2022-07-20T02:58:44.334 INFO:root:teuthology version: 1.1.1.dev383+g89d874d5 2022-07-20T02:58:44.337 DEBUG:teuthology.run:Teuthology command: teuthology --name yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436 --owner scheduled_yuriw@teuthology --description fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} --verbose -- /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/orig.config.yaml 2022-07-20T02:58:44.411 DEBUG:teuthology.report:Pushing job info to http://paddles.front.sepia.ceph.com/ 2022-07-20T02:58:44.506 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436 branch: octopus-release description: fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests} email: ceph-qa@ceph.io first_in_suite: false job_id: '6939436' kernel: kdb: true sha1: distro last_in_suite: false log-rotate: ceph-mds: 10G ceph-osd: 10G machine_type: smithi name: yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi no_nested_subset: false nuke-on-error: true openstack: - volumes: count: 4 size: 30 - machine: disk: 200 os_type: ubuntu os_version: '20.04' overrides: admin_socket: branch: octopus-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 15m rados osd op timeout: 15m mds: debug mds: 20 debug ms: 1 mds bal frag: true mds bal fragment size max: 10000 mds bal merge size: 5 mds bal split bits: 3 mds bal split size: 100 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 15m rados osd op timeout: 15m mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: debug ms: 1 debug osd: 20 osd objectstore: filestore osd op complaint time: 180 osd sloppy crc: true flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding - evicting unresponsive client - POOL_APP_NOT_ENABLED - has not responded to cap revoke by MDS for over - MDS_CLIENT_LATE_RELEASE - responding to mclientcaps sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 ceph-deploy: conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: osd default pool size: 2 osd: osd objectstore: filestore osd sloppy crc: true filestore: true fs: xfs install: ceph: flavor: default sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 workunit: branch: octopus-release sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - mon.a - mgr.y - mds.a - osd.0 - osd.1 - osd.2 - osd.3 - - mon.b - mon.c - mgr.x - mds.b - osd.4 - osd.5 - osd.6 - osd.7 - - client.0 - - client.1 - - client.2 sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 sleep_before_teardown: 0 suite: fs suite_branch: octopus-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 targets: smithi061.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkSibgklEPcg/D5L7g4g6TdZHa/6YDqG7LnZsEeMEeLuhpENnLujSS4Ssu0MC73/3Jw9mr7Aur73Ek47zSxSAIBoErrsQxODxoA92w+sJlhltq6f1QpeS2wtWSUuHhzDU8xA8GShvt1AY1y4ENdtLuAWi+rPE9LDvLc1+ACvyXdrCv27GpwiS3kCOnrg1UGRosE9GBNNOq0jJhHilZpe2C3M6GYlB6jZlqvfCilqHz0eJsONbmANoYNP+NtrzI87thP84vK2D8aJ/i6PPh3SG3V1nJ47ieoFzZK9+2U9g6WGjhrn1PVtvY/ZRvJeV3Tx3MQ2lyLTAJPghruTQGfSg1gbULvKt4nTGUBDVuclVRFmvnS1lgUht4xXGJ5aYznn9vAtZLIYIPvajuWIhmx5i92cpDt3gPz1jUhHAt7+Fg3VsnHcInPb/lx5Jf3vrde3QN/IgtxQ3BU64iFivd5870bpBJhrx0tBVKFGsdXJtPTNWyj+lWdir6efq0xWnCsHk= smithi117.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsl5o/375+L6CXtHmfxBAvSLtibmqiMrCNzqNA1GGk4T1vHRYpAmUe2zhAJT5a88q6Hf+STxMlnzG8uVzIG1wP2vF/Go6tnXbQlFRU93JrrZksVwFSiwMgDh4rtlHAPkG9ar1VpN0g23lR24Ihbe5rKr9XNFv0PsyO/vdCgWv/ZO0ef8QGCgsRZwvOxEONZBdi/gtIwggJJ00S56xmehVzmtNwSQPpYNDM46DM9HpclNhQCUDfAvkU+iDxC5Y+/wSa0im849B1JqtySSh2vHZP+stvbVp7ZxQM2NR1EYFtRjjaZ7+a8oPRG2sCDFZP7h1r4cYxMG/HMrKIbtkVih4Rn8I5yf4grweCZj3fzqtfxjhFhiHIa7vBYr82gk4+KkCkRT91knGgDCCx2HBzBtJXXK5xvXHd5qdWpY+jLttIaI3Y8V0dmc2XAgt9nLVn9X7O69liJTJpz/VMEt/JKTaO+p7Jg7uBeEJYaXclYiZDqKZhXSOMufK7aujUkWjx7nM= smithi124.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsuiOyUHlO6/aHMI2qURANqrnaYuRHNVRrnomZg6UyfYlQBIs3tgZtNBFW0lGTBQ4U4arIAW07cVXSeNEdEZBMA/JmtUF0VqhDf4NWmDV3mdVbX0xHi0mlI9Zhmihu1m6OjWx6ebiPzeKcIaOtyGcCw0AnEXZtHnJoWMjxQOgJacMPTP/dYguwDLRszpaWDO9srz5UyOwOC5X7MYMoZ/FfntMUfzIKOMvzZe4YO19DCmwxk9+osnSB/oeRa1Q7YwGZsRHrlVEp4PsriNwwjIqBm/Y0i7EiMm73VuSkaXPwP3WSxhavtfmlsXUpvqdz65hX2OQGi+2ypVCC+cXtbkIXgDJvHsliQE8LrxIvXPOupvq1zeregBXD+rU4zRpxps0UpwWbSh96HYATmx28IzSO6DmhaqnWkG6QrCJ0F4JpxqC19bD3vC2GjcQmUocIc1KfCtVwTB1ruyQEdDzml1+S1Ze7cT7IJg+kiXR9QtMrIppgn1v6ERNfsDVG6Rc0zus= smithi163.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHkJsiXzjCnt+uiHjfkqyokpi6rumJSyLxdJpLT9zTAudmlqac21M2q6eQl5edcd8WcOqdmuvonrhZtZ7uSFDJS6nA+oPZauGP8kOVAZlSQGdMWJIGtrsKM0QxiZC1BaMBAUBq7o8+lXElwrpI4CLPZf1Dy8O8QYPin76R5IE8HEt9PPELXjHWH3s/QFZqNlBqrPsv5FUzE5crQFvarhK7PQaokC4JFKHBtsn+bMLXjQitx7JrWO8K61jrHR1K7v3Jqxhkk8CXWq+nOrQB7cC5uA1LTPmdp51+aZTvEAp3BHgGJc+mbsyTGh1FtGakZOywvvcpjMmHzrWzpItkqrAgP1tf9l/cWzu/+NOmQ+B5yUSOQGrPgxxnurQy7hXA4jK7wQTi4EX2vw/03q8TnLoW3g1mN/guEZ4qEd7xSRTrC0QyEd5fd45WI/Twac2VfJDvjk1H1QIDRCAQ0gJhEBq1S0z+z2axtENKgOyPZyaiEZVcK42ZpLX7Pi+cr17XYDM= smithi179.front.sepia.ceph.com: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv/35ahvZMaabK+FQzTylGNcl63M93AHbQjzn+OqXzJAoB+Hz3xj2lTjCmLu+WhTyBd7Jm1Zy5Gtgpn7SfyVrdRz203PrDkWOILOx99XYq51g8JukNajy+7FHWjyGY/rp3pNAKWQKGuc4PfAUbVz4xDlsHjX0L7clfmVV/hG56nMEXipa/NYFS0n6Mh2NIs7gtbP/PYye4logOB1TUmUKOSpmFVe7latLRVtn7Qi0U7QaLITTxXT1eBEKSESX0h8z0WgFYLj6eKNllftzmZkl6NCjKsmUTLXkun+JHJp2QAlRd5F1BGSjA1fD8lMbuDIeeePEFgvVZRqAwkrkInHsBOafQvdDGadEoHEBKiPA95AEBr0YoS4YLE73CjX/A5SgdfXJiwS2Xdabj7hDEnIiSeFU7HgQwE7Yo/c474oCQgwf3NSye3fsnxNQ+XBS3Zj6a73Esnm7AXO5o6OlUOQhLSdDQ5NdumyzLad4uDe6zLqDJgckh4ekqimBLa9afKhk= tasks: - install: extra_packages: deb: - python3-cephfs - cephfs-shell rpm: - python3-cephfs extra_system_packages: deb: - bison - flex - libelf-dev - libssl-dev - dump - indent - libaio-dev - libtool-bin - uuid-dev - xfslibs-dev rpm: - bison - flex - elfutils-libelf-devel - openssl-devel - libacl-devel - libaio-devel - libattr-devel - libtool - libuuid-devel - xfsdump - xfsprogs - xfsprogs-devel - libaio-devel - libtool - libuuid-devel - xfsprogs-devel - ceph: null - ceph-fuse: null - cephfs_test_runner: modules: - tasks.cephfs.test_misc teuthology_branch: main teuthology_sha1: 89d874d555f56b1c1f383df72487aecec0ee6cf9 timestamp: 2022-07-19_23:28:53 tube: smithi user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.1713 2022-07-20T02:58:44.507 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa; will attempt to use it 2022-07-20T02:58:44.508 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks 2022-07-20T02:58:44.508 INFO:teuthology.run_tasks:Running task internal.check_packages... 2022-07-20T02:58:44.511 INFO:teuthology.task.internal:Checking packages... 2022-07-20T02:58:44.550 INFO:teuthology.task.internal:Checking packages for os_type 'ubuntu', flavor 'default' and ceph hash '57544da114dbbe1c44f27b2ea18e3233e4e917e0' 2022-07-20T02:58:44.550 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2022-07-20T02:58:44.552 INFO:teuthology.packaging:ref: None 2022-07-20T02:58:44.552 INFO:teuthology.packaging:tag: None 2022-07-20T02:58:44.552 INFO:teuthology.packaging:branch: octopus-release 2022-07-20T02:58:44.553 INFO:teuthology.packaging:sha1: 57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T02:58:44.553 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&ref=octopus-release 2022-07-20T02:58:44.764 INFO:teuthology.task.internal:Found packages for ceph version 15.2.16-336-g57544da1-1focal 2022-07-20T02:58:44.765 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2022-07-20T02:58:44.773 INFO:teuthology.task.internal:no buildpackages task found 2022-07-20T02:58:44.774 INFO:teuthology.run_tasks:Running task internal.save_config... 2022-07-20T02:58:44.808 INFO:teuthology.task.internal:Saving configuration 2022-07-20T02:58:44.829 INFO:teuthology.run_tasks:Running task internal.check_lock... 2022-07-20T02:58:44.857 INFO:teuthology.task.internal.check_lock:Checking locks... 2022-07-20T02:58:44.883 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi061.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:52:34.922184', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIN+f6XMPsFEmdHuMA6oGBjnrlF63ukZzHu/+7F8XLwJA'} 2022-07-20T02:58:44.910 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi117.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:52:34.921097', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIAn+otMORYZmCccppPrBcH4WuQUFZLYIYjBLm4zpc4sr'} 2022-07-20T02:58:44.967 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi124.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:52:34.920054', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFVWAeLKpoW4kj/GdciHizpRXfImtyIwbOUH1+5roupf'} 2022-07-20T02:58:44.995 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi163.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:52:34.918572', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHgc1mu/bwmrI/qvDWnWzROd43T0GJIwEVCLBzUZAbyw'} 2022-07-20T02:58:45.027 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi179.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'ubuntu', 'os_version': '20.04', 'arch': 'x86_64', 'locked': True, 'locked_since': '2022-07-20 02:52:34.923384', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIBNwAi8618Z8ebzIckROVSk3uE/QzZJpiM0+XGKlwIwb'} 2022-07-20T02:58:45.028 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2022-07-20T02:58:45.034 INFO:teuthology.task.internal:roles: ubuntu@smithi061.front.sepia.ceph.com - ['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'] 2022-07-20T02:58:45.035 INFO:teuthology.task.internal:roles: ubuntu@smithi117.front.sepia.ceph.com - ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'] 2022-07-20T02:58:45.035 INFO:teuthology.task.internal:roles: ubuntu@smithi124.front.sepia.ceph.com - ['client.0'] 2022-07-20T02:58:45.036 INFO:teuthology.task.internal:roles: ubuntu@smithi163.front.sepia.ceph.com - ['client.1'] 2022-07-20T02:58:45.036 INFO:teuthology.task.internal:roles: ubuntu@smithi179.front.sepia.ceph.com - ['client.2'] 2022-07-20T02:58:45.036 INFO:teuthology.run_tasks:Running task console_log... 2022-07-20T02:58:45.411 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f687e5d0d90>, signals=[15]) 2022-07-20T02:58:45.413 INFO:teuthology.run_tasks:Running task internal.connect... 2022-07-20T02:58:45.427 INFO:teuthology.task.internal:Opening connections... 2022-07-20T02:58:45.427 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T02:58:45.429 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:58:45.587 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T02:58:45.588 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:58:45.731 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi124.front.sepia.ceph.com 2022-07-20T02:58:45.732 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:58:45.873 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi163.front.sepia.ceph.com 2022-07-20T02:58:45.874 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:58:45.978 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi179.front.sepia.ceph.com 2022-07-20T02:58:45.980 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T02:58:46.101 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2022-07-20T02:58:46.110 DEBUG:teuthology.orchestra.run.smithi061:> uname -m 2022-07-20T02:58:46.118 INFO:teuthology.orchestra.run.smithi061.stdout:x86_64 2022-07-20T02:58:46.118 DEBUG:teuthology.orchestra.run.smithi061:> cat /etc/os-release 2022-07-20T02:58:46.175 INFO:teuthology.orchestra.run.smithi061.stdout:NAME="Ubuntu" 2022-07-20T02:58:46.176 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:58:46.176 INFO:teuthology.orchestra.run.smithi061.stdout:ID=ubuntu 2022-07-20T02:58:46.177 INFO:teuthology.orchestra.run.smithi061.stdout:ID_LIKE=debian 2022-07-20T02:58:46.177 INFO:teuthology.orchestra.run.smithi061.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:58:46.177 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_ID="20.04" 2022-07-20T02:58:46.178 INFO:teuthology.orchestra.run.smithi061.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:58:46.180 INFO:teuthology.orchestra.run.smithi061.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:58:46.181 INFO:teuthology.orchestra.run.smithi061.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:58:46.181 INFO:teuthology.orchestra.run.smithi061.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:58:46.181 INFO:teuthology.orchestra.run.smithi061.stdout:VERSION_CODENAME=focal 2022-07-20T02:58:46.181 INFO:teuthology.orchestra.run.smithi061.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:58:46.182 INFO:teuthology.lock.ops:Updating smithi061.front.sepia.ceph.com on lock server 2022-07-20T02:58:46.206 DEBUG:teuthology.orchestra.run.smithi117:> uname -m 2022-07-20T02:58:46.225 INFO:teuthology.orchestra.run.smithi117.stdout:x86_64 2022-07-20T02:58:46.226 DEBUG:teuthology.orchestra.run.smithi117:> cat /etc/os-release 2022-07-20T02:58:46.283 INFO:teuthology.orchestra.run.smithi117.stdout:NAME="Ubuntu" 2022-07-20T02:58:46.284 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:58:46.284 INFO:teuthology.orchestra.run.smithi117.stdout:ID=ubuntu 2022-07-20T02:58:46.285 INFO:teuthology.orchestra.run.smithi117.stdout:ID_LIKE=debian 2022-07-20T02:58:46.285 INFO:teuthology.orchestra.run.smithi117.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:58:46.285 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_ID="20.04" 2022-07-20T02:58:46.286 INFO:teuthology.orchestra.run.smithi117.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:58:46.286 INFO:teuthology.orchestra.run.smithi117.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:58:46.287 INFO:teuthology.orchestra.run.smithi117.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:58:46.288 INFO:teuthology.orchestra.run.smithi117.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:58:46.288 INFO:teuthology.orchestra.run.smithi117.stdout:VERSION_CODENAME=focal 2022-07-20T02:58:46.289 INFO:teuthology.orchestra.run.smithi117.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:58:46.289 INFO:teuthology.lock.ops:Updating smithi117.front.sepia.ceph.com on lock server 2022-07-20T02:58:46.318 DEBUG:teuthology.orchestra.run.smithi124:> uname -m 2022-07-20T02:58:46.335 INFO:teuthology.orchestra.run.smithi124.stdout:x86_64 2022-07-20T02:58:46.336 DEBUG:teuthology.orchestra.run.smithi124:> cat /etc/os-release 2022-07-20T02:58:46.389 INFO:teuthology.orchestra.run.smithi124.stdout:NAME="Ubuntu" 2022-07-20T02:58:46.389 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:58:46.389 INFO:teuthology.orchestra.run.smithi124.stdout:ID=ubuntu 2022-07-20T02:58:46.389 INFO:teuthology.orchestra.run.smithi124.stdout:ID_LIKE=debian 2022-07-20T02:58:46.390 INFO:teuthology.orchestra.run.smithi124.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:58:46.390 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_ID="20.04" 2022-07-20T02:58:46.390 INFO:teuthology.orchestra.run.smithi124.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:58:46.394 INFO:teuthology.orchestra.run.smithi124.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:58:46.395 INFO:teuthology.orchestra.run.smithi124.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:58:46.395 INFO:teuthology.orchestra.run.smithi124.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:58:46.395 INFO:teuthology.orchestra.run.smithi124.stdout:VERSION_CODENAME=focal 2022-07-20T02:58:46.396 INFO:teuthology.orchestra.run.smithi124.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:58:46.397 INFO:teuthology.lock.ops:Updating smithi124.front.sepia.ceph.com on lock server 2022-07-20T02:58:46.424 DEBUG:teuthology.orchestra.run.smithi163:> uname -m 2022-07-20T02:58:46.431 INFO:teuthology.orchestra.run.smithi163.stdout:x86_64 2022-07-20T02:58:46.432 DEBUG:teuthology.orchestra.run.smithi163:> cat /etc/os-release 2022-07-20T02:58:46.486 INFO:teuthology.orchestra.run.smithi163.stdout:NAME="Ubuntu" 2022-07-20T02:58:46.487 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:58:46.487 INFO:teuthology.orchestra.run.smithi163.stdout:ID=ubuntu 2022-07-20T02:58:46.487 INFO:teuthology.orchestra.run.smithi163.stdout:ID_LIKE=debian 2022-07-20T02:58:46.487 INFO:teuthology.orchestra.run.smithi163.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:58:46.488 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_ID="20.04" 2022-07-20T02:58:46.488 INFO:teuthology.orchestra.run.smithi163.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:58:46.488 INFO:teuthology.orchestra.run.smithi163.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:58:46.488 INFO:teuthology.orchestra.run.smithi163.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:58:46.489 INFO:teuthology.orchestra.run.smithi163.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:58:46.489 INFO:teuthology.orchestra.run.smithi163.stdout:VERSION_CODENAME=focal 2022-07-20T02:58:46.489 INFO:teuthology.orchestra.run.smithi163.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:58:46.490 INFO:teuthology.lock.ops:Updating smithi163.front.sepia.ceph.com on lock server 2022-07-20T02:58:46.517 DEBUG:teuthology.orchestra.run.smithi179:> uname -m 2022-07-20T02:58:46.532 INFO:teuthology.orchestra.run.smithi179.stdout:x86_64 2022-07-20T02:58:46.533 DEBUG:teuthology.orchestra.run.smithi179:> cat /etc/os-release 2022-07-20T02:58:46.583 INFO:teuthology.orchestra.run.smithi179.stdout:NAME="Ubuntu" 2022-07-20T02:58:46.584 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION="20.04.4 LTS (Focal Fossa)" 2022-07-20T02:58:46.584 INFO:teuthology.orchestra.run.smithi179.stdout:ID=ubuntu 2022-07-20T02:58:46.584 INFO:teuthology.orchestra.run.smithi179.stdout:ID_LIKE=debian 2022-07-20T02:58:46.585 INFO:teuthology.orchestra.run.smithi179.stdout:PRETTY_NAME="Ubuntu 20.04.4 LTS" 2022-07-20T02:58:46.589 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_ID="20.04" 2022-07-20T02:58:46.589 INFO:teuthology.orchestra.run.smithi179.stdout:HOME_URL="https://www.ubuntu.com/" 2022-07-20T02:58:46.590 INFO:teuthology.orchestra.run.smithi179.stdout:SUPPORT_URL="https://help.ubuntu.com/" 2022-07-20T02:58:46.590 INFO:teuthology.orchestra.run.smithi179.stdout:BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" 2022-07-20T02:58:46.591 INFO:teuthology.orchestra.run.smithi179.stdout:PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" 2022-07-20T02:58:46.592 INFO:teuthology.orchestra.run.smithi179.stdout:VERSION_CODENAME=focal 2022-07-20T02:58:46.592 INFO:teuthology.orchestra.run.smithi179.stdout:UBUNTU_CODENAME=focal 2022-07-20T02:58:46.592 INFO:teuthology.lock.ops:Updating smithi179.front.sepia.ceph.com on lock server 2022-07-20T02:58:46.611 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2022-07-20T02:58:46.625 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2022-07-20T02:58:46.631 INFO:teuthology.task.internal:Checking for old test directory... 2022-07-20T02:58:46.632 DEBUG:teuthology.orchestra.run.smithi061:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:58:46.651 DEBUG:teuthology.orchestra.run.smithi117:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:58:46.656 DEBUG:teuthology.orchestra.run.smithi124:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:58:46.662 DEBUG:teuthology.orchestra.run.smithi163:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:58:46.668 DEBUG:teuthology.orchestra.run.smithi179:> test '!' -e /home/ubuntu/cephtest 2022-07-20T02:58:46.682 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2022-07-20T02:58:46.695 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2022-07-20T02:58:46.695 DEBUG:teuthology.orchestra.run.smithi061:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:58:46.716 DEBUG:teuthology.orchestra.run.smithi117:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:58:46.723 DEBUG:teuthology.orchestra.run.smithi124:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:58:46.727 DEBUG:teuthology.orchestra.run.smithi163:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:58:46.731 DEBUG:teuthology.orchestra.run.smithi179:> test -z $(ls -A /var/lib/ceph) 2022-07-20T02:58:46.758 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2022-07-20T02:58:46.930 INFO:teuthology.run_tasks:Running task kernel... 2022-07-20T02:58:46.949 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2022-07-20T02:58:46.950 INFO:teuthology.task.kernel:config {'mon.a': {'kdb': True, 'sha1': 'distro'}, 'mon.b': {'kdb': True, 'sha1': 'distro'}, 'client.0': {'kdb': True, 'sha1': 'distro'}, 'client.1': {'kdb': True, 'sha1': 'distro'}, 'client.2': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2022-07-20T02:58:46.951 DEBUG:teuthology.orchestra.run.smithi061:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:58:46.952 DEBUG:teuthology.orchestra.run.smithi117:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:58:46.954 DEBUG:teuthology.orchestra.run.smithi124:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:58:46.954 DEBUG:teuthology.orchestra.run.smithi163:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:58:46.955 DEBUG:teuthology.orchestra.run.smithi179:> test -f /run/.containerenv -o -f /.dockerenv 2022-07-20T02:58:46.968 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:58:46.969 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:58:46.970 DEBUG:teuthology.orchestra.run.smithi117:> uname -r 2022-07-20T02:58:46.971 DEBUG:teuthology.orchestra.run.smithi061:> uname -r 2022-07-20T02:58:46.973 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:58:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:58:46.974 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T02:58:46.981 DEBUG:teuthology.orchestra.run.smithi179:> uname -r 2022-07-20T02:58:46.983 DEBUG:teuthology.orchestra.run.smithi124:> uname -r 2022-07-20T02:58:46.984 DEBUG:teuthology.orchestra.run.smithi163:> uname -r 2022-07-20T02:58:46.988 INFO:teuthology.orchestra.run.smithi117.stdout:5.4.0-122-generic 2022-07-20T02:58:46.988 INFO:teuthology.task.kernel:Running kernel on smithi117: 5.4.0-122-generic 2022-07-20T02:58:46.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get clean 2022-07-20T02:58:46.992 INFO:teuthology.orchestra.run.smithi124.stdout:5.4.0-122-generic 2022-07-20T02:58:46.992 INFO:teuthology.task.kernel:Running kernel on smithi124: 5.4.0-122-generic 2022-07-20T02:58:46.993 DEBUG:teuthology.orchestra.run.smithi124:> sudo apt-get clean 2022-07-20T02:58:47.036 INFO:teuthology.orchestra.run.smithi163.stdout:5.4.0-122-generic 2022-07-20T02:58:47.037 INFO:teuthology.orchestra.run.smithi061.stdout:5.4.0-122-generic 2022-07-20T02:58:47.037 INFO:teuthology.task.kernel:Running kernel on smithi163: 5.4.0-122-generic 2022-07-20T02:58:47.038 DEBUG:teuthology.orchestra.run.smithi163:> sudo apt-get clean 2022-07-20T02:58:47.039 INFO:teuthology.task.kernel:Running kernel on smithi061: 5.4.0-122-generic 2022-07-20T02:58:47.040 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get clean 2022-07-20T02:58:47.045 INFO:teuthology.orchestra.run.smithi179.stdout:5.4.0-122-generic 2022-07-20T02:58:47.046 INFO:teuthology.task.kernel:Running kernel on smithi179: 5.4.0-122-generic 2022-07-20T02:58:47.047 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get clean 2022-07-20T02:58:47.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2022-07-20T02:58:47.171 DEBUG:teuthology.orchestra.run.smithi124:> sudo apt-get update 2022-07-20T02:58:47.174 DEBUG:teuthology.orchestra.run.smithi163:> sudo apt-get update 2022-07-20T02:58:47.186 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2022-07-20T02:58:47.225 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2022-07-20T02:58:47.317 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:58:47.329 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:58:47.356 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:58:47.363 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:58:47.392 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:58:47.426 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:58:47.449 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:58:47.462 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:58:47.463 INFO:teuthology.orchestra.run.smithi124.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:58:47.465 INFO:teuthology.orchestra.run.smithi124.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:58:47.479 INFO:teuthology.orchestra.run.smithi163.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:58:47.480 INFO:teuthology.orchestra.run.smithi163.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:58:47.525 INFO:teuthology.orchestra.run.smithi163.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:58:47.528 INFO:teuthology.orchestra.run.smithi124.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:58:47.551 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T02:58:47.552 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T02:58:47.585 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T02:58:47.689 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:58:47.728 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:58:47.790 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:58:47.804 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:58:47.804 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:58:47.813 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:58:47.814 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:58:47.821 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:58:47.826 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:58:47.828 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:58:47.831 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:58:47.840 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:58:47.851 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:58:47.857 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:58:47.859 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:58:47.865 INFO:teuthology.orchestra.run.smithi124.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:58:47.869 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:58:47.870 INFO:teuthology.orchestra.run.smithi163.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:58:47.876 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:58:47.879 INFO:teuthology.orchestra.run.smithi179.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:58:47.920 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T02:58:47.948 INFO:teuthology.orchestra.run.smithi163.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:58:47.957 INFO:teuthology.orchestra.run.smithi124.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:58:48.006 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:58:48.057 INFO:teuthology.orchestra.run.smithi179.stdout:Get:14 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:58:48.074 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:58:48.094 INFO:teuthology.orchestra.run.smithi163.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:58:48.100 INFO:teuthology.orchestra.run.smithi124.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:58:48.221 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:58:48.254 INFO:teuthology.orchestra.run.smithi124.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:58:48.284 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:58:48.338 INFO:teuthology.orchestra.run.smithi179.stdout:Get:15 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:58:48.374 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main amd64 Packages [1,628 kB] 2022-07-20T02:58:48.389 INFO:teuthology.orchestra.run.smithi163.stdout:Get:7 http://security.ubuntu.com/ubuntu focal-security/main i386 Packages [471 kB] 2022-07-20T02:58:48.390 INFO:teuthology.orchestra.run.smithi163.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T02:58:48.405 INFO:teuthology.orchestra.run.smithi163.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:58:48.412 INFO:teuthology.orchestra.run.smithi163.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:58:48.429 INFO:teuthology.orchestra.run.smithi124.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:58:48.436 INFO:teuthology.orchestra.run.smithi163.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:58:48.442 INFO:teuthology.orchestra.run.smithi124.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:58:48.462 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:58:48.471 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:58:48.477 INFO:teuthology.orchestra.run.smithi124.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:58:48.478 INFO:teuthology.orchestra.run.smithi163.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:58:48.489 INFO:teuthology.orchestra.run.smithi163.stdout:Get:13 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:58:48.501 INFO:teuthology.orchestra.run.smithi124.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:58:48.507 INFO:teuthology.orchestra.run.smithi124.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:58:48.508 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:58:48.511 INFO:teuthology.orchestra.run.smithi124.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:58:48.512 INFO:teuthology.orchestra.run.smithi179.stdout:Get:16 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:58:48.526 INFO:teuthology.orchestra.run.smithi179.stdout:Get:17 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:58:48.529 INFO:teuthology.orchestra.run.smithi124.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:58:48.530 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:58:48.534 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:58:48.539 INFO:teuthology.orchestra.run.smithi124.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:58:48.547 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://security.ubuntu.com/ubuntu focal-security/main Translation-en [277 kB] 2022-07-20T02:58:48.552 INFO:teuthology.orchestra.run.smithi163.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:58:48.553 INFO:teuthology.orchestra.run.smithi179.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:58:48.561 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://security.ubuntu.com/ubuntu focal-security/universe i386 Packages [556 kB] 2022-07-20T02:58:48.565 INFO:teuthology.orchestra.run.smithi163.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:58:48.585 INFO:teuthology.orchestra.run.smithi179.stdout:Get:19 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:58:48.589 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/universe amd64 Packages [710 kB] 2022-07-20T02:58:48.590 INFO:teuthology.orchestra.run.smithi179.stdout:Get:20 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:58:48.594 INFO:teuthology.orchestra.run.smithi124.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:58:48.599 INFO:teuthology.orchestra.run.smithi124.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:58:48.606 INFO:teuthology.orchestra.run.smithi163.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:58:48.611 INFO:teuthology.orchestra.run.smithi163.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:58:48.621 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://security.ubuntu.com/ubuntu focal-security/universe Translation-en [128 kB] 2022-07-20T02:58:48.624 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://security.ubuntu.com/ubuntu focal-security/universe amd64 c-n-f Metadata [14.6 kB] 2022-07-20T02:58:48.624 INFO:teuthology.orchestra.run.smithi124.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:58:48.631 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T02:58:48.638 INFO:teuthology.orchestra.run.smithi163.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:58:48.642 INFO:teuthology.orchestra.run.smithi124.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:58:48.647 INFO:teuthology.orchestra.run.smithi124.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:58:48.649 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T02:58:48.652 INFO:teuthology.orchestra.run.smithi163.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:58:48.656 INFO:teuthology.orchestra.run.smithi163.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:58:48.658 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/restricted amd64 Packages [1,178 kB] 2022-07-20T02:58:48.718 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal-updates/restricted Translation-en [167 kB] 2022-07-20T02:58:48.719 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal-updates/universe i386 Packages [677 kB] 2022-07-20T02:58:48.737 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 Packages [924 kB] 2022-07-20T02:58:48.761 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal-updates/universe Translation-en [208 kB] 2022-07-20T02:58:48.766 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 c-n-f Metadata [20.9 kB] 2022-07-20T02:58:51.144 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 10.3 MB in 2s (6,224 kB/s) 2022-07-20T02:58:51.170 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 10.3 MB in 2s (6,127 kB/s) 2022-07-20T02:58:51.617 INFO:teuthology.orchestra.run.smithi163.stdout:Fetched 10.3 MB in 2s (4,752 kB/s) 2022-07-20T02:58:51.812 INFO:teuthology.orchestra.run.smithi124.stdout:Fetched 10.3 MB in 2s (4,569 kB/s) 2022-07-20T02:58:51.980 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 10.3 MB in 2s (4,390 kB/s) 2022-07-20T02:58:52.464 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-07-20T02:58:52.487 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:58:52.494 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-07-20T02:58:52.518 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:58:52.555 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-07-20T02:58:52.584 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-07-20T02:58:52.732 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-07-20T02:58:52.733 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-07-20T02:58:52.771 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-07-20T02:58:52.771 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-07-20T02:58:52.934 INFO:teuthology.orchestra.run.smithi117.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:58:52.934 INFO:teuthology.orchestra.run.smithi117.stdout:linux-image-generic set to manually installed. 2022-07-20T02:58:52.935 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:58:52.937 DEBUG:teuthology.orchestra.run.smithi117:> dpkg -s linux-image-generic 2022-07-20T02:58:52.959 INFO:teuthology.orchestra.run.smithi117.stdout:Package: linux-image-generic 2022-07-20T02:58:52.959 INFO:teuthology.orchestra.run.smithi117.stdout:Status: install ok installed 2022-07-20T02:58:52.960 INFO:teuthology.orchestra.run.smithi117.stdout:Priority: optional 2022-07-20T02:58:52.960 INFO:teuthology.orchestra.run.smithi117.stdout:Section: kernel 2022-07-20T02:58:52.961 INFO:teuthology.orchestra.run.smithi117.stdout:Installed-Size: 19 2022-07-20T02:58:52.961 INFO:teuthology.orchestra.run.smithi117.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:58:52.961 INFO:teuthology.orchestra.run.smithi117.stdout:Architecture: amd64 2022-07-20T02:58:52.962 INFO:teuthology.orchestra.run.smithi117.stdout:Source: linux-meta 2022-07-20T02:58:52.963 INFO:teuthology.orchestra.run.smithi117.stdout:Version: 5.4.0.122.123 2022-07-20T02:58:52.963 INFO:teuthology.orchestra.run.smithi117.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:58:52.964 INFO:teuthology.orchestra.run.smithi117.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:58:52.964 INFO:teuthology.orchestra.run.smithi117.stdout:Recommends: thermald 2022-07-20T02:58:52.965 INFO:teuthology.orchestra.run.smithi117.stdout:Description: Generic Linux kernel image 2022-07-20T02:58:52.965 INFO:teuthology.orchestra.run.smithi117.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:58:52.966 INFO:teuthology.orchestra.run.smithi117.stdout: available. 2022-07-20T02:58:52.966 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:58:52.967 INFO:teuthology.task.kernel:Enabling kdb on mon.b... 2022-07-20T02:58:52.967 DEBUG:teuthology.orchestra.run.smithi117:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:58:52.979 INFO:teuthology.orchestra.run.smithi179.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:58:52.979 INFO:teuthology.orchestra.run.smithi179.stdout:linux-image-generic set to manually installed. 2022-07-20T02:58:52.980 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:58:52.981 DEBUG:teuthology.orchestra.run.smithi179:> dpkg -s linux-image-generic 2022-07-20T02:58:52.982 INFO:teuthology.orchestra.run.smithi163.stdout:Reading package lists... 2022-07-20T02:58:52.988 INFO:teuthology.orchestra.run.smithi117.stdout:ttyS1 2022-07-20T02:58:53.001 INFO:teuthology.orchestra.run.smithi179.stdout:Package: linux-image-generic 2022-07-20T02:58:53.002 INFO:teuthology.orchestra.run.smithi179.stdout:Status: install ok installed 2022-07-20T02:58:53.002 INFO:teuthology.orchestra.run.smithi179.stdout:Priority: optional 2022-07-20T02:58:53.002 INFO:teuthology.orchestra.run.smithi179.stdout:Section: kernel 2022-07-20T02:58:53.003 INFO:teuthology.orchestra.run.smithi179.stdout:Installed-Size: 19 2022-07-20T02:58:53.003 INFO:teuthology.orchestra.run.smithi179.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:58:53.003 INFO:teuthology.orchestra.run.smithi179.stdout:Architecture: amd64 2022-07-20T02:58:53.003 INFO:teuthology.orchestra.run.smithi179.stdout:Source: linux-meta 2022-07-20T02:58:53.003 INFO:teuthology.orchestra.run.smithi179.stdout:Version: 5.4.0.122.123 2022-07-20T02:58:53.004 INFO:teuthology.orchestra.run.smithi179.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:58:53.004 INFO:teuthology.orchestra.run.smithi179.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:58:53.004 INFO:teuthology.orchestra.run.smithi179.stdout:Recommends: thermald 2022-07-20T02:58:53.004 INFO:teuthology.orchestra.run.smithi179.stdout:Description: Generic Linux kernel image 2022-07-20T02:58:53.005 INFO:teuthology.orchestra.run.smithi179.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:58:53.005 INFO:teuthology.orchestra.run.smithi179.stdout: available. 2022-07-20T02:58:53.005 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:58:53.005 INFO:teuthology.task.kernel:Enabling kdb on client.2... 2022-07-20T02:58:53.006 DEBUG:teuthology.orchestra.run.smithi179:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:58:53.007 DEBUG:teuthology.orchestra.run.smithi163:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:58:53.015 DEBUG:teuthology.parallel:result is None 2022-07-20T02:58:53.032 INFO:teuthology.orchestra.run.smithi179.stdout:ttyS1 2022-07-20T02:58:53.067 DEBUG:teuthology.parallel:result is None 2022-07-20T02:58:53.076 INFO:teuthology.orchestra.run.smithi163.stdout:Reading package lists... 2022-07-20T02:58:53.087 INFO:teuthology.orchestra.run.smithi124.stdout:Reading package lists... 2022-07-20T02:58:53.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:58:53.183 INFO:teuthology.orchestra.run.smithi124.stdout:Reading package lists... 2022-07-20T02:58:53.254 INFO:teuthology.orchestra.run.smithi163.stdout:Building dependency tree... 2022-07-20T02:58:53.255 INFO:teuthology.orchestra.run.smithi163.stdout:Reading state information... 2022-07-20T02:58:53.282 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-20T02:58:53.307 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y install linux-image-generic 2022-07-20T02:58:53.364 INFO:teuthology.orchestra.run.smithi124.stdout:Building dependency tree... 2022-07-20T02:58:53.365 INFO:teuthology.orchestra.run.smithi124.stdout:Reading state information... 2022-07-20T02:58:53.381 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-20T02:58:53.455 INFO:teuthology.orchestra.run.smithi163.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:58:53.456 INFO:teuthology.orchestra.run.smithi163.stdout:linux-image-generic set to manually installed. 2022-07-20T02:58:53.456 INFO:teuthology.orchestra.run.smithi163.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:58:53.457 DEBUG:teuthology.orchestra.run.smithi163:> dpkg -s linux-image-generic 2022-07-20T02:58:53.478 INFO:teuthology.orchestra.run.smithi163.stdout:Package: linux-image-generic 2022-07-20T02:58:53.479 INFO:teuthology.orchestra.run.smithi163.stdout:Status: install ok installed 2022-07-20T02:58:53.479 INFO:teuthology.orchestra.run.smithi163.stdout:Priority: optional 2022-07-20T02:58:53.479 INFO:teuthology.orchestra.run.smithi163.stdout:Section: kernel 2022-07-20T02:58:53.479 INFO:teuthology.orchestra.run.smithi163.stdout:Installed-Size: 19 2022-07-20T02:58:53.480 INFO:teuthology.orchestra.run.smithi163.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:58:53.480 INFO:teuthology.orchestra.run.smithi163.stdout:Architecture: amd64 2022-07-20T02:58:53.480 INFO:teuthology.orchestra.run.smithi163.stdout:Source: linux-meta 2022-07-20T02:58:53.481 INFO:teuthology.orchestra.run.smithi163.stdout:Version: 5.4.0.122.123 2022-07-20T02:58:53.481 INFO:teuthology.orchestra.run.smithi163.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:58:53.481 INFO:teuthology.orchestra.run.smithi163.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:58:53.481 INFO:teuthology.orchestra.run.smithi163.stdout:Recommends: thermald 2022-07-20T02:58:53.482 INFO:teuthology.orchestra.run.smithi163.stdout:Description: Generic Linux kernel image 2022-07-20T02:58:53.482 INFO:teuthology.orchestra.run.smithi163.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:58:53.482 INFO:teuthology.orchestra.run.smithi163.stdout: available. 2022-07-20T02:58:53.483 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:58:53.486 INFO:teuthology.task.kernel:Enabling kdb on client.1... 2022-07-20T02:58:53.486 DEBUG:teuthology.orchestra.run.smithi163:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:58:53.551 INFO:teuthology.orchestra.run.smithi163.stdout:ttyS1 2022-07-20T02:58:53.562 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-20T02:58:53.563 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-20T02:58:53.580 INFO:teuthology.orchestra.run.smithi124.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:58:53.580 INFO:teuthology.orchestra.run.smithi124.stdout:linux-image-generic set to manually installed. 2022-07-20T02:58:53.580 INFO:teuthology.orchestra.run.smithi124.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:58:53.583 DEBUG:teuthology.orchestra.run.smithi124:> dpkg -s linux-image-generic 2022-07-20T02:58:53.586 DEBUG:teuthology.parallel:result is None 2022-07-20T02:58:53.604 INFO:teuthology.orchestra.run.smithi124.stdout:Package: linux-image-generic 2022-07-20T02:58:53.605 INFO:teuthology.orchestra.run.smithi124.stdout:Status: install ok installed 2022-07-20T02:58:53.605 INFO:teuthology.orchestra.run.smithi124.stdout:Priority: optional 2022-07-20T02:58:53.605 INFO:teuthology.orchestra.run.smithi124.stdout:Section: kernel 2022-07-20T02:58:53.606 INFO:teuthology.orchestra.run.smithi124.stdout:Installed-Size: 19 2022-07-20T02:58:53.606 INFO:teuthology.orchestra.run.smithi124.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:58:53.606 INFO:teuthology.orchestra.run.smithi124.stdout:Architecture: amd64 2022-07-20T02:58:53.607 INFO:teuthology.orchestra.run.smithi124.stdout:Source: linux-meta 2022-07-20T02:58:53.607 INFO:teuthology.orchestra.run.smithi124.stdout:Version: 5.4.0.122.123 2022-07-20T02:58:53.607 INFO:teuthology.orchestra.run.smithi124.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:58:53.608 INFO:teuthology.orchestra.run.smithi124.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:58:53.608 INFO:teuthology.orchestra.run.smithi124.stdout:Recommends: thermald 2022-07-20T02:58:53.608 INFO:teuthology.orchestra.run.smithi124.stdout:Description: Generic Linux kernel image 2022-07-20T02:58:53.609 INFO:teuthology.orchestra.run.smithi124.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:58:53.609 INFO:teuthology.orchestra.run.smithi124.stdout: available. 2022-07-20T02:58:53.610 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:58:53.610 INFO:teuthology.task.kernel:Enabling kdb on client.0... 2022-07-20T02:58:53.610 DEBUG:teuthology.orchestra.run.smithi124:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:58:53.676 INFO:teuthology.orchestra.run.smithi124.stdout:ttyS1 2022-07-20T02:58:53.691 DEBUG:teuthology.parallel:result is None 2022-07-20T02:58:53.784 INFO:teuthology.orchestra.run.smithi061.stdout:linux-image-generic is already the newest version (5.4.0.122.123). 2022-07-20T02:58:53.784 INFO:teuthology.orchestra.run.smithi061.stdout:linux-image-generic set to manually installed. 2022-07-20T02:58:53.784 INFO:teuthology.orchestra.run.smithi061.stdout:0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T02:58:53.786 DEBUG:teuthology.orchestra.run.smithi061:> dpkg -s linux-image-generic 2022-07-20T02:58:53.807 INFO:teuthology.orchestra.run.smithi061.stdout:Package: linux-image-generic 2022-07-20T02:58:53.807 INFO:teuthology.orchestra.run.smithi061.stdout:Status: install ok installed 2022-07-20T02:58:53.808 INFO:teuthology.orchestra.run.smithi061.stdout:Priority: optional 2022-07-20T02:58:53.808 INFO:teuthology.orchestra.run.smithi061.stdout:Section: kernel 2022-07-20T02:58:53.809 INFO:teuthology.orchestra.run.smithi061.stdout:Installed-Size: 19 2022-07-20T02:58:53.809 INFO:teuthology.orchestra.run.smithi061.stdout:Maintainer: Ubuntu Kernel Team 2022-07-20T02:58:53.809 INFO:teuthology.orchestra.run.smithi061.stdout:Architecture: amd64 2022-07-20T02:58:53.809 INFO:teuthology.orchestra.run.smithi061.stdout:Source: linux-meta 2022-07-20T02:58:53.810 INFO:teuthology.orchestra.run.smithi061.stdout:Version: 5.4.0.122.123 2022-07-20T02:58:53.810 INFO:teuthology.orchestra.run.smithi061.stdout:Provides: virtualbox-guest-modules (= 6.1.32-dfsg-1~ubuntu1.20.04.1), wireguard-modules (= 1.0.20201112-1~20.04.1), zfs-modules (= 0.8.3-1ubuntu12.14) 2022-07-20T02:58:53.811 INFO:teuthology.orchestra.run.smithi061.stdout:Depends: linux-image-5.4.0-122-generic, linux-modules-extra-5.4.0-122-generic, linux-firmware, intel-microcode, amd64-microcode 2022-07-20T02:58:53.811 INFO:teuthology.orchestra.run.smithi061.stdout:Recommends: thermald 2022-07-20T02:58:53.811 INFO:teuthology.orchestra.run.smithi061.stdout:Description: Generic Linux kernel image 2022-07-20T02:58:53.812 INFO:teuthology.orchestra.run.smithi061.stdout: This package will always depend on the latest generic kernel image 2022-07-20T02:58:53.812 INFO:teuthology.orchestra.run.smithi061.stdout: available. 2022-07-20T02:58:53.812 INFO:teuthology.task.kernel:Newest distro kernel installed and running 2022-07-20T02:58:53.813 INFO:teuthology.task.kernel:Enabling kdb on mon.a... 2022-07-20T02:58:53.813 DEBUG:teuthology.orchestra.run.smithi061:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2022-07-20T02:58:53.830 INFO:teuthology.orchestra.run.smithi061.stdout:ttyS1 2022-07-20T02:58:53.863 DEBUG:teuthology.parallel:result is None 2022-07-20T02:58:53.864 INFO:teuthology.run_tasks:Running task internal.base... 2022-07-20T02:58:53.871 INFO:teuthology.task.internal:Creating test directory... 2022-07-20T02:58:53.873 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:58:53.876 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:58:53.879 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:58:53.883 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:58:53.886 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2022-07-20T02:58:53.894 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2022-07-20T02:58:53.904 INFO:teuthology.run_tasks:Running task internal.archive... 2022-07-20T02:58:53.926 INFO:teuthology.task.internal:Creating archive directory... 2022-07-20T02:58:53.927 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:58:53.930 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:58:53.934 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:58:53.937 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:58:53.940 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2022-07-20T02:58:53.957 INFO:teuthology.run_tasks:Running task internal.coredump... 2022-07-20T02:58:53.975 INFO:teuthology.task.internal:Enabling coredump saving... 2022-07-20T02:58:53.976 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:58:53.978 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:58:53.983 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:58:53.986 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:58:53.990 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2022-07-20T02:58:54.001 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.002 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.008 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.011 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.012 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.014 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.018 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.021 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.025 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.031 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2022-07-20T02:58:54.033 INFO:teuthology.run_tasks:Running task internal.sudo... 2022-07-20T02:58:54.040 INFO:teuthology.task.internal:Configuring sudo... 2022-07-20T02:58:54.040 DEBUG:teuthology.orchestra.run.smithi061:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:58:54.058 DEBUG:teuthology.orchestra.run.smithi117:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:58:54.062 DEBUG:teuthology.orchestra.run.smithi124:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:58:54.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:58:54.073 DEBUG:teuthology.orchestra.run.smithi179:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2022-07-20T02:58:54.093 INFO:teuthology.run_tasks:Running task internal.syslog... 2022-07-20T02:58:54.108 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2022-07-20T02:58:54.109 DEBUG:teuthology.orchestra.run.smithi061:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:58:54.113 DEBUG:teuthology.orchestra.run.smithi117:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:58:54.119 DEBUG:teuthology.orchestra.run.smithi124:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:58:54.129 DEBUG:teuthology.orchestra.run.smithi163:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:58:54.132 DEBUG:teuthology.orchestra.run.smithi179:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2022-07-20T02:58:54.142 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:58:54.176 DEBUG:teuthology.orchestra.run.smithi061:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:58:54.190 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T02:58:54.191 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:58:54.257 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:58:54.267 DEBUG:teuthology.orchestra.run.smithi117:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:58:54.276 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T02:58:54.279 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:58:54.300 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:58:54.308 DEBUG:teuthology.orchestra.run.smithi124:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:58:54.316 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T02:58:54.317 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:58:54.385 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:58:54.396 DEBUG:teuthology.orchestra.run.smithi163:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:58:54.407 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T02:58:54.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:58:54.463 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2022-07-20T02:58:54.474 DEBUG:teuthology.orchestra.run.smithi179:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2022-07-20T02:58:54.529 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T02:58:54.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2022-07-20T02:58:54.613 DEBUG:teuthology.orchestra.run.smithi061:> sudo service rsyslog restart 2022-07-20T02:58:54.633 DEBUG:teuthology.orchestra.run.smithi117:> sudo service rsyslog restart 2022-07-20T02:58:54.636 DEBUG:teuthology.orchestra.run.smithi124:> sudo service rsyslog restart 2022-07-20T02:58:54.641 DEBUG:teuthology.orchestra.run.smithi163:> sudo service rsyslog restart 2022-07-20T02:58:54.652 DEBUG:teuthology.orchestra.run.smithi179:> sudo service rsyslog restart 2022-07-20T02:58:54.720 INFO:teuthology.run_tasks:Running task internal.timer... 2022-07-20T02:58:54.819 INFO:teuthology.task.internal:Starting timer... 2022-07-20T02:58:54.820 INFO:teuthology.run_tasks:Running task pcp... 2022-07-20T02:58:54.833 INFO:teuthology.run_tasks:Running task selinux... 2022-07-20T02:58:54.878 DEBUG:teuthology.task.selinux:Excluding smithi061: OS 'ubuntu' does not support SELinux 2022-07-20T02:58:54.879 DEBUG:teuthology.task.selinux:Excluding smithi117: OS 'ubuntu' does not support SELinux 2022-07-20T02:58:54.879 DEBUG:teuthology.task.selinux:Excluding smithi124: OS 'ubuntu' does not support SELinux 2022-07-20T02:58:54.880 DEBUG:teuthology.task.selinux:Excluding smithi163: OS 'ubuntu' does not support SELinux 2022-07-20T02:58:54.881 DEBUG:teuthology.task.selinux:Excluding smithi179: OS 'ubuntu' does not support SELinux 2022-07-20T02:58:54.881 DEBUG:teuthology.task.selinux:Getting current SELinux state 2022-07-20T02:58:54.882 DEBUG:teuthology.task.selinux:Existing SELinux modes: {} 2022-07-20T02:58:54.882 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2022-07-20T02:58:54.883 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2022-07-20T02:58:54.932 DEBUG:teuthology.repo_utils:Setting repo remote to git://git.ceph.com/git/ceph-cm-ansible.git 2022-07-20T02:58:54.944 INFO:teuthology.repo_utils:Fetching main from origin 2022-07-20T02:58:54.999 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main to origin/main 2022-07-20T02:58:55.031 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2022-07-20T02:58:55.033 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi061.front.sepia.ceph.com,smithi117.front.sepia.ceph.com,smithi124.front.sepia.ceph.com,smithi163.front.sepia.ceph.com,smithi179.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_git_ceph-cm-ansible_main/cephlab.yml 2022-07-20T03:04:09.357 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi061.front.sepia.ceph.com'), Remote(name='ubuntu@smithi117.front.sepia.ceph.com'), Remote(name='ubuntu@smithi124.front.sepia.ceph.com'), Remote(name='ubuntu@smithi163.front.sepia.ceph.com'), Remote(name='ubuntu@smithi179.front.sepia.ceph.com')] 2022-07-20T03:04:09.363 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi061.front.sepia.ceph.com' 2022-07-20T03:04:09.367 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi061.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T03:04:09.488 DEBUG:teuthology.orchestra.run.smithi061:> true 2022-07-20T03:04:09.608 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi117.front.sepia.ceph.com' 2022-07-20T03:04:09.609 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi117.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T03:04:09.725 DEBUG:teuthology.orchestra.run.smithi117:> true 2022-07-20T03:04:09.847 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi124.front.sepia.ceph.com' 2022-07-20T03:04:09.849 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi124.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T03:04:09.958 DEBUG:teuthology.orchestra.run.smithi124:> true 2022-07-20T03:04:10.080 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi163.front.sepia.ceph.com' 2022-07-20T03:04:10.081 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi163.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T03:04:10.195 DEBUG:teuthology.orchestra.run.smithi163:> true 2022-07-20T03:04:10.320 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi179.front.sepia.ceph.com' 2022-07-20T03:04:10.322 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi179.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2022-07-20T03:04:10.428 DEBUG:teuthology.orchestra.run.smithi179:> true 2022-07-20T03:04:10.557 INFO:teuthology.run_tasks:Running task clock... 2022-07-20T03:04:10.571 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2022-07-20T03:04:10.572 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T03:04:10.574 DEBUG:teuthology.orchestra.run.smithi061:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:04:10.577 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T03:04:10.578 DEBUG:teuthology.orchestra.run.smithi117:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:04:10.581 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T03:04:10.582 DEBUG:teuthology.orchestra.run.smithi124:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:04:10.585 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T03:04:10.586 DEBUG:teuthology.orchestra.run.smithi163:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:04:10.588 INFO:teuthology.orchestra.run:Running command with timeout 360 2022-07-20T03:04:10.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T03:04:10.611 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T03:04:10.611 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Command line: ntpd -gq 2022-07-20T03:04:10.612 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: proto: precision = 0.098 usec (-23) 2022-07-20T03:04:10.613 INFO:teuthology.orchestra.run.smithi061.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.613 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.614 INFO:teuthology.orchestra.run.smithi061.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.615 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.616 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T03:04:10.616 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T03:04:10.616 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T03:04:10.616 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listen normally on 3 enp3s0f1 172.21.15.61:123 2022-07-20T03:04:10.618 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listen normally on 4 lo [::1]:123 2022-07-20T03:04:10.618 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:feb6:f8b9%5]:123 2022-07-20T03:04:10.618 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:10 ntpd[18101]: Listening on routing socket on fd #22 for interface updates 2022-07-20T03:04:10.621 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T03:04:10.622 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Command line: ntpd -gq 2022-07-20T03:04:10.623 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T03:04:10.623 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Command line: ntpd -gq 2022-07-20T03:04:10.624 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: proto: precision = 0.245 usec (-22) 2022-07-20T03:04:10.624 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T03:04:10.625 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Command line: ntpd -gq 2022-07-20T03:04:10.625 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: proto: precision = 0.123 usec (-23) 2022-07-20T03:04:10.626 INFO:teuthology.orchestra.run.smithi117.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.627 INFO:teuthology.orchestra.run.smithi124.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.628 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.629 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.630 INFO:teuthology.orchestra.run.smithi117.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.630 INFO:teuthology.orchestra.run.smithi124.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.633 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: proto: precision = 0.103 usec (-23) 2022-07-20T03:04:10.634 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: ntpd 4.2.8p12@1.3728-o (1): Starting 2022-07-20T03:04:10.635 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Command line: ntpd -gq 2022-07-20T03:04:10.635 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: proto: precision = 0.072 usec (-24) 2022-07-20T03:04:10.636 INFO:teuthology.orchestra.run.smithi163.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.637 INFO:teuthology.orchestra.run.smithi179.stderr:restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.638 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.639 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T03:04:10.639 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T03:04:10.640 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T03:04:10.640 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listen normally on 3 enp3s0f1 172.21.15.117:123 2022-07-20T03:04:10.641 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listen normally on 4 lo [::1]:123 2022-07-20T03:04:10.641 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:4463%5]:123 2022-07-20T03:04:10.641 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:10 ntpd[18046]: Listening on routing socket on fd #22 for interface updates 2022-07-20T03:04:10.641 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.642 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T03:04:10.642 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T03:04:10.642 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T03:04:10.642 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listen normally on 3 enp3s0f1 172.21.15.124:123 2022-07-20T03:04:10.643 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listen normally on 4 lo [::1]:123 2022-07-20T03:04:10.643 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe8f:d5c5%5]:123 2022-07-20T03:04:10.643 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:10 ntpd[18195]: Listening on routing socket on fd #22 for interface updates 2022-07-20T03:04:10.644 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.644 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: restrict 0.0.0.0: KOD does nothing without LIMITED. 2022-07-20T03:04:10.645 INFO:teuthology.orchestra.run.smithi163.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.645 INFO:teuthology.orchestra.run.smithi179.stderr:restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.650 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.651 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T03:04:10.651 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T03:04:10.652 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T03:04:10.652 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listen normally on 3 enp3s0f1 172.21.15.163:123 2022-07-20T03:04:10.653 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listen normally on 4 lo [::1]:123 2022-07-20T03:04:10.653 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:7d5b%5]:123 2022-07-20T03:04:10.654 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:10 ntpd[18056]: Listening on routing socket on fd #22 for interface updates 2022-07-20T03:04:10.654 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: restrict ::: KOD does nothing without LIMITED. 2022-07-20T03:04:10.655 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listen and drop on 0 v6wildcard [::]:123 2022-07-20T03:04:10.655 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listen and drop on 1 v4wildcard 0.0.0.0:123 2022-07-20T03:04:10.656 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listen normally on 2 lo 127.0.0.1:123 2022-07-20T03:04:10.656 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listen normally on 3 enp3s0f1 172.21.15.179:123 2022-07-20T03:04:10.657 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listen normally on 4 lo [::1]:123 2022-07-20T03:04:10.657 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listen normally on 5 enp3s0f1 [fe80::ec4:7aff:fe88:725b%5]:123 2022-07-20T03:04:10.658 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:10 ntpd[18044]: Listening on routing socket on fd #22 for interface updates 2022-07-20T03:04:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:11 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:11 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:11 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:11 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:11 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:11 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:11.616 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:11 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.616 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:11 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.617 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:11 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.617 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:11 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.617 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:11 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:11.618 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:11 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:11.620 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:11 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.621 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:11 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.621 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:11 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.622 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:11 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.622 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:11 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:11.623 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:11 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:11.626 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.626 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.626 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:11.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:11.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:11.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:11.629 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:11.629 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:11 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:12.619 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:12 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:12.620 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:12 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:12.620 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:12 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:12.620 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:12 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:12.620 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:12 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:12.621 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:12 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:12.625 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:12.625 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:12.625 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:12.626 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:12.626 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:12.626 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:12.626 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:12.627 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:12.627 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:12.627 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:12.627 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:12.628 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:12 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:13.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.614 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.614 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.614 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:13.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:13 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:13.619 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.619 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.619 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.620 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.620 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:13.621 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:13 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:13.624 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.624 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.624 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.624 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:13.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:13 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:13.626 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.626 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.626 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:13.627 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:13.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:13.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:13.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:13.628 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:13 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:14.622 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:14 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.623 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:14 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.624 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:14 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.624 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:14 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.625 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:14 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:14.625 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:14 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:14.632 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:14 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.632 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:14 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.633 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:14 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.633 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:14 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.634 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:14 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:14.634 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:14 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:14.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.638 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.639 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.639 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.639 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.639 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:14.640 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:14.640 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.640 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.641 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:14.641 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:14 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:14.644 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.644 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.645 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.645 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.646 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.646 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.646 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:14.647 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:14.647 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:14.648 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:14.650 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:14.650 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:14 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.614 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.614 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.619 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.620 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.620 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.621 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.621 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:15 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.626 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:15 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.626 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:15 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.626 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:15 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.627 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:15 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.627 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:15 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.627 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:15 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.631 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.632 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.632 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.633 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.633 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.634 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.635 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.636 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.636 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:15 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.638 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.639 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.639 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.639 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.640 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.640 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.640 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.641 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.641 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.642 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.642 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.643 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.643 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.643 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:15 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:15.645 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.645 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.646 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.647 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.647 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:15.648 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:15.648 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:15.648 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:15 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:16.617 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:16 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.618 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:16 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.618 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:16 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.618 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:16 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.619 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:16 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:16.619 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:16 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:16.622 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:16 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.623 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:16 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.623 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:16 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.624 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:16 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.624 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:16 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:16.625 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:16 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:16.627 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.627 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.628 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.628 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.629 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.629 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.630 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:16.630 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:16.630 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.631 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.631 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:16.632 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:16 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:16.635 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.636 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.636 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.636 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.636 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:16.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:16.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:16.637 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:16.638 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:16.638 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:16 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.613 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.614 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.615 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.616 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:17.617 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.618 INFO:teuthology.orchestra.run.smithi061.stdout:20 Jul 03:04:17 ntpd[18101]: ntpd: time slew -0.007718 s 2022-07-20T03:04:17.618 INFO:teuthology.orchestra.run.smithi061.stdout:ntpd: time slew -0.007718s 2022-07-20T03:04:17.619 INFO:teuthology.orchestra.run.smithi061.stderr:20 Jul 03:04:17 ntpd[18101]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T03:04:17.624 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.624 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.625 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:17.626 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.627 INFO:teuthology.orchestra.run.smithi124.stdout:20 Jul 03:04:17 ntpd[18195]: ntpd: time slew -0.003418 s 2022-07-20T03:04:17.628 INFO:teuthology.orchestra.run.smithi124.stdout:ntpd: time slew -0.003418s 2022-07-20T03:04:17.629 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.629 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.630 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.630 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.631 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.631 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.631 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:17.632 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.633 INFO:teuthology.orchestra.run.smithi163.stdout:20 Jul 03:04:17 ntpd[18056]: ntpd: time slew -0.003359 s 2022-07-20T03:04:17.633 INFO:teuthology.orchestra.run.smithi163.stdout:ntpd: time slew -0.003359s 2022-07-20T03:04:17.635 INFO:teuthology.orchestra.run.smithi124.stderr:20 Jul 03:04:17 ntpd[18195]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T03:04:17.636 INFO:teuthology.orchestra.run.smithi163.stderr:20 Jul 03:04:17 ntpd[18056]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T03:04:17.637 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.637 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.637 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.638 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.638 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.638 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.638 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:17.639 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.639 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.639 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.639 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:17.639 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.640 INFO:teuthology.orchestra.run.smithi117.stdout:20 Jul 03:04:17 ntpd[18046]: ntpd: time slew -0.006338 s 2022-07-20T03:04:17.640 INFO:teuthology.orchestra.run.smithi117.stdout:ntpd: time slew -0.006338s 2022-07-20T03:04:17.641 INFO:teuthology.orchestra.run.smithi117.stderr:20 Jul 03:04:17 ntpd[18046]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T03:04:17.642 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:17 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.642 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:17 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.642 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:17 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:17.643 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:17 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:17.643 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:17 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:17.643 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:17 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:17.678 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:04:17.679 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2022-07-20T03:04:17.679 INFO:teuthology.orchestra.run.smithi117.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.679 INFO:teuthology.orchestra.run.smithi117.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.680 INFO:teuthology.orchestra.run.smithi117.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.680 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.682 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:04:17.683 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2022-07-20T03:04:17.684 INFO:teuthology.orchestra.run.smithi061.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.684 INFO:teuthology.orchestra.run.smithi061.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.684 INFO:teuthology.orchestra.run.smithi061.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.685 INFO:teuthology.orchestra.run.smithi061.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.688 INFO:teuthology.orchestra.run.smithi124.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:04:17.689 INFO:teuthology.orchestra.run.smithi124.stdout:============================================================================== 2022-07-20T03:04:17.689 INFO:teuthology.orchestra.run.smithi124.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.690 INFO:teuthology.orchestra.run.smithi124.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.690 INFO:teuthology.orchestra.run.smithi124.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.691 INFO:teuthology.orchestra.run.smithi124.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.691 INFO:teuthology.orchestra.run.smithi163.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:04:17.692 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================== 2022-07-20T03:04:17.692 INFO:teuthology.orchestra.run.smithi163.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.693 INFO:teuthology.orchestra.run.smithi163.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.693 INFO:teuthology.orchestra.run.smithi163.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:17.694 INFO:teuthology.orchestra.run.smithi163.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:18.625 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:18.625 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:18.625 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:18.626 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:18.626 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: couldn't unlink /var/log/ntpstats/rawstats: Permission denied 2022-07-20T03:04:18.627 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: can't open /var/log/ntpstats/rawstats.20220720: Permission denied 2022-07-20T03:04:18.627 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: couldn't unlink /var/log/ntpstats/peerstats: Permission denied 2022-07-20T03:04:18.628 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: can't open /var/log/ntpstats/peerstats.20220720: Permission denied 2022-07-20T03:04:18.629 INFO:teuthology.orchestra.run.smithi179.stdout:20 Jul 03:04:18 ntpd[18044]: ntpd: time slew -0.004941 s 2022-07-20T03:04:18.629 INFO:teuthology.orchestra.run.smithi179.stdout:ntpd: time slew -0.004941s 2022-07-20T03:04:18.630 INFO:teuthology.orchestra.run.smithi179.stderr:20 Jul 03:04:18 ntpd[18044]: can't open /var/log/ntpstats/loopstats.20220720: Permission denied 2022-07-20T03:04:18.685 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T03:04:18.686 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== 2022-07-20T03:04:18.686 INFO:teuthology.orchestra.run.smithi179.stdout: hv01.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:18.687 INFO:teuthology.orchestra.run.smithi179.stdout: hv02.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:18.687 INFO:teuthology.orchestra.run.smithi179.stdout: hv03.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:18.687 INFO:teuthology.orchestra.run.smithi179.stdout: hv04.front.sepi .INIT. 16 u - 64 0 0.000 0.000 0.000 2022-07-20T03:04:18.688 INFO:teuthology.run_tasks:Running task install... 2022-07-20T03:04:18.699 DEBUG:teuthology.task.install:project ceph 2022-07-20T03:04:18.700 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}} 2022-07-20T03:04:18.700 DEBUG:teuthology.task.install:config {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'} 2022-07-20T03:04:18.701 INFO:teuthology.task.install:Using flavor: default 2022-07-20T03:04:18.708 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2022-07-20T03:04:18.709 INFO:teuthology.task.install:extra packages: {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']} 2022-07-20T03:04:18.710 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2022-07-20T03:04:18.761 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2022-07-20T03:04:18.768 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:18.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-07-20T03:04:18.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-key list | grep Ceph 2022-07-20T03:04:18.821 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-07-20T03:04:18.826 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:18.827 DEBUG:teuthology.orchestra.run.smithi124:> sudo lsb_release -is 2022-07-20T03:04:18.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-key list | grep Ceph 2022-07-20T03:04:18.875 INFO:teuthology.orchestra.run.smithi061.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T03:04:18.876 INFO:teuthology.orchestra.run.smithi124.stdout:Ubuntu 2022-07-20T03:04:18.882 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:18.883 DEBUG:teuthology.orchestra.run.smithi163:> sudo lsb_release -is 2022-07-20T03:04:18.884 DEBUG:teuthology.orchestra.run.smithi124:> sudo apt-key list | grep Ceph 2022-07-20T03:04:18.931 INFO:teuthology.orchestra.run.smithi117.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T03:04:18.935 INFO:teuthology.orchestra.run.smithi163.stdout:Ubuntu 2022-07-20T03:04:18.937 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T03:04:18.938 INFO:teuthology.orchestra.run.smithi061.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T03:04:18.939 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2022-07-20T03:04:18.941 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:18.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo lsb_release -is 2022-07-20T03:04:18.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo apt-key list | grep Ceph 2022-07-20T03:04:18.984 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T03:04:18.985 INFO:teuthology.orchestra.run.smithi117.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T03:04:18.986 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-07-20T03:04:18.988 INFO:teuthology.orchestra.run.smithi124.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T03:04:18.995 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2022-07-20T03:04:18.995 INFO:teuthology.orchestra.run.smithi179.stdout:Ubuntu 2022-07-20T03:04:19.000 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:19.000 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T03:04:19.001 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T03:04:19.001 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:04:19.010 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:19.011 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-key list | grep Ceph 2022-07-20T03:04:19.044 INFO:teuthology.orchestra.run.smithi163.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T03:04:19.046 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-07-20T03:04:19.047 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:19.047 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T03:04:19.048 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T03:04:19.048 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:04:19.054 INFO:teuthology.orchestra.run.smithi124.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T03:04:19.054 INFO:teuthology.orchestra.run.smithi124.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T03:04:19.055 DEBUG:teuthology.orchestra.run.smithi124:> sudo lsb_release -is 2022-07-20T03:04:19.102 INFO:teuthology.orchestra.run.smithi163.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T03:04:19.102 INFO:teuthology.orchestra.run.smithi163.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T03:04:19.103 DEBUG:teuthology.orchestra.run.smithi163:> sudo lsb_release -is 2022-07-20T03:04:19.104 INFO:teuthology.orchestra.run.smithi124.stdout:Ubuntu 2022-07-20T03:04:19.107 INFO:teuthology.orchestra.run.smithi179.stderr:Warning: apt-key output should not be parsed (stdout is not a terminal) 2022-07-20T03:04:19.108 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:19.109 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T03:04:19.109 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T03:04:19.109 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:04:19.148 INFO:teuthology.orchestra.run.smithi163.stdout:Ubuntu 2022-07-20T03:04:19.154 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:19.154 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T03:04:19.154 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T03:04:19.155 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:04:19.165 INFO:teuthology.orchestra.run.smithi179.stdout:uid [ unknown] Ceph automated package build (Ceph automated package build) 2022-07-20T03:04:19.166 INFO:teuthology.orchestra.run.smithi179.stdout:uid [ unknown] Ceph.com (release key) 2022-07-20T03:04:19.167 DEBUG:teuthology.orchestra.run.smithi179:> sudo lsb_release -is 2022-07-20T03:04:19.176 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T03:04:19.177 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T03:04:19.217 INFO:teuthology.orchestra.run.smithi179.stdout:Ubuntu 2022-07-20T03:04:19.234 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:04:19.234 INFO:teuthology.task.install.deb:Installing packages: ceph, cephadm, ceph-mds, ceph-mgr, ceph-common, ceph-fuse, ceph-test, radosgw, python3-rados, python3-rgw, python3-cephfs, python3-rbd, libcephfs2, libcephfs-dev, librados2, librbd1, rbd-fuse, python3-cephfs, cephfs-shell on remote deb x86_64 2022-07-20T03:04:19.235 INFO:teuthology.task.install.deb:Installing system (non-project) packages: bison, flex, libelf-dev, libssl-dev, dump, indent, libaio-dev, libtool-bin, uuid-dev, xfslibs-dev on remote deb x86_64 2022-07-20T03:04:19.235 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:04:19.237 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T03:04:19.238 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T03:04:19.294 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T03:04:19.294 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T03:04:19.319 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:04:19.320 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T03:04:19.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo apt-get update 2022-07-20T03:04:19.354 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T03:04:19.355 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T03:04:19.371 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:04:19.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T03:04:19.396 DEBUG:teuthology.orchestra.run.smithi117:> sudo apt-get update 2022-07-20T03:04:19.407 INFO:teuthology.task.install.deb:Pulling from https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default/ 2022-07-20T03:04:19.407 INFO:teuthology.task.install.deb:Package version is 15.2.16-336-g57544da1-1focal 2022-07-20T03:04:19.433 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:04:19.433 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T03:04:19.458 DEBUG:teuthology.orchestra.run.smithi124:> sudo apt-get update 2022-07-20T03:04:19.480 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:04:19.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T03:04:19.483 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T03:04:19.500 DEBUG:teuthology.orchestra.run.smithi163:> sudo apt-get update 2022-07-20T03:04:19.526 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:04:19.527 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/apt/sources.list.d/ceph.list 2022-07-20T03:04:19.538 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:1 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T03:04:19.538 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:2 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T03:04:19.543 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:3 http://archive.ubuntu.com/ubuntu focal-updates InRelease 2022-07-20T03:04:19.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo apt-get update 2022-07-20T03:04:19.557 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T03:04:19.560 INFO:teuthology.orchestra.run.smithi117.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal-backports InRelease 2022-07-20T03:04:19.578 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T03:04:19.592 INFO:teuthology.orchestra.run.smithi061.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T03:04:19.599 INFO:teuthology.orchestra.run.smithi124.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T03:04:19.600 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T03:04:19.604 INFO:teuthology.orchestra.run.smithi061.stdout:Ign:5 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T03:04:19.623 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T03:04:19.627 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T03:04:19.647 INFO:teuthology.orchestra.run.smithi117.stdout:Ign:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T03:04:19.659 INFO:teuthology.orchestra.run.smithi124.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T03:04:19.660 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T03:04:19.670 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T03:04:19.672 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T03:04:19.682 INFO:teuthology.orchestra.run.smithi124.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T03:04:19.699 INFO:teuthology.orchestra.run.smithi163.stdout:Ign:1 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T03:04:19.706 INFO:teuthology.orchestra.run.smithi124.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T03:04:19.710 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T03:04:19.712 INFO:teuthology.orchestra.run.smithi124.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T03:04:19.722 INFO:teuthology.orchestra.run.smithi163.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T03:04:19.728 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:1 http://security.ubuntu.com/ubuntu focal-security InRelease 2022-07-20T03:04:19.729 INFO:teuthology.orchestra.run.smithi124.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T03:04:19.747 INFO:teuthology.orchestra.run.smithi163.stdout:Ign:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T03:04:19.766 INFO:teuthology.orchestra.run.smithi163.stdout:Hit:4 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T03:04:19.767 INFO:teuthology.orchestra.run.smithi163.stdout:Get:5 http://security.ubuntu.com/ubuntu focal-security InRelease [114 kB] 2022-07-20T03:04:19.769 INFO:teuthology.orchestra.run.smithi124.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T03:04:19.781 INFO:teuthology.orchestra.run.smithi163.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T03:04:19.789 INFO:teuthology.orchestra.run.smithi179.stdout:Ign:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal InRelease 2022-07-20T03:04:19.797 INFO:teuthology.orchestra.run.smithi124.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T03:04:19.805 INFO:teuthology.orchestra.run.smithi163.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T03:04:19.815 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release [7,680 B] 2022-07-20T03:04:19.833 INFO:teuthology.orchestra.run.smithi163.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T03:04:19.837 INFO:teuthology.orchestra.run.smithi179.stdout:Ign:4 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal Release.gpg 2022-07-20T03:04:19.844 INFO:teuthology.orchestra.run.smithi179.stdout:Hit:5 http://archive.ubuntu.com/ubuntu focal InRelease 2022-07-20T03:04:19.864 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 Packages [15.8 kB] 2022-07-20T03:04:19.897 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main i386 Packages [3,078 B] 2022-07-20T03:04:19.925 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal-updates InRelease [114 kB] 2022-07-20T03:04:20.017 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T03:04:20.150 INFO:teuthology.orchestra.run.smithi124.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T03:04:20.165 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T03:04:20.177 INFO:teuthology.orchestra.run.smithi163.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T03:04:20.207 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 26.6 kB in 1s (41.3 kB/s) 2022-07-20T03:04:20.257 INFO:teuthology.orchestra.run.smithi163.stdout:Get:10 http://security.ubuntu.com/ubuntu focal-security/main amd64 c-n-f Metadata [10.8 kB] 2022-07-20T03:04:20.273 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal-backports InRelease [108 kB] 2022-07-20T03:04:20.297 INFO:teuthology.orchestra.run.smithi124.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T03:04:20.392 INFO:teuthology.orchestra.run.smithi163.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T03:04:20.424 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T03:04:20.576 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T03:04:20.594 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T03:04:20.682 INFO:teuthology.orchestra.run.smithi163.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T03:04:20.710 INFO:teuthology.orchestra.run.smithi124.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main i386 Packages [695 kB] 2022-07-20T03:04:20.716 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 Packages [1,989 kB] 2022-07-20T03:04:20.728 INFO:teuthology.orchestra.run.smithi124.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T03:04:20.845 INFO:teuthology.orchestra.run.smithi163.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T03:04:20.880 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal-updates/main Translation-en [358 kB] 2022-07-20T03:04:20.962 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 3,291 kB in 1s (2,248 kB/s) 2022-07-20T03:04:21.081 INFO:teuthology.orchestra.run.smithi124.stdout:Fetched 3,291 kB in 1s (2,244 kB/s) 2022-07-20T03:04:21.086 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-07-20T03:04:21.108 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T03:04:21.179 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-07-20T03:04:21.267 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 3,291 kB in 2s (2,169 kB/s) 2022-07-20T03:04:21.334 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-07-20T03:04:21.335 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-07-20T03:04:21.476 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:04:21.477 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:04:21.477 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:04:21.479 INFO:teuthology.orchestra.run.smithi117.stdout:The following additional packages will be installed: 2022-07-20T03:04:21.479 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:21.480 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:21.480 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T03:04:21.481 INFO:teuthology.orchestra.run.smithi117.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T03:04:21.481 INFO:teuthology.orchestra.run.smithi117.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T03:04:21.482 INFO:teuthology.orchestra.run.smithi117.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T03:04:21.482 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T03:04:21.483 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T03:04:21.483 INFO:teuthology.orchestra.run.smithi117.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T03:04:21.483 INFO:teuthology.orchestra.run.smithi117.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T03:04:21.484 INFO:teuthology.orchestra.run.smithi117.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T03:04:21.484 INFO:teuthology.orchestra.run.smithi117.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T03:04:21.485 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T03:04:21.485 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T03:04:21.486 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T03:04:21.487 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T03:04:21.487 INFO:teuthology.orchestra.run.smithi117.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T03:04:21.487 INFO:teuthology.orchestra.run.smithi117.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T03:04:21.488 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T03:04:21.488 INFO:teuthology.orchestra.run.smithi117.stdout: xmlstarlet 2022-07-20T03:04:21.489 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2022-07-20T03:04:21.489 INFO:teuthology.orchestra.run.smithi117.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T03:04:21.489 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T03:04:21.490 INFO:teuthology.orchestra.run.smithi117.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T03:04:21.490 INFO:teuthology.orchestra.run.smithi117.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T03:04:21.491 INFO:teuthology.orchestra.run.smithi117.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T03:04:21.491 INFO:teuthology.orchestra.run.smithi117.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T03:04:21.491 INFO:teuthology.orchestra.run.smithi117.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T03:04:21.493 INFO:teuthology.orchestra.run.smithi117.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T03:04:21.493 INFO:teuthology.orchestra.run.smithi117.stdout:Recommended packages: 2022-07-20T03:04:21.495 INFO:teuthology.orchestra.run.smithi117.stdout: btrfs-tools 2022-07-20T03:04:21.583 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2022-07-20T03:04:21.584 INFO:teuthology.orchestra.run.smithi117.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T03:04:21.584 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:21.584 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:21.585 INFO:teuthology.orchestra.run.smithi117.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T03:04:21.585 INFO:teuthology.orchestra.run.smithi117.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T03:04:21.586 INFO:teuthology.orchestra.run.smithi117.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T03:04:21.586 INFO:teuthology.orchestra.run.smithi117.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T03:04:21.586 INFO:teuthology.orchestra.run.smithi117.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T03:04:21.599 INFO:teuthology.orchestra.run.smithi117.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T03:04:21.600 INFO:teuthology.orchestra.run.smithi117.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T03:04:21.600 INFO:teuthology.orchestra.run.smithi117.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T03:04:21.601 INFO:teuthology.orchestra.run.smithi117.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T03:04:21.601 INFO:teuthology.orchestra.run.smithi117.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T03:04:21.602 INFO:teuthology.orchestra.run.smithi117.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T03:04:21.602 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T03:04:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T03:04:21.603 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T03:04:21.604 INFO:teuthology.orchestra.run.smithi117.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T03:04:21.604 INFO:teuthology.orchestra.run.smithi117.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T03:04:21.605 INFO:teuthology.orchestra.run.smithi117.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T03:04:21.606 INFO:teuthology.orchestra.run.smithi117.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T03:04:21.607 INFO:teuthology.orchestra.run.smithi117.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T03:04:21.608 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages will be upgraded: 2022-07-20T03:04:21.609 INFO:teuthology.orchestra.run.smithi117.stdout: librados2 librbd1 2022-07-20T03:04:21.657 INFO:teuthology.orchestra.run.smithi117.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:04:21.657 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 153 MB of archives. 2022-07-20T03:04:21.657 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T03:04:21.658 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T03:04:21.674 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T03:04:21.681 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T03:04:21.707 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T03:04:21.738 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T03:04:21.738 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T03:04:21.741 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T03:04:21.750 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T03:04:21.764 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T03:04:21.768 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T03:04:21.770 INFO:teuthology.orchestra.run.smithi117.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T03:04:21.780 INFO:teuthology.orchestra.run.smithi117.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T03:04:21.781 INFO:teuthology.orchestra.run.smithi117.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T03:04:21.781 INFO:teuthology.orchestra.run.smithi117.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T03:04:21.781 INFO:teuthology.orchestra.run.smithi117.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T03:04:21.782 INFO:teuthology.orchestra.run.smithi117.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T03:04:21.782 INFO:teuthology.orchestra.run.smithi117.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T03:04:21.782 INFO:teuthology.orchestra.run.smithi117.stdout:Get:18 http://security.ubuntu.com/ubuntu focal-security/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T03:04:21.789 INFO:teuthology.orchestra.run.smithi117.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T03:04:21.789 INFO:teuthology.orchestra.run.smithi117.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T03:04:21.793 INFO:teuthology.orchestra.run.smithi117.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T03:04:21.811 INFO:teuthology.orchestra.run.smithi117.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T03:04:21.815 INFO:teuthology.orchestra.run.smithi117.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T03:04:21.816 INFO:teuthology.orchestra.run.smithi117.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T03:04:21.816 INFO:teuthology.orchestra.run.smithi117.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T03:04:21.817 INFO:teuthology.orchestra.run.smithi117.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T03:04:21.818 INFO:teuthology.orchestra.run.smithi117.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T03:04:21.818 INFO:teuthology.orchestra.run.smithi117.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T03:04:21.822 INFO:teuthology.orchestra.run.smithi117.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T03:04:21.823 INFO:teuthology.orchestra.run.smithi117.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T03:04:21.824 INFO:teuthology.orchestra.run.smithi117.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T03:04:21.840 INFO:teuthology.orchestra.run.smithi117.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T03:04:21.841 INFO:teuthology.orchestra.run.smithi117.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T03:04:21.841 INFO:teuthology.orchestra.run.smithi117.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T03:04:21.851 INFO:teuthology.orchestra.run.smithi117.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T03:04:21.852 INFO:teuthology.orchestra.run.smithi117.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T03:04:21.852 INFO:teuthology.orchestra.run.smithi117.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T03:04:21.854 INFO:teuthology.orchestra.run.smithi117.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T03:04:21.861 INFO:teuthology.orchestra.run.smithi117.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T03:04:21.875 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-20T03:04:21.894 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T03:04:21.932 INFO:teuthology.orchestra.run.smithi117.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T03:04:21.932 INFO:teuthology.orchestra.run.smithi117.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T03:04:21.946 INFO:teuthology.orchestra.run.smithi117.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T03:04:21.947 INFO:teuthology.orchestra.run.smithi117.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T03:04:21.960 INFO:teuthology.orchestra.run.smithi117.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T03:04:21.961 INFO:teuthology.orchestra.run.smithi124.stdout:Reading package lists... 2022-07-20T03:04:21.967 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-20T03:04:21.980 DEBUG:teuthology.orchestra.run.smithi124:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T03:04:22.041 INFO:teuthology.orchestra.run.smithi117.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T03:04:22.043 INFO:teuthology.orchestra.run.smithi117.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T03:04:22.045 INFO:teuthology.orchestra.run.smithi117.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T03:04:22.045 INFO:teuthology.orchestra.run.smithi117.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T03:04:22.046 INFO:teuthology.orchestra.run.smithi117.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T03:04:22.046 INFO:teuthology.orchestra.run.smithi117.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T03:04:22.048 INFO:teuthology.orchestra.run.smithi117.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T03:04:22.049 INFO:teuthology.orchestra.run.smithi117.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T03:04:22.050 INFO:teuthology.orchestra.run.smithi124.stdout:Reading package lists... 2022-07-20T03:04:22.058 INFO:teuthology.orchestra.run.smithi117.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T03:04:22.069 INFO:teuthology.orchestra.run.smithi117.stdout:Get:54 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T03:04:22.070 INFO:teuthology.orchestra.run.smithi117.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T03:04:22.071 INFO:teuthology.orchestra.run.smithi117.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T03:04:22.073 INFO:teuthology.orchestra.run.smithi117.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T03:04:22.073 INFO:teuthology.orchestra.run.smithi117.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T03:04:22.076 INFO:teuthology.orchestra.run.smithi117.stdout:Get:59 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T03:04:22.076 INFO:teuthology.orchestra.run.smithi117.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T03:04:22.079 INFO:teuthology.orchestra.run.smithi117.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T03:04:22.081 INFO:teuthology.orchestra.run.smithi117.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T03:04:22.081 INFO:teuthology.orchestra.run.smithi117.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T03:04:22.084 INFO:teuthology.orchestra.run.smithi117.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T03:04:22.092 INFO:teuthology.orchestra.run.smithi117.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T03:04:22.092 INFO:teuthology.orchestra.run.smithi117.stdout:Get:66 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T03:04:22.096 INFO:teuthology.orchestra.run.smithi117.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T03:04:22.096 INFO:teuthology.orchestra.run.smithi117.stdout:Get:68 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T03:04:22.097 INFO:teuthology.orchestra.run.smithi117.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T03:04:22.097 INFO:teuthology.orchestra.run.smithi117.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T03:04:22.098 INFO:teuthology.orchestra.run.smithi117.stdout:Get:71 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T03:04:22.098 INFO:teuthology.orchestra.run.smithi117.stdout:Get:72 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T03:04:22.109 INFO:teuthology.orchestra.run.smithi117.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T03:04:22.110 INFO:teuthology.orchestra.run.smithi117.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T03:04:22.111 INFO:teuthology.orchestra.run.smithi117.stdout:Get:75 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T03:04:22.111 INFO:teuthology.orchestra.run.smithi117.stdout:Get:76 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T03:04:22.113 INFO:teuthology.orchestra.run.smithi117.stdout:Get:77 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T03:04:22.113 INFO:teuthology.orchestra.run.smithi117.stdout:Get:78 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T03:04:22.114 INFO:teuthology.orchestra.run.smithi117.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T03:04:22.126 INFO:teuthology.orchestra.run.smithi117.stdout:Get:80 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T03:04:22.127 INFO:teuthology.orchestra.run.smithi117.stdout:Get:81 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T03:04:22.146 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-07-20T03:04:22.147 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-20T03:04:22.147 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-20T03:04:22.166 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T03:04:22.207 INFO:teuthology.orchestra.run.smithi124.stdout:Building dependency tree... 2022-07-20T03:04:22.208 INFO:teuthology.orchestra.run.smithi124.stdout:Reading state information... 2022-07-20T03:04:22.211 INFO:teuthology.orchestra.run.smithi117.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T03:04:22.216 INFO:teuthology.orchestra.run.smithi117.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T03:04:22.227 INFO:teuthology.orchestra.run.smithi117.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T03:04:22.239 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-07-20T03:04:22.303 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:04:22.306 INFO:teuthology.orchestra.run.smithi061.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:04:22.307 INFO:teuthology.orchestra.run.smithi061.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:04:22.307 INFO:teuthology.orchestra.run.smithi061.stdout:The following additional packages will be installed: 2022-07-20T03:04:22.308 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:22.308 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:22.309 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T03:04:22.309 INFO:teuthology.orchestra.run.smithi061.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T03:04:22.309 INFO:teuthology.orchestra.run.smithi061.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T03:04:22.310 INFO:teuthology.orchestra.run.smithi061.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T03:04:22.310 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T03:04:22.311 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T03:04:22.311 INFO:teuthology.orchestra.run.smithi061.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T03:04:22.311 INFO:teuthology.orchestra.run.smithi061.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T03:04:22.312 INFO:teuthology.orchestra.run.smithi061.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T03:04:22.312 INFO:teuthology.orchestra.run.smithi061.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T03:04:22.313 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T03:04:22.313 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T03:04:22.313 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T03:04:22.314 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T03:04:22.314 INFO:teuthology.orchestra.run.smithi061.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T03:04:22.315 INFO:teuthology.orchestra.run.smithi061.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T03:04:22.315 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T03:04:22.315 INFO:teuthology.orchestra.run.smithi061.stdout: xmlstarlet 2022-07-20T03:04:22.316 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2022-07-20T03:04:22.316 INFO:teuthology.orchestra.run.smithi061.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T03:04:22.317 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T03:04:22.317 INFO:teuthology.orchestra.run.smithi061.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T03:04:22.318 INFO:teuthology.orchestra.run.smithi061.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T03:04:22.318 INFO:teuthology.orchestra.run.smithi061.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T03:04:22.320 INFO:teuthology.orchestra.run.smithi061.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T03:04:22.320 INFO:teuthology.orchestra.run.smithi061.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T03:04:22.320 INFO:teuthology.orchestra.run.smithi061.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T03:04:22.321 INFO:teuthology.orchestra.run.smithi061.stdout:Recommended packages: 2022-07-20T03:04:22.321 INFO:teuthology.orchestra.run.smithi061.stdout: btrfs-tools 2022-07-20T03:04:22.355 INFO:teuthology.orchestra.run.smithi124.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:04:22.356 INFO:teuthology.orchestra.run.smithi124.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:04:22.356 INFO:teuthology.orchestra.run.smithi124.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:04:22.356 INFO:teuthology.orchestra.run.smithi124.stdout:The following additional packages will be installed: 2022-07-20T03:04:22.357 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:22.357 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:22.358 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T03:04:22.358 INFO:teuthology.orchestra.run.smithi124.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T03:04:22.359 INFO:teuthology.orchestra.run.smithi124.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T03:04:22.361 INFO:teuthology.orchestra.run.smithi124.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T03:04:22.361 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T03:04:22.361 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T03:04:22.362 INFO:teuthology.orchestra.run.smithi124.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T03:04:22.362 INFO:teuthology.orchestra.run.smithi124.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T03:04:22.363 INFO:teuthology.orchestra.run.smithi124.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T03:04:22.363 INFO:teuthology.orchestra.run.smithi124.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T03:04:22.364 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T03:04:22.364 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T03:04:22.364 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T03:04:22.365 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T03:04:22.365 INFO:teuthology.orchestra.run.smithi124.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T03:04:22.365 INFO:teuthology.orchestra.run.smithi124.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T03:04:22.365 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T03:04:22.365 INFO:teuthology.orchestra.run.smithi124.stdout: xmlstarlet 2022-07-20T03:04:22.366 INFO:teuthology.orchestra.run.smithi124.stdout:Suggested packages: 2022-07-20T03:04:22.366 INFO:teuthology.orchestra.run.smithi124.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T03:04:22.366 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T03:04:22.367 INFO:teuthology.orchestra.run.smithi124.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T03:04:22.367 INFO:teuthology.orchestra.run.smithi124.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T03:04:22.367 INFO:teuthology.orchestra.run.smithi124.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T03:04:22.368 INFO:teuthology.orchestra.run.smithi124.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T03:04:22.368 INFO:teuthology.orchestra.run.smithi124.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T03:04:22.368 INFO:teuthology.orchestra.run.smithi124.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T03:04:22.368 INFO:teuthology.orchestra.run.smithi124.stdout:Recommended packages: 2022-07-20T03:04:22.369 INFO:teuthology.orchestra.run.smithi124.stdout: btrfs-tools 2022-07-20T03:04:22.411 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-07-20T03:04:22.412 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-07-20T03:04:22.413 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2022-07-20T03:04:22.413 INFO:teuthology.orchestra.run.smithi061.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T03:04:22.414 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:22.414 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:22.417 INFO:teuthology.orchestra.run.smithi061.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T03:04:22.417 INFO:teuthology.orchestra.run.smithi061.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T03:04:22.417 INFO:teuthology.orchestra.run.smithi061.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T03:04:22.417 INFO:teuthology.orchestra.run.smithi061.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T03:04:22.418 INFO:teuthology.orchestra.run.smithi061.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T03:04:22.418 INFO:teuthology.orchestra.run.smithi061.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T03:04:22.418 INFO:teuthology.orchestra.run.smithi061.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T03:04:22.418 INFO:teuthology.orchestra.run.smithi061.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T03:04:22.419 INFO:teuthology.orchestra.run.smithi061.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T03:04:22.419 INFO:teuthology.orchestra.run.smithi061.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T03:04:22.419 INFO:teuthology.orchestra.run.smithi061.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T03:04:22.420 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T03:04:22.420 INFO:teuthology.orchestra.run.smithi061.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T03:04:22.420 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T03:04:22.420 INFO:teuthology.orchestra.run.smithi061.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T03:04:22.421 INFO:teuthology.orchestra.run.smithi061.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T03:04:22.421 INFO:teuthology.orchestra.run.smithi061.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T03:04:22.421 INFO:teuthology.orchestra.run.smithi061.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T03:04:22.421 INFO:teuthology.orchestra.run.smithi061.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T03:04:22.421 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages will be upgraded: 2022-07-20T03:04:22.426 INFO:teuthology.orchestra.run.smithi061.stdout: librados2 librbd1 2022-07-20T03:04:22.452 INFO:teuthology.orchestra.run.smithi124.stdout:The following NEW packages will be installed: 2022-07-20T03:04:22.452 INFO:teuthology.orchestra.run.smithi124.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T03:04:22.453 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:22.453 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:22.453 INFO:teuthology.orchestra.run.smithi124.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T03:04:22.454 INFO:teuthology.orchestra.run.smithi124.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T03:04:22.454 INFO:teuthology.orchestra.run.smithi124.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T03:04:22.455 INFO:teuthology.orchestra.run.smithi124.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T03:04:22.455 INFO:teuthology.orchestra.run.smithi124.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T03:04:22.456 INFO:teuthology.orchestra.run.smithi124.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T03:04:22.456 INFO:teuthology.orchestra.run.smithi124.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T03:04:22.456 INFO:teuthology.orchestra.run.smithi124.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T03:04:22.457 INFO:teuthology.orchestra.run.smithi124.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T03:04:22.457 INFO:teuthology.orchestra.run.smithi124.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T03:04:22.457 INFO:teuthology.orchestra.run.smithi124.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T03:04:22.457 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T03:04:22.458 INFO:teuthology.orchestra.run.smithi124.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T03:04:22.458 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T03:04:22.458 INFO:teuthology.orchestra.run.smithi124.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T03:04:22.459 INFO:teuthology.orchestra.run.smithi124.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T03:04:22.459 INFO:teuthology.orchestra.run.smithi124.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T03:04:22.460 INFO:teuthology.orchestra.run.smithi124.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T03:04:22.460 INFO:teuthology.orchestra.run.smithi124.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T03:04:22.462 INFO:teuthology.orchestra.run.smithi124.stdout:The following packages will be upgraded: 2022-07-20T03:04:22.462 INFO:teuthology.orchestra.run.smithi124.stdout: librados2 librbd1 2022-07-20T03:04:22.561 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:04:22.562 INFO:teuthology.orchestra.run.smithi179.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:04:22.562 INFO:teuthology.orchestra.run.smithi179.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:04:22.562 INFO:teuthology.orchestra.run.smithi179.stdout:The following additional packages will be installed: 2022-07-20T03:04:22.563 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:22.565 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:22.565 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T03:04:22.565 INFO:teuthology.orchestra.run.smithi179.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T03:04:22.566 INFO:teuthology.orchestra.run.smithi179.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T03:04:22.566 INFO:teuthology.orchestra.run.smithi179.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T03:04:22.567 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T03:04:22.567 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T03:04:22.568 INFO:teuthology.orchestra.run.smithi179.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T03:04:22.568 INFO:teuthology.orchestra.run.smithi179.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T03:04:22.568 INFO:teuthology.orchestra.run.smithi179.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T03:04:22.568 INFO:teuthology.orchestra.run.smithi179.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T03:04:22.569 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T03:04:22.569 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T03:04:22.569 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T03:04:22.569 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T03:04:22.570 INFO:teuthology.orchestra.run.smithi179.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T03:04:22.570 INFO:teuthology.orchestra.run.smithi179.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T03:04:22.570 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T03:04:22.571 INFO:teuthology.orchestra.run.smithi179.stdout: xmlstarlet 2022-07-20T03:04:22.571 INFO:teuthology.orchestra.run.smithi179.stdout:Suggested packages: 2022-07-20T03:04:22.571 INFO:teuthology.orchestra.run.smithi179.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T03:04:22.571 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T03:04:22.571 INFO:teuthology.orchestra.run.smithi179.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T03:04:22.572 INFO:teuthology.orchestra.run.smithi179.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T03:04:22.572 INFO:teuthology.orchestra.run.smithi179.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T03:04:22.573 INFO:teuthology.orchestra.run.smithi179.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T03:04:22.573 INFO:teuthology.orchestra.run.smithi179.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T03:04:22.574 INFO:teuthology.orchestra.run.smithi179.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T03:04:22.575 INFO:teuthology.orchestra.run.smithi179.stdout:Recommended packages: 2022-07-20T03:04:22.575 INFO:teuthology.orchestra.run.smithi179.stdout: btrfs-tools 2022-07-20T03:04:22.605 INFO:teuthology.orchestra.run.smithi061.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:04:22.606 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 153 MB of archives. 2022-07-20T03:04:22.606 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T03:04:22.607 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T03:04:22.642 INFO:teuthology.orchestra.run.smithi124.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:04:22.643 INFO:teuthology.orchestra.run.smithi124.stdout:Need to get 153 MB of archives. 2022-07-20T03:04:22.643 INFO:teuthology.orchestra.run.smithi124.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T03:04:22.643 INFO:teuthology.orchestra.run.smithi124.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T03:04:22.668 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2022-07-20T03:04:22.669 INFO:teuthology.orchestra.run.smithi179.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T03:04:22.669 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:22.670 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:22.671 INFO:teuthology.orchestra.run.smithi179.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T03:04:22.671 INFO:teuthology.orchestra.run.smithi179.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T03:04:22.671 INFO:teuthology.orchestra.run.smithi179.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T03:04:22.671 INFO:teuthology.orchestra.run.smithi179.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T03:04:22.671 INFO:teuthology.orchestra.run.smithi179.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T03:04:22.672 INFO:teuthology.orchestra.run.smithi179.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T03:04:22.672 INFO:teuthology.orchestra.run.smithi179.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T03:04:22.672 INFO:teuthology.orchestra.run.smithi179.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T03:04:22.672 INFO:teuthology.orchestra.run.smithi179.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T03:04:22.673 INFO:teuthology.orchestra.run.smithi179.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T03:04:22.673 INFO:teuthology.orchestra.run.smithi179.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T03:04:22.673 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T03:04:22.673 INFO:teuthology.orchestra.run.smithi179.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T03:04:22.673 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T03:04:22.674 INFO:teuthology.orchestra.run.smithi179.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T03:04:22.674 INFO:teuthology.orchestra.run.smithi179.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T03:04:22.674 INFO:teuthology.orchestra.run.smithi179.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T03:04:22.674 INFO:teuthology.orchestra.run.smithi179.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T03:04:22.675 INFO:teuthology.orchestra.run.smithi179.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T03:04:22.675 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages will be upgraded: 2022-07-20T03:04:22.676 INFO:teuthology.orchestra.run.smithi179.stdout: librados2 librbd1 2022-07-20T03:04:22.688 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T03:04:22.689 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T03:04:22.723 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T03:04:22.724 INFO:teuthology.orchestra.run.smithi124.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T03:04:22.729 INFO:teuthology.orchestra.run.smithi124.stdout:Get:3 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T03:04:22.731 INFO:teuthology.orchestra.run.smithi179.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:04:22.732 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 153 MB of archives. 2022-07-20T03:04:22.732 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T03:04:22.732 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T03:04:22.751 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T03:04:22.758 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T03:04:22.760 INFO:teuthology.orchestra.run.smithi124.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T03:04:22.787 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T03:04:22.816 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T03:04:22.819 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T03:04:22.820 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T03:04:22.844 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T03:04:22.849 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T03:04:22.850 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T03:04:22.863 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T03:04:22.869 INFO:teuthology.orchestra.run.smithi179.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T03:04:22.870 INFO:teuthology.orchestra.run.smithi179.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T03:04:22.870 INFO:teuthology.orchestra.run.smithi179.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T03:04:22.871 INFO:teuthology.orchestra.run.smithi179.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T03:04:22.871 INFO:teuthology.orchestra.run.smithi179.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T03:04:22.872 INFO:teuthology.orchestra.run.smithi179.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T03:04:22.873 INFO:teuthology.orchestra.run.smithi179.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T03:04:22.874 INFO:teuthology.orchestra.run.smithi179.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T03:04:22.874 INFO:teuthology.orchestra.run.smithi179.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T03:04:22.887 INFO:teuthology.orchestra.run.smithi179.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T03:04:22.888 INFO:teuthology.orchestra.run.smithi179.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T03:04:22.889 INFO:teuthology.orchestra.run.smithi179.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T03:04:22.891 INFO:teuthology.orchestra.run.smithi179.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T03:04:22.893 INFO:teuthology.orchestra.run.smithi179.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T03:04:22.894 INFO:teuthology.orchestra.run.smithi124.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T03:04:22.895 INFO:teuthology.orchestra.run.smithi179.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T03:04:22.902 INFO:teuthology.orchestra.run.smithi179.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T03:04:22.903 INFO:teuthology.orchestra.run.smithi179.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T03:04:22.905 INFO:teuthology.orchestra.run.smithi179.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T03:04:22.905 INFO:teuthology.orchestra.run.smithi179.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T03:04:22.922 INFO:teuthology.orchestra.run.smithi179.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T03:04:22.923 INFO:teuthology.orchestra.run.smithi179.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T03:04:22.924 INFO:teuthology.orchestra.run.smithi179.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T03:04:22.925 INFO:teuthology.orchestra.run.smithi179.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T03:04:22.926 INFO:teuthology.orchestra.run.smithi179.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T03:04:22.927 INFO:teuthology.orchestra.run.smithi179.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T03:04:22.929 INFO:teuthology.orchestra.run.smithi179.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T03:04:22.938 INFO:teuthology.orchestra.run.smithi179.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T03:04:22.939 INFO:teuthology.orchestra.run.smithi179.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T03:04:22.988 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T03:04:22.996 INFO:teuthology.orchestra.run.smithi179.stdout:Get:39 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T03:04:22.997 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T03:04:23.007 INFO:teuthology.orchestra.run.smithi163.stdout:Fetched 3,416 kB in 2s (2,174 kB/s) 2022-07-20T03:04:23.010 INFO:teuthology.orchestra.run.smithi117.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T03:04:23.012 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T03:04:23.015 INFO:teuthology.orchestra.run.smithi179.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T03:04:23.024 INFO:teuthology.orchestra.run.smithi124.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T03:04:23.029 INFO:teuthology.orchestra.run.smithi179.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T03:04:23.031 INFO:teuthology.orchestra.run.smithi179.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T03:04:23.033 INFO:teuthology.orchestra.run.smithi124.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T03:04:23.045 INFO:teuthology.orchestra.run.smithi179.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T03:04:23.046 INFO:teuthology.orchestra.run.smithi179.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T03:04:23.048 INFO:teuthology.orchestra.run.smithi124.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T03:04:23.100 INFO:teuthology.orchestra.run.smithi117.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T03:04:23.104 INFO:teuthology.orchestra.run.smithi117.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T03:04:23.106 INFO:teuthology.orchestra.run.smithi179.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T03:04:23.106 INFO:teuthology.orchestra.run.smithi179.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T03:04:23.108 INFO:teuthology.orchestra.run.smithi179.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T03:04:23.108 INFO:teuthology.orchestra.run.smithi179.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T03:04:23.110 INFO:teuthology.orchestra.run.smithi179.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T03:04:23.111 INFO:teuthology.orchestra.run.smithi179.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T03:04:23.112 INFO:teuthology.orchestra.run.smithi179.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T03:04:23.116 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T03:04:23.122 INFO:teuthology.orchestra.run.smithi179.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T03:04:23.124 INFO:teuthology.orchestra.run.smithi179.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T03:04:23.125 INFO:teuthology.orchestra.run.smithi179.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T03:04:23.137 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T03:04:23.139 INFO:teuthology.orchestra.run.smithi179.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T03:04:23.140 INFO:teuthology.orchestra.run.smithi179.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T03:04:23.143 INFO:teuthology.orchestra.run.smithi061.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T03:04:23.145 INFO:teuthology.orchestra.run.smithi179.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T03:04:23.148 INFO:teuthology.orchestra.run.smithi179.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T03:04:23.151 INFO:teuthology.orchestra.run.smithi124.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T03:04:23.153 INFO:teuthology.orchestra.run.smithi179.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T03:04:23.153 INFO:teuthology.orchestra.run.smithi179.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T03:04:23.155 INFO:teuthology.orchestra.run.smithi117.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T03:04:23.156 INFO:teuthology.orchestra.run.smithi179.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T03:04:23.156 INFO:teuthology.orchestra.run.smithi179.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T03:04:23.158 INFO:teuthology.orchestra.run.smithi179.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T03:04:23.159 INFO:teuthology.orchestra.run.smithi179.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T03:04:23.168 INFO:teuthology.orchestra.run.smithi061.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T03:04:23.172 INFO:teuthology.orchestra.run.smithi061.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T03:04:23.174 INFO:teuthology.orchestra.run.smithi179.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T03:04:23.175 INFO:teuthology.orchestra.run.smithi124.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T03:04:23.176 INFO:teuthology.orchestra.run.smithi061.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T03:04:23.179 INFO:teuthology.orchestra.run.smithi124.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T03:04:23.192 INFO:teuthology.orchestra.run.smithi061.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T03:04:23.193 INFO:teuthology.orchestra.run.smithi061.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T03:04:23.194 INFO:teuthology.orchestra.run.smithi179.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T03:04:23.194 INFO:teuthology.orchestra.run.smithi179.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T03:04:23.194 INFO:teuthology.orchestra.run.smithi179.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T03:04:23.195 INFO:teuthology.orchestra.run.smithi179.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T03:04:23.195 INFO:teuthology.orchestra.run.smithi179.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T03:04:23.195 INFO:teuthology.orchestra.run.smithi179.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T03:04:23.195 INFO:teuthology.orchestra.run.smithi179.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T03:04:23.196 INFO:teuthology.orchestra.run.smithi061.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T03:04:23.196 INFO:teuthology.orchestra.run.smithi179.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T03:04:23.204 INFO:teuthology.orchestra.run.smithi124.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T03:04:23.211 INFO:teuthology.orchestra.run.smithi124.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T03:04:23.211 INFO:teuthology.orchestra.run.smithi124.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T03:04:23.226 INFO:teuthology.orchestra.run.smithi124.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T03:04:23.227 INFO:teuthology.orchestra.run.smithi124.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T03:04:23.229 INFO:teuthology.orchestra.run.smithi124.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T03:04:23.234 INFO:teuthology.orchestra.run.smithi061.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T03:04:23.235 INFO:teuthology.orchestra.run.smithi061.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T03:04:23.241 INFO:teuthology.orchestra.run.smithi061.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T03:04:23.267 INFO:teuthology.orchestra.run.smithi124.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T03:04:23.269 INFO:teuthology.orchestra.run.smithi124.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T03:04:23.270 INFO:teuthology.orchestra.run.smithi124.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T03:04:23.313 INFO:teuthology.orchestra.run.smithi061.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T03:04:23.315 INFO:teuthology.orchestra.run.smithi061.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T03:04:23.317 INFO:teuthology.orchestra.run.smithi061.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T03:04:23.320 INFO:teuthology.orchestra.run.smithi061.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T03:04:23.324 INFO:teuthology.orchestra.run.smithi061.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T03:04:23.324 INFO:teuthology.orchestra.run.smithi061.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T03:04:23.343 INFO:teuthology.orchestra.run.smithi061.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T03:04:23.346 INFO:teuthology.orchestra.run.smithi124.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T03:04:23.349 INFO:teuthology.orchestra.run.smithi124.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T03:04:23.350 INFO:teuthology.orchestra.run.smithi124.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T03:04:23.354 INFO:teuthology.orchestra.run.smithi124.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T03:04:23.357 INFO:teuthology.orchestra.run.smithi124.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T03:04:23.358 INFO:teuthology.orchestra.run.smithi124.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T03:04:23.376 INFO:teuthology.orchestra.run.smithi124.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T03:04:23.393 INFO:teuthology.orchestra.run.smithi061.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T03:04:23.395 INFO:teuthology.orchestra.run.smithi061.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T03:04:23.396 INFO:teuthology.orchestra.run.smithi061.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T03:04:23.396 INFO:teuthology.orchestra.run.smithi117.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T03:04:23.427 INFO:teuthology.orchestra.run.smithi124.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T03:04:23.429 INFO:teuthology.orchestra.run.smithi124.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T03:04:23.429 INFO:teuthology.orchestra.run.smithi124.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T03:04:23.474 INFO:teuthology.orchestra.run.smithi061.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T03:04:23.479 INFO:teuthology.orchestra.run.smithi061.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T03:04:23.482 INFO:teuthology.orchestra.run.smithi061.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T03:04:23.483 INFO:teuthology.orchestra.run.smithi061.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T03:04:23.486 INFO:teuthology.orchestra.run.smithi061.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T03:04:23.490 INFO:teuthology.orchestra.run.smithi061.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T03:04:23.492 INFO:teuthology.orchestra.run.smithi061.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T03:04:23.507 INFO:teuthology.orchestra.run.smithi124.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T03:04:23.513 INFO:teuthology.orchestra.run.smithi124.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T03:04:23.516 INFO:teuthology.orchestra.run.smithi124.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T03:04:23.517 INFO:teuthology.orchestra.run.smithi124.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T03:04:23.520 INFO:teuthology.orchestra.run.smithi124.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T03:04:23.523 INFO:teuthology.orchestra.run.smithi124.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T03:04:23.525 INFO:teuthology.orchestra.run.smithi124.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T03:04:23.554 INFO:teuthology.orchestra.run.smithi061.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T03:04:23.555 INFO:teuthology.orchestra.run.smithi061.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T03:04:23.586 INFO:teuthology.orchestra.run.smithi124.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T03:04:23.587 INFO:teuthology.orchestra.run.smithi124.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T03:04:23.853 INFO:teuthology.orchestra.run.smithi179.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T03:04:23.872 INFO:teuthology.orchestra.run.smithi163.stdout:Reading package lists... 2022-07-20T03:04:23.888 INFO:teuthology.orchestra.run.smithi061.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T03:04:23.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install ceph=15.2.16-336-g57544da1-1focal cephadm=15.2.16-336-g57544da1-1focal ceph-mds=15.2.16-336-g57544da1-1focal ceph-mgr=15.2.16-336-g57544da1-1focal ceph-common=15.2.16-336-g57544da1-1focal ceph-fuse=15.2.16-336-g57544da1-1focal ceph-test=15.2.16-336-g57544da1-1focal radosgw=15.2.16-336-g57544da1-1focal python3-rados=15.2.16-336-g57544da1-1focal python3-rgw=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal python3-rbd=15.2.16-336-g57544da1-1focal libcephfs2=15.2.16-336-g57544da1-1focal libcephfs-dev=15.2.16-336-g57544da1-1focal librados2=15.2.16-336-g57544da1-1focal librbd1=15.2.16-336-g57544da1-1focal rbd-fuse=15.2.16-336-g57544da1-1focal python3-cephfs=15.2.16-336-g57544da1-1focal cephfs-shell=15.2.16-336-g57544da1-1focal 2022-07-20T03:04:23.916 INFO:teuthology.orchestra.run.smithi124.stdout:Get:40 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T03:04:23.943 INFO:teuthology.orchestra.run.smithi061.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T03:04:23.946 INFO:teuthology.orchestra.run.smithi061.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T03:04:23.951 INFO:teuthology.orchestra.run.smithi061.stdout:Get:43 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T03:04:23.962 INFO:teuthology.orchestra.run.smithi163.stdout:Reading package lists... 2022-07-20T03:04:23.977 INFO:teuthology.orchestra.run.smithi124.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T03:04:23.985 INFO:teuthology.orchestra.run.smithi124.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T03:04:23.986 INFO:teuthology.orchestra.run.smithi061.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T03:04:23.986 INFO:teuthology.orchestra.run.smithi061.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T03:04:24.011 INFO:teuthology.orchestra.run.smithi061.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T03:04:24.012 INFO:teuthology.orchestra.run.smithi061.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T03:04:24.014 INFO:teuthology.orchestra.run.smithi124.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T03:04:24.015 INFO:teuthology.orchestra.run.smithi124.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T03:04:24.016 INFO:teuthology.orchestra.run.smithi124.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T03:04:24.017 INFO:teuthology.orchestra.run.smithi124.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T03:04:24.018 INFO:teuthology.orchestra.run.smithi061.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T03:04:24.018 INFO:teuthology.orchestra.run.smithi061.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T03:04:24.019 INFO:teuthology.orchestra.run.smithi061.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T03:04:24.020 INFO:teuthology.orchestra.run.smithi124.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T03:04:24.020 INFO:teuthology.orchestra.run.smithi124.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T03:04:24.023 INFO:teuthology.orchestra.run.smithi124.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T03:04:24.050 INFO:teuthology.orchestra.run.smithi061.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T03:04:24.051 INFO:teuthology.orchestra.run.smithi061.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T03:04:24.067 INFO:teuthology.orchestra.run.smithi124.stdout:Get:50 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T03:04:24.081 INFO:teuthology.orchestra.run.smithi124.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T03:04:24.083 INFO:teuthology.orchestra.run.smithi124.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T03:04:24.137 INFO:teuthology.orchestra.run.smithi061.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T03:04:24.137 INFO:teuthology.orchestra.run.smithi061.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T03:04:24.138 INFO:teuthology.orchestra.run.smithi163.stdout:Building dependency tree... 2022-07-20T03:04:24.144 INFO:teuthology.orchestra.run.smithi163.stdout:Reading state information... 2022-07-20T03:04:24.144 INFO:teuthology.orchestra.run.smithi061.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T03:04:24.144 INFO:teuthology.orchestra.run.smithi061.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T03:04:24.145 INFO:teuthology.orchestra.run.smithi061.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T03:04:24.149 INFO:teuthology.orchestra.run.smithi061.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T03:04:24.150 INFO:teuthology.orchestra.run.smithi061.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T03:04:24.152 INFO:teuthology.orchestra.run.smithi061.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T03:04:24.162 INFO:teuthology.orchestra.run.smithi124.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T03:04:24.166 INFO:teuthology.orchestra.run.smithi124.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T03:04:24.170 INFO:teuthology.orchestra.run.smithi124.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T03:04:24.173 INFO:teuthology.orchestra.run.smithi124.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T03:04:24.175 INFO:teuthology.orchestra.run.smithi124.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T03:04:24.177 INFO:teuthology.orchestra.run.smithi124.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T03:04:24.178 INFO:teuthology.orchestra.run.smithi124.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T03:04:24.180 INFO:teuthology.orchestra.run.smithi124.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T03:04:24.211 INFO:teuthology.orchestra.run.smithi061.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T03:04:24.217 INFO:teuthology.orchestra.run.smithi061.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T03:04:24.241 INFO:teuthology.orchestra.run.smithi124.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T03:04:24.248 INFO:teuthology.orchestra.run.smithi124.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T03:04:24.291 INFO:teuthology.orchestra.run.smithi061.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T03:04:24.293 INFO:teuthology.orchestra.run.smithi061.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T03:04:24.294 INFO:teuthology.orchestra.run.smithi061.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T03:04:24.296 INFO:teuthology.orchestra.run.smithi163.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:04:24.296 INFO:teuthology.orchestra.run.smithi163.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:04:24.297 INFO:teuthology.orchestra.run.smithi163.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:04:24.297 INFO:teuthology.orchestra.run.smithi163.stdout:The following additional packages will be installed: 2022-07-20T03:04:24.298 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-base ceph-mgr-cephadm ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:24.299 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:24.299 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon ceph-osd jq libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 2022-07-20T03:04:24.299 INFO:teuthology.orchestra.run.smithi163.stdout: libleveldb1d liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 2022-07-20T03:04:24.299 INFO:teuthology.orchestra.run.smithi163.stdout: librdkafka1 librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl 2022-07-20T03:04:24.300 INFO:teuthology.orchestra.run.smithi163.stdout: python3-atomicwrites python3-attr python3-bcrypt python3-blinker python3-bs4 2022-07-20T03:04:24.300 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cachetools python3-ceph-argparse python3-ceph-common 2022-07-20T03:04:24.300 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cherrypy3 python3-cmd2 python3-colorama python3-cryptography 2022-07-20T03:04:24.300 INFO:teuthology.orchestra.run.smithi163.stdout: python3-decorator python3-google-auth python3-html5lib python3-jinja2 2022-07-20T03:04:24.300 INFO:teuthology.orchestra.run.smithi163.stdout: python3-joblib python3-jwt python3-kubernetes python3-logutils python3-lxml 2022-07-20T03:04:24.301 INFO:teuthology.orchestra.run.smithi163.stdout: python3-mako python3-markupsafe python3-oauthlib python3-openssl 2022-07-20T03:04:24.301 INFO:teuthology.orchestra.run.smithi163.stdout: python3-packaging python3-paste python3-pastedeploy python3-pastescript 2022-07-20T03:04:24.301 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pecan python3-pluggy python3-prettytable python3-psutil python3-py 2022-07-20T03:04:24.301 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1 python3-pyasn1-modules python3-pyinotify python3-pyperclip 2022-07-20T03:04:24.302 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pytest python3-repoze.lru python3-requests-oauthlib python3-routes 2022-07-20T03:04:24.302 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rsa python3-scipy python3-simplegeneric python3-simplejson 2022-07-20T03:04:24.302 INFO:teuthology.orchestra.run.smithi163.stdout: python3-singledispatch python3-sklearn python3-sklearn-lib python3-soupsieve 2022-07-20T03:04:24.303 INFO:teuthology.orchestra.run.smithi163.stdout: python3-tempita python3-waitress python3-wcwidth python3-webencodings 2022-07-20T03:04:24.303 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webob python3-websocket python3-webtest python3-werkzeug socat 2022-07-20T03:04:24.303 INFO:teuthology.orchestra.run.smithi163.stdout: xmlstarlet 2022-07-20T03:04:24.303 INFO:teuthology.orchestra.run.smithi163.stdout:Suggested packages: 2022-07-20T03:04:24.304 INFO:teuthology.orchestra.run.smithi163.stdout: python3-influxdb python-attr-doc python-blinker-doc python-cryptography-doc 2022-07-20T03:04:24.304 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cryptography-vectors python3-genshi python-jinja2-doc python3-crypto 2022-07-20T03:04:24.304 INFO:teuthology.orchestra.run.smithi163.stdout: python3-lxml-dbg python-lxml-doc python3-beaker python-mako-doc 2022-07-20T03:04:24.305 INFO:teuthology.orchestra.run.smithi163.stdout: python-openssl-doc python3-openssl-dbg httpd-wsgi libapache2-mod-python 2022-07-20T03:04:24.305 INFO:teuthology.orchestra.run.smithi163.stdout: libapache2-mod-scgi libjs-mochikit python-pecan-doc python-psutil-doc 2022-07-20T03:04:24.305 INFO:teuthology.orchestra.run.smithi163.stdout: subversion python-pyinotify-doc python-scipy-doc python3-dap 2022-07-20T03:04:24.305 INFO:teuthology.orchestra.run.smithi163.stdout: python-sklearn-doc ipython3 python-waitress-doc python-webob-doc 2022-07-20T03:04:24.306 INFO:teuthology.orchestra.run.smithi163.stdout: python-webtest-doc python-werkzeug-doc python3-termcolor python3-watchdog 2022-07-20T03:04:24.306 INFO:teuthology.orchestra.run.smithi163.stdout:Recommended packages: 2022-07-20T03:04:24.306 INFO:teuthology.orchestra.run.smithi163.stdout: btrfs-tools 2022-07-20T03:04:24.307 INFO:teuthology.orchestra.run.smithi061.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T03:04:24.315 INFO:teuthology.orchestra.run.smithi061.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T03:04:24.315 INFO:teuthology.orchestra.run.smithi061.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T03:04:24.315 INFO:teuthology.orchestra.run.smithi061.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T03:04:24.315 INFO:teuthology.orchestra.run.smithi061.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T03:04:24.321 INFO:teuthology.orchestra.run.smithi124.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T03:04:24.322 INFO:teuthology.orchestra.run.smithi124.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T03:04:24.322 INFO:teuthology.orchestra.run.smithi124.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T03:04:24.324 INFO:teuthology.orchestra.run.smithi124.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T03:04:24.344 INFO:teuthology.orchestra.run.smithi124.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T03:04:24.344 INFO:teuthology.orchestra.run.smithi124.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T03:04:24.344 INFO:teuthology.orchestra.run.smithi124.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T03:04:24.345 INFO:teuthology.orchestra.run.smithi124.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T03:04:24.371 INFO:teuthology.orchestra.run.smithi061.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T03:04:24.373 INFO:teuthology.orchestra.run.smithi061.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T03:04:24.401 INFO:teuthology.orchestra.run.smithi124.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T03:04:24.402 INFO:teuthology.orchestra.run.smithi163.stdout:The following NEW packages will be installed: 2022-07-20T03:04:24.402 INFO:teuthology.orchestra.run.smithi163.stdout: ceph ceph-base ceph-common ceph-fuse ceph-mds ceph-mgr ceph-mgr-cephadm 2022-07-20T03:04:24.402 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-dashboard ceph-mgr-diskprediction-cloud 2022-07-20T03:04:24.403 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mgr-diskprediction-local ceph-mgr-k8sevents ceph-mgr-modules-core 2022-07-20T03:04:24.403 INFO:teuthology.orchestra.run.smithi163.stdout: ceph-mon ceph-osd ceph-test cephadm cephfs-shell jq libcephfs-dev libcephfs2 2022-07-20T03:04:24.404 INFO:teuthology.orchestra.run.smithi163.stdout: libjq1 libjs-sphinxdoc libjs-underscore liblbfgsb0 libleveldb1d 2022-07-20T03:04:24.404 INFO:teuthology.orchestra.run.smithi163.stdout: liblttng-ust0 liboath0 libonig5 librabbitmq4 libradosstriper1 librdkafka1 2022-07-20T03:04:24.405 INFO:teuthology.orchestra.run.smithi124.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T03:04:24.406 INFO:teuthology.orchestra.run.smithi163.stdout: librgw2 libxslt1.1 nvme-cli python-pastedeploy-tpl python3-atomicwrites 2022-07-20T03:04:24.406 INFO:teuthology.orchestra.run.smithi163.stdout: python3-attr python3-bcrypt python3-blinker python3-bs4 python3-cachetools 2022-07-20T03:04:24.408 INFO:teuthology.orchestra.run.smithi163.stdout: python3-ceph-argparse python3-ceph-common python3-cephfs python3-cherrypy3 2022-07-20T03:04:24.409 INFO:teuthology.orchestra.run.smithi163.stdout: python3-cmd2 python3-colorama python3-cryptography python3-decorator 2022-07-20T03:04:24.410 INFO:teuthology.orchestra.run.smithi163.stdout: python3-google-auth python3-html5lib python3-jinja2 python3-joblib 2022-07-20T03:04:24.411 INFO:teuthology.orchestra.run.smithi163.stdout: python3-jwt python3-kubernetes python3-logutils python3-lxml python3-mako 2022-07-20T03:04:24.411 INFO:teuthology.orchestra.run.smithi163.stdout: python3-markupsafe python3-oauthlib python3-openssl python3-packaging 2022-07-20T03:04:24.412 INFO:teuthology.orchestra.run.smithi163.stdout: python3-paste python3-pastedeploy python3-pastescript python3-pecan 2022-07-20T03:04:24.412 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pluggy python3-prettytable python3-psutil python3-py python3-pyasn1 2022-07-20T03:04:24.413 INFO:teuthology.orchestra.run.smithi163.stdout: python3-pyasn1-modules python3-pyinotify python3-pyperclip python3-pytest 2022-07-20T03:04:24.413 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rados python3-rbd python3-repoze.lru python3-requests-oauthlib 2022-07-20T03:04:24.413 INFO:teuthology.orchestra.run.smithi163.stdout: python3-rgw python3-routes python3-rsa python3-scipy python3-simplegeneric 2022-07-20T03:04:24.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-simplejson python3-singledispatch python3-sklearn 2022-07-20T03:04:24.414 INFO:teuthology.orchestra.run.smithi163.stdout: python3-sklearn-lib python3-soupsieve python3-tempita python3-waitress 2022-07-20T03:04:24.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-wcwidth python3-webencodings python3-webob python3-websocket 2022-07-20T03:04:24.415 INFO:teuthology.orchestra.run.smithi163.stdout: python3-webtest python3-werkzeug radosgw rbd-fuse socat xmlstarlet 2022-07-20T03:04:24.416 INFO:teuthology.orchestra.run.smithi163.stdout:The following packages will be upgraded: 2022-07-20T03:04:24.416 INFO:teuthology.orchestra.run.smithi163.stdout: librados2 librbd1 2022-07-20T03:04:24.426 INFO:teuthology.orchestra.run.smithi117.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T03:04:24.427 INFO:teuthology.orchestra.run.smithi117.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T03:04:24.434 INFO:teuthology.orchestra.run.smithi117.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T03:04:24.450 INFO:teuthology.orchestra.run.smithi117.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T03:04:24.451 INFO:teuthology.orchestra.run.smithi117.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T03:04:24.452 INFO:teuthology.orchestra.run.smithi061.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T03:04:24.454 INFO:teuthology.orchestra.run.smithi117.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T03:04:24.455 INFO:teuthology.orchestra.run.smithi061.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T03:04:24.467 INFO:teuthology.orchestra.run.smithi163.stdout:2 upgraded, 101 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:04:24.467 INFO:teuthology.orchestra.run.smithi163.stdout:Need to get 153 MB of archives. 2022-07-20T03:04:24.467 INFO:teuthology.orchestra.run.smithi163.stdout:After this operation, 699 MB of additional disk space will be used. 2022-07-20T03:04:24.468 INFO:teuthology.orchestra.run.smithi163.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyperclip all 1.7.0-1 [10.5 kB] 2022-07-20T03:04:24.480 INFO:teuthology.orchestra.run.smithi124.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T03:04:24.484 INFO:teuthology.orchestra.run.smithi124.stdout:Get:74 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T03:04:24.485 INFO:teuthology.orchestra.run.smithi163.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-wcwidth all 0.1.8+dfsg1-3 [17.4 kB] 2022-07-20T03:04:24.494 INFO:teuthology.orchestra.run.smithi163.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cmd2 all 0.8.5-2.1 [49.6 kB] 2022-07-20T03:04:24.522 INFO:teuthology.orchestra.run.smithi163.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/main amd64 liblttng-ust0 amd64 2.11.0-1 [160 kB] 2022-07-20T03:04:24.550 INFO:teuthology.orchestra.run.smithi163.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-prettytable all 0.7.2-5 [20.1 kB] 2022-07-20T03:04:24.551 INFO:teuthology.orchestra.run.smithi163.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 librabbitmq4 amd64 0.10.0-1 [35.8 kB] 2022-07-20T03:04:24.558 INFO:teuthology.orchestra.run.smithi163.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/universe amd64 librdkafka1 amd64 1.2.1-1ubuntu1 [410 kB] 2022-07-20T03:04:24.578 INFO:teuthology.orchestra.run.smithi163.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libleveldb1d amd64 1.22-3ubuntu2 [139 kB] 2022-07-20T03:04:24.581 INFO:teuthology.orchestra.run.smithi117.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T03:04:24.582 INFO:teuthology.orchestra.run.smithi117.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T03:04:24.583 INFO:teuthology.orchestra.run.smithi163.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 liboath0 amd64 2.6.1-1.3 [44.7 kB] 2022-07-20T03:04:24.583 INFO:teuthology.orchestra.run.smithi163.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-cryptography amd64 2.8-3ubuntu0.1 [211 kB] 2022-07-20T03:04:24.589 INFO:teuthology.orchestra.run.smithi163.stdout:Get:11 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-openssl all 19.0.0-1build1 [43.3 kB] 2022-07-20T03:04:24.589 INFO:teuthology.orchestra.run.smithi163.stdout:Get:12 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bcrypt amd64 3.1.7-2ubuntu1 [30.4 kB] 2022-07-20T03:04:24.590 INFO:teuthology.orchestra.run.smithi163.stdout:Get:13 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cherrypy3 all 8.9.1-6 [160 kB] 2022-07-20T03:04:24.593 INFO:teuthology.orchestra.run.smithi163.stdout:Get:14 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-jwt all 1.7.1-2ubuntu2.1 [18.0 kB] 2022-07-20T03:04:24.596 INFO:teuthology.orchestra.run.smithi163.stdout:Get:15 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-logutils all 0.3.3-7 [17.3 kB] 2022-07-20T03:04:24.598 INFO:teuthology.orchestra.run.smithi163.stdout:Get:16 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-markupsafe amd64 1.1.0-1build2 [13.9 kB] 2022-07-20T03:04:24.602 INFO:teuthology.orchestra.run.smithi163.stdout:Get:17 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-mako all 1.1.0+ds1-1ubuntu2 [59.1 kB] 2022-07-20T03:04:24.604 INFO:teuthology.orchestra.run.smithi163.stdout:Get:18 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplegeneric all 0.8.1-3 [11.3 kB] 2022-07-20T03:04:24.604 INFO:teuthology.orchestra.run.smithi163.stdout:Get:19 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-singledispatch all 3.4.0.3-2build1 [7,128 B] 2022-07-20T03:04:24.620 INFO:teuthology.orchestra.run.smithi163.stdout:Get:20 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webob all 1:1.8.5-2 [84.4 kB] 2022-07-20T03:04:24.621 INFO:teuthology.orchestra.run.smithi163.stdout:Get:21 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-soupsieve all 1.9.5+dfsg-1 [29.1 kB] 2022-07-20T03:04:24.622 INFO:teuthology.orchestra.run.smithi163.stdout:Get:22 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-bs4 all 4.8.2-1 [83.0 kB] 2022-07-20T03:04:24.624 INFO:teuthology.orchestra.run.smithi163.stdout:Get:23 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-waitress all 1.4.1-1ubuntu0.1 [81.4 kB] 2022-07-20T03:04:24.626 INFO:teuthology.orchestra.run.smithi163.stdout:Get:24 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-tempita all 0.5.2-6 [14.8 kB] 2022-07-20T03:04:24.626 INFO:teuthology.orchestra.run.smithi163.stdout:Get:25 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-paste all 3.4.0+dfsg1-1 [456 kB] 2022-07-20T03:04:24.634 INFO:teuthology.orchestra.run.smithi163.stdout:Get:26 http://archive.ubuntu.com/ubuntu focal/main amd64 python-pastedeploy-tpl all 2.1.0-1 [5,016 B] 2022-07-20T03:04:24.638 INFO:teuthology.orchestra.run.smithi163.stdout:Get:27 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastedeploy all 2.1.0-1 [26.5 kB] 2022-07-20T03:04:24.638 INFO:teuthology.orchestra.run.smithi163.stdout:Get:28 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webtest all 2.0.32-2 [28.2 kB] 2022-07-20T03:04:24.639 INFO:teuthology.orchestra.run.smithi163.stdout:Get:29 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pecan all 1.3.3-3ubuntu1 [85.8 kB] 2022-07-20T03:04:24.654 INFO:teuthology.orchestra.run.smithi163.stdout:Get:30 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-werkzeug all 0.16.1+dfsg1-2 [183 kB] 2022-07-20T03:04:24.656 INFO:teuthology.orchestra.run.smithi163.stdout:Get:31 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-jinja2 all 2.10.1-2 [95.5 kB] 2022-07-20T03:04:24.658 INFO:teuthology.orchestra.run.smithi163.stdout:Get:32 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-repoze.lru all 0.7-2 [12.1 kB] 2022-07-20T03:04:24.658 INFO:teuthology.orchestra.run.smithi163.stdout:Get:33 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libjs-underscore all 1.9.1~dfsg-1ubuntu0.20.04.1 [99.5 kB] 2022-07-20T03:04:24.660 INFO:teuthology.orchestra.run.smithi163.stdout:Get:34 http://archive.ubuntu.com/ubuntu focal/main amd64 libjs-sphinxdoc all 1.8.5-7ubuntu3 [97.1 kB] 2022-07-20T03:04:24.662 INFO:teuthology.orchestra.run.smithi163.stdout:Get:35 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-routes all 2.4.1-2 [88.1 kB] 2022-07-20T03:04:24.663 INFO:teuthology.orchestra.run.smithi163.stdout:Get:36 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-decorator all 4.4.2-0ubuntu1 [10.3 kB] 2022-07-20T03:04:24.671 INFO:teuthology.orchestra.run.smithi163.stdout:Get:37 http://archive.ubuntu.com/ubuntu focal/universe amd64 liblbfgsb0 amd64 3.0+dfsg.3-7build1 [29.5 kB] 2022-07-20T03:04:24.674 INFO:teuthology.orchestra.run.smithi163.stdout:Get:38 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-scipy amd64 1.3.3-3build1 [11.1 MB] 2022-07-20T03:04:24.751 INFO:teuthology.orchestra.run.smithi163.stdout:Get:39 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn-lib amd64 0.22.2.post1+dfsg-5 [2,358 kB] 2022-07-20T03:04:24.764 INFO:teuthology.orchestra.run.smithi163.stdout:Get:40 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librbd1 amd64 15.2.16-336-g57544da1-1focal [1,698 kB] 2022-07-20T03:04:24.765 INFO:teuthology.orchestra.run.smithi163.stdout:Get:41 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-joblib all 0.14.0-3 [192 kB] 2022-07-20T03:04:24.767 INFO:teuthology.orchestra.run.smithi163.stdout:Get:42 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-sklearn all 0.22.2.post1+dfsg-5 [1,773 kB] 2022-07-20T03:04:24.778 INFO:teuthology.orchestra.run.smithi163.stdout:Get:43 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-cachetools all 4.0.0-1 [9,820 B] 2022-07-20T03:04:24.828 INFO:teuthology.orchestra.run.smithi163.stdout:Get:44 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1 all 0.4.2-3build1 [46.7 kB] 2022-07-20T03:04:24.830 INFO:teuthology.orchestra.run.smithi163.stdout:Get:45 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyasn1-modules all 0.2.1-0.2build1 [32.9 kB] 2022-07-20T03:04:24.830 INFO:teuthology.orchestra.run.smithi163.stdout:Get:46 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-rsa all 4.0-3 [24.7 kB] 2022-07-20T03:04:24.832 INFO:teuthology.orchestra.run.smithi163.stdout:Get:47 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-google-auth all 1.5.1-2 [35.6 kB] 2022-07-20T03:04:24.832 INFO:teuthology.orchestra.run.smithi163.stdout:Get:48 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-blinker all 1.4+dfsg1-0.3ubuntu1 [13.2 kB] 2022-07-20T03:04:24.833 INFO:teuthology.orchestra.run.smithi163.stdout:Get:49 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-oauthlib all 3.1.0-1ubuntu2 [84.8 kB] 2022-07-20T03:04:24.834 INFO:teuthology.orchestra.run.smithi163.stdout:Get:50 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-requests-oauthlib all 1.0.0-1.1build1 [16.8 kB] 2022-07-20T03:04:24.835 INFO:teuthology.orchestra.run.smithi163.stdout:Get:51 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-websocket all 0.53.0-2ubuntu1 [32.3 kB] 2022-07-20T03:04:24.846 INFO:teuthology.orchestra.run.smithi163.stdout:Get:52 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-kubernetes all 7.0.0~a1-2 [271 kB] 2022-07-20T03:04:24.853 INFO:teuthology.orchestra.run.smithi163.stdout:Get:53 http://archive.ubuntu.com/ubuntu focal/universe amd64 libonig5 amd64 6.9.4-1 [142 kB] 2022-07-20T03:04:24.857 INFO:teuthology.orchestra.run.smithi163.stdout:Get:54 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 libjq1 amd64 1.6-1ubuntu0.20.04.1 [121 kB] 2022-07-20T03:04:24.858 INFO:teuthology.orchestra.run.smithi163.stdout:Get:55 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 jq amd64 1.6-1ubuntu0.20.04.1 [50.2 kB] 2022-07-20T03:04:24.859 INFO:teuthology.orchestra.run.smithi163.stdout:Get:56 http://archive.ubuntu.com/ubuntu focal/main amd64 socat amd64 1.7.3.3-2 [323 kB] 2022-07-20T03:04:24.866 INFO:teuthology.orchestra.run.smithi163.stdout:Get:57 http://archive.ubuntu.com/ubuntu focal/main amd64 libxslt1.1 amd64 1.1.34-4 [152 kB] 2022-07-20T03:04:24.866 INFO:teuthology.orchestra.run.smithi163.stdout:Get:58 http://archive.ubuntu.com/ubuntu focal/universe amd64 xmlstarlet amd64 1.6.1-2build1 [267 kB] 2022-07-20T03:04:24.867 INFO:teuthology.orchestra.run.smithi163.stdout:Get:59 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-colorama all 0.4.3-1build1 [23.9 kB] 2022-07-20T03:04:24.867 INFO:teuthology.orchestra.run.smithi163.stdout:Get:60 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 nvme-cli amd64 1.9-1ubuntu0.1 [283 kB] 2022-07-20T03:04:24.870 INFO:teuthology.orchestra.run.smithi163.stdout:Get:61 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-atomicwrites all 1.1.5-2build1 [7,016 B] 2022-07-20T03:04:24.882 INFO:teuthology.orchestra.run.smithi163.stdout:Get:62 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-attr all 19.3.0-2 [33.9 kB] 2022-07-20T03:04:24.883 INFO:teuthology.orchestra.run.smithi163.stdout:Get:63 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-webencodings all 0.5.1-1ubuntu1 [11.0 kB] 2022-07-20T03:04:24.883 INFO:teuthology.orchestra.run.smithi163.stdout:Get:64 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-html5lib all 1.0.1-2 [84.3 kB] 2022-07-20T03:04:24.883 INFO:teuthology.orchestra.run.smithi163.stdout:Get:65 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 python3-lxml amd64 4.5.0-1ubuntu0.5 [1,384 kB] 2022-07-20T03:04:24.901 INFO:teuthology.orchestra.run.smithi163.stdout:Get:66 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-packaging all 20.3-1 [26.8 kB] 2022-07-20T03:04:24.902 INFO:teuthology.orchestra.run.smithi163.stdout:Get:67 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pastescript all 2.0.2-4 [54.6 kB] 2022-07-20T03:04:24.902 INFO:teuthology.orchestra.run.smithi163.stdout:Get:68 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pluggy all 0.13.0-2 [18.4 kB] 2022-07-20T03:04:24.902 INFO:teuthology.orchestra.run.smithi163.stdout:Get:69 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-psutil amd64 5.5.1-1ubuntu4 [141 kB] 2022-07-20T03:04:24.902 INFO:teuthology.orchestra.run.smithi163.stdout:Get:70 http://archive.ubuntu.com/ubuntu focal-updates/universe amd64 python3-py all 1.8.1-1ubuntu0.1 [65.6 kB] 2022-07-20T03:04:24.902 INFO:teuthology.orchestra.run.smithi163.stdout:Get:71 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-pyinotify all 0.9.6-1.2ubuntu1 [24.8 kB] 2022-07-20T03:04:24.904 INFO:teuthology.orchestra.run.smithi117.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T03:04:24.905 INFO:teuthology.orchestra.run.smithi117.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T03:04:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:Get:72 http://archive.ubuntu.com/ubuntu focal/universe amd64 python3-pytest all 4.6.9-1 [178 kB] 2022-07-20T03:04:24.915 INFO:teuthology.orchestra.run.smithi163.stdout:Get:73 http://archive.ubuntu.com/ubuntu focal/main amd64 python3-simplejson amd64 3.16.0-2ubuntu2 [50.8 kB] 2022-07-20T03:04:25.183 INFO:teuthology.orchestra.run.smithi179.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T03:04:25.354 INFO:teuthology.orchestra.run.smithi163.stdout:Get:74 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librados2 amd64 15.2.16-336-g57544da1-1focal [3,257 kB] 2022-07-20T03:04:25.417 INFO:teuthology.orchestra.run.smithi179.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T03:04:25.570 INFO:teuthology.orchestra.run.smithi179.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T03:04:25.571 INFO:teuthology.orchestra.run.smithi179.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T03:04:25.630 INFO:teuthology.orchestra.run.smithi179.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T03:04:25.666 INFO:teuthology.orchestra.run.smithi179.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T03:04:25.809 INFO:teuthology.orchestra.run.smithi179.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T03:04:26.011 INFO:teuthology.orchestra.run.smithi163.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T03:04:26.079 INFO:teuthology.orchestra.run.smithi163.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T03:04:26.144 INFO:teuthology.orchestra.run.smithi163.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T03:04:26.145 INFO:teuthology.orchestra.run.smithi163.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T03:04:26.202 INFO:teuthology.orchestra.run.smithi163.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T03:04:26.204 INFO:teuthology.orchestra.run.smithi163.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T03:04:26.269 INFO:teuthology.orchestra.run.smithi163.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T03:04:26.353 INFO:teuthology.orchestra.run.smithi124.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T03:04:26.357 INFO:teuthology.orchestra.run.smithi061.stdout:Get:75 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs2 amd64 15.2.16-336-g57544da1-1focal [463 kB] 2022-07-20T03:04:26.615 INFO:teuthology.orchestra.run.smithi124.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T03:04:26.747 INFO:teuthology.orchestra.run.smithi061.stdout:Get:76 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rados amd64 15.2.16-336-g57544da1-1focal [365 kB] 2022-07-20T03:04:26.862 INFO:teuthology.orchestra.run.smithi163.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T03:04:26.878 INFO:teuthology.orchestra.run.smithi124.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T03:04:26.885 INFO:teuthology.orchestra.run.smithi124.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T03:04:26.892 INFO:teuthology.orchestra.run.smithi163.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T03:04:26.953 INFO:teuthology.orchestra.run.smithi163.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T03:04:26.957 INFO:teuthology.orchestra.run.smithi179.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T03:04:26.959 INFO:teuthology.orchestra.run.smithi124.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T03:04:26.997 INFO:teuthology.orchestra.run.smithi124.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T03:04:27.001 INFO:teuthology.orchestra.run.smithi117.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T03:04:27.003 INFO:teuthology.orchestra.run.smithi061.stdout:Get:77 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-argparse all 15.2.16-336-g57544da1-1focal [20.9 kB] 2022-07-20T03:04:27.003 INFO:teuthology.orchestra.run.smithi117.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T03:04:27.019 INFO:teuthology.orchestra.run.smithi179.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T03:04:27.047 INFO:teuthology.orchestra.run.smithi061.stdout:Get:78 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-cephfs amd64 15.2.16-336-g57544da1-1focal [156 kB] 2022-07-20T03:04:27.064 INFO:teuthology.orchestra.run.smithi117.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T03:04:27.099 INFO:teuthology.orchestra.run.smithi061.stdout:Get:79 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-ceph-common all 15.2.16-336-g57544da1-1focal [29.4 kB] 2022-07-20T03:04:27.106 INFO:teuthology.orchestra.run.smithi179.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T03:04:27.160 INFO:teuthology.orchestra.run.smithi061.stdout:Get:80 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rbd amd64 15.2.16-336-g57544da1-1focal [362 kB] 2022-07-20T03:04:27.162 INFO:teuthology.orchestra.run.smithi124.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T03:04:27.392 INFO:teuthology.orchestra.run.smithi117.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T03:04:27.403 INFO:teuthology.orchestra.run.smithi061.stdout:Get:81 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 librgw2 amd64 15.2.16-336-g57544da1-1focal [3,145 kB] 2022-07-20T03:04:27.975 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 153 MB in 6s (26.4 MB/s) 2022-07-20T03:04:27.981 INFO:teuthology.orchestra.run.smithi124.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T03:04:27.995 INFO:teuthology.orchestra.run.smithi124.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T03:04:28.067 INFO:teuthology.orchestra.run.smithi124.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T03:04:28.149 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T03:04:28.185 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T03:04:28.188 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T03:04:28.200 INFO:teuthology.orchestra.run.smithi061.stdout:Get:82 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 python3-rgw amd64 15.2.16-336-g57544da1-1focal [116 kB] 2022-07-20T03:04:28.226 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T03:04:28.227 INFO:teuthology.orchestra.run.smithi061.stdout:Get:83 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libradosstriper1 amd64 15.2.16-336-g57544da1-1focal [361 kB] 2022-07-20T03:04:28.314 INFO:teuthology.orchestra.run.smithi061.stdout:Get:84 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-common amd64 15.2.16-336-g57544da1-1focal [20.1 MB] 2022-07-20T03:04:28.529 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T03:04:28.542 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T03:04:28.588 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:04:28.679 INFO:teuthology.orchestra.run.smithi163.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T03:04:28.916 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T03:04:28.928 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T03:04:28.964 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:04:29.018 INFO:teuthology.orchestra.run.smithi163.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T03:04:29.022 INFO:teuthology.orchestra.run.smithi163.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T03:04:29.135 INFO:teuthology.orchestra.run.smithi163.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T03:04:29.260 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T03:04:29.264 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T03:04:29.326 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:04:29.597 INFO:teuthology.orchestra.run.smithi163.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T03:04:29.676 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:29.804 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:30.198 INFO:teuthology.orchestra.run.smithi179.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T03:04:30.352 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:30.468 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:30.854 INFO:teuthology.orchestra.run.smithi179.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T03:04:30.887 INFO:teuthology.orchestra.run.smithi179.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T03:04:31.062 INFO:teuthology.orchestra.run.smithi179.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T03:04:31.070 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T03:04:31.084 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:31.120 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:31.432 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rados. 2022-07-20T03:04:31.443 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:31.482 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:31.581 INFO:teuthology.orchestra.run.smithi163.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T03:04:31.629 INFO:teuthology.orchestra.run.smithi163.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T03:04:31.640 INFO:teuthology.orchestra.run.smithi163.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T03:04:31.722 INFO:teuthology.orchestra.run.smithi124.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T03:04:31.759 INFO:teuthology.orchestra.run.smithi163.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T03:04:31.761 INFO:teuthology.orchestra.run.smithi163.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T03:04:31.762 INFO:teuthology.orchestra.run.smithi163.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T03:04:31.825 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T03:04:31.839 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:31.879 INFO:teuthology.orchestra.run.smithi179.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T03:04:31.886 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:32.052 INFO:teuthology.orchestra.run.smithi163.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T03:04:32.052 INFO:teuthology.orchestra.run.smithi163.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T03:04:32.160 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T03:04:32.167 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:32.203 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:32.409 INFO:teuthology.orchestra.run.smithi061.stdout:Get:85 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-base amd64 15.2.16-336-g57544da1-1focal [5,347 kB] 2022-07-20T03:04:32.497 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T03:04:32.513 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:32.553 INFO:teuthology.orchestra.run.smithi124.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T03:04:32.559 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:32.564 INFO:teuthology.orchestra.run.smithi124.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T03:04:32.751 INFO:teuthology.orchestra.run.smithi163.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T03:04:32.752 INFO:teuthology.orchestra.run.smithi163.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T03:04:32.794 INFO:teuthology.orchestra.run.smithi124.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T03:04:32.896 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T03:04:32.906 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T03:04:32.945 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T03:04:33.194 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T03:04:33.215 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:33.248 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:33.290 INFO:teuthology.orchestra.run.smithi061.stdout:Get:86 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-modules-core all 15.2.16-336-g57544da1-1focal [163 kB] 2022-07-20T03:04:33.329 INFO:teuthology.orchestra.run.smithi061.stdout:Get:87 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr amd64 15.2.16-336-g57544da1-1focal [1,402 kB] 2022-07-20T03:04:33.578 INFO:teuthology.orchestra.run.smithi061.stdout:Get:88 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mon amd64 15.2.16-336-g57544da1-1focal [6,164 kB] 2022-07-20T03:04:33.580 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T03:04:33.595 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T03:04:33.630 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:04:33.792 INFO:teuthology.orchestra.run.smithi124.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T03:04:33.958 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T03:04:33.972 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T03:04:34.016 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:04:34.380 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package librgw2. 2022-07-20T03:04:34.398 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:34.435 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:34.666 INFO:teuthology.orchestra.run.smithi061.stdout:Get:89 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-osd amd64 15.2.16-336-g57544da1-1focal [24.0 MB] 2022-07-20T03:04:35.013 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T03:04:35.028 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:35.063 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:35.131 INFO:teuthology.orchestra.run.smithi179.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T03:04:35.131 INFO:teuthology.orchestra.run.smithi179.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T03:04:35.155 INFO:teuthology.orchestra.run.smithi179.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T03:04:35.340 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T03:04:35.355 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T03:04:35.392 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:04:35.405 INFO:teuthology.orchestra.run.smithi179.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T03:04:35.407 INFO:teuthology.orchestra.run.smithi179.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T03:04:35.419 INFO:teuthology.orchestra.run.smithi179.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T03:04:35.819 INFO:teuthology.orchestra.run.smithi179.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T03:04:35.820 INFO:teuthology.orchestra.run.smithi179.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T03:04:36.019 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liboath0. 2022-07-20T03:04:36.034 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T03:04:36.077 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T03:04:36.379 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T03:04:36.393 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:36.438 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:36.783 INFO:teuthology.orchestra.run.smithi179.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T03:04:36.783 INFO:teuthology.orchestra.run.smithi179.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T03:04:36.844 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-common. 2022-07-20T03:04:36.856 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:36.892 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:36.957 INFO:teuthology.orchestra.run.smithi124.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T03:04:36.957 INFO:teuthology.orchestra.run.smithi124.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T03:04:36.961 INFO:teuthology.orchestra.run.smithi163.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T03:04:36.962 INFO:teuthology.orchestra.run.smithi163.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T03:04:36.967 INFO:teuthology.orchestra.run.smithi124.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T03:04:37.016 INFO:teuthology.orchestra.run.smithi163.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T03:04:37.206 INFO:teuthology.orchestra.run.smithi124.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T03:04:37.207 INFO:teuthology.orchestra.run.smithi124.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T03:04:37.207 INFO:teuthology.orchestra.run.smithi124.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T03:04:37.567 INFO:teuthology.orchestra.run.smithi124.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T03:04:37.568 INFO:teuthology.orchestra.run.smithi124.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T03:04:38.259 INFO:teuthology.orchestra.run.smithi163.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T03:04:38.390 INFO:teuthology.orchestra.run.smithi124.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T03:04:38.395 INFO:teuthology.orchestra.run.smithi124.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T03:04:38.818 INFO:teuthology.orchestra.run.smithi061.stdout:Get:90 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph amd64 15.2.16-336-g57544da1-1focal [4,028 B] 2022-07-20T03:04:38.819 INFO:teuthology.orchestra.run.smithi061.stdout:Get:91 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-fuse amd64 15.2.16-336-g57544da1-1focal [537 kB] 2022-07-20T03:04:38.830 INFO:teuthology.orchestra.run.smithi061.stdout:Get:92 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mds amd64 15.2.16-336-g57544da1-1focal [2,006 kB] 2022-07-20T03:04:38.889 INFO:teuthology.orchestra.run.smithi163.stdout:Fetched 153 MB in 14s (11.0 MB/s) 2022-07-20T03:04:38.948 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-base. 2022-07-20T03:04:38.961 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:39.008 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:39.029 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T03:04:39.074 INFO:teuthology.orchestra.run.smithi163.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T03:04:39.075 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T03:04:39.077 INFO:teuthology.orchestra.run.smithi061.stdout:Get:93 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephadm amd64 15.2.16-336-g57544da1-1focal [53.2 kB] 2022-07-20T03:04:39.078 INFO:teuthology.orchestra.run.smithi061.stdout:Get:94 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-cephadm all 15.2.16-336-g57544da1-1focal [59.1 kB] 2022-07-20T03:04:39.079 INFO:teuthology.orchestra.run.smithi061.stdout:Get:95 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-dashboard all 15.2.16-336-g57544da1-1focal [3,218 kB] 2022-07-20T03:04:39.121 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T03:04:39.422 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T03:04:39.437 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T03:04:39.471 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:04:39.507 INFO:teuthology.orchestra.run.smithi061.stdout:Get:96 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-cloud all 15.2.16-336-g57544da1-1focal [34.4 kB] 2022-07-20T03:04:39.508 INFO:teuthology.orchestra.run.smithi061.stdout:Get:97 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-diskprediction-local all 15.2.16-336-g57544da1-1focal [7,891 kB] 2022-07-20T03:04:39.691 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T03:04:39.704 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T03:04:39.733 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:04:39.860 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T03:04:39.874 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T03:04:39.908 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:04:40.097 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T03:04:40.107 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T03:04:40.186 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:04:40.264 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T03:04:40.279 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T03:04:40.322 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T03:04:40.499 INFO:teuthology.orchestra.run.smithi061.stdout:Get:98 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-mgr-k8sevents all 15.2.16-336-g57544da1-1focal [4,144 B] 2022-07-20T03:04:40.516 INFO:teuthology.orchestra.run.smithi061.stdout:Get:99 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 ceph-test amd64 15.2.16-336-g57544da1-1focal [40.2 MB] 2022-07-20T03:04:40.540 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:40.614 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T03:04:40.628 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:40.666 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:40.673 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:40.883 INFO:teuthology.orchestra.run.smithi179.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T03:04:40.884 INFO:teuthology.orchestra.run.smithi179.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T03:04:40.885 INFO:teuthology.orchestra.run.smithi179.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T03:04:41.045 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T03:04:41.066 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T03:04:41.095 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:04:41.241 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:41.352 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:41.397 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T03:04:41.413 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T03:04:41.456 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:04:41.659 INFO:teuthology.orchestra.run.smithi179.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T03:04:41.794 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T03:04:41.808 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T03:04:41.851 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:04:41.919 INFO:teuthology.orchestra.run.smithi124.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T03:04:41.919 INFO:teuthology.orchestra.run.smithi124.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T03:04:41.920 INFO:teuthology.orchestra.run.smithi124.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T03:04:41.989 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T03:04:42.004 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:42.049 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:42.137 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T03:04:42.151 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T03:04:42.189 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T03:04:42.307 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 153 MB in 19s (8,057 kB/s) 2022-07-20T03:04:42.313 INFO:teuthology.orchestra.run.smithi124.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T03:04:42.375 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-rados. 2022-07-20T03:04:42.388 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:42.423 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T03:04:42.433 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:42.438 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T03:04:42.445 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T03:04:42.473 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:04:42.487 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T03:04:42.493 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T03:04:42.537 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T03:04:42.753 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T03:04:42.767 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:42.802 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:42.817 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-mako. 2022-07-20T03:04:42.830 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T03:04:42.837 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T03:04:42.850 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T03:04:42.867 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:04:42.887 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:04:42.898 INFO:teuthology.orchestra.run.smithi124.stdout:Fetched 153 MB in 20s (7,723 kB/s) 2022-07-20T03:04:43.018 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T03:04:43.060 INFO:teuthology.orchestra.run.smithi124.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T03:04:43.064 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T03:04:43.079 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T03:04:43.094 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:43.100 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T03:04:43.131 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:43.199 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T03:04:43.212 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T03:04:43.254 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T03:04:43.266 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:04:43.268 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T03:04:43.312 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:04:43.354 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T03:04:43.368 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T03:04:43.412 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:04:43.440 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T03:04:43.454 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:43.469 INFO:teuthology.orchestra.run.smithi061.stdout:Get:100 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 cephfs-shell all 15.2.16-336-g57544da1-1focal [15.9 kB] 2022-07-20T03:04:43.470 INFO:teuthology.orchestra.run.smithi061.stdout:Get:101 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 libcephfs-dev amd64 15.2.16-336-g57544da1-1focal [18.1 kB] 2022-07-20T03:04:43.471 INFO:teuthology.orchestra.run.smithi061.stdout:Get:102 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 radosgw amd64 15.2.16-336-g57544da1-1focal [9,667 kB] 2022-07-20T03:04:43.524 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:43.586 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T03:04:43.600 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T03:04:43.640 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T03:04:43.657 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T03:04:43.669 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:04:43.698 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:04:43.741 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T03:04:43.753 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T03:04:43.753 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T03:04:43.769 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T03:04:43.789 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:04:43.801 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T03:04:43.841 INFO:teuthology.orchestra.run.smithi061.stdout:Get:103 https://4.chacra.ceph.com/r/ceph/octopus/57544da114dbbe1c44f27b2ea18e3233e4e917e0/ubuntu/focal/flavors/default focal/main amd64 rbd-fuse amd64 15.2.16-336-g57544da1-1focal [61.4 kB] 2022-07-20T03:04:43.926 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webob. 2022-07-20T03:04:43.937 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:43.939 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T03:04:43.976 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T03:04:44.046 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T03:04:44.057 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:44.058 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:44.118 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T03:04:44.129 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:44.131 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T03:04:44.193 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:04:44.297 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T03:04:44.310 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T03:04:44.346 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:04:44.453 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 153 MB in 21s (7,158 kB/s) 2022-07-20T03:04:44.474 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T03:04:44.489 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T03:04:44.494 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:44.533 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:04:44.594 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T03:04:44.600 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyperclip. 2022-07-20T03:04:44.601 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T03:04:44.606 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:44.617 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:44.630 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T03:04:44.643 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 102338 files and directories currently installed.) 2022-07-20T03:04:44.646 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../000-python3-pyperclip_1.7.0-1_all.deb ... 2022-07-20T03:04:44.682 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyperclip (1.7.0-1) ... 2022-07-20T03:04:44.746 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:44.826 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T03:04:44.839 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T03:04:44.868 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:04:44.925 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T03:04:44.939 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T03:04:44.941 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-wcwidth. 2022-07-20T03:04:44.957 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../001-python3-wcwidth_0.1.8+dfsg1-3_all.deb ... 2022-07-20T03:04:44.989 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:04:44.995 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:04:45.168 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:45.179 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package librgw2. 2022-07-20T03:04:45.191 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:45.229 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:45.244 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T03:04:45.260 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T03:04:45.261 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cmd2. 2022-07-20T03:04:45.277 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../002-python3-cmd2_0.8.5-2.1_all.deb ... 2022-07-20T03:04:45.294 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:45.304 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T03:04:45.322 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:04:45.384 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T03:04:45.397 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:45.441 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:45.607 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-paste. 2022-07-20T03:04:45.620 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T03:04:45.664 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:04:45.697 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblttng-ust0:amd64. 2022-07-20T03:04:45.718 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../003-liblttng-ust0_2.11.0-1_amd64.deb ... 2022-07-20T03:04:45.735 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rados. 2022-07-20T03:04:45.748 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:45.787 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:45.793 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T03:04:45.807 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:45.808 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:04:45.865 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:45.910 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T03:04:45.914 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:45.968 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T03:04:45.982 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T03:04:45.990 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:46.023 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:04:46.088 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T03:04:46.103 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:46.138 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:46.283 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../004-librbd1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:46.285 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T03:04:46.294 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-rados. 2022-07-20T03:04:46.296 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T03:04:46.299 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T03:04:46.306 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:46.308 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T03:04:46.347 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:46.352 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:04:46.371 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:04:46.384 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librbd1 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:46.394 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T03:04:46.404 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:46.470 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:46.662 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package liboath0. 2022-07-20T03:04:46.676 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T03:04:46.680 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T03:04:46.694 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T03:04:46.696 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T03:04:46.704 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T03:04:46.726 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:46.730 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T03:04:46.756 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:46.859 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T03:04:46.868 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:46.885 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../005-librados2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:46.922 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.008 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T03:04:47.023 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:47.024 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T03:04:47.038 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:47.044 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T03:04:47.057 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T03:04:47.072 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librados2 (15.2.16-336-g57544da1-1focal) over (15.2.16-0ubuntu0.20.04.1) ... 2022-07-20T03:04:47.074 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.081 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.099 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:04:47.248 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T03:04:47.263 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T03:04:47.308 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T03:04:47.410 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T03:04:47.427 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:47.446 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T03:04:47.461 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T03:04:47.470 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.495 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:04:47.534 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T03:04:47.535 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-common. 2022-07-20T03:04:47.551 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:47.552 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:47.596 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.597 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.755 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs2. 2022-07-20T03:04:47.765 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../006-libcephfs2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:47.795 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T03:04:47.799 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T03:04:47.820 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.844 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T03:04:47.863 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T03:04:47.872 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:47.915 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:47.945 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T03:04:47.960 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T03:04:48.004 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:04:48.058 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T03:04:48.073 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:48.107 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rados. 2022-07-20T03:04:48.108 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:48.117 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../007-python3-rados_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:48.179 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:48.354 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T03:04:48.362 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T03:04:48.367 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:48.370 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T03:04:48.402 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:48.409 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:04:48.480 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-argparse. 2022-07-20T03:04:48.491 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T03:04:48.496 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../008-python3-ceph-argparse_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:48.505 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T03:04:48.538 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:04:48.545 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:48.693 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package librgw2. 2022-07-20T03:04:48.706 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:48.735 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:48.816 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cephfs. 2022-07-20T03:04:48.824 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../009-python3-cephfs_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:48.858 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:48.898 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T03:04:48.914 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T03:04:48.956 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:04:49.161 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-ceph-common. 2022-07-20T03:04:49.176 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../010-python3-ceph-common_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:49.222 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:49.235 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package librgw2. 2022-07-20T03:04:49.256 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:49.278 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:49.297 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T03:04:49.313 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:49.330 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T03:04:49.344 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:49.354 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:49.388 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:49.472 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-prettytable. 2022-07-20T03:04:49.485 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../011-python3-prettytable_0.7.2-5_all.deb ... 2022-07-20T03:04:49.515 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-prettytable (0.7.2-5) ... 2022-07-20T03:04:49.590 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-base. 2022-07-20T03:04:49.606 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:49.650 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:49.683 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T03:04:49.697 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T03:04:49.741 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:04:49.749 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rbd. 2022-07-20T03:04:49.764 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../012-python3-rbd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:49.832 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:49.846 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T03:04:49.859 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:49.896 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:50.027 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liboath0. 2022-07-20T03:04:50.043 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T03:04:50.077 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T03:04:50.179 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librabbitmq4:amd64. 2022-07-20T03:04:50.186 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../013-librabbitmq4_0.10.0-1_amd64.deb ... 2022-07-20T03:04:50.217 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T03:04:50.221 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:04:50.231 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T03:04:50.284 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:04:50.372 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T03:04:50.386 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:50.430 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:50.524 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librdkafka1:amd64. 2022-07-20T03:04:50.536 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T03:04:50.540 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../014-librdkafka1_1.2.1-1ubuntu1_amd64.deb ... 2022-07-20T03:04:50.552 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T03:04:50.567 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package liboath0. 2022-07-20T03:04:50.577 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T03:04:50.583 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:04:50.619 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T03:04:50.627 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:04:50.885 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-common. 2022-07-20T03:04:50.897 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:50.935 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:50.943 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package librgw2. 2022-07-20T03:04:50.959 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../015-librgw2_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:50.972 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T03:04:50.988 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:50.991 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T03:04:51.002 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:51.004 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T03:04:51.039 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:51.050 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T03:04:51.350 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T03:04:51.371 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:51.408 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:51.435 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-common. 2022-07-20T03:04:51.447 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:51.476 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:51.796 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rgw. 2022-07-20T03:04:51.811 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../016-python3-rgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:51.846 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:51.963 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T03:04:51.976 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T03:04:52.013 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:04:52.132 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libleveldb1d:amd64. 2022-07-20T03:04:52.149 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../017-libleveldb1d_1.22-3ubuntu2_amd64.deb ... 2022-07-20T03:04:52.180 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:04:52.344 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T03:04:52.368 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T03:04:52.416 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:04:52.426 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph. 2022-07-20T03:04:52.442 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:52.484 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:52.518 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liboath0. 2022-07-20T03:04:52.532 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../018-liboath0_2.6.1-1.3_amd64.deb ... 2022-07-20T03:04:52.567 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liboath0 (2.6.1-1.3) ... 2022-07-20T03:04:52.755 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T03:04:52.766 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T03:04:52.813 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T03:04:52.813 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:04:52.839 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:52.879 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:52.904 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libradosstriper1. 2022-07-20T03:04:52.926 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../019-libradosstriper1_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:52.972 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:53.074 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-base. 2022-07-20T03:04:53.090 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:53.098 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T03:04:53.114 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T03:04:53.138 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:53.161 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T03:04:53.197 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T03:04:53.197 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:53.224 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:53.365 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-common. 2022-07-20T03:04:53.378 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../020-ceph-common_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:53.409 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:53.425 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T03:04:53.439 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T03:04:53.483 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:04:53.536 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-base. 2022-07-20T03:04:53.547 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:53.603 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:53.743 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package cephadm. 2022-07-20T03:04:53.756 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:53.801 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:53.819 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-mako. 2022-07-20T03:04:53.832 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T03:04:53.869 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:04:54.072 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T03:04:54.085 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T03:04:54.120 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T03:04:54.121 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:04:54.133 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T03:04:54.170 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T03:04:54.214 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T03:04:54.227 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T03:04:54.272 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:04:54.462 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T03:04:54.472 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T03:04:54.489 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T03:04:54.500 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T03:04:54.505 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T03:04:54.505 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:54.516 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T03:04:54.542 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:54.555 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:04:54.558 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T03:04:54.571 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T03:04:54.600 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:04:54.778 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T03:04:54.794 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:54.827 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:54.839 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-webob. 2022-07-20T03:04:54.851 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T03:04:54.867 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T03:04:54.885 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T03:04:54.894 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T03:04:54.937 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T03:04:54.984 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T03:04:54.993 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T03:04:55.031 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T03:04:55.174 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T03:04:55.180 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T03:04:55.182 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T03:04:55.191 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T03:04:55.202 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T03:04:55.203 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T03:04:55.221 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:04:55.228 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:04:55.232 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:04:55.291 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T03:04:55.299 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:55.327 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:55.463 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-base. 2022-07-20T03:04:55.477 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../021-ceph-base_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:55.480 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T03:04:55.486 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T03:04:55.493 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T03:04:55.506 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T03:04:55.522 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:04:55.539 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:55.540 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:04:55.540 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T03:04:55.558 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T03:04:55.593 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T03:04:55.683 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T03:04:55.699 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T03:04:55.749 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:04:55.851 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T03:04:55.851 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T03:04:55.852 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-routes. 2022-07-20T03:04:55.860 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T03:04:55.875 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T03:04:55.882 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:04:55.954 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T03:04:55.971 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T03:04:56.006 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:04:56.059 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T03:04:56.066 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T03:04:56.111 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:04:56.151 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T03:04:56.173 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T03:04:56.174 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T03:04:56.191 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:56.203 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T03:04:56.227 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:56.333 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T03:04:56.347 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T03:04:56.395 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T03:04:56.396 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cryptography. 2022-07-20T03:04:56.398 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../022-python3-cryptography_2.8-3ubuntu0.1_amd64.deb ... 2022-07-20T03:04:56.442 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:04:56.465 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T03:04:56.479 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T03:04:56.480 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T03:04:56.495 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T03:04:56.523 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:04:56.546 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:04:57.210 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T03:04:57.225 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:57.268 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:57.378 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-paste. 2022-07-20T03:04:57.392 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T03:04:57.414 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-openssl. 2022-07-20T03:04:57.431 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../023-python3-openssl_19.0.0-1build1_all.deb ... 2022-07-20T03:04:57.434 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:04:57.469 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T03:04:57.476 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-openssl (19.0.0-1build1) ... 2022-07-20T03:04:57.481 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T03:04:57.505 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-mako. 2022-07-20T03:04:57.510 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T03:04:57.521 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T03:04:57.555 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T03:04:57.566 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:04:57.568 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T03:04:57.596 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:04:57.732 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-modules-core. 2022-07-20T03:04:57.747 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../024-ceph-mgr-modules-core_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:04:57.758 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T03:04:57.769 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T03:04:57.787 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:57.813 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T03:04:57.826 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T03:04:57.837 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:04:57.864 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:04:57.864 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T03:04:57.874 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T03:04:57.886 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T03:04:57.896 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T03:04:57.911 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:04:57.941 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:04:58.142 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T03:04:58.143 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T03:04:58.144 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bcrypt. 2022-07-20T03:04:58.150 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../025-python3-bcrypt_3.1.7-2ubuntu1_amd64.deb ... 2022-07-20T03:04:58.157 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T03:04:58.157 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T03:04:58.158 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-mako. 2022-07-20T03:04:58.176 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T03:04:58.197 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:04:58.198 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:04:58.209 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:04:58.216 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:04:58.251 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T03:04:58.266 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T03:04:58.309 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T03:04:58.468 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cherrypy3. 2022-07-20T03:04:58.476 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../026-python3-cherrypy3_8.9.1-6_all.deb ... 2022-07-20T03:04:58.486 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T03:04:58.502 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T03:04:58.505 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:04:58.528 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-webob. 2022-07-20T03:04:58.534 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T03:04:58.537 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T03:04:58.538 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T03:04:58.550 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T03:04:58.575 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T03:04:58.617 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:04:58.771 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T03:04:58.786 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T03:04:58.822 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:04:58.867 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jwt. 2022-07-20T03:04:58.885 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../027-python3-jwt_1.7.1-2ubuntu2.1_all.deb ... 2022-07-20T03:04:58.905 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T03:04:58.917 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:04:58.917 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T03:04:58.926 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T03:04:58.938 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T03:04:58.961 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:04:58.973 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:04:59.125 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T03:04:59.140 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T03:04:59.183 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:04:59.186 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-logutils. 2022-07-20T03:04:59.202 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../028-python3-logutils_0.3.3-7_all.deb ... 2022-07-20T03:04:59.237 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-logutils (0.3.3-7) ... 2022-07-20T03:04:59.238 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-webob. 2022-07-20T03:04:59.248 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T03:04:59.248 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T03:04:59.262 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T03:04:59.277 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T03:04:59.298 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T03:04:59.495 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T03:04:59.512 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:04:59.569 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:04:59.572 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-markupsafe. 2022-07-20T03:04:59.579 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T03:04:59.587 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../029-python3-markupsafe_1.1.0-1build2_amd64.deb ... 2022-07-20T03:04:59.594 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T03:04:59.610 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T03:04:59.628 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T03:04:59.630 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:04:59.638 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:04:59.645 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T03:04:59.659 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T03:04:59.699 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:04:59.701 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:04:59.924 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T03:04:59.940 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T03:04:59.949 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-mako. 2022-07-20T03:04:59.957 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../030-python3-mako_1.1.0+ds1-1ubuntu2_all.deb ... 2022-07-20T03:04:59.983 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:00.000 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:05:00.013 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T03:05:00.022 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T03:05:00.027 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T03:05:00.035 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:00.071 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T03:05:00.072 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:00.246 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T03:05:00.260 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T03:05:00.304 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T03:05:00.321 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplegeneric. 2022-07-20T03:05:00.333 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../031-python3-simplegeneric_0.8.1-3_all.deb ... 2022-07-20T03:05:00.343 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-paste. 2022-07-20T03:05:00.356 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T03:05:00.378 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T03:05:00.391 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:00.394 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:05:00.400 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:00.439 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:00.641 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T03:05:00.647 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-singledispatch. 2022-07-20T03:05:00.656 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T03:05:00.662 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../032-python3-singledispatch_3.4.0.3-2build1_all.deb ... 2022-07-20T03:05:00.691 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:00.697 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:05:00.729 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T03:05:00.742 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T03:05:00.778 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:00.790 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T03:05:00.805 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T03:05:00.849 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T03:05:01.007 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T03:05:01.008 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webob. 2022-07-20T03:05:01.017 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T03:05:01.028 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../033-python3-webob_1%3a1.8.5-2_all.deb ... 2022-07-20T03:05:01.028 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T03:05:01.030 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:01.048 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:01.068 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:01.068 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webob (1:1.8.5-2) ... 2022-07-20T03:05:01.144 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-paste. 2022-07-20T03:05:01.159 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T03:05:01.193 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:01.342 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T03:05:01.360 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T03:05:01.361 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:01.382 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T03:05:01.390 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:01.403 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-soupsieve. 2022-07-20T03:05:01.416 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../034-python3-soupsieve_1.9.5+dfsg-1_all.deb ... 2022-07-20T03:05:01.418 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T03:05:01.472 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:05:01.539 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T03:05:01.551 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T03:05:01.589 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:01.731 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T03:05:01.744 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T03:05:01.781 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-bs4. 2022-07-20T03:05:01.787 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:05:01.798 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../035-python3-bs4_4.8.2-1_all.deb ... 2022-07-20T03:05:01.842 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:01.843 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T03:05:01.849 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T03:05:01.883 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:02.085 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T03:05:02.098 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T03:05:02.151 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:02.209 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-waitress. 2022-07-20T03:05:02.214 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T03:05:02.227 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T03:05:02.227 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../036-python3-waitress_1.4.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:02.270 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:02.272 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T03:05:02.505 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T03:05:02.515 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:02.552 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T03:05:02.555 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:02.565 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T03:05:02.602 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:02.606 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T03:05:02.619 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T03:05:02.632 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-tempita. 2022-07-20T03:05:02.643 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../037-python3-tempita_0.5.2-6_all.deb ... 2022-07-20T03:05:02.668 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:05:02.695 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-tempita (0.5.2-6) ... 2022-07-20T03:05:02.913 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T03:05:02.922 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T03:05:02.954 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:02.975 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-paste. 2022-07-20T03:05:02.992 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../038-python3-paste_3.4.0+dfsg1-1_all.deb ... 2022-07-20T03:05:03.018 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T03:05:03.027 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:03.032 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:03.054 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T03:05:03.068 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T03:05:03.104 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:03.104 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:03.267 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T03:05:03.272 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T03:05:03.320 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:03.320 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph. 2022-07-20T03:05:03.327 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:03.352 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python-pastedeploy-tpl. 2022-07-20T03:05:03.365 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../039-python-pastedeploy-tpl_2.1.0-1_all.deb ... 2022-07-20T03:05:03.374 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:03.422 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:03.484 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T03:05:03.499 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:03.544 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:03.611 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T03:05:03.626 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T03:05:03.658 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T03:05:03.678 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:03.679 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T03:05:03.708 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:03.760 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastedeploy. 2022-07-20T03:05:03.773 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../040-python3-pastedeploy_2.1.0-1_all.deb ... 2022-07-20T03:05:03.806 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:03.980 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T03:05:04.001 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T03:05:04.025 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T03:05:04.035 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:04.038 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:04.067 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webtest. 2022-07-20T03:05:04.071 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T03:05:04.078 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:04.082 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../041-python3-webtest_2.0.32-2_all.deb ... 2022-07-20T03:05:04.091 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:04.106 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T03:05:04.115 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webtest (2.0.32-2) ... 2022-07-20T03:05:04.118 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:04.128 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:04.162 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:04.392 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T03:05:04.401 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pecan. 2022-07-20T03:05:04.420 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../042-python3-pecan_1.3.3-3ubuntu1_all.deb ... 2022-07-20T03:05:04.421 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T03:05:04.458 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:04.459 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:05:04.644 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package cephadm. 2022-07-20T03:05:04.660 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:04.697 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:04.754 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T03:05:04.781 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T03:05:04.804 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:04.853 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-werkzeug. 2022-07-20T03:05:04.865 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../043-python3-werkzeug_0.16.1+dfsg1-2_all.deb ... 2022-07-20T03:05:04.898 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:05.025 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T03:05:05.037 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T03:05:05.074 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:05.085 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T03:05:05.102 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:05.143 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:05.149 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T03:05:05.170 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T03:05:05.192 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr. 2022-07-20T03:05:05.197 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:05.208 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../044-ceph-mgr_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:05.252 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:05.386 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T03:05:05.408 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:05.435 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:05.468 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T03:05:05.482 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T03:05:05.516 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:05.711 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mon. 2022-07-20T03:05:05.726 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../045-ceph-mon_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:05.738 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T03:05:05.753 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T03:05:05.770 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:05.794 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:05.823 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T03:05:05.839 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T03:05:05.882 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:06.042 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T03:05:06.054 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T03:05:06.090 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:06.349 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T03:05:06.362 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:06.392 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:06.412 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T03:05:06.415 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T03:05:06.457 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:06.536 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph. 2022-07-20T03:05:06.550 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:06.594 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:06.730 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T03:05:06.742 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T03:05:06.773 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-routes. 2022-07-20T03:05:06.774 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-osd. 2022-07-20T03:05:06.784 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:06.786 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T03:05:06.789 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../046-ceph-osd_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:06.829 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T03:05:06.834 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:06.939 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T03:05:06.959 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:06.996 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:07.138 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T03:05:07.143 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T03:05:07.152 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:07.156 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:07.188 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:07.198 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:07.350 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T03:05:07.366 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:07.403 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:07.446 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph. 2022-07-20T03:05:07.463 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:07.465 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package jq. 2022-07-20T03:05:07.482 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:07.496 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:07.523 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:07.759 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package socat. 2022-07-20T03:05:07.773 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T03:05:07.818 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T03:05:07.824 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T03:05:07.839 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:07.912 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:07.913 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package cephadm. 2022-07-20T03:05:07.927 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:07.962 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:08.200 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T03:05:08.212 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:08.255 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:08.265 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T03:05:08.278 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T03:05:08.286 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T03:05:08.298 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T03:05:08.301 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:08.313 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T03:05:08.315 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:08.356 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:08.369 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:08.595 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T03:05:08.602 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T03:05:08.605 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T03:05:08.615 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:08.625 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T03:05:08.642 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T03:05:08.649 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:08.669 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:08.678 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:08.881 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package cephadm. 2022-07-20T03:05:08.896 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:08.912 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T03:05:08.926 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T03:05:08.930 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:08.961 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:08.983 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T03:05:08.986 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T03:05:09.031 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:09.047 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package ceph-test. 2022-07-20T03:05:09.060 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:09.096 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:09.251 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T03:05:09.263 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T03:05:09.271 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph. 2022-07-20T03:05:09.283 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T03:05:09.285 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../047-ceph_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:09.288 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T03:05:09.300 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T03:05:09.305 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T03:05:09.326 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:09.338 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:09.350 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:09.351 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:09.590 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-fuse. 2022-07-20T03:05:09.604 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../048-ceph-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:09.629 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T03:05:09.651 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T03:05:09.652 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:09.652 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:09.676 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T03:05:09.687 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:09.703 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:09.997 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-routes. 2022-07-20T03:05:10.010 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mds. 2022-07-20T03:05:10.012 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T03:05:10.023 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../049-ceph-mds_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:10.024 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T03:05:10.038 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T03:05:10.051 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T03:05:10.062 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:10.085 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:10.335 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T03:05:10.345 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T03:05:10.350 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:10.360 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T03:05:10.398 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:10.403 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:10.505 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package cephadm. 2022-07-20T03:05:10.520 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../050-cephadm_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:10.562 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:10.622 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T03:05:10.640 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T03:05:10.673 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:10.711 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T03:05:10.721 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T03:05:10.767 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:10.882 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-jinja2. 2022-07-20T03:05:10.897 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../051-python3-jinja2_2.10.1-2_all.deb ... 2022-07-20T03:05:10.942 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:10.991 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-routes. 2022-07-20T03:05:11.006 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T03:05:11.049 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T03:05:11.268 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-cephadm. 2022-07-20T03:05:11.282 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../052-ceph-mgr-cephadm_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:11.291 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T03:05:11.306 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T03:05:11.350 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T03:05:11.351 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:11.386 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T03:05:11.391 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:11.426 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T03:05:11.428 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:11.443 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:11.478 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:11.674 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-repoze.lru. 2022-07-20T03:05:11.686 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../053-python3-repoze.lru_0.7-2_all.deb ... 2022-07-20T03:05:11.730 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:11.731 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T03:05:11.746 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T03:05:11.762 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T03:05:11.777 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T03:05:11.781 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:11.821 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:11.978 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjs-underscore. 2022-07-20T03:05:11.989 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../054-libjs-underscore_1.9.1~dfsg-1ubuntu0.20.04.1_all.deb ... 2022-07-20T03:05:12.016 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:12.067 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T03:05:12.085 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T03:05:12.126 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:12.254 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjs-sphinxdoc. 2022-07-20T03:05:12.275 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../055-libjs-sphinxdoc_1.8.5-7ubuntu3_all.deb ... 2022-07-20T03:05:12.301 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:12.414 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T03:05:12.426 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T03:05:12.455 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T03:05:12.476 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:12.476 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:12.478 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T03:05:12.495 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:12.515 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:12.588 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-routes. 2022-07-20T03:05:12.600 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../056-python3-routes_2.4.1-2_all.deb ... 2022-07-20T03:05:12.614 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:12.625 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T03:05:12.628 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-routes (2.4.1-2) ... 2022-07-20T03:05:12.640 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T03:05:12.686 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:12.826 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T03:05:12.841 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T03:05:12.869 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:12.951 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T03:05:12.961 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:12.999 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-dashboard. 2022-07-20T03:05:13.002 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:13.014 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../057-ceph-mgr-dashboard_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:13.057 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:13.195 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T03:05:13.210 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T03:05:13.255 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:13.280 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T03:05:13.295 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:13.338 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:13.576 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T03:05:13.590 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T03:05:13.657 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:13.701 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T03:05:13.714 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T03:05:13.722 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T03:05:13.737 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T03:05:13.775 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:13.803 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:13.856 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T03:05:13.879 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T03:05:13.914 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:13.990 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-cloud. 2022-07-20T03:05:14.004 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../058-ceph-mgr-diskprediction-cloud_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:14.063 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T03:05:14.073 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:14.075 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T03:05:14.114 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:14.139 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T03:05:14.146 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T03:05:14.188 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:14.411 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-decorator. 2022-07-20T03:05:14.412 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T03:05:14.416 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T03:05:14.423 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../059-python3-decorator_4.4.2-0ubuntu1_all.deb ... 2022-07-20T03:05:14.445 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:14.457 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T03:05:14.468 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:14.469 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T03:05:14.502 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T03:05:14.524 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-attr. 2022-07-20T03:05:14.536 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T03:05:14.585 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T03:05:14.694 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T03:05:14.707 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T03:05:14.738 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T03:05:14.787 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblbfgsb0:amd64. 2022-07-20T03:05:14.800 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../060-liblbfgsb0_3.0+dfsg.3-7build1_amd64.deb ... 2022-07-20T03:05:14.838 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:14.867 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T03:05:14.887 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T03:05:14.914 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:14.915 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T03:05:14.926 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T03:05:14.967 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:14.982 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T03:05:15.003 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T03:05:15.037 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:15.070 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T03:05:15.082 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T03:05:15.116 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:15.198 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-scipy. 2022-07-20T03:05:15.211 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../061-python3-scipy_1.3.3-3build1_amd64.deb ... 2022-07-20T03:05:15.220 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T03:05:15.238 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T03:05:15.251 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:15.270 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:15.428 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T03:05:15.444 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T03:05:15.512 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:15.581 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T03:05:15.598 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T03:05:15.606 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T03:05:15.623 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T03:05:15.624 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T03:05:15.639 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:15.640 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:15.665 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T03:05:15.676 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:15.814 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T03:05:15.827 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T03:05:15.872 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:16.086 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T03:05:16.099 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T03:05:16.117 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T03:05:16.130 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T03:05:16.144 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:16.169 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T03:05:16.170 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T03:05:16.184 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T03:05:16.225 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:16.468 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T03:05:16.470 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T03:05:16.513 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:16.528 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T03:05:16.544 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn-lib. 2022-07-20T03:05:16.556 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../062-python3-sklearn-lib_0.22.2.post1+dfsg-5_amd64.deb ... 2022-07-20T03:05:16.557 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T03:05:16.582 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:16.590 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:16.782 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T03:05:16.794 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T03:05:16.818 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:16.874 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T03:05:16.878 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T03:05:16.888 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T03:05:16.905 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:16.935 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:16.935 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T03:05:16.939 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:16.959 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T03:05:16.989 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:17.127 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-joblib. 2022-07-20T03:05:17.139 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T03:05:17.140 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../063-python3-joblib_0.14.0-3_all.deb ... 2022-07-20T03:05:17.153 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T03:05:17.177 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-joblib (0.14.0-3) ... 2022-07-20T03:05:17.201 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:17.235 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T03:05:17.250 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T03:05:17.284 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:17.525 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T03:05:17.539 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-sklearn. 2022-07-20T03:05:17.540 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T03:05:17.545 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T03:05:17.554 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../064-python3-sklearn_0.22.2.post1+dfsg-5_all.deb ... 2022-07-20T03:05:17.559 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:17.585 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:17.589 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:17.602 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:17.629 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-py. 2022-07-20T03:05:17.645 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:17.688 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:18.013 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T03:05:18.021 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T03:05:18.065 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:18.099 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T03:05:18.120 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T03:05:18.121 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T03:05:18.126 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T03:05:18.147 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T03:05:18.157 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:18.163 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:18.164 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T03:05:18.205 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T03:05:18.332 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-diskprediction-local. 2022-07-20T03:05:18.345 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../065-ceph-mgr-diskprediction-local_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:18.380 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:18.403 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T03:05:18.419 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T03:05:18.464 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T03:05:18.465 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T03:05:18.474 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T03:05:18.520 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T03:05:18.530 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T03:05:18.534 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T03:05:18.535 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:18.546 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:18.573 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:18.590 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:18.774 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T03:05:18.786 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T03:05:18.825 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:18.899 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package jq. 2022-07-20T03:05:18.899 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:18.900 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T03:05:18.900 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T03:05:18.921 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T03:05:18.921 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T03:05:18.930 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:18.931 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:18.954 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:19.195 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package socat. 2022-07-20T03:05:19.210 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T03:05:19.218 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T03:05:19.236 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T03:05:19.262 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T03:05:19.275 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:19.276 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T03:05:19.277 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package radosgw. 2022-07-20T03:05:19.292 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:19.293 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T03:05:19.325 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T03:05:19.335 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:19.468 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-cachetools. 2022-07-20T03:05:19.493 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../066-python3-cachetools_4.0.0-1_all.deb ... 2022-07-20T03:05:19.517 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:19.577 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T03:05:19.597 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T03:05:19.621 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T03:05:19.629 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:19.637 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T03:05:19.663 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:19.742 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T03:05:19.758 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T03:05:19.786 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1. 2022-07-20T03:05:19.799 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../067-python3-pyasn1_0.4.2-3build1_all.deb ... 2022-07-20T03:05:19.800 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:19.839 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:19.969 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T03:05:19.980 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T03:05:20.027 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:20.041 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T03:05:20.053 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T03:05:20.055 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T03:05:20.067 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T03:05:20.099 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:20.112 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:20.148 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyasn1-modules. 2022-07-20T03:05:20.169 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../068-python3-pyasn1-modules_0.2.1-0.2build1_all.deb ... 2022-07-20T03:05:20.231 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:20.338 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T03:05:20.347 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T03:05:20.368 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T03:05:20.378 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:20.386 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:20.412 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T03:05:20.434 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T03:05:20.435 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:20.482 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package ceph-test. 2022-07-20T03:05:20.495 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:20.496 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:20.534 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:20.542 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-rsa. 2022-07-20T03:05:20.561 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../069-python3-rsa_4.0-3_all.deb ... 2022-07-20T03:05:20.603 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-rsa (4.0-3) ... 2022-07-20T03:05:20.714 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T03:05:20.815 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T03:05:20.830 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T03:05:20.887 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:20.920 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-google-auth. 2022-07-20T03:05:20.946 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../070-python3-google-auth_1.5.1-2_all.deb ... 2022-07-20T03:05:20.949 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T03:05:20.967 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:20.978 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:20.998 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:21.035 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:05:21.159 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T03:05:21.172 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T03:05:21.186 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:05:21.204 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:21.299 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-blinker. 2022-07-20T03:05:21.300 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T03:05:21.316 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T03:05:21.317 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../071-python3-blinker_1.4+dfsg1-0.3ubuntu1_all.deb ... 2022-07-20T03:05:21.318 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:21.349 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:21.351 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:21.505 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:21.555 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T03:05:21.571 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T03:05:21.615 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:21.625 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-oauthlib. 2022-07-20T03:05:21.639 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../072-python3-oauthlib_3.1.0-1ubuntu2_all.deb ... 2022-07-20T03:05:21.684 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:21.738 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T03:05:21.751 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:21.788 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:22.004 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-requests-oauthlib. 2022-07-20T03:05:22.019 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../073-python3-requests-oauthlib_1.0.0-1.1build1_all.deb ... 2022-07-20T03:05:22.040 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package jq. 2022-07-20T03:05:22.054 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:22.055 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:22.091 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user cephadm....done 2022-07-20T03:05:22.092 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:22.143 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T03:05:22.156 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:22.194 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:22.252 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:22.326 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package socat. 2022-07-20T03:05:22.342 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-websocket. 2022-07-20T03:05:22.342 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T03:05:22.356 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../074-python3-websocket_0.53.0-2ubuntu1_all.deb ... 2022-07-20T03:05:22.377 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T03:05:22.407 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:22.523 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T03:05:22.537 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T03:05:22.580 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:22.595 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:22.653 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-kubernetes. 2022-07-20T03:05:22.670 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../075-python3-kubernetes_7.0.0~a1-2_all.deb ... 2022-07-20T03:05:22.712 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:22.854 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T03:05:22.866 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T03:05:22.900 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:22.933 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T03:05:22.949 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:22.992 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:22.993 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T03:05:23.091 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:23.154 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T03:05:23.174 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T03:05:23.204 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package jq. 2022-07-20T03:05:23.211 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:23.227 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:23.261 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:23.290 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-mgr-k8sevents. 2022-07-20T03:05:23.309 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../076-ceph-mgr-k8sevents_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:23.348 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:23.416 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T03:05:23.555 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package socat. 2022-07-20T03:05:23.570 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T03:05:23.571 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package ceph-test. 2022-07-20T03:05:23.579 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:23.614 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T03:05:23.615 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:23.676 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libonig5:amd64. 2022-07-20T03:05:23.690 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../077-libonig5_6.9.4-1_amd64.deb ... 2022-07-20T03:05:23.727 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:23.850 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:05:24.021 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T03:05:24.040 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T03:05:24.054 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libjq1:amd64. 2022-07-20T03:05:24.059 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../078-libjq1_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:24.080 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:24.081 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T03:05:24.086 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T03:05:24.130 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:24.131 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:24.135 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:24.331 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T03:05:24.347 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:24.385 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:24.408 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package jq. 2022-07-20T03:05:24.426 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../079-jq_1.6-1ubuntu0.20.04.1_amd64.deb ... 2022-07-20T03:05:24.433 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T03:05:24.439 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:24.446 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T03:05:24.464 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:24.486 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:24.632 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T03:05:24.646 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:24.690 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:24.718 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:05:24.748 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package socat. 2022-07-20T03:05:24.762 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../080-socat_1.7.3.3-2_amd64.deb ... 2022-07-20T03:05:24.800 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking socat (1.7.3.3-2) ... 2022-07-20T03:05:24.854 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package ceph-test. 2022-07-20T03:05:24.867 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:24.903 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:24.973 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:25.119 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T03:05:25.131 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T03:05:25.196 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:25.247 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:25.323 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libxslt1.1:amd64. 2022-07-20T03:05:25.339 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../081-libxslt1.1_1.1.34-4_amd64.deb ... 2022-07-20T03:05:25.383 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:25.510 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:25.590 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T03:05:25.612 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T03:05:25.630 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T03:05:25.634 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package xmlstarlet. 2022-07-20T03:05:25.665 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:25.665 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../082-xmlstarlet_1.6.1-2build1_amd64.deb ... 2022-07-20T03:05:25.693 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:25.926 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:05:25.951 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-attr. 2022-07-20T03:05:25.965 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T03:05:26.000 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T03:05:26.053 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package ceph-test. 2022-07-20T03:05:26.059 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../083-ceph-test_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:26.097 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:26.210 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:26.353 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T03:05:26.369 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T03:05:26.412 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:26.495 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:26.714 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T03:05:26.727 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T03:05:26.763 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:26.887 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:27.057 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T03:05:27.074 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T03:05:27.108 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:27.134 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T03:05:27.147 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:27.149 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T03:05:27.195 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:27.375 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T03:05:27.382 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:27.423 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:27.456 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:27.677 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T03:05:27.678 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T03:05:27.691 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:27.698 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T03:05:27.717 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:27.738 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T03:05:27.738 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:28.039 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T03:05:28.054 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T03:05:28.055 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T03:05:28.083 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:28.101 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T03:05:28.111 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T03:05:28.169 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:28.369 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T03:05:28.427 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T03:05:28.439 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T03:05:28.484 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:28.572 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T03:05:28.587 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T03:05:28.627 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T03:05:28.628 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:28.639 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T03:05:28.669 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:28.683 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:28.795 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:05:28.804 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T03:05:28.820 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T03:05:28.861 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:28.924 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-attr. 2022-07-20T03:05:28.937 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T03:05:28.945 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T03:05:28.961 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:28.981 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T03:05:29.003 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:29.097 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:29.206 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T03:05:29.223 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:29.232 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-py. 2022-07-20T03:05:29.265 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:29.281 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:29.281 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:29.335 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T03:05:29.350 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T03:05:29.386 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:29.470 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:05:29.492 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-colorama. 2022-07-20T03:05:29.507 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../084-python3-colorama_0.4.3-1build1_all.deb ... 2022-07-20T03:05:29.570 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:29.595 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T03:05:29.614 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T03:05:29.650 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:29.673 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T03:05:29.673 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T03:05:29.682 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T03:05:29.689 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T03:05:29.729 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:29.741 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:29.743 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:29.841 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package cephfs-shell. 2022-07-20T03:05:29.853 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../085-cephfs-shell_15.2.16-336-g57544da1-1focal_all.deb ... 2022-07-20T03:05:29.894 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:29.946 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T03:05:29.968 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T03:05:30.003 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T03:05:30.074 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T03:05:30.094 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T03:05:30.106 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:05:30.156 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libcephfs-dev. 2022-07-20T03:05:30.157 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:30.165 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T03:05:30.171 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../086-libcephfs-dev_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:30.186 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T03:05:30.205 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:30.248 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:30.332 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T03:05:30.346 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T03:05:30.406 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:30.468 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T03:05:30.508 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package nvme-cli. 2022-07-20T03:05:30.521 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-attr. 2022-07-20T03:05:30.522 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../087-nvme-cli_1.9-1ubuntu0.1_amd64.deb ... 2022-07-20T03:05:30.534 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T03:05:30.579 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T03:05:30.580 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T03:05:30.586 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:30.593 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T03:05:30.594 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:30.639 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T03:05:30.845 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package radosgw. 2022-07-20T03:05:30.858 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:30.897 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:30.901 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:30.915 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T03:05:30.933 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T03:05:30.938 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T03:05:30.955 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T03:05:30.972 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:30.980 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-atomicwrites. 2022-07-20T03:05:30.994 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../088-python3-atomicwrites_1.1.5-2build1_all.deb ... 2022-07-20T03:05:31.006 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:31.038 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:31.214 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:05:31.309 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T03:05:31.315 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:31.325 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T03:05:31.339 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T03:05:31.350 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T03:05:31.351 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-attr. 2022-07-20T03:05:31.366 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../089-python3-attr_19.3.0-2_all.deb ... 2022-07-20T03:05:31.366 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:31.393 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:31.403 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-attr (19.3.0-2) ... 2022-07-20T03:05:31.574 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:31.700 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:31.718 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T03:05:31.720 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T03:05:31.731 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T03:05:31.735 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T03:05:31.747 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-webencodings. 2022-07-20T03:05:31.762 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../090-python3-webencodings_0.5.1-1ubuntu1_all.deb ... 2022-07-20T03:05:31.773 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:31.773 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:31.795 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:31.963 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T03:05:31.977 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:32.031 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:05:32.032 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:32.057 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-html5lib. 2022-07-20T03:05:32.073 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../091-python3-html5lib_1.0.1-2_all.deb ... 2022-07-20T03:05:32.115 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:32.125 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-py. 2022-07-20T03:05:32.141 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:32.183 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:32.258 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T03:05:32.274 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T03:05:32.307 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T03:05:32.315 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T03:05:32.336 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T03:05:32.476 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-lxml:amd64. 2022-07-20T03:05:32.484 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../092-python3-lxml_4.5.0-1ubuntu0.5_amd64.deb ... 2022-07-20T03:05:32.497 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T03:05:32.517 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T03:05:32.526 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:32.553 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:32.595 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:05:32.646 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T03:05:32.660 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T03:05:32.661 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:05:32.701 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:32.730 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:05:32.785 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:05:32.865 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T03:05:32.883 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T03:05:32.886 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:32.940 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T03:05:33.000 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-packaging. 2022-07-20T03:05:33.013 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../093-python3-packaging_20.3-1_all.deb ... 2022-07-20T03:05:33.033 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T03:05:33.047 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T03:05:33.075 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:33.089 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:33.129 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T03:05:33.146 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-packaging (20.3-1) ... 2022-07-20T03:05:33.277 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T03:05:33.295 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T03:05:33.368 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T03:05:33.382 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T03:05:33.430 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:33.438 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pastescript. 2022-07-20T03:05:33.447 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../094-python3-pastescript_2.0.2-4_all.deb ... 2022-07-20T03:05:33.449 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:05:33.490 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:33.501 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:33.742 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:33.770 INFO:teuthology.orchestra.run.smithi163.stdout:Adding system user cephadm....done 2022-07-20T03:05:33.827 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pluggy. 2022-07-20T03:05:33.842 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../095-python3-pluggy_0.13.0-2_all.deb ... 2022-07-20T03:05:33.864 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package radosgw. 2022-07-20T03:05:33.872 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-py. 2022-07-20T03:05:33.878 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:33.884 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:33.885 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:33.892 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:33.912 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:33.923 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:34.042 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T03:05:34.213 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-psutil. 2022-07-20T03:05:34.227 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../096-python3-psutil_5.5.1-1ubuntu4_amd64.deb ... 2022-07-20T03:05:34.244 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:34.266 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T03:05:34.277 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:34.282 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T03:05:34.296 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:34.324 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:34.420 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:05:34.536 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T03:05:34.623 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-py. 2022-07-20T03:05:34.656 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../097-python3-py_1.8.1-1ubuntu0.1_all.deb ... 2022-07-20T03:05:34.677 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:34.682 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:34.682 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T03:05:34.684 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T03:05:34.728 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T03:05:34.747 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:05:35.000 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T03:05:35.001 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T03:05:35.012 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:35.028 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pyinotify. 2022-07-20T03:05:35.047 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../098-python3-pyinotify_0.9.6-1.2ubuntu1_all.deb ... 2022-07-20T03:05:35.053 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:35.063 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T03:05:35.080 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T03:05:35.103 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:35.103 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:35.121 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:35.203 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:35.307 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T03:05:35.344 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T03:05:35.381 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-pytest. 2022-07-20T03:05:35.398 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../099-python3-pytest_4.6.9-1_all.deb ... 2022-07-20T03:05:35.413 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:05:35.439 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-pytest (4.6.9-1) ... 2022-07-20T03:05:35.522 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package radosgw. 2022-07-20T03:05:35.538 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:35.568 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:35.640 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:05:35.642 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:35.709 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:35.784 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package python3-simplejson. 2022-07-20T03:05:35.791 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:05:35.802 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../100-python3-simplejson_3.16.0-2ubuntu2_amd64.deb ... 2022-07-20T03:05:35.842 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:35.923 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:35.946 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:36.013 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:36.069 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:36.111 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:36.283 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:05:36.296 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package radosgw. 2022-07-20T03:05:36.310 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../101-radosgw_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:36.357 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:36.582 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:36.607 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T03:05:36.624 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:36.666 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:36.699 INFO:teuthology.orchestra.run.smithi179.stdout:Adding system user cephadm....done 2022-07-20T03:05:36.826 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:36.871 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:36.979 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T03:05:37.169 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:37.199 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:37.302 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:05:37.323 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T03:05:37.441 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package rbd-fuse. 2022-07-20T03:05:37.454 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../102-rbd-fuse_15.2.16-336-g57544da1-1focal_amd64.deb ... 2022-07-20T03:05:37.462 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:05:37.470 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T03:05:37.493 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:37.547 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:37.588 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:37.615 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:05:37.632 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:37.668 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:37.787 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:37.789 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-attr (19.3.0-2) ... 2022-07-20T03:05:37.794 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T03:05:37.908 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:37.941 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T03:05:38.108 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libleveldb1d:amd64 (1.22-3ubuntu2) ... 2022-07-20T03:05:38.189 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T03:05:38.194 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:38.229 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librabbitmq4:amd64 (0.10.0-1) ... 2022-07-20T03:05:38.328 INFO:teuthology.orchestra.run.smithi124.stdout:Adding system user cephadm....done 2022-07-20T03:05:38.377 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:05:38.387 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up nvme-cli (1.9-1ubuntu0.1) ... 2022-07-20T03:05:38.483 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:38.514 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:38.544 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:38.549 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:38.661 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:38.764 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:38.812 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:38.875 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:38.962 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:39.099 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:39.126 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user cephadm....done 2022-07-20T03:05:39.153 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T03:05:39.166 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:39.227 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:39.252 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:05:39.279 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:39.280 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-waitress (1.4.1-1ubuntu0.1) ... 2022-07-20T03:05:39.448 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:39.506 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:39.543 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:39.629 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T03:05:39.639 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-repoze.lru (0.7-2) ... 2022-07-20T03:05:39.694 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:39.790 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T03:05:39.791 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:39.914 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liboath0 (2.6.1-1.3) ... 2022-07-20T03:05:40.038 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:40.039 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:05:40.058 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-py (1.8.1-1ubuntu0.1) ... 2022-07-20T03:05:40.067 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T03:05:40.071 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:40.174 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T03:05:40.355 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:40.355 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:40.407 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-joblib (0.14.0-3) ... 2022-07-20T03:05:40.481 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:05:40.486 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:05:40.610 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:40.772 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:40.788 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:40.854 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jwt (1.7.1-2ubuntu2.1) ... 2022-07-20T03:05:40.902 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:05:41.059 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:41.064 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:41.131 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cachetools (4.0.0-1) ... 2022-07-20T03:05:41.192 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:05:41.193 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:41.218 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:41.345 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:41.414 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-colorama (0.4.3-1build1) ... 2022-07-20T03:05:41.455 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:41.470 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:41.502 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:41.509 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:41.683 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bcrypt (3.1.7-2ubuntu1) ... 2022-07-20T03:05:41.713 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:41.796 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:41.797 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:41.842 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:05:41.910 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T03:05:41.926 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T03:05:41.952 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyinotify (0.9.6-1.2ubuntu1) ... 2022-07-20T03:05:42.025 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:42.166 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:05:42.201 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T03:05:42.215 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T03:05:42.260 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-argparse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:42.302 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:42.363 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:42.468 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:42.483 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:42.560 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn-lib (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:42.608 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:42.647 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T03:05:42.671 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:42.714 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyperclip (1.7.0-1) ... 2022-07-20T03:05:42.754 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:42.851 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:42.851 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:42.958 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T03:05:43.008 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-markupsafe (1.1.0-1build2) ... 2022-07-20T03:05:43.019 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:05:43.125 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:43.137 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:43.266 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:43.295 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webencodings (0.5.1-1ubuntu1) ... 2022-07-20T03:05:43.306 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:43.369 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:05:43.393 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:43.442 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:43.471 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:43.601 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:43.641 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:43.767 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:43.790 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:43.891 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-psutil (5.5.1-1ubuntu4) ... 2022-07-20T03:05:43.913 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:05:44.022 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:05:44.062 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:44.063 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:05:44.199 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T03:05:44.250 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-atomicwrites (1.1.5-2build1) ... 2022-07-20T03:05:44.362 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:44.418 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:44.418 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T03:05:44.490 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:05:44.518 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:44.527 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplejson (3.16.0-2ubuntu2) ... 2022-07-20T03:05:44.599 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:05:44.658 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:44.725 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T03:05:44.748 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:05:44.768 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T03:05:44.874 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-decorator (4.4.2-0ubuntu1) ... 2022-07-20T03:05:44.900 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T03:05:45.008 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:45.115 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T03:05:45.142 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:05:45.146 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:45.185 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:05:45.187 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-jinja2 (2.10.1-2) ... 2022-07-20T03:05:45.204 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:45.421 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:45.447 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:45.489 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:45.532 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:45.549 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-tempita (0.5.2-6) ... 2022-07-20T03:05:45.580 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:45.811 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:05:45.812 INFO:teuthology.orchestra.run.smithi163.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T03:05:45.824 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:45.831 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-packaging (20.3-1) ... 2022-07-20T03:05:45.841 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:05:45.936 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:45.998 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:46.094 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:46.123 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:46.124 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python-pastedeploy-tpl (2.1.0-1) ... 2022-07-20T03:05:46.215 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:05:46.225 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-wcwidth (0.1.8+dfsg1-3) ... 2022-07-20T03:05:46.294 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:46.422 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:46.502 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:05:46.518 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-paste (3.4.0+dfsg1-1) ... 2022-07-20T03:05:46.533 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:05:46.726 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:05:46.836 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:46.860 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:46.897 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T03:05:46.935 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cmd2 (0.8.5-2.1) ... 2022-07-20T03:05:46.990 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T03:05:46.996 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:46.998 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:47.163 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T03:05:47.284 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:05:47.303 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-werkzeug (0.16.1+dfsg1-2) ... 2022-07-20T03:05:47.336 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:47.398 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:05:47.406 INFO:teuthology.orchestra.run.smithi117.stdout:Adding group ceph....done 2022-07-20T03:05:47.504 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:47.673 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T03:05:47.683 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cryptography (2.8-3ubuntu0.1) ... 2022-07-20T03:05:47.684 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:05:47.730 INFO:teuthology.orchestra.run.smithi117.stdout:Adding system user ceph....done 2022-07-20T03:05:47.765 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:47.814 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:47.874 INFO:teuthology.orchestra.run.smithi117.stdout:Setting system user ceph properties....done 2022-07-20T03:05:47.890 INFO:teuthology.orchestra.run.smithi117.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T03:05:47.896 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:48.001 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:05:48.032 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T03:05:48.064 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up socat (1.7.3.3-2) ... 2022-07-20T03:05:48.099 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:48.170 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:48.214 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:48.303 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:48.354 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T03:05:48.496 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-html5lib (1.0.1-2) ... 2022-07-20T03:05:48.520 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:05:48.580 INFO:teuthology.orchestra.run.smithi179.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T03:05:48.801 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T03:05:48.834 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:48.836 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librdkafka1:amd64 (1.2.1-1ubuntu1) ... 2022-07-20T03:05:48.937 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:48.957 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:48.973 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pluggy (0.13.0-2) ... 2022-07-20T03:05:49.075 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:05:49.085 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:05:49.214 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:05:49.218 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:05:49.218 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:05:49.272 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libxslt1.1:amd64 (1.1.34-4) ... 2022-07-20T03:05:49.339 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:49.404 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:05:49.425 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1 (0.4.2-3build1) ... 2022-07-20T03:05:49.499 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:49.499 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T03:05:49.600 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T03:05:49.747 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-singledispatch (3.4.0.3-2build1) ... 2022-07-20T03:05:49.751 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:49.788 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:05:49.853 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:49.880 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:49.953 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T03:05:49.982 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T03:05:50.003 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-logutils (0.3.3-7) ... 2022-07-20T03:05:50.074 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:50.212 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T03:05:50.262 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:50.279 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:50.302 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblttng-ust0:amd64 (2.11.0-1) ... 2022-07-20T03:05:50.369 INFO:teuthology.orchestra.run.smithi124.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T03:05:50.467 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-simplegeneric (0.8.1-3) ... 2022-07-20T03:05:50.547 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:50.602 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:50.671 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:50.672 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:50.724 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:50.754 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-prettytable (0.7.2-5) ... 2022-07-20T03:05:50.802 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:05:50.883 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:50.893 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:05:50.893 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:05:51.011 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:51.030 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-soupsieve (1.9.5+dfsg-1) ... 2022-07-20T03:05:51.154 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:05:51.342 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:51.354 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:51.363 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-websocket (0.53.0-2ubuntu1) ... 2022-07-20T03:05:51.463 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:51.513 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:51.570 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:05:51.572 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:05:51.623 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:51.659 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/python3-wsdump to provide /usr/bin/wsdump (wsdump) in auto mode 2022-07-20T03:05:51.733 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T03:05:51.790 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:51.931 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libonig5:amd64 (6.9.4-1) ... 2022-07-20T03:05:52.029 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:52.043 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:52.079 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-mako (1.1.0+ds1-1ubuntu2) ... 2022-07-20T03:05:52.146 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:52.205 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:52.333 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:52.367 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:52.408 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cherrypy3 (8.9.1-6) ... 2022-07-20T03:05:52.469 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:52.494 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T03:05:52.618 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:05:52.637 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:05:52.772 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjs-underscore (1.9.1~dfsg-1ubuntu0.20.04.1) ... 2022-07-20T03:05:52.826 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:52.859 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T03:05:52.900 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblbfgsb0:amd64 (3.0+dfsg.3-7build1) ... 2022-07-20T03:05:52.952 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:53.023 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webob (1:1.8.5-2) ... 2022-07-20T03:05:53.049 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:53.082 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:53.162 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:53.170 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:53.182 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:53.314 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:53.335 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-blinker (1.4+dfsg1-0.3ubuntu1) ... 2022-07-20T03:05:53.449 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:53.472 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:53.535 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:05:53.536 INFO:teuthology.orchestra.run.smithi117.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:05:53.612 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librados2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:53.639 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T03:05:53.760 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-scipy (1.3.3-3build1) ... 2022-07-20T03:05:53.766 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:53.889 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:53.912 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T03:05:53.943 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.072 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:54.152 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.202 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T03:05:54.217 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.218 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.312 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.344 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.374 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:54.438 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:54.555 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:54.556 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:54.583 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T03:05:54.598 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T03:05:54.690 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:54.910 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:55.029 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:55.134 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:05:55.171 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:55.202 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:55.275 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librgw2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:55.408 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjq1:amd64 (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:55.450 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:55.518 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:55.527 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pytest (4.6.9-1) ... 2022-07-20T03:05:55.636 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:55.700 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:55.723 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:05:55.831 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:55.918 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-openssl (19.0.0-1build1) ... 2022-07-20T03:05:55.931 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:55.989 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:56.049 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:56.057 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:56.114 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:56.175 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:56.261 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastedeploy (2.1.0-1) ... 2022-07-20T03:05:56.316 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:56.406 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:56.441 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T03:05:56.466 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:56.533 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T03:05:56.555 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-bs4 (4.8.2-1) ... 2022-07-20T03:05:56.816 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T03:05:56.817 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:56.915 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs2 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.049 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-oauthlib (3.1.0-1ubuntu2) ... 2022-07-20T03:05:57.123 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.127 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.228 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.267 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.390 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libradosstriper1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.448 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:57.449 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:57.504 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.554 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.555 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pyasn1-modules (0.2.1-0.2build1) ... 2022-07-20T03:05:57.655 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:57.676 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:05:57.766 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:57.866 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:57.876 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:57.891 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-sklearn (0.22.2.post1+dfsg-5) ... 2022-07-20T03:05:57.987 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:05:58.032 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:05:58.180 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:58.304 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T03:05:58.328 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T03:05:58.426 INFO:teuthology.orchestra.run.smithi117.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:05:58.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T03:05:58.525 INFO:teuthology.orchestra.run.smithi117.stdout:Reading package lists... 2022-07-20T03:05:58.601 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:58.642 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T03:05:58.644 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:05:58.701 INFO:teuthology.orchestra.run.smithi117.stdout:Building dependency tree... 2022-07-20T03:05:58.702 INFO:teuthology.orchestra.run.smithi117.stdout:Reading state information... 2022-07-20T03:05:58.857 INFO:teuthology.orchestra.run.smithi117.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T03:05:58.858 INFO:teuthology.orchestra.run.smithi117.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T03:05:58.861 INFO:teuthology.orchestra.run.smithi117.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T03:05:58.861 INFO:teuthology.orchestra.run.smithi117.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:05:58.862 INFO:teuthology.orchestra.run.smithi117.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:05:58.863 INFO:teuthology.orchestra.run.smithi117.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:05:58.864 INFO:teuthology.orchestra.run.smithi117.stdout:Suggested packages: 2022-07-20T03:05:58.864 INFO:teuthology.orchestra.run.smithi117.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T03:05:58.898 INFO:teuthology.orchestra.run.smithi117.stdout:The following NEW packages will be installed: 2022-07-20T03:05:58.899 INFO:teuthology.orchestra.run.smithi117.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T03:05:58.913 INFO:teuthology.orchestra.run.smithi117.stdout: libssl-dev 2022-07-20T03:05:58.933 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:58.978 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up librbd1 (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:58.987 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:59.075 INFO:teuthology.orchestra.run.smithi117.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:05:59.086 INFO:teuthology.orchestra.run.smithi117.stdout:Need to get 2,914 kB of archives. 2022-07-20T03:05:59.087 INFO:teuthology.orchestra.run.smithi117.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T03:05:59.087 INFO:teuthology.orchestra.run.smithi117.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T03:05:59.089 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:59.101 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libjs-sphinxdoc (1.8.5-7ubuntu3) ... 2022-07-20T03:05:59.158 INFO:teuthology.orchestra.run.smithi163.stdout:Adding group ceph....done 2022-07-20T03:05:59.211 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:59.287 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up xmlstarlet (1.6.1-2build1) ... 2022-07-20T03:05:59.333 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T03:05:59.407 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-lxml:amd64 (4.5.0-1ubuntu0.5) ... 2022-07-20T03:05:59.433 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:05:59.506 INFO:teuthology.orchestra.run.smithi163.stdout:Adding system user ceph....done 2022-07-20T03:05:59.555 INFO:teuthology.orchestra.run.smithi117.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T03:05:59.655 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:59.656 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:05:59.668 INFO:teuthology.orchestra.run.smithi163.stdout:Setting system user ceph properties....done 2022-07-20T03:05:59.683 INFO:teuthology.orchestra.run.smithi163.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T03:05:59.692 INFO:teuthology.orchestra.run.smithi117.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T03:05:59.706 INFO:teuthology.orchestra.run.smithi117.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T03:05:59.721 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T03:05:59.722 INFO:teuthology.orchestra.run.smithi117.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T03:05:59.724 INFO:teuthology.orchestra.run.smithi117.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T03:05:59.724 INFO:teuthology.orchestra.run.smithi117.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T03:05:59.747 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up jq (1.6-1ubuntu0.20.04.1) ... 2022-07-20T03:05:59.764 INFO:teuthology.orchestra.run.smithi117.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T03:05:59.765 INFO:teuthology.orchestra.run.smithi117.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T03:05:59.766 INFO:teuthology.orchestra.run.smithi117.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T03:05:59.833 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T03:05:59.875 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rsa (4.0-3) ... 2022-07-20T03:06:00.019 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.124 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.134 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.140 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-webtest (2.0.32-2) ... 2022-07-20T03:06:00.168 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T03:06:00.185 INFO:teuthology.orchestra.run.smithi117.stdout:Fetched 2,914 kB in 1s (3,133 kB/s) 2022-07-20T03:06:00.235 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:06:00.331 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package flex. 2022-07-20T03:06:00.375 INFO:teuthology.orchestra.run.smithi117.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T03:06:00.375 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:00.404 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.440 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T03:06:00.440 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-modules-core (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.504 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T03:06:00.525 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:06:00.551 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.679 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.778 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/remote-fs.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:06:00.779 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-fuse.target -> /lib/systemd/system/ceph-fuse.target. 2022-07-20T03:06:00.823 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:00.846 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:06:00.911 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package bison. 2022-07-20T03:06:00.942 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T03:06:00.973 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:01.072 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:01.073 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:01.235 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libcephfs-dev (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:01.236 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:01.348 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:01.375 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-requests-oauthlib (1.0.0-1.1build1) ... 2022-07-20T03:06:01.457 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:01.462 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T03:06:01.463 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T03:06:01.479 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:01.533 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:01.596 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T03:06:01.697 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pastescript (2.0.2-4) ... 2022-07-20T03:06:01.732 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:01.834 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package dump. 2022-07-20T03:06:01.846 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T03:06:01.879 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T03:06:01.955 INFO:teuthology.orchestra.run.smithi179.stdout:Adding group ceph....done 2022-07-20T03:06:01.971 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T03:06:02.021 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-pecan (1.3.3-3ubuntu1) ... 2022-07-20T03:06:02.096 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T03:06:02.171 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package indent. 2022-07-20T03:06:02.190 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T03:06:02.231 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T03:06:02.291 INFO:teuthology.orchestra.run.smithi179.stdout:Adding system user ceph....done 2022-07-20T03:06:02.314 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.369 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rados (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.440 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.468 INFO:teuthology.orchestra.run.smithi179.stdout:Setting system user ceph properties....done 2022-07-20T03:06:02.516 INFO:teuthology.orchestra.run.smithi179.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T03:06:02.517 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T03:06:02.531 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rbd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.531 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T03:06:02.531 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.559 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:02.633 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T03:06:02.666 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up rbd-fuse (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.747 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:02.747 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:02.770 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:02.778 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-routes (2.4.1-2) ... 2022-07-20T03:06:02.841 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T03:06:02.859 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T03:06:02.900 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:02.901 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:06:02.949 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T03:06:03.172 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-google-auth (1.5.1-2) ... 2022-07-20T03:06:03.204 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.213 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T03:06:03.230 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:03.272 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:03.397 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.440 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:03.447 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:03.482 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-rgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.558 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T03:06:03.558 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.575 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:03.608 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:03.617 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-cephfs (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.797 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.834 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:03.835 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:03.850 INFO:teuthology.orchestra.run.smithi117.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T03:06:03.867 INFO:teuthology.orchestra.run.smithi117.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T03:06:03.912 INFO:teuthology.orchestra.run.smithi117.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:03.913 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up cephfs-shell (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:03.914 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:04.028 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up python3-kubernetes (7.0.0~a1-2) ... 2022-07-20T03:06:04.210 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:04.386 INFO:teuthology.orchestra.run.smithi124.stdout:Adding group ceph....done 2022-07-20T03:06:04.478 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T03:06:04.501 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:04.606 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:04.733 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:04.733 INFO:teuthology.orchestra.run.smithi124.stdout:Adding system user ceph....done 2022-07-20T03:06:04.790 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:04.791 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:04.851 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:04.863 INFO:teuthology.orchestra.run.smithi124.stdout:Setting system user ceph properties....done 2022-07-20T03:06:04.878 INFO:teuthology.orchestra.run.smithi124.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T03:06:04.881 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T03:06:04.981 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:05.010 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-common (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:05.023 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T03:06:05.128 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:05.234 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:05.270 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T03:06:05.320 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:05.335 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T03:06:05.358 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T03:06:05.359 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:05.398 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:05.475 INFO:teuthology.orchestra.run.smithi117.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T03:06:05.520 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:05.540 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T03:06:05.542 INFO:teuthology.orchestra.run.smithi061.stdout:Adding group ceph....done 2022-07-20T03:06:05.585 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:05.586 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:05.667 INFO:teuthology.orchestra.run.smithi117.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:05.746 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:05.746 INFO:teuthology.orchestra.run.smithi163.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:05.824 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:05.826 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:05.947 INFO:teuthology.orchestra.run.smithi061.stdout:Adding system user ceph....done 2022-07-20T03:06:05.988 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:06.007 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.072 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.120 INFO:teuthology.orchestra.run.smithi061.stdout:Setting system user ceph properties....done 2022-07-20T03:06:06.134 INFO:teuthology.orchestra.run.smithi061.stdout:chown: cannot access '/var/log/ceph/*.log*': No such file or directory 2022-07-20T03:06:06.197 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.281 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:06.282 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:06.286 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target -> /lib/systemd/system/ceph.target. 2022-07-20T03:06:06.310 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:06.311 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:06.343 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.456 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.594 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/rbdmap.service -> /lib/systemd/system/rbdmap.service. 2022-07-20T03:06:06.619 INFO:teuthology.orchestra.run.smithi163.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T03:06:06.726 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.878 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:06.917 INFO:teuthology.orchestra.run.smithi163.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:06.951 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:07.014 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-test (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:07.140 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up radosgw (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:07.319 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T03:06:07.372 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:07.373 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:07.411 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:07.412 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-radosgw.target -> /lib/systemd/system/ceph-radosgw.target. 2022-07-20T03:06:07.788 INFO:teuthology.orchestra.run.smithi117.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T03:06:07.795 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:07.811 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:07.933 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.008 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:08.008 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:08.053 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-base (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.074 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.282 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:08.282 INFO:teuthology.orchestra.run.smithi179.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:08.466 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-crash.service -> /lib/systemd/system/ceph-crash.service. 2022-07-20T03:06:08.489 INFO:teuthology.orchestra.run.smithi117.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:08.491 DEBUG:teuthology.parallel:result is None 2022-07-20T03:06:08.496 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.706 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:08.707 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:08.710 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.851 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.905 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mds (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:08.976 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:09.140 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:09.141 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mds.target -> /lib/systemd/system/ceph-mds.target. 2022-07-20T03:06:09.148 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:09.193 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T03:06:09.308 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:09.544 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:09.590 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:09.771 INFO:teuthology.orchestra.run.smithi163.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:09.808 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:09.809 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mgr.target -> /lib/systemd/system/ceph-mgr.target. 2022-07-20T03:06:09.835 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:09.835 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:10.189 INFO:teuthology.orchestra.run.smithi163.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:10.192 DEBUG:teuthology.orchestra.run.smithi163:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T03:06:10.261 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-cloud (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:10.268 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:10.269 INFO:teuthology.orchestra.run.smithi163.stdout:Reading package lists... 2022-07-20T03:06:10.395 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:10.396 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-osd (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:10.445 INFO:teuthology.orchestra.run.smithi163.stdout:Building dependency tree... 2022-07-20T03:06:10.446 INFO:teuthology.orchestra.run.smithi163.stdout:Reading state information... 2022-07-20T03:06:10.536 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:10.603 INFO:teuthology.orchestra.run.smithi163.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T03:06:10.603 INFO:teuthology.orchestra.run.smithi163.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T03:06:10.603 INFO:teuthology.orchestra.run.smithi163.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T03:06:10.604 INFO:teuthology.orchestra.run.smithi163.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:06:10.604 INFO:teuthology.orchestra.run.smithi163.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:06:10.605 INFO:teuthology.orchestra.run.smithi163.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:06:10.605 INFO:teuthology.orchestra.run.smithi163.stdout:Suggested packages: 2022-07-20T03:06:10.618 INFO:teuthology.orchestra.run.smithi163.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T03:06:10.639 INFO:teuthology.orchestra.run.smithi163.stdout:The following NEW packages will be installed: 2022-07-20T03:06:10.639 INFO:teuthology.orchestra.run.smithi163.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T03:06:10.640 INFO:teuthology.orchestra.run.smithi163.stdout: libssl-dev 2022-07-20T03:06:10.764 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:10.764 INFO:teuthology.orchestra.run.smithi124.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:10.822 INFO:teuthology.orchestra.run.smithi163.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:06:10.823 INFO:teuthology.orchestra.run.smithi163.stdout:Need to get 2,914 kB of archives. 2022-07-20T03:06:10.823 INFO:teuthology.orchestra.run.smithi163.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T03:06:10.824 INFO:teuthology.orchestra.run.smithi163.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T03:06:10.886 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:10.886 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-osd.target -> /lib/systemd/system/ceph-osd.target. 2022-07-20T03:06:11.196 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:11.304 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-k8sevents (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:11.305 INFO:teuthology.orchestra.run.smithi163.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T03:06:11.326 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:11.430 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-diskprediction-local (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:11.437 INFO:teuthology.orchestra.run.smithi163.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T03:06:11.447 INFO:teuthology.orchestra.run.smithi163.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T03:06:11.458 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:11.461 INFO:teuthology.orchestra.run.smithi163.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T03:06:11.472 INFO:teuthology.orchestra.run.smithi163.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T03:06:11.472 INFO:teuthology.orchestra.run.smithi163.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T03:06:11.480 INFO:teuthology.orchestra.run.smithi163.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T03:06:11.480 INFO:teuthology.orchestra.run.smithi163.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T03:06:11.481 INFO:teuthology.orchestra.run.smithi163.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T03:06:11.576 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mon (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:11.589 INFO:teuthology.orchestra.run.smithi124.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T03:06:11.804 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/multi-user.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:11.806 INFO:teuthology.orchestra.run.smithi061.stdout:Created symlink /etc/systemd/system/ceph.target.wants/ceph-mon.target -> /lib/systemd/system/ceph-mon.target. 2022-07-20T03:06:11.911 INFO:teuthology.orchestra.run.smithi124.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:11.937 INFO:teuthology.orchestra.run.smithi163.stdout:Fetched 2,914 kB in 1s (3,128 kB/s) 2022-07-20T03:06:12.069 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package flex. 2022-07-20T03:06:12.121 INFO:teuthology.orchestra.run.smithi163.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T03:06:12.121 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:12.163 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T03:06:12.190 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:12.265 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-cephadm (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:12.390 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:12.509 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up ceph-mgr-dashboard (15.2.16-336-g57544da1-1focal) ... 2022-07-20T03:06:12.619 INFO:teuthology.orchestra.run.smithi179.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:12.620 DEBUG:teuthology.orchestra.run.smithi179:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T03:06:12.647 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package bison. 2022-07-20T03:06:12.660 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for systemd (245.4-4ubuntu3.17) ... 2022-07-20T03:06:12.660 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T03:06:12.700 INFO:teuthology.orchestra.run.smithi179.stdout:Reading package lists... 2022-07-20T03:06:12.701 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:12.880 INFO:teuthology.orchestra.run.smithi179.stdout:Building dependency tree... 2022-07-20T03:06:12.932 INFO:teuthology.orchestra.run.smithi179.stdout:Reading state information... 2022-07-20T03:06:12.935 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:13.040 INFO:teuthology.orchestra.run.smithi179.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T03:06:13.040 INFO:teuthology.orchestra.run.smithi179.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T03:06:13.041 INFO:teuthology.orchestra.run.smithi179.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T03:06:13.041 INFO:teuthology.orchestra.run.smithi179.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:06:13.041 INFO:teuthology.orchestra.run.smithi179.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:06:13.041 INFO:teuthology.orchestra.run.smithi179.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:06:13.042 INFO:teuthology.orchestra.run.smithi179.stdout:Suggested packages: 2022-07-20T03:06:13.042 INFO:teuthology.orchestra.run.smithi179.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T03:06:13.070 INFO:teuthology.orchestra.run.smithi179.stdout:The following NEW packages will be installed: 2022-07-20T03:06:13.070 INFO:teuthology.orchestra.run.smithi179.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T03:06:13.077 INFO:teuthology.orchestra.run.smithi179.stdout: libssl-dev 2022-07-20T03:06:13.117 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T03:06:13.135 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T03:06:13.236 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:13.250 INFO:teuthology.orchestra.run.smithi179.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:06:13.251 INFO:teuthology.orchestra.run.smithi179.stdout:Need to get 2,914 kB of archives. 2022-07-20T03:06:13.251 INFO:teuthology.orchestra.run.smithi179.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T03:06:13.252 INFO:teuthology.orchestra.run.smithi179.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T03:06:13.568 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package dump. 2022-07-20T03:06:13.587 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T03:06:13.635 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T03:06:13.727 INFO:teuthology.orchestra.run.smithi179.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T03:06:13.865 INFO:teuthology.orchestra.run.smithi179.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T03:06:13.870 INFO:teuthology.orchestra.run.smithi179.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T03:06:13.903 INFO:teuthology.orchestra.run.smithi179.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T03:06:13.903 INFO:teuthology.orchestra.run.smithi179.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T03:06:13.904 INFO:teuthology.orchestra.run.smithi179.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T03:06:13.904 INFO:teuthology.orchestra.run.smithi179.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T03:06:13.906 INFO:teuthology.orchestra.run.smithi179.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T03:06:13.906 INFO:teuthology.orchestra.run.smithi179.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T03:06:13.943 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package indent. 2022-07-20T03:06:13.958 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T03:06:14.003 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T03:06:14.308 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T03:06:14.320 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T03:06:14.362 INFO:teuthology.orchestra.run.smithi179.stdout:Fetched 2,914 kB in 1s (3,147 kB/s) 2022-07-20T03:06:14.363 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:14.474 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package flex. 2022-07-20T03:06:14.510 INFO:teuthology.orchestra.run.smithi179.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T03:06:14.511 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:14.587 INFO:teuthology.orchestra.run.smithi124.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:14.607 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T03:06:14.700 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T03:06:14.720 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T03:06:14.750 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:15.097 INFO:teuthology.orchestra.run.smithi124.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:15.108 DEBUG:teuthology.orchestra.run.smithi124:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T03:06:15.109 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package bison. 2022-07-20T03:06:15.127 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T03:06:15.129 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T03:06:15.142 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:15.172 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:15.180 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:15.182 INFO:teuthology.orchestra.run.smithi124.stdout:Reading package lists... 2022-07-20T03:06:15.364 INFO:teuthology.orchestra.run.smithi124.stdout:Building dependency tree... 2022-07-20T03:06:15.364 INFO:teuthology.orchestra.run.smithi124.stdout:Reading state information... 2022-07-20T03:06:15.422 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T03:06:15.428 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:15.483 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:15.520 INFO:teuthology.orchestra.run.smithi124.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T03:06:15.520 INFO:teuthology.orchestra.run.smithi124.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T03:06:15.520 INFO:teuthology.orchestra.run.smithi124.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T03:06:15.523 INFO:teuthology.orchestra.run.smithi124.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:06:15.524 INFO:teuthology.orchestra.run.smithi124.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:06:15.524 INFO:teuthology.orchestra.run.smithi124.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:06:15.525 INFO:teuthology.orchestra.run.smithi124.stdout:Suggested packages: 2022-07-20T03:06:15.525 INFO:teuthology.orchestra.run.smithi124.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T03:06:15.553 INFO:teuthology.orchestra.run.smithi124.stdout:The following NEW packages will be installed: 2022-07-20T03:06:15.555 INFO:teuthology.orchestra.run.smithi124.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T03:06:15.555 INFO:teuthology.orchestra.run.smithi124.stdout: libssl-dev 2022-07-20T03:06:15.570 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T03:06:15.587 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T03:06:15.620 INFO:teuthology.orchestra.run.smithi124.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:06:15.621 INFO:teuthology.orchestra.run.smithi124.stdout:Need to get 2,914 kB of archives. 2022-07-20T03:06:15.621 INFO:teuthology.orchestra.run.smithi124.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T03:06:15.621 INFO:teuthology.orchestra.run.smithi124.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T03:06:15.632 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:15.695 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:15.715 INFO:teuthology.orchestra.run.smithi124.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T03:06:15.746 INFO:teuthology.orchestra.run.smithi124.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T03:06:15.747 INFO:teuthology.orchestra.run.smithi124.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T03:06:15.748 INFO:teuthology.orchestra.run.smithi124.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T03:06:15.751 INFO:teuthology.orchestra.run.smithi124.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T03:06:15.752 INFO:teuthology.orchestra.run.smithi124.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T03:06:15.752 INFO:teuthology.orchestra.run.smithi124.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T03:06:15.755 INFO:teuthology.orchestra.run.smithi124.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T03:06:15.755 INFO:teuthology.orchestra.run.smithi124.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T03:06:15.785 INFO:teuthology.orchestra.run.smithi163.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T03:06:15.797 INFO:teuthology.orchestra.run.smithi163.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T03:06:15.834 INFO:teuthology.orchestra.run.smithi163.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:16.031 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package dump. 2022-07-20T03:06:16.045 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T03:06:16.046 INFO:teuthology.orchestra.run.smithi061.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:16.050 DEBUG:teuthology.orchestra.run.smithi061:> sudo DEBIAN_FRONTEND=noninteractive apt-get -y --force-yes -o Dpkg::Options::="--force-confdef" -o Dpkg::Options::="--force-confold" install bison flex libelf-dev libssl-dev dump indent libaio-dev libtool-bin uuid-dev xfslibs-dev 2022-07-20T03:06:16.097 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T03:06:16.135 INFO:teuthology.orchestra.run.smithi124.stdout:Fetched 2,914 kB in 0s (13.9 MB/s) 2022-07-20T03:06:16.136 INFO:teuthology.orchestra.run.smithi061.stdout:Reading package lists... 2022-07-20T03:06:16.262 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package flex. 2022-07-20T03:06:16.281 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T03:06:16.311 INFO:teuthology.orchestra.run.smithi124.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T03:06:16.312 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:16.322 INFO:teuthology.orchestra.run.smithi061.stdout:Building dependency tree... 2022-07-20T03:06:16.323 INFO:teuthology.orchestra.run.smithi061.stdout:Reading state information... 2022-07-20T03:06:16.350 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T03:06:16.377 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package indent. 2022-07-20T03:06:16.400 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T03:06:16.401 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:16.443 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T03:06:16.489 INFO:teuthology.orchestra.run.smithi061.stdout:libtool-bin is already the newest version (2.4.6-14). 2022-07-20T03:06:16.490 INFO:teuthology.orchestra.run.smithi061.stdout:xfslibs-dev is already the newest version (5.3.0-1ubuntu2). 2022-07-20T03:06:16.491 INFO:teuthology.orchestra.run.smithi061.stdout:uuid-dev is already the newest version (2.34-0.1ubuntu9.3). 2022-07-20T03:06:16.491 INFO:teuthology.orchestra.run.smithi061.stdout:The following packages were automatically installed and are no longer required: 2022-07-20T03:06:16.493 INFO:teuthology.orchestra.run.smithi061.stdout: libboost-iostreams1.71.0 libboost-thread1.71.0 2022-07-20T03:06:16.493 INFO:teuthology.orchestra.run.smithi061.stdout:Use 'sudo apt autoremove' to remove them. 2022-07-20T03:06:16.494 INFO:teuthology.orchestra.run.smithi061.stdout:Suggested packages: 2022-07-20T03:06:16.494 INFO:teuthology.orchestra.run.smithi061.stdout: bison-doc flex-doc indent-doc libssl-doc 2022-07-20T03:06:16.531 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:16.533 INFO:teuthology.orchestra.run.smithi061.stdout:The following NEW packages will be installed: 2022-07-20T03:06:16.533 INFO:teuthology.orchestra.run.smithi061.stdout: bison dump flex indent libaio-dev libelf-dev libfl-dev libfl2 liblzo2-2 2022-07-20T03:06:16.534 INFO:teuthology.orchestra.run.smithi061.stdout: libssl-dev 2022-07-20T03:06:16.663 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:16.706 INFO:teuthology.orchestra.run.smithi061.stdout:0 upgraded, 10 newly installed, 0 to remove and 1 not upgraded. 2022-07-20T03:06:16.706 INFO:teuthology.orchestra.run.smithi061.stdout:Need to get 2,914 kB of archives. 2022-07-20T03:06:16.706 INFO:teuthology.orchestra.run.smithi061.stdout:After this operation, 12.4 MB of additional disk space will be used. 2022-07-20T03:06:16.706 INFO:teuthology.orchestra.run.smithi061.stdout:Get:1 http://archive.ubuntu.com/ubuntu focal/main amd64 flex amd64 2.6.4-6.2 [317 kB] 2022-07-20T03:06:16.729 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T03:06:16.745 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T03:06:16.787 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:16.801 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:16.847 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package bison. 2022-07-20T03:06:16.862 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T03:06:16.926 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:16.939 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:17.024 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T03:06:17.044 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T03:06:17.045 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T03:06:17.098 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:17.118 INFO:teuthology.orchestra.run.smithi163.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T03:06:17.165 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:17.189 INFO:teuthology.orchestra.run.smithi061.stdout:Get:2 http://archive.ubuntu.com/ubuntu focal/main amd64 bison amd64 2:3.5.1+dfsg-1 [657 kB] 2022-07-20T03:06:17.244 INFO:teuthology.orchestra.run.smithi163.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T03:06:17.319 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T03:06:17.335 INFO:teuthology.orchestra.run.smithi061.stdout:Get:3 http://archive.ubuntu.com/ubuntu focal/main amd64 liblzo2-2 amd64 2.10-2 [50.8 kB] 2022-07-20T03:06:17.335 INFO:teuthology.orchestra.run.smithi061.stdout:Get:4 http://archive.ubuntu.com/ubuntu focal/universe amd64 dump amd64 0.4b46-6 [130 kB] 2022-07-20T03:06:17.345 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T03:06:17.348 INFO:teuthology.orchestra.run.smithi061.stdout:Get:5 http://archive.ubuntu.com/ubuntu focal/main amd64 indent amd64 2.2.12-1 [86.9 kB] 2022-07-20T03:06:17.356 INFO:teuthology.orchestra.run.smithi061.stdout:Get:6 http://archive.ubuntu.com/ubuntu focal/main amd64 libaio-dev amd64 0.3.112-5 [13.7 kB] 2022-07-20T03:06:17.357 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T03:06:17.361 INFO:teuthology.orchestra.run.smithi061.stdout:Get:7 http://archive.ubuntu.com/ubuntu focal/main amd64 libelf-dev amd64 0.176-1.1build1 [57.0 kB] 2022-07-20T03:06:17.362 INFO:teuthology.orchestra.run.smithi061.stdout:Get:8 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl2 amd64 2.6.4-6.2 [11.5 kB] 2022-07-20T03:06:17.362 INFO:teuthology.orchestra.run.smithi061.stdout:Get:9 http://archive.ubuntu.com/ubuntu focal/main amd64 libfl-dev amd64 2.6.4-6.2 [6,316 B] 2022-07-20T03:06:17.363 INFO:teuthology.orchestra.run.smithi061.stdout:Get:10 http://archive.ubuntu.com/ubuntu focal-updates/main amd64 libssl-dev amd64 1.1.1f-1ubuntu2.16 [1,584 kB] 2022-07-20T03:06:17.430 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T03:06:17.443 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:17.451 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:17.466 INFO:teuthology.orchestra.run.smithi163.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:17.502 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:17.593 INFO:teuthology.orchestra.run.smithi163.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:17.731 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T03:06:17.746 INFO:teuthology.orchestra.run.smithi163.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:17.746 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:17.788 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:17.796 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package dump. 2022-07-20T03:06:17.817 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T03:06:17.834 INFO:teuthology.orchestra.run.smithi061.stdout:Fetched 2,914 kB in 1s (3,130 kB/s) 2022-07-20T03:06:17.882 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T03:06:17.978 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package flex. 2022-07-20T03:06:18.030 INFO:teuthology.orchestra.run.smithi179.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T03:06:18.032 INFO:teuthology.orchestra.run.smithi061.stdout:(Reading database ... (Reading database ... 5% (Reading database ... 10% (Reading database ... 15% (Reading database ... 20% (Reading database ... 25% (Reading database ... 30% (Reading database ... 35% (Reading database ... 40% (Reading database ... 45% (Reading database ... 50% (Reading database ... 55% (Reading database ... 60% (Reading database ... 65% (Reading database ... 70% (Reading database ... 75% (Reading database ... 80% (Reading database ... 85% (Reading database ... 90% (Reading database ... 95% (Reading database ... 100% (Reading database ... 109762 files and directories currently installed.) 2022-07-20T03:06:18.032 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../0-flex_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:18.043 INFO:teuthology.orchestra.run.smithi179.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T03:06:18.070 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking flex (2.6.4-6.2) ... 2022-07-20T03:06:18.093 INFO:teuthology.orchestra.run.smithi179.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:18.183 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package indent. 2022-07-20T03:06:18.200 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T03:06:18.224 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T03:06:18.517 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package bison. 2022-07-20T03:06:18.518 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T03:06:18.527 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T03:06:18.529 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../1-bison_2%3a3.5.1+dfsg-1_amd64.deb ... 2022-07-20T03:06:18.566 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T03:06:18.568 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:18.569 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:18.687 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:18.823 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:18.839 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T03:06:18.855 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T03:06:18.897 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:18.940 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:19.002 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package liblzo2-2:amd64. 2022-07-20T03:06:19.020 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../2-liblzo2-2_2.10-2_amd64.deb ... 2022-07-20T03:06:19.079 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:19.102 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:19.194 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:19.254 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T03:06:19.269 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:19.314 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T03:06:19.335 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:19.415 INFO:teuthology.orchestra.run.smithi179.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T03:06:19.465 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package dump. 2022-07-20T03:06:19.469 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:19.479 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../3-dump_0.4b46-6_amd64.deb ... 2022-07-20T03:06:19.533 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking dump (0.4b46-6) ... 2022-07-20T03:06:19.548 INFO:teuthology.orchestra.run.smithi163.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T03:06:19.556 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T03:06:19.562 INFO:teuthology.orchestra.run.smithi179.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T03:06:19.568 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:19.623 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:19.629 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T03:06:19.801 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package indent. 2022-07-20T03:06:19.818 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../4-indent_2.2.12-1_amd64.deb ... 2022-07-20T03:06:19.869 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking indent (2.2.12-1) ... 2022-07-20T03:06:19.893 INFO:teuthology.orchestra.run.smithi124.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T03:06:19.903 INFO:teuthology.orchestra.run.smithi124.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T03:06:19.948 INFO:teuthology.orchestra.run.smithi124.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:20.148 INFO:teuthology.orchestra.run.smithi179.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:20.213 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libaio-dev:amd64. 2022-07-20T03:06:20.233 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../5-libaio-dev_0.3.112-5_amd64.deb ... 2022-07-20T03:06:20.271 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:20.300 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:20.452 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:20.462 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T03:06:20.552 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libelf-dev:amd64. 2022-07-20T03:06:20.562 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../6-libelf-dev_0.176-1.1build1_amd64.deb ... 2022-07-20T03:06:20.579 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:20.608 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:20.666 INFO:teuthology.orchestra.run.smithi163.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:20.671 DEBUG:teuthology.parallel:result is None 2022-07-20T03:06:20.704 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:20.850 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:20.912 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libfl2:amd64. 2022-07-20T03:06:20.924 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../7-libfl2_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:20.972 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:21.005 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:21.098 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:21.233 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libfl-dev:amd64. 2022-07-20T03:06:21.243 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T03:06:21.245 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../8-libfl-dev_2.6.4-6.2_amd64.deb ... 2022-07-20T03:06:21.291 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:21.331 INFO:teuthology.orchestra.run.smithi124.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T03:06:21.384 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:21.480 INFO:teuthology.orchestra.run.smithi124.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T03:06:21.547 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T03:06:21.565 INFO:teuthology.orchestra.run.smithi061.stdout:Selecting previously unselected package libssl-dev:amd64. 2022-07-20T03:06:21.583 INFO:teuthology.orchestra.run.smithi061.stdout:Preparing to unpack .../9-libssl-dev_1.1.1f-1ubuntu2.16_amd64.deb ... 2022-07-20T03:06:21.624 INFO:teuthology.orchestra.run.smithi061.stdout:Unpacking libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:21.701 INFO:teuthology.orchestra.run.smithi124.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:21.825 INFO:teuthology.orchestra.run.smithi124.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:21.974 INFO:teuthology.orchestra.run.smithi124.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:22.072 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up flex (2.6.4-6.2) ... 2022-07-20T03:06:22.212 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up liblzo2-2:amd64 (2.10-2) ... 2022-07-20T03:06:22.277 INFO:teuthology.orchestra.run.smithi179.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T03:06:22.340 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libaio-dev:amd64 (0.3.112-5) ... 2022-07-20T03:06:22.458 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libssl-dev:amd64 (1.1.1f-1ubuntu2.16) ... 2022-07-20T03:06:22.571 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libelf-dev:amd64 (0.176-1.1build1) ... 2022-07-20T03:06:22.696 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libfl2:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:22.829 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up dump (0.4b46-6) ... 2022-07-20T03:06:22.915 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/sbin/rmt-dump to provide /usr/sbin/rmt (rmt) in auto mode 2022-07-20T03:06:22.958 INFO:teuthology.orchestra.run.smithi179.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:22.960 DEBUG:teuthology.parallel:result is None 2022-07-20T03:06:22.977 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up bison (2:3.5.1+dfsg-1) ... 2022-07-20T03:06:23.071 INFO:teuthology.orchestra.run.smithi061.stdout:update-alternatives: using /usr/bin/bison.yacc to provide /usr/bin/yacc (yacc) in auto mode 2022-07-20T03:06:23.140 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up indent (2.2.12-1) ... 2022-07-20T03:06:23.291 INFO:teuthology.orchestra.run.smithi061.stdout:Setting up libfl-dev:amd64 (2.6.4-6.2) ... 2022-07-20T03:06:23.406 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for libc-bin (2.31-0ubuntu9.9) ... 2022-07-20T03:06:23.550 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for man-db (2.9.1-1) ... 2022-07-20T03:06:23.768 INFO:teuthology.orchestra.run.smithi124.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T03:06:24.512 INFO:teuthology.orchestra.run.smithi124.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:24.513 DEBUG:teuthology.parallel:result is None 2022-07-20T03:06:25.397 INFO:teuthology.orchestra.run.smithi061.stdout:Processing triggers for install-info (6.7.0.dfsg.2-5) ... 2022-07-20T03:06:26.091 INFO:teuthology.orchestra.run.smithi061.stderr:W: --force-yes is deprecated, use one of the options starting with --allow instead. 2022-07-20T03:06:26.097 DEBUG:teuthology.parallel:result is None 2022-07-20T03:06:26.099 DEBUG:teuthology.orchestra.run.smithi061:> sudo lsb_release -is 2022-07-20T03:06:26.160 INFO:teuthology.orchestra.run.smithi061.stdout:Ubuntu 2022-07-20T03:06:26.161 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:06:26.162 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:06:26.397 DEBUG:teuthology.orchestra.run.smithi061:> dpkg-query -W -f '${Version}' ceph 2022-07-20T03:06:26.435 INFO:teuthology.orchestra.run.smithi061.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T03:06:26.436 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T03:06:26.436 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T03:06:26.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo lsb_release -is 2022-07-20T03:06:26.494 INFO:teuthology.orchestra.run.smithi117.stdout:Ubuntu 2022-07-20T03:06:26.519 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:06:26.520 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:06:26.720 DEBUG:teuthology.orchestra.run.smithi117:> dpkg-query -W -f '${Version}' ceph 2022-07-20T03:06:26.762 INFO:teuthology.orchestra.run.smithi117.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T03:06:26.763 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T03:06:26.764 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T03:06:26.765 DEBUG:teuthology.orchestra.run.smithi124:> sudo lsb_release -is 2022-07-20T03:06:26.826 INFO:teuthology.orchestra.run.smithi124.stdout:Ubuntu 2022-07-20T03:06:26.831 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:06:26.833 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:06:27.106 DEBUG:teuthology.orchestra.run.smithi124:> dpkg-query -W -f '${Version}' ceph 2022-07-20T03:06:27.133 INFO:teuthology.orchestra.run.smithi124.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T03:06:27.133 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T03:06:27.134 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T03:06:27.135 DEBUG:teuthology.orchestra.run.smithi163:> sudo lsb_release -is 2022-07-20T03:06:27.188 INFO:teuthology.orchestra.run.smithi163.stdout:Ubuntu 2022-07-20T03:06:27.208 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:06:27.209 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:06:27.425 DEBUG:teuthology.orchestra.run.smithi163:> dpkg-query -W -f '${Version}' ceph 2022-07-20T03:06:27.460 INFO:teuthology.orchestra.run.smithi163.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T03:06:27.461 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T03:06:27.461 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T03:06:27.481 DEBUG:teuthology.orchestra.run.smithi179:> sudo lsb_release -is 2022-07-20T03:06:27.545 INFO:teuthology.orchestra.run.smithi179.stdout:Ubuntu 2022-07-20T03:06:27.560 DEBUG:teuthology.misc:System to be installed: Ubuntu 2022-07-20T03:06:27.561 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=ubuntu%2F20.04%2Fx86_64&sha1=57544da114dbbe1c44f27b2ea18e3233e4e917e0 2022-07-20T03:06:27.888 DEBUG:teuthology.orchestra.run.smithi179:> dpkg-query -W -f '${Version}' ceph 2022-07-20T03:06:27.934 INFO:teuthology.orchestra.run.smithi179.stdout:15.2.16-336-g57544da1-1focal 2022-07-20T03:06:27.935 INFO:teuthology.packaging:The installed version of ceph is 15.2.16-336-g57544da1-1focal 2022-07-20T03:06:27.936 INFO:teuthology.task.install:The correct ceph version 15.2.16-336-g57544da1-1focal is installed. 2022-07-20T03:06:27.937 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2022-07-20T03:06:27.938 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:27.951 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T03:06:28.002 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:28.003 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T03:06:28.021 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:06:28.021 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T03:06:28.042 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:06:28.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T03:06:28.107 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:06:28.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2022-07-20T03:06:28.135 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2022-07-20T03:06:28.136 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:28.136 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T03:06:28.156 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T03:06:28.224 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:28.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T03:06:28.246 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T03:06:28.324 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:06:28.325 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T03:06:28.348 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T03:06:28.401 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:06:28.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T03:06:28.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T03:06:28.567 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:06:28.568 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/daemon-helper 2022-07-20T03:06:28.608 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/daemon-helper 2022-07-20T03:06:28.707 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2022-07-20T03:06:28.708 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:28.708 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T03:06:28.754 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T03:06:28.821 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:28.821 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T03:06:28.879 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T03:06:28.899 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:06:28.900 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T03:06:28.921 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T03:06:28.940 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:06:28.941 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T03:06:28.968 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T03:06:29.031 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:06:29.032 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/usr/bin/adjust-ulimits 2022-07-20T03:06:29.092 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2022-07-20T03:06:29.167 INFO:teuthology.run_tasks:Running task ceph... 2022-07-20T03:06:29.290 INFO:tasks.ceph:Making ceph log dir writeable by non-root... 2022-07-20T03:06:29.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 777 /var/log/ceph 2022-07-20T03:06:29.295 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 777 /var/log/ceph 2022-07-20T03:06:29.302 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 777 /var/log/ceph 2022-07-20T03:06:29.307 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 777 /var/log/ceph 2022-07-20T03:06:29.321 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 777 /var/log/ceph 2022-07-20T03:06:29.337 INFO:tasks.ceph:Disabling ceph logrotate... 2022-07-20T03:06:29.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T03:06:29.362 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T03:06:29.383 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T03:06:29.393 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T03:06:29.398 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/logrotate.d/ceph 2022-07-20T03:06:29.428 INFO:tasks.ceph:Creating extra log directories... 2022-07-20T03:06:29.429 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T03:06:29.448 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T03:06:29.458 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T03:06:29.462 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T03:06:29.478 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/log/ceph/valgrind /var/log/ceph/profiling-logger 2022-07-20T03:06:29.533 INFO:tasks.ceph:Setting up log rotation with {'ceph-mds': '10G', 'ceph-osd': '10G'} 2022-07-20T03:06:29.534 INFO:tasks.ceph:writing logrotate stanza for ceph-mds 2022-07-20T03:06:29.534 INFO:tasks.ceph:writing logrotate stanza for ceph-osd 2022-07-20T03:06:29.535 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:29.535 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T03:06:29.592 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:29.690 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:29.690 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T03:06:29.783 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:29.832 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:06:29.833 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T03:06:29.846 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:29.887 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:06:29.889 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T03:06:29.899 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:29.941 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:06:29.942 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/logrotate.ceph-test.conf 2022-07-20T03:06:29.964 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv /home/ubuntu/cephtest/logrotate.ceph-test.conf /etc/logrotate.d/ceph-test.conf && sudo chmod 0644 /etc/logrotate.d/ceph-test.conf && sudo chown root.root /etc/logrotate.d/ceph-test.conf 2022-07-20T03:06:30.098 INFO:tasks.ceph:Creating ceph cluster ceph... 2022-07-20T03:06:30.099 INFO:tasks.ceph:config {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'fs': 'xfs', 'mkfs_options': None, 'mount_options': None, 'skip_mgr_daemons': False, 'log_ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'cpu_profile': set(), 'cluster': 'ceph', 'mon_bind_msgr2': True, 'mon_bind_addrvec': True} 2022-07-20T03:06:30.100 INFO:tasks.ceph:ctx.config {'archive_path': '/home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436', 'branch': 'octopus-release', 'description': 'fs/multiclient/{begin clusters/1-mds-3-client conf/{client mds mon osd} distros/ubuntu_latest mount/fuse objectstore-ec/filestore-xfs overrides/{frag_enable whitelist_health whitelist_wrongly_marked_down} tasks/cephfs_misc_tests}', 'email': 'ceph-qa@ceph.io', 'first_in_suite': False, 'job_id': '6939436', 'kernel': {'kdb': True, 'sha1': 'distro'}, 'last_in_suite': False, 'log-rotate': {'ceph-mds': '10G', 'ceph-osd': '10G'}, 'machine_type': 'smithi', 'name': 'yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi', 'no_nested_subset': False, 'nuke-on-error': True, 'openstack': [{'volumes': {'count': 4, 'size': 30}}, {'machine': {'disk': 200}}], 'os_type': 'ubuntu', 'os_version': '20.04', 'overrides': {'admin_socket': {'branch': 'octopus-release'}, 'ceph': {'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mds': {'debug mds': 20, 'debug ms': 1, 'mds bal frag': True, 'mds bal fragment size max': 10000, 'mds bal merge size': 5, 'mds bal split bits': 3, 'mds bal split size': 100, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': '15m', 'rados osd op timeout': '15m'}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'debug ms': 1, 'debug osd': 20, 'osd objectstore': 'filestore', 'osd op complaint time': 180, 'osd sloppy crc': True}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding', 'evicting unresponsive client', 'POOL_APP_NOT_ENABLED', 'has not responded to cap revoke by MDS for over', 'MDS_CLIENT_LATE_RELEASE', 'responding to mclientcaps'], 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}, 'ceph-deploy': {'conf': {'client': {'log file': '/var/log/ceph/ceph-$name.$pid.log'}, 'mon': {'osd default pool size': 2}, 'osd': {'osd objectstore': 'filestore', 'osd sloppy crc': True}}, 'filestore': True, 'fs': 'xfs'}, 'install': {'ceph': {'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, 'workunit': {'branch': 'octopus-release', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, 'owner': 'scheduled_yuriw@teuthology', 'priority': 75, 'repo': 'https://github.com/ceph/ceph-ci.git', 'roles': [['mon.a', 'mgr.y', 'mds.a', 'osd.0', 'osd.1', 'osd.2', 'osd.3'], ['mon.b', 'mon.c', 'mgr.x', 'mds.b', 'osd.4', 'osd.5', 'osd.6', 'osd.7'], ['client.0'], ['client.1'], ['client.2']], 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0', 'sleep_before_teardown': 0, 'suite': 'fs', 'suite_branch': 'octopus-release', 'suite_path': '/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa', 'suite_relpath': 'qa', 'suite_repo': 'https://github.com/ceph/ceph-ci.git', 'suite_sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0', 'targets': {'smithi061.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCkSibgklEPcg/D5L7g4g6TdZHa/6YDqG7LnZsEeMEeLuhpENnLujSS4Ssu0MC73/3Jw9mr7Aur73Ek47zSxSAIBoErrsQxODxoA92w+sJlhltq6f1QpeS2wtWSUuHhzDU8xA8GShvt1AY1y4ENdtLuAWi+rPE9LDvLc1+ACvyXdrCv27GpwiS3kCOnrg1UGRosE9GBNNOq0jJhHilZpe2C3M6GYlB6jZlqvfCilqHz0eJsONbmANoYNP+NtrzI87thP84vK2D8aJ/i6PPh3SG3V1nJ47ieoFzZK9+2U9g6WGjhrn1PVtvY/ZRvJeV3Tx3MQ2lyLTAJPghruTQGfSg1gbULvKt4nTGUBDVuclVRFmvnS1lgUht4xXGJ5aYznn9vAtZLIYIPvajuWIhmx5i92cpDt3gPz1jUhHAt7+Fg3VsnHcInPb/lx5Jf3vrde3QN/IgtxQ3BU64iFivd5870bpBJhrx0tBVKFGsdXJtPTNWyj+lWdir6efq0xWnCsHk=', 'smithi117.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCsl5o/375+L6CXtHmfxBAvSLtibmqiMrCNzqNA1GGk4T1vHRYpAmUe2zhAJT5a88q6Hf+STxMlnzG8uVzIG1wP2vF/Go6tnXbQlFRU93JrrZksVwFSiwMgDh4rtlHAPkG9ar1VpN0g23lR24Ihbe5rKr9XNFv0PsyO/vdCgWv/ZO0ef8QGCgsRZwvOxEONZBdi/gtIwggJJ00S56xmehVzmtNwSQPpYNDM46DM9HpclNhQCUDfAvkU+iDxC5Y+/wSa0im849B1JqtySSh2vHZP+stvbVp7ZxQM2NR1EYFtRjjaZ7+a8oPRG2sCDFZP7h1r4cYxMG/HMrKIbtkVih4Rn8I5yf4grweCZj3fzqtfxjhFhiHIa7vBYr82gk4+KkCkRT91knGgDCCx2HBzBtJXXK5xvXHd5qdWpY+jLttIaI3Y8V0dmc2XAgt9nLVn9X7O69liJTJpz/VMEt/JKTaO+p7Jg7uBeEJYaXclYiZDqKZhXSOMufK7aujUkWjx7nM=', 'smithi124.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDsuiOyUHlO6/aHMI2qURANqrnaYuRHNVRrnomZg6UyfYlQBIs3tgZtNBFW0lGTBQ4U4arIAW07cVXSeNEdEZBMA/JmtUF0VqhDf4NWmDV3mdVbX0xHi0mlI9Zhmihu1m6OjWx6ebiPzeKcIaOtyGcCw0AnEXZtHnJoWMjxQOgJacMPTP/dYguwDLRszpaWDO9srz5UyOwOC5X7MYMoZ/FfntMUfzIKOMvzZe4YO19DCmwxk9+osnSB/oeRa1Q7YwGZsRHrlVEp4PsriNwwjIqBm/Y0i7EiMm73VuSkaXPwP3WSxhavtfmlsXUpvqdz65hX2OQGi+2ypVCC+cXtbkIXgDJvHsliQE8LrxIvXPOupvq1zeregBXD+rU4zRpxps0UpwWbSh96HYATmx28IzSO6DmhaqnWkG6QrCJ0F4JpxqC19bD3vC2GjcQmUocIc1KfCtVwTB1ruyQEdDzml1+S1Ze7cT7IJg+kiXR9QtMrIppgn1v6ERNfsDVG6Rc0zus=', 'smithi163.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDHkJsiXzjCnt+uiHjfkqyokpi6rumJSyLxdJpLT9zTAudmlqac21M2q6eQl5edcd8WcOqdmuvonrhZtZ7uSFDJS6nA+oPZauGP8kOVAZlSQGdMWJIGtrsKM0QxiZC1BaMBAUBq7o8+lXElwrpI4CLPZf1Dy8O8QYPin76R5IE8HEt9PPELXjHWH3s/QFZqNlBqrPsv5FUzE5crQFvarhK7PQaokC4JFKHBtsn+bMLXjQitx7JrWO8K61jrHR1K7v3Jqxhkk8CXWq+nOrQB7cC5uA1LTPmdp51+aZTvEAp3BHgGJc+mbsyTGh1FtGakZOywvvcpjMmHzrWzpItkqrAgP1tf9l/cWzu/+NOmQ+B5yUSOQGrPgxxnurQy7hXA4jK7wQTi4EX2vw/03q8TnLoW3g1mN/guEZ4qEd7xSRTrC0QyEd5fd45WI/Twac2VfJDvjk1H1QIDRCAQ0gJhEBq1S0z+z2axtENKgOyPZyaiEZVcK42ZpLX7Pi+cr17XYDM=', 'smithi179.front.sepia.ceph.com': 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQCv/35ahvZMaabK+FQzTylGNcl63M93AHbQjzn+OqXzJAoB+Hz3xj2lTjCmLu+WhTyBd7Jm1Zy5Gtgpn7SfyVrdRz203PrDkWOILOx99XYq51g8JukNajy+7FHWjyGY/rp3pNAKWQKGuc4PfAUbVz4xDlsHjX0L7clfmVV/hG56nMEXipa/NYFS0n6Mh2NIs7gtbP/PYye4logOB1TUmUKOSpmFVe7latLRVtn7Qi0U7QaLITTxXT1eBEKSESX0h8z0WgFYLj6eKNllftzmZkl6NCjKsmUTLXkun+JHJp2QAlRd5F1BGSjA1fD8lMbuDIeeePEFgvVZRqAwkrkInHsBOafQvdDGadEoHEBKiPA95AEBr0YoS4YLE73CjX/A5SgdfXJiwS2Xdabj7hDEnIiSeFU7HgQwE7Yo/c474oCQgwf3NSye3fsnxNQ+XBS3Zj6a73Esnm7AXO5o6OlUOQhLSdDQ5NdumyzLad4uDe6zLqDJgckh4ekqimBLa9afKhk='}, 'tasks': [{'internal.check_packages': None}, {'internal.buildpackages_prep': None}, {'internal.save_config': None}, {'internal.check_lock': None}, {'internal.add_remotes': None}, {'console_log': None}, {'internal.connect': None}, {'internal.push_inventory': None}, {'internal.serialize_remote_roles': None}, {'internal.check_conflict': None}, {'internal.check_ceph_data': None}, {'internal.vm_setup': None}, {'kernel': {'kdb': True, 'sha1': 'distro'}}, {'internal.base': None}, {'internal.archive_upload': None}, {'internal.archive': None}, {'internal.coredump': None}, {'internal.sudo': None}, {'internal.syslog': None}, {'internal.timer': None}, {'pcp': None}, {'selinux': None}, {'ansible.cephlab': None}, {'clock': None}, {'install': {'extra_packages': {'deb': ['python3-cephfs', 'cephfs-shell'], 'rpm': ['python3-cephfs']}, 'extra_system_packages': {'deb': ['bison', 'flex', 'libelf-dev', 'libssl-dev', 'dump', 'indent', 'libaio-dev', 'libtool-bin', 'uuid-dev', 'xfslibs-dev'], 'rpm': ['bison', 'flex', 'elfutils-libelf-devel', 'openssl-devel', 'libacl-devel', 'libaio-devel', 'libattr-devel', 'libtool', 'libuuid-devel', 'xfsdump', 'xfsprogs', 'xfsprogs-devel', 'libaio-devel', 'libtool', 'libuuid-devel', 'xfsprogs-devel']}, 'flavor': 'default', 'sha1': '57544da114dbbe1c44f27b2ea18e3233e4e917e0'}}, {'ceph': None}, {'ceph-fuse': None}, {'cephfs_test_runner': {'modules': ['tasks.cephfs.test_misc']}}], 'teuthology_branch': 'main', 'teuthology_sha1': '89d874d555f56b1c1f383df72487aecec0ee6cf9', 'timestamp': '2022-07-19_23:28:53', 'tube': 'smithi', 'user': 'yuriw', 'verbose': True, 'worker_log': '/home/teuthworker/archive/worker_logs/dispatcher.smithi.1713'} 2022-07-20T03:06:30.103 DEBUG:teuthology.orchestra.run.smithi061:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T03:06:30.113 DEBUG:teuthology.orchestra.run.smithi117:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T03:06:30.155 DEBUG:teuthology.orchestra.run.smithi124:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T03:06:30.170 DEBUG:teuthology.orchestra.run.smithi163:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T03:06:30.195 DEBUG:teuthology.orchestra.run.smithi179:> install -d -m0755 -- /home/ubuntu/cephtest/ceph.data 2022-07-20T03:06:30.205 DEBUG:teuthology.orchestra.run.smithi061:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T03:06:30.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T03:06:30.243 DEBUG:teuthology.orchestra.run.smithi124:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T03:06:30.278 DEBUG:teuthology.orchestra.run.smithi163:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T03:06:30.297 DEBUG:teuthology.orchestra.run.smithi179:> sudo install -d -m0777 -- /var/run/ceph 2022-07-20T03:06:30.325 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:30.326 DEBUG:teuthology.orchestra.run.smithi061:> dd if=/scratch_devs of=/dev/stdout 2022-07-20T03:06:30.347 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-20T03:06:30.348 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-20T03:06:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-20T03:06:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.370 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 359 Links: 1 2022-07-20T03:06:30.371 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.371 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-20 03:03:17.775496856 +0000 2022-07-20T03:06:30.372 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-20 03:03:17.771496856 +0000 2022-07-20T03:06:30.372 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-20 03:03:17.771496856 +0000 2022-07-20T03:06:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-20T03:06:30.392 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-20T03:06:30.393 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-20T03:06:30.393 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000390781 s, 1.3 MB/s 2022-07-20T03:06:30.394 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-20T03:06:30.443 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-20T03:06:30.443 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.443 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 364 Links: 1 2022-07-20T03:06:30.444 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.444 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-20 03:03:18.891496844 +0000 2022-07-20T03:06:30.444 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-20 03:03:18.419496849 +0000 2022-07-20T03:06:30.444 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-20 03:03:18.419496849 +0000 2022-07-20T03:06:30.445 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-20T03:06:30.453 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-20T03:06:30.454 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-20T03:06:30.454 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000402804 s, 1.3 MB/s 2022-07-20T03:06:30.458 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-20T03:06:30.465 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-20T03:06:30.467 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.467 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 372 Links: 1 2022-07-20T03:06:30.468 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.468 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-20 03:03:19.535496837 +0000 2022-07-20T03:06:30.469 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-20 03:03:19.131496841 +0000 2022-07-20T03:06:30.469 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-20 03:03:19.131496841 +0000 2022-07-20T03:06:30.470 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-20T03:06:30.477 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-20T03:06:30.478 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-20T03:06:30.478 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000359462 s, 1.4 MB/s 2022-07-20T03:06:30.480 DEBUG:teuthology.orchestra.run.smithi061:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-20T03:06:30.487 INFO:teuthology.orchestra.run.smithi061.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-20T03:06:30.487 INFO:teuthology.orchestra.run.smithi061.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.488 INFO:teuthology.orchestra.run.smithi061.stdout:Device: 6h/6d Inode: 378 Links: 1 2022-07-20T03:06:30.488 INFO:teuthology.orchestra.run.smithi061.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.489 INFO:teuthology.orchestra.run.smithi061.stdout:Access: 2022-07-20 03:03:20.147496830 +0000 2022-07-20T03:06:30.489 INFO:teuthology.orchestra.run.smithi061.stdout:Modify: 2022-07-20 03:03:19.763496835 +0000 2022-07-20T03:06:30.490 INFO:teuthology.orchestra.run.smithi061.stdout:Change: 2022-07-20 03:03:19.763496835 +0000 2022-07-20T03:06:30.490 INFO:teuthology.orchestra.run.smithi061.stdout: Birth: - 2022-07-20T03:06:30.493 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records in 2022-07-20T03:06:30.494 INFO:teuthology.orchestra.run.smithi061.stderr:1+0 records out 2022-07-20T03:06:30.494 INFO:teuthology.orchestra.run.smithi061.stderr:512 bytes copied, 0.000442366 s, 1.2 MB/s 2022-07-20T03:06:30.499 INFO:tasks.ceph:osd dev map: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:30.499 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:30.500 DEBUG:teuthology.orchestra.run.smithi117:> dd if=/scratch_devs of=/dev/stdout 2022-07-20T03:06:30.513 DEBUG:teuthology.misc:devs=['/dev/vg_nvme/lv_1', '/dev/vg_nvme/lv_2', '/dev/vg_nvme/lv_3', '/dev/vg_nvme/lv_4'] 2022-07-20T03:06:30.514 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_1 && sudo dd if=/dev/vg_nvme/lv_1 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_1 2022-07-20T03:06:30.524 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_1 -> ../dm-0 2022-07-20T03:06:30.525 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.525 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 6h/6d Inode: 360 Links: 1 2022-07-20T03:06:30.530 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.531 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-07-20 03:03:25.591404325 +0000 2022-07-20T03:06:30.531 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-07-20 03:03:25.123404323 +0000 2022-07-20T03:06:30.532 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-07-20 03:03:25.123404323 +0000 2022-07-20T03:06:30.545 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-07-20T03:06:30.551 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-07-20T03:06:30.551 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-07-20T03:06:30.552 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.00040514 s, 1.3 MB/s 2022-07-20T03:06:30.552 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_2 && sudo dd if=/dev/vg_nvme/lv_2 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_2 2022-07-20T03:06:30.610 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_2 -> ../dm-1 2022-07-20T03:06:30.610 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.611 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 6h/6d Inode: 367 Links: 1 2022-07-20T03:06:30.611 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.611 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-07-20 03:03:25.847404325 +0000 2022-07-20T03:06:30.612 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-07-20 03:03:25.843404325 +0000 2022-07-20T03:06:30.612 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-07-20 03:03:25.843404325 +0000 2022-07-20T03:06:30.612 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-07-20T03:06:30.613 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-07-20T03:06:30.613 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-07-20T03:06:30.613 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000380234 s, 1.3 MB/s 2022-07-20T03:06:30.619 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_3 && sudo dd if=/dev/vg_nvme/lv_3 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_3 2022-07-20T03:06:30.627 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_3 -> ../dm-2 2022-07-20T03:06:30.628 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.629 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 6h/6d Inode: 371 Links: 1 2022-07-20T03:06:30.629 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.630 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-07-20 03:03:26.447404327 +0000 2022-07-20T03:06:30.630 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-07-20 03:03:26.443404327 +0000 2022-07-20T03:06:30.631 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-07-20 03:03:26.443404327 +0000 2022-07-20T03:06:30.631 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-07-20T03:06:30.637 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-07-20T03:06:30.639 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-07-20T03:06:30.639 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000381083 s, 1.3 MB/s 2022-07-20T03:06:30.641 DEBUG:teuthology.orchestra.run.smithi117:> stat /dev/vg_nvme/lv_4 && sudo dd if=/dev/vg_nvme/lv_4 of=/dev/null count=1 && ! mount | grep -q /dev/vg_nvme/lv_4 2022-07-20T03:06:30.699 INFO:teuthology.orchestra.run.smithi117.stdout: File: /dev/vg_nvme/lv_4 -> ../dm-3 2022-07-20T03:06:30.700 INFO:teuthology.orchestra.run.smithi117.stdout: Size: 7 Blocks: 0 IO Block: 4096 symbolic link 2022-07-20T03:06:30.700 INFO:teuthology.orchestra.run.smithi117.stdout:Device: 6h/6d Inode: 377 Links: 1 2022-07-20T03:06:30.700 INFO:teuthology.orchestra.run.smithi117.stdout:Access: (0777/lrwxrwxrwx) Uid: ( 0/ root) Gid: ( 0/ root) 2022-07-20T03:06:30.700 INFO:teuthology.orchestra.run.smithi117.stdout:Access: 2022-07-20 03:03:27.367404329 +0000 2022-07-20T03:06:30.701 INFO:teuthology.orchestra.run.smithi117.stdout:Modify: 2022-07-20 03:03:27.035404328 +0000 2022-07-20T03:06:30.701 INFO:teuthology.orchestra.run.smithi117.stdout:Change: 2022-07-20 03:03:27.035404328 +0000 2022-07-20T03:06:30.702 INFO:teuthology.orchestra.run.smithi117.stdout: Birth: - 2022-07-20T03:06:30.703 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records in 2022-07-20T03:06:30.703 INFO:teuthology.orchestra.run.smithi117.stderr:1+0 records out 2022-07-20T03:06:30.703 INFO:teuthology.orchestra.run.smithi117.stderr:512 bytes copied, 0.000400674 s, 1.3 MB/s 2022-07-20T03:06:30.704 INFO:tasks.ceph:osd dev map: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:30.705 INFO:tasks.ceph:remote_to_roles_to_devs: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-20T03:06:30.705 INFO:tasks.ceph:Generating config... 2022-07-20T03:06:30.707 INFO:tasks.ceph:[client] client mount timeout = 600 2022-07-20T03:06:30.708 INFO:tasks.ceph:[client] debug client = 20 2022-07-20T03:06:30.708 INFO:tasks.ceph:[client] debug ms = 1 2022-07-20T03:06:30.709 INFO:tasks.ceph:[client] rados mon op timeout = 15m 2022-07-20T03:06:30.709 INFO:tasks.ceph:[client] rados osd op timeout = 15m 2022-07-20T03:06:30.710 INFO:tasks.ceph:[mds] debug mds = 20 2022-07-20T03:06:30.710 INFO:tasks.ceph:[mds] debug ms = 1 2022-07-20T03:06:30.710 INFO:tasks.ceph:[mds] mds bal frag = True 2022-07-20T03:06:30.710 INFO:tasks.ceph:[mds] mds bal fragment size max = 10000 2022-07-20T03:06:30.711 INFO:tasks.ceph:[mds] mds bal merge size = 5 2022-07-20T03:06:30.711 INFO:tasks.ceph:[mds] mds bal split bits = 3 2022-07-20T03:06:30.711 INFO:tasks.ceph:[mds] mds bal split size = 100 2022-07-20T03:06:30.712 INFO:tasks.ceph:[mds] mds debug frag = True 2022-07-20T03:06:30.712 INFO:tasks.ceph:[mds] mds debug scatterstat = True 2022-07-20T03:06:30.712 INFO:tasks.ceph:[mds] mds op complaint time = 180 2022-07-20T03:06:30.713 INFO:tasks.ceph:[mds] mds verify scatter = True 2022-07-20T03:06:30.713 INFO:tasks.ceph:[mds] osd op complaint time = 180 2022-07-20T03:06:30.713 INFO:tasks.ceph:[mds] rados mon op timeout = 15m 2022-07-20T03:06:30.714 INFO:tasks.ceph:[mds] rados osd op timeout = 15m 2022-07-20T03:06:30.714 INFO:tasks.ceph:[mgr] debug mgr = 20 2022-07-20T03:06:30.714 INFO:tasks.ceph:[mgr] debug ms = 1 2022-07-20T03:06:30.715 INFO:tasks.ceph:[mon] debug mon = 20 2022-07-20T03:06:30.725 INFO:tasks.ceph:[mon] debug ms = 1 2022-07-20T03:06:30.725 INFO:tasks.ceph:[mon] debug paxos = 20 2022-07-20T03:06:30.725 INFO:tasks.ceph:[mon] mon op complaint time = 120 2022-07-20T03:06:30.726 INFO:tasks.ceph:[osd] debug ms = 1 2022-07-20T03:06:30.726 INFO:tasks.ceph:[osd] debug osd = 20 2022-07-20T03:06:30.726 INFO:tasks.ceph:[osd] osd objectstore = filestore 2022-07-20T03:06:30.727 INFO:tasks.ceph:[osd] osd op complaint time = 180 2022-07-20T03:06:30.727 INFO:tasks.ceph:[osd] osd sloppy crc = True 2022-07-20T03:06:30.728 INFO:tasks.ceph:Setting up mon.a... 2022-07-20T03:06:30.728 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring /etc/ceph/ceph.keyring 2022-07-20T03:06:30.812 INFO:teuthology.orchestra.run.smithi061.stdout:creating /etc/ceph/ceph.keyring 2022-07-20T03:06:30.813 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=mon. /etc/ceph/ceph.keyring 2022-07-20T03:06:30.895 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T03:06:30.921 DEBUG:tasks.ceph:Ceph mon addresses: [('mon.a', '172.21.15.61'), ('mon.b', '172.21.15.117'), ('mon.c', '[v2:172.21.15.117:3301,v1:172.21.15.117:6790]')] 2022-07-20T03:06:30.925 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage monmaptool --create --clobber --enable-all-features --add a 172.21.15.61 --add b 172.21.15.117 --addv c '[v2:172.21.15.117:3301,v1:172.21.15.117:6790]' --print /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:30.999 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: monmap file /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:31.035 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: generated fsid 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.036 INFO:teuthology.orchestra.run.smithi061.stdout:epoch 0 2022-07-20T03:06:31.036 INFO:teuthology.orchestra.run.smithi061.stdout:fsid 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.036 INFO:teuthology.orchestra.run.smithi061.stdout:last_changed 2022-07-20T03:06:31.001078+0000 2022-07-20T03:06:31.037 INFO:teuthology.orchestra.run.smithi061.stdout:created 2022-07-20T03:06:31.001078+0000 2022-07-20T03:06:31.037 INFO:teuthology.orchestra.run.smithi061.stdout:min_mon_release 0 (unknown) 2022-07-20T03:06:31.037 INFO:teuthology.orchestra.run.smithi061.stdout:0: [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] mon.a 2022-07-20T03:06:31.038 INFO:teuthology.orchestra.run.smithi061.stdout:1: [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] mon.b 2022-07-20T03:06:31.038 INFO:teuthology.orchestra.run.smithi061.stdout:2: [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] mon.c 2022-07-20T03:06:31.038 INFO:teuthology.orchestra.run.smithi061.stdout:monmaptool: writing epoch 0 to /home/ubuntu/cephtest/ceph.monmap (3 monitors) 2022-07-20T03:06:31.040 INFO:tasks.ceph:Writing /etc/ceph/ceph.conf for FSID 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed... 2022-07-20T03:06:31.041 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:06:31.060 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:06:31.067 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:06:31.081 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:06:31.100 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:06:31.155 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2022-07-20T03:06:31.157 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2022-07-20T03:06:31.158 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:06:31.158 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2022-07-20T03:06:31.159 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.159 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2022-07-20T03:06:31.159 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.160 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2022-07-20T03:06:31.160 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.161 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2022-07-20T03:06:31.161 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2022-07-20T03:06:31.162 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.162 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2022-07-20T03:06:31.163 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2022-07-20T03:06:31.163 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.163 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:06:31.164 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2022-07-20T03:06:31.164 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.165 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:06:31.165 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:06:31.179 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.179 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2022-07-20T03:06:31.180 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.180 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2022-07-20T03:06:31.181 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2022-07-20T03:06:31.181 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:06:31.182 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:06:31.182 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2022-07-20T03:06:31.183 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:06:31.183 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2022-07-20T03:06:31.183 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:06:31.184 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:06:31.184 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.185 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:06:31.185 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.186 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2022-07-20T03:06:31.186 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.187 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2022-07-20T03:06:31.187 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.188 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2022-07-20T03:06:31.188 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2022-07-20T03:06:31.189 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2022-07-20T03:06:31.189 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:06:31.189 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.190 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.191 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2022-07-20T03:06:31.191 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2022-07-20T03:06:31.191 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.192 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2022-07-20T03:06:31.192 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2022-07-20T03:06:31.193 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.193 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2022-07-20T03:06:31.210 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2022-07-20T03:06:31.211 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.211 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2022-07-20T03:06:31.211 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2022-07-20T03:06:31.212 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2022-07-20T03:06:31.212 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.212 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2022-07-20T03:06:31.212 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2022-07-20T03:06:31.213 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.213 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:06:31.213 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.213 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2022-07-20T03:06:31.214 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.214 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2022-07-20T03:06:31.214 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.216 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:06:31.216 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2022-07-20T03:06:31.216 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.217 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2022-07-20T03:06:31.217 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2022-07-20T03:06:31.217 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:06:31.218 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2022-07-20T03:06:31.227 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = filestore 2022-07-20T03:06:31.227 INFO:teuthology.orchestra.run.smithi163.stdout: osd op complaint time = 180 2022-07-20T03:06:31.227 INFO:teuthology.orchestra.run.smithi163.stdout: osd sloppy crc = True 2022-07-20T03:06:31.228 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.228 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2022-07-20T03:06:31.228 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:06:31.228 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2022-07-20T03:06:31.229 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2022-07-20T03:06:31.229 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2022-07-20T03:06:31.229 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.229 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.230 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2022-07-20T03:06:31.230 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.230 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2022-07-20T03:06:31.231 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:06:31.231 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2022-07-20T03:06:31.231 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2022-07-20T03:06:31.232 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2022-07-20T03:06:31.232 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2022-07-20T03:06:31.232 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2022-07-20T03:06:31.232 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.233 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2022-07-20T03:06:31.233 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2022-07-20T03:06:31.233 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.233 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.234 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:06:31.235 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:06:31.235 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:06:31.237 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.238 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:06:31.238 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:06:31.238 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:06:31.239 INFO:teuthology.orchestra.run.smithi163.stdout: mon op complaint time = 120 2022-07-20T03:06:31.239 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:06:31.239 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2022-07-20T03:06:31.239 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2022-07-20T03:06:31.240 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2022-07-20T03:06:31.240 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2022-07-20T03:06:31.240 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:06:31.241 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:06:31.241 INFO:teuthology.orchestra.run.smithi163.stdout: client mount timeout = 600 2022-07-20T03:06:31.241 INFO:teuthology.orchestra.run.smithi163.stdout: debug client = 20 2022-07-20T03:06:31.241 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:06:31.242 INFO:teuthology.orchestra.run.smithi163.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.242 INFO:teuthology.orchestra.run.smithi163.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.242 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2022-07-20T03:06:31.242 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2022-07-20T03:06:31.243 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2022-07-20T03:06:31.243 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.a] 2022-07-20T03:06:31.243 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.b] 2022-07-20T03:06:31.243 INFO:teuthology.orchestra.run.smithi163.stdout:[mds] 2022-07-20T03:06:31.244 INFO:teuthology.orchestra.run.smithi163.stdout: debug mds = 20 2022-07-20T03:06:31.244 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:06:31.244 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal frag = True 2022-07-20T03:06:31.244 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal fragment size max = 10000 2022-07-20T03:06:31.244 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal merge size = 5 2022-07-20T03:06:31.245 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal split bits = 3 2022-07-20T03:06:31.283 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal split size = 100 2022-07-20T03:06:31.285 INFO:teuthology.orchestra.run.smithi163.stdout: mds debug frag = True 2022-07-20T03:06:31.286 INFO:teuthology.orchestra.run.smithi163.stdout: mds debug scatterstat = True 2022-07-20T03:06:31.286 INFO:teuthology.orchestra.run.smithi163.stdout: mds op complaint time = 180 2022-07-20T03:06:31.287 INFO:teuthology.orchestra.run.smithi163.stdout: mds verify scatter = True 2022-07-20T03:06:31.287 INFO:teuthology.orchestra.run.smithi163.stdout: osd op complaint time = 180 2022-07-20T03:06:31.290 INFO:teuthology.orchestra.run.smithi163.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.291 INFO:teuthology.orchestra.run.smithi163.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.291 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2022-07-20T03:06:31.292 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2022-07-20T03:06:31.292 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:06:31.293 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2022-07-20T03:06:31.293 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.294 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2022-07-20T03:06:31.295 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.296 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2022-07-20T03:06:31.296 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.297 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2022-07-20T03:06:31.297 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2022-07-20T03:06:31.298 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.298 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2022-07-20T03:06:31.299 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2022-07-20T03:06:31.331 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.333 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:06:31.333 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2022-07-20T03:06:31.333 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.334 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:06:31.334 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:06:31.336 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.337 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2022-07-20T03:06:31.337 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.338 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2022-07-20T03:06:31.338 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2022-07-20T03:06:31.338 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:06:31.339 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:06:31.339 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2022-07-20T03:06:31.340 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:06:31.350 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2022-07-20T03:06:31.351 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:06:31.351 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:06:31.351 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.351 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:06:31.352 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.352 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2022-07-20T03:06:31.352 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.352 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2022-07-20T03:06:31.353 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.353 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2022-07-20T03:06:31.353 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2022-07-20T03:06:31.353 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2022-07-20T03:06:31.357 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:06:31.358 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.358 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.359 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2022-07-20T03:06:31.359 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2022-07-20T03:06:31.360 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.360 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2022-07-20T03:06:31.361 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2022-07-20T03:06:31.361 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.361 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2022-07-20T03:06:31.363 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2022-07-20T03:06:31.363 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.364 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2022-07-20T03:06:31.364 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2022-07-20T03:06:31.365 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2022-07-20T03:06:31.365 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2022-07-20T03:06:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2022-07-20T03:06:31.366 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:06:31.367 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2022-07-20T03:06:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.368 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2022-07-20T03:06:31.369 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.369 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:06:31.370 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2022-07-20T03:06:31.371 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.371 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2022-07-20T03:06:31.372 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2022-07-20T03:06:31.372 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:06:31.373 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2022-07-20T03:06:31.373 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = filestore 2022-07-20T03:06:31.374 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2022-07-20T03:06:31.374 INFO:teuthology.orchestra.run.smithi124.stdout: osd sloppy crc = True 2022-07-20T03:06:31.375 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.375 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2022-07-20T03:06:31.376 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:06:31.376 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2022-07-20T03:06:31.416 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2022-07-20T03:06:31.416 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2022-07-20T03:06:31.417 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.417 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.418 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2022-07-20T03:06:31.419 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.419 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2022-07-20T03:06:31.420 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:06:31.421 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2022-07-20T03:06:31.421 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2022-07-20T03:06:31.422 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2022-07-20T03:06:31.422 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2022-07-20T03:06:31.423 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2022-07-20T03:06:31.423 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.424 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2022-07-20T03:06:31.425 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2022-07-20T03:06:31.426 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.426 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.428 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:06:31.428 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:06:31.429 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:06:31.429 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.430 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:06:31.430 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:06:31.431 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:06:31.432 INFO:teuthology.orchestra.run.smithi124.stdout: mon op complaint time = 120 2022-07-20T03:06:31.433 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:06:31.434 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2022-07-20T03:06:31.434 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2022-07-20T03:06:31.435 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2022-07-20T03:06:31.435 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2022-07-20T03:06:31.436 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:06:31.436 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:06:31.436 INFO:teuthology.orchestra.run.smithi124.stdout: client mount timeout = 600 2022-07-20T03:06:31.436 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2022-07-20T03:06:31.437 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:06:31.437 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.437 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.437 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2022-07-20T03:06:31.438 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.b] 2022-07-20T03:06:31.438 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.c] 2022-07-20T03:06:31.438 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2022-07-20T03:06:31.438 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2022-07-20T03:06:31.439 INFO:teuthology.orchestra.run.smithi124.stdout:[mds] 2022-07-20T03:06:31.439 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds = 20 2022-07-20T03:06:31.439 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:06:31.439 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal frag = True 2022-07-20T03:06:31.440 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal fragment size max = 10000 2022-07-20T03:06:31.440 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal merge size = 5 2022-07-20T03:06:31.440 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal split bits = 3 2022-07-20T03:06:31.440 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal split size = 100 2022-07-20T03:06:31.441 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug frag = True 2022-07-20T03:06:31.441 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug scatterstat = True 2022-07-20T03:06:31.441 INFO:teuthology.orchestra.run.smithi124.stdout: mds op complaint time = 180 2022-07-20T03:06:31.441 INFO:teuthology.orchestra.run.smithi124.stdout: mds verify scatter = True 2022-07-20T03:06:31.442 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2022-07-20T03:06:31.442 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.443 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.443 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2022-07-20T03:06:31.443 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2022-07-20T03:06:31.443 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:06:31.444 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2022-07-20T03:06:31.444 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.444 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2022-07-20T03:06:31.444 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.445 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2022-07-20T03:06:31.445 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.445 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2022-07-20T03:06:31.445 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2022-07-20T03:06:31.446 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.446 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2022-07-20T03:06:31.446 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2022-07-20T03:06:31.446 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.447 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:06:31.447 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2022-07-20T03:06:31.447 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.447 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:06:31.447 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:06:31.448 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.448 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2022-07-20T03:06:31.448 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.449 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2022-07-20T03:06:31.449 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2022-07-20T03:06:31.449 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:06:31.449 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:06:31.450 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2022-07-20T03:06:31.450 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:06:31.450 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2022-07-20T03:06:31.450 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:06:31.451 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:06:31.451 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.451 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:06:31.451 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.452 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2022-07-20T03:06:31.452 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.452 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2022-07-20T03:06:31.452 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.453 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2022-07-20T03:06:31.453 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2022-07-20T03:06:31.453 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2022-07-20T03:06:31.453 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:06:31.453 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.454 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.454 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2022-07-20T03:06:31.455 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2022-07-20T03:06:31.455 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.455 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2022-07-20T03:06:31.455 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2022-07-20T03:06:31.456 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.456 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2022-07-20T03:06:31.456 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2022-07-20T03:06:31.456 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.457 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2022-07-20T03:06:31.457 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2022-07-20T03:06:31.457 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2022-07-20T03:06:31.457 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.458 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2022-07-20T03:06:31.458 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2022-07-20T03:06:31.458 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.458 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:06:31.459 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.459 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2022-07-20T03:06:31.459 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.459 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2022-07-20T03:06:31.459 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.460 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:06:31.460 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2022-07-20T03:06:31.460 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.460 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2022-07-20T03:06:31.461 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2022-07-20T03:06:31.461 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:06:31.461 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2022-07-20T03:06:31.461 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = filestore 2022-07-20T03:06:31.478 INFO:teuthology.orchestra.run.smithi117.stdout: osd op complaint time = 180 2022-07-20T03:06:31.482 INFO:teuthology.orchestra.run.smithi117.stdout: osd sloppy crc = True 2022-07-20T03:06:31.483 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.483 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2022-07-20T03:06:31.484 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:06:31.484 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2022-07-20T03:06:31.485 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2022-07-20T03:06:31.485 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2022-07-20T03:06:31.486 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.493 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.493 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2022-07-20T03:06:31.499 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.499 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2022-07-20T03:06:31.500 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:06:31.500 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2022-07-20T03:06:31.500 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2022-07-20T03:06:31.500 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2022-07-20T03:06:31.501 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2022-07-20T03:06:31.501 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2022-07-20T03:06:31.501 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.501 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2022-07-20T03:06:31.502 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2022-07-20T03:06:31.502 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.504 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.504 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:06:31.505 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:06:31.505 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:06:31.505 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.505 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:06:31.505 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:06:31.506 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:06:31.506 INFO:teuthology.orchestra.run.smithi117.stdout: mon op complaint time = 120 2022-07-20T03:06:31.506 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:06:31.506 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2022-07-20T03:06:31.513 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2022-07-20T03:06:31.514 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2022-07-20T03:06:31.514 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2022-07-20T03:06:31.514 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:06:31.515 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:06:31.515 INFO:teuthology.orchestra.run.smithi117.stdout: client mount timeout = 600 2022-07-20T03:06:31.515 INFO:teuthology.orchestra.run.smithi117.stdout: debug client = 20 2022-07-20T03:06:31.515 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:06:31.515 INFO:teuthology.orchestra.run.smithi117.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.516 INFO:teuthology.orchestra.run.smithi117.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.516 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2022-07-20T03:06:31.516 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2022-07-20T03:06:31.516 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2022-07-20T03:06:31.517 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a] 2022-07-20T03:06:31.517 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b] 2022-07-20T03:06:31.517 INFO:teuthology.orchestra.run.smithi117.stdout:[mds] 2022-07-20T03:06:31.538 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds = 20 2022-07-20T03:06:31.540 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:06:31.541 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal frag = True 2022-07-20T03:06:31.541 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal fragment size max = 10000 2022-07-20T03:06:31.541 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal merge size = 5 2022-07-20T03:06:31.549 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal split bits = 3 2022-07-20T03:06:31.549 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal split size = 100 2022-07-20T03:06:31.549 INFO:teuthology.orchestra.run.smithi117.stdout: mds debug frag = True 2022-07-20T03:06:31.549 INFO:teuthology.orchestra.run.smithi117.stdout: mds debug scatterstat = True 2022-07-20T03:06:31.550 INFO:teuthology.orchestra.run.smithi117.stdout: mds op complaint time = 180 2022-07-20T03:06:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: mds verify scatter = True 2022-07-20T03:06:31.566 INFO:teuthology.orchestra.run.smithi117.stdout: osd op complaint time = 180 2022-07-20T03:06:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.567 INFO:teuthology.orchestra.run.smithi117.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.567 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2022-07-20T03:06:31.568 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2022-07-20T03:06:31.568 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:06:31.568 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2022-07-20T03:06:31.582 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.583 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2022-07-20T03:06:31.583 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.583 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2022-07-20T03:06:31.589 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.590 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2022-07-20T03:06:31.590 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2022-07-20T03:06:31.591 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.591 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2022-07-20T03:06:31.592 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2022-07-20T03:06:31.593 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.593 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:06:31.594 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2022-07-20T03:06:31.594 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:06:31.595 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:06:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.596 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2022-07-20T03:06:31.597 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.597 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2022-07-20T03:06:31.597 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2022-07-20T03:06:31.614 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:06:31.614 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:06:31.615 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2022-07-20T03:06:31.615 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:06:31.616 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2022-07-20T03:06:31.616 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:06:31.617 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:06:31.617 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.617 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:06:31.618 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.618 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2022-07-20T03:06:31.619 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.619 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2022-07-20T03:06:31.620 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.620 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2022-07-20T03:06:31.620 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2022-07-20T03:06:31.621 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2022-07-20T03:06:31.622 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:06:31.622 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.623 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.623 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2022-07-20T03:06:31.624 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2022-07-20T03:06:31.624 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.625 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2022-07-20T03:06:31.625 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2022-07-20T03:06:31.625 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.626 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2022-07-20T03:06:31.626 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2022-07-20T03:06:31.627 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.628 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2022-07-20T03:06:31.628 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2022-07-20T03:06:31.628 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2022-07-20T03:06:31.629 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.629 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2022-07-20T03:06:31.630 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2022-07-20T03:06:31.630 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.630 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:06:31.631 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.631 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2022-07-20T03:06:31.632 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.632 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2022-07-20T03:06:31.632 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.633 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:06:31.633 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2022-07-20T03:06:31.634 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.634 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2022-07-20T03:06:31.635 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2022-07-20T03:06:31.635 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:06:31.636 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 20 2022-07-20T03:06:31.637 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = filestore 2022-07-20T03:06:31.637 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-20T03:06:31.637 INFO:teuthology.orchestra.run.smithi061.stdout: osd sloppy crc = True 2022-07-20T03:06:31.638 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.638 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2022-07-20T03:06:31.639 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:06:31.639 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2022-07-20T03:06:31.640 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-20T03:06:31.640 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-20T03:06:31.640 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.641 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.641 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2022-07-20T03:06:31.642 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.643 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2022-07-20T03:06:31.643 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:06:31.644 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-20T03:06:31.644 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2022-07-20T03:06:31.645 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-20T03:06:31.645 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2022-07-20T03:06:31.645 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2022-07-20T03:06:31.646 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.646 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2022-07-20T03:06:31.647 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2022-07-20T03:06:31.647 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.647 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.648 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:06:31.648 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:06:31.649 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:06:31.650 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.650 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:06:31.650 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:06:31.651 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:06:31.651 INFO:teuthology.orchestra.run.smithi061.stdout: mon op complaint time = 120 2022-07-20T03:06:31.652 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:06:31.652 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2022-07-20T03:06:31.653 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2022-07-20T03:06:31.653 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2022-07-20T03:06:31.658 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2022-07-20T03:06:31.658 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:06:31.659 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:06:31.659 INFO:teuthology.orchestra.run.smithi061.stdout: client mount timeout = 600 2022-07-20T03:06:31.660 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 20 2022-07-20T03:06:31.660 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:06:31.661 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.661 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.661 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2022-07-20T03:06:31.662 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.b] 2022-07-20T03:06:31.663 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.c] 2022-07-20T03:06:31.664 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-20T03:06:31.664 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-20T03:06:31.665 INFO:teuthology.orchestra.run.smithi061.stdout:[mds] 2022-07-20T03:06:31.665 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 20 2022-07-20T03:06:31.666 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:06:31.669 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal frag = True 2022-07-20T03:06:31.669 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal fragment size max = 10000 2022-07-20T03:06:31.669 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal merge size = 5 2022-07-20T03:06:31.670 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split bits = 3 2022-07-20T03:06:31.671 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split size = 100 2022-07-20T03:06:31.672 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug frag = True 2022-07-20T03:06:31.672 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug scatterstat = True 2022-07-20T03:06:31.673 INFO:teuthology.orchestra.run.smithi061.stdout: mds op complaint time = 180 2022-07-20T03:06:31.673 INFO:teuthology.orchestra.run.smithi061.stdout: mds verify scatter = True 2022-07-20T03:06:31.673 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-20T03:06:31.673 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.674 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.682 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2022-07-20T03:06:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2022-07-20T03:06:31.682 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:06:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2022-07-20T03:06:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2022-07-20T03:06:31.683 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2022-07-20T03:06:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2022-07-20T03:06:31.684 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2022-07-20T03:06:31.685 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.687 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2022-07-20T03:06:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2022-07-20T03:06:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:06:31.688 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2022-07-20T03:06:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.689 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:06:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:06:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2022-07-20T03:06:31.690 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2022-07-20T03:06:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2022-07-20T03:06:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:06:31.691 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:06:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2022-07-20T03:06:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:06:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2022-07-20T03:06:31.692 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:06:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:06:31.693 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.697 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:06:31.710 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.710 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2022-07-20T03:06:31.711 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.711 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2022-07-20T03:06:31.711 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.711 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2022-07-20T03:06:31.712 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2022-07-20T03:06:31.712 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2022-07-20T03:06:31.712 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:06:31.712 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:06:31.713 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.725 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2022-07-20T03:06:31.725 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2022-07-20T03:06:31.726 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.726 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2022-07-20T03:06:31.726 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2022-07-20T03:06:31.727 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.728 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2022-07-20T03:06:31.728 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2022-07-20T03:06:31.735 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.736 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2022-07-20T03:06:31.736 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2022-07-20T03:06:31.737 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2022-07-20T03:06:31.737 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.737 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2022-07-20T03:06:31.754 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2022-07-20T03:06:31.759 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.759 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:06:31.776 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.777 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2022-07-20T03:06:31.777 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.785 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2022-07-20T03:06:31.785 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.785 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:06:31.786 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2022-07-20T03:06:31.794 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.796 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2022-07-20T03:06:31.798 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2022-07-20T03:06:31.798 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:06:31.799 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2022-07-20T03:06:31.799 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = filestore 2022-07-20T03:06:31.800 INFO:teuthology.orchestra.run.smithi179.stdout: osd op complaint time = 180 2022-07-20T03:06:31.801 INFO:teuthology.orchestra.run.smithi179.stdout: osd sloppy crc = True 2022-07-20T03:06:31.801 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.802 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2022-07-20T03:06:31.802 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:06:31.803 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2022-07-20T03:06:31.803 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2022-07-20T03:06:31.804 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2022-07-20T03:06:31.804 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.805 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.808 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2022-07-20T03:06:31.809 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.810 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2022-07-20T03:06:31.811 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:06:31.811 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2022-07-20T03:06:31.828 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2022-07-20T03:06:31.829 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2022-07-20T03:06:31.829 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2022-07-20T03:06:31.830 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2022-07-20T03:06:31.830 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:06:31.831 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2022-07-20T03:06:31.831 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2022-07-20T03:06:31.832 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:06:31.832 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.833 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:06:31.833 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:06:31.834 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:06:31.881 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.881 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:06:31.882 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:06:31.882 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:06:31.888 INFO:teuthology.orchestra.run.smithi179.stdout: mon op complaint time = 120 2022-07-20T03:06:31.888 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:06:31.889 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2022-07-20T03:06:31.889 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2022-07-20T03:06:31.889 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2022-07-20T03:06:31.912 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2022-07-20T03:06:31.923 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:06:31.923 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:06:31.923 INFO:teuthology.orchestra.run.smithi179.stdout: client mount timeout = 600 2022-07-20T03:06:31.937 INFO:teuthology.orchestra.run.smithi179.stdout: debug client = 20 2022-07-20T03:06:31.938 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:06:31.938 INFO:teuthology.orchestra.run.smithi179.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.938 INFO:teuthology.orchestra.run.smithi179.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.939 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2022-07-20T03:06:31.939 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2022-07-20T03:06:31.940 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2022-07-20T03:06:31.940 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.a] 2022-07-20T03:06:31.941 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.b] 2022-07-20T03:06:31.942 INFO:teuthology.orchestra.run.smithi179.stdout:[mds] 2022-07-20T03:06:31.943 INFO:teuthology.orchestra.run.smithi179.stdout: debug mds = 20 2022-07-20T03:06:31.943 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:06:31.944 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal frag = True 2022-07-20T03:06:31.945 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal fragment size max = 10000 2022-07-20T03:06:31.945 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal merge size = 5 2022-07-20T03:06:31.946 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal split bits = 3 2022-07-20T03:06:31.953 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal split size = 100 2022-07-20T03:06:31.953 INFO:teuthology.orchestra.run.smithi179.stdout: mds debug frag = True 2022-07-20T03:06:31.954 INFO:teuthology.orchestra.run.smithi179.stdout: mds debug scatterstat = True 2022-07-20T03:06:31.962 INFO:teuthology.orchestra.run.smithi179.stdout: mds op complaint time = 180 2022-07-20T03:06:31.963 INFO:teuthology.orchestra.run.smithi179.stdout: mds verify scatter = True 2022-07-20T03:06:31.963 INFO:teuthology.orchestra.run.smithi179.stdout: osd op complaint time = 180 2022-07-20T03:06:31.963 INFO:teuthology.orchestra.run.smithi179.stdout: rados mon op timeout = 15m 2022-07-20T03:06:31.963 INFO:teuthology.orchestra.run.smithi179.stdout: rados osd op timeout = 15m 2022-07-20T03:06:31.964 INFO:tasks.ceph:Creating admin key on mon.a... 2022-07-20T03:06:31.964 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --gen-key --name=client.admin --cap mon 'allow *' --cap osd 'allow *' --cap mds 'allow *' --cap mgr 'allow *' /etc/ceph/ceph.keyring 2022-07-20T03:06:32.039 INFO:tasks.ceph:Copying monmap to all nodes... 2022-07-20T03:06:32.078 DEBUG:teuthology.orchestra.remote:smithi061:/etc/ceph/ceph.keyring is 206B 2022-07-20T03:06:32.236 DEBUG:teuthology.orchestra.remote:smithi061:/home/ubuntu/cephtest/ceph.monmap is 379B 2022-07-20T03:06:32.253 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T03:06:32.254 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:32.254 DEBUG:teuthology.orchestra.run.smithi061:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T03:06:32.255 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T03:06:32.284 DEBUG:teuthology.orchestra.run.smithi061:> set -ex 2022-07-20T03:06:32.285 DEBUG:teuthology.orchestra.run.smithi061:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:32.346 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T03:06:32.347 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:32.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T03:06:32.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T03:06:32.414 DEBUG:teuthology.orchestra.run.smithi117:> set -ex 2022-07-20T03:06:32.414 DEBUG:teuthology.orchestra.run.smithi117:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:32.480 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi124.front.sepia.ceph.com 2022-07-20T03:06:32.481 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:06:32.481 DEBUG:teuthology.orchestra.run.smithi124:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T03:06:32.482 DEBUG:teuthology.orchestra.run.smithi124:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T03:06:32.509 DEBUG:teuthology.orchestra.run.smithi124:> set -ex 2022-07-20T03:06:32.509 DEBUG:teuthology.orchestra.run.smithi124:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:32.558 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi163.front.sepia.ceph.com 2022-07-20T03:06:32.559 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:06:32.559 DEBUG:teuthology.orchestra.run.smithi163:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T03:06:32.560 DEBUG:teuthology.orchestra.run.smithi163:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T03:06:32.587 DEBUG:teuthology.orchestra.run.smithi163:> set -ex 2022-07-20T03:06:32.588 DEBUG:teuthology.orchestra.run.smithi163:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:32.595 INFO:tasks.ceph:Sending monmap to node ubuntu@smithi179.front.sepia.ceph.com 2022-07-20T03:06:32.596 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:06:32.611 DEBUG:teuthology.orchestra.run.smithi179:> sudo dd of=/etc/ceph/ceph.keyring 2022-07-20T03:06:32.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo chmod 0644 /etc/ceph/ceph.keyring 2022-07-20T03:06:32.647 DEBUG:teuthology.orchestra.run.smithi179:> set -ex 2022-07-20T03:06:32.648 DEBUG:teuthology.orchestra.run.smithi179:> dd of=/home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:32.722 INFO:tasks.ceph:Setting up mon nodes... 2022-07-20T03:06:32.723 INFO:tasks.ceph:Setting up mgr nodes... 2022-07-20T03:06:32.724 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mgr/ceph-y && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.y /var/lib/ceph/mgr/ceph-y/keyring 2022-07-20T03:06:32.803 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mgr/ceph-y/keyring 2022-07-20T03:06:32.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mgr/ceph-x && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mgr.x /var/lib/ceph/mgr/ceph-x/keyring 2022-07-20T03:06:32.862 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mgr/ceph-x/keyring 2022-07-20T03:06:32.866 INFO:tasks.ceph:Setting up mds nodes... 2022-07-20T03:06:32.867 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mds/ceph-a && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.a /var/lib/ceph/mds/ceph-a/keyring 2022-07-20T03:06:32.919 INFO:teuthology.orchestra.run.smithi061.stdout:creating /var/lib/ceph/mds/ceph-a/keyring 2022-07-20T03:06:32.921 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-a 2022-07-20T03:06:32.989 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mds/ceph-b && sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=mds.b /var/lib/ceph/mds/ceph-b/keyring 2022-07-20T03:06:33.051 INFO:teuthology.orchestra.run.smithi117.stdout:creating /var/lib/ceph/mds/ceph-b/keyring 2022-07-20T03:06:33.057 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mds/ceph-b 2022-07-20T03:06:33.095 INFO:tasks.ceph_client:Setting up client nodes... 2022-07-20T03:06:33.104 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.0 /etc/ceph/ceph.client.0.keyring && sudo chmod 0644 /etc/ceph/ceph.client.0.keyring 2022-07-20T03:06:33.162 INFO:teuthology.orchestra.run.smithi124.stdout:creating /etc/ceph/ceph.client.0.keyring 2022-07-20T03:06:33.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.1 /etc/ceph/ceph.client.1.keyring && sudo chmod 0644 /etc/ceph/ceph.client.1.keyring 2022-07-20T03:06:33.242 INFO:teuthology.orchestra.run.smithi163.stdout:creating /etc/ceph/ceph.client.1.keyring 2022-07-20T03:06:33.254 DEBUG:teuthology.orchestra.run.smithi179:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool --create-keyring --gen-key --name=client.2 /etc/ceph/ceph.client.2.keyring && sudo chmod 0644 /etc/ceph/ceph.client.2.keyring 2022-07-20T03:06:33.299 INFO:teuthology.orchestra.run.smithi179.stdout:creating /etc/ceph/ceph.client.2.keyring 2022-07-20T03:06:33.316 INFO:tasks.ceph:Running mkfs on osd nodes... 2022-07-20T03:06:33.317 INFO:tasks.ceph:ctx.disk_config.remote_to_roles_to_dev: {Remote(name='ubuntu@smithi061.front.sepia.ceph.com'): {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'}, Remote(name='ubuntu@smithi117.front.sepia.ceph.com'): {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'}} 2022-07-20T03:06:33.318 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-0 2022-07-20T03:06:33.350 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:33.350 INFO:tasks.ceph:role: osd.0 2022-07-20T03:06:33.351 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T03:06:33.360 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-20T03:06:33.925 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:33.926 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:33.927 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:33.927 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-20T03:06:33.928 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:33.928 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:33.929 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:33.929 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:33.929 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:33.930 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:33.937 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-20T03:06:33.938 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-0 2022-07-20T03:06:34.044 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-0 2022-07-20T03:06:34.064 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:34.065 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:34.065 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-1 2022-07-20T03:06:34.136 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:34.136 INFO:tasks.ceph:role: osd.1 2022-07-20T03:06:34.137 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T03:06:34.142 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-20T03:06:34.652 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:34.653 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:34.658 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:34.659 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-20T03:06:34.659 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:34.660 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:34.661 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:34.661 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:34.661 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:34.661 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:34.662 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-20T03:06:34.662 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-1 2022-07-20T03:06:34.695 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-1 2022-07-20T03:06:34.755 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:34.756 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:34.756 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-2 2022-07-20T03:06:34.796 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:34.798 INFO:tasks.ceph:role: osd.2 2022-07-20T03:06:34.799 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T03:06:34.800 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-20T03:06:35.364 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:35.366 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:35.367 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:35.367 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-20T03:06:35.368 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:35.368 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:35.369 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:35.369 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:35.371 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:35.371 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:35.373 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-20T03:06:35.383 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-2 2022-07-20T03:06:35.463 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-2 2022-07-20T03:06:35.480 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:35.483 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:35.484 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/osd/ceph-3 2022-07-20T03:06:35.508 INFO:tasks.ceph:roles_to_devs: {'osd.0': '/dev/vg_nvme/lv_1', 'osd.1': '/dev/vg_nvme/lv_2', 'osd.2': '/dev/vg_nvme/lv_3', 'osd.3': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:35.509 INFO:tasks.ceph:role: osd.3 2022-07-20T03:06:35.509 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T03:06:35.510 DEBUG:teuthology.orchestra.run.smithi061:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-20T03:06:36.036 INFO:teuthology.orchestra.run.smithi061.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:36.037 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:36.038 INFO:teuthology.orchestra.run.smithi061.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:36.038 INFO:teuthology.orchestra.run.smithi061.stdout: = reflink=1 2022-07-20T03:06:36.039 INFO:teuthology.orchestra.run.smithi061.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:36.039 INFO:teuthology.orchestra.run.smithi061.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:36.040 INFO:teuthology.orchestra.run.smithi061.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:36.040 INFO:teuthology.orchestra.run.smithi061.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:36.041 INFO:teuthology.orchestra.run.smithi061.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:36.041 INFO:teuthology.orchestra.run.smithi061.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:36.042 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi061.front.sepia.ceph.com -o noatime 2022-07-20T03:06:36.043 DEBUG:teuthology.orchestra.run.smithi061:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-3 2022-07-20T03:06:36.151 DEBUG:teuthology.orchestra.run.smithi061:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-3 2022-07-20T03:06:36.163 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:36.170 INFO:teuthology.orchestra.run.smithi061.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:36.171 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 0 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:36.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.243+0000 7f9e15728d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-0/keyring: can't open /var/lib/ceph/osd/ceph-0/keyring: (2) No such file or directory 2022-07-20T03:06:36.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.243+0000 7f9e15728d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-0/keyring 2022-07-20T03:06:36.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.279+0000 7f9e15728d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:36.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.279+0000 7f9e15728d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:36.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.547+0000 7f9e15728d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:36.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.551+0000 7f9e15728d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:36.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.551+0000 7f9e15728d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:36.592 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-0 2022-07-20T03:06:36.610 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 1 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:36.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.667+0000 7f6578ee2d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-1/keyring: can't open /var/lib/ceph/osd/ceph-1/keyring: (2) No such file or directory 2022-07-20T03:06:36.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.667+0000 7f6578ee2d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-1/keyring 2022-07-20T03:06:36.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.707+0000 7f6578ee2d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:36.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.707+0000 7f6578ee2d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:36.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.967+0000 7f6578ee2d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:36.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.967+0000 7f6578ee2d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:36.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:36.967+0000 7f6578ee2d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:36.987 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-1 2022-07-20T03:06:37.045 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 2 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:37.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.079+0000 7fe466296d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-2/keyring: can't open /var/lib/ceph/osd/ceph-2/keyring: (2) No such file or directory 2022-07-20T03:06:37.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.083+0000 7fe466296d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-2/keyring 2022-07-20T03:06:37.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.091+0000 7fe466296d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:37.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.091+0000 7fe466296d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:37.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.227+0000 7fe466296d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:37.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.227+0000 7fe466296d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:37.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.227+0000 7fe466296d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:37.256 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-2 2022-07-20T03:06:37.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 3 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:37.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.379+0000 7fdfba3c6d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-3/keyring: can't open /var/lib/ceph/osd/ceph-3/keyring: (2) No such file or directory 2022-07-20T03:06:37.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.379+0000 7fdfba3c6d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-3/keyring 2022-07-20T03:06:37.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.391+0000 7fdfba3c6d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:37.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.391+0000 7fdfba3c6d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:37.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.531+0000 7fdfba3c6d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:37.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.535+0000 7fdfba3c6d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:37.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:37.535+0000 7fdfba3c6d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:37.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-3 2022-07-20T03:06:37.639 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-4 2022-07-20T03:06:37.656 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:37.682 INFO:tasks.ceph:role: osd.4 2022-07-20T03:06:37.696 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T03:06:37.697 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_1 2022-07-20T03:06:38.222 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_1 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:38.223 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:38.224 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:38.224 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 2022-07-20T03:06:38.226 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:38.226 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:38.229 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:38.229 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:38.230 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:38.230 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:38.232 INFO:tasks.ceph:mount /dev/vg_nvme/lv_1 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2022-07-20T03:06:38.233 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_1 /var/lib/ceph/osd/ceph-4 2022-07-20T03:06:38.356 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-4 2022-07-20T03:06:38.415 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:38.417 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:38.418 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-5 2022-07-20T03:06:38.507 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:38.516 INFO:tasks.ceph:role: osd.5 2022-07-20T03:06:38.517 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T03:06:38.518 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_2 2022-07-20T03:06:38.999 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_2 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:38.999 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:39.000 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:39.000 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 2022-07-20T03:06:39.001 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:39.016 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:39.017 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:39.017 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:39.017 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:39.017 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:39.025 INFO:tasks.ceph:mount /dev/vg_nvme/lv_2 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2022-07-20T03:06:39.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_2 /var/lib/ceph/osd/ceph-5 2022-07-20T03:06:39.088 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-5 2022-07-20T03:06:39.123 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:39.123 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:39.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-6 2022-07-20T03:06:39.189 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:39.190 INFO:tasks.ceph:role: osd.6 2022-07-20T03:06:39.190 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T03:06:39.191 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_3 2022-07-20T03:06:39.692 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_3 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:39.696 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:39.697 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:39.697 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 2022-07-20T03:06:39.703 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:39.704 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:39.704 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:39.705 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:39.706 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:39.707 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:39.709 INFO:tasks.ceph:mount /dev/vg_nvme/lv_3 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2022-07-20T03:06:39.710 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_3 /var/lib/ceph/osd/ceph-6 2022-07-20T03:06:39.788 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-6 2022-07-20T03:06:39.827 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:39.828 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:39.829 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/osd/ceph-7 2022-07-20T03:06:39.899 INFO:tasks.ceph:roles_to_devs: {'osd.4': '/dev/vg_nvme/lv_1', 'osd.5': '/dev/vg_nvme/lv_2', 'osd.6': '/dev/vg_nvme/lv_3', 'osd.7': '/dev/vg_nvme/lv_4'} 2022-07-20T03:06:39.900 INFO:tasks.ceph:role: osd.7 2022-07-20T03:06:39.900 INFO:tasks.ceph:['mkfs.xfs', '-f', '-i', 'size=2048'] on /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T03:06:39.900 DEBUG:teuthology.orchestra.run.smithi117:> yes | sudo mkfs.xfs -f -i size=2048 /dev/vg_nvme/lv_4 2022-07-20T03:06:40.408 INFO:teuthology.orchestra.run.smithi117.stdout:meta-data=/dev/vg_nvme/lv_4 isize=2048 agcount=4, agsize=5859072 blks 2022-07-20T03:06:40.409 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 attr=2, projid32bit=1 2022-07-20T03:06:40.418 INFO:teuthology.orchestra.run.smithi117.stdout: = crc=1 finobt=1, sparse=1, rmapbt=0 2022-07-20T03:06:40.418 INFO:teuthology.orchestra.run.smithi117.stdout: = reflink=1 2022-07-20T03:06:40.418 INFO:teuthology.orchestra.run.smithi117.stdout:data = bsize=4096 blocks=23436288, imaxpct=25 2022-07-20T03:06:40.419 INFO:teuthology.orchestra.run.smithi117.stdout: = sunit=0 swidth=0 blks 2022-07-20T03:06:40.419 INFO:teuthology.orchestra.run.smithi117.stdout:naming =version 2 bsize=4096 ascii-ci=0, ftype=1 2022-07-20T03:06:40.420 INFO:teuthology.orchestra.run.smithi117.stdout:log =internal log bsize=4096 blocks=11443, version=2 2022-07-20T03:06:40.420 INFO:teuthology.orchestra.run.smithi117.stdout: = sectsz=512 sunit=0 blks, lazy-count=1 2022-07-20T03:06:40.421 INFO:teuthology.orchestra.run.smithi117.stdout:realtime =none extsz=4096 blocks=0, rtextents=0 2022-07-20T03:06:40.422 INFO:tasks.ceph:mount /dev/vg_nvme/lv_4 on ubuntu@smithi117.front.sepia.ceph.com -o noatime 2022-07-20T03:06:40.424 DEBUG:teuthology.orchestra.run.smithi117:> sudo mount -t xfs -o noatime /dev/vg_nvme/lv_4 /var/lib/ceph/osd/ceph-7 2022-07-20T03:06:40.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo /sbin/restorecon /var/lib/ceph/osd/ceph-7 2022-07-20T03:06:40.567 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:06:40.568 INFO:teuthology.orchestra.run.smithi117.stderr:sudo: /sbin/restorecon: command not found 2022-07-20T03:06:40.569 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 4 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:40.838 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:40.829+0000 7eff86754d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-4/keyring: can't open /var/lib/ceph/osd/ceph-4/keyring: (2) No such file or directory 2022-07-20T03:06:40.839 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:40.829+0000 7eff86754d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-4/keyring 2022-07-20T03:06:40.865 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:40.861+0000 7eff86754d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:40.866 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:40.861+0000 7eff86754d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:41.137 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.125+0000 7eff86754d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:41.137 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.129+0000 7eff86754d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:41.138 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.129+0000 7eff86754d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:41.174 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-4 2022-07-20T03:06:41.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 5 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:41.313 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.301+0000 7f0148557d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-5/keyring: can't open /var/lib/ceph/osd/ceph-5/keyring: (2) No such file or directory 2022-07-20T03:06:41.313 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.301+0000 7f0148557d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-5/keyring 2022-07-20T03:06:41.331 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.313+0000 7f0148557d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:41.331 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.313+0000 7f0148557d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:41.465 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.461+0000 7f0148557d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:41.466 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.461+0000 7f0148557d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:41.466 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.461+0000 7f0148557d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:41.489 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-5 2022-07-20T03:06:41.522 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 6 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:41.605 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.601+0000 7f5734635d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-6/keyring: can't open /var/lib/ceph/osd/ceph-6/keyring: (2) No such file or directory 2022-07-20T03:06:41.639 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.601+0000 7f5734635d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-6/keyring 2022-07-20T03:06:41.640 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.609+0000 7f5734635d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:41.641 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.613+0000 7f5734635d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:41.759 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.749+0000 7f5734635d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:41.759 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.749+0000 7f5734635d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:41.760 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.749+0000 7f5734635d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:41.769 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-6 2022-07-20T03:06:41.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo MALLOC_CHECK_=3 adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-osd --no-mon-config --cluster ceph --mkfs --mkkey -i 7 --monmap /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:41.870 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.861+0000 7f5f29284d80 -1 auth: error reading file: /var/lib/ceph/osd/ceph-7/keyring: can't open /var/lib/ceph/osd/ceph-7/keyring: (2) No such file or directory 2022-07-20T03:06:41.871 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.861+0000 7f5f29284d80 -1 created new key in keyring /var/lib/ceph/osd/ceph-7/keyring 2022-07-20T03:06:41.892 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.873+0000 7f5f29284d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:41.893 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:41.873+0000 7f5f29284d80 -1 journal FileJournal::_open_file : zeroing journal 2022-07-20T03:06:42.033 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:42.021+0000 7f5f29284d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:42.034 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:42.025+0000 7f5f29284d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:42.034 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:42.025+0000 7f5f29284d80 -1 journal do_read_entry(4096): bad header magic 2022-07-20T03:06:42.052 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/osd/ceph-7 2022-07-20T03:06:42.068 INFO:tasks.ceph:Reading keys from all nodes... 2022-07-20T03:06:42.069 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:42.084 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.uhyhYzEZk6 2022-07-20T03:06:42.084 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/mgr/ceph-y/keyring /tmp/tmp.uhyhYzEZk6 2022-07-20T03:06:42.147 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.uhyhYzEZk6 2022-07-20T03:06:42.247 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.uhyhYzEZk6 is 56B 2022-07-20T03:06:42.332 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.uhyhYzEZk6 2022-07-20T03:06:42.397 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:42.415 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.ypiyYq8Scg 2022-07-20T03:06:42.415 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/mds/ceph-a/keyring /tmp/tmp.ypiyYq8Scg 2022-07-20T03:06:42.444 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.ypiyYq8Scg 2022-07-20T03:06:42.616 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.ypiyYq8Scg is 56B 2022-07-20T03:06:42.735 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.ypiyYq8Scg 2022-07-20T03:06:42.772 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:42.848 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.BLOh0bepNd 2022-07-20T03:06:42.849 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-0/keyring /tmp/tmp.BLOh0bepNd 2022-07-20T03:06:42.913 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.BLOh0bepNd 2022-07-20T03:06:43.007 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.BLOh0bepNd is 56B 2022-07-20T03:06:43.112 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.BLOh0bepNd 2022-07-20T03:06:43.142 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:43.201 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.0jlapP1eSY 2022-07-20T03:06:43.209 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-1/keyring /tmp/tmp.0jlapP1eSY 2022-07-20T03:06:43.278 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.0jlapP1eSY 2022-07-20T03:06:43.495 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.0jlapP1eSY is 56B 2022-07-20T03:06:43.568 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.0jlapP1eSY 2022-07-20T03:06:43.604 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:43.775 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.ykwT1uNFe3 2022-07-20T03:06:43.776 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-2/keyring /tmp/tmp.ykwT1uNFe3 2022-07-20T03:06:43.828 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.ykwT1uNFe3 2022-07-20T03:06:44.026 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.ykwT1uNFe3 is 56B 2022-07-20T03:06:44.140 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.ykwT1uNFe3 2022-07-20T03:06:44.155 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:44.243 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.J5X2o7Y4mN 2022-07-20T03:06:44.245 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-3/keyring /tmp/tmp.J5X2o7Y4mN 2022-07-20T03:06:44.272 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.J5X2o7Y4mN 2022-07-20T03:06:44.423 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.J5X2o7Y4mN is 56B 2022-07-20T03:06:44.482 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.J5X2o7Y4mN 2022-07-20T03:06:44.496 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:44.537 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.RIOXkmb3Ko 2022-07-20T03:06:44.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/mgr/ceph-x/keyring /tmp/tmp.RIOXkmb3Ko 2022-07-20T03:06:44.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.RIOXkmb3Ko 2022-07-20T03:06:44.781 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.RIOXkmb3Ko is 56B 2022-07-20T03:06:44.856 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.RIOXkmb3Ko 2022-07-20T03:06:44.880 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:44.983 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.hN8o4asi01 2022-07-20T03:06:44.990 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/mds/ceph-b/keyring /tmp/tmp.hN8o4asi01 2022-07-20T03:06:45.026 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.hN8o4asi01 2022-07-20T03:06:45.246 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.hN8o4asi01 is 56B 2022-07-20T03:06:45.269 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.hN8o4asi01 2022-07-20T03:06:45.283 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:45.348 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.Xv0YKrPUdO 2022-07-20T03:06:45.349 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-4/keyring /tmp/tmp.Xv0YKrPUdO 2022-07-20T03:06:45.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.Xv0YKrPUdO 2022-07-20T03:06:45.496 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.Xv0YKrPUdO is 56B 2022-07-20T03:06:45.516 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.Xv0YKrPUdO 2022-07-20T03:06:45.541 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:45.595 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.TGB9v44MEK 2022-07-20T03:06:45.596 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-5/keyring /tmp/tmp.TGB9v44MEK 2022-07-20T03:06:45.697 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.TGB9v44MEK 2022-07-20T03:06:45.853 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.TGB9v44MEK is 56B 2022-07-20T03:06:45.959 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.TGB9v44MEK 2022-07-20T03:06:45.985 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:46.048 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.Si4pihgkRC 2022-07-20T03:06:46.048 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-6/keyring /tmp/tmp.Si4pihgkRC 2022-07-20T03:06:46.083 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.Si4pihgkRC 2022-07-20T03:06:46.182 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.Si4pihgkRC is 56B 2022-07-20T03:06:46.259 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.Si4pihgkRC 2022-07-20T03:06:46.276 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:46.330 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.AM4p3pP6bI 2022-07-20T03:06:46.332 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-7/keyring /tmp/tmp.AM4p3pP6bI 2022-07-20T03:06:46.370 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.AM4p3pP6bI 2022-07-20T03:06:46.508 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.AM4p3pP6bI is 56B 2022-07-20T03:06:46.591 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.AM4p3pP6bI 2022-07-20T03:06:46.654 DEBUG:teuthology.orchestra.remote:smithi124:/etc/ceph/ceph.client.0.keyring is 59B 2022-07-20T03:06:46.773 DEBUG:teuthology.orchestra.remote:smithi163:/etc/ceph/ceph.client.1.keyring is 59B 2022-07-20T03:06:47.048 DEBUG:teuthology.orchestra.remote:smithi179:/etc/ceph/ceph.client.2.keyring is 59B 2022-07-20T03:06:47.174 INFO:tasks.ceph:Adding keys to all mons... 2022-07-20T03:06:47.175 DEBUG:teuthology.orchestra.run.smithi061:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-20T03:06:47.187 DEBUG:teuthology.orchestra.run.smithi117:> sudo tee -a /etc/ceph/ceph.keyring 2022-07-20T03:06:47.229 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.y] 2022-07-20T03:06:47.229 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q== 2022-07-20T03:06:47.264 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-20T03:06:47.265 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw== 2022-07-20T03:06:47.265 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.0] 2022-07-20T03:06:47.267 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ== 2022-07-20T03:06:47.267 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.1] 2022-07-20T03:06:47.271 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw== 2022-07-20T03:06:47.273 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.2] 2022-07-20T03:06:47.280 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA== 2022-07-20T03:06:47.280 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.3] 2022-07-20T03:06:47.281 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw== 2022-07-20T03:06:47.281 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr.x] 2022-07-20T03:06:47.282 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw== 2022-07-20T03:06:47.282 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-20T03:06:47.282 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg== 2022-07-20T03:06:47.283 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.4] 2022-07-20T03:06:47.283 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg== 2022-07-20T03:06:47.366 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.5] 2022-07-20T03:06:47.366 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w== 2022-07-20T03:06:47.367 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.6] 2022-07-20T03:06:47.367 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ== 2022-07-20T03:06:47.368 INFO:teuthology.orchestra.run.smithi061.stdout:[osd.7] 2022-07-20T03:06:47.368 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew== 2022-07-20T03:06:47.369 INFO:teuthology.orchestra.run.smithi061.stdout:[client.0] 2022-07-20T03:06:47.369 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA== 2022-07-20T03:06:47.370 INFO:teuthology.orchestra.run.smithi061.stdout:[client.1] 2022-07-20T03:06:47.370 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg== 2022-07-20T03:06:47.371 INFO:teuthology.orchestra.run.smithi061.stdout:[client.2] 2022-07-20T03:06:47.371 INFO:teuthology.orchestra.run.smithi061.stdout: key = AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg== 2022-07-20T03:06:47.372 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.y] 2022-07-20T03:06:47.372 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q== 2022-07-20T03:06:47.373 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a] 2022-07-20T03:06:47.373 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw== 2022-07-20T03:06:47.374 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.0] 2022-07-20T03:06:47.374 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ== 2022-07-20T03:06:47.375 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.1] 2022-07-20T03:06:47.375 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw== 2022-07-20T03:06:47.376 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.2] 2022-07-20T03:06:47.376 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA== 2022-07-20T03:06:47.377 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.3] 2022-07-20T03:06:47.377 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw== 2022-07-20T03:06:47.377 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr.x] 2022-07-20T03:06:47.401 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw== 2022-07-20T03:06:47.401 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b] 2022-07-20T03:06:47.401 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg== 2022-07-20T03:06:47.401 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.4] 2022-07-20T03:06:47.402 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg== 2022-07-20T03:06:47.402 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.5] 2022-07-20T03:06:47.402 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w== 2022-07-20T03:06:47.402 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.6] 2022-07-20T03:06:47.403 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ== 2022-07-20T03:06:47.403 INFO:teuthology.orchestra.run.smithi117.stdout:[osd.7] 2022-07-20T03:06:47.403 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew== 2022-07-20T03:06:47.403 INFO:teuthology.orchestra.run.smithi117.stdout:[client.0] 2022-07-20T03:06:47.403 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA== 2022-07-20T03:06:47.404 INFO:teuthology.orchestra.run.smithi117.stdout:[client.1] 2022-07-20T03:06:47.404 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg== 2022-07-20T03:06:47.404 INFO:teuthology.orchestra.run.smithi117.stdout:[client.2] 2022-07-20T03:06:47.404 INFO:teuthology.orchestra.run.smithi117.stdout: key = AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg== 2022-07-20T03:06:47.418 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T03:06:47.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.y --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T03:06:47.521 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T03:06:47.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.a --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T03:06:47.617 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.671 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.0 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.731 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.796 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.1 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.854 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.2 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.940 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:47.978 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.3 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.025 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T03:06:48.139 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mgr.x --cap mon 'allow profile mgr' --cap osd 'allow *' --cap mds 'allow *' 2022-07-20T03:06:48.265 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T03:06:48.277 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=mds.b --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' --cap mds allow 2022-07-20T03:06:48.362 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.372 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.4 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.433 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.440 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.5 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.522 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.529 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.6 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.580 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.587 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=osd.7 --cap mon 'allow *' --cap mgr 'allow *' --cap osd 'allow *' 2022-07-20T03:06:48.720 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T03:06:48.723 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.0 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T03:06:48.837 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T03:06:48.854 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.1 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T03:06:48.967 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T03:06:48.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-authtool /etc/ceph/ceph.keyring --name=client.2 --cap mon 'allow rw' --cap mgr 'allow r' --cap osd 'allow rwx' --cap mds allow 2022-07-20T03:06:49.036 INFO:tasks.ceph:Running mkfs on mon nodes... 2022-07-20T03:06:49.043 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /var/lib/ceph/mon/ceph-a 2022-07-20T03:06:49.111 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i a --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-20T03:06:49.213 DEBUG:teuthology.orchestra.run.smithi061:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-a 2022-07-20T03:06:49.309 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-b 2022-07-20T03:06:49.364 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i b --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-20T03:06:49.444 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-b 2022-07-20T03:06:49.510 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /var/lib/ceph/mon/ceph-c 2022-07-20T03:06:49.572 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph-mon --cluster ceph --mkfs -i c --monmap /home/ubuntu/cephtest/ceph.monmap --keyring /etc/ceph/ceph.keyring 2022-07-20T03:06:49.704 DEBUG:teuthology.orchestra.run.smithi117:> sudo chown -R ceph:ceph /var/lib/ceph/mon/ceph-c 2022-07-20T03:06:49.730 DEBUG:teuthology.orchestra.run.smithi061:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:49.734 DEBUG:teuthology.orchestra.run.smithi117:> rm -- /home/ubuntu/cephtest/ceph.monmap 2022-07-20T03:06:49.745 INFO:tasks.ceph:Starting mon daemons in cluster ceph... 2022-07-20T03:06:49.746 INFO:tasks.ceph.mon.a:Restarting daemon 2022-07-20T03:06:49.746 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i a 2022-07-20T03:06:49.879 INFO:tasks.ceph.mon.a:Started 2022-07-20T03:06:49.880 INFO:tasks.ceph.mon.b:Restarting daemon 2022-07-20T03:06:49.880 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i b 2022-07-20T03:06:49.934 INFO:tasks.ceph.mon.b:Started 2022-07-20T03:06:49.956 INFO:tasks.ceph.mon.c:Restarting daemon 2022-07-20T03:06:49.957 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mon -f --cluster ceph -i c 2022-07-20T03:06:49.965 INFO:tasks.ceph.mon.c:Started 2022-07-20T03:06:49.965 INFO:tasks.ceph:Starting mgr daemons in cluster ceph... 2022-07-20T03:06:49.966 INFO:tasks.ceph.mgr.y:Restarting daemon 2022-07-20T03:06:49.966 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i y 2022-07-20T03:06:49.972 INFO:tasks.ceph.mgr.y:Started 2022-07-20T03:06:49.972 INFO:tasks.ceph.mgr.x:Restarting daemon 2022-07-20T03:06:49.973 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mgr -f --cluster ceph -i x 2022-07-20T03:06:50.040 INFO:tasks.ceph.mgr.x:Started 2022-07-20T03:06:50.041 INFO:tasks.ceph:Setting crush tunables to default 2022-07-20T03:06:50.057 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd crush tunables default 2022-07-20T03:06:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.163+0000 7f5ca8930700 1 Processor -- start 2022-07-20T03:06:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.163+0000 7f5ca8930700 1 -- start start 2022-07-20T03:06:50.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca8930700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 0x7f5ca41407d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca8930700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5ca4141110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca8930700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ca40b4440 con 0x7f5ca40b6060 2022-07-20T03:06:50.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca8930700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ca4143fd0 con 0x7f5ca40b51e0 2022-07-20T03:06:50.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca8930700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4144270 con 0x7f5ca4140cc0 2022-07-20T03:06:50.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5ca40b6060 0x7f5ca4144710 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38420/0 (socket says 172.21.15.61:38420) 2022-07-20T03:06:50.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca37fe700 1 -- 172.21.15.61:0/1798991443 learned_addr learned my addr 172.21.15.61:0/1798991443 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:06:50.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca2ffd700 1 --2- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 0x7f5ca41407d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5ca4141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca2ffd700 1 -- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 msgr2=0x7f5ca41407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 14 2022-07-20T03:06:50.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca2ffd700 1 -- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 msgr2=0x7f5ca41407d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-20T03:06:50.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca2ffd700 1 --2- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 0x7f5ca41407d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-20T03:06:50.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca3fff700 1 -- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 msgr2=0x7f5ca4141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_bulk peer close file descriptor 15 2022-07-20T03:06:50.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca2ffd700 1 --2- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 0x7f5ca41407d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-20T03:06:50.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca3fff700 1 -- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 msgr2=0x7f5ca4141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).read_until read failed 2022-07-20T03:06:50.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5ca4141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_read_frame_preamble_main read frame preamble failed r=-1 ((1) Operation not permitted) 2022-07-20T03:06:50.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.167+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5ca4141110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0)._fault waiting 0.200000 2022-07-20T03:06:50.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2375668370 0 0) 0x7f5ca40b4440 con 0x7f5ca40b6060 2022-07-20T03:06:50.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c90003260 con 0x7f5ca40b6060 2022-07-20T03:06:50.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1258437958 0 0) 0x7f5c90003260 con 0x7f5ca40b6060 2022-07-20T03:06:50.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 msgr2=0x7f5ca4141110 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-20T03:06:50.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5ca4141110 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 msgr2=0x7f5ca41407d0 unknown :-1 s=STATE_CONNECTING l=0).mark_down 2022-07-20T03:06:50.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 --2- 172.21.15.61:0/1798991443 >> v2:172.21.15.117:3300/0 conn(0x7f5ca40b51e0 0x7f5ca41407d0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca4144e20 con 0x7f5ca40b6060 2022-07-20T03:06:50.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f5c94001680 con 0x7f5ca40b6060 2022-07-20T03:06:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 >> v1:172.21.15.61:6789/0 conn(0x7f5ca40b6060 legacy=0x7f5ca4144710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c90003750 0x7f5c90003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5c90004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c900049f0 0x7f5c90004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4143fd0 con 0x7f5c90003750 2022-07-20T03:06:50.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4144270 con 0x7f5c900049f0 2022-07-20T03:06:50.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c9000b7d0 con 0x7f5ca4140cc0 2022-07-20T03:06:50.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca37fe700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c90003750 0x7f5c90003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca2ffd700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5c90004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.191+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c900049f0 0x7f5c90004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca37fe700 1 -- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 msgr2=0x7f5c90004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:50.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca37fe700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca4140cc0 0x7f5c90004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca37fe700 1 -- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c900049f0 msgr2=0x7f5c90004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:50.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca37fe700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c900049f0 0x7f5c90004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca37fe700 1 -- 172.21.15.61:0/1798991443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c90003260 con 0x7f5c90003750 2022-07-20T03:06:50.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca37fe700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c90003750 0x7f5c90003b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5c94001390 tx=0x7f5ca40ad500).ready entity=mon.0 client_cookie=b7f8643cc91afa88 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:50.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c94007170 con 0x7f5c90003750 2022-07-20T03:06:50.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c940072d0 con 0x7f5c90003750 2022-07-20T03:06:50.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.195+0000 7f5ca27fc700 1 -- 172.21.15.61:0/1798991443 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9400c8c0 con 0x7f5c90003750 2022-07-20T03:06:50.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca8930700 1 -- 172.21.15.61:0/1798991443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c90003750 msgr2=0x7f5c90003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:50.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca8930700 1 --2- 172.21.15.61:0/1798991443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c90003750 0x7f5c90003b70 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f5c94001390 tx=0x7f5ca40ad500).stop 2022-07-20T03:06:50.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca3fff700 1 -- 172.21.15.61:0/1798991443 reap_dead start 2022-07-20T03:06:50.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca8930700 1 -- 172.21.15.61:0/1798991443 shutdown_connections 2022-07-20T03:06:50.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca8930700 1 -- 172.21.15.61:0/1798991443 >> 172.21.15.61:0/1798991443 conn(0x7f5ca4063d10 msgr2=0x7f5ca40b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:50.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca8930700 1 -- 172.21.15.61:0/1798991443 shutdown_connections 2022-07-20T03:06:50.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.199+0000 7f5ca8930700 1 -- 172.21.15.61:0/1798991443 wait complete. 2022-07-20T03:06:50.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 Processor -- start 2022-07-20T03:06:50.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 -- start start 2022-07-20T03:06:50.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c90003750 0x7f5ca4064480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca40aa9d0 0x7f5ca4142740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 0x7f5ca4065260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:50.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40abb70 con 0x7f5ca4064db0 2022-07-20T03:06:50.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca40ac2a0 con 0x7f5c90003750 2022-07-20T03:06:50.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ca4065750 con 0x7f5ca40aa9d0 2022-07-20T03:06:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 0x7f5ca4065260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca2ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca40aa9d0 0x7f5ca4142740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca37fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c90003750 0x7f5ca4064480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 0x7f5ca4065260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:41850/0 (socket says 172.21.15.61:41850) 2022-07-20T03:06:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca2ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca40aa9d0 0x7f5ca4142740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:56752/0 (socket says 172.21.15.61:56752) 2022-07-20T03:06:50.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 -- 172.21.15.61:0/3489415042 learned_addr learned my addr 172.21.15.61:0/3489415042 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:06:50.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 -- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca40aa9d0 msgr2=0x7f5ca4142740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:50.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca40aa9d0 0x7f5ca4142740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 -- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c90003750 msgr2=0x7f5ca4064480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:50.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c90003750 0x7f5ca4064480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 -- 172.21.15.61:0/3489415042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ca40aae30 con 0x7f5ca4064db0 2022-07-20T03:06:50.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca37fe700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c90003750 0x7f5ca4064480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:50.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca3fff700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 0x7f5ca4065260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5ca40669f0 tx=0x7f5c9c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:50.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9c019070 con 0x7f5ca4064db0 2022-07-20T03:06:50.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c9c0147e0 con 0x7f5ca4064db0 2022-07-20T03:06:50.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ca40675e0 con 0x7f5ca4064db0 2022-07-20T03:06:50.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c9c00c070 con 0x7f5ca4064db0 2022-07-20T03:06:50.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.203+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ca4142c30 con 0x7f5ca4064db0 2022-07-20T03:06:50.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.207+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ca40b40c0 con 0x7f5ca4064db0 2022-07-20T03:06:50.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.211+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 1) v1 ==== 345+0+0 (secure 0 0 0) 0x7f5c9c012040 con 0x7f5ca4064db0 2022-07-20T03:06:50.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.211+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5c9c005a70 con 0x7f5ca4064db0 2022-07-20T03:06:50.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.211+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f5c9c00f040 con 0x7f5ca4064db0 2022-07-20T03:06:50.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.271+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f5ca40574e0 con 0x7f5ca4064db0 2022-07-20T03:06:50.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca0ff9700 1 -- 172.21.15.61:0/3489415042 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v2) v1 ==== 124+0+0 (secure 0 0 0) 0x7f5c9c015480 con 0x7f5ca4064db0 2022-07-20T03:06:50.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 msgr2=0x7f5ca4065260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:50.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 0x7f5ca4065260 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f5ca40669f0 tx=0x7f5c9c008040).stop 2022-07-20T03:06:50.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 shutdown_connections 2022-07-20T03:06:50.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ca4064db0 0x7f5ca4065260 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ca40aa9d0 0x7f5ca4142740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 --2- 172.21.15.61:0/3489415042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c90003750 0x7f5ca4064480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 >> 172.21.15.61:0/3489415042 conn(0x7f5ca4063d10 msgr2=0x7f5ca41410d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 shutdown_connections 2022-07-20T03:06:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:06:50.279+0000 7f5ca8930700 1 -- 172.21.15.61:0/3489415042 wait complete. 2022-07-20T03:06:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:adjusted tunables profile to default 2022-07-20T03:06:50.331 INFO:tasks.ceph:Starting osd daemons in cluster ceph... 2022-07-20T03:06:50.332 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:50.348 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.ouywOucFQ2 2022-07-20T03:06:50.349 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-0/fsid /tmp/tmp.ouywOucFQ2 2022-07-20T03:06:50.384 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.ouywOucFQ2 2022-07-20T03:06:50.498 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.ouywOucFQ2 is 37B 2022-07-20T03:06:50.620 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.ouywOucFQ2 2022-07-20T03:06:50.653 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:50.725 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.39ZTm1mb3A 2022-07-20T03:06:50.730 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-1/fsid /tmp/tmp.39ZTm1mb3A 2022-07-20T03:06:50.761 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.39ZTm1mb3A 2022-07-20T03:06:50.945 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.39ZTm1mb3A is 37B 2022-07-20T03:06:51.026 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.39ZTm1mb3A 2022-07-20T03:06:51.101 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:51.183 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.RYun6tuAdf 2022-07-20T03:06:51.183 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-2/fsid /tmp/tmp.RYun6tuAdf 2022-07-20T03:06:51.212 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.RYun6tuAdf 2022-07-20T03:06:51.346 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.RYun6tuAdf is 37B 2022-07-20T03:06:51.400 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.RYun6tuAdf 2022-07-20T03:06:51.414 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T03:06:51.471 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.hWxzb4pOTS 2022-07-20T03:06:51.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo cp /var/lib/ceph/osd/ceph-3/fsid /tmp/tmp.hWxzb4pOTS 2022-07-20T03:06:51.502 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.hWxzb4pOTS 2022-07-20T03:06:51.726 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.hWxzb4pOTS is 37B 2022-07-20T03:06:51.818 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.hWxzb4pOTS 2022-07-20T03:06:51.848 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:51.876 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.tkBUetRwEW 2022-07-20T03:06:51.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-4/fsid /tmp/tmp.tkBUetRwEW 2022-07-20T03:06:52.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.tkBUetRwEW 2022-07-20T03:06:52.257 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.tkBUetRwEW is 37B 2022-07-20T03:06:52.303 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.tkBUetRwEW 2022-07-20T03:06:52.380 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:52.436 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.uJWCAxEcwS 2022-07-20T03:06:52.437 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-5/fsid /tmp/tmp.uJWCAxEcwS 2022-07-20T03:06:52.523 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.uJWCAxEcwS 2022-07-20T03:06:52.684 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.uJWCAxEcwS is 37B 2022-07-20T03:06:52.853 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.uJWCAxEcwS 2022-07-20T03:06:52.889 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:52.930 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.xPcb7R39fj 2022-07-20T03:06:52.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-6/fsid /tmp/tmp.xPcb7R39fj 2022-07-20T03:06:52.960 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.xPcb7R39fj 2022-07-20T03:06:53.125 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.xPcb7R39fj is 37B 2022-07-20T03:06:53.282 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.xPcb7R39fj 2022-07-20T03:06:53.342 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T03:06:53.386 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.6vYLEVkZeB 2022-07-20T03:06:53.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo cp /var/lib/ceph/osd/ceph-7/fsid /tmp/tmp.6vYLEVkZeB 2022-07-20T03:06:53.432 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.6vYLEVkZeB 2022-07-20T03:06:53.572 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.6vYLEVkZeB is 37B 2022-07-20T03:06:53.619 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.6vYLEVkZeB 2022-07-20T03:06:53.747 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 2994e86d-2708-44df-bf9c-63667f52059f 0 2022-07-20T03:06:53.964 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 Processor -- start 2022-07-20T03:06:53.965 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 -- start start 2022-07-20T03:06:53.965 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:53.972 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3050061ac0 0x7f305005c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:53.972 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3050063c00 con 0x7f3050065650 2022-07-20T03:06:53.973 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30500b6c40 con 0x7f3050061ac0 2022-07-20T03:06:53.973 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3050146c90 con 0x7f30500b6860 2022-07-20T03:06:53.973 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:53.974 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305005bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:59944/0 (socket says 172.21.15.117:59944) 2022-07-20T03:06:53.974 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 -- 172.21.15.117:0/3295182952 learned_addr learned my addr 172.21.15.117:0/3295182952 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:53.974 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f3047fff700 1 --2- 172.21.15.117:0/3295182952 >> v2:172.21.15.61:3300/0 conn(0x7f3050061ac0 0x7f305005c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:53.974 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 -- 172.21.15.117:0/3295182952 >> v2:172.21.15.61:3300/0 conn(0x7f3050061ac0 msgr2=0x7f305005c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:53.975 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 --2- 172.21.15.117:0/3295182952 >> v2:172.21.15.61:3300/0 conn(0x7f3050061ac0 0x7f305005c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:53.975 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 -- 172.21.15.117:0/3295182952 >> v1:172.21.15.61:6789/0 conn(0x7f3050065650 legacy=0x7f3050145be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:53.975 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 -- 172.21.15.117:0/3295182952 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3050063e30 con 0x7f30500b6860 2022-07-20T03:06:53.975 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.905+0000 7f304e59c700 1 --2- 172.21.15.117:0/3295182952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305005bd90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f3040002f70 tx=0x7f3040006eb0).ready entity=mon.2 client_cookie=8d92d8131107504 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:53.975 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304dd9b700 1 -- 172.21.15.117:0/3295182952 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f304000d570 con 0x7f30500b6860 2022-07-20T03:06:53.976 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304dd9b700 1 -- 172.21.15.117:0/3295182952 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040000e90 con 0x7f30500b6860 2022-07-20T03:06:53.976 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304dd9b700 1 -- 172.21.15.117:0/3295182952 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f304000d900 con 0x7f30500b6860 2022-07-20T03:06:53.976 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 -- 172.21.15.117:0/3295182952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 msgr2=0x7f305005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:53.976 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 --2- 172.21.15.117:0/3295182952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305005bd90 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f3040002f70 tx=0x7f3040006eb0).stop 2022-07-20T03:06:53.977 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 -- 172.21.15.117:0/3295182952 shutdown_connections 2022-07-20T03:06:53.977 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 --2- 172.21.15.117:0/3295182952 >> v2:172.21.15.61:3300/0 conn(0x7f3050061ac0 0x7f305005c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:53.977 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 --2- 172.21.15.117:0/3295182952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305005bd90 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:53.978 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 -- 172.21.15.117:0/3295182952 >> 172.21.15.117:0/3295182952 conn(0x7f30500b9dd0 msgr2=0x7f30500b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:53.978 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 -- 172.21.15.117:0/3295182952 shutdown_connections 2022-07-20T03:06:53.979 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 -- 172.21.15.117:0/3295182952 wait complete. 2022-07-20T03:06:53.979 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 Processor -- start 2022-07-20T03:06:53.979 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.909+0000 7f304f59e700 1 -- start start 2022-07-20T03:06:53.979 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 0x7f305014c8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:53.980 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3050065650 0x7f3050155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:53.980 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305014a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:53.980 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3050144f90 con 0x7f3050065650 2022-07-20T03:06:53.980 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f305014cf80 con 0x7f3050061ac0 2022-07-20T03:06:53.981 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3050143f60 con 0x7f30500b6860 2022-07-20T03:06:53.981 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 0x7f305014c8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:53.982 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 0x7f305014c8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44336/0 (socket says 172.21.15.117:44336) 2022-07-20T03:06:53.983 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 -- 172.21.15.117:0/849152133 learned_addr learned my addr 172.21.15.117:0/849152133 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:53.983 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304ed9d700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305014a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:53.983 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f3047fff700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3050065650 0x7f3050155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:53.983 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 -- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 msgr2=0x7f305014a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:53.983 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305014a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:53.984 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 -- 172.21.15.117:0/849152133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3050065650 msgr2=0x7f3050155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:53.984 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3050065650 0x7f3050155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:53.984 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 -- 172.21.15.117:0/849152133 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f304000b1e0 con 0x7f3050061ac0 2022-07-20T03:06:53.984 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304e59c700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 0x7f305014c8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3040002f70 tx=0x7f3040015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:53.985 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30400117e0 con 0x7f3050061ac0 2022-07-20T03:06:53.985 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3040011940 con 0x7f3050061ac0 2022-07-20T03:06:53.985 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3050143b60 con 0x7f3050061ac0 2022-07-20T03:06:53.987 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3040011ca0 con 0x7f3050061ac0 2022-07-20T03:06:53.987 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3050148df0 con 0x7f3050061ac0 2022-07-20T03:06:53.987 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 2) v1 ==== 32894+0+0 (secure 0 0 0) 0x7f3040025030 con 0x7f3050061ac0 2022-07-20T03:06:53.987 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 754+0+0 (secure 0 0 0) 0x7f304003b200 con 0x7f3050061ac0 2022-07-20T03:06:53.988 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.913+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30500b3e30 con 0x7f3050061ac0 2022-07-20T03:06:53.995 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.921+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+67674 (secure 0 0 0) 0x7f3040023070 con 0x7f3050061ac0 2022-07-20T03:06:53.997 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.985+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", "id": 0} v 0) v1 -- 0x7f30500574e0 con 0x7f3050061ac0 2022-07-20T03:06:53.998 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.993+0000 7f30477fe700 1 -- 172.21.15.117:0/849152133 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", "id": 0}]=0 v3) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3040019d80 con 0x7f3050061ac0 2022-07-20T03:06:53.999 INFO:teuthology.orchestra.run.smithi117.stdout:0 2022-07-20T03:06:54.018 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 msgr2=0x7f305014c8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.021 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 0x7f305014c8b0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3040002f70 tx=0x7f3040015000).stop 2022-07-20T03:06:54.022 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 shutdown_connections 2022-07-20T03:06:54.022 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30500b6860 0x7f305014a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.023 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3050065650 0x7f3050155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.023 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 --2- 172.21.15.117:0/849152133 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3050061ac0 0x7f305014c8b0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.024 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 >> 172.21.15.117:0/849152133 conn(0x7f30500b9dd0 msgr2=0x7f3050063fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:54.043 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 shutdown_connections 2022-07-20T03:06:54.044 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:53.997+0000 7f304f59e700 1 -- 172.21.15.117:0/849152133 wait complete. 2022-07-20T03:06:54.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 9cd27797-c189-4c94-9e22-769611620680 1 2022-07-20T03:06:54.188 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 Processor -- start 2022-07-20T03:06:54.189 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 -- start start 2022-07-20T03:06:54.204 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fedb413a770 0x7fedb4067120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.205 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 0x7fedb414b120 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.205 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedb40b4340 con 0x7fedb4067610 2022-07-20T03:06:54.205 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedb4066680 con 0x7fedb413a770 2022-07-20T03:06:54.205 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedba40c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40b1d80 con 0x7fedb405c4a0 2022-07-20T03:06:54.235 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedb9c0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 0x7fedb414b120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.235 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedb9c0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 0x7fedb414b120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:59970/0 (socket says 172.21.15.117:59970) 2022-07-20T03:06:54.235 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedb9c0b700 1 -- 172.21.15.117:0/3409674143 learned_addr learned my addr 172.21.15.117:0/3409674143 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:54.235 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedb940a700 1 --2- 172.21.15.117:0/3409674143 >> v2:172.21.15.61:3300/0 conn(0x7fedb413a770 0x7fedb4067120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.236 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedb9c0b700 1 -- 172.21.15.117:0/3409674143 >> v2:172.21.15.61:3300/0 conn(0x7fedb413a770 msgr2=0x7fedb4067120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:54.236 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.185+0000 7fedb9c0b700 1 --2- 172.21.15.117:0/3409674143 >> v2:172.21.15.61:3300/0 conn(0x7fedb413a770 0x7fedb4067120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.236 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedb9c0b700 1 -- 172.21.15.117:0/3409674143 >> v1:172.21.15.61:6789/0 conn(0x7fedb4067610 legacy=0x7fedb405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.236 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedb9c0b700 1 -- 172.21.15.117:0/3409674143 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedb40b1ae0 con 0x7fedb405c4a0 2022-07-20T03:06:54.237 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedb940a700 1 --2- 172.21.15.117:0/3409674143 >> v2:172.21.15.61:3300/0 conn(0x7fedb413a770 0x7fedb4067120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:06:54.237 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedb9c0b700 1 --2- 172.21.15.117:0/3409674143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 0x7fedb414b120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fedac0032f0 tx=0x7fedac009290).ready entity=mon.2 client_cookie=effd7b5215646402 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.237 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fed9ffff700 1 -- 172.21.15.117:0/3409674143 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedac00b8a0 con 0x7fedb405c4a0 2022-07-20T03:06:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fed9ffff700 1 -- 172.21.15.117:0/3409674143 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedac00ba00 con 0x7fedb405c4a0 2022-07-20T03:06:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fed9ffff700 1 -- 172.21.15.117:0/3409674143 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedac00bb70 con 0x7fedb405c4a0 2022-07-20T03:06:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 -- 172.21.15.117:0/3409674143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 msgr2=0x7fedb414b120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.238 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 --2- 172.21.15.117:0/3409674143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 0x7fedb414b120 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fedac0032f0 tx=0x7fedac009290).stop 2022-07-20T03:06:54.239 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 -- 172.21.15.117:0/3409674143 shutdown_connections 2022-07-20T03:06:54.239 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 --2- 172.21.15.117:0/3409674143 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb405c4a0 0x7fedb414b120 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.239 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 --2- 172.21.15.117:0/3409674143 >> v2:172.21.15.61:3300/0 conn(0x7fedb413a770 0x7fedb4067120 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.239 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 -- 172.21.15.117:0/3409674143 >> 172.21.15.117:0/3409674143 conn(0x7fedb4064c00 msgr2=0x7fedb4064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 -- 172.21.15.117:0/3409674143 shutdown_connections 2022-07-20T03:06:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 -- 172.21.15.117:0/3409674143 wait complete. 2022-07-20T03:06:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 Processor -- start 2022-07-20T03:06:54.253 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.189+0000 7fedba40c700 1 -- start start 2022-07-20T03:06:54.254 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb405c4a0 0x7fedb40b6300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.254 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedb4067610 0x7fedb40ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.254 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 0x7fedb41502e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.254 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb413ca90 con 0x7fedb405c4a0 2022-07-20T03:06:54.255 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedb40add80 con 0x7fedb4067610 2022-07-20T03:06:54.255 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedb413c590 con 0x7fedb413a770 2022-07-20T03:06:54.256 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb9c0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 0x7fedb41502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.256 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedb4067610 0x7fedb40ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.257 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb940a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb405c4a0 0x7fedb40b6300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.257 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb9c0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 0x7fedb41502e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:59976/0 (socket says 172.21.15.117:59976) 2022-07-20T03:06:54.257 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb9c0b700 1 -- 172.21.15.117:0/2292458055 learned_addr learned my addr 172.21.15.117:0/2292458055 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:54.300 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 -- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 msgr2=0x7fedb41502e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.300 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 0x7fedb41502e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.301 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 -- 172.21.15.117:0/2292458055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb405c4a0 msgr2=0x7fedb40b6300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.301 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb405c4a0 0x7fedb40b6300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.301 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 -- 172.21.15.117:0/2292458055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedac009530 con 0x7fedb4067610 2022-07-20T03:06:54.301 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb940a700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb405c4a0 0x7fedb40b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:54.302 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb9c0b700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 0x7fedb41502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:54.305 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedb8c09700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedb4067610 0x7fedb40ad890 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fedb413bcb0 tx=0x7fedb0007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.305 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedb0012400 con 0x7fedb4067610 2022-07-20T03:06:54.306 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedb0012560 con 0x7fedb4067610 2022-07-20T03:06:54.306 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedb413c7f0 con 0x7fedb4067610 2022-07-20T03:06:54.306 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedb00128c0 con 0x7fedb4067610 2022-07-20T03:06:54.306 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.193+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedb414fde0 con 0x7fedb4067610 2022-07-20T03:06:54.307 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.197+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedb413d870 con 0x7fedb4067610 2022-07-20T03:06:54.307 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.197+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7fedb0019030 con 0x7fedb4067610 2022-07-20T03:06:54.307 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.197+0000 7feda6ffd700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed90053a30 0x7fed90055ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.307 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.197+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 869+0+0 (secure 0 0 0) 0x7fedb00698c0 con 0x7fedb4067610 2022-07-20T03:06:54.307 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.197+0000 7fedb940a700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed90053a30 0x7fed90055ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.308 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.197+0000 7fedb940a700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed90053a30 0x7fed90055ef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feda800c000 tx=0x7feda8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.308 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.205+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fedb413d870 con 0x7fedb4067610 2022-07-20T03:06:54.340 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.333+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "9cd27797-c189-4c94-9e22-769611620680", "id": 1} v 0) v1 -- 0x7fedb413d870 con 0x7fedb4067610 2022-07-20T03:06:54.354 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7feda6ffd700 1 -- 172.21.15.117:0/2292458055 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "9cd27797-c189-4c94-9e22-769611620680", "id": 1}]=0 v4) v1 ==== 112+0+2 (secure 0 0 0) 0x7fedb413d870 con 0x7fedb4067610 2022-07-20T03:06:54.355 INFO:teuthology.orchestra.run.smithi117.stdout:1 2022-07-20T03:06:54.356 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed90053a30 msgr2=0x7fed90055ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.357 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed90053a30 0x7fed90055ef0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feda800c000 tx=0x7feda8008040).stop 2022-07-20T03:06:54.357 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedb4067610 msgr2=0x7fedb40ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.357 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedb4067610 0x7fedb40ad890 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fedb413bcb0 tx=0x7fedb0007b90).stop 2022-07-20T03:06:54.357 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 shutdown_connections 2022-07-20T03:06:54.359 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed90053a30 0x7fed90055ef0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.360 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedb4067610 0x7fedb40ad890 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.360 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedb413a770 0x7fedb41502e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.360 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 --2- 172.21.15.117:0/2292458055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedb405c4a0 0x7fedb40b6300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.361 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 >> 172.21.15.117:0/2292458055 conn(0x7fedb4064c00 msgr2=0x7fedb4064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:54.361 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 shutdown_connections 2022-07-20T03:06:54.361 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.345+0000 7fedba40c700 1 -- 172.21.15.117:0/2292458055 wait complete. 2022-07-20T03:06:54.375 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 637945c3-b1e0-4d86-933a-2723e56b0705 2 2022-07-20T03:06:54.502 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 Processor -- start 2022-07-20T03:06:54.503 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 -- start start 2022-07-20T03:06:54.503 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3f800b9b70 0x7f3f800b9f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.503 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3f800b00f0 0x7f3f8013c8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.504 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f801462b0 con 0x7f3f8013cde0 2022-07-20T03:06:54.504 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f800b3f80 con 0x7f3f800b9b70 2022-07-20T03:06:54.504 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f858a0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f801464e0 con 0x7f3f800b00f0 2022-07-20T03:06:54.504 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f7f7fe700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f3f8013cde0 0x7f3f80142f10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52282/0 (socket says 172.21.15.117:52282) 2022-07-20T03:06:54.505 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f7f7fe700 1 -- 172.21.15.117:0/182026816 learned_addr learned my addr 172.21.15.117:0/182026816 (peer_addr_for_me v1:172.21.15.117:0/0) 2022-07-20T03:06:54.505 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f7e7fc700 1 --2- 172.21.15.117:0/182026816 >> v2:172.21.15.117:3300/0 conn(0x7f3f800b00f0 0x7f3f8013c8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.506 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.485+0000 7f3f7effd700 1 --2- 172.21.15.117:0/182026816 >> v2:172.21.15.61:3300/0 conn(0x7f3f800b9b70 0x7f3f800b9f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.506 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7dffb700 1 -- 172.21.15.117:0/182026816 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2518564570 0 0) 0x7f3f801462b0 con 0x7f3f8013cde0 2022-07-20T03:06:54.506 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7dffb700 1 -- 172.21.15.117:0/182026816 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3f64003260 con 0x7f3f8013cde0 2022-07-20T03:06:54.507 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/182026816 >> v2:172.21.15.61:3300/0 conn(0x7f3f800b9b70 msgr2=0x7f3f800b9f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:54.507 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7e7fc700 1 --2- 172.21.15.117:0/182026816 >> v2:172.21.15.61:3300/0 conn(0x7f3f800b9b70 0x7f3f800b9f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.507 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/182026816 >> v1:172.21.15.117:6789/0 conn(0x7f3f8013cde0 legacy=0x7f3f80142f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.507 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/182026816 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f800b47d0 con 0x7f3f800b00f0 2022-07-20T03:06:54.508 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7e7fc700 1 --2- 172.21.15.117:0/182026816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f8013c8f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3f70002f70 tx=0x7f3f70006df0).ready entity=mon.1 client_cookie=f9180343caa08f4d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.508 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7dffb700 1 -- 172.21.15.117:0/182026816 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7000d570 con 0x7f3f800b00f0 2022-07-20T03:06:54.508 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7dffb700 1 -- 172.21.15.117:0/182026816 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f70000e90 con 0x7f3f800b00f0 2022-07-20T03:06:54.508 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f7dffb700 1 -- 172.21.15.117:0/182026816 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7000d8c0 con 0x7f3f800b00f0 2022-07-20T03:06:54.509 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 -- 172.21.15.117:0/182026816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 msgr2=0x7f3f8013c8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.509 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 --2- 172.21.15.117:0/182026816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f8013c8f0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f3f70002f70 tx=0x7f3f70006df0).stop 2022-07-20T03:06:54.509 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 -- 172.21.15.117:0/182026816 shutdown_connections 2022-07-20T03:06:54.509 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 --2- 172.21.15.117:0/182026816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f8013c8f0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.510 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 --2- 172.21.15.117:0/182026816 >> v2:172.21.15.61:3300/0 conn(0x7f3f800b9b70 0x7f3f800b9f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.510 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 -- 172.21.15.117:0/182026816 >> 172.21.15.117:0/182026816 conn(0x7f3f80067980 msgr2=0x7f3f800b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:54.510 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 -- 172.21.15.117:0/182026816 shutdown_connections 2022-07-20T03:06:54.510 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 -- 172.21.15.117:0/182026816 wait complete. 2022-07-20T03:06:54.529 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 Processor -- start 2022-07-20T03:06:54.529 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.489+0000 7f3f858a0700 1 -- start start 2022-07-20T03:06:54.530 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f80152c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.530 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 0x7f3f800b4aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.530 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f8013cde0 0x7f3f801441b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.531 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80145a30 con 0x7f3f8013cde0 2022-07-20T03:06:54.531 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f801524f0 con 0x7f3f800b00f0 2022-07-20T03:06:54.531 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f80152790 con 0x7f3f800b9b70 2022-07-20T03:06:54.531 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 0x7f3f800b4aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.532 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7effd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f80152c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.532 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 0x7f3f800b4aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:59986/0 (socket says 172.21.15.117:59986) 2022-07-20T03:06:54.532 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/801744820 learned_addr learned my addr 172.21.15.117:0/801744820 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:54.532 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7f7fe700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f8013cde0 0x7f3f801441b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.533 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 msgr2=0x7f3f80152c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.533 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f80152c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.533 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/801744820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f8013cde0 msgr2=0x7f3f801441b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.533 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f8013cde0 0x7f3f801441b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.553 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 -- 172.21.15.117:0/801744820 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f7000b180 con 0x7f3f800b9b70 2022-07-20T03:06:54.553 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7effd700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f80152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:54.553 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7e7fc700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 0x7f3f800b4aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3f70002f70 tx=0x7f3f70014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.564 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f70013e90 con 0x7f3f800b9b70 2022-07-20T03:06:54.564 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f7000de40 con 0x7f3f800b9b70 2022-07-20T03:06:54.564 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f80153120 con 0x7f3f800b9b70 2022-07-20T03:06:54.565 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f7001e5f0 con 0x7f3f800b9b70 2022-07-20T03:06:54.565 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f800b4f90 con 0x7f3f800b9b70 2022-07-20T03:06:54.565 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7f3f70025030 con 0x7f3f800b9b70 2022-07-20T03:06:54.565 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f800574e0 con 0x7f3f800b9b70 2022-07-20T03:06:54.566 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f6ffff700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f5c053910 0x7f3f5c055dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.566 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1360+0+0 (secure 0 0 0) 0x7f3f7006b790 con 0x7f3f800b9b70 2022-07-20T03:06:54.566 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.493+0000 7f3f7effd700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f5c053910 0x7f3f5c055dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.566 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.497+0000 7f3f7effd700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f5c053910 0x7f3f5c055dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f3f7400c000 tx=0x7f3f74008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.567 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.501+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f70043a00 con 0x7f3f800b9b70 2022-07-20T03:06:54.640 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.637+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd new", "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", "id": 2} v 0) v1 -- 0x7f3f80069a60 con 0x7f3f800b9b70 2022-07-20T03:06:54.650 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.645+0000 7f3f6ffff700 1 -- 172.21.15.117:0/801744820 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", "id": 2}]=0 v5) v1 ==== 112+0+2 (secure 0 0 0) 0x7f3f70023030 con 0x7f3f800b9b70 2022-07-20T03:06:54.651 INFO:teuthology.orchestra.run.smithi117.stdout:2 2022-07-20T03:06:54.661 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f5c053910 msgr2=0x7f3f5c055dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.662 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f5c053910 0x7f3f5c055dd0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f3f7400c000 tx=0x7f3f74008040).stop 2022-07-20T03:06:54.663 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 msgr2=0x7f3f800b4aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.664 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 0x7f3f800b4aa0 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f3f70002f70 tx=0x7f3f70014000).stop 2022-07-20T03:06:54.664 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 shutdown_connections 2022-07-20T03:06:54.665 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f5c053910 0x7f3f5c055dd0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.665 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f8013cde0 0x7f3f801441b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.671 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f800b9b70 0x7f3f800b4aa0 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.674 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 --2- 172.21.15.117:0/801744820 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f800b00f0 0x7f3f80152c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.675 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 >> 172.21.15.117:0/801744820 conn(0x7f3f80067980 msgr2=0x7f3f80139ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:54.675 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 shutdown_connections 2022-07-20T03:06:54.676 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.649+0000 7f3f858a0700 1 -- 172.21.15.117:0/801744820 wait complete. 2022-07-20T03:06:54.678 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new b6079eb6-2fd9-4f0e-831f-83b051fc040f 3 2022-07-20T03:06:54.844 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.817+0000 7fe08b049700 1 Processor -- start 2022-07-20T03:06:54.887 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.817+0000 7fe08b049700 1 -- start start 2022-07-20T03:06:54.887 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe0840b5970 0x7fe0840ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.913 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840af8c0 0x7fe0840ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:54.923 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0840b0b80 con 0x7fe0840aeae0 2022-07-20T03:06:54.924 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe08405d4d0 con 0x7fe0840b5970 2022-07-20T03:06:54.924 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840b5d50 con 0x7fe0840af8c0 2022-07-20T03:06:54.925 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a848700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe0840aeae0 0x7fe08405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52292/0 (socket says 172.21.15.117:52292) 2022-07-20T03:06:54.925 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a047700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe0840b5970 0x7fe0840ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.928 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a848700 1 -- 172.21.15.117:0/1483823442 learned_addr learned my addr 172.21.15.117:0/1483823442 (peer_addr_for_me v1:172.21.15.117:0/0) 2022-07-20T03:06:54.928 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe089846700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840af8c0 0x7fe0840ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:54.928 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a047700 1 -- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840af8c0 msgr2=0x7fe0840ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:54.929 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a047700 1 --2- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840af8c0 0x7fe0840ae5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:54.929 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a047700 1 -- 172.21.15.117:0/1483823442 >> v1:172.21.15.117:6789/0 conn(0x7fe0840aeae0 legacy=0x7fe08405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.944 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a047700 1 -- 172.21.15.117:0/1483823442 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0840b0db0 con 0x7fe0840b5970 2022-07-20T03:06:54.956 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe089045700 1 -- 172.21.15.117:0/1483823442 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3972538658 0 0) 0x7fe0840b0b80 con 0x7fe0840aeae0 2022-07-20T03:06:54.957 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08a047700 1 --2- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840b5970 0x7fe0840ae100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe074002f70 tx=0x7fe074006df0).ready entity=mon.1 client_cookie=9c09e575ab4a21b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:54.957 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe089045700 1 -- 172.21.15.117:0/1483823442 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe07400d570 con 0x7fe0840b5970 2022-07-20T03:06:54.958 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe089045700 1 -- 172.21.15.117:0/1483823442 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe074000e90 con 0x7fe0840b5970 2022-07-20T03:06:54.979 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe089045700 1 -- 172.21.15.117:0/1483823442 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe07400d8c0 con 0x7fe0840b5970 2022-07-20T03:06:54.980 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840b5970 msgr2=0x7fe0840ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:54.981 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 --2- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840b5970 0x7fe0840ae100 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fe074002f70 tx=0x7fe074006df0).stop 2022-07-20T03:06:54.981 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- 172.21.15.117:0/1483823442 shutdown_connections 2022-07-20T03:06:54.981 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 --2- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840af8c0 0x7fe0840ae5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.005 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 --2- 172.21.15.117:0/1483823442 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840b5970 0x7fe0840ae100 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.007 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- 172.21.15.117:0/1483823442 >> 172.21.15.117:0/1483823442 conn(0x7fe084063490 msgr2=0x7fe0840b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:55.008 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- 172.21.15.117:0/1483823442 shutdown_connections 2022-07-20T03:06:55.019 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.821+0000 7fe08b049700 1 -- 172.21.15.117:0/1483823442 wait complete. 2022-07-20T03:06:55.020 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 Processor -- start 2022-07-20T03:06:55.020 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 -- start start 2022-07-20T03:06:55.020 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 0x7fe0841111b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.021 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840af8c0 0x7fe084159810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.021 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0840b5970 0x7fe084159d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.027 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0841474f0 con 0x7fe0840b5970 2022-07-20T03:06:55.030 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0840676c0 con 0x7fe0840af8c0 2022-07-20T03:06:55.031 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe084147720 con 0x7fe0840aeae0 2022-07-20T03:06:55.031 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 0x7fe0841111b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.032 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe089846700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840af8c0 0x7fe084159810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.032 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 0x7fe0841111b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:59998/0 (socket says 172.21.15.117:59998) 2022-07-20T03:06:55.033 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 -- 172.21.15.117:0/866411718 learned_addr learned my addr 172.21.15.117:0/866411718 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:55.033 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a848700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0840b5970 0x7fe084159d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.034 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 -- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840af8c0 msgr2=0x7fe084159810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.048 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840af8c0 0x7fe084159810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.048 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 -- 172.21.15.117:0/866411718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0840b5970 msgr2=0x7fe084159d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.049 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0840b5970 0x7fe084159d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.049 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 -- 172.21.15.117:0/866411718 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe07400b180 con 0x7fe0840aeae0 2022-07-20T03:06:55.049 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe089846700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840af8c0 0x7fe084159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:55.069 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08a047700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 0x7fe0841111b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe074002f70 tx=0x7fe074014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.069 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe07400dea0 con 0x7fe0840aeae0 2022-07-20T03:06:55.070 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe074013de0 con 0x7fe0840aeae0 2022-07-20T03:06:55.072 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe084067340 con 0x7fe0840aeae0 2022-07-20T03:06:55.073 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe07401e5f0 con 0x7fe0840aeae0 2022-07-20T03:06:55.073 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.825+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe084066d30 con 0x7fe0840aeae0 2022-07-20T03:06:55.074 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.829+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7fe074025030 con 0x7fe0840aeae0 2022-07-20T03:06:55.074 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.829+0000 7fe0837fe700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0640539a0 0x7fe064055e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.075 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.829+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe084067880 con 0x7fe0840aeae0 2022-07-20T03:06:55.075 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.829+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1499+0+0 (secure 0 0 0) 0x7fe07406b830 con 0x7fe0840aeae0 2022-07-20T03:06:55.076 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.829+0000 7fe089846700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0640539a0 0x7fe064055e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.076 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.829+0000 7fe089846700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0640539a0 0x7fe064055e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe07800b000 tx=0x7fe078006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.077 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.837+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe084067880 con 0x7fe0840aeae0 2022-07-20T03:06:55.082 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.965+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd new", "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", "id": 3} v 0) v1 -- 0x7fe084067880 con 0x7fe0840aeae0 2022-07-20T03:06:55.083 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.977+0000 7fe0837fe700 1 -- 172.21.15.117:0/866411718 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", "id": 3}]=0 v6) v1 ==== 112+0+2 (secure 0 0 0) 0x7fe084067880 con 0x7fe0840aeae0 2022-07-20T03:06:55.084 INFO:teuthology.orchestra.run.smithi117.stdout:3 2022-07-20T03:06:55.113 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.977+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0640539a0 msgr2=0x7fe064055e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.115 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.977+0000 7fe08b049700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0640539a0 0x7fe064055e60 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe07800b000 tx=0x7fe078006df0).stop 2022-07-20T03:06:55.116 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.977+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 msgr2=0x7fe0841111b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.116 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.977+0000 7fe08b049700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 0x7fe0841111b0 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe074002f70 tx=0x7fe074014000).stop 2022-07-20T03:06:55.117 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 shutdown_connections 2022-07-20T03:06:55.117 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0640539a0 0x7fe064055e60 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.117 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0840af8c0 0x7fe084159810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.118 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0840b5970 0x7fe084159d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.118 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 --2- 172.21.15.117:0/866411718 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0840aeae0 0x7fe0841111b0 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.119 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 >> 172.21.15.117:0/866411718 conn(0x7fe084063490 msgr2=0x7fe0840b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:55.119 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 shutdown_connections 2022-07-20T03:06:55.120 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:54.981+0000 7fe08b049700 1 -- 172.21.15.117:0/866411718 wait complete. 2022-07-20T03:06:55.120 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 1f342a61-db71-49da-a367-b177e2e65fdd 4 2022-07-20T03:06:55.307 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daf59e700 1 Processor -- start 2022-07-20T03:06:55.308 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daf59e700 1 -- start start 2022-07-20T03:06:55.308 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daf59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0db00b6800 0x7f0db00aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.309 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daf59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00ae280 con 0x7f0db00b3cb0 2022-07-20T03:06:55.309 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daf59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0db00b6be0 con 0x7f0db00ad0e0 2022-07-20T03:06:55.309 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daf59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0db00ae4b0 con 0x7f0db00b6800 2022-07-20T03:06:55.317 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daed9d700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f0db00ad0e0 0x7f0db01418f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52308/0 (socket says 172.21.15.117:52308) 2022-07-20T03:06:55.317 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0daed9d700 1 -- 172.21.15.117:0/1405618606 learned_addr learned my addr 172.21.15.117:0/1405618606 (peer_addr_for_me v1:172.21.15.117:0/0) 2022-07-20T03:06:55.318 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dae59c700 1 --2- 172.21.15.117:0/1405618606 >> v2:172.21.15.61:3300/0 conn(0x7f0db00b6800 0x7f0db00aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.318 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 121008682 0 0) 0x7f0db00b6be0 con 0x7f0db00ad0e0 2022-07-20T03:06:55.319 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d90003260 con 0x7f0db00ad0e0 2022-07-20T03:06:55.319 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2595742500 0 0) 0x7f0db00ae280 con 0x7f0db00b3cb0 2022-07-20T03:06:55.320 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db00b6be0 con 0x7f0db00b3cb0 2022-07-20T03:06:55.320 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3077555101 0 0) 0x7f0d90003260 con 0x7f0db00ad0e0 2022-07-20T03:06:55.320 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dae59c700 1 -- 172.21.15.117:0/1405618606 >> v1:172.21.15.117:6789/0 conn(0x7f0db00ad0e0 legacy=0x7f0db01418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.321 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dae59c700 1 -- 172.21.15.117:0/1405618606 >> v1:172.21.15.61:6789/0 conn(0x7f0db00b3cb0 legacy=0x7f0db0145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.321 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dae59c700 1 -- 172.21.15.117:0/1405618606 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db00ae750 con 0x7f0db00b6800 2022-07-20T03:06:55.332 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.241+0000 7f0dae59c700 1 --2- 172.21.15.117:0/1405618606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b6800 0x7f0db00aebf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f0da8001490 tx=0x7f0d98006df0).ready entity=mon.0 client_cookie=b678135bb0a074f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.333 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0da80019e0 con 0x7f0db00b6800 2022-07-20T03:06:55.333 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d98000e90 con 0x7f0db00b6800 2022-07-20T03:06:55.333 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0dad59a700 1 -- 172.21.15.117:0/1405618606 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d98012460 con 0x7f0db00b6800 2022-07-20T03:06:55.341 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- 172.21.15.117:0/1405618606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b6800 msgr2=0x7f0db00aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.341 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 --2- 172.21.15.117:0/1405618606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b6800 0x7f0db00aebf0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f0da8001490 tx=0x7f0d98006df0).stop 2022-07-20T03:06:55.341 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- 172.21.15.117:0/1405618606 shutdown_connections 2022-07-20T03:06:55.342 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 --2- 172.21.15.117:0/1405618606 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b6800 0x7f0db00aebf0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.342 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- 172.21.15.117:0/1405618606 >> 172.21.15.117:0/1405618606 conn(0x7f0db00b9dd0 msgr2=0x7f0db00b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:55.342 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- 172.21.15.117:0/1405618606 shutdown_connections 2022-07-20T03:06:55.351 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- 172.21.15.117:0/1405618606 wait complete. 2022-07-20T03:06:55.352 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 Processor -- start 2022-07-20T03:06:55.352 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- start start 2022-07-20T03:06:55.364 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 0x7f0db00b7480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.365 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b3cb0 0x7f0db013fe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.365 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db00b6800 0x7f0db0140350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.366 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db00b1f80 con 0x7f0db00b3cb0 2022-07-20T03:06:55.366 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0152060 con 0x7f0db00ad0e0 2022-07-20T03:06:55.369 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.245+0000 7f0daf59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db0152300 con 0x7f0db00b6800 2022-07-20T03:06:55.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0dae59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 0x7f0db00b7480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db00b6800 0x7f0db0140350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0dae59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 0x7f0db00b7480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44398/0 (socket says 172.21.15.117:44398) 2022-07-20T03:06:55.371 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0dae59c700 1 -- 172.21.15.117:0/769453971 learned_addr learned my addr 172.21.15.117:0/769453971 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:55.371 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0dadd9b700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b3cb0 0x7f0db013fe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.371 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 -- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 msgr2=0x7f0db00b7480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.371 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 0x7f0db00b7480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.372 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 -- 172.21.15.117:0/769453971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b3cb0 msgr2=0x7f0db013fe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.372 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b3cb0 0x7f0db013fe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.372 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 -- 172.21.15.117:0/769453971 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d9800b180 con 0x7f0db00b6800 2022-07-20T03:06:55.372 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0dadd9b700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b3cb0 0x7f0db013fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:55.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0dae59c700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 0x7f0db00b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:06:55.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daed9d700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db00b6800 0x7f0db0140350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0da8001bc0 tx=0x7f0da8001980).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0da8004ab0 con 0x7f0db00b6800 2022-07-20T03:06:55.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0da8004c10 con 0x7f0db00b6800 2022-07-20T03:06:55.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0db01525a0 con 0x7f0db00b6800 2022-07-20T03:06:55.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0da8007170 con 0x7f0db00b6800 2022-07-20T03:06:55.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0db0140840 con 0x7f0db00b6800 2022-07-20T03:06:55.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0db0151ea0 con 0x7f0db00b6800 2022-07-20T03:06:55.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.249+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7f0da801b070 con 0x7f0db00b6800 2022-07-20T03:06:55.375 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.253+0000 7f0da77fe700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d94053a80 0x7f0d94055f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.375 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.253+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(6..6 src has 1..6) v4 ==== 1614+0+0 (secure 0 0 0) 0x7f0da806a3f0 con 0x7f0db00b6800 2022-07-20T03:06:55.375 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.253+0000 7f0dae59c700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d94053a80 0x7f0d94055f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.375 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.253+0000 7f0dae59c700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d94053a80 0x7f0d94055f40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0d9801a000 tx=0x7f0d98014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.376 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.261+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0da8042560 con 0x7f0db00b6800 2022-07-20T03:06:55.399 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.397+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", "id": 4} v 0) v1 -- 0x7f0db0152730 con 0x7f0db00b6800 2022-07-20T03:06:55.407 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0da77fe700 1 -- 172.21.15.117:0/769453971 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", "id": 4}]=0 v7) v1 ==== 112+0+2 (secure 0 0 0) 0x7f0da8045b80 con 0x7f0db00b6800 2022-07-20T03:06:55.408 INFO:teuthology.orchestra.run.smithi117.stdout:4 2022-07-20T03:06:55.429 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d94053a80 msgr2=0x7f0d94055f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.430 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d94053a80 0x7f0d94055f40 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f0d9801a000 tx=0x7f0d98014000).stop 2022-07-20T03:06:55.430 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db00b6800 msgr2=0x7f0db0140350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.431 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db00b6800 0x7f0db0140350 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f0da8001bc0 tx=0x7f0da8001980).stop 2022-07-20T03:06:55.431 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 shutdown_connections 2022-07-20T03:06:55.431 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d94053a80 0x7f0d94055f40 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.431 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db00b6800 0x7f0db0140350 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.432 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db00b3cb0 0x7f0db013fe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.432 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 --2- 172.21.15.117:0/769453971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db00ad0e0 0x7f0db00b7480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.432 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 >> 172.21.15.117:0/769453971 conn(0x7f0db00b9dd0 msgr2=0x7f0db0142eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:55.432 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 shutdown_connections 2022-07-20T03:06:55.433 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.405+0000 7f0daf59e700 1 -- 172.21.15.117:0/769453971 wait complete. 2022-07-20T03:06:55.433 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new ae86667a-16a2-48c1-9182-7694f9e01b8b 5 2022-07-20T03:06:55.540 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 Processor -- start 2022-07-20T03:06:55.595 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 -- start start 2022-07-20T03:06:55.596 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4cd00b9eb0 0x7f4cd005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.617 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4cd005c990 0x7f4cd0146430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.617 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4cd00b0c20 con 0x7f4cd00b5320 2022-07-20T03:06:55.627 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd00ba290 con 0x7f4cd005c990 2022-07-20T03:06:55.628 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd81ac700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4cd0142e00 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.628 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd5f48700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4cd00b9eb0 0x7f4cd005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.628 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd5747700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f4cd00b5320 0x7f4cd005c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52318/0 (socket says 172.21.15.117:52318) 2022-07-20T03:06:55.628 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd5747700 1 -- 172.21.15.117:0/3216473887 learned_addr learned my addr 172.21.15.117:0/3216473887 (peer_addr_for_me v1:172.21.15.117:0/0) 2022-07-20T03:06:55.629 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd6749700 1 --2- 172.21.15.117:0/3216473887 >> v2:172.21.15.61:3300/0 conn(0x7f4cd005c990 0x7f4cd0146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.629 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd5f48700 1 -- 172.21.15.117:0/3216473887 >> v2:172.21.15.61:3300/0 conn(0x7f4cd005c990 msgr2=0x7f4cd0146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:55.629 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd5f48700 1 --2- 172.21.15.117:0/3216473887 >> v2:172.21.15.61:3300/0 conn(0x7f4cd005c990 0x7f4cd0146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.629 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd5f48700 1 -- 172.21.15.117:0/3216473887 >> v1:172.21.15.117:6789/0 conn(0x7f4cd00b5320 legacy=0x7f4cd005c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.630 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd5f48700 1 -- 172.21.15.117:0/3216473887 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cd00afed0 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.630 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.537+0000 7f4cd4f46700 1 -- 172.21.15.117:0/3216473887 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3159836550 0 0) 0x7f4cd00b0c20 con 0x7f4cd00b5320 2022-07-20T03:06:55.630 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd5f48700 1 --2- 172.21.15.117:0/3216473887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd005bd90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4ccc002f70 tx=0x7f4ccc006df0).ready entity=mon.1 client_cookie=2fd39f8a9dd0b1e4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.630 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd4f46700 1 -- 172.21.15.117:0/3216473887 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ccc00d570 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.631 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd4f46700 1 -- 172.21.15.117:0/3216473887 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ccc000e90 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.631 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd4f46700 1 -- 172.21.15.117:0/3216473887 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ccc00d8c0 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.631 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3216473887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 msgr2=0x7f4cd005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.631 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3216473887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd005bd90 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7f4ccc002f70 tx=0x7f4ccc006df0).stop 2022-07-20T03:06:55.632 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3216473887 shutdown_connections 2022-07-20T03:06:55.632 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3216473887 >> v2:172.21.15.61:3300/0 conn(0x7f4cd005c990 0x7f4cd0146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.632 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3216473887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd005bd90 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.632 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3216473887 >> 172.21.15.117:0/3216473887 conn(0x7f4cd0067980 msgr2=0x7f4cd00b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:55.632 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3216473887 shutdown_connections 2022-07-20T03:06:55.633 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.541+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3216473887 wait complete. 2022-07-20T03:06:55.633 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 Processor -- start 2022-07-20T03:06:55.648 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 -- start start 2022-07-20T03:06:55.648 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4cd005c990 0x7f4cd013fe70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.649 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cd00b5320 0x7f4cd0140b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.649 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd013b8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.654 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd01553a0 con 0x7f4cd00b5320 2022-07-20T03:06:55.654 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd0147900 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.655 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4cd0140420 con 0x7f4cd005c990 2022-07-20T03:06:55.704 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd5f48700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4cd005c990 0x7f4cd013fe70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.704 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd013b8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.708 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd013b8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44416/0 (socket says 172.21.15.117:44416) 2022-07-20T03:06:55.710 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd5f48700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4cd005c990 0x7f4cd013fe70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:60026/0 (socket says 172.21.15.117:60026) 2022-07-20T03:06:55.710 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 -- 172.21.15.117:0/3021267592 learned_addr learned my addr 172.21.15.117:0/3021267592 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:55.727 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd5747700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cd00b5320 0x7f4cd0140b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.729 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 -- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4cd005c990 msgr2=0x7f4cd013fe70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.733 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4cd005c990 0x7f4cd013fe70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.733 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 -- 172.21.15.117:0/3021267592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cd00b5320 msgr2=0x7f4cd0140b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.734 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cd00b5320 0x7f4cd0140b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.734 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 -- 172.21.15.117:0/3021267592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ccc00b180 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.734 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd6749700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd013b8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f4cc8006e00 tx=0x7f4cc80091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.735 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc800c640 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.735 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cc800c7a0 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.735 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4cd0147b30 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.735 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cc800cb00 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.736 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.545+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4cd013e2b0 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.736 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.549+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4cd00b0170 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.736 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.549+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 3) v1 ==== 65884+0+0 (secure 0 0 0) 0x7f4cc8022070 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.737 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.549+0000 7f4cbeffd700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cb00539d0 0x7f4cb0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:55.737 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.549+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(7..7 src has 1..7) v4 ==== 1729+0+0 (secure 0 0 0) 0x7f4cc806b230 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.737 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.549+0000 7f4cd5f48700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cb00539d0 0x7f4cb0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:55.737 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.549+0000 7f4cd5f48700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cb00539d0 0x7f4cb0055e90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ccc01a000 tx=0x7f4ccc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:55.751 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.557+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4cc8043320 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.754 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.685+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", "id": 5} v 0) v1 -- 0x7f4cd00574e0 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.776 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cbeffd700 1 -- 172.21.15.117:0/3021267592 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", "id": 5}]=0 v8) v1 ==== 112+0+2 (secure 0 0 0) 0x7f4cc8046940 con 0x7f4cd00b9eb0 2022-07-20T03:06:55.799 INFO:teuthology.orchestra.run.smithi117.stdout:5 2022-07-20T03:06:55.816 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cb00539d0 msgr2=0x7f4cb0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.816 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cb00539d0 0x7f4cb0055e90 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f4ccc01a000 tx=0x7f4ccc014000).stop 2022-07-20T03:06:55.817 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 msgr2=0x7f4cd013b8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:55.817 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd013b8e0 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f4cc8006e00 tx=0x7f4cc80091d0).stop 2022-07-20T03:06:55.825 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 shutdown_connections 2022-07-20T03:06:55.826 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cb00539d0 0x7f4cb0055e90 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.844 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4cd00b9eb0 0x7f4cd013b8e0 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.845 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4cd00b5320 0x7f4cd0140b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.845 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 --2- 172.21.15.117:0/3021267592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4cd005c990 0x7f4cd013fe70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:55.846 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 >> 172.21.15.117:0/3021267592 conn(0x7f4cd0067980 msgr2=0x7f4cd0143a30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:55.846 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 shutdown_connections 2022-07-20T03:06:55.847 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.697+0000 7f4cd81ac700 1 -- 172.21.15.117:0/3021267592 wait complete. 2022-07-20T03:06:55.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 1558d94a-b96d-4bbb-86ad-34fe05792eb8 6 2022-07-20T03:06:56.002 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.997+0000 7fef87436700 1 Processor -- start 2022-07-20T03:06:56.003 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:55.997+0000 7fef87436700 1 -- start start 2022-07-20T03:06:56.036 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fef800b9eb0 0x7fef8005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.039 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fef800b5320 0x7fef8005c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.039 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef800b0c20 con 0x7fef8005c770 2022-07-20T03:06:56.040 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef800ba290 con 0x7fef800b9eb0 2022-07-20T03:06:56.040 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef80142e00 con 0x7fef800b5320 2022-07-20T03:06:56.040 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef849d1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fef800b5320 0x7fef8005c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.041 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef859d3700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fef8005c770 0x7fef80141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.117:52334/0 (socket says 172.21.15.117:52334) 2022-07-20T03:06:56.041 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef859d3700 1 -- 172.21.15.117:0/58689809 learned_addr learned my addr 172.21.15.117:0/58689809 (peer_addr_for_me v1:172.21.15.117:0/0) 2022-07-20T03:06:56.042 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef851d2700 1 --2- 172.21.15.117:0/58689809 >> v2:172.21.15.61:3300/0 conn(0x7fef800b9eb0 0x7fef8005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.042 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef849d1700 1 -- 172.21.15.117:0/58689809 >> v2:172.21.15.61:3300/0 conn(0x7fef800b9eb0 msgr2=0x7fef8005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:56.043 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef849d1700 1 --2- 172.21.15.117:0/58689809 >> v2:172.21.15.61:3300/0 conn(0x7fef800b9eb0 0x7fef8005bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.043 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef849d1700 1 -- 172.21.15.117:0/58689809 >> v1:172.21.15.117:6789/0 conn(0x7fef8005c770 legacy=0x7fef80141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.043 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef7ffff700 1 -- 172.21.15.117:0/58689809 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1523609685 0 0) 0x7fef800b0c20 con 0x7fef8005c770 2022-07-20T03:06:56.043 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef849d1700 1 -- 172.21.15.117:0/58689809 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef800afed0 con 0x7fef800b5320 2022-07-20T03:06:56.043 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef849d1700 1 --2- 172.21.15.117:0/58689809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef800b5320 0x7fef8005c280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fef74002f70 tx=0x7fef74006df0).ready entity=mon.1 client_cookie=8e64fa0b5526876e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:56.044 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef7ffff700 1 -- 172.21.15.117:0/58689809 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7400d570 con 0x7fef800b5320 2022-07-20T03:06:56.044 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef7ffff700 1 -- 172.21.15.117:0/58689809 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef74000e90 con 0x7fef800b5320 2022-07-20T03:06:56.044 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef7ffff700 1 -- 172.21.15.117:0/58689809 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7400d8c0 con 0x7fef800b5320 2022-07-20T03:06:56.044 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- 172.21.15.117:0/58689809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef800b5320 msgr2=0x7fef8005c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.045 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 --2- 172.21.15.117:0/58689809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef800b5320 0x7fef8005c280 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7fef74002f70 tx=0x7fef74006df0).stop 2022-07-20T03:06:56.045 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- 172.21.15.117:0/58689809 shutdown_connections 2022-07-20T03:06:56.045 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 --2- 172.21.15.117:0/58689809 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef800b5320 0x7fef8005c280 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.045 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 --2- 172.21.15.117:0/58689809 >> v2:172.21.15.61:3300/0 conn(0x7fef800b9eb0 0x7fef8005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.046 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- 172.21.15.117:0/58689809 >> 172.21.15.117:0/58689809 conn(0x7fef80067980 msgr2=0x7fef800b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:56.059 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- 172.21.15.117:0/58689809 shutdown_connections 2022-07-20T03:06:56.059 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.001+0000 7fef87436700 1 -- 172.21.15.117:0/58689809 wait complete. 2022-07-20T03:06:56.059 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 Processor -- start 2022-07-20T03:06:56.059 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 -- start start 2022-07-20T03:06:56.059 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 0x7fef800aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.060 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef800b5320 0x7fef800b8880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.060 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef800b9eb0 0x7fef8013d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.060 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80145190 con 0x7fef800b5320 2022-07-20T03:06:56.060 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef80145430 con 0x7fef8005c770 2022-07-20T03:06:56.061 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef801456d0 con 0x7fef800b9eb0 2022-07-20T03:06:56.061 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 0x7fef800aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.061 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 0x7fef800aecb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44426/0 (socket says 172.21.15.117:44426) 2022-07-20T03:06:56.061 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 -- 172.21.15.117:0/1592019134 learned_addr learned my addr 172.21.15.117:0/1592019134 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:56.062 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef859d3700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef800b9eb0 0x7fef8013d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.062 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef849d1700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef800b5320 0x7fef800b8880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.062 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 -- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef800b9eb0 msgr2=0x7fef8013d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.062 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef800b9eb0 0x7fef8013d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.063 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 -- 172.21.15.117:0/1592019134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef800b5320 msgr2=0x7fef800b8880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.063 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef800b5320 0x7fef800b8880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.063 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 -- 172.21.15.117:0/1592019134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef7400b180 con 0x7fef8005c770 2022-07-20T03:06:56.064 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef851d2700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 0x7fef800aecb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fef78006e00 tx=0x7fef780091d0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:56.064 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7800c850 con 0x7fef8005c770 2022-07-20T03:06:56.064 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef7800c9b0 con 0x7fef8005c770 2022-07-20T03:06:56.064 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef80145970 con 0x7fef8005c770 2022-07-20T03:06:56.064 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef7800cd10 con 0x7fef8005c770 2022-07-20T03:06:56.065 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.005+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef800af1a0 con 0x7fef8005c770 2022-07-20T03:06:56.065 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.009+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef800b0170 con 0x7fef8005c770 2022-07-20T03:06:56.065 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.009+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fef78022070 con 0x7fef8005c770 2022-07-20T03:06:56.065 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.009+0000 7fef7e7fc700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fef60053ad0 0x7fef60055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.066 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.009+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(8..8 src has 1..8) v4 ==== 1844+0+0 (secure 0 0 0) 0x7fef7806b5f0 con 0x7fef8005c770 2022-07-20T03:06:56.066 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.009+0000 7fef849d1700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fef60053ad0 0x7fef60055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.066 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.009+0000 7fef849d1700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fef60053ad0 0x7fef60055f90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fef74018000 tx=0x7fef74014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:56.066 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.017+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fef78043530 con 0x7fef8005c770 2022-07-20T03:06:56.162 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.157+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd new", "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", "id": 6} v 0) v1 -- 0x7fef800574e0 con 0x7fef8005c770 2022-07-20T03:06:56.184 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.169+0000 7fef7e7fc700 1 -- 172.21.15.117:0/1592019134 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", "id": 6}]=0 v9) v1 ==== 112+0+2 (secure 0 0 0) 0x7fef78016070 con 0x7fef8005c770 2022-07-20T03:06:56.185 INFO:teuthology.orchestra.run.smithi117.stdout:6 2022-07-20T03:06:56.195 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fef60053ad0 msgr2=0x7fef60055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.203 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fef60053ad0 0x7fef60055f90 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fef74018000 tx=0x7fef74014000).stop 2022-07-20T03:06:56.205 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 msgr2=0x7fef800aecb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.206 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 0x7fef800aecb0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fef78006e00 tx=0x7fef780091d0).stop 2022-07-20T03:06:56.206 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 shutdown_connections 2022-07-20T03:06:56.207 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fef60053ad0 0x7fef60055f90 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.207 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef800b9eb0 0x7fef8013d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.208 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef800b5320 0x7fef800b8880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.208 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 --2- 172.21.15.117:0/1592019134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef8005c770 0x7fef800aecb0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.209 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 >> 172.21.15.117:0/1592019134 conn(0x7fef80067980 msgr2=0x7fef800b5d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:56.209 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 shutdown_connections 2022-07-20T03:06:56.210 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.173+0000 7fef87436700 1 -- 172.21.15.117:0/1592019134 wait complete. 2022-07-20T03:06:56.212 DEBUG:teuthology.orchestra.run.smithi117:> sudo ceph --cluster ceph osd new 1f056769-9505-445e-8c01-b2276cd63a2c 7 2022-07-20T03:06:56.364 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 Processor -- start 2022-07-20T03:06:56.364 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 -- start start 2022-07-20T03:06:56.365 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f97040b9eb0 0x7f970405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.365 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f97040b5320 0x7f970405c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.365 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 0x7f9704141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.365 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97040b0c20 con 0x7f97040b5320 2022-07-20T03:06:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f97040ba290 con 0x7f97040b9eb0 2022-07-20T03:06:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97095ae700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f970405cbc0 con 0x7f970405c770 2022-07-20T03:06:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.329+0000 7f97037fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 0x7f9704141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.366 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f9702ffd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f97040b9eb0 0x7f970405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.367 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 0x7f9704141d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:60048/0 (socket says 172.21.15.117:60048) 2022-07-20T03:06:56.367 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 -- 172.21.15.117:0/3390840079 learned_addr learned my addr 172.21.15.117:0/3390840079 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:56.367 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97027fc700 1 --2- 172.21.15.117:0/3390840079 >> v2:172.21.15.61:3300/0 conn(0x7f97040b5320 0x7f970405c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.367 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 -- 172.21.15.117:0/3390840079 >> v2:172.21.15.117:3300/0 conn(0x7f97040b9eb0 msgr2=0x7f970405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:56.367 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 --2- 172.21.15.117:0/3390840079 >> v2:172.21.15.117:3300/0 conn(0x7f97040b9eb0 0x7f970405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 -- 172.21.15.117:0/3390840079 >> v2:172.21.15.61:3300/0 conn(0x7f97040b5320 msgr2=0x7f970405c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:06:56.368 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 --2- 172.21.15.117:0/3390840079 >> v2:172.21.15.61:3300/0 conn(0x7f97040b5320 0x7f970405c280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.369 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 -- 172.21.15.117:0/3390840079 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9704142e00 con 0x7f970405c770 2022-07-20T03:06:56.369 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97037fe700 1 --2- 172.21.15.117:0/3390840079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 0x7f9704141d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f96f4002f70 tx=0x7f96f4006eb0).ready entity=mon.2 client_cookie=d56bfb6bf2e25ca0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f9701ffb700 1 -- 172.21.15.117:0/3390840079 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96f400d570 con 0x7f970405c770 2022-07-20T03:06:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f9701ffb700 1 -- 172.21.15.117:0/3390840079 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96f4000e90 con 0x7f970405c770 2022-07-20T03:06:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f9701ffb700 1 -- 172.21.15.117:0/3390840079 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96f400d900 con 0x7f970405c770 2022-07-20T03:06:56.370 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 -- 172.21.15.117:0/3390840079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 msgr2=0x7f9704141d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 --2- 172.21.15.117:0/3390840079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 0x7f9704141d60 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f96f4002f70 tx=0x7f96f4006eb0).stop 2022-07-20T03:06:56.373 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 -- 172.21.15.117:0/3390840079 shutdown_connections 2022-07-20T03:06:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 --2- 172.21.15.117:0/3390840079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f970405c770 0x7f9704141d60 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 --2- 172.21.15.117:0/3390840079 >> v2:172.21.15.61:3300/0 conn(0x7f97040b5320 0x7f970405c280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 --2- 172.21.15.117:0/3390840079 >> v2:172.21.15.117:3300/0 conn(0x7f97040b9eb0 0x7f970405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.374 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 -- 172.21.15.117:0/3390840079 >> 172.21.15.117:0/3390840079 conn(0x7f9704067980 msgr2=0x7f97040b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:56.375 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 -- 172.21.15.117:0/3390840079 shutdown_connections 2022-07-20T03:06:56.377 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.333+0000 7f97095ae700 1 -- 172.21.15.117:0/3390840079 wait complete. 2022-07-20T03:06:56.378 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 Processor -- start 2022-07-20T03:06:56.378 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 -- start start 2022-07-20T03:06:56.380 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f970405c770 0x7f9704151d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.380 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 0x7f97040af320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.380 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97040b9eb0 0x7f9704114550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.380 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9704114d80 con 0x7f970405c770 2022-07-20T03:06:56.419 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9704115020 con 0x7f97040b9eb0 2022-07-20T03:06:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9704155340 con 0x7f97040b5320 2022-07-20T03:06:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97037fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97040b9eb0 0x7f9704114550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 0x7f97040af320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.420 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 0x7f97040af320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.117:60054/0 (socket says 172.21.15.117:60054) 2022-07-20T03:06:56.421 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97037fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97040b9eb0 0x7f9704114550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.117:44440/0 (socket says 172.21.15.117:44440) 2022-07-20T03:06:56.421 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 -- 172.21.15.117:0/151498732 learned_addr learned my addr 172.21.15.117:0/151498732 (peer_addr_for_me v2:172.21.15.117:0/0) 2022-07-20T03:06:56.421 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f9702ffd700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f970405c770 0x7f9704151d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.421 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 -- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97040b9eb0 msgr2=0x7f9704114550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97040b9eb0 0x7f9704114550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 -- 172.21.15.117:0/151498732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f970405c770 msgr2=0x7f9704151d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f970405c770 0x7f9704151d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.422 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 -- 172.21.15.117:0/151498732 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96f400b1e0 con 0x7f97040b5320 2022-07-20T03:06:56.423 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97027fc700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 0x7f97040af320 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f96ec00c000 tx=0x7f96ec006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:56.423 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96ec011750 con 0x7f97040b5320 2022-07-20T03:06:56.423 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96ec0118b0 con 0x7f97040b5320 2022-07-20T03:06:56.423 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f97040af810 con 0x7f97040b5320 2022-07-20T03:06:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f96ec011c10 con 0x7f97040b5320 2022-07-20T03:06:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.337+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f97041557e0 con 0x7f97040b5320 2022-07-20T03:06:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.341+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f97040afed0 con 0x7f97040b5320 2022-07-20T03:06:56.424 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.341+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f96ec011d70 con 0x7f97040b5320 2022-07-20T03:06:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.341+0000 7f96fbfff700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f96e80539b0 0x7f96e8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.341+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(9..9 src has 1..9) v4 ==== 1959+0+0 (secure 0 0 0) 0x7f96ec06b270 con 0x7f97040b5320 2022-07-20T03:06:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.341+0000 7f9702ffd700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f96e80539b0 0x7f96e8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:56.425 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.341+0000 7f9702ffd700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f96e80539b0 0x7f96e8055e70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f96fc00b000 tx=0x7f96fc0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:56.426 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.349+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f96ec043180 con 0x7f97040b5320 2022-07-20T03:06:56.492 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.477+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd new", "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", "id": 7} v 0) v1 -- 0x7f97040574e0 con 0x7f97040b5320 2022-07-20T03:06:56.540 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.489+0000 7f96fbfff700 1 -- 172.21.15.117:0/151498732 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd new", "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", "id": 7}]=0 v10) v1 ==== 112+0+2 (secure 0 0 0) 0x7f96ec0467a0 con 0x7f97040b5320 2022-07-20T03:06:56.573 INFO:teuthology.orchestra.run.smithi117.stdout:7 2022-07-20T03:06:56.591 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f96e80539b0 msgr2=0x7f96e8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.592 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f96e80539b0 0x7f96e8055e70 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f96fc00b000 tx=0x7f96fc0091d0).stop 2022-07-20T03:06:56.592 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 msgr2=0x7f97040af320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:56.593 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 0x7f97040af320 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f96ec00c000 tx=0x7f96ec006df0).stop 2022-07-20T03:06:56.594 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 shutdown_connections 2022-07-20T03:06:56.594 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f96e80539b0 0x7f96e8055e70 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.594 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f97040b9eb0 0x7f9704114550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.595 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f97040b5320 0x7f97040af320 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.595 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 --2- 172.21.15.117:0/151498732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f970405c770 0x7f9704151d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:56.596 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 >> 172.21.15.117:0/151498732 conn(0x7f9704067980 msgr2=0x7f9704142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:56.596 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 shutdown_connections 2022-07-20T03:06:56.597 INFO:teuthology.orchestra.run.smithi117.stderr:2022-07-20T03:06:56.493+0000 7f97095ae700 1 -- 172.21.15.117:0/151498732 wait complete. 2022-07-20T03:06:56.608 INFO:tasks.ceph.osd.0:Restarting daemon 2022-07-20T03:06:56.609 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 0 2022-07-20T03:06:56.666 INFO:tasks.ceph.osd.0:Started 2022-07-20T03:06:56.667 INFO:tasks.ceph.osd.1:Restarting daemon 2022-07-20T03:06:56.668 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 1 2022-07-20T03:06:56.708 INFO:tasks.ceph.osd.1:Started 2022-07-20T03:06:56.709 INFO:tasks.ceph.osd.2:Restarting daemon 2022-07-20T03:06:56.710 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 2 2022-07-20T03:06:56.743 INFO:tasks.ceph.osd.2:Started 2022-07-20T03:06:56.749 INFO:tasks.ceph.osd.3:Restarting daemon 2022-07-20T03:06:56.750 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 3 2022-07-20T03:06:56.752 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:56.735+0000 7fb56c87ed80 -1 Falling back to public interface 2022-07-20T03:06:56.760 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:56.751+0000 7fb56c87ed80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:56.761 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:56.751+0000 7fb56c87ed80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.781 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:56.751+0000 7fb56c87ed80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.804 INFO:tasks.ceph.osd.3:Started 2022-07-20T03:06:56.806 INFO:tasks.ceph.osd.4:Restarting daemon 2022-07-20T03:06:56.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 4 2022-07-20T03:06:56.814 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:56.755+0000 7fb56c87ed80 -1 osd.0 0 log_to_monitors {default=true} 2022-07-20T03:06:56.815 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:56.783+0000 7f3651760d80 -1 Falling back to public interface 2022-07-20T03:06:56.816 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:56.787+0000 7fc873aacd80 -1 Falling back to public interface 2022-07-20T03:06:56.817 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:56.803+0000 7f3651760d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:56.818 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:56.803+0000 7f3651760d80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.818 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:56.803+0000 7f3651760d80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.820 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:56.803+0000 7fc873aacd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:56.820 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:56.803+0000 7fc873aacd80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.821 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:56.803+0000 7fc873aacd80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.823 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:56.811+0000 7f3651760d80 -1 osd.1 0 log_to_monitors {default=true} 2022-07-20T03:06:56.824 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:56.811+0000 7fc873aacd80 -1 osd.2 0 log_to_monitors {default=true} 2022-07-20T03:06:56.846 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:56.827+0000 7f6bea98ad80 -1 Falling back to public interface 2022-07-20T03:06:56.847 INFO:tasks.ceph.osd.4:Started 2022-07-20T03:06:56.848 INFO:tasks.ceph.osd.5:Restarting daemon 2022-07-20T03:06:56.848 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 5 2022-07-20T03:06:56.888 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:56.847+0000 7f6bea98ad80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:56.889 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:56.847+0000 7f6bea98ad80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.889 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:56.847+0000 7f6bea98ad80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.892 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:56.855+0000 7f6bea98ad80 -1 osd.3 0 log_to_monitors {default=true} 2022-07-20T03:06:56.894 INFO:tasks.ceph.osd.5:Started 2022-07-20T03:06:56.895 INFO:tasks.ceph.osd.6:Restarting daemon 2022-07-20T03:06:56.896 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 6 2022-07-20T03:06:56.899 INFO:tasks.ceph.osd.6:Started 2022-07-20T03:06:56.900 INFO:tasks.ceph.osd.7:Restarting daemon 2022-07-20T03:06:56.901 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-osd -f --cluster ceph -i 7 2022-07-20T03:06:56.918 INFO:tasks.ceph.osd.7:Started 2022-07-20T03:06:56.919 INFO:tasks.ceph:Waiting for OSDs to come up 2022-07-20T03:06:56.928 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:56.905+0000 7fb9f184ed80 -1 Falling back to public interface 2022-07-20T03:06:56.940 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:56.921+0000 7fb9f184ed80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:56.941 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:56.921+0000 7fb9f184ed80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.941 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:56.921+0000 7fb9f184ed80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:56.946 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:56.929+0000 7fb9f184ed80 -1 osd.4 0 log_to_monitors {default=true} 2022-07-20T03:06:56.974 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:56.969+0000 7f0be2b69d80 -1 Falling back to public interface 2022-07-20T03:06:56.978 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:56.973+0000 7fab31492d80 -1 Falling back to public interface 2022-07-20T03:06:56.983 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:56.977+0000 7f426873dd80 -1 Falling back to public interface 2022-07-20T03:06:57.001 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:56.985+0000 7f0be2b69d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:57.002 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:56.985+0000 7f0be2b69d80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:57.002 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:56.985+0000 7f0be2b69d80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:57.004 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:56.989+0000 7f0be2b69d80 -1 osd.6 0 log_to_monitors {default=true} 2022-07-20T03:06:57.007 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:56.993+0000 7fab31492d80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:57.007 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:56.993+0000 7fab31492d80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:57.008 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:56.993+0000 7fab31492d80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:57.008 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:56.997+0000 7fab31492d80 -1 osd.5 0 log_to_monitors {default=true} 2022-07-20T03:06:57.012 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:56.997+0000 7f426873dd80 -1 journal FileJournal::_open: disabling aio for non-block journal. Use journal_force_aio to force use of aio anyway 2022-07-20T03:06:57.012 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:56.997+0000 7f426873dd80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:57.013 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:56.997+0000 7f426873dd80 -1 journal do_read_entry(8192): bad header magic 2022-07-20T03:06:57.018 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:57.001+0000 7f426873dd80 -1 osd.7 0 log_to_monitors {default=true} 2022-07-20T03:06:57.753 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-20T03:06:57.884 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 Processor -- start 2022-07-20T03:06:57.884 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 -- start start 2022-07-20T03:06:57.885 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f54d0060bd0 0x7f54d00b1800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:57.885 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d005cd70 con 0x7f54d00ae920 2022-07-20T03:06:57.924 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54d0147070 con 0x7f54d005c990 2022-07-20T03:06:57.924 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54d00b27c0 con 0x7f54d0060bd0 2022-07-20T03:06:57.924 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54cf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f54d00ae920 0x7f54d005f780 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38518/0 (socket says 172.21.15.61:38518) 2022-07-20T03:06:57.924 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54cf7fe700 1 -- 172.21.15.61:0/1527208124 learned_addr learned my addr 172.21.15.61:0/1527208124 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:06:57.925 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d4eea700 1 --2- 172.21.15.61:0/1527208124 >> v2:172.21.15.61:3300/0 conn(0x7f54d0060bd0 0x7f54d00b1800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:57.925 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54ceffd700 1 -- 172.21.15.61:0/1527208124 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3975346332 0 0) 0x7f54d005cd70 con 0x7f54d00ae920 2022-07-20T03:06:57.925 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54ceffd700 1 -- 172.21.15.61:0/1527208124 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b0003260 con 0x7f54d00ae920 2022-07-20T03:06:57.925 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54ceffd700 1 -- 172.21.15.61:0/1527208124 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2372126058 0 0) 0x7f54b0003260 con 0x7f54d00ae920 2022-07-20T03:06:57.926 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d4eea700 1 -- 172.21.15.61:0/1527208124 >> v1:172.21.15.117:6789/0 conn(0x7f54d005c990 legacy=0x7f54d01477b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:57.926 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d4eea700 1 -- 172.21.15.61:0/1527208124 >> v1:172.21.15.61:6789/0 conn(0x7f54d00ae920 legacy=0x7f54d005f780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:57.926 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d4eea700 1 -- 172.21.15.61:0/1527208124 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54d0147310 con 0x7f54d0060bd0 2022-07-20T03:06:57.926 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d4eea700 1 --2- 172.21.15.61:0/1527208124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d0060bd0 0x7f54d00b1800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f54c8001b80 tx=0x7f54b8006df0).ready entity=mon.0 client_cookie=f7e7c2e241bc0d66 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:57.926 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54ceffd700 1 -- 172.21.15.61:0/1527208124 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54c80019e0 con 0x7f54d0060bd0 2022-07-20T03:06:57.927 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54ceffd700 1 -- 172.21.15.61:0/1527208124 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8000e90 con 0x7f54d0060bd0 2022-07-20T03:06:57.927 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54ceffd700 1 -- 172.21.15.61:0/1527208124 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54b800d4f0 con 0x7f54d0060bd0 2022-07-20T03:06:57.927 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 -- 172.21.15.61:0/1527208124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d0060bd0 msgr2=0x7f54d00b1800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:57.927 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 --2- 172.21.15.61:0/1527208124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d0060bd0 0x7f54d00b1800 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f54c8001b80 tx=0x7f54b8006df0).stop 2022-07-20T03:06:57.928 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.863+0000 7f54d694d700 1 -- 172.21.15.61:0/1527208124 shutdown_connections 2022-07-20T03:06:57.928 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 --2- 172.21.15.61:0/1527208124 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d0060bd0 0x7f54d00b1800 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:57.928 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- 172.21.15.61:0/1527208124 >> 172.21.15.61:0/1527208124 conn(0x7f54d00b6150 msgr2=0x7f54d00b6530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:57.928 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- 172.21.15.61:0/1527208124 shutdown_connections 2022-07-20T03:06:57.929 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- 172.21.15.61:0/1527208124 wait complete. 2022-07-20T03:06:57.929 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 Processor -- start 2022-07-20T03:06:57.929 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- start start 2022-07-20T03:06:57.929 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 0x7f54d01562b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:57.929 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d0060bd0 0x7f54d0141ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:57.947 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d00ae920 0x7f54d00b5190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:57.947 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0142100 con 0x7f54d005c990 2022-07-20T03:06:57.948 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54d01423a0 con 0x7f54d0060bd0 2022-07-20T03:06:57.948 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d694d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54d0065230 con 0x7f54d00ae920 2022-07-20T03:06:57.949 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 0x7f54d01562b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:57.949 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 0x7f54d01562b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:41924/0 (socket says 172.21.15.61:41924) 2022-07-20T03:06:57.949 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 -- 172.21.15.61:0/2705426585 learned_addr learned my addr 172.21.15.61:0/2705426585 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:06:57.984 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cf7fe700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d0060bd0 0x7f54d0141ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:57.985 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54d4eea700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d00ae920 0x7f54d00b5190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:57.985 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 -- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d00ae920 msgr2=0x7f54d00b5190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:57.985 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d00ae920 0x7f54d00b5190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:57.985 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 -- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d0060bd0 msgr2=0x7f54d0141ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:57.986 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d0060bd0 0x7f54d0141ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:57.989 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 -- 172.21.15.61:0/2705426585 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b800b180 con 0x7f54d005c990 2022-07-20T03:06:57.990 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.867+0000 7f54cffff700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 0x7f54d01562b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f54d00b6ff0 tx=0x7f54c0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:57.990 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54c000ce30 con 0x7f54d005c990 2022-07-20T03:06:57.991 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c000f470 con 0x7f54d005c990 2022-07-20T03:06:57.992 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54d0065510 con 0x7f54d005c990 2022-07-20T03:06:57.992 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54c000f7d0 con 0x7f54d005c990 2022-07-20T03:06:57.992 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54d01569d0 con 0x7f54d005c990 2022-07-20T03:06:57.993 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54d00611c0 con 0x7f54d005c990 2022-07-20T03:06:57.993 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f54c0019030 con 0x7f54d005c990 2022-07-20T03:06:57.993 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cd7fa700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54b4053920 0x7f54b4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:06:57.993 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(11..11 src has 1..11) v4 ==== 2312+0+0 (secure 0 0 0) 0x7f54c006a540 con 0x7f54d005c990 2022-07-20T03:06:57.994 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.871+0000 7f54cf7fe700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54b4053920 0x7f54b4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:06:57.994 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.875+0000 7f54cf7fe700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54b4053920 0x7f54b4055de0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f54c800b000 tx=0x7f54c8000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:06:58.000 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:57.875+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54c00421e0 con 0x7f54d005c990 2022-07-20T03:06:58.020 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.003+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f54d00574e0 con 0x7f54d005c990 2022-07-20T03:06:58.021 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.003+0000 7f54cd7fa700 1 -- 172.21.15.61:0/2705426585 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v11) v1 ==== 74+0+8274 (secure 0 0 0) 0x7f54c0045800 con 0x7f54d005c990 2022-07-20T03:06:58.022 INFO:teuthology.misc.health.smithi061.stdout: 2022-07-20T03:06:58.022 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":11,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:06:57.490018+0000","last_up_change":"0.000000","last_in_change":"0.000000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":3,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"5","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":0,"in":0,"weight":0,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":0,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[]},"cluster_addrs":{"addrvec":[]},"heartbeat_back_addrs":{"addrvec":[]},"heartbeat_front_addrs":{"addrvec":[]},"public_addr":"(unrecognized address family 0)/0","cluster_addr":"(unrecognized address family 0)/0","heartbeat_back_addr":"(unrecognized address family 0)/0","heartbeat_front_addr":"(unrecognized address family 0)/0","state":["exists","new"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":0,"old_weight":0,"last_purged_snaps_scrub":"0.000000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:06:58.028 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54b4053920 msgr2=0x7f54b4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:58.029 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54b4053920 0x7f54b4055de0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f54c800b000 tx=0x7f54c8000f40).stop 2022-07-20T03:06:58.033 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 msgr2=0x7f54d01562b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:06:58.033 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 0x7f54d01562b0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f54d00b6ff0 tx=0x7f54c0007c50).stop 2022-07-20T03:06:58.033 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 shutdown_connections 2022-07-20T03:06:58.034 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54b4053920 0x7f54b4055de0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:58.034 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54d0060bd0 0x7f54d0141ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:58.035 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54d00ae920 0x7f54d00b5190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:58.035 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 --2- 172.21.15.61:0/2705426585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54d005c990 0x7f54d01562b0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:06:58.036 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 >> 172.21.15.61:0/2705426585 conn(0x7f54d00b6150 msgr2=0x7f54d00b32d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:06:58.037 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 shutdown_connections 2022-07-20T03:06:58.037 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:06:58.007+0000 7f54d694d700 1 -- 172.21.15.61:0/2705426585 wait complete. 2022-07-20T03:06:58.038 DEBUG:teuthology.misc:0 of 8 OSDs are up 2022-07-20T03:06:58.510 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:58.507+0000 7f363b826700 -1 osd.1 0 waiting for initial osdmap 2022-07-20T03:06:58.511 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:58.507+0000 7f6bd4a50700 -1 osd.3 0 waiting for initial osdmap 2022-07-20T03:06:58.512 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:58.507+0000 7fc85db72700 -1 osd.2 0 waiting for initial osdmap 2022-07-20T03:06:58.513 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:58.507+0000 7fb556944700 -1 osd.0 0 waiting for initial osdmap 2022-07-20T03:06:58.514 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:58.505+0000 7fb9db914700 -1 osd.4 0 waiting for initial osdmap 2022-07-20T03:06:58.515 INFO:tasks.ceph.osd.1.smithi061.stderr:2022-07-20T03:06:58.511+0000 7f364aca6700 -1 osd.1 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.516 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:58.509+0000 7f4252803700 -1 osd.7 0 waiting for initial osdmap 2022-07-20T03:06:58.517 INFO:tasks.ceph.osd.0.smithi061.stderr:2022-07-20T03:06:58.511+0000 7fb565dc4700 -1 osd.0 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.517 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:58.509+0000 7fab1b558700 -1 osd.5 0 waiting for initial osdmap 2022-07-20T03:06:58.527 INFO:tasks.ceph.osd.4.smithi117.stderr:2022-07-20T03:06:58.509+0000 7fb9ead94700 -1 osd.4 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.529 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:58.509+0000 7f0bccc2f700 -1 osd.6 0 waiting for initial osdmap 2022-07-20T03:06:58.529 INFO:tasks.ceph.osd.6.smithi117.stderr:2022-07-20T03:06:58.513+0000 7f0bdc0af700 -1 osd.6 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.530 INFO:tasks.ceph.osd.7.smithi117.stderr:2022-07-20T03:06:58.513+0000 7f4261c83700 -1 osd.7 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.531 INFO:tasks.ceph.osd.5.smithi117.stderr:2022-07-20T03:06:58.513+0000 7fab2a9d8700 -1 osd.5 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.550 INFO:tasks.ceph.osd.3.smithi061.stderr:2022-07-20T03:06:58.535+0000 7f6be3ed0700 -1 osd.3 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:06:58.551 INFO:tasks.ceph.osd.2.smithi061.stderr:2022-07-20T03:06:58.547+0000 7fc86cff2700 -1 osd.2 12 set_numa_affinity unable to identify public interface '' numa node: (2) No such file or directory 2022-07-20T03:07:00.110 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:00.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:00.171 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:00.224 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:00.244 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:04.882 DEBUG:teuthology.orchestra.run.smithi061:> adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:05.009 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 Processor -- start 2022-07-20T03:07:05.009 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 -- start start 2022-07-20T03:07:05.010 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c0601d0 0x7f9d6c0b1470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.011 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d6c13efd0 0x7f9d6c05bdc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.011 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d6c142680 con 0x7f9d6c13f410 2022-07-20T03:07:05.012 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6c0ae910 con 0x7f9d6c13efd0 2022-07-20T03:07:05.012 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d720ae700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c0605b0 con 0x7f9d6c0601d0 2022-07-20T03:07:05.013 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9d6c13f410 0x7f9d6c142d50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38724/0 (socket says 172.21.15.61:38724) 2022-07-20T03:07:05.013 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6bfff700 1 -- 172.21.15.61:0/1506917246 learned_addr learned my addr 172.21.15.61:0/1506917246 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:05.013 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6affd700 1 --2- 172.21.15.61:0/1506917246 >> v2:172.21.15.117:3300/0 conn(0x7f9d6c13efd0 0x7f9d6c05bdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.014 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6b7fe700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c0601d0 0x7f9d6c0b1470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.015 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6a7fc700 1 -- 172.21.15.61:0/1506917246 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427066226 0 0) 0x7f9d6c142680 con 0x7f9d6c13f410 2022-07-20T03:07:05.015 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6a7fc700 1 -- 172.21.15.61:0/1506917246 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d50003260 con 0x7f9d6c13f410 2022-07-20T03:07:05.016 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6affd700 1 -- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c0601d0 msgr2=0x7f9d6c0b1470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:05.016 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6affd700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c0601d0 0x7f9d6c0b1470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.017 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6affd700 1 -- 172.21.15.61:0/1506917246 >> v1:172.21.15.61:6789/0 conn(0x7f9d6c13f410 legacy=0x7f9d6c142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.017 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6affd700 1 -- 172.21.15.61:0/1506917246 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6c1428b0 con 0x7f9d6c13efd0 2022-07-20T03:07:05.018 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6b7fe700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c0601d0 0x7f9d6c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:05.018 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6affd700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13efd0 0x7f9d6c05bdc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9d5c0032f0 tx=0x7f9d5c0091d0).ready entity=mon.1 client_cookie=e84757a4a67c4372 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:05.019 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6a7fc700 1 -- 172.21.15.61:0/1506917246 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5c00b600 con 0x7f9d6c13efd0 2022-07-20T03:07:05.019 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6a7fc700 1 -- 172.21.15.61:0/1506917246 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5c00b760 con 0x7f9d6c13efd0 2022-07-20T03:07:05.020 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.987+0000 7f9d6a7fc700 1 -- 172.21.15.61:0/1506917246 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5c00b8d0 con 0x7f9d6c13efd0 2022-07-20T03:07:05.020 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13efd0 msgr2=0x7f9d6c05bdc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.021 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13efd0 0x7f9d6c05bdc0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f9d5c0032f0 tx=0x7f9d5c0091d0).stop 2022-07-20T03:07:05.054 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- 172.21.15.61:0/1506917246 shutdown_connections 2022-07-20T03:07:05.054 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13efd0 0x7f9d6c05bdc0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.055 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 --2- 172.21.15.61:0/1506917246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c0601d0 0x7f9d6c0b1470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.055 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- 172.21.15.61:0/1506917246 >> 172.21.15.61:0/1506917246 conn(0x7f9d6c05c650 msgr2=0x7f9d6c05f780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:05.056 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- 172.21.15.61:0/1506917246 shutdown_connections 2022-07-20T03:07:05.057 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- 172.21.15.61:0/1506917246 wait complete. 2022-07-20T03:07:05.057 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 Processor -- start 2022-07-20T03:07:05.057 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- start start 2022-07-20T03:07:05.057 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 0x7f9d6c136320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.058 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c13efd0 0x7f9d6c0b44f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.059 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13f410 0x7f9d6c131370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.059 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c1356a0 con 0x7f9d6c0601d0 2022-07-20T03:07:05.059 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c135940 con 0x7f9d6c13f410 2022-07-20T03:07:05.060 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d720ae700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6c135be0 con 0x7f9d6c13efd0 2022-07-20T03:07:05.060 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 0x7f9d6c136320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.080 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 0x7f9d6c136320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42128/0 (socket says 172.21.15.61:42128) 2022-07-20T03:07:05.080 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 -- 172.21.15.61:0/825995767 learned_addr learned my addr 172.21.15.61:0/825995767 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:05.080 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6bfff700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13f410 0x7f9d6c131370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.081 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6affd700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c13efd0 0x7f9d6c0b44f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.081 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 -- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c13efd0 msgr2=0x7f9d6c0b44f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.081 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c13efd0 0x7f9d6c0b44f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.082 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 -- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13f410 msgr2=0x7f9d6c131370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.082 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13f410 0x7f9d6c131370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.082 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.991+0000 7f9d6b7fe700 1 -- 172.21.15.61:0/825995767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5400b180 con 0x7f9d6c0601d0 2022-07-20T03:07:05.083 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d6b7fe700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 0x7f9d6c136320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9d5400c000 tx=0x7f9d54006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:05.083 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5400c7e0 con 0x7f9d6c0601d0 2022-07-20T03:07:05.083 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5400c940 con 0x7f9d6c0601d0 2022-07-20T03:07:05.083 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5c009470 con 0x7f9d6c0601d0 2022-07-20T03:07:05.084 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5400cca0 con 0x7f9d6c0601d0 2022-07-20T03:07:05.084 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6c13ea30 con 0x7f9d6c0601d0 2022-07-20T03:07:05.084 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d6c0b1d90 con 0x7f9d6c0601d0 2022-07-20T03:07:05.084 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d5401c460 con 0x7f9d6c0601d0 2022-07-20T03:07:05.085 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d68ff9700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053a40 0x7f9d44055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.085 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.995+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7f9d5406c070 con 0x7f9d6c0601d0 2022-07-20T03:07:05.085 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.999+0000 7f9d6affd700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053a40 0x7f9d44055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.085 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:04.999+0000 7f9d6affd700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053a40 0x7f9d44055f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9d5c01b000 tx=0x7f9d5c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:05.086 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.003+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d540432f0 con 0x7f9d6c0601d0 2022-07-20T03:07:05.167 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.135+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d6c06b200 con 0x7f9d6c0601d0 2022-07-20T03:07:05.168 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d68ff9700 1 -- 172.21.15.61:0/825995767 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v15) v1 ==== 74+0+11853 (secure 0 0 0) 0x7f9d5400ce00 con 0x7f9d6c0601d0 2022-07-20T03:07:05.185 INFO:teuthology.misc.health.smithi061.stdout: 2022-07-20T03:07:05.186 INFO:teuthology.misc.health.smithi061.stdout:{"epoch":15,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:01.535823+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":1,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":13,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:05.188 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053a40 msgr2=0x7f9d44055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.188 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053a40 0x7f9d44055f00 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f9d5c01b000 tx=0x7f9d5c016000).stop 2022-07-20T03:07:05.188 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 msgr2=0x7f9d6c136320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.189 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 0x7f9d6c136320 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f9d5400c000 tx=0x7f9d54006df0).stop 2022-07-20T03:07:05.189 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 shutdown_connections 2022-07-20T03:07:05.189 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053a40 0x7f9d44055f00 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.189 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d6c13f410 0x7f9d6c131370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.190 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d6c13efd0 0x7f9d6c0b44f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.190 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 --2- 172.21.15.61:0/825995767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d6c0601d0 0x7f9d6c136320 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.190 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 >> 172.21.15.61:0/825995767 conn(0x7f9d6c05c650 msgr2=0x7f9d6c0b1f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:05.190 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 shutdown_connections 2022-07-20T03:07:05.191 INFO:teuthology.misc.health.smithi061.stderr:2022-07-20T03:07:05.139+0000 7f9d720ae700 1 -- 172.21.15.61:0/825995767 wait complete. 2022-07-20T03:07:05.191 DEBUG:teuthology.misc:8 of 8 OSDs are up 2022-07-20T03:07:05.196 INFO:tasks.ceph:Creating RBD pool 2022-07-20T03:07:05.197 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd pool create rbd 8 2022-07-20T03:07:05.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.303+0000 7ffbf2e18700 1 Processor -- start 2022-07-20T03:07:05.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.303+0000 7ffbf2e18700 1 -- start start 2022-07-20T03:07:05.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf2e18700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffbec0ab0c0 0x7ffbec142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf2e18700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec0ac840 con 0x7ffbec0b4d70 2022-07-20T03:07:05.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf2e18700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffbec142fb0 con 0x7ffbec0ac460 2022-07-20T03:07:05.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf2e18700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffbec143240 con 0x7ffbec0ab0c0 2022-07-20T03:07:05.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffbec0ab0c0 0x7ffbec142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffbec0ab0c0 0x7ffbec142ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42138/0 (socket says 172.21.15.61:42138) 2022-07-20T03:07:05.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 -- 172.21.15.61:0/3071993278 learned_addr learned my addr 172.21.15.61:0/3071993278 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:05.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 -- 172.21.15.61:0/3071993278 >> v1:172.21.15.117:6789/0 conn(0x7ffbec0ac460 legacy=0x7ffbec13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 -- 172.21.15.61:0/3071993278 >> v1:172.21.15.61:6789/0 conn(0x7ffbec0b4d70 legacy=0x7ffbec0b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 -- 172.21.15.61:0/3071993278 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbec1434e0 con 0x7ffbec0ab0c0 2022-07-20T03:07:05.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf1615700 1 --2- 172.21.15.61:0/3071993278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec142ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ffbe0002f70 tx=0x7ffbe0006eb0).ready entity=mon.0 client_cookie=b7afe58b9b51866e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:05.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf0e14700 1 -- 172.21.15.61:0/3071993278 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe000d660 con 0x7ffbec0ab0c0 2022-07-20T03:07:05.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf0e14700 1 -- 172.21.15.61:0/3071993278 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0000e90 con 0x7ffbec0ab0c0 2022-07-20T03:07:05.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.307+0000 7ffbf0e14700 1 -- 172.21.15.61:0/3071993278 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe000d9f0 con 0x7ffbec0ab0c0 2022-07-20T03:07:05.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 -- 172.21.15.61:0/3071993278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ab0c0 msgr2=0x7ffbec142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/3071993278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec142ac0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ffbe0002f70 tx=0x7ffbe0006eb0).stop 2022-07-20T03:07:05.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 -- 172.21.15.61:0/3071993278 shutdown_connections 2022-07-20T03:07:05.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/3071993278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec142ac0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 -- 172.21.15.61:0/3071993278 >> 172.21.15.61:0/3071993278 conn(0x7ffbec0632b0 msgr2=0x7ffbec063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:05.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 -- 172.21.15.61:0/3071993278 shutdown_connections 2022-07-20T03:07:05.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 -- 172.21.15.61:0/3071993278 wait complete. 2022-07-20T03:07:05.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 Processor -- start 2022-07-20T03:07:05.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.311+0000 7ffbf2e18700 1 -- start start 2022-07-20T03:07:05.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec145260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 0x7ffbec0641e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffbec0b4d70 0x7ffbec0646d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec142ad0 con 0x7ffbec0ac460 2022-07-20T03:07:05.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec142160 con 0x7ffbec0ab0c0 2022-07-20T03:07:05.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffbec0671c0 con 0x7ffbec0b4d70 2022-07-20T03:07:05.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 0x7ffbec0641e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 0x7ffbec0641e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42144/0 (socket says 172.21.15.61:42144) 2022-07-20T03:07:05.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 -- 172.21.15.61:0/1152989165 learned_addr learned my addr 172.21.15.61:0/1152989165 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:05.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1e16700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec145260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2617700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffbec0b4d70 0x7ffbec0646d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 -- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffbec0b4d70 msgr2=0x7ffbec0646d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffbec0b4d70 0x7ffbec0646d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 -- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec0ab0c0 msgr2=0x7ffbec145260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec145260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 -- 172.21.15.61:0/1152989165 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffbe000b1e0 con 0x7ffbec0ac460 2022-07-20T03:07:05.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2617700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffbec0b4d70 0x7ffbec0646d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:05.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf1615700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 0x7ffbec0641e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ffbec142c10 tx=0x7ffbe0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:05.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe0006b40 con 0x7ffbec0ac460 2022-07-20T03:07:05.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffbe0011c70 con 0x7ffbec0ac460 2022-07-20T03:07:05.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbec142390 con 0x7ffbec0ac460 2022-07-20T03:07:05.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbec155940 con 0x7ffbec0ac460 2022-07-20T03:07:05.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffbe001e460 con 0x7ffbec0ac460 2022-07-20T03:07:05.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.315+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbec143780 con 0x7ffbec0ac460 2022-07-20T03:07:05.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.319+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffbe0025030 con 0x7ffbec0ac460 2022-07-20T03:07:05.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.319+0000 7ffbdeffd700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffbcc053ad0 0x7ffbcc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:05.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.319+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(15..15 src has 1..15) v4 ==== 4900+0+0 (secure 0 0 0) 0x7ffbe006ce70 con 0x7ffbec0ac460 2022-07-20T03:07:05.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.319+0000 7ffbf1e16700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffbcc053ad0 0x7ffbcc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:05.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.319+0000 7ffbf1e16700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffbcc053ad0 0x7ffbcc055f90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ffbe800b000 tx=0x7ffbe8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:05.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.323+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffbe0044150 con 0x7ffbec0ac460 2022-07-20T03:07:05.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.455+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "rbd", "pg_num": 8} v 0) v1 -- 0x7ffbec0574e0 con 0x7ffbec0ac460 2022-07-20T03:07:05.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.931+0000 7ffbdeffd700 1 -- 172.21.15.61:0/1152989165 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "rbd", "pg_num": 8}]=0 pool 'rbd' created v16) v1 ==== 109+0+0 (secure 0 0 0) 0x7ffbe0023070 con 0x7ffbec0ac460 2022-07-20T03:07:05.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.935+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffbcc053ad0 msgr2=0x7ffbcc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.935+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffbcc053ad0 0x7ffbcc055f90 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7ffbe800b000 tx=0x7ffbe8009040).stop 2022-07-20T03:07:05.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 msgr2=0x7ffbec0641e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:05.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 0x7ffbec0641e0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7ffbec142c10 tx=0x7ffbe0015000).stop 2022-07-20T03:07:05.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 shutdown_connections 2022-07-20T03:07:05.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffbcc053ad0 0x7ffbcc055f90 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffbec0b4d70 0x7ffbec0646d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffbec0ac460 0x7ffbec0641e0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 --2- 172.21.15.61:0/1152989165 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffbec0ab0c0 0x7ffbec145260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:05.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 >> 172.21.15.61:0/1152989165 conn(0x7ffbec0632b0 msgr2=0x7ffbec0b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:05.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 shutdown_connections 2022-07-20T03:07:05.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:05.939+0000 7ffbf2e18700 1 -- 172.21.15.61:0/1152989165 wait complete. 2022-07-20T03:07:05.958 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'rbd' created 2022-07-20T03:07:05.958 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph osd pool application enable rbd rbd --yes-i-really-mean-it 2022-07-20T03:07:06.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b1caa700 1 Processor -- start 2022-07-20T03:07:06.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b1caa700 1 -- start start 2022-07-20T03:07:06.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b1caa700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe0ac0b5670 0x7fe0ac05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:06.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b1caa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac05ce10 con 0x7fe0ac0af030 2022-07-20T03:07:06.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b1caa700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ac0afa80 con 0x7fe0ac0b05f0 2022-07-20T03:07:06.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b1caa700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ac140c50 con 0x7fe0ac0b5670 2022-07-20T03:07:06.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b0ca8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe0ac0af030 0x7fe0ac0afee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38748/0 (socket says 172.21.15.61:38748) 2022-07-20T03:07:06.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b0ca8700 1 -- 172.21.15.61:0/852125112 learned_addr learned my addr 172.21.15.61:0/852125112 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:06.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b14a9700 1 --2- 172.21.15.61:0/852125112 >> v2:172.21.15.117:3300/0 conn(0x7fe0ac0b5670 0x7fe0ac05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:06.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0ab7fe700 1 -- 172.21.15.61:0/852125112 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4761797 0 0) 0x7fe0ac05ce10 con 0x7fe0ac0af030 2022-07-20T03:07:06.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0ab7fe700 1 -- 172.21.15.61:0/852125112 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe08c003260 con 0x7fe0ac0af030 2022-07-20T03:07:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b14a9700 1 -- 172.21.15.61:0/852125112 >> v1:172.21.15.117:6789/0 conn(0x7fe0ac0b05f0 legacy=0x7fe0ac13d650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b14a9700 1 -- 172.21.15.61:0/852125112 >> v1:172.21.15.61:6789/0 conn(0x7fe0ac0af030 legacy=0x7fe0ac0afee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b14a9700 1 -- 172.21.15.61:0/852125112 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ac0af410 con 0x7fe0ac0b5670 2022-07-20T03:07:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0b14a9700 1 --2- 172.21.15.61:0/852125112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac05bd90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe09c002f70 tx=0x7fe09c006df0).ready entity=mon.1 client_cookie=c7daa16aa5cf85c6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0ab7fe700 1 -- 172.21.15.61:0/852125112 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe09c00d610 con 0x7fe0ac0b5670 2022-07-20T03:07:06.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0ab7fe700 1 -- 172.21.15.61:0/852125112 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c000e90 con 0x7fe0ac0b5670 2022-07-20T03:07:06.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.067+0000 7fe0ab7fe700 1 -- 172.21.15.61:0/852125112 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe09c00d960 con 0x7fe0ac0b5670 2022-07-20T03:07:06.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- 172.21.15.61:0/852125112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 msgr2=0x7fe0ac05bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/852125112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac05bd90 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fe09c002f70 tx=0x7fe09c006df0).stop 2022-07-20T03:07:06.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- 172.21.15.61:0/852125112 shutdown_connections 2022-07-20T03:07:06.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/852125112 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac05bd90 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- 172.21.15.61:0/852125112 >> 172.21.15.61:0/852125112 conn(0x7fe0ac0634e0 msgr2=0x7fe0ac0b4f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:06.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- 172.21.15.61:0/852125112 shutdown_connections 2022-07-20T03:07:06.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- 172.21.15.61:0/852125112 wait complete. 2022-07-20T03:07:06.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 Processor -- start 2022-07-20T03:07:06.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- start start 2022-07-20T03:07:06.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ac0af030 0x7fe0ac132d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:06.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ac0b05f0 0x7fe0ac12ec70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:06.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac12faf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:06.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac14f760 con 0x7fe0ac0af030 2022-07-20T03:07:06.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac1323a0 con 0x7fe0ac0b5670 2022-07-20T03:07:06.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b1caa700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ac132640 con 0x7fe0ac0b05f0 2022-07-20T03:07:06.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b0ca8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ac0af030 0x7fe0ac132d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:06.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b0ca8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ac0af030 0x7fe0ac132d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42156/0 (socket says 172.21.15.61:42156) 2022-07-20T03:07:06.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b14a9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac12faf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0abfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ac0b05f0 0x7fe0ac12ec70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.071+0000 7fe0b0ca8700 1 -- 172.21.15.61:0/4131331622 learned_addr learned my addr 172.21.15.61:0/4131331622 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b14a9700 1 -- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ac0b05f0 msgr2=0x7fe0ac12ec70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b14a9700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ac0b05f0 0x7fe0ac12ec70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b14a9700 1 -- 172.21.15.61:0/4131331622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ac0af030 msgr2=0x7fe0ac132d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b14a9700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ac0af030 0x7fe0ac132d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b14a9700 1 -- 172.21.15.61:0/4131331622 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09c00b180 con 0x7fe0ac0b5670 2022-07-20T03:07:06.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b14a9700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac12faf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe09c002f70 tx=0x7fe09c014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:06.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe09c016070 con 0x7fe0ac0b5670 2022-07-20T03:07:06.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe09c00d290 con 0x7fe0ac0b5670 2022-07-20T03:07:06.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ac1328e0 con 0x7fe0ac0b5670 2022-07-20T03:07:06.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe09c013bf0 con 0x7fe0ac0b5670 2022-07-20T03:07:06.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ac151d30 con 0x7fe0ac0b5670 2022-07-20T03:07:06.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0ac132a70 con 0x7fe0ac0b5670 2022-07-20T03:07:06.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe09c00f030 con 0x7fe0ac0b5670 2022-07-20T03:07:06.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0a9ffb700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe090053ad0 0x7fe090055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:06.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(16..16 src has 1..16) v4 ==== 5242+0+0 (secure 0 0 0) 0x7fe09c06c5f0 con 0x7fe0ac0b5670 2022-07-20T03:07:06.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.075+0000 7fe0b0ca8700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe090053ad0 0x7fe090055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:06.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.079+0000 7fe0b0ca8700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe090053ad0 0x7fe090055f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe0a400d000 tx=0x7fe0a4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:06.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.087+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe09c0437f0 con 0x7fe0ac0b5670 2022-07-20T03:07:06.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.211+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe0ac0574e0 con 0x7fe0ac0b5670 2022-07-20T03:07:06.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.943+0000 7fe0a9ffb700 1 -- 172.21.15.61:0/4131331622 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool application enable", "pool": "rbd", "app": "rbd", "yes_i_really_mean_it": true}]=0 enabled application 'rbd' on pool 'rbd' v17) v1 ==== 173+0+0 (secure 0 0 0) 0x7fe09c023020 con 0x7fe0ac0b5670 2022-07-20T03:07:06.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe090053ad0 msgr2=0x7fe090055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe090053ad0 0x7fe090055f90 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fe0a400d000 tx=0x7fe0a4009040).stop 2022-07-20T03:07:06.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 msgr2=0x7fe0ac12faf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:06.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac12faf0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe09c002f70 tx=0x7fe09c014000).stop 2022-07-20T03:07:06.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 shutdown_connections 2022-07-20T03:07:06.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe090053ad0 0x7fe090055f90 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ac0b5670 0x7fe0ac12faf0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ac0b05f0 0x7fe0ac12ec70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 --2- 172.21.15.61:0/4131331622 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ac0af030 0x7fe0ac132d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:06.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 >> 172.21.15.61:0/4131331622 conn(0x7fe0ac0634e0 msgr2=0x7fe0ac13fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:06.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 shutdown_connections 2022-07-20T03:07:07.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:06.951+0000 7fe0b1caa700 1 -- 172.21.15.61:0/4131331622 wait complete. 2022-07-20T03:07:07.006 INFO:teuthology.orchestra.run.smithi061.stderr:enabled application 'rbd' on pool 'rbd' 2022-07-20T03:07:07.009 INFO:tasks.ceph:Starting mds daemons in cluster ceph... 2022-07-20T03:07:07.009 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:07:07.032 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:07:07.081 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:07:07.082 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:07:07.094 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:07:07.134 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:07:07.135 INFO:tasks.ceph:Setting up CephFS filesystem... 2022-07-20T03:07:07.136 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:07.159 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:07:07.186 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:07:07.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5137d9a700 1 Processor -- start 2022-07-20T03:07:07.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5137d9a700 1 -- start start 2022-07-20T03:07:07.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5137d9a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f51300b1e00 0x7f513014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5137d9a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f513005d480 con 0x7f513005c3f0 2022-07-20T03:07:07.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5137d9a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51300b21e0 con 0x7f51300acef0 2022-07-20T03:07:07.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5137d9a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f513005cf40 con 0x7f51300b1e00 2022-07-20T03:07:07.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135335700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f513005c3f0 0x7f51300ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38780/0 (socket says 172.21.15.61:38780) 2022-07-20T03:07:07.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135335700 1 -- 172.21.15.61:0/619292348 learned_addr learned my addr 172.21.15.61:0/619292348 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:07.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135b36700 1 --2- 172.21.15.61:0/619292348 >> v2:172.21.15.117:3300/0 conn(0x7f51300b1e00 0x7f513014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486263814 0 0) 0x7f513005d480 con 0x7f513005c3f0 2022-07-20T03:07:07.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5118003260 con 0x7f513005c3f0 2022-07-20T03:07:07.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1367090973 0 0) 0x7f51300b21e0 con 0x7f51300acef0 2022-07-20T03:07:07.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f513005d480 con 0x7f51300acef0 2022-07-20T03:07:07.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135b36700 1 -- 172.21.15.61:0/619292348 >> v1:172.21.15.117:6789/0 conn(0x7f51300acef0 legacy=0x7f5130146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135b36700 1 -- 172.21.15.61:0/619292348 >> v1:172.21.15.61:6789/0 conn(0x7f513005c3f0 legacy=0x7f51300ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135b36700 1 -- 172.21.15.61:0/619292348 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f513005d1e0 con 0x7f51300b1e00 2022-07-20T03:07:07.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5135b36700 1 --2- 172.21.15.61:0/619292348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300b1e00 0x7f513014a6d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f512c002f70 tx=0x7f512c006df0).ready entity=mon.1 client_cookie=5c258089d3a1384 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:07.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.263+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f512c00d540 con 0x7f51300b1e00 2022-07-20T03:07:07.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f512c000e90 con 0x7f51300b1e00 2022-07-20T03:07:07.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5134b34700 1 -- 172.21.15.61:0/619292348 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f512c00d890 con 0x7f51300b1e00 2022-07-20T03:07:07.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- 172.21.15.61:0/619292348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300b1e00 msgr2=0x7f513014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 --2- 172.21.15.61:0/619292348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300b1e00 0x7f513014a6d0 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f512c002f70 tx=0x7f512c006df0).stop 2022-07-20T03:07:07.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- 172.21.15.61:0/619292348 shutdown_connections 2022-07-20T03:07:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 --2- 172.21.15.61:0/619292348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300b1e00 0x7f513014a6d0 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- 172.21.15.61:0/619292348 >> 172.21.15.61:0/619292348 conn(0x7f51300611c0 msgr2=0x7f5130062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:07.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- 172.21.15.61:0/619292348 shutdown_connections 2022-07-20T03:07:07.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- 172.21.15.61:0/619292348 wait complete. 2022-07-20T03:07:07.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 Processor -- start 2022-07-20T03:07:07.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- start start 2022-07-20T03:07:07.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300acef0 0x7f5130063dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51300b1e00 0x7f5130157180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 0x7f51300b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5130156a40 con 0x7f51301493d0 2022-07-20T03:07:07.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5130156ce0 con 0x7f51300acef0 2022-07-20T03:07:07.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5137d9a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5130156760 con 0x7f51300b1e00 2022-07-20T03:07:07.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.267+0000 7f5136337700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 0x7f51300b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 0x7f51300b2ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42192/0 (socket says 172.21.15.61:42192) 2022-07-20T03:07:07.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 -- 172.21.15.61:0/2290899802 learned_addr learned my addr 172.21.15.61:0/2290899802 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:07.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5135335700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51300b1e00 0x7f5130157180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5135b36700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300acef0 0x7f5130063dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 -- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51300b1e00 msgr2=0x7f5130157180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51300b1e00 0x7f5130157180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 -- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300acef0 msgr2=0x7f5130063dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300acef0 0x7f5130063dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 -- 172.21.15.61:0/2290899802 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f512c00b180 con 0x7f51301493d0 2022-07-20T03:07:07.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5136337700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 0x7f51300b2ec0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5128001490 tx=0x7f5128007dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:07.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5128002240 con 0x7f51301493d0 2022-07-20T03:07:07.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51280023a0 con 0x7f51301493d0 2022-07-20T03:07:07.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5128001650 con 0x7f51301493d0 2022-07-20T03:07:07.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5130157670 con 0x7f51301493d0 2022-07-20T03:07:07.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f513014a670 con 0x7f51301493d0 2022-07-20T03:07:07.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51300b04b0 con 0x7f51301493d0 2022-07-20T03:07:07.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.271+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5128010030 con 0x7f51301493d0 2022-07-20T03:07:07.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.275+0000 7f511effd700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5110053a40 0x7f5110055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.275+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f512806baa0 con 0x7f51301493d0 2022-07-20T03:07:07.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.275+0000 7f5135b36700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5110053a40 0x7f5110055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.275+0000 7f5135b36700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5110053a40 0x7f5110055f00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f512c01a000 tx=0x7f512c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:07.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.283+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5128042c90 con 0x7f51301493d0 2022-07-20T03:07:07.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.403+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f51300574e0 con 0x7f51301493d0 2022-07-20T03:07:07.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f511effd700 1 -- 172.21.15.61:0/2290899802 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v17) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f51280462b0 con 0x7f51301493d0 2022-07-20T03:07:07.412 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:07.413 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:06.930126+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":2,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"17","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":0,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:07.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5110053a40 msgr2=0x7f5110055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5110053a40 0x7f5110055f00 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f512c01a000 tx=0x7f512c014000).stop 2022-07-20T03:07:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 msgr2=0x7f51300b2ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 0x7f51300b2ec0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f5128001490 tx=0x7f5128007dd0).stop 2022-07-20T03:07:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 shutdown_connections 2022-07-20T03:07:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5110053a40 0x7f5110055f00 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51301493d0 0x7f51300b2ec0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51300b1e00 0x7f5130157180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 --2- 172.21.15.61:0/2290899802 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51300acef0 0x7f5130063dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 >> 172.21.15.61:0/2290899802 conn(0x7f513005c3f0 msgr2=0x7f51300afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:07.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 shutdown_connections 2022-07-20T03:07:07.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.407+0000 7f5137d9a700 1 -- 172.21.15.61:0/2290899802 wait complete. 2022-07-20T03:07:07.459 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 32769, 'flags_names': 'hashpspool,creating', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '17', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:07:07.460 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:07.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 Processor -- start 2022-07-20T03:07:07.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 -- start start 2022-07-20T03:07:07.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f591c0b9da0 0x7f591c1468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f591c1419a0 0x7f591c141e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f591c0ba180 con 0x7f591c0b3cf0 2022-07-20T03:07:07.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c0ad5b0 con 0x7f591c1419a0 2022-07-20T03:07:07.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922e03700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f591c1461b0 con 0x7f591c0b9da0 2022-07-20T03:07:07.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922602700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f591c1419a0 0x7f591c141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922602700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f591c1419a0 0x7f591c141e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42198/0 (socket says 172.21.15.61:42198) 2022-07-20T03:07:07.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5921600700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f591c0b3cf0 0x7f591c142a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38798/0 (socket says 172.21.15.61:38798) 2022-07-20T03:07:07.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5922602700 1 -- 172.21.15.61:0/3512141581 learned_addr learned my addr 172.21.15.61:0/3512141581 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.587+0000 7f5921e01700 1 --2- 172.21.15.61:0/3512141581 >> v2:172.21.15.117:3300/0 conn(0x7f591c0b9da0 0x7f591c1468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922602700 1 -- 172.21.15.61:0/3512141581 >> v2:172.21.15.117:3300/0 conn(0x7f591c0b9da0 msgr2=0x7f591c1468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922602700 1 --2- 172.21.15.61:0/3512141581 >> v2:172.21.15.117:3300/0 conn(0x7f591c0b9da0 0x7f591c1468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922602700 1 -- 172.21.15.61:0/3512141581 >> v1:172.21.15.61:6789/0 conn(0x7f591c0b3cf0 legacy=0x7f591c142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922602700 1 -- 172.21.15.61:0/3512141581 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f591c146450 con 0x7f591c1419a0 2022-07-20T03:07:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5921e01700 1 --2- 172.21.15.61:0/3512141581 >> v2:172.21.15.117:3300/0 conn(0x7f591c0b9da0 0x7f591c1468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922602700 1 --2- 172.21.15.61:0/3512141581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c141e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f59140032f0 tx=0x7f5914009290).ready entity=mon.0 client_cookie=c552a026d1d95985 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5920dff700 1 -- 172.21.15.61:0/3512141581 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f591400b870 con 0x7f591c1419a0 2022-07-20T03:07:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5920dff700 1 -- 172.21.15.61:0/3512141581 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591400b9d0 con 0x7f591c1419a0 2022-07-20T03:07:07.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5920dff700 1 -- 172.21.15.61:0/3512141581 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f591400bb40 con 0x7f591c1419a0 2022-07-20T03:07:07.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 -- 172.21.15.61:0/3512141581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 msgr2=0x7f591c141e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 --2- 172.21.15.61:0/3512141581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c141e50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f59140032f0 tx=0x7f5914009290).stop 2022-07-20T03:07:07.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 -- 172.21.15.61:0/3512141581 shutdown_connections 2022-07-20T03:07:07.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 --2- 172.21.15.61:0/3512141581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c141e50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 --2- 172.21.15.61:0/3512141581 >> v2:172.21.15.117:3300/0 conn(0x7f591c0b9da0 0x7f591c1468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 -- 172.21.15.61:0/3512141581 >> 172.21.15.61:0/3512141581 conn(0x7f591c0678c0 msgr2=0x7f591c0b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:07.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 -- 172.21.15.61:0/3512141581 shutdown_connections 2022-07-20T03:07:07.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 -- 172.21.15.61:0/3512141581 wait complete. 2022-07-20T03:07:07.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 Processor -- start 2022-07-20T03:07:07.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.591+0000 7f5922e03700 1 -- start start 2022-07-20T03:07:07.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f591c0b3cf0 0x7f591c0af1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f591c0b9da0 0x7f591c0af6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c13d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c0b1780 con 0x7f591c1419a0 2022-07-20T03:07:07.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f591c148ff0 con 0x7f591c0b3cf0 2022-07-20T03:07:07.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f591c147a30 con 0x7f591c0b9da0 2022-07-20T03:07:07.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c13d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c13d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42206/0 (socket says 172.21.15.61:42206) 2022-07-20T03:07:07.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 -- 172.21.15.61:0/2538553270 learned_addr learned my addr 172.21.15.61:0/2538553270 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:07.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5921e01700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f591c0b3cf0 0x7f591c0af1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5921600700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f591c0b9da0 0x7f591c0af6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 -- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f591c0b9da0 msgr2=0x7f591c0af6b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f591c0b9da0 0x7f591c0af6b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 -- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f591c0b3cf0 msgr2=0x7f591c0af1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f591c0b3cf0 0x7f591c0af1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 -- 172.21.15.61:0/2538553270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5914009530 con 0x7f591c1419a0 2022-07-20T03:07:07.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922602700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c13d7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f59140032f0 tx=0x7f5914016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:07.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f591400bdb0 con 0x7f591c1419a0 2022-07-20T03:07:07.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f591401f420 con 0x7f591c1419a0 2022-07-20T03:07:07.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f591c147cd0 con 0x7f591c1419a0 2022-07-20T03:07:07.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f591401f780 con 0x7f591c1419a0 2022-07-20T03:07:07.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f591c0b1b30 con 0x7f591c1419a0 2022-07-20T03:07:07.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.595+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f591c148a60 con 0x7f591c1419a0 2022-07-20T03:07:07.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.599+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f591400f070 con 0x7f591c1419a0 2022-07-20T03:07:07.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.599+0000 7f590affd700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f58fc053ad0 0x7f58fc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:07.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.599+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(17..17 src has 1..17) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f591406c800 con 0x7f591c1419a0 2022-07-20T03:07:07.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.599+0000 7f5921e01700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f58fc053ad0 0x7f58fc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:07.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.599+0000 7f5921e01700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f58fc053ad0 0x7f58fc055f90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f591800c000 tx=0x7f5918008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:07.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.607+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f591c148a60 con 0x7f591c1419a0 2022-07-20T03:07:07.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.735+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f591c148a60 con 0x7f591c1419a0 2022-07-20T03:07:07.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.735+0000 7f590affd700 1 -- 172.21.15.61:0/2538553270 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v17) v1 ==== 110+0+10 (secure 0 0 0) 0x7f591c148a60 con 0x7f591c1419a0 2022-07-20T03:07:07.876 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:07.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.735+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f58fc053ad0 msgr2=0x7f58fc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f58fc053ad0 0x7f58fc055f90 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f591800c000 tx=0x7f5918008040).stop 2022-07-20T03:07:07.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 msgr2=0x7f591c13d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:07.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c13d7f0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f59140032f0 tx=0x7f5914016000).stop 2022-07-20T03:07:07.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 shutdown_connections 2022-07-20T03:07:07.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f58fc053ad0 0x7f58fc055f90 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f591c1419a0 0x7f591c13d7f0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f591c0b9da0 0x7f591c0af6b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 --2- 172.21.15.61:0/2538553270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f591c0b3cf0 0x7f591c0af1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:07.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 >> 172.21.15.61:0/2538553270 conn(0x7f591c0678c0 msgr2=0x7f591c0b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:07.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 shutdown_connections 2022-07-20T03:07:07.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:07.739+0000 7f5922e03700 1 -- 172.21.15.61:0/2538553270 wait complete. 2022-07-20T03:07:07.908 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:08.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 Processor -- start 2022-07-20T03:07:08.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 -- start start 2022-07-20T03:07:08.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4d100af030 0x7f4d1005c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100b5690 0x7f4d10140730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d100af410 con 0x7f4d10140c20 2022-07-20T03:07:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d10144050 con 0x7f4d100af030 2022-07-20T03:07:08.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1005be30 con 0x7f4d100b5690 2022-07-20T03:07:08.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4d10140c20 0x7f4d101296f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38808/0 (socket says 172.21.15.61:38808) 2022-07-20T03:07:08.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0ed9d700 1 -- 172.21.15.61:0/3340579002 learned_addr learned my addr 172.21.15.61:0/3340579002 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:08.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0dd9b700 1 --2- 172.21.15.61:0/3340579002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100b5690 0x7f4d10140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0e59c700 1 --2- 172.21.15.61:0/3340579002 >> v2:172.21.15.117:3300/0 conn(0x7f4d100af030 0x7f4d1005c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0d59a700 1 -- 172.21.15.61:0/3340579002 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 568523646 0 0) 0x7f4d100af410 con 0x7f4d10140c20 2022-07-20T03:07:08.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0d59a700 1 -- 172.21.15.61:0/3340579002 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4cf0003260 con 0x7f4d10140c20 2022-07-20T03:07:08.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0d59a700 1 -- 172.21.15.61:0/3340579002 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 584829631 0 0) 0x7f4cf0003260 con 0x7f4d10140c20 2022-07-20T03:07:08.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0dd9b700 1 -- 172.21.15.61:0/3340579002 >> v2:172.21.15.117:3300/0 conn(0x7f4d100af030 msgr2=0x7f4d1005c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:08.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0dd9b700 1 --2- 172.21.15.61:0/3340579002 >> v2:172.21.15.117:3300/0 conn(0x7f4d100af030 0x7f4d1005c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0dd9b700 1 -- 172.21.15.61:0/3340579002 >> v1:172.21.15.61:6789/0 conn(0x7f4d10140c20 legacy=0x7f4d101296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0dd9b700 1 -- 172.21.15.61:0/3340579002 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1005c0d0 con 0x7f4d100b5690 2022-07-20T03:07:08.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.027+0000 7f4d0dd9b700 1 --2- 172.21.15.61:0/3340579002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100b5690 0x7f4d10140730 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f4cf8002f70 tx=0x7f4cf8006df0).ready entity=mon.2 client_cookie=f70fd95086ccd254 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0d59a700 1 -- 172.21.15.61:0/3340579002 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cf800d4a0 con 0x7f4d100b5690 2022-07-20T03:07:08.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0d59a700 1 -- 172.21.15.61:0/3340579002 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4cf8000e90 con 0x7f4d100b5690 2022-07-20T03:07:08.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0d59a700 1 -- 172.21.15.61:0/3340579002 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4cf800d7f0 con 0x7f4d100b5690 2022-07-20T03:07:08.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3340579002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100b5690 msgr2=0x7f4d10140730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3340579002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100b5690 0x7f4d10140730 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f4cf8002f70 tx=0x7f4cf8006df0).stop 2022-07-20T03:07:08.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3340579002 shutdown_connections 2022-07-20T03:07:08.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3340579002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100b5690 0x7f4d10140730 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3340579002 >> v2:172.21.15.117:3300/0 conn(0x7f4d100af030 0x7f4d1005c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3340579002 >> 172.21.15.61:0/3340579002 conn(0x7f4d100634c0 msgr2=0x7f4d100b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:08.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3340579002 shutdown_connections 2022-07-20T03:07:08.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3340579002 wait complete. 2022-07-20T03:07:08.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 Processor -- start 2022-07-20T03:07:08.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- start start 2022-07-20T03:07:08.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100af030 0x7f4d100b46c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d100b5690 0x7f4d1013fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 0x7f4d101401b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d100aea40 con 0x7f4d10140c20 2022-07-20T03:07:08.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1012f400 con 0x7f4d100b5690 2022-07-20T03:07:08.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1012f6a0 con 0x7f4d100af030 2022-07-20T03:07:08.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.031+0000 7f4d0ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 0x7f4d101401b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 0x7f4d101401b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42220/0 (socket says 172.21.15.61:42220) 2022-07-20T03:07:08.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 -- 172.21.15.61:0/3665548762 learned_addr learned my addr 172.21.15.61:0/3665548762 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:08.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0e59c700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100af030 0x7f4d100b46c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0dd9b700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d100b5690 0x7f4d1013fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 -- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100af030 msgr2=0x7f4d100b46c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100af030 0x7f4d100b46c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 -- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d100b5690 msgr2=0x7f4d1013fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d100b5690 0x7f4d1013fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 -- 172.21.15.61:0/3665548762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4cf800b180 con 0x7f4d10140c20 2022-07-20T03:07:08.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0dd9b700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d100b5690 0x7f4d1013fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:08.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0ed9d700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 0x7f4d101401b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4d08001250 tx=0x7f4d08007d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d08004ab0 con 0x7f4d10140c20 2022-07-20T03:07:08.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d101406a0 con 0x7f4d10140c20 2022-07-20T03:07:08.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d10152bd0 con 0x7f4d10140c20 2022-07-20T03:07:08.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d08004c10 con 0x7f4d10140c20 2022-07-20T03:07:08.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d08004d80 con 0x7f4d10140c20 2022-07-20T03:07:08.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d100ae4d0 con 0x7f4d10140c20 2022-07-20T03:07:08.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.035+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4d08007170 con 0x7f4d10140c20 2022-07-20T03:07:08.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.039+0000 7f4d077fe700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cf40539b0 0x7f4cf4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.039+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4d0806b6e0 con 0x7f4d10140c20 2022-07-20T03:07:08.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.039+0000 7f4d0e59c700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cf40539b0 0x7f4cf4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.039+0000 7f4d0e59c700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cf40539b0 0x7f4cf4055e70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4d0000b000 tx=0x7f4d00006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.043+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4d080428d0 con 0x7f4d10140c20 2022-07-20T03:07:08.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.171+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4d10143d40 con 0x7f4d10140c20 2022-07-20T03:07:08.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.171+0000 7f4d077fe700 1 -- 172.21.15.61:0/3665548762 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v18) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4d08045ef0 con 0x7f4d10140c20 2022-07-20T03:07:08.180 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cf40539b0 msgr2=0x7f4cf4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cf40539b0 0x7f4cf4055e70 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4d0000b000 tx=0x7f4d00006df0).stop 2022-07-20T03:07:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 msgr2=0x7f4d101401b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 0x7f4d101401b0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f4d08001250 tx=0x7f4d08007d70).stop 2022-07-20T03:07:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 shutdown_connections 2022-07-20T03:07:08.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cf40539b0 0x7f4cf4055e70 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d100b5690 0x7f4d1013fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d10140c20 0x7f4d101401b0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 --2- 172.21.15.61:0/3665548762 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d100af030 0x7f4d100b46c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 >> 172.21.15.61:0/3665548762 conn(0x7f4d100634c0 msgr2=0x7f4d101294c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:08.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 shutdown_connections 2022-07-20T03:07:08.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.175+0000 7f4d0f59e700 1 -- 172.21.15.61:0/3665548762 wait complete. 2022-07-20T03:07:08.281 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:07:08.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.399+0000 7f7ebf51a700 1 Processor -- start 2022-07-20T03:07:08.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.399+0000 7f7ebf51a700 1 -- start start 2022-07-20T03:07:08.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7eb813a5b0 0x7f7eb813a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7eb8140810 0x7f7eb805d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb80b20c0 0x7f7eb80b3650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb8141ac0 con 0x7f7eb8140810 2022-07-20T03:07:08.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7eb80b4410 con 0x7f7eb813a5b0 2022-07-20T03:07:08.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb80b4170 con 0x7f7eb80b20c0 2022-07-20T03:07:08.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7eb8140810 0x7f7eb805d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7eb8140810 0x7f7eb805d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42224/0 (socket says 172.21.15.61:42224) 2022-07-20T03:07:08.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 -- 172.21.15.61:0/3927954767 learned_addr learned my addr 172.21.15.61:0/3927954767 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:08.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebed19700 1 --2- 172.21.15.61:0/3927954767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb80b20c0 0x7f7eb80b3650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebe518700 1 --2- 172.21.15.61:0/3927954767 >> v2:172.21.15.117:3300/0 conn(0x7f7eb813a5b0 0x7f7eb813a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 -- 172.21.15.61:0/3927954767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb80b20c0 msgr2=0x7f7eb80b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:08.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 --2- 172.21.15.61:0/3927954767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb80b20c0 0x7f7eb80b3650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 -- 172.21.15.61:0/3927954767 >> v2:172.21.15.117:3300/0 conn(0x7f7eb813a5b0 msgr2=0x7f7eb813a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:08.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 --2- 172.21.15.61:0/3927954767 >> v2:172.21.15.117:3300/0 conn(0x7f7eb813a5b0 0x7f7eb813a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 -- 172.21.15.61:0/3927954767 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb805bfc0 con 0x7f7eb8140810 2022-07-20T03:07:08.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebdd17700 1 --2- 172.21.15.61:0/3927954767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb8140810 0x7f7eb805d0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7eb80ae0f0 tx=0x7f7eb4009290).ready entity=mon.0 client_cookie=14034f28fa773308 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebd516700 1 -- 172.21.15.61:0/3927954767 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb400b860 con 0x7f7eb8140810 2022-07-20T03:07:08.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebd516700 1 -- 172.21.15.61:0/3927954767 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb400b9c0 con 0x7f7eb8140810 2022-07-20T03:07:08.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebd516700 1 -- 172.21.15.61:0/3927954767 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb400bb30 con 0x7f7eb8140810 2022-07-20T03:07:08.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- 172.21.15.61:0/3927954767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb8140810 msgr2=0x7f7eb805d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/3927954767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb8140810 0x7f7eb805d0c0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f7eb80ae0f0 tx=0x7f7eb4009290).stop 2022-07-20T03:07:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- 172.21.15.61:0/3927954767 shutdown_connections 2022-07-20T03:07:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/3927954767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb80b20c0 0x7f7eb80b3650 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/3927954767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb8140810 0x7f7eb805d0c0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/3927954767 >> v2:172.21.15.117:3300/0 conn(0x7f7eb813a5b0 0x7f7eb813a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- 172.21.15.61:0/3927954767 >> 172.21.15.61:0/3927954767 conn(0x7f7eb8064be0 msgr2=0x7f7eb8064fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:08.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- 172.21.15.61:0/3927954767 shutdown_connections 2022-07-20T03:07:08.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- 172.21.15.61:0/3927954767 wait complete. 2022-07-20T03:07:08.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 Processor -- start 2022-07-20T03:07:08.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.403+0000 7f7ebf51a700 1 -- start start 2022-07-20T03:07:08.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 0x7f7eb813b9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7eb813a5b0 0x7f7eb813d860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb8140810 0x7f7eb813dd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb81506f0 con 0x7f7eb80b20c0 2022-07-20T03:07:08.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb813e240 con 0x7f7eb813a5b0 2022-07-20T03:07:08.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7eb8150b20 con 0x7f7eb8140810 2022-07-20T03:07:08.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 0x7f7eb813b9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 0x7f7eb813b9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42230/0 (socket says 172.21.15.61:42230) 2022-07-20T03:07:08.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 -- 172.21.15.61:0/519066326 learned_addr learned my addr 172.21.15.61:0/519066326 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:08.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebdd17700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7eb813a5b0 0x7f7eb813d860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebed19700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb8140810 0x7f7eb813dd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 -- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb8140810 msgr2=0x7f7eb813dd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb8140810 0x7f7eb813dd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 -- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7eb813a5b0 msgr2=0x7f7eb813d860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7eb813a5b0 0x7f7eb813d860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 -- 172.21.15.61:0/519066326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb4009530 con 0x7f7eb80b20c0 2022-07-20T03:07:08.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebed19700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb8140810 0x7f7eb813dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:08.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebe518700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 0x7f7eb813b9e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7eac0030b0 tx=0x7f7eac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eac011780 con 0x7f7eb80b20c0 2022-07-20T03:07:08.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eac0118e0 con 0x7f7eb80b20c0 2022-07-20T03:07:08.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7eb80b5ea0 con 0x7f7eb80b20c0 2022-07-20T03:07:08.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eac011c40 con 0x7f7eb80b20c0 2022-07-20T03:07:08.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7eb813c3d0 con 0x7f7eb80b20c0 2022-07-20T03:07:08.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.407+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7eb80b3b40 con 0x7f7eb80b20c0 2022-07-20T03:07:08.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.411+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7eac01c460 con 0x7f7eb80b20c0 2022-07-20T03:07:08.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.411+0000 7f7eab7fe700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e980539b0 0x7f7e98055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.411+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7eac06bd10 con 0x7f7eb80b20c0 2022-07-20T03:07:08.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.411+0000 7f7ebdd17700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e980539b0 0x7f7e98055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.411+0000 7f7ebdd17700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e980539b0 0x7f7e98055e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7eb401b000 tx=0x7f7eb4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.415+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7eac042f40 con 0x7f7eb80b20c0 2022-07-20T03:07:08.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.539+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7eb80574e0 con 0x7f7eb80b20c0 2022-07-20T03:07:08.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7eab7fe700 1 -- 172.21.15.61:0/519066326 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v18) v1 ==== 84+0+143 (secure 0 0 0) 0x7f7eac046560 con 0x7f7eb80b20c0 2022-07-20T03:07:08.547 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:08.548 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:07:08.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:08.549 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:07:08.549 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:07:08.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:08.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:08.551 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:07:08.551 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:07:08.552 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:08.552 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:07:08.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e980539b0 msgr2=0x7f7e98055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e980539b0 0x7f7e98055e70 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f7eb401b000 tx=0x7f7eb4016000).stop 2022-07-20T03:07:08.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 msgr2=0x7f7eb813b9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 0x7f7eb813b9e0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f7eac0030b0 tx=0x7f7eac008040).stop 2022-07-20T03:07:08.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 shutdown_connections 2022-07-20T03:07:08.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e980539b0 0x7f7e98055e70 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7eb813a5b0 0x7f7eb813d860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7eb8140810 0x7f7eb813dd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 --2- 172.21.15.61:0/519066326 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7eb80b20c0 0x7f7eb813b9e0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 >> 172.21.15.61:0/519066326 conn(0x7f7eb8064be0 msgr2=0x7f7eb8144040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:08.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 shutdown_connections 2022-07-20T03:07:08.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.543+0000 7f7ebf51a700 1 -- 172.21.15.61:0/519066326 wait complete. 2022-07-20T03:07:08.563 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:07:08.566 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:07:08.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 Processor -- start 2022-07-20T03:07:08.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 -- start start 2022-07-20T03:07:08.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f62c809df40 0x7f62c8145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f62c8060ee0 0x7f62c809cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62c80607e0 con 0x7f62c80b4d60 2022-07-20T03:07:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c8060d20 con 0x7f62c809df40 2022-07-20T03:07:08.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62cea5f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f62c8060a80 con 0x7f62c8060ee0 2022-07-20T03:07:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62c7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f62c809df40 0x7f62c8145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.679+0000 7f62c7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f62c809df40 0x7f62c8145680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42238/0 (socket says 172.21.15.61:42238) 2022-07-20T03:07:08.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c7fff700 1 -- 172.21.15.61:0/1753780360 learned_addr learned my addr 172.21.15.61:0/1753780360 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:08.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62ccffc700 1 --2- 172.21.15.61:0/1753780360 >> v2:172.21.15.117:3300/0 conn(0x7f62c8060ee0 0x7f62c809cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c7fff700 1 -- 172.21.15.61:0/1753780360 >> v2:172.21.15.117:3300/0 conn(0x7f62c8060ee0 msgr2=0x7f62c809cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:08.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c7fff700 1 --2- 172.21.15.61:0/1753780360 >> v2:172.21.15.117:3300/0 conn(0x7f62c8060ee0 0x7f62c809cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c7fff700 1 -- 172.21.15.61:0/1753780360 >> v1:172.21.15.61:6789/0 conn(0x7f62c80b4d60 legacy=0x7f62c8145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c6ffd700 1 -- 172.21.15.61:0/1753780360 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973378318 0 0) 0x7f62c80607e0 con 0x7f62c80b4d60 2022-07-20T03:07:08.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c7fff700 1 -- 172.21.15.61:0/1753780360 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62c809e320 con 0x7f62c809df40 2022-07-20T03:07:08.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c7fff700 1 --2- 172.21.15.61:0/1753780360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c8145680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f62bc002f70 tx=0x7f62bc006e50).ready entity=mon.0 client_cookie=84a8592f7e4f69d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c6ffd700 1 -- 172.21.15.61:0/1753780360 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62bc00d610 con 0x7f62c809df40 2022-07-20T03:07:08.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c6ffd700 1 -- 172.21.15.61:0/1753780360 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62bc000e90 con 0x7f62c809df40 2022-07-20T03:07:08.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62c6ffd700 1 -- 172.21.15.61:0/1753780360 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62bc00d960 con 0x7f62c809df40 2022-07-20T03:07:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 -- 172.21.15.61:0/1753780360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 msgr2=0x7f62c8145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1753780360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c8145680 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f62bc002f70 tx=0x7f62bc006e50).stop 2022-07-20T03:07:08.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 -- 172.21.15.61:0/1753780360 shutdown_connections 2022-07-20T03:07:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1753780360 >> v2:172.21.15.117:3300/0 conn(0x7f62c8060ee0 0x7f62c809cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1753780360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c8145680 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 -- 172.21.15.61:0/1753780360 >> 172.21.15.61:0/1753780360 conn(0x7f62c80b5f30 msgr2=0x7f62c80b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:08.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 -- 172.21.15.61:0/1753780360 shutdown_connections 2022-07-20T03:07:08.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 -- 172.21.15.61:0/1753780360 wait complete. 2022-07-20T03:07:08.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 Processor -- start 2022-07-20T03:07:08.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.683+0000 7f62cea5f700 1 -- start start 2022-07-20T03:07:08.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62c8060ee0 0x7f62c814a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c809af80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62c80b4d60 0x7f62c809b470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80b2b20 con 0x7f62c809df40 2022-07-20T03:07:08.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62c8090670 con 0x7f62c80b4d60 2022-07-20T03:07:08.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62c80b3050 con 0x7f62c8060ee0 2022-07-20T03:07:08.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c809af80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c809af80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42246/0 (socket says 172.21.15.61:42246) 2022-07-20T03:07:08.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 -- 172.21.15.61:0/1477219886 learned_addr learned my addr 172.21.15.61:0/1477219886 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:08.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c7fff700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62c8060ee0 0x7f62c814a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62ccffc700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62c80b4d60 0x7f62c809b470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 -- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62c8060ee0 msgr2=0x7f62c814a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62c8060ee0 0x7f62c814a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 -- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62c80b4d60 msgr2=0x7f62c809b470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:08.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62c80b4d60 0x7f62c809b470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:08.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 -- 172.21.15.61:0/1477219886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc00b1e0 con 0x7f62c809df40 2022-07-20T03:07:08.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c77fe700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c809af80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f62b8000ed0 tx=0x7f62b8007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62b8007e90 con 0x7f62c809df40 2022-07-20T03:07:08.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62b8012400 con 0x7f62c809df40 2022-07-20T03:07:08.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62b8012760 con 0x7f62c809df40 2022-07-20T03:07:08.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62c8151f40 con 0x7f62c809df40 2022-07-20T03:07:08.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.687+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62c809b960 con 0x7f62c809df40 2022-07-20T03:07:08.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.691+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f62b8019030 con 0x7f62c809df40 2022-07-20T03:07:08.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.691+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62c8090830 con 0x7f62c809df40 2022-07-20T03:07:08.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.691+0000 7f62c57fa700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62a80539b0 0x7f62a8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:08.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.691+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(18..18 src has 1..18) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f62b806af30 con 0x7f62c809df40 2022-07-20T03:07:08.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.691+0000 7f62c7fff700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62a80539b0 0x7f62a8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:08.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.691+0000 7f62c7fff700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62a80539b0 0x7f62a8055e70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f62bc01a000 tx=0x7f62bc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:08.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.695+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62b80420e0 con 0x7f62c809df40 2022-07-20T03:07:08.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:08.823+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f62c80574e0 con 0x7f62c809df40 2022-07-20T03:07:09.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.191+0000 7f62c57fa700 1 -- 172.21.15.61:0/1477219886 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v19) v1 ==== 133+0+0 (secure 0 0 0) 0x7f62b8045700 con 0x7f62c809df40 2022-07-20T03:07:09.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62a80539b0 msgr2=0x7f62a8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:09.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62a80539b0 0x7f62a8055e70 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f62bc01a000 tx=0x7f62bc014000).stop 2022-07-20T03:07:09.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 msgr2=0x7f62c809af80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:09.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c809af80 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f62b8000ed0 tx=0x7f62b8007c20).stop 2022-07-20T03:07:09.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 shutdown_connections 2022-07-20T03:07:09.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62a80539b0 0x7f62a8055e70 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62c80b4d60 0x7f62c809b470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62c809df40 0x7f62c809af80 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 --2- 172.21.15.61:0/1477219886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62c8060ee0 0x7f62c814a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 >> 172.21.15.61:0/1477219886 conn(0x7f62c80b5f30 msgr2=0x7f62c8145630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:09.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 shutdown_connections 2022-07-20T03:07:09.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.199+0000 7f62cea5f700 1 -- 172.21.15.61:0/1477219886 wait complete. 2022-07-20T03:07:09.212 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:07:09.225 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:07:09.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 Processor -- start 2022-07-20T03:07:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 -- start start 2022-07-20T03:07:09.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f67f405bf30 0x7f67f405d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:09.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f67f4143ea0 0x7f67f4144330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:09.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f40b4230 con 0x7f67f40adfa0 2022-07-20T03:07:09.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f40677b0 con 0x7f67f405bf30 2022-07-20T03:07:09.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f9347700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f40b2950 con 0x7f67f4143ea0 2022-07-20T03:07:09.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f67f405bf30 0x7f67f405d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:09.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f67f405bf30 0x7f67f405d0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42252/0 (socket says 172.21.15.61:42252) 2022-07-20T03:07:09.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 -- 172.21.15.61:0/2874579629 learned_addr learned my addr 172.21.15.61:0/2874579629 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:09.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f8b46700 1 --2- 172.21.15.61:0/2874579629 >> v2:172.21.15.117:3300/0 conn(0x7f67f4143ea0 0x7f67f4144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:09.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 -- 172.21.15.61:0/2874579629 >> v2:172.21.15.117:3300/0 conn(0x7f67f4143ea0 msgr2=0x7f67f4144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:09.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 --2- 172.21.15.61:0/2874579629 >> v2:172.21.15.117:3300/0 conn(0x7f67f4143ea0 0x7f67f4144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 -- 172.21.15.61:0/2874579629 >> v1:172.21.15.117:6789/0 conn(0x7f67f40adfa0 legacy=0x7f67f413a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:09.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.351+0000 7f67f3fff700 1 -- 172.21.15.61:0/2874579629 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f40b3850 con 0x7f67f405bf30 2022-07-20T03:07:09.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f3fff700 1 --2- 172.21.15.61:0/2874579629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f405d0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f67dc002f70 tx=0x7f67dc006eb0).ready entity=mon.0 client_cookie=93513b82a6f15de0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:09.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f2ffd700 1 -- 172.21.15.61:0/2874579629 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67dc00d5c0 con 0x7f67f405bf30 2022-07-20T03:07:09.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f2ffd700 1 -- 172.21.15.61:0/2874579629 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc000e90 con 0x7f67f405bf30 2022-07-20T03:07:09.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f2ffd700 1 -- 172.21.15.61:0/2874579629 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67dc00d950 con 0x7f67f405bf30 2022-07-20T03:07:09.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 -- 172.21.15.61:0/2874579629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 msgr2=0x7f67f405d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:09.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 --2- 172.21.15.61:0/2874579629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f405d0d0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f67dc002f70 tx=0x7f67dc006eb0).stop 2022-07-20T03:07:09.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 -- 172.21.15.61:0/2874579629 shutdown_connections 2022-07-20T03:07:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 --2- 172.21.15.61:0/2874579629 >> v2:172.21.15.117:3300/0 conn(0x7f67f4143ea0 0x7f67f4144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 --2- 172.21.15.61:0/2874579629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f405d0d0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 -- 172.21.15.61:0/2874579629 >> 172.21.15.61:0/2874579629 conn(0x7f67f4064b90 msgr2=0x7f67f4064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:09.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 -- 172.21.15.61:0/2874579629 shutdown_connections 2022-07-20T03:07:09.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 -- 172.21.15.61:0/2874579629 wait complete. 2022-07-20T03:07:09.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 Processor -- start 2022-07-20T03:07:09.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.355+0000 7f67f9347700 1 -- start start 2022-07-20T03:07:09.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f40b0700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:09.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f40adfa0 0x7f67f405f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f4143ea0 0x7f67f413cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:09.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f414ea10 con 0x7f67f405bf30 2022-07-20T03:07:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f40b0bf0 con 0x7f67f4143ea0 2022-07-20T03:07:09.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f40afb20 con 0x7f67f40adfa0 2022-07-20T03:07:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f40b0700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f40b0700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42258/0 (socket says 172.21.15.61:42258) 2022-07-20T03:07:09.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 -- 172.21.15.61:0/2006303046 learned_addr learned my addr 172.21.15.61:0/2006303046 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f37fe700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f40adfa0 0x7f67f405f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:09.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f8b46700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f4143ea0 0x7f67f413cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 -- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f40adfa0 msgr2=0x7f67f405f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:09.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f40adfa0 0x7f67f405f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 -- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f4143ea0 msgr2=0x7f67f413cce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:09.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f4143ea0 0x7f67f413cce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 -- 172.21.15.61:0/2006303046 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67dc00b1e0 con 0x7f67f405bf30 2022-07-20T03:07:09.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f3fff700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f40b0700 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f67f40ae530 tx=0x7f67dc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:09.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67dc0116e0 con 0x7f67f405bf30 2022-07-20T03:07:09.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67dc011840 con 0x7f67f405bf30 2022-07-20T03:07:09.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f40ae610 con 0x7f67f405bf30 2022-07-20T03:07:09.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67dc011ba0 con 0x7f67f405bf30 2022-07-20T03:07:09.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.359+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f405fed0 con 0x7f67f405bf30 2022-07-20T03:07:09.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.363+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f40ae7a0 con 0x7f67f405bf30 2022-07-20T03:07:09.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.363+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f67dc025030 con 0x7f67f405bf30 2022-07-20T03:07:09.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.363+0000 7f67f17fa700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67d8053ad0 0x7f67d8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:09.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.363+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(19..19 src has 1..19) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f67dc06cd40 con 0x7f67f405bf30 2022-07-20T03:07:09.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.363+0000 7f67f37fe700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67d8053ad0 0x7f67d8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:09.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.363+0000 7f67f37fe700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67d8053ad0 0x7f67d8055f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f67e400d000 tx=0x7f67e4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:09.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.371+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67dc043da0 con 0x7f67f405bf30 2022-07-20T03:07:09.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:09.499+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f67f40af940 con 0x7f67f405bf30 2022-07-20T03:07:10.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.195+0000 7f67f17fa700 1 -- 172.21.15.61:0/2006303046 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v20) v1 ==== 125+0+0 (secure 0 0 0) 0x7f67dc023020 con 0x7f67f405bf30 2022-07-20T03:07:10.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67d8053ad0 msgr2=0x7f67d8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67d8053ad0 0x7f67d8055f90 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f67e400d000 tx=0x7f67e4007b90).stop 2022-07-20T03:07:10.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 msgr2=0x7f67f40b0700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f40b0700 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f67f40ae530 tx=0x7f67dc015000).stop 2022-07-20T03:07:10.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 shutdown_connections 2022-07-20T03:07:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67d8053ad0 0x7f67d8055f90 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.203+0000 7f67f9347700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f4143ea0 0x7f67f413cce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.207+0000 7f67f9347700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f40adfa0 0x7f67f405f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.207+0000 7f67f9347700 1 --2- 172.21.15.61:0/2006303046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f405bf30 0x7f67f40b0700 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.207+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 >> 172.21.15.61:0/2006303046 conn(0x7f67f4064b90 msgr2=0x7f67f413ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:10.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.207+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 shutdown_connections 2022-07-20T03:07:10.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.207+0000 7f67f9347700 1 -- 172.21.15.61:0/2006303046 wait complete. 2022-07-20T03:07:10.319 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:07:10.319 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:07:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 Processor -- start 2022-07-20T03:07:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 -- start start 2022-07-20T03:07:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ffb70098ae0 0x7ffb700abe60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffb700ac470 0x7ffb70061b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb700aba30 con 0x7ffb700b35c0 2022-07-20T03:07:10.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb700b39a0 con 0x7ffb700ac470 2022-07-20T03:07:10.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.527+0000 7ffb74945700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb7009c000 con 0x7ffb70098ae0 2022-07-20T03:07:10.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffb700ac470 0x7ffb70061b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffb700ac470 0x7ffb70061b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42270/0 (socket says 172.21.15.61:42270) 2022-07-20T03:07:10.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 -- 172.21.15.61:0/3701844044 learned_addr learned my addr 172.21.15.61:0/3701844044 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:10.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6e59c700 1 --2- 172.21.15.61:0/3701844044 >> v2:172.21.15.117:3300/0 conn(0x7ffb70098ae0 0x7ffb700abe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 -- 172.21.15.61:0/3701844044 >> v2:172.21.15.117:3300/0 conn(0x7ffb70098ae0 msgr2=0x7ffb700abe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:10.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 --2- 172.21.15.61:0/3701844044 >> v2:172.21.15.117:3300/0 conn(0x7ffb70098ae0 0x7ffb700abe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 -- 172.21.15.61:0/3701844044 >> v1:172.21.15.117:6789/0 conn(0x7ffb700b35c0 legacy=0x7ffb70061440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 -- 172.21.15.61:0/3701844044 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb700acbc0 con 0x7ffb700ac470 2022-07-20T03:07:10.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6e59c700 1 --2- 172.21.15.61:0/3701844044 >> v2:172.21.15.117:3300/0 conn(0x7ffb70098ae0 0x7ffb700abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:10.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6f59e700 1 --2- 172.21.15.61:0/3701844044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700ac470 0x7ffb70061b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ffb700602b0 tx=0x7ffb68009290).ready entity=mon.0 client_cookie=4143ebf91f63ea85 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:10.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6dd9b700 1 -- 172.21.15.61:0/3701844044 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6800b860 con 0x7ffb700ac470 2022-07-20T03:07:10.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6dd9b700 1 -- 172.21.15.61:0/3701844044 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6800b9c0 con 0x7ffb700ac470 2022-07-20T03:07:10.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb6dd9b700 1 -- 172.21.15.61:0/3701844044 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6800bb30 con 0x7ffb700ac470 2022-07-20T03:07:10.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 -- 172.21.15.61:0/3701844044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700ac470 msgr2=0x7ffb70061b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 --2- 172.21.15.61:0/3701844044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700ac470 0x7ffb70061b50 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7ffb700602b0 tx=0x7ffb68009290).stop 2022-07-20T03:07:10.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 -- 172.21.15.61:0/3701844044 shutdown_connections 2022-07-20T03:07:10.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 --2- 172.21.15.61:0/3701844044 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700ac470 0x7ffb70061b50 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 --2- 172.21.15.61:0/3701844044 >> v2:172.21.15.117:3300/0 conn(0x7ffb70098ae0 0x7ffb700abe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 -- 172.21.15.61:0/3701844044 >> 172.21.15.61:0/3701844044 conn(0x7ffb700b6b20 msgr2=0x7ffb700b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:10.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 -- 172.21.15.61:0/3701844044 shutdown_connections 2022-07-20T03:07:10.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.531+0000 7ffb74945700 1 -- 172.21.15.61:0/3701844044 wait complete. 2022-07-20T03:07:10.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 Processor -- start 2022-07-20T03:07:10.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 -- start start 2022-07-20T03:07:10.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb70098ae0 0x7ffb70064e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb700ac470 0x7ffb700668d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 0x7ffb70066dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb70065bb0 con 0x7ffb700b35c0 2022-07-20T03:07:10.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700af620 con 0x7ffb700ac470 2022-07-20T03:07:10.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb700aebf0 con 0x7ffb70098ae0 2022-07-20T03:07:10.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 0x7ffb70066dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 0x7ffb70066dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42276/0 (socket says 172.21.15.61:42276) 2022-07-20T03:07:10.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 -- 172.21.15.61:0/533230772 learned_addr learned my addr 172.21.15.61:0/533230772 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:10.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6e59c700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb700ac470 0x7ffb700668d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6ed9d700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb70098ae0 0x7ffb70064e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 -- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb70098ae0 msgr2=0x7ffb70064e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb70098ae0 0x7ffb70064e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 -- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb700ac470 msgr2=0x7ffb700668d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb700ac470 0x7ffb700668d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 -- 172.21.15.61:0/533230772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb68009530 con 0x7ffb700b35c0 2022-07-20T03:07:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6e59c700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb700ac470 0x7ffb700668d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb6f59e700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 0x7ffb70066dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ffb700602b0 tx=0x7ffb68016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb680032f0 con 0x7ffb700b35c0 2022-07-20T03:07:10.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb6800bda0 con 0x7ffb700b35c0 2022-07-20T03:07:10.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb6801f610 con 0x7ffb700b35c0 2022-07-20T03:07:10.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb700aee20 con 0x7ffb700b35c0 2022-07-20T03:07:10.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.535+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb700ae860 con 0x7ffb700b35c0 2022-07-20T03:07:10.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.539+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb700ad100 con 0x7ffb700b35c0 2022-07-20T03:07:10.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.539+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffb68026030 con 0x7ffb700b35c0 2022-07-20T03:07:10.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.539+0000 7ffb67fff700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb54053ad0 0x7ffb54055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.539+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(20..20 src has 1..20) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ffb6806ca20 con 0x7ffb700b35c0 2022-07-20T03:07:10.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.539+0000 7ffb6ed9d700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb54053ad0 0x7ffb54055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.539+0000 7ffb6ed9d700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb54053ad0 0x7ffb54055f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ffb6000d000 tx=0x7ffb60007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:10.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.547+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffb680438e0 con 0x7ffb700b35c0 2022-07-20T03:07:10.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.711+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ffb700574e0 con 0x7ffb700b35c0 2022-07-20T03:07:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.719+0000 7ffb67fff700 1 -- 172.21.15.61:0/533230772 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 3 and data pool 4 v4) v1 ==== 187+0+0 (secure 0 0 0) 0x7ffb680240d0 con 0x7ffb700b35c0 2022-07-20T03:07:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb54053ad0 msgr2=0x7ffb54055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb54053ad0 0x7ffb54055f90 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ffb6000d000 tx=0x7ffb60007b70).stop 2022-07-20T03:07:10.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 msgr2=0x7ffb70066dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 0x7ffb70066dc0 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7ffb700602b0 tx=0x7ffb68016000).stop 2022-07-20T03:07:10.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 shutdown_connections 2022-07-20T03:07:10.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb54053ad0 0x7ffb54055f90 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb700ac470 0x7ffb700668d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb700b35c0 0x7ffb70066dc0 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 --2- 172.21.15.61:0/533230772 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb70098ae0 0x7ffb70064e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 >> 172.21.15.61:0/533230772 conn(0x7ffb700b6b20 msgr2=0x7ffb7009af90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:10.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 shutdown_connections 2022-07-20T03:07:10.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.723+0000 7ffb74945700 1 -- 172.21.15.61:0/533230772 wait complete. 2022-07-20T03:07:10.768 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 3 and data pool 4 2022-07-20T03:07:10.768 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:10.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 Processor -- start 2022-07-20T03:07:10.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- start start 2022-07-20T03:07:10.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa65c0b5650 0x7fa65c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c05bd90 0x7fa65c142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:10.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65c0aea30 con 0x7fa65c0b1200 2022-07-20T03:07:10.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65c142970 con 0x7fa65c0b5650 2022-07-20T03:07:10.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c0aec60 con 0x7fa65c05bd90 2022-07-20T03:07:10.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa65c0b5650 0x7fa65c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa65c0b5650 0x7fa65c0b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42282/0 (socket says 172.21.15.61:42282) 2022-07-20T03:07:10.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 -- 172.21.15.61:0/1512530561 learned_addr learned my addr 172.21.15.61:0/1512530561 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662924700 1 --2- 172.21.15.61:0/1512530561 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c05bd90 0x7fa65c142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:10.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 -- 172.21.15.61:0/1512530561 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c05bd90 msgr2=0x7fa65c142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:10.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 --2- 172.21.15.61:0/1512530561 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c05bd90 0x7fa65c142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:10.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 -- 172.21.15.61:0/1512530561 >> v1:172.21.15.117:6789/0 conn(0x7fa65c0b1200 legacy=0x7fa65c05c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:10.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 -- 172.21.15.61:0/1512530561 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65c0af180 con 0x7fa65c0b5650 2022-07-20T03:07:10.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa662123700 1 --2- 172.21.15.61:0/1512530561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b5650 0x7fa65c0b5a30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa658002f70 tx=0x7fa658006eb0).ready entity=mon.0 client_cookie=346d9402ba6b0af2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:10.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa661121700 1 -- 172.21.15.61:0/1512530561 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa65800d5c0 con 0x7fa65c0b5650 2022-07-20T03:07:11.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa661121700 1 -- 172.21.15.61:0/1512530561 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa658000e90 con 0x7fa65c0b5650 2022-07-20T03:07:11.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa661121700 1 -- 172.21.15.61:0/1512530561 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa65800d950 con 0x7fa65c0b5650 2022-07-20T03:07:11.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- 172.21.15.61:0/1512530561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b5650 msgr2=0x7fa65c0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 --2- 172.21.15.61:0/1512530561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b5650 0x7fa65c0b5a30 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa658002f70 tx=0x7fa658006eb0).stop 2022-07-20T03:07:11.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- 172.21.15.61:0/1512530561 shutdown_connections 2022-07-20T03:07:11.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 --2- 172.21.15.61:0/1512530561 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c05bd90 0x7fa65c142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 --2- 172.21.15.61:0/1512530561 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b5650 0x7fa65c0b5a30 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- 172.21.15.61:0/1512530561 >> 172.21.15.61:0/1512530561 conn(0x7fa65c0634e0 msgr2=0x7fa65c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:11.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- 172.21.15.61:0/1512530561 shutdown_connections 2022-07-20T03:07:11.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.879+0000 7fa663125700 1 -- 172.21.15.61:0/1512530561 wait complete. 2022-07-20T03:07:11.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 Processor -- start 2022-07-20T03:07:11.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- start start 2022-07-20T03:07:11.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65c05bd90 0x7fa65c141670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 0x7fa65c10f590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c0b5650 0x7fa65c10fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c066760 con 0x7fa65c0b1200 2022-07-20T03:07:11.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c064a10 con 0x7fa65c05bd90 2022-07-20T03:07:11.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa65c065990 con 0x7fa65c0b5650 2022-07-20T03:07:11.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 0x7fa65c10f590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 0x7fa65c10f590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42288/0 (socket says 172.21.15.61:42288) 2022-07-20T03:07:11.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 -- 172.21.15.61:0/2354098951 learned_addr learned my addr 172.21.15.61:0/2354098951 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:11.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa662123700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65c05bd90 0x7fa65c141670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 -- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c0b5650 msgr2=0x7fa65c10fa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c0b5650 0x7fa65c10fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:07:11.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 -- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65c05bd90 msgr2=0x7fa65c141670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65c05bd90 0x7fa65c141670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 -- 172.21.15.61:0/2354098951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa65800b1e0 con 0x7fa65c0b1200 2022-07-20T03:07:11.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa662123700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65c05bd90 0x7fa65c141670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:11.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa661922700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 0x7fa65c10f590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa65c0608e0 tx=0x7fa64c007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:11.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64c00ce30 con 0x7fa65c0b1200 2022-07-20T03:07:11.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa65c066990 con 0x7fa65c0b1200 2022-07-20T03:07:11.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64c00f470 con 0x7fa65c0b1200 2022-07-20T03:07:11.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa65c141b60 con 0x7fa65c0b1200 2022-07-20T03:07:11.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64c00f7d0 con 0x7fa65c0b1200 2022-07-20T03:07:11.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.883+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa65c066b20 con 0x7fa65c0b1200 2022-07-20T03:07:11.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.887+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa64c019030 con 0x7fa65c0b1200 2022-07-20T03:07:11.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.887+0000 7fa64b7fe700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa63c053af0 0x7fa63c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.887+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa64c06b450 con 0x7fa65c0b1200 2022-07-20T03:07:11.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.887+0000 7fa662123700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa63c053af0 0x7fa63c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.887+0000 7fa662123700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa63c053af0 0x7fa63c055fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa65801a000 tx=0x7fa658015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:11.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:10.891+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa64c042310 con 0x7fa65c0b1200 2022-07-20T03:07:11.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.015+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa65c0574e0 con 0x7fa65c0b1200 2022-07-20T03:07:11.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.019+0000 7fa64b7fe700 1 -- 172.21.15.61:0/2354098951 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16099 (secure 0 0 0) 0x7fa64c045930 con 0x7fa65c0b1200 2022-07-20T03:07:11.061 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:11.062 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":21,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:10.718404+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:11.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa63c053af0 msgr2=0x7fa63c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa63c053af0 0x7fa63c055fb0 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa65801a000 tx=0x7fa658015000).stop 2022-07-20T03:07:11.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 msgr2=0x7fa65c10f590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 0x7fa65c10f590 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7fa65c0608e0 tx=0x7fa64c007c00).stop 2022-07-20T03:07:11.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 shutdown_connections 2022-07-20T03:07:11.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa63c053af0 0x7fa63c055fb0 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa65c0b5650 0x7fa65c10fa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:07:11.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65c0b1200 0x7fa65c10f590 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 --2- 172.21.15.61:0/2354098951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa65c05bd90 0x7fa65c141670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 >> 172.21.15.61:0/2354098951 conn(0x7fa65c0634e0 msgr2=0x7fa65c0afd30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:11.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 shutdown_connections 2022-07-20T03:07:11.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.023+0000 7fa663125700 1 -- 172.21.15.61:0/2354098951 wait complete. 2022-07-20T03:07:11.135 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:11.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 Processor -- start 2022-07-20T03:07:11.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 -- start start 2022-07-20T03:07:11.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180b5650 0x7f75180b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f75180af8e0 0x7f751805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7518147010 con 0x7f75180adff0 2022-07-20T03:07:11.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75181472b0 con 0x7f75180af8e0 2022-07-20T03:07:11.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751d6d9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7518146d40 con 0x7f75180b5650 2022-07-20T03:07:11.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f75177fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f75180adff0 0x7f7518147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38892/0 (socket says 172.21.15.61:38892) 2022-07-20T03:07:11.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f75177fe700 1 -- 172.21.15.61:0/3908352155 learned_addr learned my addr 172.21.15.61:0/3908352155 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:11.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f751ced8700 1 --2- 172.21.15.61:0/3908352155 >> v2:172.21.15.117:3300/0 conn(0x7f75180af8e0 0x7f751805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f7517fff700 1 --2- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180b5650 0x7f75180b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.275+0000 7f7516ffd700 1 -- 172.21.15.61:0/3908352155 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3277757413 0 0) 0x7f7518147010 con 0x7f75180adff0 2022-07-20T03:07:11.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f7516ffd700 1 -- 172.21.15.61:0/3908352155 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74f8003260 con 0x7f75180adff0 2022-07-20T03:07:11.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751ced8700 1 -- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180b5650 msgr2=0x7f75180b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751ced8700 1 --2- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180b5650 0x7f75180b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751ced8700 1 -- 172.21.15.61:0/3908352155 >> v1:172.21.15.61:6789/0 conn(0x7f75180adff0 legacy=0x7f7518147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751ced8700 1 -- 172.21.15.61:0/3908352155 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180acae0 con 0x7f75180af8e0 2022-07-20T03:07:11.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751ced8700 1 --2- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180af8e0 0x7f751805bd90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f75100032f0 tx=0x7f75100091d0).ready entity=mon.1 client_cookie=7a1e33734110bf65 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:11.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f7516ffd700 1 -- 172.21.15.61:0/3908352155 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f751000b600 con 0x7f75180af8e0 2022-07-20T03:07:11.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f7516ffd700 1 -- 172.21.15.61:0/3908352155 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f751000b760 con 0x7f75180af8e0 2022-07-20T03:07:11.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f7516ffd700 1 -- 172.21.15.61:0/3908352155 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f751000b8d0 con 0x7f75180af8e0 2022-07-20T03:07:11.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 -- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180af8e0 msgr2=0x7f751805bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 --2- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180af8e0 0x7f751805bd90 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f75100032f0 tx=0x7f75100091d0).stop 2022-07-20T03:07:11.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 -- 172.21.15.61:0/3908352155 shutdown_connections 2022-07-20T03:07:11.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 --2- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180af8e0 0x7f751805bd90 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 --2- 172.21.15.61:0/3908352155 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180b5650 0x7f75180b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 -- 172.21.15.61:0/3908352155 >> 172.21.15.61:0/3908352155 conn(0x7f75180634e0 msgr2=0x7f75180b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:11.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 -- 172.21.15.61:0/3908352155 shutdown_connections 2022-07-20T03:07:11.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.279+0000 7f751d6d9700 1 -- 172.21.15.61:0/3908352155 wait complete. 2022-07-20T03:07:11.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 Processor -- start 2022-07-20T03:07:11.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 -- start start 2022-07-20T03:07:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180adff0 0x7f75181111d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 0x7f7518144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b5650 0x7f75180648b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518110930 con 0x7f75180af8e0 2022-07-20T03:07:11.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518110bd0 con 0x7f75180b5650 2022-07-20T03:07:11.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751d6d9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7518110e70 con 0x7f75180adff0 2022-07-20T03:07:11.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 0x7f7518144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 0x7f7518144ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42302/0 (socket says 172.21.15.61:42302) 2022-07-20T03:07:11.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 -- 172.21.15.61:0/2187789567 learned_addr learned my addr 172.21.15.61:0/2187789567 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:11.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f7517fff700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180adff0 0x7f75181111d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f751ced8700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b5650 0x7f75180648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 -- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180adff0 msgr2=0x7f75181111d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180adff0 0x7f75181111d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 -- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b5650 msgr2=0x7f75180648b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b5650 0x7f75180648b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.283+0000 7f75177fe700 1 -- 172.21.15.61:0/2187789567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7510009470 con 0x7f75180af8e0 2022-07-20T03:07:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75177fe700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 0x7f7518144ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7508001290 tx=0x7f7508002850).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f750800ce80 con 0x7f75180af8e0 2022-07-20T03:07:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f75080070d0 con 0x7f75180af8e0 2022-07-20T03:07:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75181105b0 con 0x7f75180af8e0 2022-07-20T03:07:11.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f75080054e0 con 0x7f75180af8e0 2022-07-20T03:07:11.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f751810f3f0 con 0x7f75180af8e0 2022-07-20T03:07:11.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f750801a060 con 0x7f75180af8e0 2022-07-20T03:07:11.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7518159aa0 con 0x7f75180af8e0 2022-07-20T03:07:11.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75157fa700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74fc0539b0 0x7f74fc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:11.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.287+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(21..21 src has 1..21) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f750806b650 con 0x7f75180af8e0 2022-07-20T03:07:11.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.291+0000 7f7517fff700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74fc0539b0 0x7f74fc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:11.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.291+0000 7f7517fff700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74fc0539b0 0x7f74fc055e70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f750000b000 tx=0x7f7500006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:11.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.291+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f75080424a0 con 0x7f75180af8e0 2022-07-20T03:07:11.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.415+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f751805c280 con 0x7f75180af8e0 2022-07-20T03:07:11.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f75157fa700 1 -- 172.21.15.61:0/2187789567 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v21) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f7508018070 con 0x7f75180af8e0 2022-07-20T03:07:11.660 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:11.660 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":21,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:10.718404+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:11.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74fc0539b0 msgr2=0x7f74fc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74fc0539b0 0x7f74fc055e70 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f750000b000 tx=0x7f7500006df0).stop 2022-07-20T03:07:11.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 msgr2=0x7f7518144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:11.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 0x7f7518144ff0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f7508001290 tx=0x7f7508002850).stop 2022-07-20T03:07:11.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 shutdown_connections 2022-07-20T03:07:11.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74fc0539b0 0x7f74fc055e70 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b5650 0x7f75180648b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180af8e0 0x7f7518144ff0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 --2- 172.21.15.61:0/2187789567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f75180adff0 0x7f75181111d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:11.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 >> 172.21.15.61:0/2187789567 conn(0x7f75180634e0 msgr2=0x7f75180ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:11.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 shutdown_connections 2022-07-20T03:07:11.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.419+0000 7f751d6d9700 1 -- 172.21.15.61:0/2187789567 wait complete. 2022-07-20T03:07:11.693 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:07:12.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 Processor -- start 2022-07-20T03:07:12.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 -- start start 2022-07-20T03:07:12.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 0x7f1e340b2700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 0x7f1e340b2bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e34144a50 con 0x7f1e340b20e0 2022-07-20T03:07:12.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e3405d4c0 con 0x7f1e3405d0e0 2022-07-20T03:07:12.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e3a336700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3414af10 con 0x7f1e340b36b0 2022-07-20T03:07:12.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 0x7f1e340b2bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e33fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 0x7f1e340b2700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e337fe700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1e340b20e0 0x7f1e341472f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:47410/0 (socket says 172.21.15.61:47410) 2022-07-20T03:07:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e337fe700 1 -- 172.21.15.61:0/797373944 learned_addr learned my addr 172.21.15.61:0/797373944 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e33fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 0x7f1e340b2700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:51988/0 (socket says 172.21.15.61:51988) 2022-07-20T03:07:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 0x7f1e340b2bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:57216/0 (socket says 172.21.15.61:57216) 2022-07-20T03:07:12.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 -- 172.21.15.61:0/797373944 >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 msgr2=0x7f1e340b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:12.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 --2- 172.21.15.61:0/797373944 >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 0x7f1e340b2700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 -- 172.21.15.61:0/797373944 >> v1:172.21.15.117:6789/0 conn(0x7f1e340b20e0 legacy=0x7f1e341472f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 -- 172.21.15.61:0/797373944 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e3414b420 con 0x7f1e340b36b0 2022-07-20T03:07:12.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e33fff700 1 --2- 172.21.15.61:0/797373944 >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 0x7f1e340b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:12.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e388d3700 1 --2- 172.21.15.61:0/797373944 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 0x7f1e340b2bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1e2c0032f0 tx=0x7f1e2c0091d0).ready entity=mon.2 client_cookie=3b75dbc6ff891cd0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:12.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.935+0000 7f1e32ffd700 1 -- 172.21.15.61:0/797373944 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e2c00b6c0 con 0x7f1e340b36b0 2022-07-20T03:07:12.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e32ffd700 1 -- 172.21.15.61:0/797373944 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e2c00b820 con 0x7f1e340b36b0 2022-07-20T03:07:12.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e32ffd700 1 -- 172.21.15.61:0/797373944 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e2c00b990 con 0x7f1e340b36b0 2022-07-20T03:07:12.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 -- 172.21.15.61:0/797373944 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 msgr2=0x7f1e340b2bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 --2- 172.21.15.61:0/797373944 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 0x7f1e340b2bf0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f1e2c0032f0 tx=0x7f1e2c0091d0).stop 2022-07-20T03:07:12.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 -- 172.21.15.61:0/797373944 shutdown_connections 2022-07-20T03:07:12.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 --2- 172.21.15.61:0/797373944 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e340b36b0 0x7f1e340b2bf0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 --2- 172.21.15.61:0/797373944 >> v2:172.21.15.117:3300/0 conn(0x7f1e3405d0e0 0x7f1e340b2700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 -- 172.21.15.61:0/797373944 >> 172.21.15.61:0/797373944 conn(0x7f1e340b7d30 msgr2=0x7f1e340b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:12.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 -- 172.21.15.61:0/797373944 shutdown_connections 2022-07-20T03:07:12.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 -- 172.21.15.61:0/797373944 wait complete. 2022-07-20T03:07:12.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 Processor -- start 2022-07-20T03:07:12.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.939+0000 7f1e3a336700 1 -- start start 2022-07-20T03:07:12.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e3405d0e0 0x7f1e340ad3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 0x7f1e34146610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e34159820 0x7f1e341589d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e34159cd0 con 0x7f1e340b36b0 2022-07-20T03:07:12.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e34159f30 con 0x7f1e34159820 2022-07-20T03:07:12.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e3415a1d0 con 0x7f1e3405d0e0 2022-07-20T03:07:12.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 0x7f1e34146610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 0x7f1e34146610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42316/0 (socket says 172.21.15.61:42316) 2022-07-20T03:07:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 -- 172.21.15.61:0/2288351033 learned_addr learned my addr 172.21.15.61:0/2288351033 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e33fff700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e3405d0e0 0x7f1e340ad3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e388d3700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e34159820 0x7f1e341589d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 -- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e3405d0e0 msgr2=0x7f1e340ad3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e3405d0e0 0x7f1e340ad3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 -- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e34159820 msgr2=0x7f1e341589d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e34159820 0x7f1e341589d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 -- 172.21.15.61:0/2288351033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e2400b180 con 0x7f1e340b36b0 2022-07-20T03:07:12.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e337fe700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 0x7f1e34146610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1e34064670 tx=0x7f1e1c007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:12.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c0070d0 con 0x7f1e340b36b0 2022-07-20T03:07:12.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e1c007230 con 0x7f1e340b36b0 2022-07-20T03:07:12.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e1c005610 con 0x7f1e340b36b0 2022-07-20T03:07:12.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e2c009470 con 0x7f1e340b36b0 2022-07-20T03:07:12.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.943+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e34158ec0 con 0x7f1e340b36b0 2022-07-20T03:07:12.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.947+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e3414b180 con 0x7f1e340b36b0 2022-07-20T03:07:12.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.947+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1e1c010070 con 0x7f1e340b36b0 2022-07-20T03:07:12.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.947+0000 7f1e317fa700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e18053920 0x7f1e18055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.947+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(22..22 src has 1..22) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1e1c06b420 con 0x7f1e340b36b0 2022-07-20T03:07:12.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.947+0000 7f1e33fff700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e18053920 0x7f1e18055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.947+0000 7f1e33fff700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e18053920 0x7f1e18055de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f1e2400b000 tx=0x7f1e24006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:12.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:11.955+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e1c042380 con 0x7f1e340b36b0 2022-07-20T03:07:12.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.123+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f1e340574e0 con 0x7f1e340b36b0 2022-07-20T03:07:12.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.751+0000 7f1e317fa700 1 -- 172.21.15.61:0/2288351033 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v7) v1 ==== 118+0+0 (secure 0 0 0) 0x7f1e1c018070 con 0x7f1e340b36b0 2022-07-20T03:07:12.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.751+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e18053920 msgr2=0x7f1e18055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.751+0000 7f1e3a336700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e18053920 0x7f1e18055de0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f1e2400b000 tx=0x7f1e24006df0).stop 2022-07-20T03:07:12.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.751+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 msgr2=0x7f1e34146610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.751+0000 7f1e3a336700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 0x7f1e34146610 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f1e34064670 tx=0x7f1e1c007c40).stop 2022-07-20T03:07:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 shutdown_connections 2022-07-20T03:07:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e18053920 0x7f1e18055de0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e34159820 0x7f1e341589d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e340b36b0 0x7f1e34146610 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 --2- 172.21.15.61:0/2288351033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e3405d0e0 0x7f1e340ad3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 >> 172.21.15.61:0/2288351033 conn(0x7f1e340b7d30 msgr2=0x7f1e34147930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:12.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 shutdown_connections 2022-07-20T03:07:12.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.755+0000 7f1e3a336700 1 -- 172.21.15.61:0/2288351033 wait complete. 2022-07-20T03:07:12.807 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:12.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.923+0000 7f84a359e700 1 Processor -- start 2022-07-20T03:07:12.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.923+0000 7f84a359e700 1 -- start start 2022-07-20T03:07:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84a40b7b70 0x7f84a40b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a40b31e0 0x7f84a413d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84a40b7f50 con 0x7f84a4066160 2022-07-20T03:07:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84a40b2800 con 0x7f84a40b7b70 2022-07-20T03:07:12.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a40b2aa0 con 0x7f84a40b31e0 2022-07-20T03:07:12.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84a40b7b70 0x7f84a40b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84a40b7b70 0x7f84a40b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42322/0 (socket says 172.21.15.61:42322) 2022-07-20T03:07:12.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 -- 172.21.15.61:0/3517210049 learned_addr learned my addr 172.21.15.61:0/3517210049 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:12.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a1d9b700 1 --2- 172.21.15.61:0/3517210049 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a40b31e0 0x7f84a413d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:12.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 -- 172.21.15.61:0/3517210049 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a40b31e0 msgr2=0x7f84a413d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:12.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 --2- 172.21.15.61:0/3517210049 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a40b31e0 0x7f84a413d810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:12.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 -- 172.21.15.61:0/3517210049 >> v1:172.21.15.117:6789/0 conn(0x7f84a4066160 legacy=0x7f84a413dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 -- 172.21.15.61:0/3517210049 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84a40b2d40 con 0x7f84a40b7b70 2022-07-20T03:07:12.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a259c700 1 --2- 172.21.15.61:0/3517210049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b7b70 0x7f84a40b1ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f849c0032f0 tx=0x7f849c009290).ready entity=mon.0 client_cookie=5690dc7626b1bd26 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:12.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a159a700 1 -- 172.21.15.61:0/3517210049 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f849c00b8a0 con 0x7f84a40b7b70 2022-07-20T03:07:12.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a159a700 1 -- 172.21.15.61:0/3517210049 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f849c00ba00 con 0x7f84a40b7b70 2022-07-20T03:07:12.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a159a700 1 -- 172.21.15.61:0/3517210049 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f849c00bb70 con 0x7f84a40b7b70 2022-07-20T03:07:12.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- 172.21.15.61:0/3517210049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b7b70 msgr2=0x7f84a40b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:12.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 --2- 172.21.15.61:0/3517210049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b7b70 0x7f84a40b1ac0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f849c0032f0 tx=0x7f849c009290).stop 2022-07-20T03:07:13.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- 172.21.15.61:0/3517210049 shutdown_connections 2022-07-20T03:07:13.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 --2- 172.21.15.61:0/3517210049 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a40b31e0 0x7f84a413d810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 --2- 172.21.15.61:0/3517210049 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b7b70 0x7f84a40b1ac0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- 172.21.15.61:0/3517210049 >> 172.21.15.61:0/3517210049 conn(0x7f84a40656b0 msgr2=0x7f84a40b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:13.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- 172.21.15.61:0/3517210049 shutdown_connections 2022-07-20T03:07:13.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.927+0000 7f84a359e700 1 -- 172.21.15.61:0/3517210049 wait complete. 2022-07-20T03:07:13.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 Processor -- start 2022-07-20T03:07:13.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 -- start start 2022-07-20T03:07:13.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a4066160 0x7f84a4153530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 0x7f84a4111660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84a40b7b70 0x7f84a40ada70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4153d20 con 0x7f84a40b31e0 2022-07-20T03:07:13.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4153fc0 con 0x7f84a40b7b70 2022-07-20T03:07:13.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84a4154260 con 0x7f84a4066160 2022-07-20T03:07:13.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 0x7f84a4111660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 0x7f84a4111660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42330/0 (socket says 172.21.15.61:42330) 2022-07-20T03:07:13.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a259c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a4066160 0x7f84a4153530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.931+0000 7f84a1d9b700 1 -- 172.21.15.61:0/3749936781 learned_addr learned my addr 172.21.15.61:0/3749936781 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:13.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a2d9d700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84a40b7b70 0x7f84a40ada70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a1d9b700 1 -- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a4066160 msgr2=0x7f84a4153530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a1d9b700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a4066160 0x7f84a4153530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a1d9b700 1 -- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84a40b7b70 msgr2=0x7f84a40ada70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a1d9b700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84a40b7b70 0x7f84a40ada70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a1d9b700 1 -- 172.21.15.61:0/3749936781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c009530 con 0x7f84a40b31e0 2022-07-20T03:07:13.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a259c700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a4066160 0x7f84a4153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:13.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a1d9b700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 0x7f84a4111660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f848c0030b0 tx=0x7f848c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848c011780 con 0x7f84a40b31e0 2022-07-20T03:07:13.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848c0118e0 con 0x7f84a40b31e0 2022-07-20T03:07:13.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84a40af4e0 con 0x7f84a40b31e0 2022-07-20T03:07:13.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848c011c40 con 0x7f84a40b31e0 2022-07-20T03:07:13.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.935+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84a4111b50 con 0x7f84a40b31e0 2022-07-20T03:07:13.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.939+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f848c012420 con 0x7f84a40b31e0 2022-07-20T03:07:13.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.939+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84a40574e0 con 0x7f84a40b31e0 2022-07-20T03:07:13.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.939+0000 7f849b7fe700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84880539d0 0x7f8488055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.939+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f848c06bc00 con 0x7f84a40b31e0 2022-07-20T03:07:13.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.939+0000 7f84a259c700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84880539d0 0x7f8488055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.939+0000 7f84a259c700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84880539d0 0x7f8488055e90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f849c01b000 tx=0x7f849c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:12.943+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f848c071020 con 0x7f84a40b31e0 2022-07-20T03:07:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84a4068630 con 0x7f84a40b31e0 2022-07-20T03:07:13.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f849b7fe700 1 -- 172.21.15.61:0/3749936781 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f848c042b00 con 0x7f84a40b31e0 2022-07-20T03:07:13.166 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:13.167 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:12.741818+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.117:6835/725875269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":725875269},{"type":"v1","addr":"172.21.15.117:6835","nonce":725875269}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:13.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84880539d0 msgr2=0x7f8488055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84880539d0 0x7f8488055e90 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f849c01b000 tx=0x7f849c016000).stop 2022-07-20T03:07:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 msgr2=0x7f84a4111660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 0x7f84a4111660 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f848c0030b0 tx=0x7f848c008040).stop 2022-07-20T03:07:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 shutdown_connections 2022-07-20T03:07:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84880539d0 0x7f8488055e90 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84a40b7b70 0x7f84a40ada70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84a40b31e0 0x7f84a4111660 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 --2- 172.21.15.61:0/3749936781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84a4066160 0x7f84a4153530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 >> 172.21.15.61:0/3749936781 conn(0x7f84a40656b0 msgr2=0x7f84a4142830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:13.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 shutdown_connections 2022-07-20T03:07:13.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.115+0000 7f84a359e700 1 -- 172.21.15.61:0/3749936781 wait complete. 2022-07-20T03:07:13.202 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2022-07-20T03:07:13.203 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:13.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.311+0000 7f5c79ce3700 1 Processor -- start 2022-07-20T03:07:13.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.311+0000 7f5c79ce3700 1 -- start start 2022-07-20T03:07:13.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5c740b20c0 0x7f5c7405d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c740b4410 con 0x7f5c74140810 2022-07-20T03:07:13.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c74141ac0 con 0x7f5c740b20c0 2022-07-20T03:07:13.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c740b4170 con 0x7f5c7413a5b0 2022-07-20T03:07:13.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5c740b20c0 0x7f5c7405d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5c740b20c0 0x7f5c7405d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42336/0 (socket says 172.21.15.61:42336) 2022-07-20T03:07:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 -- 172.21.15.61:0/4080208295 learned_addr learned my addr 172.21.15.61:0/4080208295 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c78ce1700 1 --2- 172.21.15.61:0/4080208295 >> v2:172.21.15.117:3300/0 conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 -- 172.21.15.61:0/4080208295 >> v2:172.21.15.117:3300/0 conn(0x7f5c7413a5b0 msgr2=0x7f5c7413a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:13.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 --2- 172.21.15.61:0/4080208295 >> v2:172.21.15.117:3300/0 conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 -- 172.21.15.61:0/4080208295 >> v1:172.21.15.61:6789/0 conn(0x7f5c74140810 legacy=0x7f5c740b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c737fe700 1 -- 172.21.15.61:0/4080208295 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 281251689 0 0) 0x7f5c740b4410 con 0x7f5c74140810 2022-07-20T03:07:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 -- 172.21.15.61:0/4080208295 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c7405bfc0 con 0x7f5c740b20c0 2022-07-20T03:07:13.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c78ce1700 1 --2- 172.21.15.61:0/4080208295 >> v2:172.21.15.117:3300/0 conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c794e2700 1 --2- 172.21.15.61:0/4080208295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740b20c0 0x7f5c7405d0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5c740ae0f0 tx=0x7f5c6c009290).ready entity=mon.0 client_cookie=1664e44819fb2d6f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c737fe700 1 -- 172.21.15.61:0/4080208295 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c6c00b860 con 0x7f5c740b20c0 2022-07-20T03:07:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c737fe700 1 -- 172.21.15.61:0/4080208295 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c6c00b9c0 con 0x7f5c740b20c0 2022-07-20T03:07:13.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c737fe700 1 -- 172.21.15.61:0/4080208295 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c6c00bb30 con 0x7f5c740b20c0 2022-07-20T03:07:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- 172.21.15.61:0/4080208295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740b20c0 msgr2=0x7f5c7405d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/4080208295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740b20c0 0x7f5c7405d0c0 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f5c740ae0f0 tx=0x7f5c6c009290).stop 2022-07-20T03:07:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- 172.21.15.61:0/4080208295 shutdown_connections 2022-07-20T03:07:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/4080208295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c740b20c0 0x7f5c7405d0c0 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/4080208295 >> v2:172.21.15.117:3300/0 conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- 172.21.15.61:0/4080208295 >> 172.21.15.61:0/4080208295 conn(0x7f5c74064be0 msgr2=0x7f5c74064fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- 172.21.15.61:0/4080208295 shutdown_connections 2022-07-20T03:07:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.315+0000 7f5c79ce3700 1 -- 172.21.15.61:0/4080208295 wait complete. 2022-07-20T03:07:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 Processor -- start 2022-07-20T03:07:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 -- start start 2022-07-20T03:07:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c740b20c0 0x7f5c741461a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c74140810 0x7f5c740aec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740ae560 con 0x7f5c7413a5b0 2022-07-20T03:07:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c7413c8b0 con 0x7f5c740b20c0 2022-07-20T03:07:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c740ae790 con 0x7f5c74140810 2022-07-20T03:07:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42340/0 (socket says 172.21.15.61:42340) 2022-07-20T03:07:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 -- 172.21.15.61:0/1707390114 learned_addr learned my addr 172.21.15.61:0/1707390114 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c794e2700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c74140810 0x7f5c740aec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c78ce1700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c740b20c0 0x7f5c741461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 -- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c74140810 msgr2=0x7f5c740aec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c74140810 0x7f5c740aec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 -- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c740b20c0 msgr2=0x7f5c741461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c740b20c0 0x7f5c741461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 -- 172.21.15.61:0/1707390114 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c6c009530 con 0x7f5c7413a5b0 2022-07-20T03:07:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c73fff700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 0x7f5c7413a990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5c60000ed0 tx=0x7f5c60007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c60012400 con 0x7f5c7413a5b0 2022-07-20T03:07:13.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c60012560 con 0x7f5c7413a5b0 2022-07-20T03:07:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c740b1470 con 0x7f5c7413a5b0 2022-07-20T03:07:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c600128c0 con 0x7f5c7413a5b0 2022-07-20T03:07:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.319+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c7413b0a0 con 0x7f5c7413a5b0 2022-07-20T03:07:13.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.323+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c740574e0 con 0x7f5c7413a5b0 2022-07-20T03:07:13.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.323+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5c60019030 con 0x7f5c7413a5b0 2022-07-20T03:07:13.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.323+0000 7f5c71ffb700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c54053a40 0x7f5c54055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.323+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5c6006b280 con 0x7f5c7413a5b0 2022-07-20T03:07:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.323+0000 7f5c78ce1700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c54053a40 0x7f5c54055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.323+0000 7f5c78ce1700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c54053a40 0x7f5c54055f00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f5c6800c000 tx=0x7f5c68008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.327+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5c600420d0 con 0x7f5c7413a5b0 2022-07-20T03:07:13.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.459+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5c740699b0 con 0x7f5c7413a5b0 2022-07-20T03:07:13.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.459+0000 7f5c71ffb700 1 -- 172.21.15.61:0/1707390114 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f5c600456f0 con 0x7f5c7413a5b0 2022-07-20T03:07:13.473 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:13.482 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c54053a40 msgr2=0x7f5c54055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c54053a40 0x7f5c54055f00 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f5c6800c000 tx=0x7f5c68008040).stop 2022-07-20T03:07:13.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 msgr2=0x7f5c7413a990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 0x7f5c7413a990 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5c60000ed0 tx=0x7f5c60007c50).stop 2022-07-20T03:07:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 shutdown_connections 2022-07-20T03:07:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c54053a40 0x7f5c54055f00 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c74140810 0x7f5c740aec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c7413a5b0 0x7f5c7413a990 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 --2- 172.21.15.61:0/1707390114 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c740b20c0 0x7f5c741461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 >> 172.21.15.61:0/1707390114 conn(0x7f5c74064be0 msgr2=0x7f5c7413fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:13.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 shutdown_connections 2022-07-20T03:07:13.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.463+0000 7f5c79ce3700 1 -- 172.21.15.61:0/1707390114 wait complete. 2022-07-20T03:07:13.524 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:13.561 INFO:tasks.daemonwatchdog.daemon_watchdog:watchdog starting 2022-07-20T03:07:13.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 Processor -- start 2022-07-20T03:07:13.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- start start 2022-07-20T03:07:13.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f41280ac8a0 0x7f41280b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4128060ec0 con 0x7f41280b21f0 2022-07-20T03:07:13.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f412813fb20 con 0x7f41281438b0 2022-07-20T03:07:13.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4128143d60 con 0x7f41280ac8a0 2022-07-20T03:07:13.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412c811700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f41280b21f0 0x7f4128140260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38946/0 (socket says 172.21.15.61:38946) 2022-07-20T03:07:13.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412c811700 1 -- 172.21.15.61:0/2990480030 learned_addr learned my addr 172.21.15.61:0/2990480030 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:13.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411ffff700 1 --2- 172.21.15.61:0/2990480030 >> v2:172.21.15.117:3300/0 conn(0x7f41280ac8a0 0x7f41280b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411f7fe700 1 -- 172.21.15.61:0/2990480030 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2891621089 0 0) 0x7f4128060ec0 con 0x7f41280b21f0 2022-07-20T03:07:13.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411f7fe700 1 -- 172.21.15.61:0/2990480030 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4114003260 con 0x7f41280b21f0 2022-07-20T03:07:13.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411ffff700 1 -- 172.21.15.61:0/2990480030 >> v1:172.21.15.117:6789/0 conn(0x7f41281438b0 legacy=0x7f412813ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411ffff700 1 -- 172.21.15.61:0/2990480030 >> v1:172.21.15.61:6789/0 conn(0x7f41280b21f0 legacy=0x7f4128140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411ffff700 1 -- 172.21.15.61:0/2990480030 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f412813fdc0 con 0x7f41280ac8a0 2022-07-20T03:07:13.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411ffff700 1 --2- 172.21.15.61:0/2990480030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b25d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f41280625f0 tx=0x7f4118006df0).ready entity=mon.1 client_cookie=6bee87fac28a34dd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411f7fe700 1 -- 172.21.15.61:0/2990480030 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f411800d610 con 0x7f41280ac8a0 2022-07-20T03:07:13.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411f7fe700 1 -- 172.21.15.61:0/2990480030 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4118000e90 con 0x7f41280ac8a0 2022-07-20T03:07:13.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f411f7fe700 1 -- 172.21.15.61:0/2990480030 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f411800d960 con 0x7f41280ac8a0 2022-07-20T03:07:13.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- 172.21.15.61:0/2990480030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 msgr2=0x7f41280b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 --2- 172.21.15.61:0/2990480030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b25d0 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f41280625f0 tx=0x7f4118006df0).stop 2022-07-20T03:07:13.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- 172.21.15.61:0/2990480030 shutdown_connections 2022-07-20T03:07:13.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 --2- 172.21.15.61:0/2990480030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b25d0 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- 172.21.15.61:0/2990480030 >> 172.21.15.61:0/2990480030 conn(0x7f41280b36b0 msgr2=0x7f41280b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:13.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- 172.21.15.61:0/2990480030 shutdown_connections 2022-07-20T03:07:13.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.671+0000 7f412ea75700 1 -- 172.21.15.61:0/2990480030 wait complete. 2022-07-20T03:07:13.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 Processor -- start 2022-07-20T03:07:13.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 -- start start 2022-07-20T03:07:13.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b1180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 0x7f4128098580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41281438b0 0x7f4128140e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4128142320 con 0x7f41281438b0 2022-07-20T03:07:13.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f412809c1a0 con 0x7f41280ac8a0 2022-07-20T03:07:13.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4128142580 con 0x7f41280b21f0 2022-07-20T03:07:13.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 0x7f4128098580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412c811700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b1180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412d012700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41281438b0 0x7f4128140e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 0x7f4128098580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:57260/0 (socket says 172.21.15.61:57260) 2022-07-20T03:07:13.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 -- 172.21.15.61:0/540914203 learned_addr learned my addr 172.21.15.61:0/540914203 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:13.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412c811700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b1180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:52034/0 (socket says 172.21.15.61:52034) 2022-07-20T03:07:13.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 -- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 msgr2=0x7f41280b1180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b1180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 -- 172.21.15.61:0/540914203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41281438b0 msgr2=0x7f4128140e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41281438b0 0x7f4128140e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 -- 172.21.15.61:0/540914203 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f411800b180 con 0x7f41280b21f0 2022-07-20T03:07:13.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411ffff700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 0x7f4128098580 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f41280625f0 tx=0x7f4118014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4118013d60 con 0x7f41280b21f0 2022-07-20T03:07:13.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f41280980e0 con 0x7f41280b21f0 2022-07-20T03:07:13.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f411800d290 con 0x7f41280b21f0 2022-07-20T03:07:13.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.675+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f411801e5f0 con 0x7f41280b21f0 2022-07-20T03:07:13.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4128142b60 con 0x7f41280b21f0 2022-07-20T03:07:13.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f41280574e0 con 0x7f41280b21f0 2022-07-20T03:07:13.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f411801c070 con 0x7f41280b21f0 2022-07-20T03:07:13.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f411dffb700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4108053ad0 0x7f4108055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:13.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f411806d030 con 0x7f41280b21f0 2022-07-20T03:07:13.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f412c811700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4108053ad0 0x7f4108055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:13.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.679+0000 7f412c811700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4108053ad0 0x7f4108055f90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f412400d000 tx=0x7f4124009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:13.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.691+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4118043f90 con 0x7f41280b21f0 2022-07-20T03:07:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.819+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f412805c400 con 0x7f41280b21f0 2022-07-20T03:07:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f411dffb700 1 -- 172.21.15.61:0/540914203 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f4118023020 con 0x7f41280b21f0 2022-07-20T03:07:13.880 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:13.894 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4108053ad0 msgr2=0x7f4108055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4108053ad0 0x7f4108055f90 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f412400d000 tx=0x7f4124009040).stop 2022-07-20T03:07:13.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 msgr2=0x7f4128098580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:13.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 0x7f4128098580 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f41280625f0 tx=0x7f4118014000).stop 2022-07-20T03:07:13.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 shutdown_connections 2022-07-20T03:07:13.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4108053ad0 0x7f4108055f90 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f41281438b0 0x7f4128140e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f41280b21f0 0x7f4128098580 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 --2- 172.21.15.61:0/540914203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f41280ac8a0 0x7f41280b1180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:13.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 >> 172.21.15.61:0/540914203 conn(0x7f41280b36b0 msgr2=0x7f41280b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:13.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 shutdown_connections 2022-07-20T03:07:13.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:13.823+0000 7f412ea75700 1 -- 172.21.15.61:0/540914203 wait complete. 2022-07-20T03:07:13.925 INFO:tasks.ceph.ceph_manager.ceph:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:13.926 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:14.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56160c9700 1 Processor -- start 2022-07-20T03:07:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56160c9700 1 -- start start 2022-07-20T03:07:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56160c9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f56100b3cf0 0x7f5610142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56160c9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56100ba180 con 0x7f56100b9da0 2022-07-20T03:07:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56160c9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f56101461b0 con 0x7f5610142f40 2022-07-20T03:07:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56160c9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56100ad5b0 con 0x7f56100b3cf0 2022-07-20T03:07:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56150c7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f56100b9da0 0x7f56101468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38960/0 (socket says 172.21.15.61:38960) 2022-07-20T03:07:14.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56150c7700 1 -- 172.21.15.61:0/2747762296 learned_addr learned my addr 172.21.15.61:0/2747762296 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56148c6700 1 --2- 172.21.15.61:0/2747762296 >> v2:172.21.15.117:3300/0 conn(0x7f56100b3cf0 0x7f5610142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f5607fff700 1 -- 172.21.15.61:0/2747762296 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3681887513 0 0) 0x7f56100ba180 con 0x7f56100b9da0 2022-07-20T03:07:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f5607fff700 1 -- 172.21.15.61:0/2747762296 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f55fc003260 con 0x7f56100b9da0 2022-07-20T03:07:14.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f5607fff700 1 -- 172.21.15.61:0/2747762296 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 566560359 0 0) 0x7f55fc003260 con 0x7f56100b9da0 2022-07-20T03:07:14.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56148c6700 1 -- 172.21.15.61:0/2747762296 >> v1:172.21.15.117:6789/0 conn(0x7f5610142f40 legacy=0x7f56101419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56148c6700 1 -- 172.21.15.61:0/2747762296 >> v1:172.21.15.61:6789/0 conn(0x7f56100b9da0 legacy=0x7f56101468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.063+0000 7f56148c6700 1 -- 172.21.15.61:0/2747762296 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5610146450 con 0x7f56100b3cf0 2022-07-20T03:07:14.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56148c6700 1 --2- 172.21.15.61:0/2747762296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f5610142a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5608001b80 tx=0x7f560c006df0).ready entity=mon.1 client_cookie=4095139b742144a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f5607fff700 1 -- 172.21.15.61:0/2747762296 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f56080019e0 con 0x7f56100b3cf0 2022-07-20T03:07:14.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f5607fff700 1 -- 172.21.15.61:0/2747762296 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f560c000e90 con 0x7f56100b3cf0 2022-07-20T03:07:14.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f5607fff700 1 -- 172.21.15.61:0/2747762296 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f560c00d4f0 con 0x7f56100b3cf0 2022-07-20T03:07:14.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 -- 172.21.15.61:0/2747762296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 msgr2=0x7f5610142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 --2- 172.21.15.61:0/2747762296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f5610142a50 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f5608001b80 tx=0x7f560c006df0).stop 2022-07-20T03:07:14.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 -- 172.21.15.61:0/2747762296 shutdown_connections 2022-07-20T03:07:14.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 --2- 172.21.15.61:0/2747762296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f5610142a50 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 -- 172.21.15.61:0/2747762296 >> 172.21.15.61:0/2747762296 conn(0x7f56100678c0 msgr2=0x7f56100b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:14.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 -- 172.21.15.61:0/2747762296 shutdown_connections 2022-07-20T03:07:14.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 -- 172.21.15.61:0/2747762296 wait complete. 2022-07-20T03:07:14.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 Processor -- start 2022-07-20T03:07:14.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.067+0000 7f56160c9700 1 -- start start 2022-07-20T03:07:14.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f56101410d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56100b9da0 0x7f5610113190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 0x7f5610155390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f56100b08f0 con 0x7f5610142f40 2022-07-20T03:07:14.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5610151db0 con 0x7f56100b3cf0 2022-07-20T03:07:14.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5610113680 con 0x7f56100b9da0 2022-07-20T03:07:14.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 0x7f5610155390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 0x7f5610155390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42372/0 (socket says 172.21.15.61:42372) 2022-07-20T03:07:14.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 -- 172.21.15.61:0/3163745109 learned_addr learned my addr 172.21.15.61:0/3163745109 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:14.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56148c6700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56100b9da0 0x7f5610113190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56150c7700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f56101410d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 -- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56100b9da0 msgr2=0x7f5610113190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56100b9da0 0x7f5610113190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 -- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 msgr2=0x7f56101410d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f56101410d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 -- 172.21.15.61:0/3163745109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f560c00b180 con 0x7f5610142f40 2022-07-20T03:07:14.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56158c8700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 0x7f5610155390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f56100af6f0 tx=0x7f5600007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5600001d50 con 0x7f5610142f40 2022-07-20T03:07:14.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5600007030 con 0x7f5610142f40 2022-07-20T03:07:14.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5610155880 con 0x7f5610142f40 2022-07-20T03:07:14.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5600007390 con 0x7f5610142f40 2022-07-20T03:07:14.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f56100afde0 con 0x7f5610142f40 2022-07-20T03:07:14.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f56100667f0 con 0x7f5610142f40 2022-07-20T03:07:14.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.071+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5600010030 con 0x7f5610142f40 2022-07-20T03:07:14.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.075+0000 7f56067fc700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f55f0053a40 0x7f55f0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.075+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f560006baa0 con 0x7f5610142f40 2022-07-20T03:07:14.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.075+0000 7f56150c7700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f55f0053a40 0x7f55f0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.075+0000 7f56150c7700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f55f0053a40 0x7f55f0055f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f560800b000 tx=0x7f5608000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.079+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56000428f0 con 0x7f5610142f40 2022-07-20T03:07:14.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.203+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f56100574e0 con 0x7f5610142f40 2022-07-20T03:07:14.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.203+0000 7f56067fc700 1 -- 172.21.15.61:0/3163745109 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5600045f10 con 0x7f5610142f40 2022-07-20T03:07:14.288 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:14.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f55f0053a40 msgr2=0x7f55f0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f55f0053a40 0x7f55f0055f00 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f560800b000 tx=0x7f5608000f40).stop 2022-07-20T03:07:14.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 msgr2=0x7f5610155390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 0x7f5610155390 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f56100af6f0 tx=0x7f5600007c20).stop 2022-07-20T03:07:14.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 shutdown_connections 2022-07-20T03:07:14.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f55f0053a40 0x7f55f0055f00 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5610142f40 0x7f5610155390 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56100b9da0 0x7f5610113190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 --2- 172.21.15.61:0/3163745109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f56100b3cf0 0x7f56101410d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 >> 172.21.15.61:0/3163745109 conn(0x7f56100678c0 msgr2=0x7f56100b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:14.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 shutdown_connections 2022-07-20T03:07:14.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.207+0000 7f56160c9700 1 -- 172.21.15.61:0/3163745109 wait complete. 2022-07-20T03:07:14.294 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:14.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 Processor -- start 2022-07-20T03:07:14.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 -- start start 2022-07-20T03:07:14.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f44fc0b2ec0 0x7f44fc0b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44fc05d160 0x7f44fc142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44fc0b3f10 con 0x7f44fc142700 2022-07-20T03:07:14.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc0b4ea0 con 0x7f44fc05d160 2022-07-20T03:07:14.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4501f79700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44fc142b30 con 0x7f44fc0b2ec0 2022-07-20T03:07:14.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f44f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44fc05d160 0x7f44fc142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f44f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44fc05d160 0x7f44fc142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42376/0 (socket says 172.21.15.61:42376) 2022-07-20T03:07:14.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f44f3fff700 1 -- 172.21.15.61:0/1636266359 learned_addr learned my addr 172.21.15.61:0/1636266359 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:14.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.427+0000 7f4500f77700 1 --2- 172.21.15.61:0/1636266359 >> v2:172.21.15.117:3300/0 conn(0x7f44fc0b2ec0 0x7f44fc0b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f3fff700 1 -- 172.21.15.61:0/1636266359 >> v2:172.21.15.117:3300/0 conn(0x7f44fc0b2ec0 msgr2=0x7f44fc0b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:14.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f3fff700 1 --2- 172.21.15.61:0/1636266359 >> v2:172.21.15.117:3300/0 conn(0x7f44fc0b2ec0 0x7f44fc0b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f3fff700 1 -- 172.21.15.61:0/1636266359 >> v1:172.21.15.117:6789/0 conn(0x7f44fc142700 legacy=0x7f44fc1461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f3fff700 1 -- 172.21.15.61:0/1636266359 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44fc0b3790 con 0x7f44fc05d160 2022-07-20T03:07:14.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4500f77700 1 --2- 172.21.15.61:0/1636266359 >> v2:172.21.15.117:3300/0 conn(0x7f44fc0b2ec0 0x7f44fc0b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:14.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f3fff700 1 --2- 172.21.15.61:0/1636266359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc142210 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f44ec002f70 tx=0x7f44ec006eb0).ready entity=mon.0 client_cookie=401fc7de4c77a88b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f37fe700 1 -- 172.21.15.61:0/1636266359 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44ec00d5c0 con 0x7f44fc05d160 2022-07-20T03:07:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f37fe700 1 -- 172.21.15.61:0/1636266359 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44ec000e90 con 0x7f44fc05d160 2022-07-20T03:07:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f44f37fe700 1 -- 172.21.15.61:0/1636266359 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44ec00d950 con 0x7f44fc05d160 2022-07-20T03:07:14.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 -- 172.21.15.61:0/1636266359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 msgr2=0x7f44fc142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 --2- 172.21.15.61:0/1636266359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc142210 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f44ec002f70 tx=0x7f44ec006eb0).stop 2022-07-20T03:07:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 -- 172.21.15.61:0/1636266359 shutdown_connections 2022-07-20T03:07:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 --2- 172.21.15.61:0/1636266359 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc142210 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 --2- 172.21.15.61:0/1636266359 >> v2:172.21.15.117:3300/0 conn(0x7f44fc0b2ec0 0x7f44fc0b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 -- 172.21.15.61:0/1636266359 >> 172.21.15.61:0/1636266359 conn(0x7f44fc066e20 msgr2=0x7f44fc067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:14.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 -- 172.21.15.61:0/1636266359 shutdown_connections 2022-07-20T03:07:14.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.431+0000 7f4501f79700 1 -- 172.21.15.61:0/1636266359 wait complete. 2022-07-20T03:07:14.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 Processor -- start 2022-07-20T03:07:14.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 -- start start 2022-07-20T03:07:14.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc0adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44fc0b2ec0 0x7f44fc12fad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44fc142700 0x7f44fc12ffc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc1460c0 con 0x7f44fc05d160 2022-07-20T03:07:14.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0ad920 con 0x7f44fc142700 2022-07-20T03:07:14.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44fc0acb80 con 0x7f44fc0b2ec0 2022-07-20T03:07:14.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc0adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc0adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42382/0 (socket says 172.21.15.61:42382) 2022-07-20T03:07:14.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 -- 172.21.15.61:0/166280443 learned_addr learned my addr 172.21.15.61:0/166280443 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:14.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f44f3fff700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44fc0b2ec0 0x7f44fc12fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501778700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44fc142700 0x7f44fc12ffc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 -- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44fc0b2ec0 msgr2=0x7f44fc12fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44fc0b2ec0 0x7f44fc12fad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 -- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44fc142700 msgr2=0x7f44fc12ffc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44fc142700 0x7f44fc12ffc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 -- 172.21.15.61:0/166280443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44f800b180 con 0x7f44fc05d160 2022-07-20T03:07:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4500f77700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc0adff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f44f800c000 tx=0x7f44f8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44f800c7e0 con 0x7f44fc05d160 2022-07-20T03:07:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44f800c940 con 0x7f44fc05d160 2022-07-20T03:07:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44ec00b1e0 con 0x7f44fc05d160 2022-07-20T03:07:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44f800cca0 con 0x7f44fc05d160 2022-07-20T03:07:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.435+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44fc130530 con 0x7f44fc05d160 2022-07-20T03:07:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.439+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f44f801c460 con 0x7f44fc05d160 2022-07-20T03:07:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.439+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44fc146c20 con 0x7f44fc05d160 2022-07-20T03:07:14.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.439+0000 7f44f1ffb700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44dc053ad0 0x7f44dc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.439+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f44f806c4c0 con 0x7f44fc05d160 2022-07-20T03:07:14.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.439+0000 7f44f3fff700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44dc053ad0 0x7f44dc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.439+0000 7f44f3fff700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44dc053ad0 0x7f44dc055f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f44ec01c000 tx=0x7f44ec015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.443+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44f80432b0 con 0x7f44fc05d160 2022-07-20T03:07:14.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f44fc0b2620 con 0x7f44fc05d160 2022-07-20T03:07:14.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f44f1ffb700 1 -- 172.21.15.61:0/166280443 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f44f800ce00 con 0x7f44fc05d160 2022-07-20T03:07:14.575 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:14.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44dc053ad0 msgr2=0x7f44dc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44dc053ad0 0x7f44dc055f90 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f44ec01c000 tx=0x7f44ec015000).stop 2022-07-20T03:07:14.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 msgr2=0x7f44fc0adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc0adff0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f44f800c000 tx=0x7f44f8006df0).stop 2022-07-20T03:07:14.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 shutdown_connections 2022-07-20T03:07:14.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44dc053ad0 0x7f44dc055f90 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44fc142700 0x7f44fc12ffc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44fc0b2ec0 0x7f44fc12fad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 --2- 172.21.15.61:0/166280443 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44fc05d160 0x7f44fc0adff0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 >> 172.21.15.61:0/166280443 conn(0x7f44fc066e20 msgr2=0x7f44fc05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:14.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 shutdown_connections 2022-07-20T03:07:14.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.571+0000 7f4501f79700 1 -- 172.21.15.61:0/166280443 wait complete. 2022-07-20T03:07:14.612 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:14.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810f337700 1 Processor -- start 2022-07-20T03:07:14.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810f337700 1 -- start start 2022-07-20T03:07:14.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810f337700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f81080b3670 0x7f81080ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810f337700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81080b3a50 con 0x7f8108099300 2022-07-20T03:07:14.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810f337700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81081430a0 con 0x7f81081437e0 2022-07-20T03:07:14.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810f337700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8108099e60 con 0x7f81080b3670 2022-07-20T03:07:14.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810d0d3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f81080b3670 0x7f81080ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810d8d4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8108099300 0x7f8108099750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:38990/0 (socket says 172.21.15.61:38990) 2022-07-20T03:07:14.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810d8d4700 1 -- 172.21.15.61:0/3788975828 learned_addr learned my addr 172.21.15.61:0/3788975828 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810d0d3700 1 -- 172.21.15.61:0/3788975828 >> v1:172.21.15.117:6789/0 conn(0x7f81081437e0 legacy=0x7f810813f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810d0d3700 1 -- 172.21.15.61:0/3788975828 >> v1:172.21.15.61:6789/0 conn(0x7f8108099300 legacy=0x7f8108099750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f80fbfff700 1 -- 172.21.15.61:0/3788975828 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420981616 0 0) 0x7f81080b3a50 con 0x7f8108099300 2022-07-20T03:07:14.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.755+0000 7f810d0d3700 1 -- 172.21.15.61:0/3788975828 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8108143340 con 0x7f81080b3670 2022-07-20T03:07:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810d0d3700 1 --2- 172.21.15.61:0/3788975828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81080b3670 0x7f81080ac180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f80fc002f70 tx=0x7f80fc006e80).ready entity=mon.0 client_cookie=74fc3de9e49a63e2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f80fbfff700 1 -- 172.21.15.61:0/3788975828 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80fc00bac0 con 0x7f81080b3670 2022-07-20T03:07:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f80fbfff700 1 -- 172.21.15.61:0/3788975828 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc000e90 con 0x7f81080b3670 2022-07-20T03:07:14.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f80fbfff700 1 -- 172.21.15.61:0/3788975828 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80fc00be10 con 0x7f81080b3670 2022-07-20T03:07:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 -- 172.21.15.61:0/3788975828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81080b3670 msgr2=0x7f81080ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 --2- 172.21.15.61:0/3788975828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81080b3670 0x7f81080ac180 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f80fc002f70 tx=0x7f80fc006e80).stop 2022-07-20T03:07:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 -- 172.21.15.61:0/3788975828 shutdown_connections 2022-07-20T03:07:14.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 --2- 172.21.15.61:0/3788975828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81080b3670 0x7f81080ac180 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 -- 172.21.15.61:0/3788975828 >> 172.21.15.61:0/3788975828 conn(0x7f81080b6c50 msgr2=0x7f81080b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 -- 172.21.15.61:0/3788975828 shutdown_connections 2022-07-20T03:07:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.759+0000 7f810f337700 1 -- 172.21.15.61:0/3788975828 wait complete. 2022-07-20T03:07:14.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 Processor -- start 2022-07-20T03:07:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 -- start start 2022-07-20T03:07:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 0x7f810809b800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81080b3670 0x7f810813d520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81081437e0 0x7f810813da10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8108140560 con 0x7f8108099300 2022-07-20T03:07:14.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f81080b00d0 con 0x7f81081437e0 2022-07-20T03:07:14.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81080afe30 con 0x7f81080b3670 2022-07-20T03:07:14.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 0x7f810809b800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 0x7f810809b800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42396/0 (socket says 172.21.15.61:42396) 2022-07-20T03:07:14.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 -- 172.21.15.61:0/2610153943 learned_addr learned my addr 172.21.15.61:0/2610153943 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:14.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d8d4700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81081437e0 0x7f810813da10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810c8d2700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81080b3670 0x7f810813d520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 -- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81080b3670 msgr2=0x7f810813d520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81080b3670 0x7f810813d520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 -- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81081437e0 msgr2=0x7f810813da10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81081437e0 0x7f810813da10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 -- 172.21.15.61:0/2610153943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80fc00d410 con 0x7f8108099300 2022-07-20T03:07:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810d0d3700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 0x7f810809b800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f80fc002f70 tx=0x7f81080aba60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80fc020070 con 0x7f8108099300 2022-07-20T03:07:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f80fc0197e0 con 0x7f8108099300 2022-07-20T03:07:14.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f81080b0310 con 0x7f8108099300 2022-07-20T03:07:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80fc019b40 con 0x7f8108099300 2022-07-20T03:07:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.763+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8108098ad0 con 0x7f8108099300 2022-07-20T03:07:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.767+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81080af780 con 0x7f8108099300 2022-07-20T03:07:14.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.767+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f80fc019ca0 con 0x7f8108099300 2022-07-20T03:07:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.767+0000 7f80faffd700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f80e4053ad0 0x7f80e4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.767+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f80fc06fed0 con 0x7f8108099300 2022-07-20T03:07:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.767+0000 7f810c8d2700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f80e4053ad0 0x7f80e4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:14.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.767+0000 7f810c8d2700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f80e4053ad0 0x7f80e4055f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f810400b000 tx=0x7f8104009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:14.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.771+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f81080af780 con 0x7f8108099300 2022-07-20T03:07:14.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.895+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8108098ad0 con 0x7f8108099300 2022-07-20T03:07:14.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f80faffd700 1 -- 172.21.15.61:0/2610153943 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8108098ad0 con 0x7f8108099300 2022-07-20T03:07:14.905 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:14.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f80e4053ad0 msgr2=0x7f80e4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f80e4053ad0 0x7f80e4055f90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f810400b000 tx=0x7f8104009040).stop 2022-07-20T03:07:14.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 msgr2=0x7f810809b800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:14.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 0x7f810809b800 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f80fc002f70 tx=0x7f81080aba60).stop 2022-07-20T03:07:14.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 shutdown_connections 2022-07-20T03:07:14.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f80e4053ad0 0x7f80e4055f90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81081437e0 0x7f810813da10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81080b3670 0x7f810813d520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 --2- 172.21.15.61:0/2610153943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8108099300 0x7f810809b800 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:14.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 >> 172.21.15.61:0/2610153943 conn(0x7f81080b6c50 msgr2=0x7f810805fad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:14.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 shutdown_connections 2022-07-20T03:07:14.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:14.899+0000 7f810f337700 1 -- 172.21.15.61:0/2610153943 wait complete. 2022-07-20T03:07:14.923 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:15.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.071+0000 7f081ffff700 1 Processor -- start 2022-07-20T03:07:15.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.071+0000 7f081ffff700 1 -- start start 2022-07-20T03:07:15.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f08200ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08200ac840 con 0x7f0820142ac0 2022-07-20T03:07:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081ffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0820143240 con 0x7f08200b4d70 2022-07-20T03:07:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08201434e0 con 0x7f08200ac460 2022-07-20T03:07:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f0817fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0820142ac0 0x7f082013c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39002/0 (socket says 172.21.15.61:39002) 2022-07-20T03:07:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f0817fff700 1 -- 172.21.15.61:0/3881499255 learned_addr learned my addr 172.21.15.61:0/3881499255 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:15.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081effd700 1 --2- 172.21.15.61:0/3881499255 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f08200ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081e7fc700 1 -- 172.21.15.61:0/3881499255 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998498962 0 0) 0x7f08200ac840 con 0x7f0820142ac0 2022-07-20T03:07:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081e7fc700 1 -- 172.21.15.61:0/3881499255 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0800003260 con 0x7f0820142ac0 2022-07-20T03:07:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081e7fc700 1 -- 172.21.15.61:0/3881499255 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3381844175 0 0) 0x7f0820143240 con 0x7f08200b4d70 2022-07-20T03:07:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081effd700 1 -- 172.21.15.61:0/3881499255 >> v1:172.21.15.117:6789/0 conn(0x7f08200b4d70 legacy=0x7f08200b5200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081effd700 1 -- 172.21.15.61:0/3881499255 >> v1:172.21.15.61:6789/0 conn(0x7f0820142ac0 legacy=0x7f082013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081effd700 1 -- 172.21.15.61:0/3881499255 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0820143780 con 0x7f08200ac460 2022-07-20T03:07:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081effd700 1 --2- 172.21.15.61:0/3881499255 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f08200ab0c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f0810002f70 tx=0x7f0810006df0).ready entity=mon.2 client_cookie=3a106d53b0b25524 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081e7fc700 1 -- 172.21.15.61:0/3881499255 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081000d570 con 0x7f08200ac460 2022-07-20T03:07:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081e7fc700 1 -- 172.21.15.61:0/3881499255 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0810000e90 con 0x7f08200ac460 2022-07-20T03:07:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081e7fc700 1 -- 172.21.15.61:0/3881499255 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081000d8c0 con 0x7f08200ac460 2022-07-20T03:07:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081ffff700 1 -- 172.21.15.61:0/3881499255 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 msgr2=0x7f08200ab0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.075+0000 7f081ffff700 1 --2- 172.21.15.61:0/3881499255 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f08200ab0c0 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f0810002f70 tx=0x7f0810006df0).stop 2022-07-20T03:07:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 -- 172.21.15.61:0/3881499255 shutdown_connections 2022-07-20T03:07:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 --2- 172.21.15.61:0/3881499255 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f08200ab0c0 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 -- 172.21.15.61:0/3881499255 >> 172.21.15.61:0/3881499255 conn(0x7f08200632b0 msgr2=0x7f0820063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 -- 172.21.15.61:0/3881499255 shutdown_connections 2022-07-20T03:07:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 -- 172.21.15.61:0/3881499255 wait complete. 2022-07-20T03:07:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 Processor -- start 2022-07-20T03:07:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.079+0000 7f081ffff700 1 -- start start 2022-07-20T03:07:15.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f082013fc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 0x7f0820064520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0820142ac0 0x7f0820141ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08200afbc0 con 0x7f08200b4d70 2022-07-20T03:07:15.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0820156060 con 0x7f0820142ac0 2022-07-20T03:07:15.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08200af720 con 0x7f08200ac460 2022-07-20T03:07:15.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 0x7f0820064520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 0x7f0820064520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42412/0 (socket says 172.21.15.61:42412) 2022-07-20T03:07:15.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 -- 172.21.15.61:0/2951442045 learned_addr learned my addr 172.21.15.61:0/2951442045 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:15.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081effd700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f082013fc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081f7fe700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0820142ac0 0x7f0820141ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 -- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 msgr2=0x7f082013fc10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f082013fc10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 -- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0820142ac0 msgr2=0x7f0820141ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0820142ac0 0x7f0820141ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 -- 172.21.15.61:0/2951442045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f081000b180 con 0x7f08200b4d70 2022-07-20T03:07:15.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f0817fff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 0x7f0820064520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0808001110 tx=0x7f0808007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f080800cdd0 con 0x7f08200b4d70 2022-07-20T03:07:15.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08080070d0 con 0x7f08200b4d70 2022-07-20T03:07:15.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0820155d80 con 0x7f08200b4d70 2022-07-20T03:07:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0808007430 con 0x7f08200b4d70 2022-07-20T03:07:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08200af200 con 0x7f08200b4d70 2022-07-20T03:07:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.083+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0808010070 con 0x7f08200b4d70 2022-07-20T03:07:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.087+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08201426f0 con 0x7f08200b4d70 2022-07-20T03:07:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.087+0000 7f081cff9700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0804053ad0 0x7f0804055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.087+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f080806b490 con 0x7f08200b4d70 2022-07-20T03:07:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.087+0000 7f081effd700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0804053ad0 0x7f0804055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.087+0000 7f081effd700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0804053ad0 0x7f0804055f90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f081001a000 tx=0x7f0810014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.091+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08080422e0 con 0x7f08200b4d70 2022-07-20T03:07:15.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.219+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0820156220 con 0x7f08200b4d70 2022-07-20T03:07:15.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.223+0000 7f081cff9700 1 -- 172.21.15.61:0/2951442045 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0808045900 con 0x7f08200b4d70 2022-07-20T03:07:15.225 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:15.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0804053ad0 msgr2=0x7f0804055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0804053ad0 0x7f0804055f90 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f081001a000 tx=0x7f0810014000).stop 2022-07-20T03:07:15.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 msgr2=0x7f0820064520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 0x7f0820064520 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f0808001110 tx=0x7f0808007aa0).stop 2022-07-20T03:07:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 shutdown_connections 2022-07-20T03:07:15.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0804053ad0 0x7f0804055f90 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0820142ac0 0x7f0820141ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08200b4d70 0x7f0820064520 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 --2- 172.21.15.61:0/2951442045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08200ac460 0x7f082013fc10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 >> 172.21.15.61:0/2951442045 conn(0x7f08200632b0 msgr2=0x7f0820060ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:15.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 shutdown_connections 2022-07-20T03:07:15.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.227+0000 7f081ffff700 1 -- 172.21.15.61:0/2951442045 wait complete. 2022-07-20T03:07:15.246 INFO:tasks.ceph:Waiting until ceph daemons up and pgs clean... 2022-07-20T03:07:15.246 INFO:tasks.ceph.ceph_manager.ceph:waiting for mgr available 2022-07-20T03:07:15.247 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mgr dump --format=json 2022-07-20T03:07:15.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 Processor -- start 2022-07-20T03:07:15.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 -- start start 2022-07-20T03:07:15.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0b5650 0x7f8f6c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8f6c0af8e0 0x7f8f6c05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f6c147010 con 0x7f8f6c0adff0 2022-07-20T03:07:15.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f6c1472b0 con 0x7f8f6c0af8e0 2022-07-20T03:07:15.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f70d9e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c146d40 con 0x7f8f6c0b5650 2022-07-20T03:07:15.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6bfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8f6c0af8e0 0x7f8f6c05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6affd700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f8f6c0adff0 0x7f8f6c147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:47520/0 (socket says 172.21.15.61:47520) 2022-07-20T03:07:15.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6affd700 1 -- 172.21.15.61:0/812256378 learned_addr learned my addr 172.21.15.61:0/812256378 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:15.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6b7fe700 1 --2- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0b5650 0x7f8f6c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6a7fc700 1 -- 172.21.15.61:0/812256378 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3139438321 0 0) 0x7f8f6c147010 con 0x7f8f6c0adff0 2022-07-20T03:07:15.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6a7fc700 1 -- 172.21.15.61:0/812256378 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8f50003260 con 0x7f8f6c0adff0 2022-07-20T03:07:15.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6bfff700 1 -- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0b5650 msgr2=0x7f8f6c0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:15.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6bfff700 1 --2- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0b5650 0x7f8f6c0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6bfff700 1 -- 172.21.15.61:0/812256378 >> v1:172.21.15.117:6789/0 conn(0x7f8f6c0adff0 legacy=0x7f8f6c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6bfff700 1 -- 172.21.15.61:0/812256378 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c0acae0 con 0x7f8f6c0af8e0 2022-07-20T03:07:15.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6bfff700 1 --2- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0af8e0 0x7f8f6c05bd90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8f640032f0 tx=0x7f8f640091d0).ready entity=mon.1 client_cookie=ddc2963dad249be2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.363+0000 7f8f6a7fc700 1 -- 172.21.15.61:0/812256378 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6400b600 con 0x7f8f6c0af8e0 2022-07-20T03:07:15.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f6a7fc700 1 -- 172.21.15.61:0/812256378 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f6400b760 con 0x7f8f6c0af8e0 2022-07-20T03:07:15.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f6a7fc700 1 -- 172.21.15.61:0/812256378 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6400b8d0 con 0x7f8f6c0af8e0 2022-07-20T03:07:15.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0af8e0 msgr2=0x7f8f6c05bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0af8e0 0x7f8f6c05bd90 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8f640032f0 tx=0x7f8f640091d0).stop 2022-07-20T03:07:15.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- 172.21.15.61:0/812256378 shutdown_connections 2022-07-20T03:07:15.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0af8e0 0x7f8f6c05bd90 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/812256378 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0b5650 0x7f8f6c0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- 172.21.15.61:0/812256378 >> 172.21.15.61:0/812256378 conn(0x7f8f6c0634e0 msgr2=0x7f8f6c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:15.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- 172.21.15.61:0/812256378 shutdown_connections 2022-07-20T03:07:15.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- 172.21.15.61:0/812256378 wait complete. 2022-07-20T03:07:15.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 Processor -- start 2022-07-20T03:07:15.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- start start 2022-07-20T03:07:15.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0adff0 0x7f8f6c10ee70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0af8e0 0x7f8f6c14cf20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 0x7f8f6c111750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c111c40 con 0x7f8f6c0b5650 2022-07-20T03:07:15.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c10e730 con 0x7f8f6c0adff0 2022-07-20T03:07:15.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.367+0000 7f8f70d9e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f6c10e9d0 con 0x7f8f6c0af8e0 2022-07-20T03:07:15.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 0x7f8f6c111750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 0x7f8f6c111750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42428/0 (socket says 172.21.15.61:42428) 2022-07-20T03:07:15.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 -- 172.21.15.61:0/3272193758 learned_addr learned my addr 172.21.15.61:0/3272193758 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:15.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6affd700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0af8e0 0x7f8f6c14cf20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6b7fe700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0adff0 0x7f8f6c10ee70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 -- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0af8e0 msgr2=0x7f8f6c14cf20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0af8e0 0x7f8f6c14cf20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 -- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0adff0 msgr2=0x7f8f6c10ee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0adff0 0x7f8f6c10ee70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 -- 172.21.15.61:0/3272193758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f64009470 con 0x7f8f6c0b5650 2022-07-20T03:07:15.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f6bfff700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 0x7f8f6c111750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8f640032f0 tx=0x7f8f64016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6400b8d0 con 0x7f8f6c0b5650 2022-07-20T03:07:15.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f64021670 con 0x7f8f6c0b5650 2022-07-20T03:07:15.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f6c111e60 con 0x7f8f6c0b5650 2022-07-20T03:07:15.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f6400b600 con 0x7f8f6c0b5650 2022-07-20T03:07:15.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6c146450 con 0x7f8f6c0b5650 2022-07-20T03:07:15.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.371+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f6c10eb90 con 0x7f8f6c0b5650 2022-07-20T03:07:15.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.375+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f6400f070 con 0x7f8f6c0b5650 2022-07-20T03:07:15.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.375+0000 7f8f68ff9700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f44053a40 0x7f8f44055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.375+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8f6406d150 con 0x7f8f6c0b5650 2022-07-20T03:07:15.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.375+0000 7f8f6b7fe700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f44053a40 0x7f8f44055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.375+0000 7f8f6b7fe700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f44053a40 0x7f8f44055f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8f5c00b000 tx=0x7f8f5c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.383+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f64044160 con 0x7f8f6c0b5650 2022-07-20T03:07:15.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.539+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mgr dump", "format": "json"} v 0) v1 -- 0x7f8f6c0574e0 con 0x7f8f6c0b5650 2022-07-20T03:07:15.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f68ff9700 1 -- 172.21.15.61:0/3272193758 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mgr dump", "format": "json"}]=0 v4) v1 ==== 74+0+131710 (secure 0 0 0) 0x7f8f64015020 con 0x7f8f6c0b5650 2022-07-20T03:07:15.588 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:15.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f44053a40 msgr2=0x7f8f44055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f44053a40 0x7f8f44055f00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8f5c00b000 tx=0x7f8f5c006df0).stop 2022-07-20T03:07:15.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 msgr2=0x7f8f6c111750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 0x7f8f6c111750 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f8f640032f0 tx=0x7f8f64016000).stop 2022-07-20T03:07:15.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 shutdown_connections 2022-07-20T03:07:15.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f44053a40 0x7f8f44055f00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f6c0b5650 0x7f8f6c111750 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f6c0af8e0 0x7f8f6c14cf20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 --2- 172.21.15.61:0/3272193758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f6c0adff0 0x7f8f6c10ee70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.547+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 >> 172.21.15.61:0/3272193758 conn(0x7f8f6c0634e0 msgr2=0x7f8f6c0ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:15.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.551+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 shutdown_connections 2022-07-20T03:07:15.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.551+0000 7f8f70d9e700 1 -- 172.21.15.61:0/3272193758 wait complete. 2022-07-20T03:07:15.664 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":4,"active_gid":4103,"active_name":"x","active_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6800","nonce":22827},{"type":"v1","addr":"172.21.15.117:6801","nonce":22827}]},"active_addr":"172.21.15.117:6801/22827","active_change":"2022-07-20T03:06:53.008251+0000","active_mgr_features":4540138292840890367,"available":true,"standbys":[{"gid":4104,"name":"y","mgr_features":4540138292840890367,"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}]}],"modules":["iostat","restful"],"available_modules":[{"name":"alerts","can_run":true,"error_string":"","module_options":{"interval":{"name":"interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"How frequently to reexamine health status","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"smtp_destination":{"name":"smtp_destination","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Email address to send alerts to","long_desc":"","tags":[],"see_also":[]},"smtp_from_name":{"name":"smtp_from_name","type":"str","level":"advanced","flags":1,"default_value":"Ceph","min":"","max":"","enum_allowed":[],"desc":"Email From: name","long_desc":"","tags":[],"see_also":[]},"smtp_host":{"name":"smtp_host","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_password":{"name":"smtp_password","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"Password to authenticate with","long_desc":"","tags":[],"see_also":[]},"smtp_port":{"name":"smtp_port","type":"int","level":"advanced","flags":1,"default_value":"465","min":"","max":"","enum_allowed":[],"desc":"SMTP port","long_desc":"","tags":[],"see_also":[]},"smtp_sender":{"name":"smtp_sender","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"SMTP envelope sender","long_desc":"","tags":[],"see_also":[]},"smtp_ssl":{"name":"smtp_ssl","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Use SSL to connect to SMTP server","long_desc":"","tags":[],"see_also":[]},"smtp_user":{"name":"smtp_user","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"User to authenticate as","long_desc":"","tags":[],"see_also":[]}}},{"name":"balancer","can_run":true,"error_string":"","module_options":{"active":{"name":"active","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"automatically balance PGs across cluster","long_desc":"","tags":[],"see_also":[]},"begin_time":{"name":"begin_time","type":"str","level":"advanced","flags":1,"default_value":"0000","min":"","max":"","enum_allowed":[],"desc":"beginning time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"begin_weekday":{"name":"begin_weekday","type":"uint","level":"advanced","flags":1,"default_value":"0","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to this day of the week or later","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"crush_compat_max_iterations":{"name":"crush_compat_max_iterations","type":"uint","level":"advanced","flags":1,"default_value":"25","min":"1","max":"250","enum_allowed":[],"desc":"maximum number of iterations to attempt optimization","long_desc":"","tags":[],"see_also":[]},"crush_compat_metrics":{"name":"crush_compat_metrics","type":"str","level":"advanced","flags":1,"default_value":"pgs,objects,bytes","min":"","max":"","enum_allowed":[],"desc":"metrics with which to calculate OSD utilization","long_desc":"Value is a list of one or more of \"pgs\", \"objects\", or \"bytes\", and indicates which metrics to use to balance utilization.","tags":[],"see_also":[]},"crush_compat_step":{"name":"crush_compat_step","type":"float","level":"advanced","flags":1,"default_value":"0.5","min":"0.001","max":"0.999","enum_allowed":[],"desc":"aggressiveness of optimization","long_desc":".99 is very aggressive, .01 is less aggressive","tags":[],"see_also":[]},"end_time":{"name":"end_time","type":"str","level":"advanced","flags":1,"default_value":"2400","min":"","max":"","enum_allowed":[],"desc":"ending time of day to automatically balance","long_desc":"This is a time of day in the format HHMM.","tags":[],"see_also":[]},"end_weekday":{"name":"end_weekday","type":"uint","level":"advanced","flags":1,"default_value":"7","min":"0","max":"7","enum_allowed":[],"desc":"Restrict automatic balancing to days of the week earlier than this","long_desc":"0 or 7 = Sunday, 1 = Monday, etc.","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_score":{"name":"min_score","type":"float","level":"advanced","flags":1,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"minimum score, below which no optimization is attempted","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":1,"default_value":"none","min":"","max":"","enum_allowed":["crush-compat","none","upmap"],"desc":"Balancer mode","long_desc":"","tags":[],"see_also":[]},"pool_ids":{"name":"pool_ids","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"pools which the automatic balancing will be limited to","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and attempt optimization","long_desc":"","tags":[],"see_also":[]},"upmap_max_deviation":{"name":"upmap_max_deviation","type":"int","level":"advanced","flags":1,"default_value":"5","min":"1","max":"","enum_allowed":[],"desc":"deviation below which no optimization is attempted","long_desc":"If the number of PGs are within this count then no optimization is attempted","tags":[],"see_also":[]},"upmap_max_optimizations":{"name":"upmap_max_optimizations","type":"uint","level":"advanced","flags":1,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"maximum upmap optimizations to make per attempt","long_desc":"","tags":[],"see_also":[]}}},{"name":"cephadm","can_run":true,"error_string":"","module_options":{"allow_ptrace":{"name":"allow_ptrace","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"allow SYS_PTRACE capability on ceph containers","long_desc":"The SYS_PTRACE capability is needed to attach to a process with gdb or strace. Enabling this options can allow debugging daemons that encounter problems at runtime.","tags":[],"see_also":[]},"config_dashboard":{"name":"config_dashboard","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"manage configs like API endpoints in Dashboard.","long_desc":"","tags":[],"see_also":[]},"container_image_alertmanager":{"name":"container_image_alertmanager","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/alertmanager:v0.20.0","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_base":{"name":"container_image_base","type":"str","level":"advanced","flags":1,"default_value":"quay.io/ceph/ceph","min":"","max":"","enum_allowed":[],"desc":"Container image name, without the tag","long_desc":"","tags":[],"see_also":[]},"container_image_grafana":{"name":"container_image_grafana","type":"str","level":"advanced","flags":0,"default_value":"quay.io/ceph/ceph-grafana:6.7.4","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_node_exporter":{"name":"container_image_node_exporter","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/node-exporter:v0.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_image_prometheus":{"name":"container_image_prometheus","type":"str","level":"advanced","flags":0,"default_value":"quay.io/prometheus/prometheus:v2.18.1","min":"","max":"","enum_allowed":[],"desc":"Prometheus container image","long_desc":"","tags":[],"see_also":[]},"container_init":{"name":"container_init","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Run podman/docker with `--init`","long_desc":"","tags":[],"see_also":[]},"daemon_cache_timeout":{"name":"daemon_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"seconds to cache service (daemon) inventory","long_desc":"","tags":[],"see_also":[]},"device_cache_timeout":{"name":"device_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"1800","min":"","max":"","enum_allowed":[],"desc":"seconds to cache device inventory","long_desc":"","tags":[],"see_also":[]},"facts_cache_timeout":{"name":"facts_cache_timeout","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"seconds to cache host facts data","long_desc":"","tags":[],"see_also":[]},"host_check_interval":{"name":"host_check_interval","type":"secs","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to perform a host check","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"log to the \"cephadm\" cluster log channel\"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"manage_etc_ceph_ceph_conf":{"name":"manage_etc_ceph_ceph_conf","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Manage and own /etc/ceph/ceph.conf on the hosts.","long_desc":"","tags":[],"see_also":[]},"migration_current":{"name":"migration_current","type":"int","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"internal - do not modify","long_desc":"","tags":[],"see_also":[]},"mode":{"name":"mode","type":"str","level":"advanced","flags":0,"default_value":"root","min":"","max":"","enum_allowed":["cephadm-package","root"],"desc":"mode for remote execution of cephadm","long_desc":"","tags":[],"see_also":[]},"prometheus_alerts_path":{"name":"prometheus_alerts_path","type":"str","level":"advanced","flags":0,"default_value":"/etc/prometheus/ceph/ceph_default_alerts.yml","min":"","max":"","enum_allowed":[],"desc":"location of alerts to include in prometheus deployments","long_desc":"","tags":[],"see_also":[]},"registry_password":{"name":"registry_password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository password","long_desc":"","tags":[],"see_also":[]},"registry_url":{"name":"registry_url","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository url","long_desc":"","tags":[],"see_also":[]},"registry_username":{"name":"registry_username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"Custom repository username","long_desc":"","tags":[],"see_also":[]},"ssh_config_file":{"name":"ssh_config_file","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"customized SSH config file to connect to managed hosts","long_desc":"","tags":[],"see_also":[]},"use_repo_digest":{"name":"use_repo_digest","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Automatically convert image tags to image digest. Make sure all daemons use the same image","long_desc":"","tags":[],"see_also":[]},"warn_on_failed_host_check":{"name":"warn_on_failed_host_check","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if the host check fails","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_daemons":{"name":"warn_on_stray_daemons","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected that are not managed by cephadm","long_desc":"","tags":[],"see_also":[]},"warn_on_stray_hosts":{"name":"warn_on_stray_hosts","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"raise a health warning if daemons are detected on a host that is not managed by cephadm","long_desc":"","tags":[],"see_also":[]}}},{"name":"crash","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"retain_interval":{"name":"retain_interval","type":"secs","level":"advanced","flags":1,"default_value":"31536000","min":"","max":"","enum_allowed":[],"desc":"how long to retain crashes before pruning them","long_desc":"","tags":[],"see_also":[]},"warn_recent_interval":{"name":"warn_recent_interval","type":"secs","level":"advanced","flags":1,"default_value":"1209600","min":"","max":"","enum_allowed":[],"desc":"time interval in which to warn about recent crashes","long_desc":"","tags":[],"see_also":[]}}},{"name":"dashboard","can_run":true,"error_string":"","module_options":{"ACCOUNT_LOCKOUT_ATTEMPTS":{"name":"ACCOUNT_LOCKOUT_ATTEMPTS","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_HOST":{"name":"ALERTMANAGER_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ALERTMANAGER_API_SSL_VERIFY":{"name":"ALERTMANAGER_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_ENABLED":{"name":"AUDIT_API_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"AUDIT_API_LOG_PAYLOAD":{"name":"AUDIT_API_LOG_PAYLOAD","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ENABLE_BROWSABLE_API":{"name":"ENABLE_BROWSABLE_API","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_cephfs":{"name":"FEATURE_TOGGLE_cephfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_iscsi":{"name":"FEATURE_TOGGLE_iscsi","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_mirroring":{"name":"FEATURE_TOGGLE_mirroring","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_nfs":{"name":"FEATURE_TOGGLE_nfs","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rbd":{"name":"FEATURE_TOGGLE_rbd","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"FEATURE_TOGGLE_rgw":{"name":"FEATURE_TOGGLE_rgw","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE":{"name":"GANESHA_CLUSTERS_RADOS_POOL_NAMESPACE","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_PASSWORD":{"name":"GRAFANA_API_PASSWORD","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_SSL_VERIFY":{"name":"GRAFANA_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_URL":{"name":"GRAFANA_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_API_USERNAME":{"name":"GRAFANA_API_USERNAME","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_FRONTEND_API_URL":{"name":"GRAFANA_FRONTEND_API_URL","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"GRAFANA_UPDATE_DASHBOARDS":{"name":"GRAFANA_UPDATE_DASHBOARDS","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ISCSI_API_SSL_VERIFICATION":{"name":"ISCSI_API_SSL_VERIFICATION","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_HOST":{"name":"PROMETHEUS_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PROMETHEUS_API_SSL_VERIFY":{"name":"PROMETHEUS_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_COMPLEXITY_ENABLED":{"name":"PWD_POLICY_CHECK_COMPLEXITY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED":{"name":"PWD_POLICY_CHECK_EXCLUSION_LIST_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_LENGTH_ENABLED":{"name":"PWD_POLICY_CHECK_LENGTH_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_OLDPWD_ENABLED":{"name":"PWD_POLICY_CHECK_OLDPWD_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_REPETITIVE_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED":{"name":"PWD_POLICY_CHECK_SEQUENTIAL_CHARS_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_CHECK_USERNAME_ENABLED":{"name":"PWD_POLICY_CHECK_USERNAME_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_ENABLED":{"name":"PWD_POLICY_ENABLED","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_EXCLUSION_LIST":{"name":"PWD_POLICY_EXCLUSION_LIST","type":"str","level":"advanced","flags":0,"default_value":"osd,host,dashboard,pool,block,nfs,ceph,monitors,gateway,logs,crush,maps","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_COMPLEXITY":{"name":"PWD_POLICY_MIN_COMPLEXITY","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"PWD_POLICY_MIN_LENGTH":{"name":"PWD_POLICY_MIN_LENGTH","type":"int","level":"advanced","flags":0,"default_value":"8","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"REST_REQUESTS_TIMEOUT":{"name":"REST_REQUESTS_TIMEOUT","type":"int","level":"advanced","flags":0,"default_value":"45","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ACCESS_KEY":{"name":"RGW_API_ACCESS_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_ADMIN_RESOURCE":{"name":"RGW_API_ADMIN_RESOURCE","type":"str","level":"advanced","flags":0,"default_value":"admin","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_HOST":{"name":"RGW_API_HOST","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_PORT":{"name":"RGW_API_PORT","type":"int","level":"advanced","flags":0,"default_value":"80","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SCHEME":{"name":"RGW_API_SCHEME","type":"str","level":"advanced","flags":0,"default_value":"http","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SECRET_KEY":{"name":"RGW_API_SECRET_KEY","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_SSL_VERIFY":{"name":"RGW_API_SSL_VERIFY","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"RGW_API_USER_ID":{"name":"RGW_API_USER_ID","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_SPAN":{"name":"USER_PWD_EXPIRATION_SPAN","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_1":{"name":"USER_PWD_EXPIRATION_WARNING_1","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"USER_PWD_EXPIRATION_WARNING_2":{"name":"USER_PWD_EXPIRATION_WARNING_2","type":"int","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"crt_file":{"name":"crt_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"debug":{"name":"debug","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"Enable/disable debug options","long_desc":"","tags":[],"see_also":[]},"jwt_token_ttl":{"name":"jwt_token_ttl","type":"int","level":"advanced","flags":0,"default_value":"28800","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"motd":{"name":"motd","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"The message of the day","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"::","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"int","level":"advanced","flags":0,"default_value":"8080","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl_server_port":{"name":"ssl_server_port","type":"int","level":"advanced","flags":0,"default_value":"8443","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_behaviour":{"name":"standby_behaviour","type":"str","level":"advanced","flags":0,"default_value":"redirect","min":"","max":"","enum_allowed":["error","redirect"],"desc":"","long_desc":"","tags":[],"see_also":[]},"standby_error_status_code":{"name":"standby_error_status_code","type":"int","level":"advanced","flags":0,"default_value":"500","min":"400","max":"599","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url_prefix":{"name":"url_prefix","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"devicehealth","can_run":true,"error_string":"","module_options":{"enable_monitoring":{"name":"enable_monitoring","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"monitor device health metrics","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mark_out_threshold":{"name":"mark_out_threshold","type":"secs","level":"advanced","flags":1,"default_value":"2419200","min":"","max":"","enum_allowed":[],"desc":"automatically mark OSD if it may fail before this long","long_desc":"","tags":[],"see_also":[]},"pool_name":{"name":"pool_name","type":"str","level":"advanced","flags":1,"default_value":"device_health_metrics","min":"","max":"","enum_allowed":[],"desc":"name of pool in which to store device health metrics","long_desc":"","tags":[],"see_also":[]},"retention_period":{"name":"retention_period","type":"secs","level":"advanced","flags":1,"default_value":"15552000","min":"","max":"","enum_allowed":[],"desc":"how long to retain device health metrics","long_desc":"","tags":[],"see_also":[]},"scrape_frequency":{"name":"scrape_frequency","type":"secs","level":"advanced","flags":1,"default_value":"86400","min":"","max":"","enum_allowed":[],"desc":"how frequently to scrape device health metrics","long_desc":"","tags":[],"see_also":[]},"self_heal":{"name":"self_heal","type":"bool","level":"advanced","flags":1,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"preemptively heal cluster around devices that may fail","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"secs","level":"advanced","flags":1,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"how frequently to wake up and check device health","long_desc":"","tags":[],"see_also":[]},"warn_threshold":{"name":"warn_threshold","type":"secs","level":"advanced","flags":1,"default_value":"7257600","min":"","max":"","enum_allowed":[],"desc":"raise health warning if OSD may fail before this long","long_desc":"","tags":[],"see_also":[]}}},{"name":"diskprediction_cloud","can_run":true,"error_string":"","module_options":{"diskprediction_cert_context":{"name":"diskprediction_cert_context","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_default_authority":{"name":"diskprediction_default_authority","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_password":{"name":"diskprediction_password","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_port":{"name":"diskprediction_port","type":"str","level":"advanced","flags":0,"default_value":"31400","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_retrieve_prediction_interval":{"name":"diskprediction_retrieve_prediction_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_server":{"name":"diskprediction_server","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_ssl_target_name_override":{"name":"diskprediction_ssl_target_name_override","type":"str","level":"advanced","flags":0,"default_value":"localhost","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_metrics_interval":{"name":"diskprediction_upload_metrics_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_upload_smart_interval":{"name":"diskprediction_upload_smart_interval","type":"str","level":"advanced","flags":0,"default_value":"43200","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"diskprediction_user":{"name":"diskprediction_user","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"600","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"influx","can_run":false,"error_string":"influxdb python module not found","module_options":{"batch_size":{"name":"batch_size","type":"str","level":"advanced","flags":0,"default_value":"5000","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"database":{"name":"database","type":"str","level":"advanced","flags":0,"default_value":"ceph","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"hostname":{"name":"hostname","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"str","level":"advanced","flags":0,"default_value":"30","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"password":{"name":"password","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"port":{"name":"port","type":"str","level":"advanced","flags":0,"default_value":"8086","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"ssl":{"name":"ssl","type":"str","level":"advanced","flags":0,"default_value":"false","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"threads":{"name":"threads","type":"str","level":"advanced","flags":0,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"username":{"name":"username","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"verify_ssl":{"name":"verify_ssl","type":"str","level":"advanced","flags":0,"default_value":"true","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"insights","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"iostat","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"localpool","can_run":true,"error_string":"","module_options":{"failure_domain":{"name":"failure_domain","type":"str","level":"advanced","flags":1,"default_value":"host","min":"","max":"","enum_allowed":[],"desc":"failure domain for any created local pool","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"min_size":{"name":"min_size","type":"int","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"default min_size for any created local pool","long_desc":"","tags":[],"see_also":[]},"num_rep":{"name":"num_rep","type":"int","level":"advanced","flags":1,"default_value":"3","min":"","max":"","enum_allowed":[],"desc":"default replica count for any created local pool","long_desc":"","tags":[],"see_also":[]},"pg_num":{"name":"pg_num","type":"int","level":"advanced","flags":1,"default_value":"128","min":"","max":"","enum_allowed":[],"desc":"default pg_num for any created local pool","long_desc":"","tags":[],"see_also":[]},"prefix":{"name":"prefix","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"name prefix for any created local pool","long_desc":"","tags":[],"see_also":[]},"subtree":{"name":"subtree","type":"str","level":"advanced","flags":1,"default_value":"rack","min":"","max":"","enum_allowed":[],"desc":"CRUSH level for which to create a local pool","long_desc":"","tags":[],"see_also":[]}}},{"name":"orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"orchestrator":{"name":"orchestrator","type":"str","level":"advanced","flags":1,"default_value":"None","min":"","max":"","enum_allowed":["cephadm","rook","test_orchestrator"],"desc":"Orchestrator backend","long_desc":"","tags":[],"see_also":[]}}},{"name":"osd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"pg_autoscaler","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"sleep_interval":{"name":"sleep_interval","type":"str","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"progress","can_run":true,"error_string":"","module_options":{"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_completed_events":{"name":"max_completed_events","type":"int","level":"advanced","flags":1,"default_value":"50","min":"","max":"","enum_allowed":[],"desc":"number of past completed events to remember","long_desc":"","tags":[],"see_also":[]},"persist_interval":{"name":"persist_interval","type":"secs","level":"advanced","flags":1,"default_value":"5","min":"","max":"","enum_allowed":[],"desc":"how frequently to persist completed events","long_desc":"","tags":[],"see_also":[]}}},{"name":"prometheus","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools":{"name":"rbd_stats_pools","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rbd_stats_pools_refresh_interval":{"name":"rbd_stats_pools_refresh_interval","type":"int","level":"advanced","flags":0,"default_value":"300","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"scrape_interval":{"name":"scrape_interval","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"stale_cache_strategy":{"name":"stale_cache_strategy","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"rbd_support","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_snap_create":{"name":"max_concurrent_snap_create","type":"int","level":"advanced","flags":0,"default_value":"10","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"mirror_snapshot_schedule":{"name":"mirror_snapshot_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"trash_purge_schedule":{"name":"trash_purge_schedule","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"restful","can_run":true,"error_string":"","module_options":{"key_file":{"name":"key_file","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_addr":{"name":"server_addr","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"server_port":{"name":"server_port","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"selftest","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption1":{"name":"roption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"roption2":{"name":"roption2","type":"str","level":"advanced","flags":0,"default_value":"xyz","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption1":{"name":"rwoption1","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption2":{"name":"rwoption2","type":"int","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption3":{"name":"rwoption3","type":"float","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption4":{"name":"rwoption4","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption5":{"name":"rwoption5","type":"bool","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"rwoption6":{"name":"rwoption6","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testkey":{"name":"testkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testlkey":{"name":"testlkey","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"testnewline":{"name":"testnewline","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"status","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telegraf","can_run":true,"error_string":"","module_options":{"address":{"name":"address","type":"str","level":"advanced","flags":0,"default_value":"unixgram:///tmp/telegraf.sock","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"15","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"telemetry","can_run":true,"error_string":"","module_options":{"channel_basic":{"name":"channel_basic","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"Share basic cluster information (size, version)","long_desc":"","tags":[],"see_also":[]},"channel_crash":{"name":"channel_crash","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_device":{"name":"channel_device","type":"bool","level":"advanced","flags":0,"default_value":"True","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"channel_ident":{"name":"channel_ident","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"contact":{"name":"contact","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"description":{"name":"description","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"device_url":{"name":"device_url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/device","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"enabled":{"name":"enabled","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"int","level":"advanced","flags":0,"default_value":"24","min":"8","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"last_opt_revision":{"name":"last_opt_revision","type":"int","level":"advanced","flags":0,"default_value":"1","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"leaderboard":{"name":"leaderboard","type":"bool","level":"advanced","flags":0,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"organization":{"name":"organization","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"proxy":{"name":"proxy","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"url":{"name":"url","type":"str","level":"advanced","flags":0,"default_value":"https://telemetry.ceph.com/report","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"test_orchestrator","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}},{"name":"volumes","can_run":true,"error_string":"","module_options":{"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"max_concurrent_clones":{"name":"max_concurrent_clones","type":"int","level":"advanced","flags":0,"default_value":"4","min":"","max":"","enum_allowed":[],"desc":"Number of asynchronous cloner threads","long_desc":"","tags":[],"see_also":[]},"snapshot_clone_delay":{"name":"snapshot_clone_delay","type":"int","level":"advanced","flags":0,"default_value":"0","min":"","max":"","enum_allowed":[],"desc":"Delay clone begin operation by snapshot_clone_delay seconds","long_desc":"","tags":[],"see_also":[]}}},{"name":"zabbix","can_run":true,"error_string":"","module_options":{"discovery_interval":{"name":"discovery_interval","type":"str","level":"advanced","flags":0,"default_value":"100","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"identifier":{"name":"identifier","type":"str","level":"advanced","flags":0,"default_value":"","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"interval":{"name":"interval","type":"secs","level":"advanced","flags":0,"default_value":"60","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_level":{"name":"log_level","type":"str","level":"advanced","flags":1,"default_value":"","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster":{"name":"log_to_cluster","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_cluster_level":{"name":"log_to_cluster_level","type":"str","level":"advanced","flags":1,"default_value":"info","min":"","max":"","enum_allowed":["","critical","debug","error","info","warning"],"desc":"","long_desc":"","tags":[],"see_also":[]},"log_to_file":{"name":"log_to_file","type":"bool","level":"advanced","flags":1,"default_value":"False","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_host":{"name":"zabbix_host","type":"str","level":"advanced","flags":0,"default_value":"None","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_port":{"name":"zabbix_port","type":"int","level":"advanced","flags":0,"default_value":"10051","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]},"zabbix_sender":{"name":"zabbix_sender","type":"str","level":"advanced","flags":0,"default_value":"/usr/bin/zabbix_sender","min":"","max":"","enum_allowed":[],"desc":"","long_desc":"","tags":[],"see_also":[]}}}],"services":{},"always_on_modules":{"nautilus":["balancer","crash","devicehealth","orchestrator_cli","progress","rbd_support","status","volumes"],"octopus":["balancer","crash","devicehealth","orchestrator","pg_autoscaler","progress","rbd_support","status","telemetry","volumes"],"last_failure_osd_epoch":0,"active_clients":[{"addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":4138659466}]},{"addrvec":[{"type":"v2","addr":"172.21.15.117:0","nonce":2712642449}]}]}} 2022-07-20T03:07:15.726 INFO:tasks.ceph.ceph_manager.ceph:mgr available! 2022-07-20T03:07:15.727 INFO:tasks.ceph.ceph_manager.ceph:waiting for all up 2022-07-20T03:07:15.728 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:15.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 Processor -- start 2022-07-20T03:07:15.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 -- start start 2022-07-20T03:07:15.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29581424e0 0x7f29580b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 0x7f29580ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29581433d0 con 0x7f29580b0570 2022-07-20T03:07:15.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2958143b10 con 0x7f29580aea10 2022-07-20T03:07:15.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295e1a1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2958143670 con 0x7f29581424e0 2022-07-20T03:07:15.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295d9a0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 0x7f29580ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295d9a0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 0x7f29580ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:52114/0 (socket says 172.21.15.61:52114) 2022-07-20T03:07:15.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295c99e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29581424e0 0x7f29580b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295d9a0700 1 -- 172.21.15.61:0/1492403244 learned_addr learned my addr 172.21.15.61:0/1492403244 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:15.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295d19f700 1 --1- 172.21.15.61:0/1492403244 >> v1:172.21.15.117:6789/0 conn(0x7f29580b0570 0x7f2958141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:47534/0 (socket says 172.21.15.61:47534) 2022-07-20T03:07:15.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295c99e700 1 -- 172.21.15.61:0/1492403244 >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 msgr2=0x7f29580ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:15.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295c99e700 1 --2- 172.21.15.61:0/1492403244 >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 0x7f29580ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295c99e700 1 -- 172.21.15.61:0/1492403244 >> v1:172.21.15.117:6789/0 conn(0x7f29580b0570 legacy=0x7f2958141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295c99e700 1 -- 172.21.15.61:0/1492403244 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29580b1010 con 0x7f29581424e0 2022-07-20T03:07:15.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.843+0000 7f295d9a0700 1 --2- 172.21.15.61:0/1492403244 >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 0x7f29580ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:15.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295c99e700 1 --2- 172.21.15.61:0/1492403244 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29581424e0 0x7f29580b0950 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f2954002f70 tx=0x7f2954006df0).ready entity=mon.2 client_cookie=a22401a3cb41d744 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f294ffff700 1 -- 172.21.15.61:0/1492403244 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f295400d570 con 0x7f29581424e0 2022-07-20T03:07:15.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f294ffff700 1 -- 172.21.15.61:0/1492403244 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2954000e90 con 0x7f29581424e0 2022-07-20T03:07:15.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f294ffff700 1 -- 172.21.15.61:0/1492403244 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f295400d8c0 con 0x7f29581424e0 2022-07-20T03:07:15.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 -- 172.21.15.61:0/1492403244 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29581424e0 msgr2=0x7f29580b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1492403244 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29581424e0 0x7f29580b0950 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f2954002f70 tx=0x7f2954006df0).stop 2022-07-20T03:07:15.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 -- 172.21.15.61:0/1492403244 shutdown_connections 2022-07-20T03:07:15.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1492403244 >> v2:172.21.15.117:3300/0 conn(0x7f29580aea10 0x7f29580ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1492403244 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29581424e0 0x7f29580b0950 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 -- 172.21.15.61:0/1492403244 >> 172.21.15.61:0/1492403244 conn(0x7f29580b9360 msgr2=0x7f29580b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:15.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 -- 172.21.15.61:0/1492403244 shutdown_connections 2022-07-20T03:07:15.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 -- 172.21.15.61:0/1492403244 wait complete. 2022-07-20T03:07:15.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 Processor -- start 2022-07-20T03:07:15.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.847+0000 7f295e1a1700 1 -- start start 2022-07-20T03:07:15.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 0x7f29580b4d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29580b0570 0x7f2958112550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29581424e0 0x7f2958151ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2958114cb0 con 0x7f29580aea10 2022-07-20T03:07:15.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29580b45b0 con 0x7f29581424e0 2022-07-20T03:07:15.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29580b4850 con 0x7f29580b0570 2022-07-20T03:07:15.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 0x7f29580b4d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 0x7f29580b4d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42436/0 (socket says 172.21.15.61:42436) 2022-07-20T03:07:15.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 -- 172.21.15.61:0/1633626938 learned_addr learned my addr 172.21.15.61:0/1633626938 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:15.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295c99e700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29580b0570 0x7f2958112550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d9a0700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29581424e0 0x7f2958151ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 -- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29580b0570 msgr2=0x7f2958112550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29580b0570 0x7f2958112550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 -- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29581424e0 msgr2=0x7f2958151ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:15.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29581424e0 0x7f2958151ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:15.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 -- 172.21.15.61:0/1633626938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2950009530 con 0x7f29580aea10 2022-07-20T03:07:15.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295d19f700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 0x7f29580b4d60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f29581439f0 tx=0x7f2948007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29480070d0 con 0x7f29580aea10 2022-07-20T03:07:15.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2948007230 con 0x7f29580aea10 2022-07-20T03:07:15.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295400b180 con 0x7f29580aea10 2022-07-20T03:07:15.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2948005740 con 0x7f29580aea10 2022-07-20T03:07:15.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.851+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2958112a40 con 0x7f29580aea10 2022-07-20T03:07:15.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.855+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29580b3680 con 0x7f29580aea10 2022-07-20T03:07:15.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.855+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2948010070 con 0x7f29580aea10 2022-07-20T03:07:15.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.855+0000 7f294e7fc700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2940053a60 0x7f2940055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:15.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.855+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f294806b3a0 con 0x7f29580aea10 2022-07-20T03:07:15.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.855+0000 7f295c99e700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2940053a60 0x7f2940055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:15.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.855+0000 7f295c99e700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2940053a60 0x7f2940055f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f295401a000 tx=0x7f2954014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:15.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.863+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29580b3680 con 0x7f29580aea10 2022-07-20T03:07:15.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.991+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29580b3680 con 0x7f29580aea10 2022-07-20T03:07:16.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.991+0000 7f294e7fc700 1 -- 172.21.15.61:0/1633626938 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f29580b3680 con 0x7f29580aea10 2022-07-20T03:07:16.016 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:16.017 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:16.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2940053a60 msgr2=0x7f2940055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2940053a60 0x7f2940055f20 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f295401a000 tx=0x7f2954014000).stop 2022-07-20T03:07:16.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 msgr2=0x7f29580b4d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 0x7f29580b4d60 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f29581439f0 tx=0x7f2948007c00).stop 2022-07-20T03:07:16.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 shutdown_connections 2022-07-20T03:07:16.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2940053a60 0x7f2940055f20 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29581424e0 0x7f2958151ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29580b0570 0x7f2958112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 --2- 172.21.15.61:0/1633626938 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29580aea10 0x7f29580b4d60 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 >> 172.21.15.61:0/1633626938 conn(0x7f29580b9360 msgr2=0x7f295813f250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:16.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 shutdown_connections 2022-07-20T03:07:16.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:15.995+0000 7f295e1a1700 1 -- 172.21.15.61:0/1633626938 wait complete. 2022-07-20T03:07:16.024 INFO:tasks.ceph.ceph_manager.ceph:all up! 2022-07-20T03:07:16.024 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90eef84700 1 Processor -- start 2022-07-20T03:07:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90eef84700 1 -- start start 2022-07-20T03:07:16.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90eef84700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e80b1ef0 0x7f90e80605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90eef84700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e8140940 con 0x7f90e80ab180 2022-07-20T03:07:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90eef84700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f90e80b22d0 con 0x7f90e813f980 2022-07-20T03:07:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90eef84700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e805fbc0 con 0x7f90e80b1ef0 2022-07-20T03:07:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ee783700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f90e80ab180 0x7f90e80ab5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39042/0 (socket says 172.21.15.61:39042) 2022-07-20T03:07:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90edf82700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e80b1ef0 0x7f90e80605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ee783700 1 -- 172.21.15.61:0/2478909864 learned_addr learned my addr 172.21.15.61:0/2478909864 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ecf80700 1 -- 172.21.15.61:0/2478909864 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282494521 0 0) 0x7f90e8140940 con 0x7f90e80ab180 2022-07-20T03:07:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ecf80700 1 -- 172.21.15.61:0/2478909864 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90d0003260 con 0x7f90e80ab180 2022-07-20T03:07:16.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ecf80700 1 -- 172.21.15.61:0/2478909864 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1440785991 0 0) 0x7f90d0003260 con 0x7f90e80ab180 2022-07-20T03:07:16.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90edf82700 1 -- 172.21.15.61:0/2478909864 >> v1:172.21.15.117:6789/0 conn(0x7f90e813f980 legacy=0x7f90e80abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90edf82700 1 -- 172.21.15.61:0/2478909864 >> v1:172.21.15.61:6789/0 conn(0x7f90e80ab180 legacy=0x7f90e80ab5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90edf82700 1 -- 172.21.15.61:0/2478909864 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e805fe60 con 0x7f90e80b1ef0 2022-07-20T03:07:16.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90edf82700 1 --2- 172.21.15.61:0/2478909864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e80605a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f90e0001800 tx=0x7f90d8006df0).ready entity=mon.0 client_cookie=39481b96f4f6fcb4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ecf80700 1 -- 172.21.15.61:0/2478909864 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e0001660 con 0x7f90e80b1ef0 2022-07-20T03:07:16.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ecf80700 1 -- 172.21.15.61:0/2478909864 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8000e90 con 0x7f90e80b1ef0 2022-07-20T03:07:16.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.135+0000 7f90ecf80700 1 -- 172.21.15.61:0/2478909864 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d800d4f0 con 0x7f90e80b1ef0 2022-07-20T03:07:16.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- 172.21.15.61:0/2478909864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 msgr2=0x7f90e80605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 --2- 172.21.15.61:0/2478909864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e80605a0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f90e0001800 tx=0x7f90d8006df0).stop 2022-07-20T03:07:16.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- 172.21.15.61:0/2478909864 shutdown_connections 2022-07-20T03:07:16.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 --2- 172.21.15.61:0/2478909864 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e80605a0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- 172.21.15.61:0/2478909864 >> 172.21.15.61:0/2478909864 conn(0x7f90e80b2ca0 msgr2=0x7f90e80b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:16.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- 172.21.15.61:0/2478909864 shutdown_connections 2022-07-20T03:07:16.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- 172.21.15.61:0/2478909864 wait complete. 2022-07-20T03:07:16.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 Processor -- start 2022-07-20T03:07:16.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- start start 2022-07-20T03:07:16.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 0x7f90e81432b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e80b1ef0 0x7f90e8156040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e813f980 0x7f90e8156530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80ae190 con 0x7f90e80ab180 2022-07-20T03:07:16.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80ae3c0 con 0x7f90e813f980 2022-07-20T03:07:16.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80ae660 con 0x7f90e80b1ef0 2022-07-20T03:07:16.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 0x7f90e81432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 0x7f90e81432b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42452/0 (socket says 172.21.15.61:42452) 2022-07-20T03:07:16.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 -- 172.21.15.61:0/4132000739 learned_addr learned my addr 172.21.15.61:0/4132000739 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:16.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90ee783700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e813f980 0x7f90e8156530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90ed781700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e80b1ef0 0x7f90e8156040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 -- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e80b1ef0 msgr2=0x7f90e8156040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e80b1ef0 0x7f90e8156040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 -- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e813f980 msgr2=0x7f90e8156530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e813f980 0x7f90e8156530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 -- 172.21.15.61:0/4132000739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d800b180 con 0x7f90e80ab180 2022-07-20T03:07:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90ee783700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e813f980 0x7f90e8156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:16.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90edf82700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 0x7f90e81432b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f90e0001800 tx=0x7f90d8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d8016070 con 0x7f90e80ab180 2022-07-20T03:07:16.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8002f50 con 0x7f90e80ab180 2022-07-20T03:07:16.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.139+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e81437a0 con 0x7f90e80ab180 2022-07-20T03:07:16.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d800d4f0 con 0x7f90e80ab180 2022-07-20T03:07:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e80af5a0 con 0x7f90e80ab180 2022-07-20T03:07:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e8060100 con 0x7f90e80ab180 2022-07-20T03:07:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f90d8025030 con 0x7f90e80ab180 2022-07-20T03:07:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90d6ffd700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90c8053ad0 0x7f90c8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f90d806cc90 con 0x7f90e80ab180 2022-07-20T03:07:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90ed781700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90c8053ad0 0x7f90c8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.143+0000 7f90ed781700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90c8053ad0 0x7f90c8055f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f90e400d000 tx=0x7f90e4007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.151+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90d8043bf0 con 0x7f90e80ab180 2022-07-20T03:07:16.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.283+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f90e80574e0 con 0x7f90e80ab180 2022-07-20T03:07:16.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.283+0000 7f90d6ffd700 1 -- 172.21.15.61:0/4132000739 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f90d8023020 con 0x7f90e80ab180 2022-07-20T03:07:16.319 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:16.320 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.283+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90c8053ad0 msgr2=0x7f90c8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.283+0000 7f90eef84700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90c8053ad0 0x7f90c8055f90 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f90e400d000 tx=0x7f90e4007f10).stop 2022-07-20T03:07:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.283+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 msgr2=0x7f90e81432b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.283+0000 7f90eef84700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 0x7f90e81432b0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f90e0001800 tx=0x7f90d8014000).stop 2022-07-20T03:07:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 shutdown_connections 2022-07-20T03:07:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90c8053ad0 0x7f90c8055f90 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e813f980 0x7f90e8156530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e80b1ef0 0x7f90e8156040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 --2- 172.21.15.61:0/4132000739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80ab180 0x7f90e81432b0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 >> 172.21.15.61:0/4132000739 conn(0x7f90e80b2ca0 msgr2=0x7f90e813d590 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:16.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 shutdown_connections 2022-07-20T03:07:16.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.287+0000 7f90eef84700 1 -- 172.21.15.61:0/4132000739 wait complete. 2022-07-20T03:07:16.344 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 flush_pg_stats 2022-07-20T03:07:16.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 Processor -- start 2022-07-20T03:07:16.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 -- start start 2022-07-20T03:07:16.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f77180accc0 0x7f77181432b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77181437a0 0x7f7718143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7718098a80 con 0x7f7718095c70 2022-07-20T03:07:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7718097250 con 0x7f77181437a0 2022-07-20T03:07:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771e40f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77180ad0a0 con 0x7f77180accc0 2022-07-20T03:07:16.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771cc0c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77181437a0 0x7f7718143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771dc0e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7718095c70 0x7f77180960c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39060/0 (socket says 172.21.15.61:39060) 2022-07-20T03:07:16.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771dc0e700 1 -- 172.21.15.61:0/3742570860 learned_addr learned my addr 172.21.15.61:0/3742570860 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:16.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771d40d700 1 --2- 172.21.15.61:0/3742570860 >> v2:172.21.15.117:3300/0 conn(0x7f77180accc0 0x7f77181432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f7707fff700 1 -- 172.21.15.61:0/3742570860 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1772229243 0 0) 0x7f7718098a80 con 0x7f7718095c70 2022-07-20T03:07:16.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f7707fff700 1 -- 172.21.15.61:0/3742570860 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7700003260 con 0x7f7718095c70 2022-07-20T03:07:16.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771cc0c700 1 -- 172.21.15.61:0/3742570860 >> v2:172.21.15.117:3300/0 conn(0x7f77180accc0 msgr2=0x7f77181432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:16.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771cc0c700 1 --2- 172.21.15.61:0/3742570860 >> v2:172.21.15.117:3300/0 conn(0x7f77180accc0 0x7f77181432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771cc0c700 1 -- 172.21.15.61:0/3742570860 >> v1:172.21.15.61:6789/0 conn(0x7f7718095c70 legacy=0x7f77180960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.467+0000 7f771cc0c700 1 -- 172.21.15.61:0/3742570860 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77180abca0 con 0x7f77181437a0 2022-07-20T03:07:16.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771cc0c700 1 --2- 172.21.15.61:0/3742570860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77181437a0 0x7f7718143bf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7708002f70 tx=0x7f7708006df0).ready entity=mon.0 client_cookie=d023636046188162 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f7707fff700 1 -- 172.21.15.61:0/3742570860 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f770800d4f0 con 0x7f77181437a0 2022-07-20T03:07:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f7707fff700 1 -- 172.21.15.61:0/3742570860 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7708000e90 con 0x7f77181437a0 2022-07-20T03:07:16.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f7707fff700 1 -- 172.21.15.61:0/3742570860 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f770800d840 con 0x7f77181437a0 2022-07-20T03:07:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 -- 172.21.15.61:0/3742570860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77181437a0 msgr2=0x7f7718143bf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 --2- 172.21.15.61:0/3742570860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77181437a0 0x7f7718143bf0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f7708002f70 tx=0x7f7708006df0).stop 2022-07-20T03:07:16.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 -- 172.21.15.61:0/3742570860 shutdown_connections 2022-07-20T03:07:16.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 --2- 172.21.15.61:0/3742570860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77181437a0 0x7f7718143bf0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 --2- 172.21.15.61:0/3742570860 >> v2:172.21.15.117:3300/0 conn(0x7f77180accc0 0x7f77181432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 -- 172.21.15.61:0/3742570860 >> 172.21.15.61:0/3742570860 conn(0x7f77180b6210 msgr2=0x7f77180b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:16.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 -- 172.21.15.61:0/3742570860 shutdown_connections 2022-07-20T03:07:16.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 -- 172.21.15.61:0/3742570860 wait complete. 2022-07-20T03:07:16.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 Processor -- start 2022-07-20T03:07:16.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 -- start start 2022-07-20T03:07:16.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.471+0000 7f771e40f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7718095c70 0x7f77180b1da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77180accc0 0x7f7718067480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77181437a0 0x7f7718065f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7718065210 con 0x7f7718095c70 2022-07-20T03:07:16.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f771809a110 con 0x7f77181437a0 2022-07-20T03:07:16.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f771809b150 con 0x7f77180accc0 2022-07-20T03:07:16.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7718095c70 0x7f77180b1da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7718095c70 0x7f77180b1da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42464/0 (socket says 172.21.15.61:42464) 2022-07-20T03:07:16.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 -- 172.21.15.61:0/4240014615 learned_addr learned my addr 172.21.15.61:0/4240014615 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:16.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771cc0c700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77180accc0 0x7f7718067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771dc0e700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77181437a0 0x7f7718065f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 -- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77180accc0 msgr2=0x7f7718067480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77180accc0 0x7f7718067480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 -- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77181437a0 msgr2=0x7f7718065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77181437a0 0x7f7718065f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 -- 172.21.15.61:0/4240014615 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f770800b180 con 0x7f7718095c70 2022-07-20T03:07:16.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771d40d700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7718095c70 0x7f77180b1da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f771400c000 tx=0x7f7714006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7714010780 con 0x7f7718095c70 2022-07-20T03:07:16.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77140108e0 con 0x7f7718095c70 2022-07-20T03:07:16.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7718066fa0 con 0x7f7718095c70 2022-07-20T03:07:16.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7714010c40 con 0x7f7718095c70 2022-07-20T03:07:16.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7718099c20 con 0x7f7718095c70 2022-07-20T03:07:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f771809afd0 con 0x7f7718095c70 2022-07-20T03:07:16.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.475+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f771401c460 con 0x7f7718095c70 2022-07-20T03:07:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f77067fc700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76f8053af0 0x7f76f8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f771406c200 con 0x7f7718095c70 2022-07-20T03:07:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f77067fc700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f76f805ae50 0x7f76f805d2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:16.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 --> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f76f805d890 con 0x7f76f805ae50 2022-07-20T03:07:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f771406e890 con 0x7f7718095c70 2022-07-20T03:07:16.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f771cc0c700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76f8053af0 0x7f76f8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f771dc0e700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f76f805ae50 0x7f76f805d2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:16.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f771cc0c700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76f8053af0 0x7f76f8055fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f770801a000 tx=0x7f7708014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f771dc0e700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f76f805ae50 0x7f76f805d2b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:16.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.479+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== osd.0 v2:172.21.15.61:6800/22894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f76f805d890 con 0x7f76f805ae50 2022-07-20T03:07:16.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 --> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f77180666c0 con 0x7f76f805ae50 2022-07-20T03:07:16.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f77067fc700 1 -- 172.21.15.61:0/4240014615 <== osd.0 v2:172.21.15.61:6800/22894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f77180666c0 con 0x7f76f805ae50 2022-07-20T03:07:16.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f76f805ae50 msgr2=0x7f76f805d2b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f76f805ae50 0x7f76f805d2b0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:16.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76f8053af0 msgr2=0x7f76f8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76f8053af0 0x7f76f8055fb0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f770801a000 tx=0x7f7708014000).stop 2022-07-20T03:07:16.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7718095c70 msgr2=0x7f77180b1da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:16.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 --2- 172.21.15.61:0/4240014615 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7718095c70 0x7f77180b1da0 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f771400c000 tx=0x7f7714006df0).stop 2022-07-20T03:07:16.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771dc0e700 1 -- 172.21.15.61:0/4240014615 reap_dead start 2022-07-20T03:07:16.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 shutdown_connections 2022-07-20T03:07:16.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 >> 172.21.15.61:0/4240014615 conn(0x7f77180b6210 msgr2=0x7f77180960c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:16.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 shutdown_connections 2022-07-20T03:07:16.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.499+0000 7f771e40f700 1 -- 172.21.15.61:0/4240014615 wait complete. 2022-07-20T03:07:16.825 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:16.826 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 flush_pg_stats 2022-07-20T03:07:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e62d17700 1 Processor -- start 2022-07-20T03:07:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e62d17700 1 -- start start 2022-07-20T03:07:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e62d17700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9e5c097a80 0x7f9e5c061080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e62d17700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c14a750 con 0x7f9e5c060ca0 2022-07-20T03:07:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e62d17700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e5c0b2840 con 0x7f9e5c0afb80 2022-07-20T03:07:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e62d17700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e5c148430 con 0x7f9e5c097a80 2022-07-20T03:07:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9e5c060ca0 0x7f9e5c14ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39072/0 (socket says 172.21.15.61:39072) 2022-07-20T03:07:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 -- 172.21.15.61:0/4102515594 learned_addr learned my addr 172.21.15.61:0/4102515594 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5bfff700 1 --2- 172.21.15.61:0/4102515594 >> v2:172.21.15.117:3300/0 conn(0x7f9e5c097a80 0x7f9e5c061080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 137323011 0 0) 0x7f9e5c14a750 con 0x7f9e5c060ca0 2022-07-20T03:07:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e40003260 con 0x7f9e5c060ca0 2022-07-20T03:07:17.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1172098136 0 0) 0x7f9e40003260 con 0x7f9e5c060ca0 2022-07-20T03:07:17.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 >> v2:172.21.15.117:3300/0 conn(0x7f9e5c097a80 msgr2=0x7f9e5c061080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:17.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 --2- 172.21.15.61:0/4102515594 >> v2:172.21.15.117:3300/0 conn(0x7f9e5c097a80 0x7f9e5c061080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 >> v1:172.21.15.117:6789/0 conn(0x7f9e5c0afb80 legacy=0x7f9e5c146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c0ac110 con 0x7f9e5c060ca0 2022-07-20T03:07:17.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f9e4c0019e0 con 0x7f9e5c060ca0 2022-07-20T03:07:17.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 >> v1:172.21.15.61:6789/0 conn(0x7f9e5c060ca0 legacy=0x7f9e5c14ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e40003750 0x7f9e40003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40004500 0x7f9e40004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e40004e70 0x7f9e4000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4000bcf0 con 0x7f9e40003750 2022-07-20T03:07:17.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4000be60 con 0x7f9e40004500 2022-07-20T03:07:17.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e4000c020 con 0x7f9e40004e70 2022-07-20T03:07:17.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e5bfff700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40004500 0x7f9e40004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e612b4700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e40004e70 0x7f9e4000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e40003750 0x7f9e40003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 -- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e40004e70 msgr2=0x7f9e4000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:17.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e40004e70 0x7f9e4000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 -- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40004500 msgr2=0x7f9e40004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:17.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40004500 0x7f9e40004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e60ab3700 1 -- 172.21.15.61:0/4102515594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e40003260 con 0x7f9e40003750 2022-07-20T03:07:17.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.943+0000 7f9e612b4700 1 -- 172.21.15.61:0/4102515594 reap_dead start 2022-07-20T03:07:17.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e40003750 0x7f9e40003b70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9e4c0016f0 tx=0x7f9e5c0ac2a0).ready entity=mon.0 client_cookie=c458c7f42fff026c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e4c00cbd0 con 0x7f9e40003750 2022-07-20T03:07:17.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e4c00cd30 con 0x7f9e40003750 2022-07-20T03:07:17.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e5b7fe700 1 -- 172.21.15.61:0/4102515594 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e4c00cea0 con 0x7f9e40003750 2022-07-20T03:07:17.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- 172.21.15.61:0/4102515594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e40003750 msgr2=0x7f9e40003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e40003750 0x7f9e40003b70 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f9e4c0016f0 tx=0x7f9e5c0ac2a0).stop 2022-07-20T03:07:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- 172.21.15.61:0/4102515594 shutdown_connections 2022-07-20T03:07:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 --2- 172.21.15.61:0/4102515594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e40003750 0x7f9e40003b70 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- 172.21.15.61:0/4102515594 >> 172.21.15.61:0/4102515594 conn(0x7f9e5c0b5cb0 msgr2=0x7f9e5c0b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- 172.21.15.61:0/4102515594 shutdown_connections 2022-07-20T03:07:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- 172.21.15.61:0/4102515594 wait complete. 2022-07-20T03:07:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 Processor -- start 2022-07-20T03:07:17.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- start start 2022-07-20T03:07:17.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40003750 0x7f9e5c09ac20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e5c14d270 0x7f9e5c149ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e5c156420 0x7f9e5c1568a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c098a40 con 0x7f9e5c156420 2022-07-20T03:07:17.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c098ce0 con 0x7f9e40003750 2022-07-20T03:07:17.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e62d17700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e5c09a780 con 0x7f9e5c14d270 2022-07-20T03:07:17.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e5c156420 0x7f9e5c1568a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e5c156420 0x7f9e5c1568a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42488/0 (socket says 172.21.15.61:42488) 2022-07-20T03:07:17.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 -- 172.21.15.61:0/2035174888 learned_addr learned my addr 172.21.15.61:0/2035174888 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:17.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e5bfff700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e5c14d270 0x7f9e5c149ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40003750 0x7f9e5c09ac20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 -- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e5c14d270 msgr2=0x7f9e5c149ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e5c14d270 0x7f9e5c149ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 -- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40003750 msgr2=0x7f9e5c09ac20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40003750 0x7f9e5c09ac20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 -- 172.21.15.61:0/2035174888 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e5c098730 con 0x7f9e5c156420 2022-07-20T03:07:17.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e40003750 0x7f9e5c09ac20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:17.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.947+0000 7f9e612b4700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e5c156420 0x7f9e5c1568a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9e54000b60 tx=0x7f9e54007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e54007030 con 0x7f9e5c156420 2022-07-20T03:07:17.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e54007190 con 0x7f9e5c156420 2022-07-20T03:07:17.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e5c156d90 con 0x7f9e5c156420 2022-07-20T03:07:17.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e54006510 con 0x7f9e5c156420 2022-07-20T03:07:17.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e5c099150 con 0x7f9e5c156420 2022-07-20T03:07:17.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9e5401f070 con 0x7f9e5c156420 2022-07-20T03:07:17.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f9e5c09a940 con 0x7f9e5c156420 2022-07-20T03:07:17.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9e34053a60 0x7f9e34055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9e5406b470 con 0x7f9e5c156420 2022-07-20T03:07:17.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f9e3405ad40 0x7f9e3405d1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 --> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9e3405d780 con 0x7f9e3405ad40 2022-07-20T03:07:17.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f9e5406dae0 con 0x7f9e5c156420 2022-07-20T03:07:17.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9e34053a60 0x7f9e34055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e5bfff700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f9e3405ad40 0x7f9e3405d1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e60ab3700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9e34053a60 0x7f9e34055f20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9e4c017000 tx=0x7f9e4c013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.951+0000 7f9e5bfff700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f9e3405ad40 0x7f9e3405d1a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.955+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== osd.1 v2:172.21.15.61:6808/22998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f9e3405d780 con 0x7f9e3405ad40 2022-07-20T03:07:17.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 --> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f9e5c0abe40 con 0x7f9e3405ad40 2022-07-20T03:07:17.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e59ffb700 1 -- 172.21.15.61:0/2035174888 <== osd.1 v2:172.21.15.61:6808/22998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f9e5c0abe40 con 0x7f9e3405ad40 2022-07-20T03:07:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f9e3405ad40 msgr2=0x7f9e3405d1a0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f9e3405ad40 0x7f9e3405d1a0 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9e34053a60 msgr2=0x7f9e34055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9e34053a60 0x7f9e34055f20 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f9e4c017000 tx=0x7f9e4c013000).stop 2022-07-20T03:07:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e5c156420 msgr2=0x7f9e5c1568a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 --2- 172.21.15.61:0/2035174888 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e5c156420 0x7f9e5c1568a0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f9e54000b60 tx=0x7f9e54007a00).stop 2022-07-20T03:07:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e612b4700 1 -- 172.21.15.61:0/2035174888 reap_dead start 2022-07-20T03:07:17.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 shutdown_connections 2022-07-20T03:07:17.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 >> 172.21.15.61:0/2035174888 conn(0x7f9e5c0b5cb0 msgr2=0x7f9e5c0b3650 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:17.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 shutdown_connections 2022-07-20T03:07:17.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:16.971+0000 7f9e62d17700 1 -- 172.21.15.61:0/2035174888 wait complete. 2022-07-20T03:07:17.095 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:17.096 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 flush_pg_stats 2022-07-20T03:07:17.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 Processor -- start 2022-07-20T03:07:17.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 -- start start 2022-07-20T03:07:17.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f76280b2310 0x7f76281433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7628146b40 0x7f762813cc90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7628147020 con 0x7f76281438e0 2022-07-20T03:07:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280b1840 con 0x7f7628146b40 2022-07-20T03:07:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762d0fc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76280afbc0 con 0x7f76280b2310 2022-07-20T03:07:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762c8fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7628146b40 0x7f762813cc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762c8fb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7628146b40 0x7f762813cc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42498/0 (socket says 172.21.15.61:42498) 2022-07-20T03:07:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762c8fb700 1 -- 172.21.15.61:0/3539386605 learned_addr learned my addr 172.21.15.61:0/3539386605 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:17.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f7627fff700 1 --2- 172.21.15.61:0/3539386605 >> v2:172.21.15.117:3300/0 conn(0x7f76280b2310 0x7f76281433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762c8fb700 1 -- 172.21.15.61:0/3539386605 >> v2:172.21.15.117:3300/0 conn(0x7f76280b2310 msgr2=0x7f76281433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:17.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.211+0000 7f762c8fb700 1 --2- 172.21.15.61:0/3539386605 >> v2:172.21.15.117:3300/0 conn(0x7f76280b2310 0x7f76281433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762c8fb700 1 -- 172.21.15.61:0/3539386605 >> v1:172.21.15.117:6789/0 conn(0x7f76281438e0 legacy=0x7f762813c580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f7626ffd700 1 -- 172.21.15.61:0/3539386605 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1249351989 0 0) 0x7f7628147020 con 0x7f76281438e0 2022-07-20T03:07:17.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762c8fb700 1 -- 172.21.15.61:0/3539386605 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628142cb0 con 0x7f7628146b40 2022-07-20T03:07:17.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762c8fb700 1 --2- 172.21.15.61:0/3539386605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7628146b40 0x7f762813cc90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f7620002f70 tx=0x7f7620006eb0).ready entity=mon.0 client_cookie=8fa23fc90b8ca9fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f7626ffd700 1 -- 172.21.15.61:0/3539386605 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f762000d5c0 con 0x7f7628146b40 2022-07-20T03:07:17.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f7626ffd700 1 -- 172.21.15.61:0/3539386605 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7620000e90 con 0x7f7628146b40 2022-07-20T03:07:17.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f7626ffd700 1 -- 172.21.15.61:0/3539386605 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f762000d910 con 0x7f7628146b40 2022-07-20T03:07:17.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 -- 172.21.15.61:0/3539386605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7628146b40 msgr2=0x7f762813cc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 --2- 172.21.15.61:0/3539386605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7628146b40 0x7f762813cc90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f7620002f70 tx=0x7f7620006eb0).stop 2022-07-20T03:07:17.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 -- 172.21.15.61:0/3539386605 shutdown_connections 2022-07-20T03:07:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 --2- 172.21.15.61:0/3539386605 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7628146b40 0x7f762813cc90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 --2- 172.21.15.61:0/3539386605 >> v2:172.21.15.117:3300/0 conn(0x7f76280b2310 0x7f76281433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 -- 172.21.15.61:0/3539386605 >> 172.21.15.61:0/3539386605 conn(0x7f7628063150 msgr2=0x7f7628063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 -- 172.21.15.61:0/3539386605 shutdown_connections 2022-07-20T03:07:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.215+0000 7f762d0fc700 1 -- 172.21.15.61:0/3539386605 wait complete. 2022-07-20T03:07:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 Processor -- start 2022-07-20T03:07:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 -- start start 2022-07-20T03:07:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76280b2310 0x7f7628064cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76281438e0 0x7f762813e5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7628146b40 0x7f762813eaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f762813db00 con 0x7f76281438e0 2022-07-20T03:07:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76280ac6c0 con 0x7f7628146b40 2022-07-20T03:07:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762d0fc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7628155860 con 0x7f76280b2310 2022-07-20T03:07:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76281438e0 0x7f762813e5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76281438e0 0x7f762813e5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42504/0 (socket says 172.21.15.61:42504) 2022-07-20T03:07:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 -- 172.21.15.61:0/3816065209 learned_addr learned my addr 172.21.15.61:0/3816065209 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f7627fff700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76280b2310 0x7f7628064cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f762c8fb700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7628146b40 0x7f762813eaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 -- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76280b2310 msgr2=0x7f7628064cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76280b2310 0x7f7628064cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 -- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7628146b40 msgr2=0x7f762813eaa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7628146b40 0x7f762813eaa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 -- 172.21.15.61:0/3816065209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762000b1e0 con 0x7f76281438e0 2022-07-20T03:07:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.219+0000 7f76277fe700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76281438e0 0x7f762813e5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f761c000ed0 tx=0x7f761c007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f761c012400 con 0x7f76281438e0 2022-07-20T03:07:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f761c012560 con 0x7f76281438e0 2022-07-20T03:07:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7628155b00 con 0x7f76281438e0 2022-07-20T03:07:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f761c0128c0 con 0x7f76281438e0 2022-07-20T03:07:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76281552e0 con 0x7f76281438e0 2022-07-20T03:07:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7628142f50 con 0x7f76281438e0 2022-07-20T03:07:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f761c019030 con 0x7f76281438e0 2022-07-20T03:07:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7608053a40 0x7f7608055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f761c06b140 con 0x7f76281438e0 2022-07-20T03:07:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f760805ad20 0x7f760805d180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 --> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f760805d760 con 0x7f760805ad20 2022-07-20T03:07:17.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f761c06d7b0 con 0x7f76281438e0 2022-07-20T03:07:17.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f7627fff700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7608053a40 0x7f7608055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f762c8fb700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f760805ad20 0x7f760805d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f762c8fb700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f760805ad20 0x7f760805d180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.223+0000 7f7627fff700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7608053a40 0x7f7608055f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f761800b000 tx=0x7f76180091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.227+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== osd.2 v2:172.21.15.61:6815/23001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f760805d760 con 0x7f760805ad20 2022-07-20T03:07:17.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.239+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 --> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f762813f420 con 0x7f760805ad20 2022-07-20T03:07:17.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f76257fa700 1 -- 172.21.15.61:0/3816065209 <== osd.2 v2:172.21.15.61:6815/23001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f762813f420 con 0x7f760805ad20 2022-07-20T03:07:17.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f760805ad20 msgr2=0x7f760805d180 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f760805ad20 0x7f760805d180 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7608053a40 msgr2=0x7f7608055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7608053a40 0x7f7608055f00 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f761800b000 tx=0x7f76180091d0).stop 2022-07-20T03:07:17.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76281438e0 msgr2=0x7f762813e5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 --2- 172.21.15.61:0/3816065209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76281438e0 0x7f762813e5b0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f761c000ed0 tx=0x7f761c007a90).stop 2022-07-20T03:07:17.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762c8fb700 1 -- 172.21.15.61:0/3816065209 reap_dead start 2022-07-20T03:07:17.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 shutdown_connections 2022-07-20T03:07:17.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 >> 172.21.15.61:0/3816065209 conn(0x7f7628063150 msgr2=0x7f76280b5fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:17.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 shutdown_connections 2022-07-20T03:07:17.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.243+0000 7f762d0fc700 1 -- 172.21.15.61:0/3816065209 wait complete. 2022-07-20T03:07:17.507 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:17.508 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 flush_pg_stats 2022-07-20T03:07:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f8ca5700 1 Processor -- start 2022-07-20T03:07:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f8ca5700 1 -- start start 2022-07-20T03:07:17.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f8ca5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6f413f900 0x7fe6f41281f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f8ca5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f413fe00 con 0x7fe6f40b2db0 2022-07-20T03:07:17.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f8ca5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f41400a0 con 0x7fe6f40657d0 2022-07-20T03:07:17.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f8ca5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f4140340 con 0x7fe6f413f900 2022-07-20T03:07:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6f413f900 0x7fe6f41281f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe6f40b2db0 0x7fe6f4127ae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39110/0 (socket says 172.21.15.61:39110) 2022-07-20T03:07:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f259c700 1 -- 172.21.15.61:0/2937108201 learned_addr learned my addr 172.21.15.61:0/2937108201 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:17.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f1d9b700 1 --2- 172.21.15.61:0/2937108201 >> v2:172.21.15.61:3300/0 conn(0x7fe6f413f900 0x7fe6f41281f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42512/0 (socket says 172.21.15.61:42512) 2022-07-20T03:07:17.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f1d9b700 1 -- 172.21.15.61:0/2937108201 >> v1:172.21.15.117:6789/0 conn(0x7fe6f40657d0 legacy=0x7fe6f40b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f159a700 1 -- 172.21.15.61:0/2937108201 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 946892972 0 0) 0x7fe6f413fe00 con 0x7fe6f40b2db0 2022-07-20T03:07:17.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f1d9b700 1 -- 172.21.15.61:0/2937108201 >> v1:172.21.15.61:6789/0 conn(0x7fe6f40b2db0 legacy=0x7fe6f4127ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.619+0000 7fe6f1d9b700 1 -- 172.21.15.61:0/2937108201 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f40b3190 con 0x7fe6f413f900 2022-07-20T03:07:17.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f1d9b700 1 --2- 172.21.15.61:0/2937108201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f413f900 0x7fe6f41281f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe6ec001110 tx=0x7fe6dc0091d0).ready entity=mon.0 client_cookie=19111b8f800cc158 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f159a700 1 -- 172.21.15.61:0/2937108201 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6dc00b600 con 0x7fe6f413f900 2022-07-20T03:07:17.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f159a700 1 -- 172.21.15.61:0/2937108201 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6dc00b760 con 0x7fe6f413f900 2022-07-20T03:07:17.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f159a700 1 -- 172.21.15.61:0/2937108201 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6dc00b8d0 con 0x7fe6f413f900 2022-07-20T03:07:17.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/2937108201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f413f900 msgr2=0x7fe6f41281f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 --2- 172.21.15.61:0/2937108201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f413f900 0x7fe6f41281f0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fe6ec001110 tx=0x7fe6dc0091d0).stop 2022-07-20T03:07:17.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/2937108201 shutdown_connections 2022-07-20T03:07:17.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 --2- 172.21.15.61:0/2937108201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f413f900 0x7fe6f41281f0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/2937108201 >> 172.21.15.61:0/2937108201 conn(0x7fe6f4063bb0 msgr2=0x7fe6f4063f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:17.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/2937108201 shutdown_connections 2022-07-20T03:07:17.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.623+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/2937108201 wait complete. 2022-07-20T03:07:17.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 Processor -- start 2022-07-20T03:07:17.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 -- start start 2022-07-20T03:07:17.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f40657d0 0x7fe6f40ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f40b2db0 0x7fe6f41266f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f413f900 0x7fe6f41432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40af630 con 0x7fe6f40657d0 2022-07-20T03:07:17.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40aee10 con 0x7fe6f413f900 2022-07-20T03:07:17.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f40af0f0 con 0x7fe6f40b2db0 2022-07-20T03:07:17.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f40657d0 0x7fe6f40ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f40657d0 0x7fe6f40ad590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42518/0 (socket says 172.21.15.61:42518) 2022-07-20T03:07:17.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 -- 172.21.15.61:0/1059647537 learned_addr learned my addr 172.21.15.61:0/1059647537 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f1d9b700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f40b2db0 0x7fe6f41266f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f2d9d700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f413f900 0x7fe6f41432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 -- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f40b2db0 msgr2=0x7fe6f41266f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f40b2db0 0x7fe6f41266f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 -- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f413f900 msgr2=0x7fe6f41432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f413f900 0x7fe6f41432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 -- 172.21.15.61:0/1059647537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6dc009470 con 0x7fe6f40657d0 2022-07-20T03:07:17.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f2d9d700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f413f900 0x7fe6f41432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:17.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f259c700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f40657d0 0x7fe6f40ad590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe6e4000bc0 tx=0x7fe6e4007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6e40072e0 con 0x7fe6f40657d0 2022-07-20T03:07:17.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e4007440 con 0x7fe6f40657d0 2022-07-20T03:07:17.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f40af390 con 0x7fe6f40657d0 2022-07-20T03:07:17.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6e4005800 con 0x7fe6f40657d0 2022-07-20T03:07:17.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.627+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f4141a30 con 0x7fe6f40657d0 2022-07-20T03:07:17.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe6f40b02e0 con 0x7fe6f40657d0 2022-07-20T03:07:17.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe6e4005960 con 0x7fe6f40657d0 2022-07-20T03:07:17.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6eb7fe700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6cc053a40 0x7fe6cc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe6e406e1f0 con 0x7fe6f40657d0 2022-07-20T03:07:17.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6eb7fe700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7fe6cc05ada0 0x7fe6cc05d200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:17.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 --> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe6cc05d7e0 con 0x7fe6cc05ada0 2022-07-20T03:07:17.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe6e4070880 con 0x7fe6f40657d0 2022-07-20T03:07:17.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6f1d9b700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6cc053a40 0x7fe6cc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6f2d9d700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7fe6cc05ada0 0x7fe6cc05d200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:17.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6f2d9d700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7fe6cc05ada0 0x7fe6cc05d200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.631+0000 7fe6f1d9b700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6cc053a40 0x7fe6cc055f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fe6dc01b000 tx=0x7fe6dc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:17.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.635+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== osd.3 v2:172.21.15.61:6824/23120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fe6cc05d7e0 con 0x7fe6cc05ada0 2022-07-20T03:07:17.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 --> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fe6f40574e0 con 0x7fe6cc05ada0 2022-07-20T03:07:17.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6eb7fe700 1 -- 172.21.15.61:0/1059647537 <== osd.3 v2:172.21.15.61:6824/23120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fe6f40574e0 con 0x7fe6cc05ada0 2022-07-20T03:07:17.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7fe6cc05ada0 msgr2=0x7fe6cc05d200 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7fe6cc05ada0 0x7fe6cc05d200 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:17.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6cc053a40 msgr2=0x7fe6cc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6cc053a40 0x7fe6cc055f00 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fe6dc01b000 tx=0x7fe6dc016000).stop 2022-07-20T03:07:17.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f40657d0 msgr2=0x7fe6f40ad590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:17.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 --2- 172.21.15.61:0/1059647537 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f40657d0 0x7fe6f40ad590 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fe6e4000bc0 tx=0x7fe6e4007af0).stop 2022-07-20T03:07:17.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f2d9d700 1 -- 172.21.15.61:0/1059647537 reap_dead start 2022-07-20T03:07:17.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 shutdown_connections 2022-07-20T03:07:17.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 >> 172.21.15.61:0/1059647537 conn(0x7fe6f4063bb0 msgr2=0x7fe6f40b53c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:17.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 shutdown_connections 2022-07-20T03:07:17.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:17.651+0000 7fe6f8ca5700 1 -- 172.21.15.61:0/1059647537 wait complete. 2022-07-20T03:07:17.821 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:17.822 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 flush_pg_stats 2022-07-20T03:07:18.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 Processor -- start 2022-07-20T03:07:18.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 -- start start 2022-07-20T03:07:18.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1aa80b2db0 0x7f1aa8127ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 0x7f1aa80b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aa813fe00 con 0x7f1aa813f900 2022-07-20T03:07:18.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aa81400a0 con 0x7f1aa80b2db0 2022-07-20T03:07:18.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.035+0000 7f1aadcce700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa8140340 con 0x7f1aa80657d0 2022-07-20T03:07:18.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa77fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1aa80b2db0 0x7f1aa8127ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 0x7f1aa80b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 0x7f1aa80b0af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:57436/0 (socket says 172.21.15.61:57436) 2022-07-20T03:07:18.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa77fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1aa80b2db0 0x7f1aa8127ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:52208/0 (socket says 172.21.15.61:52208) 2022-07-20T03:07:18.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 -- 172.21.15.61:0/3973905542 learned_addr learned my addr 172.21.15.61:0/3973905542 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:18.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1a9effd700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1aa813f900 0x7f1aa8127fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:47632/0 (socket says 172.21.15.61:47632) 2022-07-20T03:07:18.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa6ffd700 1 -- 172.21.15.61:0/3973905542 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2622908687 0 0) 0x7f1aa813fe00 con 0x7f1aa813f900 2022-07-20T03:07:18.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa6ffd700 1 -- 172.21.15.61:0/3973905542 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a88003260 con 0x7f1aa813f900 2022-07-20T03:07:18.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 -- 172.21.15.61:0/3973905542 >> v2:172.21.15.117:3300/0 conn(0x7f1aa80b2db0 msgr2=0x7f1aa8127ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:18.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 --2- 172.21.15.61:0/3973905542 >> v2:172.21.15.117:3300/0 conn(0x7f1aa80b2db0 0x7f1aa8127ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 -- 172.21.15.61:0/3973905542 >> v1:172.21.15.117:6789/0 conn(0x7f1aa813f900 legacy=0x7f1aa8127fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 -- 172.21.15.61:0/3973905542 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa80b3190 con 0x7f1aa80657d0 2022-07-20T03:07:18.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa7fff700 1 --2- 172.21.15.61:0/3973905542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 0x7f1aa80b0af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f1aa00032f0 tx=0x7f1aa000b530).ready entity=mon.2 client_cookie=5fed4202a35e06dd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa6ffd700 1 -- 172.21.15.61:0/3973905542 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa000dd70 con 0x7f1aa80657d0 2022-07-20T03:07:18.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa6ffd700 1 -- 172.21.15.61:0/3973905542 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1aa0010470 con 0x7f1aa80657d0 2022-07-20T03:07:18.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aa6ffd700 1 -- 172.21.15.61:0/3973905542 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1aa00107d0 con 0x7f1aa80657d0 2022-07-20T03:07:18.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 -- 172.21.15.61:0/3973905542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 msgr2=0x7f1aa80b0af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 --2- 172.21.15.61:0/3973905542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 0x7f1aa80b0af0 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f1aa00032f0 tx=0x7f1aa000b530).stop 2022-07-20T03:07:18.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 -- 172.21.15.61:0/3973905542 shutdown_connections 2022-07-20T03:07:18.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 --2- 172.21.15.61:0/3973905542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa80657d0 0x7f1aa80b0af0 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 --2- 172.21.15.61:0/3973905542 >> v2:172.21.15.117:3300/0 conn(0x7f1aa80b2db0 0x7f1aa8127ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 -- 172.21.15.61:0/3973905542 >> 172.21.15.61:0/3973905542 conn(0x7f1aa8063bb0 msgr2=0x7f1aa8063f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:18.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 -- 172.21.15.61:0/3973905542 shutdown_connections 2022-07-20T03:07:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.039+0000 7f1aadcce700 1 -- 172.21.15.61:0/3973905542 wait complete. 2022-07-20T03:07:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 Processor -- start 2022-07-20T03:07:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 -- start start 2022-07-20T03:07:18.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aa80657d0 0x7f1aa80b4620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aa80b2db0 0x7f1aa80ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa813f900 0x7f1aa81433e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80b0260 con 0x7f1aa80b2db0 2022-07-20T03:07:18.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa80b0000 con 0x7f1aa80657d0 2022-07-20T03:07:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1aadcce700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aa8142f40 con 0x7f1aa813f900 2022-07-20T03:07:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1a9effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aa80b2db0 0x7f1aa80ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.043+0000 7f1a9effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aa80b2db0 0x7f1aa80ad590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42536/0 (socket says 172.21.15.61:42536) 2022-07-20T03:07:18.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1a9effd700 1 -- 172.21.15.61:0/217880075 learned_addr learned my addr 172.21.15.61:0/217880075 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aa80657d0 0x7f1aa80b4620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa7fff700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa813f900 0x7f1aa81433e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 -- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa813f900 msgr2=0x7f1aa81433e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aa813f900 0x7f1aa81433e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 -- 172.21.15.61:0/217880075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aa80b2db0 msgr2=0x7f1aa80ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aa80b2db0 0x7f1aa80ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 -- 172.21.15.61:0/217880075 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aa000b7d0 con 0x7f1aa80657d0 2022-07-20T03:07:18.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1a9effd700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aa80b2db0 0x7f1aa80ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:18.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa77fe700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aa80657d0 0x7f1aa80b4620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1aa80aff20 tx=0x7f1a98006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a98010780 con 0x7f1aa80657d0 2022-07-20T03:07:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a980108e0 con 0x7f1aa80657d0 2022-07-20T03:07:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aa80ada80 con 0x7f1aa80657d0 2022-07-20T03:07:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a98010c40 con 0x7f1aa80657d0 2022-07-20T03:07:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aa80aeaf0 con 0x7f1aa80657d0 2022-07-20T03:07:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1aa80af210 con 0x7f1aa80657d0 2022-07-20T03:07:18.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.047+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1a9801c460 con 0x7f1aa80657d0 2022-07-20T03:07:18.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa57fa700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a8c053ad0 0x7f1a8c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1a9806c210 con 0x7f1aa80657d0 2022-07-20T03:07:18.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1a9effd700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a8c053ad0 0x7f1a8c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa57fa700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f1a8c05adb0 0x7f1a8c05d210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 --> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1a8c05d7f0 con 0x7f1a8c05adb0 2022-07-20T03:07:18.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1a9806e8a0 con 0x7f1aa80657d0 2022-07-20T03:07:18.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa7fff700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f1a8c05adb0 0x7f1a8c05d210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1a9effd700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a8c053ad0 0x7f1a8c055f90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1a9000d000 tx=0x7f1a90007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa7fff700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f1a8c05adb0 0x7f1a8c05d210 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.051+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== osd.4 v2:172.21.15.117:6802/23184 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f1a8c05d7f0 con 0x7f1a8c05adb0 2022-07-20T03:07:18.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 --> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1aa8143b90 con 0x7f1a8c05adb0 2022-07-20T03:07:18.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aa57fa700 1 -- 172.21.15.61:0/217880075 <== osd.4 v2:172.21.15.117:6802/23184 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1aa8143b90 con 0x7f1a8c05adb0 2022-07-20T03:07:18.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f1a8c05adb0 msgr2=0x7f1a8c05d210 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f1a8c05adb0 0x7f1a8c05d210 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a8c053ad0 msgr2=0x7f1a8c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a8c053ad0 0x7f1a8c055f90 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f1a9000d000 tx=0x7f1a90007ad0).stop 2022-07-20T03:07:18.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aa80657d0 msgr2=0x7f1aa80b4620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 --2- 172.21.15.61:0/217880075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aa80657d0 0x7f1aa80b4620 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7f1aa80aff20 tx=0x7f1a98006df0).stop 2022-07-20T03:07:18.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aa7fff700 1 -- 172.21.15.61:0/217880075 reap_dead start 2022-07-20T03:07:18.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 shutdown_connections 2022-07-20T03:07:18.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 >> 172.21.15.61:0/217880075 conn(0x7f1aa8063bb0 msgr2=0x7f1aa805f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:18.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 shutdown_connections 2022-07-20T03:07:18.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.071+0000 7f1aadcce700 1 -- 172.21.15.61:0/217880075 wait complete. 2022-07-20T03:07:18.224 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:18.224 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 flush_pg_stats 2022-07-20T03:07:18.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.359+0000 7f1af4065700 1 Processor -- start 2022-07-20T03:07:18.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.359+0000 7f1af4065700 1 -- start start 2022-07-20T03:07:18.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1aec05c960 0x7f1aec060070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1aec146e20 0x7f1aec13c9e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1aec142f90 con 0x7f1aec13ced0 2022-07-20T03:07:18.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec05f840 con 0x7f1aec146e20 2022-07-20T03:07:18.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1aec05fae0 con 0x7f1aec05c960 2022-07-20T03:07:18.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1aec146e20 0x7f1aec13c9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1aec146e20 0x7f1aec13c9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42546/0 (socket says 172.21.15.61:42546) 2022-07-20T03:07:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 -- 172.21.15.61:0/860899040 learned_addr learned my addr 172.21.15.61:0/860899040 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1e01700 1 --2- 172.21.15.61:0/860899040 >> v2:172.21.15.117:3300/0 conn(0x7f1aec05c960 0x7f1aec060070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af0dff700 1 -- 172.21.15.61:0/860899040 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2739923166 0 0) 0x7f1aec142f90 con 0x7f1aec13ced0 2022-07-20T03:07:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af0dff700 1 -- 172.21.15.61:0/860899040 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1ad4003260 con 0x7f1aec13ced0 2022-07-20T03:07:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 -- 172.21.15.61:0/860899040 >> v2:172.21.15.117:3300/0 conn(0x7f1aec05c960 msgr2=0x7f1aec060070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 --2- 172.21.15.61:0/860899040 >> v2:172.21.15.117:3300/0 conn(0x7f1aec05c960 0x7f1aec060070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 -- 172.21.15.61:0/860899040 >> v1:172.21.15.117:6789/0 conn(0x7f1aec13ced0 legacy=0x7f1aec1433c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 -- 172.21.15.61:0/860899040 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1aec144050 con 0x7f1aec146e20 2022-07-20T03:07:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af1600700 1 --2- 172.21.15.61:0/860899040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec146e20 0x7f1aec13c9e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1aec0b2990 tx=0x7f1adc006eb0).ready entity=mon.0 client_cookie=31577fe189bc1198 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af0dff700 1 -- 172.21.15.61:0/860899040 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1adc00e6b0 con 0x7f1aec146e20 2022-07-20T03:07:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af0dff700 1 -- 172.21.15.61:0/860899040 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1adc000e90 con 0x7f1aec146e20 2022-07-20T03:07:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af0dff700 1 -- 172.21.15.61:0/860899040 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1adc00ea40 con 0x7f1aec146e20 2022-07-20T03:07:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 -- 172.21.15.61:0/860899040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec146e20 msgr2=0x7f1aec13c9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 --2- 172.21.15.61:0/860899040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec146e20 0x7f1aec13c9e0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f1aec0b2990 tx=0x7f1adc006eb0).stop 2022-07-20T03:07:18.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 -- 172.21.15.61:0/860899040 shutdown_connections 2022-07-20T03:07:18.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 --2- 172.21.15.61:0/860899040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec146e20 0x7f1aec13c9e0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 --2- 172.21.15.61:0/860899040 >> v2:172.21.15.117:3300/0 conn(0x7f1aec05c960 0x7f1aec060070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.363+0000 7f1af4065700 1 -- 172.21.15.61:0/860899040 >> 172.21.15.61:0/860899040 conn(0x7f1aec063bb0 msgr2=0x7f1aec063f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:18.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 -- 172.21.15.61:0/860899040 shutdown_connections 2022-07-20T03:07:18.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 -- 172.21.15.61:0/860899040 wait complete. 2022-07-20T03:07:18.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 Processor -- start 2022-07-20T03:07:18.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 -- start start 2022-07-20T03:07:18.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec05c960 0x7f1aec064e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aec13ced0 0x7f1aec065330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aec146e20 0x7f1aec139da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec13f720 con 0x7f1aec05c960 2022-07-20T03:07:18.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec156780 con 0x7f1aec146e20 2022-07-20T03:07:18.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af4065700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1aec1564e0 con 0x7f1aec13ced0 2022-07-20T03:07:18.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af1e01700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec05c960 0x7f1aec064e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af1e01700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec05c960 0x7f1aec064e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42550/0 (socket says 172.21.15.61:42550) 2022-07-20T03:07:18.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af1e01700 1 -- 172.21.15.61:0/3690597090 learned_addr learned my addr 172.21.15.61:0/3690597090 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:18.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af1600700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aec13ced0 0x7f1aec065330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.367+0000 7f1af2602700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aec146e20 0x7f1aec139da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af1e01700 1 -- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aec13ced0 msgr2=0x7f1aec065330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af1e01700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1aec13ced0 0x7f1aec065330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af1e01700 1 -- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aec146e20 msgr2=0x7f1aec139da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af1e01700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1aec146e20 0x7f1aec139da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af1e01700 1 -- 172.21.15.61:0/3690597090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1adc00b1e0 con 0x7f1aec05c960 2022-07-20T03:07:18.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af1e01700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec05c960 0x7f1aec064e40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f1aec140340 tx=0x7f1ae8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ae8010780 con 0x7f1aec05c960 2022-07-20T03:07:18.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ae80108e0 con 0x7f1aec05c960 2022-07-20T03:07:18.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1aec0afe40 con 0x7f1aec05c960 2022-07-20T03:07:18.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ae8010c40 con 0x7f1aec05c960 2022-07-20T03:07:18.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1aec0b02e0 con 0x7f1aec05c960 2022-07-20T03:07:18.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1aec0affd0 con 0x7f1aec05c960 2022-07-20T03:07:18.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1ae801c460 con 0x7f1aec05c960 2022-07-20T03:07:18.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.371+0000 7f1adaffd700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1acc053ad0 0x7f1acc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ae806c320 con 0x7f1aec05c960 2022-07-20T03:07:18.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1af1600700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1acc053ad0 0x7f1acc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1adaffd700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f1acc05ae30 0x7f1acc05d290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 --> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1acc05d870 con 0x7f1acc05ae30 2022-07-20T03:07:18.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ae806e990 con 0x7f1aec05c960 2022-07-20T03:07:18.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1af2602700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f1acc05ae30 0x7f1acc05d290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1af1600700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1acc053ad0 0x7f1acc055f90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1adc01c000 tx=0x7f1adc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1af2602700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f1acc05ae30 0x7f1acc05d290 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.375+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== osd.5 v2:172.21.15.117:6818/23294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f1acc05d870 con 0x7f1acc05ae30 2022-07-20T03:07:18.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 --> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f1aec13a550 con 0x7f1acc05ae30 2022-07-20T03:07:18.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1adaffd700 1 -- 172.21.15.61:0/3690597090 <== osd.5 v2:172.21.15.117:6818/23294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f1aec13a550 con 0x7f1acc05ae30 2022-07-20T03:07:18.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f1acc05ae30 msgr2=0x7f1acc05d290 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f1acc05ae30 0x7f1acc05d290 crc :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1acc053ad0 msgr2=0x7f1acc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1acc053ad0 0x7f1acc055f90 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f1adc01c000 tx=0x7f1adc015000).stop 2022-07-20T03:07:18.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec05c960 msgr2=0x7f1aec064e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.391+0000 7f1af4065700 1 --2- 172.21.15.61:0/3690597090 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1aec05c960 0x7f1aec064e40 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f1aec140340 tx=0x7f1ae8006df0).stop 2022-07-20T03:07:18.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.395+0000 7f1af2602700 1 -- 172.21.15.61:0/3690597090 reap_dead start 2022-07-20T03:07:18.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.395+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 shutdown_connections 2022-07-20T03:07:18.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.395+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 >> 172.21.15.61:0/3690597090 conn(0x7f1aec063bb0 msgr2=0x7f1aec13aed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:18.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.395+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 shutdown_connections 2022-07-20T03:07:18.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.395+0000 7f1af4065700 1 -- 172.21.15.61:0/3690597090 wait complete. 2022-07-20T03:07:18.616 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:18.625 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 flush_pg_stats 2022-07-20T03:07:18.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.751+0000 7fa6b6505700 1 Processor -- start 2022-07-20T03:07:18.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.751+0000 7fa6b6505700 1 -- start start 2022-07-20T03:07:18.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa6b00af8c0 0x7fa6b00ae810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa6b0143a10 0x7fa6b0143e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6b00b0b80 con 0x7fa6b00b5970 2022-07-20T03:07:18.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b005d4d0 con 0x7fa6b0143a10 2022-07-20T03:07:18.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa6b00b5d50 con 0x7fa6b00af8c0 2022-07-20T03:07:18.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa6b0143a10 0x7fa6b0143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa6b0143a10 0x7fa6b0143e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42564/0 (socket says 172.21.15.61:42564) 2022-07-20T03:07:18.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 -- 172.21.15.61:0/1638967317 learned_addr learned my addr 172.21.15.61:0/1638967317 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:18.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5503700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa6b00b5970 0x7fa6b00ae100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39160/0 (socket says 172.21.15.61:39160) 2022-07-20T03:07:18.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b4d02700 1 --2- 172.21.15.61:0/1638967317 >> v2:172.21.15.117:3300/0 conn(0x7fa6b00af8c0 0x7fa6b00ae810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 -- 172.21.15.61:0/1638967317 >> v2:172.21.15.117:3300/0 conn(0x7fa6b00af8c0 msgr2=0x7fa6b00ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:18.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 --2- 172.21.15.61:0/1638967317 >> v2:172.21.15.117:3300/0 conn(0x7fa6b00af8c0 0x7fa6b00ae810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 -- 172.21.15.61:0/1638967317 >> v1:172.21.15.61:6789/0 conn(0x7fa6b00b5970 legacy=0x7fa6b00ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 -- 172.21.15.61:0/1638967317 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6b00b0db0 con 0x7fa6b0143a10 2022-07-20T03:07:18.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b5d04700 1 --2- 172.21.15.61:0/1638967317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0143e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa6a8002f70 tx=0x7fa6a8006eb0).ready entity=mon.0 client_cookie=368b97f0331ab845 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6a7fff700 1 -- 172.21.15.61:0/1638967317 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a800d5c0 con 0x7fa6b0143a10 2022-07-20T03:07:18.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6a7fff700 1 -- 172.21.15.61:0/1638967317 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8000e90 con 0x7fa6b0143a10 2022-07-20T03:07:18.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6a7fff700 1 -- 172.21.15.61:0/1638967317 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a800d950 con 0x7fa6b0143a10 2022-07-20T03:07:18.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- 172.21.15.61:0/1638967317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 msgr2=0x7fa6b0143e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 --2- 172.21.15.61:0/1638967317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0143e50 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fa6a8002f70 tx=0x7fa6a8006eb0).stop 2022-07-20T03:07:18.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- 172.21.15.61:0/1638967317 shutdown_connections 2022-07-20T03:07:18.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 --2- 172.21.15.61:0/1638967317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0143e50 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 --2- 172.21.15.61:0/1638967317 >> v2:172.21.15.117:3300/0 conn(0x7fa6b00af8c0 0x7fa6b00ae810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- 172.21.15.61:0/1638967317 >> 172.21.15.61:0/1638967317 conn(0x7fa6b0063490 msgr2=0x7fa6b00b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:18.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- 172.21.15.61:0/1638967317 shutdown_connections 2022-07-20T03:07:18.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.755+0000 7fa6b6505700 1 -- 172.21.15.61:0/1638967317 wait complete. 2022-07-20T03:07:18.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 Processor -- start 2022-07-20T03:07:18.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 -- start start 2022-07-20T03:07:18.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b00af8c0 0x7fa6b01595e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b00b5970 0x7fa6b014bee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b0144350 con 0x7fa6b0143a10 2022-07-20T03:07:18.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b01461f0 con 0x7fa6b00b5970 2022-07-20T03:07:18.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6b01450e0 con 0x7fa6b00af8c0 2022-07-20T03:07:18.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42568/0 (socket says 172.21.15.61:42568) 2022-07-20T03:07:18.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 -- 172.21.15.61:0/3926601405 learned_addr learned my addr 172.21.15.61:0/3926601405 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:18.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5503700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b00af8c0 0x7fa6b01595e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b4d02700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b00b5970 0x7fa6b014bee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 -- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b00af8c0 msgr2=0x7fa6b01595e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa6b00af8c0 0x7fa6b01595e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 -- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b00b5970 msgr2=0x7fa6b014bee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b00b5970 0x7fa6b014bee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 -- 172.21.15.61:0/3926601405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6a800b1e0 con 0x7fa6b0143a10 2022-07-20T03:07:18.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b4d02700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6b00b5970 0x7fa6b014bee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:18.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b5d04700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0142630 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa6a8002f70 tx=0x7fa6a8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a80117e0 con 0x7fa6b0143a10 2022-07-20T03:07:18.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa6a8011940 con 0x7fa6b0143a10 2022-07-20T03:07:18.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa6b0144ce0 con 0x7fa6b0143a10 2022-07-20T03:07:18.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa6a8011ca0 con 0x7fa6b0143a10 2022-07-20T03:07:18.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.759+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6b0140dd0 con 0x7fa6b0143a10 2022-07-20T03:07:18.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fa6b0146070 con 0x7fa6b0143a10 2022-07-20T03:07:18.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa6a801a040 con 0x7fa6b0143a10 2022-07-20T03:07:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6a67fc700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa698053af0 0x7fa698055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa6a806ced0 con 0x7fa6b0143a10 2022-07-20T03:07:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6a67fc700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fa69805ae20 0x7fa69805d280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 --> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fa69805d860 con 0x7fa69805ae20 2022-07-20T03:07:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7fa6a806f470 con 0x7fa6b0143a10 2022-07-20T03:07:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6b5503700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa698053af0 0x7fa698055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6b4d02700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fa69805ae20 0x7fa69805d280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6b4d02700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fa69805ae20 0x7fa69805d280 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.763+0000 7fa6b5503700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa698053af0 0x7fa698055fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6a000b000 tx=0x7fa6a0007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.767+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== osd.6 v2:172.21.15.117:6810/23293 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fa69805d860 con 0x7fa69805ae20 2022-07-20T03:07:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 --> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7fa6b0145360 con 0x7fa69805ae20 2022-07-20T03:07:18.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6a67fc700 1 -- 172.21.15.61:0/3926601405 <== osd.6 v2:172.21.15.117:6810/23293 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7fa6b0145360 con 0x7fa69805ae20 2022-07-20T03:07:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fa69805ae20 msgr2=0x7fa69805d280 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fa69805ae20 0x7fa69805d280 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa698053af0 msgr2=0x7fa698055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa698053af0 0x7fa698055fb0 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fa6a000b000 tx=0x7fa6a0007160).stop 2022-07-20T03:07:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 msgr2=0x7fa6b0142630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 --2- 172.21.15.61:0/3926601405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa6b0143a10 0x7fa6b0142630 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7fa6a8002f70 tx=0x7fa6a8015000).stop 2022-07-20T03:07:18.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b5d04700 1 -- 172.21.15.61:0/3926601405 reap_dead start 2022-07-20T03:07:18.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 shutdown_connections 2022-07-20T03:07:18.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 >> 172.21.15.61:0/3926601405 conn(0x7fa6b0063490 msgr2=0x7fa6b00b2e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:18.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 shutdown_connections 2022-07-20T03:07:18.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.783+0000 7fa6b6505700 1 -- 172.21.15.61:0/3926601405 wait complete. 2022-07-20T03:07:18.837 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:18.838 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 flush_pg_stats 2022-07-20T03:07:19.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 Processor -- start 2022-07-20T03:07:19.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 -- start start 2022-07-20T03:07:19.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f33fc0b2170 0x7f33fc0b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f33fc0b2550 0x7f33fc0aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f33fc0b5020 con 0x7f33fc0b3d40 2022-07-20T03:07:19.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc0b3640 con 0x7f33fc0b2550 2022-07-20T03:07:19.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f34008be700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f33fc0add00 con 0x7f33fc0b2170 2022-07-20T03:07:19.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f33f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f33fc0b2550 0x7f33fc0aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f33f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f33fc0b2550 0x7f33fc0aeae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42576/0 (socket says 172.21.15.61:42576) 2022-07-20T03:07:19.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.951+0000 7f33f3fff700 1 -- 172.21.15.61:0/1439062977 learned_addr learned my addr 172.21.15.61:0/1439062977 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:19.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33fb7fe700 1 --2- 172.21.15.61:0/1439062977 >> v2:172.21.15.117:3300/0 conn(0x7f33fc0b2170 0x7f33fc0b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33f3fff700 1 -- 172.21.15.61:0/1439062977 >> v2:172.21.15.117:3300/0 conn(0x7f33fc0b2170 msgr2=0x7f33fc0b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:19.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33f3fff700 1 --2- 172.21.15.61:0/1439062977 >> v2:172.21.15.117:3300/0 conn(0x7f33fc0b2170 0x7f33fc0b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33f3fff700 1 -- 172.21.15.61:0/1439062977 >> v1:172.21.15.117:6789/0 conn(0x7f33fc0b3d40 legacy=0x7f33fc13e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:19.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33f3fff700 1 -- 172.21.15.61:0/1439062977 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33fc0adad0 con 0x7f33fc0b2550 2022-07-20T03:07:19.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33fb7fe700 1 --2- 172.21.15.61:0/1439062977 >> v2:172.21.15.117:3300/0 conn(0x7f33fc0b2170 0x7f33fc0b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:19.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33f3fff700 1 --2- 172.21.15.61:0/1439062977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc0aeae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f33fc0ad240 tx=0x7f33e4006eb0).ready entity=mon.0 client_cookie=6cbe40964a0d2525 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33faffd700 1 -- 172.21.15.61:0/1439062977 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33e400d5c0 con 0x7f33fc0b2550 2022-07-20T03:07:19.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33faffd700 1 -- 172.21.15.61:0/1439062977 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e4000e90 con 0x7f33fc0b2550 2022-07-20T03:07:19.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f33faffd700 1 -- 172.21.15.61:0/1439062977 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33e400d910 con 0x7f33fc0b2550 2022-07-20T03:07:19.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 -- 172.21.15.61:0/1439062977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 msgr2=0x7f33fc0aeae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 --2- 172.21.15.61:0/1439062977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc0aeae0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f33fc0ad240 tx=0x7f33e4006eb0).stop 2022-07-20T03:07:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 -- 172.21.15.61:0/1439062977 shutdown_connections 2022-07-20T03:07:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 --2- 172.21.15.61:0/1439062977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc0aeae0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 --2- 172.21.15.61:0/1439062977 >> v2:172.21.15.117:3300/0 conn(0x7f33fc0b2170 0x7f33fc0b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 -- 172.21.15.61:0/1439062977 >> 172.21.15.61:0/1439062977 conn(0x7f33fc063150 msgr2=0x7f33fc063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:19.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 -- 172.21.15.61:0/1439062977 shutdown_connections 2022-07-20T03:07:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.955+0000 7f34008be700 1 -- 172.21.15.61:0/1439062977 wait complete. 2022-07-20T03:07:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 Processor -- start 2022-07-20T03:07:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 -- start start 2022-07-20T03:07:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f33fc0b2170 0x7f33fc155940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc064ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0b3d40 0x7f33fc064fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc066cd0 con 0x7f33fc0b2550 2022-07-20T03:07:19.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc155f90 con 0x7f33fc0b2170 2022-07-20T03:07:19.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f33fc156230 con 0x7f33fc0b3d40 2022-07-20T03:07:19.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc064ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc064ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42584/0 (socket says 172.21.15.61:42584) 2022-07-20T03:07:19.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 -- 172.21.15.61:0/1753536750 learned_addr learned my addr 172.21.15.61:0/1753536750 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:19.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33fb7fe700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f33fc0b2170 0x7f33fc155940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33fbfff700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0b3d40 0x7f33fc064fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 -- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0b3d40 msgr2=0x7f33fc064fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f33fc0b3d40 0x7f33fc064fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 -- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f33fc0b2170 msgr2=0x7f33fc155940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f33fc0b2170 0x7f33fc155940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 -- 172.21.15.61:0/1753536750 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33e400b1e0 con 0x7f33fc0b2550 2022-07-20T03:07:19.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f3fff700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc064ab0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f33fc0ad240 tx=0x7f33e4015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33e4002f50 con 0x7f33fc0b2550 2022-07-20T03:07:19.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f33e400d940 con 0x7f33fc0b2550 2022-07-20T03:07:19.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33fc0669c0 con 0x7f33fc0b2550 2022-07-20T03:07:19.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.959+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f33e4011d70 con 0x7f33fc0b2550 2022-07-20T03:07:19.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33fc0a8df0 con 0x7f33fc0b2550 2022-07-20T03:07:19.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f33fc066b50 con 0x7f33fc0b2550 2022-07-20T03:07:19.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f33e4025030 con 0x7f33fc0b2550 2022-07-20T03:07:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33f97fa700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f33e00539b0 0x7f33e0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f33e406cfc0 con 0x7f33fc0b2550 2022-07-20T03:07:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33f97fa700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7f33e005ad10 0x7f33e005d170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33fb7fe700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f33e00539b0 0x7f33e0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 --> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f33e005d7f0 con 0x7f33e005ad10 2022-07-20T03:07:19.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=23) v2 ==== 24+0+0 (secure 0 0 0) 0x7f33e406f650 con 0x7f33fc0b2550 2022-07-20T03:07:19.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.963+0000 7f33fbfff700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7f33e005ad10 0x7f33e005d170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.967+0000 7f33fb7fe700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f33e00539b0 0x7f33e0055e70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f33ec00d000 tx=0x7f33ec008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.967+0000 7f33fbfff700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7f33e005ad10 0x7f33e005d170 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.967+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== osd.7 v2:172.21.15.117:6825/23295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f33e005d7f0 con 0x7f33e005ad10 2022-07-20T03:07:19.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 --> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] -- command(tid 2: {"prefix": "flush_pg_stats"}) v1 -- 0x7f33fc0b38a0 con 0x7f33e005ad10 2022-07-20T03:07:19.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f33f97fa700 1 -- 172.21.15.61:0/1753536750 <== osd.7 v2:172.21.15.117:6825/23295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+11 (crc 0 0 0) 0x7f33fc0b38a0 con 0x7f33e005ad10 2022-07-20T03:07:19.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7f33e005ad10 msgr2=0x7f33e005d170 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7f33e005ad10 0x7f33e005d170 crc :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f33e00539b0 msgr2=0x7f33e0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f33e00539b0 0x7f33e0055e70 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f33ec00d000 tx=0x7f33ec008040).stop 2022-07-20T03:07:19.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 msgr2=0x7f33fc064ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 --2- 172.21.15.61:0/1753536750 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f33fc0b2550 0x7f33fc064ab0 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f33fc0ad240 tx=0x7f33e4015000).stop 2022-07-20T03:07:19.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f33fbfff700 1 -- 172.21.15.61:0/1753536750 reap_dead start 2022-07-20T03:07:19.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 shutdown_connections 2022-07-20T03:07:19.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 >> 172.21.15.61:0/1753536750 conn(0x7f33fc063150 msgr2=0x7f33fc0abe60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:19.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 shutdown_connections 2022-07-20T03:07:19.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:18.983+0000 7f34008be700 1 -- 172.21.15.61:0/1753536750 wait complete. 2022-07-20T03:07:19.077 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:19.078 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.0 2022-07-20T03:07:19.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 Processor -- start 2022-07-20T03:07:19.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 -- start start 2022-07-20T03:07:19.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2364060ca0 0x7f2364061080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 0x7f236414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 0x7f236414b070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2364148430 con 0x7f23640afb80 2022-07-20T03:07:19.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f236414a750 con 0x7f2364060ca0 2022-07-20T03:07:19.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.191+0000 7f236a792700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23640b2840 con 0x7f2364097a80 2022-07-20T03:07:19.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2368d2f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 0x7f236414b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 0x7f236414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2363fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2364060ca0 0x7f2364061080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 0x7f236414ab80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:57498/0 (socket says 172.21.15.61:57498) 2022-07-20T03:07:19.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2368d2f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 0x7f236414b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42596/0 (socket says 172.21.15.61:42596) 2022-07-20T03:07:19.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 -- 172.21.15.61:0/807930504 learned_addr learned my addr 172.21.15.61:0/807930504 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:19.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 -- 172.21.15.61:0/807930504 >> v2:172.21.15.117:3300/0 conn(0x7f2364060ca0 msgr2=0x7f2364061080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:19.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 --2- 172.21.15.61:0/807930504 >> v2:172.21.15.117:3300/0 conn(0x7f2364060ca0 0x7f2364061080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 -- 172.21.15.61:0/807930504 >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 msgr2=0x7f236414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:19.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 --2- 172.21.15.61:0/807930504 >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 0x7f236414b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 -- 172.21.15.61:0/807930504 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23640ac110 con 0x7f2364097a80 2022-07-20T03:07:19.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2368d2f700 1 --2- 172.21.15.61:0/807930504 >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 0x7f236414b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:19.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f23637fe700 1 --2- 172.21.15.61:0/807930504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 0x7f236414ab80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f23640ab300 tx=0x7f2354009290).ready entity=mon.2 client_cookie=fcbe2cf0dffc260d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2362ffd700 1 -- 172.21.15.61:0/807930504 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f235400b680 con 0x7f2364097a80 2022-07-20T03:07:19.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2362ffd700 1 -- 172.21.15.61:0/807930504 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f235400b7e0 con 0x7f2364097a80 2022-07-20T03:07:19.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f2362ffd700 1 -- 172.21.15.61:0/807930504 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f235400b950 con 0x7f2364097a80 2022-07-20T03:07:19.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 -- 172.21.15.61:0/807930504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 msgr2=0x7f236414ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 --2- 172.21.15.61:0/807930504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 0x7f236414ab80 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7f23640ab300 tx=0x7f2354009290).stop 2022-07-20T03:07:19.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 -- 172.21.15.61:0/807930504 shutdown_connections 2022-07-20T03:07:19.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 --2- 172.21.15.61:0/807930504 >> v2:172.21.15.61:3300/0 conn(0x7f23640afb80 0x7f236414b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 --2- 172.21.15.61:0/807930504 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364097a80 0x7f236414ab80 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 --2- 172.21.15.61:0/807930504 >> v2:172.21.15.117:3300/0 conn(0x7f2364060ca0 0x7f2364061080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 -- 172.21.15.61:0/807930504 >> 172.21.15.61:0/807930504 conn(0x7f23640b5cb0 msgr2=0x7f23640b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 -- 172.21.15.61:0/807930504 shutdown_connections 2022-07-20T03:07:19.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.195+0000 7f236a792700 1 -- 172.21.15.61:0/807930504 wait complete. 2022-07-20T03:07:19.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 Processor -- start 2022-07-20T03:07:19.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 -- start start 2022-07-20T03:07:19.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364060ca0 0x7f2364149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 0x7f236414bdd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23640afb80 0x7f236414c2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640af630 con 0x7f2364097a80 2022-07-20T03:07:19.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23640ad9d0 con 0x7f23640afb80 2022-07-20T03:07:19.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f236a792700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23640af390 con 0x7f2364060ca0 2022-07-20T03:07:19.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f23637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 0x7f236414bdd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f23637fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 0x7f236414bdd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42600/0 (socket says 172.21.15.61:42600) 2022-07-20T03:07:19.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f23637fe700 1 -- 172.21.15.61:0/494055168 learned_addr learned my addr 172.21.15.61:0/494055168 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:19.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.199+0000 7f2368d2f700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23640afb80 0x7f236414c2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f2363fff700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364060ca0 0x7f2364149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23637fe700 1 -- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364060ca0 msgr2=0x7f2364149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23637fe700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364060ca0 0x7f2364149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23637fe700 1 -- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23640afb80 msgr2=0x7f236414c2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23637fe700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23640afb80 0x7f236414c2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23637fe700 1 -- 172.21.15.61:0/494055168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f235c00b180 con 0x7f2364097a80 2022-07-20T03:07:19.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23637fe700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 0x7f236414bdd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f23640ab300 tx=0x7f2354016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2354021620 con 0x7f2364097a80 2022-07-20T03:07:19.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2354021780 con 0x7f2364097a80 2022-07-20T03:07:19.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2354021ae0 con 0x7f2364097a80 2022-07-20T03:07:19.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2354009530 con 0x7f2364097a80 2022-07-20T03:07:19.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2364063bb0 con 0x7f2364097a80 2022-07-20T03:07:19.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23640abe40 con 0x7f2364097a80 2022-07-20T03:07:19.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f235400f070 con 0x7f2364097a80 2022-07-20T03:07:19.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23617fa700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f23440539b0 0x7f2344055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f235406d380 con 0x7f2364097a80 2022-07-20T03:07:19.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.203+0000 7f2363fff700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f23440539b0 0x7f2344055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.207+0000 7f2363fff700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f23440539b0 0x7f2344055e70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f235800b000 tx=0x7f2358006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.211+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2354044240 con 0x7f2364097a80 2022-07-20T03:07:19.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.339+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 0} v 0) v1 -- 0x7f23640574e0 con 0x7f2364097a80 2022-07-20T03:07:19.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f23617fa700 1 -- 172.21.15.61:0/494055168 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 0}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2354015070 con 0x7f2364097a80 2022-07-20T03:07:19.362 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:19.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f23440539b0 msgr2=0x7f2344055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f23440539b0 0x7f2344055e70 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f235800b000 tx=0x7f2358006df0).stop 2022-07-20T03:07:19.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 msgr2=0x7f236414bdd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 0x7f236414bdd0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f23640ab300 tx=0x7f2354016000).stop 2022-07-20T03:07:19.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 shutdown_connections 2022-07-20T03:07:19.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f23440539b0 0x7f2344055e70 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23640afb80 0x7f236414c2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2364097a80 0x7f236414bdd0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 --2- 172.21.15.61:0/494055168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2364060ca0 0x7f2364149b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 >> 172.21.15.61:0/494055168 conn(0x7f23640b5cb0 msgr2=0x7f236414aaa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:19.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 shutdown_connections 2022-07-20T03:07:19.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.343+0000 7f236a792700 1 -- 172.21.15.61:0/494055168 wait complete. 2022-07-20T03:07:19.380 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.0 2022-07-20T03:07:19.381 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.1 2022-07-20T03:07:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.507+0000 7f3a93c05700 1 Processor -- start 2022-07-20T03:07:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.507+0000 7f3a93c05700 1 -- start start 2022-07-20T03:07:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.507+0000 7f3a93c05700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c061080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c14a750 con 0x7f3a8c060ca0 2022-07-20T03:07:19.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a8c0b2840 con 0x7f3a8c0afb80 2022-07-20T03:07:19.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c148430 con 0x7f3a8c097a80 2022-07-20T03:07:19.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a919a1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3a8c060ca0 0x7f3a8c14ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39204/0 (socket says 172.21.15.61:39204) 2022-07-20T03:07:19.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a919a1700 1 -- 172.21.15.61:0/4127396621 learned_addr learned my addr 172.21.15.61:0/4127396621 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:19.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a911a0700 1 --2- 172.21.15.61:0/4127396621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c061080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a9099f700 1 -- 172.21.15.61:0/4127396621 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520551020 0 0) 0x7f3a8c14a750 con 0x7f3a8c060ca0 2022-07-20T03:07:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a9099f700 1 -- 172.21.15.61:0/4127396621 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3a70003260 con 0x7f3a8c060ca0 2022-07-20T03:07:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a9099f700 1 -- 172.21.15.61:0/4127396621 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1581756003 0 0) 0x7f3a8c0b2840 con 0x7f3a8c0afb80 2022-07-20T03:07:19.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a911a0700 1 -- 172.21.15.61:0/4127396621 >> v1:172.21.15.117:6789/0 conn(0x7f3a8c0afb80 legacy=0x7f3a8c146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a911a0700 1 -- 172.21.15.61:0/4127396621 >> v1:172.21.15.61:6789/0 conn(0x7f3a8c060ca0 legacy=0x7f3a8c14ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a911a0700 1 -- 172.21.15.61:0/4127396621 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a8c0ac110 con 0x7f3a8c097a80 2022-07-20T03:07:19.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a911a0700 1 --2- 172.21.15.61:0/4127396621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c061080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3a7c002f70 tx=0x7f3a7c006df0).ready entity=mon.2 client_cookie=f0c6ecf264c45eb6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a9099f700 1 -- 172.21.15.61:0/4127396621 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a7c00d570 con 0x7f3a8c097a80 2022-07-20T03:07:19.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a9099f700 1 -- 172.21.15.61:0/4127396621 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a7c000e90 con 0x7f3a8c097a80 2022-07-20T03:07:19.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a9099f700 1 -- 172.21.15.61:0/4127396621 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a7c00d8c0 con 0x7f3a8c097a80 2022-07-20T03:07:19.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- 172.21.15.61:0/4127396621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 msgr2=0x7f3a8c061080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 --2- 172.21.15.61:0/4127396621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c061080 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f3a7c002f70 tx=0x7f3a7c006df0).stop 2022-07-20T03:07:19.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- 172.21.15.61:0/4127396621 shutdown_connections 2022-07-20T03:07:19.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 --2- 172.21.15.61:0/4127396621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c061080 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- 172.21.15.61:0/4127396621 >> 172.21.15.61:0/4127396621 conn(0x7f3a8c0b5cb0 msgr2=0x7f3a8c0b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:19.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- 172.21.15.61:0/4127396621 shutdown_connections 2022-07-20T03:07:19.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.511+0000 7f3a93c05700 1 -- 172.21.15.61:0/4127396621 wait complete. 2022-07-20T03:07:19.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 Processor -- start 2022-07-20T03:07:19.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 -- start start 2022-07-20T03:07:19.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c060ca0 0x7f3a8c0aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c066150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 0x7f3a8c144c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c1561a0 con 0x7f3a8c0afb80 2022-07-20T03:07:19.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c063d30 con 0x7f3a8c060ca0 2022-07-20T03:07:19.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a8c063fd0 con 0x7f3a8c097a80 2022-07-20T03:07:19.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 0x7f3a8c144c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 0x7f3a8c144c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42616/0 (socket says 172.21.15.61:42616) 2022-07-20T03:07:19.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 -- 172.21.15.61:0/2895149823 learned_addr learned my addr 172.21.15.61:0/2895149823 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:19.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a911a0700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c066150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a919a1700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c060ca0 0x7f3a8c0aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 -- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 msgr2=0x7f3a8c066150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c066150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 -- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c060ca0 msgr2=0x7f3a8c0aea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c060ca0 0x7f3a8c0aea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 -- 172.21.15.61:0/2895149823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a7c00b180 con 0x7f3a8c0afb80 2022-07-20T03:07:19.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a921a2700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 0x7f3a8c144c20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3a88000bc0 tx=0x7f3a88007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a88001d10 con 0x7f3a8c0afb80 2022-07-20T03:07:19.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3a88007030 con 0x7f3a8c0afb80 2022-07-20T03:07:19.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a8c064270 con 0x7f3a8c0afb80 2022-07-20T03:07:19.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3a88007390 con 0x7f3a8c0afb80 2022-07-20T03:07:19.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.515+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a8c099bc0 con 0x7f3a8c0afb80 2022-07-20T03:07:19.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.519+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a8c063b90 con 0x7f3a8c0afb80 2022-07-20T03:07:19.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.519+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3a88010030 con 0x7f3a8c0afb80 2022-07-20T03:07:19.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.519+0000 7f3a82ffd700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a74053920 0x7f3a74055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:19.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.519+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3a8806ba00 con 0x7f3a8c0afb80 2022-07-20T03:07:19.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.519+0000 7f3a919a1700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a74053920 0x7f3a74055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:19.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.519+0000 7f3a919a1700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a74053920 0x7f3a74055de0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3a8400d000 tx=0x7f3a84009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:19.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.523+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3a8c063b90 con 0x7f3a8c0afb80 2022-07-20T03:07:19.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.647+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 1} v 0) v1 -- 0x7f3a8c063b90 con 0x7f3a8c0afb80 2022-07-20T03:07:19.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.647+0000 7f3a82ffd700 1 -- 172.21.15.61:0/2895149823 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 1}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f3a8c063b90 con 0x7f3a8c0afb80 2022-07-20T03:07:19.863 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:19.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a74053920 msgr2=0x7f3a74055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a74053920 0x7f3a74055de0 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f3a8400d000 tx=0x7f3a84009040).stop 2022-07-20T03:07:19.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 msgr2=0x7f3a8c144c20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:19.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 0x7f3a8c144c20 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f3a88000bc0 tx=0x7f3a88007be0).stop 2022-07-20T03:07:19.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 shutdown_connections 2022-07-20T03:07:19.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a74053920 0x7f3a74055de0 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a8c0afb80 0x7f3a8c144c20 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a8c097a80 0x7f3a8c066150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 --2- 172.21.15.61:0/2895149823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a8c060ca0 0x7f3a8c0aea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:19.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 >> 172.21.15.61:0/2895149823 conn(0x7f3a8c0b5cb0 msgr2=0x7f3a8c1460d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 shutdown_connections 2022-07-20T03:07:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:19.651+0000 7f3a93c05700 1 -- 172.21.15.61:0/2895149823 wait complete. 2022-07-20T03:07:19.874 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.1 2022-07-20T03:07:19.875 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.2 2022-07-20T03:07:20.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 Processor -- start 2022-07-20T03:07:20.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 -- start start 2022-07-20T03:07:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7cfc0b23d0 0x7f7cfc061b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc062010 0x7f7cfc1430b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7cfc060250 con 0x7f7cfc05c960 2022-07-20T03:07:20.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7cfc0b12c0 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc062650 con 0x7f7cfc062010 2022-07-20T03:07:20.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7cfc0b23d0 0x7f7cfc061b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7cfc0b23d0 0x7f7cfc061b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42622/0 (socket says 172.21.15.61:42622) 2022-07-20T03:07:20.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 -- 172.21.15.61:0/3265437884 learned_addr learned my addr 172.21.15.61:0/3265437884 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:20.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfad9d700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc062010 0x7f7cfc1430b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 -- 172.21.15.61:0/3265437884 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc062010 msgr2=0x7f7cfc1430b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:20.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc062010 0x7f7cfc1430b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 -- 172.21.15.61:0/3265437884 >> v1:172.21.15.117:6789/0 conn(0x7f7cfc05c960 legacy=0x7f7cfc0b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 -- 172.21.15.61:0/3265437884 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cfc05cd40 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf959a700 1 -- 172.21.15.61:0/3265437884 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2729740507 0 0) 0x7f7cfc060250 con 0x7f7cfc05c960 2022-07-20T03:07:20.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cfad9d700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc062010 0x7f7cfc1430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:20.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf9d9b700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc0b23d0 0x7f7cfc061b20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f7cfc096250 tx=0x7f7cf0006eb0).ready entity=mon.0 client_cookie=7e8b2acb4b224b5f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf959a700 1 -- 172.21.15.61:0/3265437884 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf000d5c0 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf959a700 1 -- 172.21.15.61:0/3265437884 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf0000e90 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.003+0000 7f7cf959a700 1 -- 172.21.15.61:0/3265437884 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf000d910 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3265437884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc0b23d0 msgr2=0x7f7cfc061b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc0b23d0 0x7f7cfc061b20 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f7cfc096250 tx=0x7f7cf0006eb0).stop 2022-07-20T03:07:20.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3265437884 shutdown_connections 2022-07-20T03:07:20.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc062010 0x7f7cfc1430b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3265437884 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc0b23d0 0x7f7cfc061b20 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3265437884 >> 172.21.15.61:0/3265437884 conn(0x7f7cfc0b6210 msgr2=0x7f7cfc0b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:20.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3265437884 shutdown_connections 2022-07-20T03:07:20.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3265437884 wait complete. 2022-07-20T03:07:20.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 Processor -- start 2022-07-20T03:07:20.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- start start 2022-07-20T03:07:20.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc05c960 0x7f7cfc1436c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cfc062010 0x7f7cfc155c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc0b23d0 0x7f7cfc156110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc156600 con 0x7f7cfc05c960 2022-07-20T03:07:20.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc09a670 con 0x7f7cfc062010 2022-07-20T03:07:20.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7cfc09a910 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc05c960 0x7f7cfc1436c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc05c960 0x7f7cfc1436c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42626/0 (socket says 172.21.15.61:42626) 2022-07-20T03:07:20.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfa59c700 1 -- 172.21.15.61:0/3012586625 learned_addr learned my addr 172.21.15.61:0/3012586625 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:20.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc0b23d0 0x7f7cfc156110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cf9d9b700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cfc062010 0x7f7cfc155c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 -- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cfc062010 msgr2=0x7f7cfc155c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cfc062010 0x7f7cfc155c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 -- 172.21.15.61:0/3012586625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc05c960 msgr2=0x7f7cfc1436c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc05c960 0x7f7cfc1436c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 -- 172.21.15.61:0/3012586625 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7cf4009470 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.007+0000 7f7cfad9d700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc0b23d0 0x7f7cfc156110 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f7cf400c000 tx=0x7f7cf40091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf400c6a0 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7cf000b1e0 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7cf400c800 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7cf400cb60 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7cfc143bb0 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7cfc094a90 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7cf400ccc0 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7ceb7fe700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7cd8053af0 0x7f7cd8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7cf406c890 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7cfa59c700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7cd8053af0 0x7f7cd8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.011+0000 7f7cfa59c700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7cd8053af0 0x7f7cd8055fb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7cec00d000 tx=0x7f7cec007ac0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.023+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7cf4043660 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.147+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 2} v 0) v1 -- 0x7f7cfc060480 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.147+0000 7f7ceb7fe700 1 -- 172.21.15.61:0/3012586625 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 2}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f7cf4016090 con 0x7f7cfc0b23d0 2022-07-20T03:07:20.230 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:20.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7cd8053af0 msgr2=0x7f7cd8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7cd8053af0 0x7f7cd8055fb0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f7cec00d000 tx=0x7f7cec007ac0).stop 2022-07-20T03:07:20.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc0b23d0 msgr2=0x7f7cfc156110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc0b23d0 0x7f7cfc156110 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f7cf400c000 tx=0x7f7cf40091d0).stop 2022-07-20T03:07:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 shutdown_connections 2022-07-20T03:07:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7cd8053af0 0x7f7cd8055fb0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7cfc062010 0x7f7cfc155c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7cfc0b23d0 0x7f7cfc156110 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 --2- 172.21.15.61:0/3012586625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7cfc05c960 0x7f7cfc1436c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 >> 172.21.15.61:0/3012586625 conn(0x7f7cfc0b6210 msgr2=0x7f7cfc066cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:20.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 shutdown_connections 2022-07-20T03:07:20.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.151+0000 7f7cfb59e700 1 -- 172.21.15.61:0/3012586625 wait complete. 2022-07-20T03:07:20.235 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.2 2022-07-20T03:07:20.235 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.3 2022-07-20T03:07:20.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.343+0000 7f6aa6a77700 1 Processor -- start 2022-07-20T03:07:20.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.343+0000 7f6aa6a77700 1 -- start start 2022-07-20T03:07:20.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6aa00afb80 0x7f6aa0142080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa0146340 con 0x7f6aa0060fe0 2022-07-20T03:07:20.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6aa00613c0 con 0x7f6aa00b02a0 2022-07-20T03:07:20.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6aa00ac110 con 0x7f6aa00afb80 2022-07-20T03:07:20.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa4813700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6aa0060fe0 0x7f6aa00b08b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39232/0 (socket says 172.21.15.61:39232) 2022-07-20T03:07:20.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa4813700 1 -- 172.21.15.61:0/3629742721 learned_addr learned my addr 172.21.15.61:0/3629742721 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:20.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa5014700 1 --2- 172.21.15.61:0/3629742721 >> v2:172.21.15.117:3300/0 conn(0x7f6aa00afb80 0x7f6aa0142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6a9f7fe700 1 -- 172.21.15.61:0/3629742721 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1599239776 0 0) 0x7f6aa0146340 con 0x7f6aa0060fe0 2022-07-20T03:07:20.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6a9f7fe700 1 -- 172.21.15.61:0/3629742721 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6a80003260 con 0x7f6aa0060fe0 2022-07-20T03:07:20.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa5014700 1 -- 172.21.15.61:0/3629742721 >> v1:172.21.15.117:6789/0 conn(0x7f6aa00b02a0 legacy=0x7f6aa0146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6a9f7fe700 1 -- 172.21.15.61:0/3629742721 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 678220748 0 0) 0x7f6aa00613c0 con 0x7f6aa00b02a0 2022-07-20T03:07:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa5014700 1 -- 172.21.15.61:0/3629742721 >> v1:172.21.15.61:6789/0 conn(0x7f6aa0060fe0 legacy=0x7f6aa00b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa5014700 1 -- 172.21.15.61:0/3629742721 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6aa0146570 con 0x7f6aa00afb80 2022-07-20T03:07:20.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa5014700 1 --2- 172.21.15.61:0/3629742721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa00afb80 0x7f6aa0142080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6a98002f70 tx=0x7f6a98006df0).ready entity=mon.1 client_cookie=abb5a05b1391a8a0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6a9f7fe700 1 -- 172.21.15.61:0/3629742721 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a9800d610 con 0x7f6aa00afb80 2022-07-20T03:07:20.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6a9f7fe700 1 -- 172.21.15.61:0/3629742721 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a98000e90 con 0x7f6aa00afb80 2022-07-20T03:07:20.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6a9f7fe700 1 -- 172.21.15.61:0/3629742721 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a9800d960 con 0x7f6aa00afb80 2022-07-20T03:07:20.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- 172.21.15.61:0/3629742721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa00afb80 msgr2=0x7f6aa0142080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/3629742721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa00afb80 0x7f6aa0142080 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f6a98002f70 tx=0x7f6a98006df0).stop 2022-07-20T03:07:20.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- 172.21.15.61:0/3629742721 shutdown_connections 2022-07-20T03:07:20.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/3629742721 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa00afb80 0x7f6aa0142080 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- 172.21.15.61:0/3629742721 >> 172.21.15.61:0/3629742721 conn(0x7f6aa00b5cb0 msgr2=0x7f6aa00b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:20.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- 172.21.15.61:0/3629742721 shutdown_connections 2022-07-20T03:07:20.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.347+0000 7f6aa6a77700 1 -- 172.21.15.61:0/3629742721 wait complete. 2022-07-20T03:07:20.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 Processor -- start 2022-07-20T03:07:20.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 -- start start 2022-07-20T03:07:20.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa0060fe0 0x7f6aa014a100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 0x7f6aa0155e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aa00b02a0 0x7f6aa00aecb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa00aeab0 con 0x7f6aa00afb80 2022-07-20T03:07:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa0149630 con 0x7f6aa0060fe0 2022-07-20T03:07:20.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6aa01556d0 con 0x7f6aa00b02a0 2022-07-20T03:07:20.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 0x7f6aa0155e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 0x7f6aa0155e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42642/0 (socket says 172.21.15.61:42642) 2022-07-20T03:07:20.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 -- 172.21.15.61:0/1401554491 learned_addr learned my addr 172.21.15.61:0/1401554491 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:20.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa5014700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aa00b02a0 0x7f6aa00aecb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa4813700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa0060fe0 0x7f6aa014a100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 -- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aa00b02a0 msgr2=0x7f6aa00aecb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aa00b02a0 0x7f6aa00aecb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 -- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa0060fe0 msgr2=0x7f6aa014a100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa0060fe0 0x7f6aa014a100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 -- 172.21.15.61:0/1401554491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a9800b180 con 0x7f6aa00afb80 2022-07-20T03:07:20.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9ffff700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 0x7f6aa0155e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f6a90000f40 tx=0x7f6a9000c880).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a90006aa0 con 0x7f6aa00afb80 2022-07-20T03:07:20.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a90006c00 con 0x7f6aa00afb80 2022-07-20T03:07:20.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a90007030 con 0x7f6aa00afb80 2022-07-20T03:07:20.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6aa0155970 con 0x7f6aa00afb80 2022-07-20T03:07:20.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.351+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6aa00af5e0 con 0x7f6aa00afb80 2022-07-20T03:07:20.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.355+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6a90010030 con 0x7f6aa00afb80 2022-07-20T03:07:20.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.355+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6aa0155b00 con 0x7f6aa00afb80 2022-07-20T03:07:20.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.355+0000 7f6a9dffb700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a84053a60 0x7f6a84055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.355+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a9006bda0 con 0x7f6aa00afb80 2022-07-20T03:07:20.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.355+0000 7f6aa4813700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a84053a60 0x7f6a84055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.355+0000 7f6aa4813700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a84053a60 0x7f6a84055f20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6a8800b000 tx=0x7f6a88009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.359+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a90042d00 con 0x7f6aa00afb80 2022-07-20T03:07:20.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.487+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 3} v 0) v1 -- 0x7f6aa00abe40 con 0x7f6aa00afb80 2022-07-20T03:07:20.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.487+0000 7f6a9dffb700 1 -- 172.21.15.61:0/1401554491 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 3}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f6a90046320 con 0x7f6aa00afb80 2022-07-20T03:07:20.579 INFO:teuthology.orchestra.run.smithi061.stdout:55834574853 2022-07-20T03:07:20.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a84053a60 msgr2=0x7f6a84055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a84053a60 0x7f6a84055f20 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f6a8800b000 tx=0x7f6a88009040).stop 2022-07-20T03:07:20.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 msgr2=0x7f6aa0155e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 0x7f6aa0155e10 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f6a90000f40 tx=0x7f6a9000c880).stop 2022-07-20T03:07:20.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 shutdown_connections 2022-07-20T03:07:20.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a84053a60 0x7f6a84055f20 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6aa00b02a0 0x7f6aa00aecb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6aa00afb80 0x7f6aa0155e10 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 --2- 172.21.15.61:0/1401554491 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6aa0060fe0 0x7f6aa014a100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 >> 172.21.15.61:0/1401554491 conn(0x7f6aa00b5cb0 msgr2=0x7f6aa0144240 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:20.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 shutdown_connections 2022-07-20T03:07:20.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.491+0000 7f6aa6a77700 1 -- 172.21.15.61:0/1401554491 wait complete. 2022-07-20T03:07:20.627 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574853 for osd.3 2022-07-20T03:07:20.628 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.4 2022-07-20T03:07:20.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 Processor -- start 2022-07-20T03:07:20.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 -- start start 2022-07-20T03:07:20.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d80b08b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa8d80b02a0 0x7fa8d8146a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8d8146340 con 0x7fa8d80afb80 2022-07-20T03:07:20.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa8d80613c0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8dde73700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80ac110 con 0x7fa8d8060fe0 2022-07-20T03:07:20.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8d6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa8d80b02a0 0x7fa8d8146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8d6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa8d80b02a0 0x7fa8d8146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42648/0 (socket says 172.21.15.61:42648) 2022-07-20T03:07:20.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8d6ffd700 1 -- 172.21.15.61:0/7092993 learned_addr learned my addr 172.21.15.61:0/7092993 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:20.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.767+0000 7fa8d77fe700 1 --2- 172.21.15.61:0/7092993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d80b08b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d6ffd700 1 -- 172.21.15.61:0/7092993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 msgr2=0x7fa8d80b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:20.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d6ffd700 1 --2- 172.21.15.61:0/7092993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d80b08b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d6ffd700 1 -- 172.21.15.61:0/7092993 >> v1:172.21.15.117:6789/0 conn(0x7fa8d80afb80 legacy=0x7fa8d8142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d6ffd700 1 -- 172.21.15.61:0/7092993 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8d8146570 con 0x7fa8d80b02a0 2022-07-20T03:07:20.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d6ffd700 1 --2- 172.21.15.61:0/7092993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8146a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa8c80032f0 tx=0x7fa8c8009290).ready entity=mon.0 client_cookie=3d21a9196543afe0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d67fc700 1 -- 172.21.15.61:0/7092993 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c800b8a0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d67fc700 1 -- 172.21.15.61:0/7092993 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c800ba00 con 0x7fa8d80b02a0 2022-07-20T03:07:20.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8d67fc700 1 -- 172.21.15.61:0/7092993 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c800bb70 con 0x7fa8d80b02a0 2022-07-20T03:07:20.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 -- 172.21.15.61:0/7092993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 msgr2=0x7fa8d8146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 --2- 172.21.15.61:0/7092993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8146a10 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa8c80032f0 tx=0x7fa8c8009290).stop 2022-07-20T03:07:20.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 -- 172.21.15.61:0/7092993 shutdown_connections 2022-07-20T03:07:20.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 --2- 172.21.15.61:0/7092993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8146a10 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 --2- 172.21.15.61:0/7092993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d80b08b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 -- 172.21.15.61:0/7092993 >> 172.21.15.61:0/7092993 conn(0x7fa8d80b5cb0 msgr2=0x7fa8d80b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:20.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 -- 172.21.15.61:0/7092993 shutdown_connections 2022-07-20T03:07:20.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 -- 172.21.15.61:0/7092993 wait complete. 2022-07-20T03:07:20.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 Processor -- start 2022-07-20T03:07:20.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.771+0000 7fa8dde73700 1 -- start start 2022-07-20T03:07:20.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d8143130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8d80afb80 0x7fa8d8067070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8067560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80acd00 con 0x7fa8d80b02a0 2022-07-20T03:07:20.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80ae520 con 0x7fa8d80afb80 2022-07-20T03:07:20.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8d80ae7b0 con 0x7fa8d8060fe0 2022-07-20T03:07:20.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8067560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42656/0 (socket says 172.21.15.61:42656) 2022-07-20T03:07:20.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 -- 172.21.15.61:0/3233008273 learned_addr learned my addr 172.21.15.61:0/3233008273 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:20.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d77fe700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d8143130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d6ffd700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8d80afb80 0x7fa8d8067070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 -- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 msgr2=0x7fa8d8143130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d8143130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 -- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8d80afb80 msgr2=0x7fa8d8067070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8d80afb80 0x7fa8d8067070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 -- 172.21.15.61:0/3233008273 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8009530 con 0x7fa8d80b02a0 2022-07-20T03:07:20.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d6ffd700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8d80afb80 0x7fa8d8067070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:20.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d7fff700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8067560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa8d80b2d80 tx=0x7fa8d0007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d0012400 con 0x7fa8d80b02a0 2022-07-20T03:07:20.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8d80ad5e0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8d0012560 con 0x7fa8d80b02a0 2022-07-20T03:07:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8d00128c0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8d8067a50 con 0x7fa8d80b02a0 2022-07-20T03:07:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8d80abe40 con 0x7fa8d80b02a0 2022-07-20T03:07:20.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa8d0019030 con 0x7fa8d80b02a0 2022-07-20T03:07:20.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d4ff9700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8b8053ad0 0x7fa8b8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:20.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.775+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa8d006b2e0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.779+0000 7fa8d77fe700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8b8053ad0 0x7fa8b8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:20.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.779+0000 7fa8d77fe700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8b8053ad0 0x7fa8b8055f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa8cc00b000 tx=0x7fa8cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:20.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.783+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8d00421a0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.907+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 4} v 0) v1 -- 0x7fa8d80574e0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8d4ff9700 1 -- 172.21.15.61:0/3233008273 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 4}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fa8d00457c0 con 0x7fa8d80b02a0 2022-07-20T03:07:20.929 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:20.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8b8053ad0 msgr2=0x7fa8b8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8b8053ad0 0x7fa8b8055f90 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fa8cc00b000 tx=0x7fa8cc006df0).stop 2022-07-20T03:07:20.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 msgr2=0x7fa8d8067560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:20.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8067560 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fa8d80b2d80 tx=0x7fa8d0007b90).stop 2022-07-20T03:07:20.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 shutdown_connections 2022-07-20T03:07:20.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8b8053ad0 0x7fa8b8055f90 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8d80afb80 0x7fa8d8067070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8d80b02a0 0x7fa8d8067560 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 --2- 172.21.15.61:0/3233008273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8d8060fe0 0x7fa8d8143130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:20.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 >> 172.21.15.61:0/3233008273 conn(0x7fa8d80b5cb0 msgr2=0x7fa8d80aafc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:20.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 shutdown_connections 2022-07-20T03:07:20.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:20.911+0000 7fa8dde73700 1 -- 172.21.15.61:0/3233008273 wait complete. 2022-07-20T03:07:20.947 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574854 for osd.4 2022-07-20T03:07:20.948 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.5 2022-07-20T03:07:21.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3f0d2700 1 Processor -- start 2022-07-20T03:07:21.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3f0d2700 1 -- start start 2022-07-20T03:07:21.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3f0d2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fed3814f9d0 0x7fed38151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3f0d2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed380b0b80 con 0x7fed380af8c0 2022-07-20T03:07:21.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3f0d2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed3805d4d0 con 0x7fed380b5970 2022-07-20T03:07:21.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3f0d2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed380b5d50 con 0x7fed3814f9d0 2022-07-20T03:07:21.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d8cf700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fed380af8c0 0x7fed380ae810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39260/0 (socket says 172.21.15.61:39260) 2022-07-20T03:07:21.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d8cf700 1 -- 172.21.15.61:0/570727307 learned_addr learned my addr 172.21.15.61:0/570727307 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:21.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3e8d1700 1 --2- 172.21.15.61:0/570727307 >> v2:172.21.15.117:3300/0 conn(0x7fed3814f9d0 0x7fed38151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1817268922 0 0) 0x7fed380b0b80 con 0x7fed380af8c0 2022-07-20T03:07:21.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed24003260 con 0x7fed380af8c0 2022-07-20T03:07:21.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3346382846 0 0) 0x7fed3805d4d0 con 0x7fed380b5970 2022-07-20T03:07:21.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed380b0b80 con 0x7fed380b5970 2022-07-20T03:07:21.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3e8d1700 1 -- 172.21.15.61:0/570727307 >> v1:172.21.15.117:6789/0 conn(0x7fed380b5970 legacy=0x7fed380ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3e8d1700 1 -- 172.21.15.61:0/570727307 >> v1:172.21.15.61:6789/0 conn(0x7fed380af8c0 legacy=0x7fed380ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 977371038 0 0) 0x7fed24003260 con 0x7fed380af8c0 2022-07-20T03:07:21.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3e8d1700 1 -- 172.21.15.61:0/570727307 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed380b0db0 con 0x7fed3814f9d0 2022-07-20T03:07:21.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3e8d1700 1 --2- 172.21.15.61:0/570727307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed3814f9d0 0x7fed38151dc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fed28001490 tx=0x7fed30006df0).ready entity=mon.1 client_cookie=6062824d4528bc5e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed280019e0 con 0x7fed3814f9d0 2022-07-20T03:07:21.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed30000e90 con 0x7fed3814f9d0 2022-07-20T03:07:21.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.059+0000 7fed3d0ce700 1 -- 172.21.15.61:0/570727307 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed30012460 con 0x7fed3814f9d0 2022-07-20T03:07:21.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- 172.21.15.61:0/570727307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed3814f9d0 msgr2=0x7fed38151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/570727307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed3814f9d0 0x7fed38151dc0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fed28001490 tx=0x7fed30006df0).stop 2022-07-20T03:07:21.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- 172.21.15.61:0/570727307 shutdown_connections 2022-07-20T03:07:21.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/570727307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed3814f9d0 0x7fed38151dc0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- 172.21.15.61:0/570727307 >> 172.21.15.61:0/570727307 conn(0x7fed38063490 msgr2=0x7fed380b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:21.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- 172.21.15.61:0/570727307 shutdown_connections 2022-07-20T03:07:21.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- 172.21.15.61:0/570727307 wait complete. 2022-07-20T03:07:21.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 Processor -- start 2022-07-20T03:07:21.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- start start 2022-07-20T03:07:21.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed380af8c0 0x7fed38142390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 0x7fed3810f030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3814f9d0 0x7fed38111520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed381107f0 con 0x7fed380b5970 2022-07-20T03:07:21.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed3810e420 con 0x7fed380af8c0 2022-07-20T03:07:21.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3f0d2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed38110150 con 0x7fed3814f9d0 2022-07-20T03:07:21.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3d8cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 0x7fed3810f030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3d8cf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 0x7fed3810f030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42668/0 (socket says 172.21.15.61:42668) 2022-07-20T03:07:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.063+0000 7fed3d8cf700 1 -- 172.21.15.61:0/4060442211 learned_addr learned my addr 172.21.15.61:0/4060442211 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed380af8c0 0x7fed38142390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e8d1700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3814f9d0 0x7fed38111520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 -- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3814f9d0 msgr2=0x7fed38111520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3814f9d0 0x7fed38111520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 -- 172.21.15.61:0/4060442211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 msgr2=0x7fed3810f030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 0x7fed3810f030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 -- 172.21.15.61:0/4060442211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed3000b180 con 0x7fed380af8c0 2022-07-20T03:07:21.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3d8cf700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 0x7fed3810f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:21.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3e0d0700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed380af8c0 0x7fed38142390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fed3810e560 tx=0x7fed2c007a20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c012400 con 0x7fed380af8c0 2022-07-20T03:07:21.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c012560 con 0x7fed380af8c0 2022-07-20T03:07:21.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed38063b60 con 0x7fed380af8c0 2022-07-20T03:07:21.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c0128c0 con 0x7fed380af8c0 2022-07-20T03:07:21.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed38142880 con 0x7fed380af8c0 2022-07-20T03:07:21.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fed2c019030 con 0x7fed380af8c0 2022-07-20T03:07:21.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed3813ffd0 con 0x7fed380af8c0 2022-07-20T03:07:21.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed377fe700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed18053a60 0x7fed18055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.067+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fed2c06b160 con 0x7fed380af8c0 2022-07-20T03:07:21.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.071+0000 7fed3d8cf700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed18053a60 0x7fed18055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.071+0000 7fed3d8cf700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed18053a60 0x7fed18055f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fed2800d000 tx=0x7fed28000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.075+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed2c042020 con 0x7fed380af8c0 2022-07-20T03:07:21.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.203+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 5} v 0) v1 -- 0x7fed380acae0 con 0x7fed380af8c0 2022-07-20T03:07:21.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed377fe700 1 -- 172.21.15.61:0/4060442211 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 5}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7fed2c045640 con 0x7fed380af8c0 2022-07-20T03:07:21.224 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:21.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed18053a60 msgr2=0x7fed18055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed18053a60 0x7fed18055f20 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fed2800d000 tx=0x7fed28000f40).stop 2022-07-20T03:07:21.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed380af8c0 msgr2=0x7fed38142390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed380af8c0 0x7fed38142390 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7fed3810e560 tx=0x7fed2c007a20).stop 2022-07-20T03:07:21.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 shutdown_connections 2022-07-20T03:07:21.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed18053a60 0x7fed18055f20 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3814f9d0 0x7fed38111520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed380b5970 0x7fed3810f030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 --2- 172.21.15.61:0/4060442211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed380af8c0 0x7fed38142390 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 >> 172.21.15.61:0/4060442211 conn(0x7fed38063490 msgr2=0x7fed380ae9f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:21.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 shutdown_connections 2022-07-20T03:07:21.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.207+0000 7fed3f0d2700 1 -- 172.21.15.61:0/4060442211 wait complete. 2022-07-20T03:07:21.391 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.5 2022-07-20T03:07:21.399 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.6 2022-07-20T03:07:21.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 Processor -- start 2022-07-20T03:07:21.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 -- start start 2022-07-20T03:07:21.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca005c7c0 0x7f8ca00ac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8ca00b23c0 0x7f8ca00ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ca00ab250 0x7f8ca00acd40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca0096440 con 0x7f8ca00ab250 2022-07-20T03:07:21.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ca0098cf0 con 0x7f8ca00b23c0 2022-07-20T03:07:21.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00ac0f0 con 0x7f8ca005c7c0 2022-07-20T03:07:21.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ca00ab250 0x7f8ca00acd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ca00ab250 0x7f8ca00acd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42676/0 (socket says 172.21.15.61:42676) 2022-07-20T03:07:21.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 -- 172.21.15.61:0/3685502770 learned_addr learned my addr 172.21.15.61:0/3685502770 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:21.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9e59c700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca005c7c0 0x7f8ca00ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9dd9b700 1 --2- 172.21.15.61:0/3685502770 >> v2:172.21.15.117:3300/0 conn(0x7f8ca00b23c0 0x7f8ca00ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 -- 172.21.15.61:0/3685502770 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca005c7c0 msgr2=0x7f8ca00ac360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:21.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca005c7c0 0x7f8ca00ac360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 -- 172.21.15.61:0/3685502770 >> v2:172.21.15.117:3300/0 conn(0x7f8ca00b23c0 msgr2=0x7f8ca00ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:21.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 --2- 172.21.15.61:0/3685502770 >> v2:172.21.15.117:3300/0 conn(0x7f8ca00b23c0 0x7f8ca00ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 -- 172.21.15.61:0/3685502770 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ca0095bc0 con 0x7f8ca00ab250 2022-07-20T03:07:21.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9dd9b700 1 --2- 172.21.15.61:0/3685502770 >> v2:172.21.15.117:3300/0 conn(0x7f8ca00b23c0 0x7f8ca00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:21.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9e59c700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca005c7c0 0x7f8ca00ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:21.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9ed9d700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca00ab250 0x7f8ca00acd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f8c90002f70 tx=0x7f8c90006eb0).ready entity=mon.0 client_cookie=41fb9fa6556f6243 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9d59a700 1 -- 172.21.15.61:0/3685502770 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c9000d570 con 0x7f8ca00ab250 2022-07-20T03:07:21.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9d59a700 1 -- 172.21.15.61:0/3685502770 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c90000e90 con 0x7f8ca00ab250 2022-07-20T03:07:21.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.515+0000 7f8c9d59a700 1 -- 172.21.15.61:0/3685502770 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c9000d900 con 0x7f8ca00ab250 2022-07-20T03:07:21.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- 172.21.15.61:0/3685502770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca00ab250 msgr2=0x7f8ca00acd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca00ab250 0x7f8ca00acd40 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f8c90002f70 tx=0x7f8c90006eb0).stop 2022-07-20T03:07:21.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- 172.21.15.61:0/3685502770 shutdown_connections 2022-07-20T03:07:21.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca00ab250 0x7f8ca00acd40 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/3685502770 >> v2:172.21.15.117:3300/0 conn(0x7f8ca00b23c0 0x7f8ca00ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/3685502770 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca005c7c0 0x7f8ca00ac360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- 172.21.15.61:0/3685502770 >> 172.21.15.61:0/3685502770 conn(0x7f8ca00b6210 msgr2=0x7f8ca00b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:21.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- 172.21.15.61:0/3685502770 shutdown_connections 2022-07-20T03:07:21.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- 172.21.15.61:0/3685502770 wait complete. 2022-07-20T03:07:21.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 Processor -- start 2022-07-20T03:07:21.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- start start 2022-07-20T03:07:21.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 0x7f8ca0156cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ca00ab250 0x7f8ca009bb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca00b23c0 0x7f8ca009c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca00650e0 con 0x7f8ca005c7c0 2022-07-20T03:07:21.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca01571a0 con 0x7f8ca00ab250 2022-07-20T03:07:21.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ca0064ba0 con 0x7f8ca00b23c0 2022-07-20T03:07:21.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 0x7f8ca0156cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 0x7f8ca0156cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42680/0 (socket says 172.21.15.61:42680) 2022-07-20T03:07:21.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9e59c700 1 -- 172.21.15.61:0/1427816569 learned_addr learned my addr 172.21.15.61:0/1427816569 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:21.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9dd9b700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ca00ab250 0x7f8ca009bb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.519+0000 7f8c9ed9d700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca00b23c0 0x7f8ca009c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9e59c700 1 -- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca00b23c0 msgr2=0x7f8ca009c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9e59c700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca00b23c0 0x7f8ca009c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9e59c700 1 -- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ca00ab250 msgr2=0x7f8ca009bb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9e59c700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ca00ab250 0x7f8ca009bb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9e59c700 1 -- 172.21.15.61:0/1427816569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8c9000b1e0 con 0x7f8ca005c7c0 2022-07-20T03:07:21.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9e59c700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 0x7f8ca0156cb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8c98006cc0 tx=0x7f8c98008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c98013620 con 0x7f8ca005c7c0 2022-07-20T03:07:21.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8c98013780 con 0x7f8ca005c7c0 2022-07-20T03:07:21.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ca0064e40 con 0x7f8ca005c7c0 2022-07-20T03:07:21.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8c98013ae0 con 0x7f8ca005c7c0 2022-07-20T03:07:21.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ca0099b90 con 0x7f8ca005c7c0 2022-07-20T03:07:21.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ca00abc50 con 0x7f8ca005c7c0 2022-07-20T03:07:21.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8c98013c40 con 0x7f8ca005c7c0 2022-07-20T03:07:21.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.523+0000 7f8c8f7fe700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8c7c053a40 0x7f8c7c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.527+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8c9806be40 con 0x7f8ca005c7c0 2022-07-20T03:07:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.527+0000 7f8c9dd9b700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8c7c053a40 0x7f8c7c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.527+0000 7f8c9dd9b700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8c7c053a40 0x7f8c7c055f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8c9400c000 tx=0x7f8c94008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.527+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8c98042c30 con 0x7f8ca005c7c0 2022-07-20T03:07:21.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 6} v 0) v1 -- 0x7f8ca00574e0 con 0x7f8ca005c7c0 2022-07-20T03:07:21.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c8f7fe700 1 -- 172.21.15.61:0/1427816569 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 6}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f8c98046250 con 0x7f8ca005c7c0 2022-07-20T03:07:21.724 INFO:teuthology.orchestra.run.smithi061.stdout:55834574855 2022-07-20T03:07:21.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8c7c053a40 msgr2=0x7f8c7c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8c7c053a40 0x7f8c7c055f00 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f8c9400c000 tx=0x7f8c94008040).stop 2022-07-20T03:07:21.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 msgr2=0x7f8ca0156cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 0x7f8ca0156cb0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f8c98006cc0 tx=0x7f8c98008040).stop 2022-07-20T03:07:21.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 shutdown_connections 2022-07-20T03:07:21.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8c7c053a40 0x7f8c7c055f00 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ca00ab250 0x7f8ca009bb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ca00b23c0 0x7f8ca009c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 --2- 172.21.15.61:0/1427816569 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ca005c7c0 0x7f8ca0156cb0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 >> 172.21.15.61:0/1427816569 conn(0x7f8ca00b6210 msgr2=0x7f8ca00b51a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:21.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 shutdown_connections 2022-07-20T03:07:21.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.659+0000 7f8c9f59e700 1 -- 172.21.15.61:0/1427816569 wait complete. 2022-07-20T03:07:21.814 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574854 got 55834574855 for osd.6 2022-07-20T03:07:21.815 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd last-stat-seq osd.7 2022-07-20T03:07:21.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.919+0000 7f2496628700 1 Processor -- start 2022-07-20T03:07:21.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.919+0000 7f2496628700 1 -- start start 2022-07-20T03:07:21.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2496628700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249005c7c0 0x7f24900b1390 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2496628700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f24900ab250 0x7f24901462c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2496628700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24900b0900 con 0x7f24901467b0 2022-07-20T03:07:21.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2496628700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24900b2a30 con 0x7f24900ab250 2022-07-20T03:07:21.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2496628700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2490094c50 con 0x7f249005c7c0 2022-07-20T03:07:21.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2495e27700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f24901467b0 0x7f24901420a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39290/0 (socket says 172.21.15.61:39290) 2022-07-20T03:07:21.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2495e27700 1 -- 172.21.15.61:0/3062372397 learned_addr learned my addr 172.21.15.61:0/3062372397 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:21.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2495626700 1 --2- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249005c7c0 0x7f24900b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2494e25700 1 --2- 172.21.15.61:0/3062372397 >> v2:172.21.15.117:3300/0 conn(0x7f24900ab250 0x7f24901462c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f247ffff700 1 -- 172.21.15.61:0/3062372397 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3319639175 0 0) 0x7f24900b0900 con 0x7f24901467b0 2022-07-20T03:07:21.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f247ffff700 1 -- 172.21.15.61:0/3062372397 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2478003260 con 0x7f24901467b0 2022-07-20T03:07:21.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2494e25700 1 -- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249005c7c0 msgr2=0x7f24900b1390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:21.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2494e25700 1 --2- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249005c7c0 0x7f24900b1390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2494e25700 1 -- 172.21.15.61:0/3062372397 >> v1:172.21.15.61:6789/0 conn(0x7f24901467b0 legacy=0x7f24901420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2494e25700 1 -- 172.21.15.61:0/3062372397 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24900b0c50 con 0x7f24900ab250 2022-07-20T03:07:21.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f2494e25700 1 --2- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24900ab250 0x7f24901462c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2480002f70 tx=0x7f2480006df0).ready entity=mon.1 client_cookie=4c8b861f05f7f713 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.923+0000 7f247ffff700 1 -- 172.21.15.61:0/3062372397 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f248000d4a0 con 0x7f24900ab250 2022-07-20T03:07:21.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f247ffff700 1 -- 172.21.15.61:0/3062372397 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2480000e90 con 0x7f24900ab250 2022-07-20T03:07:21.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f247ffff700 1 -- 172.21.15.61:0/3062372397 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f248000d7f0 con 0x7f24900ab250 2022-07-20T03:07:21.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 -- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24900ab250 msgr2=0x7f24901462c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 --2- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24900ab250 0x7f24901462c0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f2480002f70 tx=0x7f2480006df0).stop 2022-07-20T03:07:21.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 -- 172.21.15.61:0/3062372397 shutdown_connections 2022-07-20T03:07:21.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 --2- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24900ab250 0x7f24901462c0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 --2- 172.21.15.61:0/3062372397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249005c7c0 0x7f24900b1390 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 -- 172.21.15.61:0/3062372397 >> 172.21.15.61:0/3062372397 conn(0x7f24900b6210 msgr2=0x7f24900b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:21.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 -- 172.21.15.61:0/3062372397 shutdown_connections 2022-07-20T03:07:21.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 -- 172.21.15.61:0/3062372397 wait complete. 2022-07-20T03:07:21.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.927+0000 7f2496628700 1 Processor -- start 2022-07-20T03:07:21.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 -- start start 2022-07-20T03:07:21.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249005c7c0 0x7f24900acaf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24900ab250 0x7f2490155ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 0x7f2490156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2490141130 con 0x7f24901467b0 2022-07-20T03:07:21.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2490067790 con 0x7f249005c7c0 2022-07-20T03:07:21.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2490066930 con 0x7f24900ab250 2022-07-20T03:07:21.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 0x7f2490156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 0x7f2490156190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42698/0 (socket says 172.21.15.61:42698) 2022-07-20T03:07:21.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 -- 172.21.15.61:0/344258780 learned_addr learned my addr 172.21.15.61:0/344258780 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:21.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2494e25700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24900ab250 0x7f2490155ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495626700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249005c7c0 0x7f24900acaf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 -- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24900ab250 msgr2=0x7f2490155ca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24900ab250 0x7f2490155ca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 -- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249005c7c0 msgr2=0x7f24900acaf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:21.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249005c7c0 0x7f24900acaf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 -- 172.21.15.61:0/344258780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248000b180 con 0x7f24901467b0 2022-07-20T03:07:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2495e27700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 0x7f2490156190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f2490065cc0 tx=0x7f2488007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2488005ba0 con 0x7f24901467b0 2022-07-20T03:07:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2488005d00 con 0x7f24901467b0 2022-07-20T03:07:21.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24880072c0 con 0x7f24901467b0 2022-07-20T03:07:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24900679c0 con 0x7f24901467b0 2022-07-20T03:07:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.931+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24901557d0 con 0x7f24901467b0 2022-07-20T03:07:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.935+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24900b0ef0 con 0x7f24901467b0 2022-07-20T03:07:21.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.935+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2488010070 con 0x7f24901467b0 2022-07-20T03:07:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.935+0000 7f247e7fc700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2470053a40 0x7f2470055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.935+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f248806b8d0 con 0x7f24901467b0 2022-07-20T03:07:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.935+0000 7f2495626700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2470053a40 0x7f2470055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:21.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.935+0000 7f2495626700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2470053a40 0x7f2470055f00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f248c00b000 tx=0x7f248c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:21.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:21.939+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2488042720 con 0x7f24901467b0 2022-07-20T03:07:22.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.067+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd last-stat-seq", "id": 7} v 0) v1 -- 0x7f24900574e0 con 0x7f24901467b0 2022-07-20T03:07:22.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.067+0000 7f247e7fc700 1 -- 172.21.15.61:0/344258780 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd last-stat-seq", "id": 7}]=0 v0) v1 ==== 74+0+12 (secure 0 0 0) 0x7f2488045d40 con 0x7f24901467b0 2022-07-20T03:07:22.075 INFO:teuthology.orchestra.run.smithi061.stdout:55834574854 2022-07-20T03:07:22.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2470053a40 msgr2=0x7f2470055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2470053a40 0x7f2470055f00 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f248c00b000 tx=0x7f248c006df0).stop 2022-07-20T03:07:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 msgr2=0x7f2490156190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 0x7f2490156190 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f2490065cc0 tx=0x7f2488007dc0).stop 2022-07-20T03:07:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 shutdown_connections 2022-07-20T03:07:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2470053a40 0x7f2470055f00 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24901467b0 0x7f2490156190 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24900ab250 0x7f2490155ca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 --2- 172.21.15.61:0/344258780 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249005c7c0 0x7f24900acaf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 >> 172.21.15.61:0/344258780 conn(0x7f24900b6210 msgr2=0x7f24901417d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:22.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 shutdown_connections 2022-07-20T03:07:22.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.071+0000 7f2496628700 1 -- 172.21.15.61:0/344258780 wait complete. 2022-07-20T03:07:22.107 INFO:tasks.ceph.ceph_manager.ceph:need seq 55834574853 got 55834574854 for osd.7 2022-07-20T03:07:22.107 INFO:tasks.ceph.ceph_manager.ceph:waiting for clean 2022-07-20T03:07:22.108 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T03:07:22.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 Processor -- start 2022-07-20T03:07:22.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 -- start start 2022-07-20T03:07:22.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbdd005c3f0 0x7fbdd014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbdd00acef0 0x7fbdd0146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd005d480 con 0x7fbdd00b1e00 2022-07-20T03:07:22.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd00b21e0 con 0x7fbdd005c3f0 2022-07-20T03:07:22.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdd4bd1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbdd005cf40 con 0x7fbdd00acef0 2022-07-20T03:07:22.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdcdd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbdd005c3f0 0x7fbdd014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdcdd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbdd005c3f0 0x7fbdd014a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42702/0 (socket says 172.21.15.61:42702) 2022-07-20T03:07:22.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdce59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbdd00b1e00 0x7fbdd00ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39302/0 (socket says 172.21.15.61:39302) 2022-07-20T03:07:22.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.239+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/1416274680 learned_addr learned my addr 172.21.15.61:0/1416274680 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:22.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdced9d700 1 --2- 172.21.15.61:0/1416274680 >> v2:172.21.15.117:3300/0 conn(0x7fbdd00acef0 0x7fbdd0146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/1416274680 >> v2:172.21.15.117:3300/0 conn(0x7fbdd00acef0 msgr2=0x7fbdd0146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcdd9b700 1 --2- 172.21.15.61:0/1416274680 >> v2:172.21.15.117:3300/0 conn(0x7fbdd00acef0 0x7fbdd0146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/1416274680 >> v1:172.21.15.61:6789/0 conn(0x7fbdd00b1e00 legacy=0x7fbdd00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcd59a700 1 -- 172.21.15.61:0/1416274680 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2304949082 0 0) 0x7fbdd005d480 con 0x7fbdd00b1e00 2022-07-20T03:07:22.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/1416274680 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd005d1e0 con 0x7fbdd005c3f0 2022-07-20T03:07:22.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcdd9b700 1 --2- 172.21.15.61:0/1416274680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd005c3f0 0x7fbdd014a6d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbdc0002f70 tx=0x7fbdc0006eb0).ready entity=mon.0 client_cookie=5a7f2c6d3c804394 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:22.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcd59a700 1 -- 172.21.15.61:0/1416274680 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc000d5c0 con 0x7fbdd005c3f0 2022-07-20T03:07:22.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcd59a700 1 -- 172.21.15.61:0/1416274680 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc0000e90 con 0x7fbdd005c3f0 2022-07-20T03:07:22.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdcd59a700 1 -- 172.21.15.61:0/1416274680 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc000d910 con 0x7fbdd005c3f0 2022-07-20T03:07:22.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/1416274680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd005c3f0 msgr2=0x7fbdd014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/1416274680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd005c3f0 0x7fbdd014a6d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fbdc0002f70 tx=0x7fbdc0006eb0).stop 2022-07-20T03:07:22.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/1416274680 shutdown_connections 2022-07-20T03:07:22.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/1416274680 >> v2:172.21.15.117:3300/0 conn(0x7fbdd00acef0 0x7fbdd0146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/1416274680 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd005c3f0 0x7fbdd014a6d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/1416274680 >> 172.21.15.61:0/1416274680 conn(0x7fbdd00611c0 msgr2=0x7fbdd0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:22.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/1416274680 shutdown_connections 2022-07-20T03:07:22.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.243+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/1416274680 wait complete. 2022-07-20T03:07:22.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 Processor -- start 2022-07-20T03:07:22.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 -- start start 2022-07-20T03:07:22.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00acef0 0x7fbdd014ce60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 0x7fbdd009e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00639e0 0x7fbdd01564b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00b5ee0 con 0x7fbdd00b1e00 2022-07-20T03:07:22.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd01569a0 con 0x7fbdd00acef0 2022-07-20T03:07:22.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdd4bd1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd009e720 con 0x7fbdd00639e0 2022-07-20T03:07:22.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 0x7fbdd009e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 0x7fbdd009e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42710/0 (socket says 172.21.15.61:42710) 2022-07-20T03:07:22.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/2930103257 learned_addr learned my addr 172.21.15.61:0/2930103257 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:22.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdce59c700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00acef0 0x7fbdd014ce60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdced9d700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00639e0 0x7fbdd01564b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00639e0 msgr2=0x7fbdd01564b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00639e0 0x7fbdd01564b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00acef0 msgr2=0x7fbdd014ce60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00acef0 0x7fbdd014ce60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/2930103257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc000b1e0 con 0x7fbdd00b1e00 2022-07-20T03:07:22.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.247+0000 7fbdced9d700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00639e0 0x7fbdd01564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:22.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdcdd9b700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 0x7fbdd009e080 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fbdc0002f70 tx=0x7fbdc0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:22.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc0011690 con 0x7fbdd00b1e00 2022-07-20T03:07:22.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc00117f0 con 0x7fbdd00b1e00 2022-07-20T03:07:22.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc0011b50 con 0x7fbdd00b1e00 2022-07-20T03:07:22.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd0159c50 con 0x7fbdd00b1e00 2022-07-20T03:07:22.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd015a310 con 0x7fbdd00b1e00 2022-07-20T03:07:22.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd009e570 con 0x7fbdd00b1e00 2022-07-20T03:07:22.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbdc001a040 con 0x7fbdd00b1e00 2022-07-20T03:07:22.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdbf7fe700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdac053a40 0x7fbdac055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.251+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbdc006cc20 con 0x7fbdd00b1e00 2022-07-20T03:07:22.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.255+0000 7fbdce59c700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdac053a40 0x7fbdac055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.255+0000 7fbdce59c700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdac053a40 0x7fbdac055f00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbdc400d000 tx=0x7fbdc4007bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:22.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.259+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbdc0043ba0 con 0x7fbdd00b1e00 2022-07-20T03:07:22.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.379+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fbdd0159ad0 con 0x7fbdac053a40 2022-07-20T03:07:22.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdbf7fe700 1 -- 172.21.15.61:0/2930103257 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82204 (secure 0 0 0) 0x7fbdc4015410 con 0x7fbdac053a40 2022-07-20T03:07:22.471 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:22.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdac053a40 msgr2=0x7fbdac055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdac053a40 0x7fbdac055f00 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fbdc400d000 tx=0x7fbdc4007bc0).stop 2022-07-20T03:07:22.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 msgr2=0x7fbdd009e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 0x7fbdd009e080 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7fbdc0002f70 tx=0x7fbdc0015000).stop 2022-07-20T03:07:22.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 shutdown_connections 2022-07-20T03:07:22.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdac053a40 0x7fbdac055f00 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00639e0 0x7fbdd01564b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b1e00 0x7fbdd009e080 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 --2- 172.21.15.61:0/2930103257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00acef0 0x7fbdd014ce60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 >> 172.21.15.61:0/2930103257 conn(0x7fbdd005c3f0 msgr2=0x7fbdd00ae4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:22.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 shutdown_connections 2022-07-20T03:07:22.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.391+0000 7fbdd4bd1700 1 -- 172.21.15.61:0/2930103257 wait complete. 2022-07-20T03:07:22.502 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-20T03:07:22.502 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2022-07-20T03:07:21.023366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6366880,"kb_used_data":743228160,"kb_used_omap":566,"kb_used_meta":0,"kb_avail":743228160,"statfs":{"total":767585320960,"available":761065635840,"internally_reserved":0,"allocated":761065635840,"data_stored":761065635840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":579816,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001386"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.737179+0000","last_change":"2022-07-20T03:07:11.736955+0000","last_active":"2022-07-20T03:07:11.737179+0000","last_peered":"2022-07-20T03:07:11.737179+0000","last_clean":"2022-07-20T03:07:11.737179+0000","last_became_active":"2022-07-20T03:07:11.736673+0000","last_became_peered":"2022-07-20T03:07:11.736673+0000","last_unstale":"2022-07-20T03:07:11.737179+0000","last_undegraded":"2022-07-20T03:07:11.737179+0000","last_fullsized":"2022-07-20T03:07:11.737179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.732886+0000","last_change":"2022-07-20T03:07:00.764096+0000","last_active":"2022-07-20T03:07:10.732886+0000","last_peered":"2022-07-20T03:07:10.732886+0000","last_clean":"2022-07-20T03:07:10.732886+0000","last_became_active":"2022-07-20T03:07:00.763305+0000","last_became_peered":"2022-07-20T03:07:00.763305+0000","last_unstale":"2022-07-20T03:07:10.732886+0000","last_undegraded":"2022-07-20T03:07:10.732886+0000","last_fullsized":"2022-07-20T03:07:10.732886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_clean_scrub_stamp":"2022-07-20T03:06:54.339924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.731741+0000","last_change":"2022-07-20T03:07:06.947888+0000","last_active":"2022-07-20T03:07:11.731741+0000","last_peered":"2022-07-20T03:07:11.731741+0000","last_clean":"2022-07-20T03:07:11.731741+0000","last_became_active":"2022-07-20T03:07:06.947545+0000","last_became_peered":"2022-07-20T03:07:06.947545+0000","last_unstale":"2022-07-20T03:07:11.731741+0000","last_undegraded":"2022-07-20T03:07:11.731741+0000","last_fullsized":"2022-07-20T03:07:11.731741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.735163+0000","last_change":"2022-07-20T03:07:10.209192+0000","last_active":"2022-07-20T03:07:10.735163+0000","last_peered":"2022-07-20T03:07:10.735163+0000","last_clean":"2022-07-20T03:07:10.735163+0000","last_became_active":"2022-07-20T03:07:10.208021+0000","last_became_peered":"2022-07-20T03:07:10.208021+0000","last_unstale":"2022-07-20T03:07:10.735163+0000","last_undegraded":"2022-07-20T03:07:10.735163+0000","last_fullsized":"2022-07-20T03:07:10.735163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.194262+0000","last_change":"2022-07-20T03:07:10.732683+0000","last_active":"2022-07-20T03:07:12.194262+0000","last_peered":"2022-07-20T03:07:12.194262+0000","last_clean":"2022-07-20T03:07:12.194262+0000","last_became_active":"2022-07-20T03:07:10.732251+0000","last_became_peered":"2022-07-20T03:07:10.732251+0000","last_unstale":"2022-07-20T03:07:12.194262+0000","last_undegraded":"2022-07-20T03:07:12.194262+0000","last_fullsized":"2022-07-20T03:07:12.194262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.732768+0000","last_change":"2022-07-20T03:07:06.947578+0000","last_active":"2022-07-20T03:07:10.732768+0000","last_peered":"2022-07-20T03:07:10.732768+0000","last_clean":"2022-07-20T03:07:10.732768+0000","last_became_active":"2022-07-20T03:07:06.947295+0000","last_became_peered":"2022-07-20T03:07:06.947295+0000","last_unstale":"2022-07-20T03:07:10.732768+0000","last_undegraded":"2022-07-20T03:07:10.732768+0000","last_fullsized":"2022-07-20T03:07:10.732768+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.194129+0000","last_change":"2022-07-20T03:07:10.211286+0000","last_active":"2022-07-20T03:07:12.194129+0000","last_peered":"2022-07-20T03:07:12.194129+0000","last_clean":"2022-07-20T03:07:12.194129+0000","last_became_active":"2022-07-20T03:07:10.209725+0000","last_became_peered":"2022-07-20T03:07:10.209725+0000","last_unstale":"2022-07-20T03:07:12.194129+0000","last_undegraded":"2022-07-20T03:07:12.194129+0000","last_fullsized":"2022-07-20T03:07:12.194129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.197060+0000","last_change":"2022-07-20T03:07:10.729635+0000","last_active":"2022-07-20T03:07:12.197060+0000","last_peered":"2022-07-20T03:07:12.197060+0000","last_clean":"2022-07-20T03:07:12.197060+0000","last_became_active":"2022-07-20T03:07:10.729233+0000","last_became_peered":"2022-07-20T03:07:10.729233+0000","last_unstale":"2022-07-20T03:07:12.197060+0000","last_undegraded":"2022-07-20T03:07:12.197060+0000","last_fullsized":"2022-07-20T03:07:12.197060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.734717+0000","last_change":"2022-07-20T03:07:06.951565+0000","last_active":"2022-07-20T03:07:11.734717+0000","last_peered":"2022-07-20T03:07:11.734717+0000","last_clean":"2022-07-20T03:07:11.734717+0000","last_became_active":"2022-07-20T03:07:06.951117+0000","last_became_peered":"2022-07-20T03:07:06.951117+0000","last_unstale":"2022-07-20T03:07:11.734717+0000","last_undegraded":"2022-07-20T03:07:11.734717+0000","last_fullsized":"2022-07-20T03:07:11.734717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196138+0000","last_change":"2022-07-20T03:07:10.209074+0000","last_active":"2022-07-20T03:07:12.196138+0000","last_peered":"2022-07-20T03:07:12.196138+0000","last_clean":"2022-07-20T03:07:12.196138+0000","last_became_active":"2022-07-20T03:07:10.208047+0000","last_became_peered":"2022-07-20T03:07:10.208047+0000","last_unstale":"2022-07-20T03:07:12.196138+0000","last_undegraded":"2022-07-20T03:07:12.196138+0000","last_fullsized":"2022-07-20T03:07:12.196138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196798+0000","last_change":"2022-07-20T03:07:10.740104+0000","last_active":"2022-07-20T03:07:12.196798+0000","last_peered":"2022-07-20T03:07:12.196798+0000","last_clean":"2022-07-20T03:07:12.196798+0000","last_became_active":"2022-07-20T03:07:10.739711+0000","last_became_peered":"2022-07-20T03:07:10.739711+0000","last_unstale":"2022-07-20T03:07:12.196798+0000","last_undegraded":"2022-07-20T03:07:12.196798+0000","last_fullsized":"2022-07-20T03:07:12.196798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.731988+0000","last_change":"2022-07-20T03:07:06.946761+0000","last_active":"2022-07-20T03:07:11.731988+0000","last_peered":"2022-07-20T03:07:11.731988+0000","last_clean":"2022-07-20T03:07:11.731988+0000","last_became_active":"2022-07-20T03:07:06.946278+0000","last_became_peered":"2022-07-20T03:07:06.946278+0000","last_unstale":"2022-07-20T03:07:11.731988+0000","last_undegraded":"2022-07-20T03:07:11.731988+0000","last_fullsized":"2022-07-20T03:07:11.731988+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.732562+0000","last_change":"2022-07-20T03:07:10.207459+0000","last_active":"2022-07-20T03:07:11.732562+0000","last_peered":"2022-07-20T03:07:11.732562+0000","last_clean":"2022-07-20T03:07:11.732562+0000","last_became_active":"2022-07-20T03:07:10.207042+0000","last_became_peered":"2022-07-20T03:07:10.207042+0000","last_unstale":"2022-07-20T03:07:11.732562+0000","last_undegraded":"2022-07-20T03:07:11.732562+0000","last_fullsized":"2022-07-20T03:07:11.732562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196571+0000","last_change":"2022-07-20T03:07:10.730218+0000","last_active":"2022-07-20T03:07:12.196571+0000","last_peered":"2022-07-20T03:07:12.196571+0000","last_clean":"2022-07-20T03:07:12.196571+0000","last_became_active":"2022-07-20T03:07:10.729821+0000","last_became_peered":"2022-07-20T03:07:10.729821+0000","last_unstale":"2022-07-20T03:07:12.196571+0000","last_undegraded":"2022-07-20T03:07:12.196571+0000","last_fullsized":"2022-07-20T03:07:12.196571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.731990+0000","last_change":"2022-07-20T03:07:10.204487+0000","last_active":"2022-07-20T03:07:11.731990+0000","last_peered":"2022-07-20T03:07:11.731990+0000","last_clean":"2022-07-20T03:07:11.731990+0000","last_became_active":"2022-07-20T03:07:10.203986+0000","last_became_peered":"2022-07-20T03:07:10.203986+0000","last_unstale":"2022-07-20T03:07:11.731990+0000","last_undegraded":"2022-07-20T03:07:11.731990+0000","last_fullsized":"2022-07-20T03:07:11.731990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196944+0000","last_change":"2022-07-20T03:07:06.951643+0000","last_active":"2022-07-20T03:07:12.196944+0000","last_peered":"2022-07-20T03:07:12.196944+0000","last_clean":"2022-07-20T03:07:12.196944+0000","last_became_active":"2022-07-20T03:07:06.951291+0000","last_became_peered":"2022-07-20T03:07:06.951291+0000","last_unstale":"2022-07-20T03:07:12.196944+0000","last_undegraded":"2022-07-20T03:07:12.196944+0000","last_fullsized":"2022-07-20T03:07:12.196944+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.194035+0000","last_change":"2022-07-20T03:07:10.732961+0000","last_active":"2022-07-20T03:07:12.194035+0000","last_peered":"2022-07-20T03:07:12.194035+0000","last_clean":"2022-07-20T03:07:12.194035+0000","last_became_active":"2022-07-20T03:07:10.732470+0000","last_became_peered":"2022-07-20T03:07:10.732470+0000","last_unstale":"2022-07-20T03:07:12.194035+0000","last_undegraded":"2022-07-20T03:07:12.194035+0000","last_fullsized":"2022-07-20T03:07:12.194035+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196820+0000","last_change":"2022-07-20T03:07:10.211560+0000","last_active":"2022-07-20T03:07:12.196820+0000","last_peered":"2022-07-20T03:07:12.196820+0000","last_clean":"2022-07-20T03:07:12.196820+0000","last_became_active":"2022-07-20T03:07:10.211088+0000","last_became_peered":"2022-07-20T03:07:10.211088+0000","last_unstale":"2022-07-20T03:07:12.196820+0000","last_undegraded":"2022-07-20T03:07:12.196820+0000","last_fullsized":"2022-07-20T03:07:12.196820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.731982+0000","last_change":"2022-07-20T03:07:06.943629+0000","last_active":"2022-07-20T03:07:10.731982+0000","last_peered":"2022-07-20T03:07:10.731982+0000","last_clean":"2022-07-20T03:07:10.731982+0000","last_became_active":"2022-07-20T03:07:06.942970+0000","last_became_peered":"2022-07-20T03:07:06.942970+0000","last_unstale":"2022-07-20T03:07:10.731982+0000","last_undegraded":"2022-07-20T03:07:10.731982+0000","last_fullsized":"2022-07-20T03:07:10.731982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.740129+0000","last_change":"2022-07-20T03:07:10.737196+0000","last_active":"2022-07-20T03:07:10.740129+0000","last_peered":"2022-07-20T03:07:10.740129+0000","last_clean":"2022-07-20T03:07:10.740129+0000","last_became_active":"2022-07-20T03:07:10.736744+0000","last_became_peered":"2022-07-20T03:07:10.736744+0000","last_unstale":"2022-07-20T03:07:10.740129+0000","last_undegraded":"2022-07-20T03:07:10.740129+0000","last_fullsized":"2022-07-20T03:07:10.740129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.741321+0000","last_change":"2022-07-20T03:07:10.208376+0000","last_active":"2022-07-20T03:07:10.741321+0000","last_peered":"2022-07-20T03:07:10.741321+0000","last_clean":"2022-07-20T03:07:10.741321+0000","last_became_active":"2022-07-20T03:07:10.206566+0000","last_became_peered":"2022-07-20T03:07:10.206566+0000","last_unstale":"2022-07-20T03:07:10.741321+0000","last_undegraded":"2022-07-20T03:07:10.741321+0000","last_fullsized":"2022-07-20T03:07:10.741321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196983+0000","last_change":"2022-07-20T03:07:06.950534+0000","last_active":"2022-07-20T03:07:12.196983+0000","last_peered":"2022-07-20T03:07:12.196983+0000","last_clean":"2022-07-20T03:07:12.196983+0000","last_became_active":"2022-07-20T03:07:06.949930+0000","last_became_peered":"2022-07-20T03:07:06.949930+0000","last_unstale":"2022-07-20T03:07:12.196983+0000","last_undegraded":"2022-07-20T03:07:12.196983+0000","last_fullsized":"2022-07-20T03:07:12.196983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.732445+0000","last_change":"2022-07-20T03:07:10.726332+0000","last_active":"2022-07-20T03:07:11.732445+0000","last_peered":"2022-07-20T03:07:11.732445+0000","last_clean":"2022-07-20T03:07:11.732445+0000","last_became_active":"2022-07-20T03:07:10.725868+0000","last_became_peered":"2022-07-20T03:07:10.725868+0000","last_unstale":"2022-07-20T03:07:11.732445+0000","last_undegraded":"2022-07-20T03:07:11.732445+0000","last_fullsized":"2022-07-20T03:07:11.732445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196362+0000","last_change":"2022-07-20T03:07:10.211194+0000","last_active":"2022-07-20T03:07:12.196362+0000","last_peered":"2022-07-20T03:07:12.196362+0000","last_clean":"2022-07-20T03:07:12.196362+0000","last_became_active":"2022-07-20T03:07:10.210872+0000","last_became_peered":"2022-07-20T03:07:10.210872+0000","last_unstale":"2022-07-20T03:07:12.196362+0000","last_undegraded":"2022-07-20T03:07:12.196362+0000","last_fullsized":"2022-07-20T03:07:12.196362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.734902+0000","last_change":"2022-07-20T03:07:06.946858+0000","last_active":"2022-07-20T03:07:11.734902+0000","last_peered":"2022-07-20T03:07:11.734902+0000","last_clean":"2022-07-20T03:07:11.734902+0000","last_became_active":"2022-07-20T03:07:06.946555+0000","last_became_peered":"2022-07-20T03:07:06.946555+0000","last_unstale":"2022-07-20T03:07:11.734902+0000","last_undegraded":"2022-07-20T03:07:11.734902+0000","last_fullsized":"2022-07-20T03:07:11.734902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795860,"kb_used_data":92903520,"kb_used_omap":63,"kb_used_meta":0,"kb_avail":92903520,"statfs":{"total":95948165120,"available":95133204480,"internally_reserved":0,"allocated":95133204480,"data_stored":95133204480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64831,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795852,"kb_used_data":92903528,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903528,"statfs":{"total":95948165120,"available":95133212672,"internally_reserved":0,"allocated":95133212672,"data_stored":95133212672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43355,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795860,"kb_used_data":92903520,"kb_used_omap":77,"kb_used_meta":0,"kb_avail":92903520,"statfs":{"total":95948165120,"available":95133204480,"internally_reserved":0,"allocated":95133204480,"data_stored":95133204480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79443,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903524,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":0,"allocated":95133208576,"data_stored":95133208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58670,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795868,"kb_used_data":92903512,"kb_used_omap":65,"kb_used_meta":0,"kb_avail":92903512,"statfs":{"total":95948165120,"available":95133196288,"internally_reserved":0,"allocated":95133196288,"data_stored":95133196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66802,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795864,"kb_used_data":92903516,"kb_used_omap":72,"kb_used_meta":0,"kb_avail":92903516,"statfs":{"total":95948165120,"available":95133200384,"internally_reserved":0,"allocated":95133200384,"data_stored":95133200384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74125,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795864,"kb_used_data":92903516,"kb_used_omap":102,"kb_used_meta":0,"kb_avail":92903516,"statfs":{"total":95948165120,"available":95133200384,"internally_reserved":0,"allocated":95133200384,"data_stored":95133200384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104682,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903524,"kb_used_omap":85,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":0,"allocated":95133208576,"data_stored":95133208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87908,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]}]}]}} 2022-07-20T03:07:22.522 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T03:07:22.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.627+0000 7f0aabf5f700 1 Processor -- start 2022-07-20T03:07:22.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.627+0000 7f0aabf5f700 1 -- start start 2022-07-20T03:07:22.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0aa4142e30 0x7f0aa40b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40acff0 con 0x7f0aa40b2540 2022-07-20T03:07:22.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0aa40ad4c0 con 0x7f0aa40b18a0 2022-07-20T03:07:22.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0aa40ad220 con 0x7f0aa4142e30 2022-07-20T03:07:22.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa94fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0aa4142e30 0x7f0aa40b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa9cfb700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0aa40b2540 0x7f0aa4142720 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39316/0 (socket says 172.21.15.61:39316) 2022-07-20T03:07:22.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa94fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0aa4142e30 0x7f0aa40b0380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42716/0 (socket says 172.21.15.61:42716) 2022-07-20T03:07:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa9cfb700 1 -- 172.21.15.61:0/1593963773 learned_addr learned my addr 172.21.15.61:0/1593963773 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa94fa700 1 -- 172.21.15.61:0/1593963773 >> v1:172.21.15.117:6789/0 conn(0x7f0aa40b18a0 legacy=0x7f0aa413bc40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa94fa700 1 -- 172.21.15.61:0/1593963773 >> v1:172.21.15.61:6789/0 conn(0x7f0aa40b2540 legacy=0x7f0aa4142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa94fa700 1 -- 172.21.15.61:0/1593963773 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aa4142280 con 0x7f0aa4142e30 2022-07-20T03:07:22.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa94fa700 1 --2- 172.21.15.61:0/1593963773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa40b0380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0a94002f70 tx=0x7f0a94006df0).ready entity=mon.0 client_cookie=b3a2a3d226da9627 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:22.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa8cf9700 1 -- 172.21.15.61:0/1593963773 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9400d610 con 0x7f0aa4142e30 2022-07-20T03:07:22.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa8cf9700 1 -- 172.21.15.61:0/1593963773 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a94000e90 con 0x7f0aa4142e30 2022-07-20T03:07:22.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aa8cf9700 1 -- 172.21.15.61:0/1593963773 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9400d960 con 0x7f0aa4142e30 2022-07-20T03:07:22.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- 172.21.15.61:0/1593963773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 msgr2=0x7f0aa40b0380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/1593963773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa40b0380 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f0a94002f70 tx=0x7f0a94006df0).stop 2022-07-20T03:07:22.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- 172.21.15.61:0/1593963773 shutdown_connections 2022-07-20T03:07:22.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/1593963773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa40b0380 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- 172.21.15.61:0/1593963773 >> 172.21.15.61:0/1593963773 conn(0x7f0aa4062f30 msgr2=0x7f0aa4063340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:22.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- 172.21.15.61:0/1593963773 shutdown_connections 2022-07-20T03:07:22.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.631+0000 7f0aabf5f700 1 -- 172.21.15.61:0/1593963773 wait complete. 2022-07-20T03:07:22.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 Processor -- start 2022-07-20T03:07:22.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 -- start start 2022-07-20T03:07:22.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0aa40b18a0 0x7f0aa4064490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0aa40b2540 0x7f0aa41534e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa4139e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41539d0 con 0x7f0aa4142e30 2022-07-20T03:07:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa41399f0 con 0x7f0aa40b18a0 2022-07-20T03:07:22.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0aa4138210 con 0x7f0aa40b2540 2022-07-20T03:07:22.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa4139e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa4139e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42726/0 (socket says 172.21.15.61:42726) 2022-07-20T03:07:22.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aa9cfb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0aa40b18a0 0x7f0aa4064490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 -- 172.21.15.61:0/3621519209 learned_addr learned my addr 172.21.15.61:0/3621519209 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:22.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aa94fa700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0aa40b2540 0x7f0aa41534e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 -- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0aa40b2540 msgr2=0x7f0aa41534e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0aa40b2540 0x7f0aa41534e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 -- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0aa40b18a0 msgr2=0x7f0aa4064490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:22.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0aa40b18a0 0x7f0aa4064490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:22.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 -- 172.21.15.61:0/3621519209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a9400b180 con 0x7f0aa4142e30 2022-07-20T03:07:22.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aa9cfb700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0aa40b18a0 0x7f0aa4064490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:22.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aaa4fc700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa4139e50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0aa40b2f10 tx=0x7f0a9c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:22.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9c00f470 con 0x7f0aa4142e30 2022-07-20T03:07:22.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0aa40672f0 con 0x7f0aa4142e30 2022-07-20T03:07:22.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a9c00f5d0 con 0x7f0aa4142e30 2022-07-20T03:07:22.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a9c00f930 con 0x7f0aa4142e30 2022-07-20T03:07:22.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.635+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0aa4137230 con 0x7f0aa4142e30 2022-07-20T03:07:22.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.639+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0aa40670b0 con 0x7f0aa4142e30 2022-07-20T03:07:22.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.639+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0a9c019030 con 0x7f0aa4142e30 2022-07-20T03:07:22.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.639+0000 7f0a92ffd700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a84053920 0x7f0a84055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:22.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.639+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0a9c06b580 con 0x7f0aa4142e30 2022-07-20T03:07:22.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.639+0000 7f0aa9cfb700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a84053920 0x7f0a84055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:22.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.639+0000 7f0aa9cfb700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a84053920 0x7f0a84055de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0aa000d000 tx=0x7f0aa0007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:22.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.647+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0aa40670b0 con 0x7f0aa4142e30 2022-07-20T03:07:22.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.771+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f0aa4139bb0 con 0x7f0a84053920 2022-07-20T03:07:22.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.779+0000 7f0a92ffd700 1 -- 172.21.15.61:0/3621519209 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+82204 (secure 0 0 0) 0x7f0aa0004520 con 0x7f0a84053920 2022-07-20T03:07:22.990 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:23.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.779+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a84053920 msgr2=0x7f0a84055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.779+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a84053920 0x7f0a84055de0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f0aa000d000 tx=0x7f0aa0007b80).stop 2022-07-20T03:07:23.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.779+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 msgr2=0x7f0aa4139e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.779+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa4139e50 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f0aa40b2f10 tx=0x7f0a9c007f40).stop 2022-07-20T03:07:23.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 shutdown_connections 2022-07-20T03:07:23.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a84053920 0x7f0a84055de0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0aa4142e30 0x7f0aa4139e50 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0aa40b2540 0x7f0aa41534e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 --2- 172.21.15.61:0/3621519209 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0aa40b18a0 0x7f0aa4064490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 >> 172.21.15.61:0/3621519209 conn(0x7f0aa4062f30 msgr2=0x7f0aa4135ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:23.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 shutdown_connections 2022-07-20T03:07:23.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:22.783+0000 7f0aabf5f700 1 -- 172.21.15.61:0/3621519209 wait complete. 2022-07-20T03:07:23.049 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-20T03:07:23.049 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":35,"stamp":"2022-07-20T03:07:21.023366+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":50,"acting":50,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":50,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6366880,"kb_used_data":743228160,"kb_used_omap":566,"kb_used_meta":0,"kb_avail":743228160,"statfs":{"total":767585320960,"available":761065635840,"internally_reserved":0,"allocated":761065635840,"data_stored":761065635840,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":579816,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"4.001386"},"pg_stats":[{"pgid":"4.5","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.737179+0000","last_change":"2022-07-20T03:07:11.736955+0000","last_active":"2022-07-20T03:07:11.737179+0000","last_peered":"2022-07-20T03:07:11.737179+0000","last_clean":"2022-07-20T03:07:11.737179+0000","last_became_active":"2022-07-20T03:07:11.736673+0000","last_became_peered":"2022-07-20T03:07:11.736673+0000","last_unstale":"2022-07-20T03:07:11.737179+0000","last_undegraded":"2022-07-20T03:07:11.737179+0000","last_fullsized":"2022-07-20T03:07:11.737179+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":22,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,4],"acting":[6,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]},{"pgid":"1.0","version":"0'0","reported_seq":16,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.732886+0000","last_change":"2022-07-20T03:07:00.764096+0000","last_active":"2022-07-20T03:07:10.732886+0000","last_peered":"2022-07-20T03:07:10.732886+0000","last_clean":"2022-07-20T03:07:10.732886+0000","last_became_active":"2022-07-20T03:07:00.763305+0000","last_became_peered":"2022-07-20T03:07:00.763305+0000","last_unstale":"2022-07-20T03:07:10.732886+0000","last_undegraded":"2022-07-20T03:07:10.732886+0000","last_fullsized":"2022-07-20T03:07:10.732886+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_clean_scrub_stamp":"2022-07-20T03:06:54.339924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.731741+0000","last_change":"2022-07-20T03:07:06.947888+0000","last_active":"2022-07-20T03:07:11.731741+0000","last_peered":"2022-07-20T03:07:11.731741+0000","last_clean":"2022-07-20T03:07:11.731741+0000","last_became_active":"2022-07-20T03:07:06.947545+0000","last_became_peered":"2022-07-20T03:07:06.947545+0000","last_unstale":"2022-07-20T03:07:11.731741+0000","last_undegraded":"2022-07-20T03:07:11.731741+0000","last_fullsized":"2022-07-20T03:07:11.731741+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.2","version":"21'1","reported_seq":11,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.735163+0000","last_change":"2022-07-20T03:07:10.209192+0000","last_active":"2022-07-20T03:07:10.735163+0000","last_peered":"2022-07-20T03:07:10.735163+0000","last_clean":"2022-07-20T03:07:10.735163+0000","last_became_active":"2022-07-20T03:07:10.208021+0000","last_became_peered":"2022-07-20T03:07:10.208021+0000","last_unstale":"2022-07-20T03:07:10.735163+0000","last_undegraded":"2022-07-20T03:07:10.735163+0000","last_fullsized":"2022-07-20T03:07:10.735163+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":1,"ondisk_log_size":1,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,5],"acting":[3,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"4.6","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.194262+0000","last_change":"2022-07-20T03:07:10.732683+0000","last_active":"2022-07-20T03:07:12.194262+0000","last_peered":"2022-07-20T03:07:12.194262+0000","last_clean":"2022-07-20T03:07:12.194262+0000","last_became_active":"2022-07-20T03:07:10.732251+0000","last_became_peered":"2022-07-20T03:07:10.732251+0000","last_unstale":"2022-07-20T03:07:12.194262+0000","last_undegraded":"2022-07-20T03:07:12.194262+0000","last_fullsized":"2022-07-20T03:07:12.194262+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.732768+0000","last_change":"2022-07-20T03:07:06.947578+0000","last_active":"2022-07-20T03:07:10.732768+0000","last_peered":"2022-07-20T03:07:10.732768+0000","last_clean":"2022-07-20T03:07:10.732768+0000","last_became_active":"2022-07-20T03:07:06.947295+0000","last_became_peered":"2022-07-20T03:07:06.947295+0000","last_unstale":"2022-07-20T03:07:10.732768+0000","last_undegraded":"2022-07-20T03:07:10.732768+0000","last_fullsized":"2022-07-20T03:07:10.732768+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"3.1","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.194129+0000","last_change":"2022-07-20T03:07:10.211286+0000","last_active":"2022-07-20T03:07:12.194129+0000","last_peered":"2022-07-20T03:07:12.194129+0000","last_clean":"2022-07-20T03:07:12.194129+0000","last_became_active":"2022-07-20T03:07:10.209725+0000","last_became_peered":"2022-07-20T03:07:10.209725+0000","last_unstale":"2022-07-20T03:07:12.194129+0000","last_undegraded":"2022-07-20T03:07:12.194129+0000","last_fullsized":"2022-07-20T03:07:12.194129+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,4],"acting":[0,4],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"4.7","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.197060+0000","last_change":"2022-07-20T03:07:10.729635+0000","last_active":"2022-07-20T03:07:12.197060+0000","last_peered":"2022-07-20T03:07:12.197060+0000","last_clean":"2022-07-20T03:07:12.197060+0000","last_became_active":"2022-07-20T03:07:10.729233+0000","last_became_peered":"2022-07-20T03:07:10.729233+0000","last_unstale":"2022-07-20T03:07:12.197060+0000","last_undegraded":"2022-07-20T03:07:12.197060+0000","last_fullsized":"2022-07-20T03:07:12.197060+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.734717+0000","last_change":"2022-07-20T03:07:06.951565+0000","last_active":"2022-07-20T03:07:11.734717+0000","last_peered":"2022-07-20T03:07:11.734717+0000","last_clean":"2022-07-20T03:07:11.734717+0000","last_became_active":"2022-07-20T03:07:06.951117+0000","last_became_peered":"2022-07-20T03:07:06.951117+0000","last_unstale":"2022-07-20T03:07:11.734717+0000","last_undegraded":"2022-07-20T03:07:11.734717+0000","last_fullsized":"2022-07-20T03:07:11.734717+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"3.0","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196138+0000","last_change":"2022-07-20T03:07:10.209074+0000","last_active":"2022-07-20T03:07:12.196138+0000","last_peered":"2022-07-20T03:07:12.196138+0000","last_clean":"2022-07-20T03:07:12.196138+0000","last_became_active":"2022-07-20T03:07:10.208047+0000","last_became_peered":"2022-07-20T03:07:10.208047+0000","last_unstale":"2022-07-20T03:07:12.196138+0000","last_undegraded":"2022-07-20T03:07:12.196138+0000","last_fullsized":"2022-07-20T03:07:12.196138+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":12,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":4,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.4","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196798+0000","last_change":"2022-07-20T03:07:10.740104+0000","last_active":"2022-07-20T03:07:12.196798+0000","last_peered":"2022-07-20T03:07:12.196798+0000","last_clean":"2022-07-20T03:07:12.196798+0000","last_became_active":"2022-07-20T03:07:10.739711+0000","last_became_peered":"2022-07-20T03:07:10.739711+0000","last_unstale":"2022-07-20T03:07:12.196798+0000","last_undegraded":"2022-07-20T03:07:12.196798+0000","last_fullsized":"2022-07-20T03:07:12.196798+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":14,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.731988+0000","last_change":"2022-07-20T03:07:06.946761+0000","last_active":"2022-07-20T03:07:11.731988+0000","last_peered":"2022-07-20T03:07:11.731988+0000","last_clean":"2022-07-20T03:07:11.731988+0000","last_became_active":"2022-07-20T03:07:06.946278+0000","last_became_peered":"2022-07-20T03:07:06.946278+0000","last_unstale":"2022-07-20T03:07:11.731988+0000","last_undegraded":"2022-07-20T03:07:11.731988+0000","last_fullsized":"2022-07-20T03:07:11.731988+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"3.3","version":"21'5","reported_seq":16,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.732562+0000","last_change":"2022-07-20T03:07:10.207459+0000","last_active":"2022-07-20T03:07:11.732562+0000","last_peered":"2022-07-20T03:07:11.732562+0000","last_clean":"2022-07-20T03:07:11.732562+0000","last_became_active":"2022-07-20T03:07:10.207042+0000","last_became_peered":"2022-07-20T03:07:10.207042+0000","last_unstale":"2022-07-20T03:07:11.732562+0000","last_undegraded":"2022-07-20T03:07:11.732562+0000","last_fullsized":"2022-07-20T03:07:11.732562+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":5,"ondisk_log_size":5,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":34,"num_objects":3,"num_object_clones":0,"num_object_copies":6,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":3,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":6,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,0],"acting":[4,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"4.2","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196571+0000","last_change":"2022-07-20T03:07:10.730218+0000","last_active":"2022-07-20T03:07:12.196571+0000","last_peered":"2022-07-20T03:07:12.196571+0000","last_clean":"2022-07-20T03:07:12.196571+0000","last_became_active":"2022-07-20T03:07:10.729821+0000","last_became_peered":"2022-07-20T03:07:10.729821+0000","last_unstale":"2022-07-20T03:07:12.196571+0000","last_undegraded":"2022-07-20T03:07:12.196571+0000","last_fullsized":"2022-07-20T03:07:12.196571+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,5],"acting":[1,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"3.5","version":"21'6","reported_seq":17,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.731990+0000","last_change":"2022-07-20T03:07:10.204487+0000","last_active":"2022-07-20T03:07:11.731990+0000","last_peered":"2022-07-20T03:07:11.731990+0000","last_clean":"2022-07-20T03:07:11.731990+0000","last_became_active":"2022-07-20T03:07:10.203986+0000","last_became_peered":"2022-07-20T03:07:10.203986+0000","last_unstale":"2022-07-20T03:07:11.731990+0000","last_undegraded":"2022-07-20T03:07:11.731990+0000","last_fullsized":"2022-07-20T03:07:11.731990+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":6,"ondisk_log_size":6,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":160,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,3],"acting":[5,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196944+0000","last_change":"2022-07-20T03:07:06.951643+0000","last_active":"2022-07-20T03:07:12.196944+0000","last_peered":"2022-07-20T03:07:12.196944+0000","last_clean":"2022-07-20T03:07:12.196944+0000","last_became_active":"2022-07-20T03:07:06.951291+0000","last_became_peered":"2022-07-20T03:07:06.951291+0000","last_unstale":"2022-07-20T03:07:12.196944+0000","last_undegraded":"2022-07-20T03:07:12.196944+0000","last_fullsized":"2022-07-20T03:07:12.196944+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.3","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.194035+0000","last_change":"2022-07-20T03:07:10.732961+0000","last_active":"2022-07-20T03:07:12.194035+0000","last_peered":"2022-07-20T03:07:12.194035+0000","last_clean":"2022-07-20T03:07:12.194035+0000","last_became_active":"2022-07-20T03:07:10.732470+0000","last_became_peered":"2022-07-20T03:07:10.732470+0000","last_unstale":"2022-07-20T03:07:12.194035+0000","last_undegraded":"2022-07-20T03:07:12.194035+0000","last_fullsized":"2022-07-20T03:07:12.194035+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,5],"acting":[0,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"3.4","version":"21'8","reported_seq":19,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196820+0000","last_change":"2022-07-20T03:07:10.211560+0000","last_active":"2022-07-20T03:07:12.196820+0000","last_peered":"2022-07-20T03:07:12.196820+0000","last_clean":"2022-07-20T03:07:12.196820+0000","last_became_active":"2022-07-20T03:07:10.211088+0000","last_became_peered":"2022-07-20T03:07:10.211088+0000","last_unstale":"2022-07-20T03:07:12.196820+0000","last_undegraded":"2022-07-20T03:07:12.196820+0000","last_fullsized":"2022-07-20T03:07:12.196820+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":8,"ondisk_log_size":8,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":1535,"num_objects":5,"num_object_clones":0,"num_object_copies":10,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":5,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":9,"num_write_kb":3,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,2],"acting":[1,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":13,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.731982+0000","last_change":"2022-07-20T03:07:06.943629+0000","last_active":"2022-07-20T03:07:10.731982+0000","last_peered":"2022-07-20T03:07:10.731982+0000","last_clean":"2022-07-20T03:07:10.731982+0000","last_became_active":"2022-07-20T03:07:06.942970+0000","last_became_peered":"2022-07-20T03:07:06.942970+0000","last_unstale":"2022-07-20T03:07:10.731982+0000","last_undegraded":"2022-07-20T03:07:10.731982+0000","last_fullsized":"2022-07-20T03:07:10.731982+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"4.0","version":"0'0","reported_seq":9,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.740129+0000","last_change":"2022-07-20T03:07:10.737196+0000","last_active":"2022-07-20T03:07:10.740129+0000","last_peered":"2022-07-20T03:07:10.740129+0000","last_clean":"2022-07-20T03:07:10.740129+0000","last_became_active":"2022-07-20T03:07:10.736744+0000","last_became_peered":"2022-07-20T03:07:10.736744+0000","last_unstale":"2022-07-20T03:07:10.740129+0000","last_undegraded":"2022-07-20T03:07:10.740129+0000","last_fullsized":"2022-07-20T03:07:10.740129+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"3.7","version":"21'7","reported_seq":17,"reported_epoch":21,"state":"active+clean","last_fresh":"2022-07-20T03:07:10.741321+0000","last_change":"2022-07-20T03:07:10.208376+0000","last_active":"2022-07-20T03:07:10.741321+0000","last_peered":"2022-07-20T03:07:10.741321+0000","last_clean":"2022-07-20T03:07:10.741321+0000","last_became_active":"2022-07-20T03:07:10.206566+0000","last_became_peered":"2022-07-20T03:07:10.206566+0000","last_unstale":"2022-07-20T03:07:10.741321+0000","last_undegraded":"2022-07-20T03:07:10.741321+0000","last_fullsized":"2022-07-20T03:07:10.741321+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":7,"ondisk_log_size":7,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":575,"num_objects":4,"num_object_clones":0,"num_object_copies":8,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":4,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":8,"num_write_kb":2,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[3,7],"acting":[3,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":3,"acting_primary":3,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":13,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196983+0000","last_change":"2022-07-20T03:07:06.950534+0000","last_active":"2022-07-20T03:07:12.196983+0000","last_peered":"2022-07-20T03:07:12.196983+0000","last_clean":"2022-07-20T03:07:12.196983+0000","last_became_active":"2022-07-20T03:07:06.949930+0000","last_became_peered":"2022-07-20T03:07:06.949930+0000","last_unstale":"2022-07-20T03:07:12.196983+0000","last_undegraded":"2022-07-20T03:07:12.196983+0000","last_fullsized":"2022-07-20T03:07:12.196983+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"4.1","version":"0'0","reported_seq":10,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.732445+0000","last_change":"2022-07-20T03:07:10.726332+0000","last_active":"2022-07-20T03:07:11.732445+0000","last_peered":"2022-07-20T03:07:11.732445+0000","last_clean":"2022-07-20T03:07:11.732445+0000","last_became_active":"2022-07-20T03:07:10.725868+0000","last_became_peered":"2022-07-20T03:07:10.725868+0000","last_unstale":"2022-07-20T03:07:11.732445+0000","last_undegraded":"2022-07-20T03:07:11.732445+0000","last_fullsized":"2022-07-20T03:07:11.732445+0000","mapping_epoch":20,"log_start":"0'0","ondisk_log_start":"0'0","created":20,"last_epoch_clean":21,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:10.192190+0000","last_clean_scrub_stamp":"2022-07-20T03:07:10.192190+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[4,5],"acting":[4,5],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":4,"acting_primary":4,"purged_snaps":[]},{"pgid":"3.6","version":"22'4","reported_seq":15,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:12.196362+0000","last_change":"2022-07-20T03:07:10.211194+0000","last_active":"2022-07-20T03:07:12.196362+0000","last_peered":"2022-07-20T03:07:12.196362+0000","last_clean":"2022-07-20T03:07:12.196362+0000","last_became_active":"2022-07-20T03:07:10.210872+0000","last_became_peered":"2022-07-20T03:07:10.210872+0000","last_unstale":"2022-07-20T03:07:12.196362+0000","last_undegraded":"2022-07-20T03:07:12.196362+0000","last_fullsized":"2022-07-20T03:07:12.196362+0000","mapping_epoch":19,"log_start":"0'0","ondisk_log_start":"0'0","created":19,"last_epoch_clean":20,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:09.189287+0000","last_clean_scrub_stamp":"2022-07-20T03:07:09.189287+0000","log_size":4,"ondisk_log_size":4,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":2,"num_object_clones":0,"num_object_copies":4,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":2,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":4,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":2,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[0,1],"acting":[0,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":0,"acting_primary":0,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":11,"reported_epoch":22,"state":"active+clean","last_fresh":"2022-07-20T03:07:11.734902+0000","last_change":"2022-07-20T03:07:06.946858+0000","last_active":"2022-07-20T03:07:11.734902+0000","last_peered":"2022-07-20T03:07:11.734902+0000","last_clean":"2022-07-20T03:07:11.734902+0000","last_became_active":"2022-07-20T03:07:06.946555+0000","last_became_peered":"2022-07-20T03:07:06.946555+0000","last_unstale":"2022-07-20T03:07:11.734902+0000","last_undegraded":"2022-07-20T03:07:11.734902+0000","last_fullsized":"2022-07-20T03:07:11.734902+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":4,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":3,"num_pg":8,"stat_sum":{"num_bytes":2304,"num_objects":22,"num_object_clones":0,"num_object_copies":44,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":22,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":45,"num_write_kb":13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":14,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":39,"ondisk_log_size":39,"up":16,"acting":16,"num_store_stats":0},{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834574854,"num_pgs":6,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795860,"kb_used_data":92903520,"kb_used_omap":63,"kb_used_meta":0,"kb_avail":92903520,"statfs":{"total":95948165120,"available":95133204480,"internally_reserved":0,"allocated":95133204480,"data_stored":95133204480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":64831,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.872}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.68799999999999994}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72699999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88100000000000001}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92100000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.091}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0489999999999999}]}]},{"osd":6,"up_from":13,"seq":55834574855,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795852,"kb_used_data":92903528,"kb_used_omap":42,"kb_used_meta":0,"kb_avail":92903528,"statfs":{"total":95948165120,"available":95133212672,"internally_reserved":0,"allocated":95133212672,"data_stored":95133212672,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":43355,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.69299999999999995}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.071}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77600000000000002}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84099999999999997}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.196}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1779999999999999}]}]},{"osd":5,"up_from":13,"seq":55834574854,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795860,"kb_used_data":92903520,"kb_used_omap":77,"kb_used_meta":0,"kb_avail":92903520,"statfs":{"total":95948165120,"available":95133204480,"internally_reserved":0,"allocated":95133204480,"data_stored":95133204480,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":79443,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.97599999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.67100000000000004}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.60899999999999999}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.747}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79900000000000004}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.89600000000000002}]}]},{"osd":4,"up_from":13,"seq":55834574855,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903524,"kb_used_omap":57,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":0,"allocated":95133208576,"data_stored":95133208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":58670,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0840000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.83599999999999997}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.77000000000000002}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.79300000000000004}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.877}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.94699999999999995}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.92900000000000005}]}]},{"osd":3,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795868,"kb_used_data":92903512,"kb_used_omap":65,"kb_used_meta":0,"kb_avail":92903512,"statfs":{"total":95948165120,"available":95133196288,"internally_reserved":0,"allocated":95133196288,"data_stored":95133196288,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":66802,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.50800000000000001}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.61899999999999999}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52900000000000003}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.72899999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.80100000000000005}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.63400000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":2,"up_from":13,"seq":55834574854,"num_pgs":5,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795864,"kb_used_data":92903516,"kb_used_omap":72,"kb_used_meta":0,"kb_avail":92903516,"statfs":{"total":95948165120,"available":95133200384,"internally_reserved":0,"allocated":95133200384,"data_stored":95133200384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":74125,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.98099999999999998}]},{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.74099999999999999}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84899999999999998}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.88900000000000001}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.099}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.82899999999999996}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.069}]}]},{"osd":1,"up_from":13,"seq":55834574854,"num_pgs":11,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795864,"kb_used_data":92903516,"kb_used_omap":102,"kb_used_meta":0,"kb_avail":92903516,"statfs":{"total":95948165120,"available":95133200384,"internally_reserved":0,"allocated":95133200384,"data_stored":95133200384,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":104682,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.52300000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.76100000000000001}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.81499999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.84999999999999998}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.86099999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.998}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.1000000000000001}]}]},{"osd":0,"up_from":13,"seq":55834574855,"num_pgs":8,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":795856,"kb_used_data":92903524,"kb_used_omap":85,"kb_used_meta":0,"kb_avail":92903524,"statfs":{"total":95948165120,"available":95133208576,"internally_reserved":0,"allocated":95133208576,"data_stored":95133208576,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":87908,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.45700000000000002}]},{"osd":2,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.54300000000000004}]},{"osd":3,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.57299999999999995}]},{"osd":4,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.028}]},{"osd":5,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":0.99399999999999999}]},{"osd":6,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.0600000000000001}]},{"osd":7,"last update":"Thu Jan 1 00:00:00 1970","interfaces":[{"interface":"back","average":{"1min":0,"5min":0,"15min":0},"min":{"1min":0,"5min":0,"15min":0},"max":{"1min":0,"5min":0,"15min":0},"last":1.115}]}]}]}} 2022-07-20T03:07:23.051 INFO:tasks.ceph.ceph_manager.ceph:clean! 2022-07-20T03:07:23.051 INFO:tasks.ceph:Waiting until ceph cluster ceph is healthy... 2022-07-20T03:07:23.052 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy 2022-07-20T03:07:23.052 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph health --format=json 2022-07-20T03:07:23.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 Processor -- start 2022-07-20T03:07:23.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 -- start start 2022-07-20T03:07:23.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb1900b5b30 0x7fb1900aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b13e0 0x7fb1901463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1900b5f10 con 0x7fb1900ace90 2022-07-20T03:07:23.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1900afc40 con 0x7fb1900b5b30 2022-07-20T03:07:23.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb195aad700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1900afee0 con 0x7fb1900b13e0 2022-07-20T03:07:23.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb187fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb1900ace90 0x7fb1900b0cd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39328/0 (socket says 172.21.15.61:39328) 2022-07-20T03:07:23.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb18f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb1900b5b30 0x7fb1900aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb187fff700 1 -- 172.21.15.61:0/1214395929 learned_addr learned my addr 172.21.15.61:0/1214395929 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.159+0000 7fb18ffff700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b13e0 0x7fb1901463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18effd700 1 -- 172.21.15.61:0/1214395929 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 322295354 0 0) 0x7fb1900b5f10 con 0x7fb1900ace90 2022-07-20T03:07:23.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18effd700 1 -- 172.21.15.61:0/1214395929 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb170003260 con 0x7fb1900ace90 2022-07-20T03:07:23.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18f7fe700 1 -- 172.21.15.61:0/1214395929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b13e0 msgr2=0x7fb1901463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:23.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18f7fe700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b13e0 0x7fb1901463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18f7fe700 1 -- 172.21.15.61:0/1214395929 >> v1:172.21.15.61:6789/0 conn(0x7fb1900ace90 legacy=0x7fb1900b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18f7fe700 1 -- 172.21.15.61:0/1214395929 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1900ad490 con 0x7fb1900b5b30 2022-07-20T03:07:23.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18ffff700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b13e0 0x7fb1901463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:23.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18f7fe700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900b5b30 0x7fb1900aefb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb180002f70 tx=0x7fb180006df0).ready entity=mon.0 client_cookie=7f33ecb1d7234c2d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:23.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18effd700 1 -- 172.21.15.61:0/1214395929 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb18000d4a0 con 0x7fb1900b5b30 2022-07-20T03:07:23.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18effd700 1 -- 172.21.15.61:0/1214395929 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb180000e90 con 0x7fb1900b5b30 2022-07-20T03:07:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb18effd700 1 -- 172.21.15.61:0/1214395929 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb18000d7f0 con 0x7fb1900b5b30 2022-07-20T03:07:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 -- 172.21.15.61:0/1214395929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900b5b30 msgr2=0x7fb1900aefb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900b5b30 0x7fb1900aefb0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fb180002f70 tx=0x7fb180006df0).stop 2022-07-20T03:07:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 -- 172.21.15.61:0/1214395929 shutdown_connections 2022-07-20T03:07:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b13e0 0x7fb1901463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 --2- 172.21.15.61:0/1214395929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900b5b30 0x7fb1900aefb0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 -- 172.21.15.61:0/1214395929 >> 172.21.15.61:0/1214395929 conn(0x7fb1900635b0 msgr2=0x7fb1900b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 -- 172.21.15.61:0/1214395929 shutdown_connections 2022-07-20T03:07:23.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.163+0000 7fb195aad700 1 -- 172.21.15.61:0/1214395929 wait complete. 2022-07-20T03:07:23.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 Processor -- start 2022-07-20T03:07:23.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 -- start start 2022-07-20T03:07:23.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 0x7fb190066ca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1900b13e0 0x7fb1900675e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b5b30 0x7fb190143c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014ab70 con 0x7fb1900ace90 2022-07-20T03:07:23.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014a630 con 0x7fb1900b13e0 2022-07-20T03:07:23.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb195aad700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb19014a8d0 con 0x7fb1900b5b30 2022-07-20T03:07:23.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 0x7fb190066ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 0x7fb190066ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42736/0 (socket says 172.21.15.61:42736) 2022-07-20T03:07:23.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 -- 172.21.15.61:0/3777796276 learned_addr learned my addr 172.21.15.61:0/3777796276 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:23.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18ffff700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b5b30 0x7fb190143c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb187fff700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1900b13e0 0x7fb1900675e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 -- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b5b30 msgr2=0x7fb190143c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b5b30 0x7fb190143c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 -- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1900b13e0 msgr2=0x7fb1900675e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1900b13e0 0x7fb1900675e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb18f7fe700 1 -- 172.21.15.61:0/3777796276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb188009470 con 0x7fb1900ace90 2022-07-20T03:07:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.167+0000 7fb187fff700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1900b13e0 0x7fb1900675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:23.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18f7fe700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 0x7fb190066ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb180002f70 tx=0x7fb180014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:23.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb180016070 con 0x7fb1900ace90 2022-07-20T03:07:23.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb180013a40 con 0x7fb1900ace90 2022-07-20T03:07:23.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb180013da0 con 0x7fb1900ace90 2022-07-20T03:07:23.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb18000b180 con 0x7fb1900ace90 2022-07-20T03:07:23.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb19014b1a0 con 0x7fb1900ace90 2022-07-20T03:07:23.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1900574e0 con 0x7fb1900ace90 2022-07-20T03:07:23.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb180025030 con 0x7fb1900ace90 2022-07-20T03:07:23.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18d7fa700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb174053a40 0x7fb174055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb18006cf40 con 0x7fb1900ace90 2022-07-20T03:07:23.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.171+0000 7fb187fff700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb174053a40 0x7fb174055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.175+0000 7fb187fff700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb174053a40 0x7fb174055f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb17800b000 tx=0x7fb178009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:23.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.179+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb180043ea0 con 0x7fb1900ace90 2022-07-20T03:07:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.351+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "health", "format": "json"} v 0) v1 -- 0x7fb190067ea0 con 0x7fb1900ace90 2022-07-20T03:07:23.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.351+0000 7fb18d7fa700 1 -- 172.21.15.61:0/3777796276 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "health", "format": "json"}]=0 v0) v1 ==== 72+0+45 (secure 0 0 0) 0x7fb180023020 con 0x7fb1900ace90 2022-07-20T03:07:23.396 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.351+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb174053a40 msgr2=0x7fb174055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.351+0000 7fb195aad700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb174053a40 0x7fb174055f00 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fb17800b000 tx=0x7fb178009040).stop 2022-07-20T03:07:23.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.351+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 msgr2=0x7fb190066ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.351+0000 7fb195aad700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 0x7fb190066ca0 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fb180002f70 tx=0x7fb180014000).stop 2022-07-20T03:07:23.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 shutdown_connections 2022-07-20T03:07:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb174053a40 0x7fb174055f00 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1900b13e0 0x7fb1900675e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1900b5b30 0x7fb190143c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 --2- 172.21.15.61:0/3777796276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1900ace90 0x7fb190066ca0 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 >> 172.21.15.61:0/3777796276 conn(0x7fb1900635b0 msgr2=0x7fb190147110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:23.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 shutdown_connections 2022-07-20T03:07:23.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.355+0000 7fb195aad700 1 -- 172.21.15.61:0/3777796276 wait complete. 2022-07-20T03:07:23.475 INFO:teuthology.orchestra.run.smithi061.stdout:{"status":"HEALTH_OK","checks":{},"mutes":[]} 2022-07-20T03:07:23.475 INFO:tasks.ceph.ceph_manager.ceph:wait_until_healthy done 2022-07-20T03:07:23.512 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.643+0000 7febe9cc3700 1 Processor -- start 2022-07-20T03:07:23.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- start start 2022-07-20T03:07:23.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40608b0 0x7febe405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7febe40afa10 0x7febe40b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febe40b42a0 con 0x7febe40b3de0 2022-07-20T03:07:23.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febe4060c90 con 0x7febe40afa10 2022-07-20T03:07:23.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40af410 con 0x7febe40608b0 2022-07-20T03:07:23.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7febe40afa10 0x7febe40b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7febe40afa10 0x7febe40b46d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42748/0 (socket says 172.21.15.61:42748) 2022-07-20T03:07:23.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 -- 172.21.15.61:0/2577069514 learned_addr learned my addr 172.21.15.61:0/2577069514 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:23.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe37fe700 1 --2- 172.21.15.61:0/2577069514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40608b0 0x7febe405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 -- 172.21.15.61:0/2577069514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40608b0 msgr2=0x7febe405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:23.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 --2- 172.21.15.61:0/2577069514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40608b0 0x7febe405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 -- 172.21.15.61:0/2577069514 >> v1:172.21.15.117:6789/0 conn(0x7febe40b3de0 legacy=0x7febe4146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 -- 172.21.15.61:0/2577069514 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febe40af6b0 con 0x7febe40afa10 2022-07-20T03:07:23.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe27fc700 1 -- 172.21.15.61:0/2577069514 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3307053029 0 0) 0x7febe40b42a0 con 0x7febe40b3de0 2022-07-20T03:07:23.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe2ffd700 1 --2- 172.21.15.61:0/2577069514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40afa10 0x7febe40b46d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7febd8002f70 tx=0x7febd8006eb0).ready entity=mon.0 client_cookie=236d448f27aba5f9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:23.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe27fc700 1 -- 172.21.15.61:0/2577069514 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd800d570 con 0x7febe40afa10 2022-07-20T03:07:23.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe27fc700 1 -- 172.21.15.61:0/2577069514 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8000e90 con 0x7febe40afa10 2022-07-20T03:07:23.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe27fc700 1 -- 172.21.15.61:0/2577069514 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd800d8c0 con 0x7febe40afa10 2022-07-20T03:07:23.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- 172.21.15.61:0/2577069514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40afa10 msgr2=0x7febe40b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 --2- 172.21.15.61:0/2577069514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40afa10 0x7febe40b46d0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7febd8002f70 tx=0x7febd8006eb0).stop 2022-07-20T03:07:23.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- 172.21.15.61:0/2577069514 shutdown_connections 2022-07-20T03:07:23.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 --2- 172.21.15.61:0/2577069514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40afa10 0x7febe40b46d0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 --2- 172.21.15.61:0/2577069514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40608b0 0x7febe405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- 172.21.15.61:0/2577069514 >> 172.21.15.61:0/2577069514 conn(0x7febe40b5ee0 msgr2=0x7febe40b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:23.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- 172.21.15.61:0/2577069514 shutdown_connections 2022-07-20T03:07:23.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.647+0000 7febe9cc3700 1 -- 172.21.15.61:0/2577069514 wait complete. 2022-07-20T03:07:23.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 Processor -- start 2022-07-20T03:07:23.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 -- start start 2022-07-20T03:07:23.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe40608b0 0x7febe4091bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40afa10 0x7febe4151c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 0x7febe4152120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe4152eb0 con 0x7febe40b3de0 2022-07-20T03:07:23.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febe40915f0 con 0x7febe40608b0 2022-07-20T03:07:23.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febe40944b0 con 0x7febe40afa10 2022-07-20T03:07:23.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 0x7febe4152120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 0x7febe4152120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42754/0 (socket says 172.21.15.61:42754) 2022-07-20T03:07:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 -- 172.21.15.61:0/3720433696 learned_addr learned my addr 172.21.15.61:0/3720433696 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe2ffd700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40afa10 0x7febe4151c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe37fe700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe40608b0 0x7febe4091bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 -- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40afa10 msgr2=0x7febe4151c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40afa10 0x7febe4151c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 -- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe40608b0 msgr2=0x7febe4091bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe40608b0 0x7febe4091bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 -- 172.21.15.61:0/3720433696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd800b1e0 con 0x7febe40b3de0 2022-07-20T03:07:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe3fff700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 0x7febe4152120 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7febdc000f40 tx=0x7febdc007f60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febdc006aa0 con 0x7febe40b3de0 2022-07-20T03:07:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febe4094750 con 0x7febe40b3de0 2022-07-20T03:07:23.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc006c00 con 0x7febe40b3de0 2022-07-20T03:07:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febdc007030 con 0x7febe40b3de0 2022-07-20T03:07:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.651+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febe408ffb0 con 0x7febe40b3de0 2022-07-20T03:07:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.655+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febe40920a0 con 0x7febe40b3de0 2022-07-20T03:07:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.655+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7febdc010070 con 0x7febe40b3de0 2022-07-20T03:07:23.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.655+0000 7febe0ff9700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febc4053ad0 0x7febc4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:23.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.655+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7febdc06bb80 con 0x7febe40b3de0 2022-07-20T03:07:23.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.655+0000 7febe37fe700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febc4053ad0 0x7febc4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:23.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.655+0000 7febe37fe700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febc4053ad0 0x7febc4055f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7febd400b000 tx=0x7febd4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:23.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.663+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febe40920a0 con 0x7febe40b3de0 2022-07-20T03:07:23.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.791+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7febe40920a0 con 0x7febe40b3de0 2022-07-20T03:07:23.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe0ff9700 1 -- 172.21.15.61:0/3720433696 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7febe40920a0 con 0x7febe40b3de0 2022-07-20T03:07:23.821 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:23.822 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febc4053ad0 msgr2=0x7febc4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febc4053ad0 0x7febc4055f90 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7febd400b000 tx=0x7febd4006df0).stop 2022-07-20T03:07:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 msgr2=0x7febe4152120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:23.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 0x7febe4152120 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7febdc000f40 tx=0x7febdc007f60).stop 2022-07-20T03:07:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 shutdown_connections 2022-07-20T03:07:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febc4053ad0 0x7febc4055f90 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febe40b3de0 0x7febe4152120 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febe40afa10 0x7febe4151c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 --2- 172.21.15.61:0/3720433696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febe40608b0 0x7febe4091bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 >> 172.21.15.61:0/3720433696 conn(0x7febe40b5ee0 msgr2=0x7febe40ae600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 shutdown_connections 2022-07-20T03:07:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.795+0000 7febe9cc3700 1 -- 172.21.15.61:0/3720433696 wait complete. 2022-07-20T03:07:23.827 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:23.828 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:24.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.939+0000 7fdd4a9f6700 1 Processor -- start 2022-07-20T03:07:24.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.939+0000 7fdd4a9f6700 1 -- start start 2022-07-20T03:07:24.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd4a9f6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdd440b34d0 0x7fdd44061ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd4a9f6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd4a9f6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd4405f840 con 0x7fdd440ab180 2022-07-20T03:07:24.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd4a9f6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd44142b90 con 0x7fdd440b34d0 2022-07-20T03:07:24.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd4a9f6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd4405fa70 con 0x7fdd44142720 2022-07-20T03:07:24.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd437fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdd440ab180 0x7fdd44142010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39358/0 (socket says 172.21.15.61:39358) 2022-07-20T03:07:24.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd437fe700 1 -- 172.21.15.61:0/3309492508 learned_addr learned my addr 172.21.15.61:0/3309492508 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:24.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd48f93700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd43fff700 1 --2- 172.21.15.61:0/3309492508 >> v2:172.21.15.117:3300/0 conn(0x7fdd440b34d0 0x7fdd44061ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd42ffd700 1 -- 172.21.15.61:0/3309492508 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2883595485 0 0) 0x7fdd4405f840 con 0x7fdd440ab180 2022-07-20T03:07:24.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd42ffd700 1 -- 172.21.15.61:0/3309492508 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdd24003260 con 0x7fdd440ab180 2022-07-20T03:07:24.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd43fff700 1 -- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 msgr2=0x7fdd44142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:24.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd43fff700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd43fff700 1 -- 172.21.15.61:0/3309492508 >> v1:172.21.15.61:6789/0 conn(0x7fdd440ab180 legacy=0x7fdd44142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd43fff700 1 -- 172.21.15.61:0/3309492508 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd4405fd10 con 0x7fdd440b34d0 2022-07-20T03:07:24.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd48f93700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:24.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd43fff700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44061ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdd44060870 tx=0x7fdd34006df0).ready entity=mon.1 client_cookie=a00599181ef83a45 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd42ffd700 1 -- 172.21.15.61:0/3309492508 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd3400d570 con 0x7fdd440b34d0 2022-07-20T03:07:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd42ffd700 1 -- 172.21.15.61:0/3309492508 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34000e90 con 0x7fdd440b34d0 2022-07-20T03:07:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.943+0000 7fdd42ffd700 1 -- 172.21.15.61:0/3309492508 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd3400d8c0 con 0x7fdd440b34d0 2022-07-20T03:07:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 msgr2=0x7fdd44061ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44061ba0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7fdd44060870 tx=0x7fdd34006df0).stop 2022-07-20T03:07:24.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/3309492508 shutdown_connections 2022-07-20T03:07:24.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/3309492508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44061ba0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/3309492508 >> 172.21.15.61:0/3309492508 conn(0x7fdd440b6c50 msgr2=0x7fdd440b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:24.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/3309492508 shutdown_connections 2022-07-20T03:07:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/3309492508 wait complete. 2022-07-20T03:07:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 Processor -- start 2022-07-20T03:07:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- start start 2022-07-20T03:07:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 0x7fdd440b2350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44149ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44155200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44148830 con 0x7fdd440ab180 2022-07-20T03:07:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd440992d0 con 0x7fdd440b34d0 2022-07-20T03:07:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd4a9f6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd44099570 con 0x7fdd44142720 2022-07-20T03:07:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 0x7fdd440b2350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 0x7fdd440b2350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42764/0 (socket says 172.21.15.61:42764) 2022-07-20T03:07:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd43fff700 1 -- 172.21.15.61:0/640332243 learned_addr learned my addr 172.21.15.61:0/640332243 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd48f93700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44155200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.947+0000 7fdd437fe700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44149ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd43fff700 1 -- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 msgr2=0x7fdd44155200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd43fff700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44155200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd43fff700 1 -- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 msgr2=0x7fdd44149ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd43fff700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44149ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd43fff700 1 -- 172.21.15.61:0/640332243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd3c009470 con 0x7fdd440ab180 2022-07-20T03:07:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd48f93700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd437fe700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd43fff700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 0x7fdd440b2350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdd44060870 tx=0x7fdd34014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd34016070 con 0x7fdd440ab180 2022-07-20T03:07:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd34013a40 con 0x7fdd440ab180 2022-07-20T03:07:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd34013da0 con 0x7fdd440ab180 2022-07-20T03:07:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd3400b180 con 0x7fdd440ab180 2022-07-20T03:07:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd4409ae30 con 0x7fdd440ab180 2022-07-20T03:07:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd4409b530 con 0x7fdd440ab180 2022-07-20T03:07:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.951+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdd3400f030 con 0x7fdd440ab180 2022-07-20T03:07:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.955+0000 7fdd417fa700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd280539b0 0x7fdd28055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.955+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdd3406cd00 con 0x7fdd440ab180 2022-07-20T03:07:24.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.955+0000 7fdd437fe700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd280539b0 0x7fdd28055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.955+0000 7fdd437fe700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd280539b0 0x7fdd28055e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fdd2c00b000 tx=0x7fdd2c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:23.963+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd4409b530 con 0x7fdd440ab180 2022-07-20T03:07:24.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.091+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdd4409b530 con 0x7fdd440ab180 2022-07-20T03:07:24.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.091+0000 7fdd417fa700 1 -- 172.21.15.61:0/640332243 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdd4409b530 con 0x7fdd440ab180 2022-07-20T03:07:24.240 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:24.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd280539b0 msgr2=0x7fdd28055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd280539b0 0x7fdd28055e70 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fdd2c00b000 tx=0x7fdd2c009040).stop 2022-07-20T03:07:24.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 msgr2=0x7fdd440b2350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 0x7fdd440b2350 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fdd44060870 tx=0x7fdd34014000).stop 2022-07-20T03:07:24.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 shutdown_connections 2022-07-20T03:07:24.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd280539b0 0x7fdd28055e70 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd440b34d0 0x7fdd44149ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd44142720 0x7fdd44155200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 --2- 172.21.15.61:0/640332243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd440ab180 0x7fdd440b2350 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 >> 172.21.15.61:0/640332243 conn(0x7fdd440b6c50 msgr2=0x7fdd44061020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:24.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 shutdown_connections 2022-07-20T03:07:24.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.095+0000 7fdd4a9f6700 1 -- 172.21.15.61:0/640332243 wait complete. 2022-07-20T03:07:24.247 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:24.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.363+0000 7f3e628fc700 1 Processor -- start 2022-07-20T03:07:24.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.363+0000 7f3e628fc700 1 -- start start 2022-07-20T03:07:24.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 0x7f3e5c13a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 0x7f3e5c13aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3e5c065f20 con 0x7f3e5c0b48d0 2022-07-20T03:07:24.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3e5c0b29c0 con 0x7f3e5c143f00 2022-07-20T03:07:24.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c0661c0 con 0x7f3e5c05cda0 2022-07-20T03:07:24.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e620fb700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 0x7f3e5c13aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 0x7f3e5c13a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e610f9700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f3e5c0b48d0 0x7f3e5c0b38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:47874/0 (socket says 172.21.15.61:47874) 2022-07-20T03:07:24.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 0x7f3e5c13a590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:57676/0 (socket says 172.21.15.61:57676) 2022-07-20T03:07:24.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e610f9700 1 -- 172.21.15.61:0/2506023065 learned_addr learned my addr 172.21.15.61:0/2506023065 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:24.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e620fb700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 0x7f3e5c13aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:52456/0 (socket says 172.21.15.61:52456) 2022-07-20T03:07:24.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 -- 172.21.15.61:0/2506023065 >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 msgr2=0x7f3e5c13aa80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:24.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 --2- 172.21.15.61:0/2506023065 >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 0x7f3e5c13aa80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 -- 172.21.15.61:0/2506023065 >> v1:172.21.15.117:6789/0 conn(0x7f3e5c0b48d0 legacy=0x7f3e5c0b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e608f8700 1 -- 172.21.15.61:0/2506023065 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4107033492 0 0) 0x7f3e5c065f20 con 0x7f3e5c0b48d0 2022-07-20T03:07:24.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 -- 172.21.15.61:0/2506023065 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e5c066460 con 0x7f3e5c05cda0 2022-07-20T03:07:24.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e620fb700 1 --2- 172.21.15.61:0/2506023065 >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 0x7f3e5c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:24.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e618fa700 1 --2- 172.21.15.61:0/2506023065 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 0x7f3e5c13a590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f3e58002f70 tx=0x7f3e58006df0).ready entity=mon.2 client_cookie=15acc482ac802ec1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e608f8700 1 -- 172.21.15.61:0/2506023065 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5800d570 con 0x7f3e5c05cda0 2022-07-20T03:07:24.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e608f8700 1 -- 172.21.15.61:0/2506023065 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e58000e90 con 0x7f3e5c05cda0 2022-07-20T03:07:24.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e608f8700 1 -- 172.21.15.61:0/2506023065 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5800d8c0 con 0x7f3e5c05cda0 2022-07-20T03:07:24.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- 172.21.15.61:0/2506023065 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 msgr2=0x7f3e5c13a590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 --2- 172.21.15.61:0/2506023065 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 0x7f3e5c13a590 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f3e58002f70 tx=0x7f3e58006df0).stop 2022-07-20T03:07:24.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- 172.21.15.61:0/2506023065 shutdown_connections 2022-07-20T03:07:24.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 --2- 172.21.15.61:0/2506023065 >> v2:172.21.15.117:3300/0 conn(0x7f3e5c143f00 0x7f3e5c13aa80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 --2- 172.21.15.61:0/2506023065 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c05cda0 0x7f3e5c13a590 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- 172.21.15.61:0/2506023065 >> 172.21.15.61:0/2506023065 conn(0x7f3e5c064b90 msgr2=0x7f3e5c064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:24.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- 172.21.15.61:0/2506023065 shutdown_connections 2022-07-20T03:07:24.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.367+0000 7f3e628fc700 1 -- 172.21.15.61:0/2506023065 wait complete. 2022-07-20T03:07:24.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 Processor -- start 2022-07-20T03:07:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 -- start start 2022-07-20T03:07:24.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e5c05cda0 0x7f3e5c142790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c143f00 0x7f3e5c05fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 0x7f3e5c0602b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c143d50 con 0x7f3e5c142c80 2022-07-20T03:07:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c05f840 con 0x7f3e5c05cda0 2022-07-20T03:07:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3e5c143ab0 con 0x7f3e5c143f00 2022-07-20T03:07:24.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 0x7f3e5c0602b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 0x7f3e5c0602b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42782/0 (socket says 172.21.15.61:42782) 2022-07-20T03:07:24.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 -- 172.21.15.61:0/838046539 learned_addr learned my addr 172.21.15.61:0/838046539 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:24.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e618fa700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e5c05cda0 0x7f3e5c142790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e610f9700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c143f00 0x7f3e5c05fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 -- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c143f00 msgr2=0x7f3e5c05fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c143f00 0x7f3e5c05fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 -- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e5c05cda0 msgr2=0x7f3e5c142790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e5c05cda0 0x7f3e5c142790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 -- 172.21.15.61:0/838046539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3e54009470 con 0x7f3e5c142c80 2022-07-20T03:07:24.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e620fb700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 0x7f3e5c0602b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3e5400c000 tx=0x7f3e540091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5400ee60 con 0x7f3e5c142c80 2022-07-20T03:07:24.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3e5400c850 con 0x7f3e5c142c80 2022-07-20T03:07:24.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3e5800b180 con 0x7f3e5c142c80 2022-07-20T03:07:24.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3e5400cbb0 con 0x7f3e5c142c80 2022-07-20T03:07:24.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.371+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3e5c0af6e0 con 0x7f3e5c142c80 2022-07-20T03:07:24.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.375+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3e54022070 con 0x7f3e5c142c80 2022-07-20T03:07:24.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.375+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3e5c065ba0 con 0x7f3e5c142c80 2022-07-20T03:07:24.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.375+0000 7f3e4effd700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3e3c053a60 0x7f3e3c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.375+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3e5406c5a0 con 0x7f3e5c142c80 2022-07-20T03:07:24.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.375+0000 7f3e618fa700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3e3c053a60 0x7f3e3c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.375+0000 7f3e618fa700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3e3c053a60 0x7f3e3c055f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3e5801a000 tx=0x7f3e58014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.383+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3e54043500 con 0x7f3e5c142c80 2022-07-20T03:07:24.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.515+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3e5c0574e0 con 0x7f3e5c142c80 2022-07-20T03:07:24.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e4effd700 1 -- 172.21.15.61:0/838046539 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3e540160a0 con 0x7f3e5c142c80 2022-07-20T03:07:24.639 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:24.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3e3c053a60 msgr2=0x7f3e3c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3e3c053a60 0x7f3e3c055f20 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3e5801a000 tx=0x7f3e58014000).stop 2022-07-20T03:07:24.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 msgr2=0x7f3e5c0602b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 0x7f3e5c0602b0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f3e5400c000 tx=0x7f3e540091d0).stop 2022-07-20T03:07:24.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 shutdown_connections 2022-07-20T03:07:24.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3e3c053a60 0x7f3e3c055f20 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3e5c142c80 0x7f3e5c0602b0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3e5c143f00 0x7f3e5c05fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 --2- 172.21.15.61:0/838046539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3e5c05cda0 0x7f3e5c142790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 >> 172.21.15.61:0/838046539 conn(0x7f3e5c064b90 msgr2=0x7f3e5c066ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:24.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 shutdown_connections 2022-07-20T03:07:24.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.519+0000 7f3e628fc700 1 -- 172.21.15.61:0/838046539 wait complete. 2022-07-20T03:07:24.650 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.763+0000 7f24b9306700 1 Processor -- start 2022-07-20T03:07:24.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.763+0000 7f24b9306700 1 -- start start 2022-07-20T03:07:24.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b9306700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f24b40ab270 0x7f24b41425f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b9306700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b4146260 con 0x7f24b405f840 2022-07-20T03:07:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b9306700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b4146500 con 0x7f24b40acbe0 2022-07-20T03:07:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b9306700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b41470b0 con 0x7f24b40ab270 2022-07-20T03:07:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f24b405f840 0x7f24b4141ee0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39386/0 (socket says 172.21.15.61:39386) 2022-07-20T03:07:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b37fe700 1 -- 172.21.15.61:0/2968028682 learned_addr learned my addr 172.21.15.61:0/2968028682 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 --2- 172.21.15.61:0/2968028682 >> v2:172.21.15.117:3300/0 conn(0x7f24b40ab270 0x7f24b41425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3316162851 0 0) 0x7f24b4146260 con 0x7f24b405f840 2022-07-20T03:07:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2494003260 con 0x7f24b405f840 2022-07-20T03:07:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 692732426 0 0) 0x7f2494003260 con 0x7f24b405f840 2022-07-20T03:07:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 >> v2:172.21.15.117:3300/0 conn(0x7f24b40ab270 msgr2=0x7f24b41425f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 --2- 172.21.15.61:0/2968028682 >> v2:172.21.15.117:3300/0 conn(0x7f24b40ab270 0x7f24b41425f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 >> v1:172.21.15.117:6789/0 conn(0x7f24b40acbe0 legacy=0x7f24b41469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b40ab6d0 con 0x7f24b405f840 2022-07-20T03:07:24.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f249c001660 con 0x7f24b405f840 2022-07-20T03:07:24.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 >> v1:172.21.15.61:6789/0 conn(0x7f24b405f840 legacy=0x7f24b4141ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2494003750 0x7f2494003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004500 0x7f2494004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2494004e70 0x7f249400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249400bcf0 con 0x7f2494004e70 2022-07-20T03:07:24.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249400be60 con 0x7f2494003750 2022-07-20T03:07:24.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249400c020 con 0x7f2494004500 2022-07-20T03:07:24.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2494004e70 0x7f249400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b37fe700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004500 0x7f2494004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b3fff700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2494003750 0x7f2494003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 -- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004500 msgr2=0x7f2494004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:24.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004500 0x7f2494004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 -- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2494003750 msgr2=0x7f2494003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:24.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2494003750 0x7f2494003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 -- 172.21.15.61:0/2968028682 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2494003260 con 0x7f2494004e70 2022-07-20T03:07:24.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 -- 172.21.15.61:0/2968028682 reap_dead start 2022-07-20T03:07:24.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b8b05700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2494004e70 0x7f249400b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f24ac00c000 tx=0x7f24ac0091d0).ready entity=mon.0 client_cookie=8c710c7ff215789a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.767+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24ac00e7e0 con 0x7f2494004e70 2022-07-20T03:07:24.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac00e940 con 0x7f2494004e70 2022-07-20T03:07:24.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b2ffd700 1 -- 172.21.15.61:0/2968028682 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24ac00eca0 con 0x7f2494004e70 2022-07-20T03:07:24.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/2968028682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2494004e70 msgr2=0x7f249400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2494004e70 0x7f249400b750 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f24ac00c000 tx=0x7f24ac0091d0).stop 2022-07-20T03:07:24.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/2968028682 shutdown_connections 2022-07-20T03:07:24.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 --2- 172.21.15.61:0/2968028682 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2494004e70 0x7f249400b750 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/2968028682 >> 172.21.15.61:0/2968028682 conn(0x7f24b40b2af0 msgr2=0x7f24b40b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:24.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/2968028682 shutdown_connections 2022-07-20T03:07:24.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/2968028682 wait complete. 2022-07-20T03:07:24.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 Processor -- start 2022-07-20T03:07:24.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- start start 2022-07-20T03:07:24.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004e70 0x7f24b4096dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b409bda0 0x7f24b4151b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 0x7f24b4152510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40976c0 con 0x7f24b4152060 2022-07-20T03:07:24.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b409afa0 con 0x7f24b409bda0 2022-07-20T03:07:24.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b40965a0 con 0x7f2494004e70 2022-07-20T03:07:24.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 0x7f24b4152510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 0x7f24b4152510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42802/0 (socket says 172.21.15.61:42802) 2022-07-20T03:07:24.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 -- 172.21.15.61:0/4226753140 learned_addr learned my addr 172.21.15.61:0/4226753140 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:24.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b37fe700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b409bda0 0x7f24b4151b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b3fff700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004e70 0x7f24b4096dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 -- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004e70 msgr2=0x7f24b4096dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004e70 0x7f24b4096dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 -- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b409bda0 msgr2=0x7f24b4151b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b409bda0 0x7f24b4151b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 -- 172.21.15.61:0/4226753140 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b4096840 con 0x7f24b4152060 2022-07-20T03:07:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b37fe700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b409bda0 0x7f24b4151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b8b05700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 0x7f24b4152510 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f24ac00c000 tx=0x7f24ac019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24ac009600 con 0x7f24b4152060 2022-07-20T03:07:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24ac01de00 con 0x7f24b4152060 2022-07-20T03:07:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b4097420 con 0x7f24b4152060 2022-07-20T03:07:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24ac01b640 con 0x7f24b4152060 2022-07-20T03:07:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.771+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b409b9a0 con 0x7f24b4152060 2022-07-20T03:07:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.775+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b40acfc0 con 0x7f24b4152060 2022-07-20T03:07:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.775+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f24ac018020 con 0x7f24b4152060 2022-07-20T03:07:24.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.775+0000 7f24b17fa700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24980539d0 0x7f2498055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:24.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.775+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f24ac06dd70 con 0x7f24b4152060 2022-07-20T03:07:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.775+0000 7f24b3fff700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24980539d0 0x7f2498055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.775+0000 7f24b3fff700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24980539d0 0x7f2498055e90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f24a400d000 tx=0x7f24a4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.783+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24ac044d10 con 0x7f24b4152060 2022-07-20T03:07:24.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.907+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f24b40574e0 con 0x7f24b4152060 2022-07-20T03:07:24.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.907+0000 7f24b17fa700 1 -- 172.21.15.61:0/4226753140 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f24ac0482f0 con 0x7f24b4152060 2022-07-20T03:07:24.924 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:24.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24980539d0 msgr2=0x7f2498055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24980539d0 0x7f2498055e90 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f24a400d000 tx=0x7f24a4009040).stop 2022-07-20T03:07:24.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 msgr2=0x7f24b4152510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:24.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 0x7f24b4152510 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f24ac00c000 tx=0x7f24ac019000).stop 2022-07-20T03:07:24.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 shutdown_connections 2022-07-20T03:07:24.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24980539d0 0x7f2498055e90 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b409bda0 0x7f24b4151b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b4152060 0x7f24b4152510 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 --2- 172.21.15.61:0/4226753140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2494004e70 0x7f24b4096dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:24.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 >> 172.21.15.61:0/4226753140 conn(0x7f24b40b2af0 msgr2=0x7f24b4144d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:24.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 shutdown_connections 2022-07-20T03:07:24.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:24.911+0000 7f24b9306700 1 -- 172.21.15.61:0/4226753140 wait complete. 2022-07-20T03:07:24.935 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:25.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.059+0000 7f6c29da3700 1 Processor -- start 2022-07-20T03:07:25.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c29da3700 1 -- start start 2022-07-20T03:07:25.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c29da3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6c24140860 0x7f6c240675f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c29da3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6c24063270 0x7f6c2405d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c29da3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c240b16f0 con 0x7f6c2405cda0 2022-07-20T03:07:25.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c29da3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c240624b0 con 0x7f6c24140860 2022-07-20T03:07:25.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c29da3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c24140d30 con 0x7f6c24063270 2022-07-20T03:07:25.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c22ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6c24140860 0x7f6c240675f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c237fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6c2405cda0 0x7f6c24061da0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39404/0 (socket says 172.21.15.61:39404) 2022-07-20T03:07:25.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c237fe700 1 -- 172.21.15.61:0/3588001956 learned_addr learned my addr 172.21.15.61:0/3588001956 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:25.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c23fff700 1 --2- 172.21.15.61:0/3588001956 >> v2:172.21.15.117:3300/0 conn(0x7f6c24063270 0x7f6c2405d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c22ffd700 1 -- 172.21.15.61:0/3588001956 >> v2:172.21.15.117:3300/0 conn(0x7f6c24063270 msgr2=0x7f6c2405d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:25.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c22ffd700 1 --2- 172.21.15.61:0/3588001956 >> v2:172.21.15.117:3300/0 conn(0x7f6c24063270 0x7f6c2405d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c22ffd700 1 -- 172.21.15.61:0/3588001956 >> v1:172.21.15.61:6789/0 conn(0x7f6c2405cda0 legacy=0x7f6c24061da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c227fc700 1 -- 172.21.15.61:0/3588001956 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2201862111 0 0) 0x7f6c240b16f0 con 0x7f6c2405cda0 2022-07-20T03:07:25.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c22ffd700 1 -- 172.21.15.61:0/3588001956 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c24140fd0 con 0x7f6c24140860 2022-07-20T03:07:25.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c22ffd700 1 --2- 172.21.15.61:0/3588001956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c24140860 0x7f6c240675f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f6c240663f0 tx=0x7f6c14006df0).ready entity=mon.0 client_cookie=a41a5193675c8302 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:25.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c227fc700 1 -- 172.21.15.61:0/3588001956 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c1400d5c0 con 0x7f6c24140860 2022-07-20T03:07:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c227fc700 1 -- 172.21.15.61:0/3588001956 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c14000e90 con 0x7f6c24140860 2022-07-20T03:07:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.063+0000 7f6c227fc700 1 -- 172.21.15.61:0/3588001956 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c1400d910 con 0x7f6c24140860 2022-07-20T03:07:25.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 -- 172.21.15.61:0/3588001956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c24140860 msgr2=0x7f6c240675f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3588001956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c24140860 0x7f6c240675f0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f6c240663f0 tx=0x7f6c14006df0).stop 2022-07-20T03:07:25.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 -- 172.21.15.61:0/3588001956 shutdown_connections 2022-07-20T03:07:25.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3588001956 >> v2:172.21.15.117:3300/0 conn(0x7f6c24063270 0x7f6c2405d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3588001956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c24140860 0x7f6c240675f0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 -- 172.21.15.61:0/3588001956 >> 172.21.15.61:0/3588001956 conn(0x7f6c240b7da0 msgr2=0x7f6c240b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 -- 172.21.15.61:0/3588001956 shutdown_connections 2022-07-20T03:07:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 -- 172.21.15.61:0/3588001956 wait complete. 2022-07-20T03:07:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 Processor -- start 2022-07-20T03:07:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.067+0000 7f6c29da3700 1 -- start start 2022-07-20T03:07:25.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 0x7f6c24143ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c24063270 0x7f6c241531d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c24140860 0x7f6c241536c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c241430a0 con 0x7f6c2405cda0 2022-07-20T03:07:25.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c2409c8c0 con 0x7f6c24063270 2022-07-20T03:07:25.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c24150a00 con 0x7f6c24140860 2022-07-20T03:07:25.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 0x7f6c24143ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 0x7f6c24143ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42812/0 (socket says 172.21.15.61:42812) 2022-07-20T03:07:25.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 -- 172.21.15.61:0/3721532191 learned_addr learned my addr 172.21.15.61:0/3721532191 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:25.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c23fff700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c24140860 0x7f6c241536c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c22ffd700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c24063270 0x7f6c241531d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 -- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c24140860 msgr2=0x7f6c241536c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c24140860 0x7f6c241536c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 -- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c24063270 msgr2=0x7f6c241531d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c24063270 0x7f6c241531d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 -- 172.21.15.61:0/3721532191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c1400b180 con 0x7f6c2405cda0 2022-07-20T03:07:25.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c23fff700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c24140860 0x7f6c241536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:25.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c237fe700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 0x7f6c24143ac0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6c0c000ed0 tx=0x7f6c0c007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:25.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c0c0070d0 con 0x7f6c2405cda0 2022-07-20T03:07:25.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0c007230 con 0x7f6c2405cda0 2022-07-20T03:07:25.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c24143fb0 con 0x7f6c2405cda0 2022-07-20T03:07:25.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c0c005610 con 0x7f6c2405cda0 2022-07-20T03:07:25.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c24142400 con 0x7f6c2405cda0 2022-07-20T03:07:25.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.071+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c24150c20 con 0x7f6c2405cda0 2022-07-20T03:07:25.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.075+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6c0c010070 con 0x7f6c2405cda0 2022-07-20T03:07:25.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.075+0000 7f6c20ff9700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c080539b0 0x7f6c08055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.075+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6c0c06b640 con 0x7f6c2405cda0 2022-07-20T03:07:25.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.075+0000 7f6c22ffd700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c080539b0 0x7f6c08055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.075+0000 7f6c22ffd700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c080539b0 0x7f6c08055e70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6c1401a000 tx=0x7f6c14014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:25.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.083+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6c0c042500 con 0x7f6c2405cda0 2022-07-20T03:07:25.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6c241432a0 con 0x7f6c2405cda0 2022-07-20T03:07:25.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c20ff9700 1 -- 172.21.15.61:0/3721532191 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6c0c045b20 con 0x7f6c2405cda0 2022-07-20T03:07:25.541 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:25.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c080539b0 msgr2=0x7f6c08055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c080539b0 0x7f6c08055e70 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f6c1401a000 tx=0x7f6c14014000).stop 2022-07-20T03:07:25.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 msgr2=0x7f6c24143ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 0x7f6c24143ac0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f6c0c000ed0 tx=0x7f6c0c007ac0).stop 2022-07-20T03:07:25.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 shutdown_connections 2022-07-20T03:07:25.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c080539b0 0x7f6c08055e70 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c24063270 0x7f6c241531d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c24140860 0x7f6c241536c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 --2- 172.21.15.61:0/3721532191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c2405cda0 0x7f6c24143ac0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 >> 172.21.15.61:0/3721532191 conn(0x7f6c240b7da0 msgr2=0x7f6c240625f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:25.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 shutdown_connections 2022-07-20T03:07:25.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.211+0000 7f6c29da3700 1 -- 172.21.15.61:0/3721532191 wait complete. 2022-07-20T03:07:25.624 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:25.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 Processor -- start 2022-07-20T03:07:25.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- start start 2022-07-20T03:07:25.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21340b1620 0x7f2134142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21341426b0 con 0x7f2134142fd0 2022-07-20T03:07:25.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21340b1a00 con 0x7f213413ee40 2022-07-20T03:07:25.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2134060b50 con 0x7f21340b1620 2022-07-20T03:07:25.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21340b1620 0x7f2134142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21340b1620 0x7f2134142ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42820/0 (socket says 172.21.15.61:42820) 2022-07-20T03:07:25.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21327fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2134142fd0 0x7f213413e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39418/0 (socket says 172.21.15.61:39418) 2022-07-20T03:07:25.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 -- 172.21.15.61:0/4034979439 learned_addr learned my addr 172.21.15.61:0/4034979439 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:25.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 -- 172.21.15.61:0/4034979439 >> v1:172.21.15.117:6789/0 conn(0x7f213413ee40 legacy=0x7f21341499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 -- 172.21.15.61:0/4034979439 >> v1:172.21.15.61:6789/0 conn(0x7f2134142fd0 legacy=0x7f213413e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 -- 172.21.15.61:0/4034979439 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f213405f840 con 0x7f21340b1620 2022-07-20T03:07:25.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2132ffd700 1 --2- 172.21.15.61:0/4034979439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21340b1620 0x7f2134142ae0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2124002f70 tx=0x7f2124006e40).ready entity=mon.0 client_cookie=e10c8e8a7a95cc9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:25.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2131ffb700 1 -- 172.21.15.61:0/4034979439 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212400bd20 con 0x7f21340b1620 2022-07-20T03:07:25.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2131ffb700 1 -- 172.21.15.61:0/4034979439 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2124000e90 con 0x7f21340b1620 2022-07-20T03:07:25.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f2131ffb700 1 -- 172.21.15.61:0/4034979439 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21240155e0 con 0x7f21340b1620 2022-07-20T03:07:25.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- 172.21.15.61:0/4034979439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21340b1620 msgr2=0x7f2134142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 --2- 172.21.15.61:0/4034979439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21340b1620 0x7f2134142ae0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f2124002f70 tx=0x7f2124006e40).stop 2022-07-20T03:07:25.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- 172.21.15.61:0/4034979439 shutdown_connections 2022-07-20T03:07:25.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 --2- 172.21.15.61:0/4034979439 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21340b1620 0x7f2134142ae0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- 172.21.15.61:0/4034979439 >> 172.21.15.61:0/4034979439 conn(0x7f21340b23d0 msgr2=0x7f21340b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:25.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- 172.21.15.61:0/4034979439 shutdown_connections 2022-07-20T03:07:25.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.739+0000 7f21398da700 1 -- 172.21.15.61:0/4034979439 wait complete. 2022-07-20T03:07:25.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 Processor -- start 2022-07-20T03:07:25.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 -- start start 2022-07-20T03:07:25.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21340b1620 0x7f21340b4050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 0x7f2134143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2134142fd0 0x7f2134143f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:25.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134146260 con 0x7f213413ee40 2022-07-20T03:07:25.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2134146500 con 0x7f2134142fd0 2022-07-20T03:07:25.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2134145fa0 con 0x7f21340b1620 2022-07-20T03:07:25.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 0x7f2134143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 0x7f2134143a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42828/0 (socket says 172.21.15.61:42828) 2022-07-20T03:07:25.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 -- 172.21.15.61:0/3793942534 learned_addr learned my addr 172.21.15.61:0/3793942534 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:25.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f2132ffd700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21340b1620 0x7f21340b4050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21337fe700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2134142fd0 0x7f2134143f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:25.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 -- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21340b1620 msgr2=0x7f21340b4050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21340b1620 0x7f21340b4050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 -- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2134142fd0 msgr2=0x7f2134143f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:25.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2134142fd0 0x7f2134143f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:25.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 -- 172.21.15.61:0/3793942534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f212400d4d0 con 0x7f213413ee40 2022-07-20T03:07:25.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f2132ffd700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21340b1620 0x7f21340b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:25.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21327fc700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 0x7f2134143a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f2134062630 tx=0x7f212c001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:25.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212c005ae0 con 0x7f213413ee40 2022-07-20T03:07:26.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f212c005c40 con 0x7f213413ee40 2022-07-20T03:07:26.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21340b54c0 con 0x7f213413ee40 2022-07-20T03:07:26.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f212c0070d0 con 0x7f213413ee40 2022-07-20T03:07:26.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.743+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21340b5190 con 0x7f213413ee40 2022-07-20T03:07:26.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.747+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f212c010070 con 0x7f213413ee40 2022-07-20T03:07:26.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.747+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f213405fae0 con 0x7f213413ee40 2022-07-20T03:07:26.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.747+0000 7f212bfff700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2118053a40 0x7f2118055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:26.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.747+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f212c06b840 con 0x7f213413ee40 2022-07-20T03:07:26.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.747+0000 7f2132ffd700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2118053a40 0x7f2118055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:26.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.747+0000 7f2132ffd700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2118053a40 0x7f2118055f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f212400b000 tx=0x7f2124018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:26.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.751+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f212c0427a0 con 0x7f213413ee40 2022-07-20T03:07:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.915+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f21340574e0 con 0x7f213413ee40 2022-07-20T03:07:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.915+0000 7f212bfff700 1 -- 172.21.15.61:0/3793942534 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f212c045dc0 con 0x7f213413ee40 2022-07-20T03:07:26.131 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:26.132 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:12.741818+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.117:6835/725875269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":725875269},{"type":"v1","addr":"172.21.15.117:6835","nonce":725875269}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:26.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2118053a40 msgr2=0x7f2118055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:26.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2118053a40 0x7f2118055f00 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f212400b000 tx=0x7f2124018000).stop 2022-07-20T03:07:26.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 msgr2=0x7f2134143a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:26.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 0x7f2134143a10 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f2134062630 tx=0x7f212c001400).stop 2022-07-20T03:07:26.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 shutdown_connections 2022-07-20T03:07:26.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2118053a40 0x7f2118055f00 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:26.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2134142fd0 0x7f2134143f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:26.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f213413ee40 0x7f2134143a10 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:26.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 --2- 172.21.15.61:0/3793942534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21340b1620 0x7f21340b4050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:26.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 >> 172.21.15.61:0/3793942534 conn(0x7f21340b23d0 msgr2=0x7f213405fce0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:26.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 shutdown_connections 2022-07-20T03:07:26.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:25.919+0000 7f21398da700 1 -- 172.21.15.61:0/3793942534 wait complete. 2022-07-20T03:07:26.167 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2022-07-20T03:07:26.383 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:07:10.718708+0000', 'modified': '2022-07-20T03:07:12.741818+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.117:6835/725875269', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 725875269}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 725875269}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:07:26.384 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:07:26.395 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:07:26.512 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":4273,"want_state":"up:active","state":"up:active","rank_uptime":15.780591064999999,"mdsmap_epoch":7,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":19.322822503000001} 2022-07-20T03:07:26.513 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 4273, "mdsmap_epoch": 7, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 15.780591065, "state": "up:active", "uptime": 19.322822503, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:07:26.513 INFO:teuthology.run_tasks:Running task ceph-fuse... 2022-07-20T03:07:26.751 INFO:tasks.ceph_fuse:Running ceph_fuse task... 2022-07-20T03:07:26.752 INFO:tasks.ceph_fuse:config is None 2022-07-20T03:07:26.753 INFO:tasks.ceph_fuse:new config is {'client.0': None, 'client.1': None, 'client.2': None} 2022-07-20T03:07:26.753 INFO:tasks.ceph_fuse:Mounting ceph-fuse clients... 2022-07-20T03:07:26.753 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:26.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.883+0000 7ff29b6ab700 1 Processor -- start 2022-07-20T03:07:26.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.883+0000 7ff29b6ab700 1 -- start start 2022-07-20T03:07:26.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940b1e00 0x7ff29414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:26.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff2940acef0 0x7ff294146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:26.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff29405d480 con 0x7ff29405c3f0 2022-07-20T03:07:26.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2940b21e0 con 0x7ff2940acef0 2022-07-20T03:07:26.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29405cf40 con 0x7ff2940b1e00 2022-07-20T03:07:26.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff298c46700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff29405c3f0 0x7ff2940ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39432/0 (socket says 172.21.15.61:39432) 2022-07-20T03:07:27.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff298c46700 1 -- 172.21.15.61:0/3153630218 learned_addr learned my addr 172.21.15.61:0/3153630218 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:27.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299447700 1 --2- 172.21.15.61:0/3153630218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940b1e00 0x7ff29414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299c48700 1 --2- 172.21.15.61:0/3153630218 >> v2:172.21.15.117:3300/0 conn(0x7ff2940acef0 0x7ff294146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff28bfff700 1 -- 172.21.15.61:0/3153630218 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3983047321 0 0) 0x7ff29405d480 con 0x7ff29405c3f0 2022-07-20T03:07:27.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff28bfff700 1 -- 172.21.15.61:0/3153630218 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff278003260 con 0x7ff29405c3f0 2022-07-20T03:07:27.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299447700 1 -- 172.21.15.61:0/3153630218 >> v2:172.21.15.117:3300/0 conn(0x7ff2940acef0 msgr2=0x7ff294146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:27.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299447700 1 --2- 172.21.15.61:0/3153630218 >> v2:172.21.15.117:3300/0 conn(0x7ff2940acef0 0x7ff294146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299447700 1 -- 172.21.15.61:0/3153630218 >> v1:172.21.15.61:6789/0 conn(0x7ff29405c3f0 legacy=0x7ff2940ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299447700 1 -- 172.21.15.61:0/3153630218 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff29405d1e0 con 0x7ff2940b1e00 2022-07-20T03:07:27.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff299447700 1 --2- 172.21.15.61:0/3153630218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940b1e00 0x7ff29414a6d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff284002f70 tx=0x7ff284006df0).ready entity=mon.2 client_cookie=71206554c73985db server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff28bfff700 1 -- 172.21.15.61:0/3153630218 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff28400d4a0 con 0x7ff2940b1e00 2022-07-20T03:07:27.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff28bfff700 1 -- 172.21.15.61:0/3153630218 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff284000e90 con 0x7ff2940b1e00 2022-07-20T03:07:27.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff28bfff700 1 -- 172.21.15.61:0/3153630218 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff28400d7f0 con 0x7ff2940b1e00 2022-07-20T03:07:27.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3153630218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940b1e00 msgr2=0x7ff29414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3153630218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940b1e00 0x7ff29414a6d0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7ff284002f70 tx=0x7ff284006df0).stop 2022-07-20T03:07:27.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3153630218 shutdown_connections 2022-07-20T03:07:27.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3153630218 >> v2:172.21.15.117:3300/0 conn(0x7ff2940acef0 0x7ff294146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3153630218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940b1e00 0x7ff29414a6d0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3153630218 >> 172.21.15.61:0/3153630218 conn(0x7ff2940611c0 msgr2=0x7ff294062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:27.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3153630218 shutdown_connections 2022-07-20T03:07:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.887+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3153630218 wait complete. 2022-07-20T03:07:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 Processor -- start 2022-07-20T03:07:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 -- start start 2022-07-20T03:07:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2940acef0 0x7ff2941467c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 0x7ff29409ec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940a05a0 0x7ff2940a0a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2940a0f20 con 0x7ff2940b1e00 2022-07-20T03:07:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff29409e560 con 0x7ff2940acef0 2022-07-20T03:07:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff29b6ab700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff29409f150 con 0x7ff2940a05a0 2022-07-20T03:07:27.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff298c46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 0x7ff29409ec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff298c46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 0x7ff29409ec60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42842/0 (socket says 172.21.15.61:42842) 2022-07-20T03:07:27.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.891+0000 7ff298c46700 1 -- 172.21.15.61:0/3006111300 learned_addr learned my addr 172.21.15.61:0/3006111300 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:27.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff299447700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2940acef0 0x7ff2941467c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff299c48700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940a05a0 0x7ff2940a0a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff298c46700 1 -- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940a05a0 msgr2=0x7ff2940a0a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff298c46700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940a05a0 0x7ff2940a0a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff298c46700 1 -- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2940acef0 msgr2=0x7ff2941467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff298c46700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2940acef0 0x7ff2941467c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff298c46700 1 -- 172.21.15.61:0/3006111300 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff28400b180 con 0x7ff2940b1e00 2022-07-20T03:07:27.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff298c46700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 0x7ff29409ec60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff29409f2e0 tx=0x7ff290007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff29000cea0 con 0x7ff2940b1e00 2022-07-20T03:07:27.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2900070d0 con 0x7ff2940b1e00 2022-07-20T03:07:27.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff290007430 con 0x7ff2940b1e00 2022-07-20T03:07:27.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff29409e7c0 con 0x7ff2940b1e00 2022-07-20T03:07:27.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff29414a4e0 con 0x7ff2940b1e00 2022-07-20T03:07:27.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2941436e0 con 0x7ff2940b1e00 2022-07-20T03:07:27.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff29001a070 con 0x7ff2940b1e00 2022-07-20T03:07:27.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff28a7fc700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff27c053ad0 0x7ff27c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.895+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff29006b140 con 0x7ff2940b1e00 2022-07-20T03:07:27.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.899+0000 7ff299447700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff27c053ad0 0x7ff27c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.899+0000 7ff299447700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff27c053ad0 0x7ff27c055f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff28401a000 tx=0x7ff284014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:26.903+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff290041f90 con 0x7ff2940b1e00 2022-07-20T03:07:27.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.031+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff29409e950 con 0x7ff2940b1e00 2022-07-20T03:07:27.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff28a7fc700 1 -- 172.21.15.61:0/3006111300 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7ff2900455b0 con 0x7ff2940b1e00 2022-07-20T03:07:27.085 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:27.094 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:27.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff27c053ad0 msgr2=0x7ff27c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff27c053ad0 0x7ff27c055f90 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7ff28401a000 tx=0x7ff284014000).stop 2022-07-20T03:07:27.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 msgr2=0x7ff29409ec60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 0x7ff29409ec60 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7ff29409f2e0 tx=0x7ff290007a60).stop 2022-07-20T03:07:27.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 shutdown_connections 2022-07-20T03:07:27.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff27c053ad0 0x7ff27c055f90 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2940a05a0 0x7ff2940a0a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2940b1e00 0x7ff29409ec60 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 --2- 172.21.15.61:0/3006111300 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2940acef0 0x7ff2941467c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 >> 172.21.15.61:0/3006111300 conn(0x7ff29405c3f0 msgr2=0x7ff2940b46c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:27.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 shutdown_connections 2022-07-20T03:07:27.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.035+0000 7ff29b6ab700 1 -- 172.21.15.61:0/3006111300 wait complete. 2022-07-20T03:07:27.118 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:27.120 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 Processor -- start 2022-07-20T03:07:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 -- start start 2022-07-20T03:07:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c41473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29c40b1400 0x7f29c4147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29c405cc70 con 0x7f29c40af940 2022-07-20T03:07:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29c40b1070 con 0x7f29c40b1400 2022-07-20T03:07:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c40b5dd0 con 0x7f29c40b59f0 2022-07-20T03:07:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29c40b1400 0x7f29c4147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29c40b1400 0x7f29c4147890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42850/0 (socket says 172.21.15.61:42850) 2022-07-20T03:07:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 -- 172.21.15.61:0/20334410 learned_addr learned my addr 172.21.15.61:0/20334410 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c259c700 1 --2- 172.21.15.61:0/20334410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c41473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 -- 172.21.15.61:0/20334410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 msgr2=0x7f29c41473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 --2- 172.21.15.61:0/20334410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c41473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 -- 172.21.15.61:0/20334410 >> v1:172.21.15.117:6789/0 conn(0x7f29c40af940 legacy=0x7f29c414a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.259+0000 7f29c2d9d700 1 -- 172.21.15.61:0/20334410 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29c405c760 con 0x7f29c40b1400 2022-07-20T03:07:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c2d9d700 1 --2- 172.21.15.61:0/20334410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40b1400 0x7f29c4147890 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f29bc0032f0 tx=0x7f29bc009290).ready entity=mon.0 client_cookie=b84474ad4b814cdb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c159a700 1 -- 172.21.15.61:0/20334410 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc00b8a0 con 0x7f29c40b1400 2022-07-20T03:07:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c159a700 1 -- 172.21.15.61:0/20334410 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29bc00ba00 con 0x7f29c40b1400 2022-07-20T03:07:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c159a700 1 -- 172.21.15.61:0/20334410 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29bc00bb70 con 0x7f29c40b1400 2022-07-20T03:07:27.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 -- 172.21.15.61:0/20334410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40b1400 msgr2=0x7f29c4147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 --2- 172.21.15.61:0/20334410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40b1400 0x7f29c4147890 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f29bc0032f0 tx=0x7f29bc009290).stop 2022-07-20T03:07:27.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 -- 172.21.15.61:0/20334410 shutdown_connections 2022-07-20T03:07:27.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 --2- 172.21.15.61:0/20334410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40b1400 0x7f29c4147890 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 --2- 172.21.15.61:0/20334410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c41473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 -- 172.21.15.61:0/20334410 >> 172.21.15.61:0/20334410 conn(0x7f29c4063480 msgr2=0x7f29c40b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 -- 172.21.15.61:0/20334410 shutdown_connections 2022-07-20T03:07:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 -- 172.21.15.61:0/20334410 wait complete. 2022-07-20T03:07:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 Processor -- start 2022-07-20T03:07:27.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.263+0000 7f29c359e700 1 -- start start 2022-07-20T03:07:27.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 0x7f29c410f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c40b1400 0x7f29c413e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c413ead0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4064ea0 con 0x7f29c40af940 2022-07-20T03:07:27.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29c410f0d0 con 0x7f29c40b1400 2022-07-20T03:07:27.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29c4111f00 con 0x7f29c40b59f0 2022-07-20T03:07:27.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 0x7f29c410f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 0x7f29c410f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42854/0 (socket says 172.21.15.61:42854) 2022-07-20T03:07:27.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 -- 172.21.15.61:0/3573607330 learned_addr learned my addr 172.21.15.61:0/3573607330 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:27.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c1d9b700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c40b1400 0x7f29c413e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c2d9d700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c413ead0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 -- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 msgr2=0x7f29c413ead0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c413ead0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 -- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c40b1400 msgr2=0x7f29c413e5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c40b1400 0x7f29c413e5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 -- 172.21.15.61:0/3573607330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29bc009530 con 0x7f29c40af940 2022-07-20T03:07:27.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c259c700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 0x7f29c410f870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f29c410f260 tx=0x7f29ac008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29ac011780 con 0x7f29c40af940 2022-07-20T03:07:27.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29ac0118e0 con 0x7f29c40af940 2022-07-20T03:07:27.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29c41121a0 con 0x7f29c40af940 2022-07-20T03:07:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f29ac011c40 con 0x7f29c40af940 2022-07-20T03:07:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29c410e190 con 0x7f29c40af940 2022-07-20T03:07:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.267+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29c4156760 con 0x7f29c40af940 2022-07-20T03:07:27.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.271+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f29ac01c460 con 0x7f29c40af940 2022-07-20T03:07:27.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.271+0000 7f29bb7fe700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f29a80539b0 0x7f29a8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.271+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f29ac06bf10 con 0x7f29c40af940 2022-07-20T03:07:27.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.271+0000 7f29c1d9b700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f29a80539b0 0x7f29a8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.271+0000 7f29c1d9b700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f29a80539b0 0x7f29a8055e70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f29b400d000 tx=0x7f29b4007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.275+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29ac042e10 con 0x7f29c40af940 2022-07-20T03:07:27.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.407+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f29c405c9d0 con 0x7f29c40af940 2022-07-20T03:07:27.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.407+0000 7f29bb7fe700 1 -- 172.21.15.61:0/3573607330 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f29ac046430 con 0x7f29c40af940 2022-07-20T03:07:27.472 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:27.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f29a80539b0 msgr2=0x7f29a8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f29a80539b0 0x7f29a8055e70 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f29b400d000 tx=0x7f29b4007b50).stop 2022-07-20T03:07:27.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 msgr2=0x7f29c410f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 0x7f29c410f870 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f29c410f260 tx=0x7f29ac008040).stop 2022-07-20T03:07:27.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 shutdown_connections 2022-07-20T03:07:27.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f29a80539b0 0x7f29a8055e70 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29c40b1400 0x7f29c413e5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29c40b59f0 0x7f29c413ead0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 --2- 172.21.15.61:0/3573607330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29c40af940 0x7f29c410f870 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 >> 172.21.15.61:0/3573607330 conn(0x7f29c4063480 msgr2=0x7f29c40b42e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:27.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 shutdown_connections 2022-07-20T03:07:27.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.411+0000 7f29c359e700 1 -- 172.21.15.61:0/3573607330 wait complete. 2022-07-20T03:07:27.562 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:27.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.679+0000 7f428bab3700 1 Processor -- start 2022-07-20T03:07:27.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.679+0000 7f428bab3700 1 -- start start 2022-07-20T03:07:27.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f42840988b0 0x7f42841422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f4284060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f42840afdb0 con 0x7f42841462a0 2022-07-20T03:07:27.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f42840b0050 con 0x7f42840988b0 2022-07-20T03:07:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428405fcd0 con 0x7f42841427b0 2022-07-20T03:07:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428904e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f42840988b0 0x7f42841422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f4284060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428904e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f42840988b0 0x7f42841422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:52544/0 (socket says 172.21.15.61:52544) 2022-07-20T03:07:27.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f4284060170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:57768/0 (socket says 172.21.15.61:57768) 2022-07-20T03:07:27.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428904e700 1 -- 172.21.15.61:0/3064114238 learned_addr learned my addr 172.21.15.61:0/3064114238 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:27.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 -- 172.21.15.61:0/3064114238 >> v2:172.21.15.117:3300/0 conn(0x7f42840988b0 msgr2=0x7f42841422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:27.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 --2- 172.21.15.61:0/3064114238 >> v2:172.21.15.117:3300/0 conn(0x7f42840988b0 0x7f42841422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 -- 172.21.15.61:0/3064114238 >> v1:172.21.15.117:6789/0 conn(0x7f42841462a0 legacy=0x7f4284146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 -- 172.21.15.61:0/3064114238 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f42840afae0 con 0x7f42841427b0 2022-07-20T03:07:27.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428a050700 1 --2- 172.21.15.61:0/3064114238 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f4284060170 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f427c0032f0 tx=0x7f427c0091d0).ready entity=mon.2 client_cookie=7f6b00f5166db269 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428884d700 1 -- 172.21.15.61:0/3064114238 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f427c00b6c0 con 0x7f42841427b0 2022-07-20T03:07:27.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428884d700 1 -- 172.21.15.61:0/3064114238 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f427c00b820 con 0x7f42841427b0 2022-07-20T03:07:27.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428884d700 1 -- 172.21.15.61:0/3064114238 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f427c00b990 con 0x7f42841427b0 2022-07-20T03:07:27.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- 172.21.15.61:0/3064114238 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 msgr2=0x7f4284060170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 --2- 172.21.15.61:0/3064114238 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f4284060170 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f427c0032f0 tx=0x7f427c0091d0).stop 2022-07-20T03:07:27.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- 172.21.15.61:0/3064114238 shutdown_connections 2022-07-20T03:07:27.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 --2- 172.21.15.61:0/3064114238 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f4284060170 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 --2- 172.21.15.61:0/3064114238 >> v2:172.21.15.117:3300/0 conn(0x7f42840988b0 0x7f42841422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- 172.21.15.61:0/3064114238 >> 172.21.15.61:0/3064114238 conn(0x7f42840b5f00 msgr2=0x7f42840b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:27.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- 172.21.15.61:0/3064114238 shutdown_connections 2022-07-20T03:07:27.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.683+0000 7f428bab3700 1 -- 172.21.15.61:0/3064114238 wait complete. 2022-07-20T03:07:27.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 Processor -- start 2022-07-20T03:07:27.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 -- start start 2022-07-20T03:07:27.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42840988b0 0x7f428413f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f42841555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 0x7f42840b2ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4284149480 con 0x7f42841462a0 2022-07-20T03:07:27.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4284149220 con 0x7f42840988b0 2022-07-20T03:07:27.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428bab3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f42840b2a40 con 0x7f42841427b0 2022-07-20T03:07:27.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428a050700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 0x7f42840b2ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428a050700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 0x7f42840b2ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42870/0 (socket says 172.21.15.61:42870) 2022-07-20T03:07:27.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428904e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f42841555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428a050700 1 -- 172.21.15.61:0/3200449653 learned_addr learned my addr 172.21.15.61:0/3200449653 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:27.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428984f700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42840988b0 0x7f428413f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428904e700 1 -- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42840988b0 msgr2=0x7f428413f690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428904e700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42840988b0 0x7f428413f690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428904e700 1 -- 172.21.15.61:0/3200449653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 msgr2=0x7f42840b2ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428904e700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 0x7f42840b2ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428904e700 1 -- 172.21.15.61:0/3200449653 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f427c009470 con 0x7f42841427b0 2022-07-20T03:07:27.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.687+0000 7f428a050700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 0x7f42840b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:27.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f428904e700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f42841555d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f428000c000 tx=0x7f4280008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4280013400 con 0x7f42841427b0 2022-07-20T03:07:27.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4284148710 con 0x7f42841427b0 2022-07-20T03:07:27.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4280013560 con 0x7f42841427b0 2022-07-20T03:07:27.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f42800138c0 con 0x7f42841427b0 2022-07-20T03:07:27.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4284155c60 con 0x7f42841427b0 2022-07-20T03:07:27.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4284146d90 con 0x7f42841427b0 2022-07-20T03:07:27.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4280013a20 con 0x7f42841427b0 2022-07-20T03:07:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f4276ffd700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4264053920 0x7f4264055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f428006bc60 con 0x7f42841427b0 2022-07-20T03:07:27.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.691+0000 7f428984f700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4264053920 0x7f4264055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:27.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.695+0000 7f428984f700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4264053920 0x7f4264055de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f427800d000 tx=0x7f4278009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:27.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.703+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4280042b20 con 0x7f42841427b0 2022-07-20T03:07:27.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.827+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f42840b2c10 con 0x7f42841427b0 2022-07-20T03:07:27.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f4276ffd700 1 -- 172.21.15.61:0/3200449653 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4280013cb0 con 0x7f42841427b0 2022-07-20T03:07:27.832 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:27.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4264053920 msgr2=0x7f4264055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4264053920 0x7f4264055de0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f427800d000 tx=0x7f4278009040).stop 2022-07-20T03:07:27.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 msgr2=0x7f42841555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:27.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f42841555d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f428000c000 tx=0x7f4280008040).stop 2022-07-20T03:07:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 shutdown_connections 2022-07-20T03:07:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4264053920 0x7f4264055de0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f42841462a0 0x7f42840b2ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f42841427b0 0x7f42841555d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 --2- 172.21.15.61:0/3200449653 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42840988b0 0x7f428413f690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:27.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.831+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 >> 172.21.15.61:0/3200449653 conn(0x7f42840b5f00 msgr2=0x7f428405c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:27.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.835+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 shutdown_connections 2022-07-20T03:07:27.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.835+0000 7f428bab3700 1 -- 172.21.15.61:0/3200449653 wait complete. 2022-07-20T03:07:27.853 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:28.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 Processor -- start 2022-07-20T03:07:28.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 -- start start 2022-07-20T03:07:28.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f82740b9da0 0x7f82741468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f82740b3cf0 0x7f8274142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8274142f40 0x7f827405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82740ad5b0 con 0x7f82740b3cf0 2022-07-20T03:07:28.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f82740ba180 con 0x7f82740b9da0 2022-07-20T03:07:28.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f827359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f82741461b0 con 0x7f8274142f40 2022-07-20T03:07:28.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.963+0000 7f8271d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f82740b3cf0 0x7f8274142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f82740b3cf0 0x7f8274142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42880/0 (socket says 172.21.15.61:42880) 2022-07-20T03:07:28.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 -- 172.21.15.61:0/723164154 learned_addr learned my addr 172.21.15.61:0/723164154 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:28.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827259c700 1 --2- 172.21.15.61:0/723164154 >> v2:172.21.15.117:3300/0 conn(0x7f82740b9da0 0x7f82741468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8272d9d700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8274142f40 0x7f827405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 -- 172.21.15.61:0/723164154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8274142f40 msgr2=0x7f827405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:28.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8274142f40 0x7f827405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 -- 172.21.15.61:0/723164154 >> v2:172.21.15.117:3300/0 conn(0x7f82740b9da0 msgr2=0x7f82741468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:28.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 --2- 172.21.15.61:0/723164154 >> v2:172.21.15.117:3300/0 conn(0x7f82740b9da0 0x7f82741468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 -- 172.21.15.61:0/723164154 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8274146450 con 0x7f82740b3cf0 2022-07-20T03:07:28.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8272d9d700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8274142f40 0x7f827405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:28.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827259c700 1 --2- 172.21.15.61:0/723164154 >> v2:172.21.15.117:3300/0 conn(0x7f82740b9da0 0x7f82741468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:28.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f8271d9b700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82740b3cf0 0x7f8274142a50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f825c002f70 tx=0x7f825c006eb0).ready entity=mon.0 client_cookie=ea25a3fd71cfe217 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:28.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827159a700 1 -- 172.21.15.61:0/723164154 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f825c00d570 con 0x7f82740b3cf0 2022-07-20T03:07:28.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827159a700 1 -- 172.21.15.61:0/723164154 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f825c000e90 con 0x7f82740b3cf0 2022-07-20T03:07:28.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827159a700 1 -- 172.21.15.61:0/723164154 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f825c00d900 con 0x7f82740b3cf0 2022-07-20T03:07:28.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 -- 172.21.15.61:0/723164154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82740b3cf0 msgr2=0x7f8274142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82740b3cf0 0x7f8274142a50 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f825c002f70 tx=0x7f825c006eb0).stop 2022-07-20T03:07:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 -- 172.21.15.61:0/723164154 shutdown_connections 2022-07-20T03:07:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8274142f40 0x7f827405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 --2- 172.21.15.61:0/723164154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f82740b3cf0 0x7f8274142a50 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 --2- 172.21.15.61:0/723164154 >> v2:172.21.15.117:3300/0 conn(0x7f82740b9da0 0x7f82741468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 -- 172.21.15.61:0/723164154 >> 172.21.15.61:0/723164154 conn(0x7f82740678c0 msgr2=0x7f82740b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:28.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 -- 172.21.15.61:0/723164154 shutdown_connections 2022-07-20T03:07:28.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.967+0000 7f827359e700 1 -- 172.21.15.61:0/723164154 wait complete. 2022-07-20T03:07:28.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 Processor -- start 2022-07-20T03:07:28.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 -- start start 2022-07-20T03:07:28.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82740b3cf0 0x7f82740b8230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82740b9da0 0x7f8274145280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 0x7f8274155040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8274145770 con 0x7f8274142f40 2022-07-20T03:07:28.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f827413deb0 con 0x7f82740b9da0 2022-07-20T03:07:28.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8274144de0 con 0x7f82740b3cf0 2022-07-20T03:07:28.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 0x7f8274155040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 0x7f8274155040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42886/0 (socket says 172.21.15.61:42886) 2022-07-20T03:07:28.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 -- 172.21.15.61:0/2301797500 learned_addr learned my addr 172.21.15.61:0/2301797500 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:28.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8271d9b700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82740b9da0 0x7f8274145280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827259c700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82740b3cf0 0x7f82740b8230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 -- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82740b3cf0 msgr2=0x7f82740b8230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82740b3cf0 0x7f82740b8230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 -- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82740b9da0 msgr2=0x7f8274145280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82740b9da0 0x7f8274145280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 -- 172.21.15.61:0/2301797500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f825c00b1e0 con 0x7f8274142f40 2022-07-20T03:07:28.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f8272d9d700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 0x7f8274155040 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f826c006cc0 tx=0x7f826c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:28.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f826c011780 con 0x7f8274142f40 2022-07-20T03:07:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f826c0118e0 con 0x7f8274142f40 2022-07-20T03:07:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f827413db30 con 0x7f8274142f40 2022-07-20T03:07:28.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f826c011c40 con 0x7f8274142f40 2022-07-20T03:07:28.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.971+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f82741441b0 con 0x7f8274142f40 2022-07-20T03:07:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.975+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f827413dd20 con 0x7f8274142f40 2022-07-20T03:07:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.975+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f826c012420 con 0x7f8274142f40 2022-07-20T03:07:28.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.975+0000 7f826b7fe700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8258053920 0x7f8258055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.975+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f826c06be70 con 0x7f8274142f40 2022-07-20T03:07:28.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.975+0000 7f827259c700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8258053920 0x7f8258055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.975+0000 7f827259c700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8258053920 0x7f8258055de0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f826400c000 tx=0x7f8264008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:28.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:27.979+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f826c042d70 con 0x7f8274142f40 2022-07-20T03:07:28.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.107+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8274145d20 con 0x7f8274142f40 2022-07-20T03:07:28.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.107+0000 7f826b7fe700 1 -- 172.21.15.61:0/2301797500 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8274145d20 con 0x7f8274142f40 2022-07-20T03:07:28.473 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8258053920 msgr2=0x7f8258055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8258053920 0x7f8258055de0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f826400c000 tx=0x7f8264008040).stop 2022-07-20T03:07:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 msgr2=0x7f8274155040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 0x7f8274155040 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f826c006cc0 tx=0x7f826c008040).stop 2022-07-20T03:07:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 shutdown_connections 2022-07-20T03:07:28.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8258053920 0x7f8258055de0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f82740b9da0 0x7f8274145280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8274142f40 0x7f8274155040 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 --2- 172.21.15.61:0/2301797500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f82740b3cf0 0x7f82740b8230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 >> 172.21.15.61:0/2301797500 conn(0x7f82740678c0 msgr2=0x7f82740b4c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:28.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 shutdown_connections 2022-07-20T03:07:28.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.111+0000 7f827359e700 1 -- 172.21.15.61:0/2301797500 wait complete. 2022-07-20T03:07:28.509 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:28.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.643+0000 7f6f2975a700 1 Processor -- start 2022-07-20T03:07:28.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.643+0000 7f6f2975a700 1 -- start start 2022-07-20T03:07:28.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f2405cf40 0x7f6f240afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f24147c90 0x7f6f240b0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6f240b1220 0x7f6f24099080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f241472b0 con 0x7f6f24147c90 2022-07-20T03:07:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f24147550 con 0x7f6f240b1220 2022-07-20T03:07:28.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241477f0 con 0x7f6f2405cf40 2022-07-20T03:07:28.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f24147c90 0x7f6f240b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f24147c90 0x7f6f240b0210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42892/0 (socket says 172.21.15.61:42892) 2022-07-20T03:07:28.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 -- 172.21.15.61:0/1117806928 learned_addr learned my addr 172.21.15.61:0/1117806928 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:28.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f237fe700 1 --2- 172.21.15.61:0/1117806928 >> v2:172.21.15.117:3300/0 conn(0x7f6f240b1220 0x7f6f24099080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f22ffd700 1 --2- 172.21.15.61:0/1117806928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f2405cf40 0x7f6f240afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 -- 172.21.15.61:0/1117806928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f2405cf40 msgr2=0x7f6f240afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:28.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 --2- 172.21.15.61:0/1117806928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f2405cf40 0x7f6f240afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 -- 172.21.15.61:0/1117806928 >> v2:172.21.15.117:3300/0 conn(0x7f6f240b1220 msgr2=0x7f6f24099080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:28.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 --2- 172.21.15.61:0/1117806928 >> v2:172.21.15.117:3300/0 conn(0x7f6f240b1220 0x7f6f24099080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 -- 172.21.15.61:0/1117806928 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f24146fa0 con 0x7f6f24147c90 2022-07-20T03:07:28.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f237fe700 1 --2- 172.21.15.61:0/1117806928 >> v2:172.21.15.117:3300/0 conn(0x7f6f240b1220 0x7f6f24099080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:28.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f227fc700 1 --2- 172.21.15.61:0/1117806928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f24147c90 0x7f6f240b0210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6f14002f70 tx=0x7f6f14006eb0).ready entity=mon.0 client_cookie=762062f738030375 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:28.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f21ffb700 1 -- 172.21.15.61:0/1117806928 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f1400d570 con 0x7f6f24147c90 2022-07-20T03:07:28.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f21ffb700 1 -- 172.21.15.61:0/1117806928 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14000e90 con 0x7f6f24147c90 2022-07-20T03:07:28.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f21ffb700 1 -- 172.21.15.61:0/1117806928 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f1400d900 con 0x7f6f24147c90 2022-07-20T03:07:28.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- 172.21.15.61:0/1117806928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f24147c90 msgr2=0x7f6f240b0210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- 172.21.15.61:0/1117806928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f24147c90 0x7f6f240b0210 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6f14002f70 tx=0x7f6f14006eb0).stop 2022-07-20T03:07:28.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- 172.21.15.61:0/1117806928 shutdown_connections 2022-07-20T03:07:28.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- 172.21.15.61:0/1117806928 >> v2:172.21.15.117:3300/0 conn(0x7f6f240b1220 0x7f6f24099080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- 172.21.15.61:0/1117806928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f24147c90 0x7f6f240b0210 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 --2- 172.21.15.61:0/1117806928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f2405cf40 0x7f6f240afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- 172.21.15.61:0/1117806928 >> 172.21.15.61:0/1117806928 conn(0x7f6f240b5b70 msgr2=0x7f6f240b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- 172.21.15.61:0/1117806928 shutdown_connections 2022-07-20T03:07:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.647+0000 7f6f2975a700 1 -- 172.21.15.61:0/1117806928 wait complete. 2022-07-20T03:07:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 Processor -- start 2022-07-20T03:07:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 -- start start 2022-07-20T03:07:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f2405cf40 0x7f6f24143f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 0x7f6f24159800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f24147c90 0x7f6f24159cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f240b4640 con 0x7f6f2405cf40 2022-07-20T03:07:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f241402b0 con 0x7f6f240b1220 2022-07-20T03:07:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f2975a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f240b42c0 con 0x7f6f24147c90 2022-07-20T03:07:28.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f22ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f2405cf40 0x7f6f24143f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 0x7f6f24159800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f237fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f24147c90 0x7f6f24159cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f22ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f2405cf40 0x7f6f24143f90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42898/0 (socket says 172.21.15.61:42898) 2022-07-20T03:07:28.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 0x7f6f24159800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:52576/0 (socket says 172.21.15.61:52576) 2022-07-20T03:07:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f22ffd700 1 -- 172.21.15.61:0/3541631067 learned_addr learned my addr 172.21.15.61:0/3541631067 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 -- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f24147c90 msgr2=0x7f6f24159cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f24147c90 0x7f6f24159cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 -- 172.21.15.61:0/3541631067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f2405cf40 msgr2=0x7f6f24143f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f2405cf40 0x7f6f24143f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.651+0000 7f6f227fc700 1 -- 172.21.15.61:0/3541631067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f1c009470 con 0x7f6f240b1220 2022-07-20T03:07:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f227fc700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 0x7f6f24159800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f6f14002f70 tx=0x7f6f14015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f140117e0 con 0x7f6f240b1220 2022-07-20T03:07:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f1400b1e0 con 0x7f6f240b1220 2022-07-20T03:07:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f14011940 con 0x7f6f240b1220 2022-07-20T03:07:28.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f14011ca0 con 0x7f6f240b1220 2022-07-20T03:07:28.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f241406b0 con 0x7f6f240b1220 2022-07-20T03:07:28.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f2405d320 con 0x7f6f240b1220 2022-07-20T03:07:28.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6f1401a040 con 0x7f6f240b1220 2022-07-20T03:07:28.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f13fff700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f00053a60 0x7f6f00055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:28.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6f1406cbd0 con 0x7f6f240b1220 2022-07-20T03:07:28.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.655+0000 7f6f22ffd700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f00053a60 0x7f6f00055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:28.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.659+0000 7f6f22ffd700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f00053a60 0x7f6f00055f20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6f1800b000 tx=0x7f6f18006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:28.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.667+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f14043a90 con 0x7f6f240b1220 2022-07-20T03:07:28.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.795+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f6f240574e0 con 0x7f6f240b1220 2022-07-20T03:07:28.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.795+0000 7f6f13fff700 1 -- 172.21.15.61:0/3541631067 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6f140470b0 con 0x7f6f240b1220 2022-07-20T03:07:28.833 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:28.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f00053a60 msgr2=0x7f6f00055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f00053a60 0x7f6f00055f20 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f6f1800b000 tx=0x7f6f18006df0).stop 2022-07-20T03:07:28.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 msgr2=0x7f6f24159800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:28.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 0x7f6f24159800 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f6f14002f70 tx=0x7f6f14015000).stop 2022-07-20T03:07:28.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 shutdown_connections 2022-07-20T03:07:28.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f00053a60 0x7f6f00055f20 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f240b1220 0x7f6f24159800 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f24147c90 0x7f6f24159cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 --2- 172.21.15.61:0/3541631067 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f2405cf40 0x7f6f24143f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:28.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 >> 172.21.15.61:0/3541631067 conn(0x7f6f240b5b70 msgr2=0x7f6f2414d250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:28.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 shutdown_connections 2022-07-20T03:07:28.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:28.799+0000 7f6f2975a700 1 -- 172.21.15.61:0/3541631067 wait complete. 2022-07-20T03:07:28.888 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:07:28.889 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:29.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 Processor -- start 2022-07-20T03:07:29.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- start start 2022-07-20T03:07:29.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3bb80aea20 0x7f3bb8140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:29.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb80aee00 con 0x7f3bb80b1a70 2022-07-20T03:07:29.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3bb8144470 con 0x7f3bb80675f0 2022-07-20T03:07:29.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3bb8144710 con 0x7f3bb80aea20 2022-07-20T03:07:29.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3bb80b1a70 0x7f3bb80b1f20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39506/0 (socket says 172.21.15.61:39506) 2022-07-20T03:07:29.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb77fe700 1 -- 172.21.15.61:0/2989159573 learned_addr learned my addr 172.21.15.61:0/2989159573 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:29.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb6ffd700 1 --2- 172.21.15.61:0/2989159573 >> v2:172.21.15.117:3300/0 conn(0x7f3bb80aea20 0x7f3bb8140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:29.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb6ffd700 1 -- 172.21.15.61:0/2989159573 >> v1:172.21.15.117:6789/0 conn(0x7f3bb80675f0 legacy=0x7f3bb8140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb6ffd700 1 -- 172.21.15.61:0/2989159573 >> v1:172.21.15.61:6789/0 conn(0x7f3bb80b1a70 legacy=0x7f3bb80b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb67fc700 1 -- 172.21.15.61:0/2989159573 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1756114463 0 0) 0x7f3bb80aee00 con 0x7f3bb80b1a70 2022-07-20T03:07:29.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb6ffd700 1 -- 172.21.15.61:0/2989159573 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bb81449b0 con 0x7f3bb80aea20 2022-07-20T03:07:29.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb6ffd700 1 --2- 172.21.15.61:0/2989159573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb8140a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3bb80b1750 tx=0x7f3ba8006df0).ready entity=mon.1 client_cookie=96bb093f47ce9103 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:29.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb67fc700 1 -- 172.21.15.61:0/2989159573 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ba800d610 con 0x7f3bb80aea20 2022-07-20T03:07:29.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb67fc700 1 -- 172.21.15.61:0/2989159573 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ba8000e90 con 0x7f3bb80aea20 2022-07-20T03:07:29.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bb67fc700 1 -- 172.21.15.61:0/2989159573 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ba800d960 con 0x7f3bb80aea20 2022-07-20T03:07:29.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- 172.21.15.61:0/2989159573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 msgr2=0x7f3bb8140a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2989159573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb8140a90 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f3bb80b1750 tx=0x7f3ba8006df0).stop 2022-07-20T03:07:29.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- 172.21.15.61:0/2989159573 shutdown_connections 2022-07-20T03:07:29.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2989159573 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb8140a90 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.011+0000 7f3bbd397700 1 -- 172.21.15.61:0/2989159573 >> 172.21.15.61:0/2989159573 conn(0x7f3bb80611c0 msgr2=0x7f3bb8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:29.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 -- 172.21.15.61:0/2989159573 shutdown_connections 2022-07-20T03:07:29.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 -- 172.21.15.61:0/2989159573 wait complete. 2022-07-20T03:07:29.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 Processor -- start 2022-07-20T03:07:29.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 -- start start 2022-07-20T03:07:29.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb80642a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:29.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bb80b1a70 0x7f3bb8140b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:29.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 0x7f3bb8155990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:29.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb8152f50 con 0x7f3bb8141020 2022-07-20T03:07:29.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb81531f0 con 0x7f3bb80aea20 2022-07-20T03:07:29.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bbd397700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3bb8141580 con 0x7f3bb80b1a70 2022-07-20T03:07:29.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bb77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 0x7f3bb8155990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:29.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bb77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 0x7f3bb8155990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42912/0 (socket says 172.21.15.61:42912) 2022-07-20T03:07:29.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.015+0000 7f3bb77fe700 1 -- 172.21.15.61:0/2776798708 learned_addr learned my addr 172.21.15.61:0/2776798708 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:29.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb6ffd700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb80642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:29.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bae7fc700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bb80b1a70 0x7f3bb8140b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:29.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb77fe700 1 -- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bb80b1a70 msgr2=0x7f3bb8140b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb77fe700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bb80b1a70 0x7f3bb8140b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb77fe700 1 -- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 msgr2=0x7f3bb80642a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb77fe700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb80642a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb77fe700 1 -- 172.21.15.61:0/2776798708 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ba800b180 con 0x7f3bb8141020 2022-07-20T03:07:29.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb6ffd700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb80642a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:29.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb77fe700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 0x7f3bb8155990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3bb0000f40 tx=0x7f3bb0001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:29.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bb0005ba0 con 0x7f3bb8141020 2022-07-20T03:07:29.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3bb0005d00 con 0x7f3bb8141020 2022-07-20T03:07:29.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bb813fa00 con 0x7f3bb8141020 2022-07-20T03:07:29.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3bb00072c0 con 0x7f3bb8141020 2022-07-20T03:07:29.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bb813ffa0 con 0x7f3bb8141020 2022-07-20T03:07:29.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bb81533b0 con 0x7f3bb8141020 2022-07-20T03:07:29.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.019+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3bb001a060 con 0x7f3bb8141020 2022-07-20T03:07:29.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.023+0000 7f3bb4ff9700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b9c053ad0 0x7f3b9c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:29.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.023+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3bb006b440 con 0x7f3bb8141020 2022-07-20T03:07:29.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.023+0000 7f3bb6ffd700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b9c053ad0 0x7f3b9c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:29.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.023+0000 7f3bb6ffd700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b9c053ad0 0x7f3b9c055f90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3ba801c000 tx=0x7f3ba8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:29.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.023+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3bb00423a0 con 0x7f3bb8141020 2022-07-20T03:07:29.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.187+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3bb80574e0 con 0x7f3bb8141020 2022-07-20T03:07:29.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bb4ff9700 1 -- 172.21.15.61:0/2776798708 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f3bb0018020 con 0x7f3bb8141020 2022-07-20T03:07:29.195 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:29.195 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:12.741818+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.117:6835/725875269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":725875269},{"type":"v1","addr":"172.21.15.117:6835","nonce":725875269}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:29.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b9c053ad0 msgr2=0x7f3b9c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b9c053ad0 0x7f3b9c055f90 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f3ba801c000 tx=0x7f3ba8015000).stop 2022-07-20T03:07:29.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 msgr2=0x7f3bb8155990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:29.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 0x7f3bb8155990 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f3bb0000f40 tx=0x7f3bb0001360).stop 2022-07-20T03:07:29.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 shutdown_connections 2022-07-20T03:07:29.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b9c053ad0 0x7f3b9c055f90 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3bb8141020 0x7f3bb8155990 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3bb80b1a70 0x7f3bb8140b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 --2- 172.21.15.61:0/2776798708 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3bb80aea20 0x7f3bb80642a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:29.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 >> 172.21.15.61:0/2776798708 conn(0x7f3bb80675f0 msgr2=0x7f3bb80b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:29.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 shutdown_connections 2022-07-20T03:07:29.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:29.191+0000 7f3bbd397700 1 -- 172.21.15.61:0/2776798708 wait complete. 2022-07-20T03:07:29.213 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2022-07-20T03:07:29.440 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:07:10.718708+0000', 'modified': '2022-07-20T03:07:12.741818+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.117:6835/725875269', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 725875269}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 725875269}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:07:29.442 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:07:29.443 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:07:29.634 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":4273,"want_state":"up:active","state":"up:active","rank_uptime":18.864966318,"mdsmap_epoch":7,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":22.407196916} 2022-07-20T03:07:29.635 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 4273, "mdsmap_epoch": 7, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 18.864966318, "state": "up:active", "uptime": 22.407196916, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:07:29.635 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:07:29.636 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:07:29.636 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:07:29.637 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:29.639 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:29.655 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:07:29.672 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:07:29.673 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:07:29.805 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:07:29.808 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:07:29.835 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:29.836 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:07:29.877 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:07:29.911 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:29.939 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:07:29.972 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:07:29.972 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:07:30.022 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:07:30.008+0000 7f1b88292080 -1 init, newargv = 0x5605edf02f30 newargc=9ceph-fuse[22297]: starting ceph client 2022-07-20T03:07:30.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr: 2022-07-20T03:07:30.023 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22297]: starting fuse 2022-07-20T03:07:30.035 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:07:30.035 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:07:30.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:30.036 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:07:30.086 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:07:30.086 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:07:30.087 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:30.087 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:07:30.088 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:07:30.088 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:07:30.088 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:07:30.089 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:07:30.089 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:07:30.090 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:07:30.090 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:07:30.091 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:07:30.091 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:07:30.092 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:07:30.092 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:07:30.092 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:07:30.093 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:07:30.093 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:07:30.093 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:07:30.094 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:07:30.094 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:07:30.094 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:07:30.095 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:07:30.095 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:07:30.098 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:07:30.099 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:07:30.183 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22297.asok 2022-07-20T03:07:30.184 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22297.asok 2022-07-20T03:07:30.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:30.202 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22297.asok status 2022-07-20T03:07:30.307 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:30.337 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:30.343 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:30.443 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:07:30.444 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:07:30.448 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:07:30.449 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:07:30.449 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:07:30.449 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:07:30.449 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:07:30.459 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22297", 2022-07-20T03:07:30.460 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:07:30.460 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:07:30.460 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:07:30.460 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:07:30.461 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 4448, 2022-07-20T03:07:30.461 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:07:30.461 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:07:30.461 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:07:30.461 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 4448 2022-07-20T03:07:30.462 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:07:30.462 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:07:30.462 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:07:30.462 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:07:30.463 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1160859655 2022-07-20T03:07:30.463 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:07:30.463 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:07:30.463 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:07:30.464 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:07:30.464 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:07:30.464 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1160859655 2022-07-20T03:07:30.464 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:07:30.464 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.4448 172.21.15.124:0/1160859655", 2022-07-20T03:07:30.465 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/1160859655", 2022-07-20T03:07:30.487 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:07:30.487 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 7, 2022-07-20T03:07:30.487 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 23, 2022-07-20T03:07:30.487 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:07:30.487 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:07:30.488 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:07:30.488 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:07:30.489 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:30.491 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:30.503 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:07:30.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 Processor -- start 2022-07-20T03:07:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 -- start start 2022-07-20T03:07:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f249c0b1ef0 0x7f249c142f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:30.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f249c05fa50 0x7f249c13e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:30.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f249c060980 con 0x7f249c0ac3a0 2022-07-20T03:07:30.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249c0b22d0 con 0x7f249c0b1ef0 2022-07-20T03:07:30.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a1da6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f249c098810 con 0x7f249c05fa50 2022-07-20T03:07:30.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a0da4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f249c0b1ef0 0x7f249c142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:30.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a0da4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f249c0b1ef0 0x7f249c142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42918/0 (socket says 172.21.15.61:42918) 2022-07-20T03:07:30.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a0da4700 1 -- 172.21.15.61:0/1912514546 learned_addr learned my addr 172.21.15.61:0/1912514546 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:30.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.599+0000 7f24a15a5700 1 --2- 172.21.15.61:0/1912514546 >> v2:172.21.15.117:3300/0 conn(0x7f249c05fa50 0x7f249c13e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:30.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a0da4700 1 -- 172.21.15.61:0/1912514546 >> v2:172.21.15.117:3300/0 conn(0x7f249c05fa50 msgr2=0x7f249c13e9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:30.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a0da4700 1 --2- 172.21.15.61:0/1912514546 >> v2:172.21.15.117:3300/0 conn(0x7f249c05fa50 0x7f249c13e9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a0da4700 1 -- 172.21.15.61:0/1912514546 >> v1:172.21.15.117:6789/0 conn(0x7f249c0ac3a0 legacy=0x7f249c143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:30.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a0da4700 1 -- 172.21.15.61:0/1912514546 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f249c060bb0 con 0x7f249c0b1ef0 2022-07-20T03:07:30.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a0da4700 1 --2- 172.21.15.61:0/1912514546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c0b1ef0 0x7f249c142f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f248c002f70 tx=0x7f248c006eb0).ready entity=mon.0 client_cookie=7d9f7053e6933d62 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:30.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f249b7fe700 1 -- 172.21.15.61:0/1912514546 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f248c00d610 con 0x7f249c0b1ef0 2022-07-20T03:07:30.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f249b7fe700 1 -- 172.21.15.61:0/1912514546 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c000e90 con 0x7f249c0b1ef0 2022-07-20T03:07:30.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f249b7fe700 1 -- 172.21.15.61:0/1912514546 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f248c00d9a0 con 0x7f249c0b1ef0 2022-07-20T03:07:30.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 -- 172.21.15.61:0/1912514546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c0b1ef0 msgr2=0x7f249c142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:30.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 --2- 172.21.15.61:0/1912514546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c0b1ef0 0x7f249c142f80 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f248c002f70 tx=0x7f248c006eb0).stop 2022-07-20T03:07:30.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 -- 172.21.15.61:0/1912514546 shutdown_connections 2022-07-20T03:07:30.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 --2- 172.21.15.61:0/1912514546 >> v2:172.21.15.117:3300/0 conn(0x7f249c05fa50 0x7f249c13e9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 --2- 172.21.15.61:0/1912514546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c0b1ef0 0x7f249c142f80 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 -- 172.21.15.61:0/1912514546 >> 172.21.15.61:0/1912514546 conn(0x7f249c0b2ca0 msgr2=0x7f249c0b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:30.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 -- 172.21.15.61:0/1912514546 shutdown_connections 2022-07-20T03:07:30.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 -- 172.21.15.61:0/1912514546 wait complete. 2022-07-20T03:07:30.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 Processor -- start 2022-07-20T03:07:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.603+0000 7f24a1da6700 1 -- start start 2022-07-20T03:07:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 0x7f249c09aab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249c0ac3a0 0x7f249c09afa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:30.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249c0b1ef0 0x7f249c09b490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:30.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c09bd50 con 0x7f249c05fa50 2022-07-20T03:07:30.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0ae120 con 0x7f249c0b1ef0 2022-07-20T03:07:30.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f249c0ae6a0 con 0x7f249c0ac3a0 2022-07-20T03:07:30.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 0x7f249c09aab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:30.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a15a5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249c0b1ef0 0x7f249c09b490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:30.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 0x7f249c09aab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42924/0 (socket says 172.21.15.61:42924) 2022-07-20T03:07:30.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 -- 172.21.15.61:0/3148644239 learned_addr learned my addr 172.21.15.61:0/3148644239 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:30.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f249bfff700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249c0ac3a0 0x7f249c09afa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:30.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 -- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249c0ac3a0 msgr2=0x7f249c09afa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:30.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249c0ac3a0 0x7f249c09afa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 -- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249c0b1ef0 msgr2=0x7f249c09b490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:30.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249c0b1ef0 0x7f249c09b490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 -- 172.21.15.61:0/3148644239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f248c00b1e0 con 0x7f249c05fa50 2022-07-20T03:07:30.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a0da4700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 0x7f249c09aab0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f248c002f70 tx=0x7f248c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:30.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f248c013bd0 con 0x7f249c05fa50 2022-07-20T03:07:30.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f248c00d290 con 0x7f249c05fa50 2022-07-20T03:07:30.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f249c141890 con 0x7f249c05fa50 2022-07-20T03:07:30.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f248c01e400 con 0x7f249c05fa50 2022-07-20T03:07:30.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.607+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f249c0993d0 con 0x7f249c05fa50 2022-07-20T03:07:30.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.611+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f249c09be90 con 0x7f249c05fa50 2022-07-20T03:07:30.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.611+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f248c025030 con 0x7f249c05fa50 2022-07-20T03:07:30.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.611+0000 7f2499ffb700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24740539d0 0x7f2474055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:30.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.611+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f248c06c9a0 con 0x7f249c05fa50 2022-07-20T03:07:30.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.611+0000 7f249bfff700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24740539d0 0x7f2474055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:30.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.611+0000 7f249bfff700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24740539d0 0x7f2474055e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f248400d000 tx=0x7f2484007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:30.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.619+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f248c043880 con 0x7f249c05fa50 2022-07-20T03:07:30.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.747+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f249c062270 con 0x7f249c05fa50 2022-07-20T03:07:30.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.747+0000 7f2499ffb700 1 -- 172.21.15.61:0/3148644239 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f248c023020 con 0x7f249c05fa50 2022-07-20T03:07:30.766 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:30.767 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:30.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24740539d0 msgr2=0x7f2474055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:30.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24740539d0 0x7f2474055e90 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f248400d000 tx=0x7f2484007b90).stop 2022-07-20T03:07:30.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 msgr2=0x7f249c09aab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:30.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 0x7f249c09aab0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f248c002f70 tx=0x7f248c014000).stop 2022-07-20T03:07:30.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 shutdown_connections 2022-07-20T03:07:30.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24740539d0 0x7f2474055e90 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f249c0b1ef0 0x7f249c09b490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f249c0ac3a0 0x7f249c09afa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 --2- 172.21.15.61:0/3148644239 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f249c05fa50 0x7f249c09aab0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:30.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 >> 172.21.15.61:0/3148644239 conn(0x7f249c0b2ca0 msgr2=0x7f249c1404b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:30.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 shutdown_connections 2022-07-20T03:07:30.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.751+0000 7f24a1da6700 1 -- 172.21.15.61:0/3148644239 wait complete. 2022-07-20T03:07:30.784 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:30.785 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:30.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c824cb700 1 Processor -- start 2022-07-20T03:07:30.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c824cb700 1 -- start start 2022-07-20T03:07:31.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c824cb700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6c7c0b7bb0 0x7f6c7c1408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c824cb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c05c4c0 con 0x7f6c7c0addd0 2022-07-20T03:07:31.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c824cb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c7c0b7f90 con 0x7f6c7c0b3770 2022-07-20T03:07:31.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c824cb700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c7c05bd90 con 0x7f6c7c0b7bb0 2022-07-20T03:07:31.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c81cca700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6c7c0addd0 0x7f6c7c13d620 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39534/0 (socket says 172.21.15.61:39534) 2022-07-20T03:07:31.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c81cca700 1 -- 172.21.15.61:0/528467333 learned_addr learned my addr 172.21.15.61:0/528467333 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:31.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c814c9700 1 --2- 172.21.15.61:0/528467333 >> v2:172.21.15.117:3300/0 conn(0x7f6c7c0b7bb0 0x7f6c7c1408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c73fff700 1 -- 172.21.15.61:0/528467333 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1495698846 0 0) 0x7f6c7c05c4c0 con 0x7f6c7c0addd0 2022-07-20T03:07:31.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.939+0000 7f6c73fff700 1 -- 172.21.15.61:0/528467333 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6c60003260 con 0x7f6c7c0addd0 2022-07-20T03:07:31.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c814c9700 1 -- 172.21.15.61:0/528467333 >> v1:172.21.15.117:6789/0 conn(0x7f6c7c0b3770 legacy=0x7f6c7c05ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c814c9700 1 -- 172.21.15.61:0/528467333 >> v1:172.21.15.61:6789/0 conn(0x7f6c7c0addd0 legacy=0x7f6c7c13d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c814c9700 1 -- 172.21.15.61:0/528467333 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c05bf80 con 0x7f6c7c0b7bb0 2022-07-20T03:07:31.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c814c9700 1 --2- 172.21.15.61:0/528467333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c1408d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6c740032f0 tx=0x7f6c740091d0).ready entity=mon.1 client_cookie=16b346f14d206f3f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c73fff700 1 -- 172.21.15.61:0/528467333 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c7400b6c0 con 0x7f6c7c0b7bb0 2022-07-20T03:07:31.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c73fff700 1 -- 172.21.15.61:0/528467333 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c7400b820 con 0x7f6c7c0b7bb0 2022-07-20T03:07:31.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c73fff700 1 -- 172.21.15.61:0/528467333 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c7400b990 con 0x7f6c7c0b7bb0 2022-07-20T03:07:31.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 -- 172.21.15.61:0/528467333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 msgr2=0x7f6c7c1408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 --2- 172.21.15.61:0/528467333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c1408d0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f6c740032f0 tx=0x7f6c740091d0).stop 2022-07-20T03:07:31.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 -- 172.21.15.61:0/528467333 shutdown_connections 2022-07-20T03:07:31.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 --2- 172.21.15.61:0/528467333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c1408d0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 -- 172.21.15.61:0/528467333 >> 172.21.15.61:0/528467333 conn(0x7f6c7c065720 msgr2=0x7f6c7c0b7160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:31.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 -- 172.21.15.61:0/528467333 shutdown_connections 2022-07-20T03:07:31.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 -- 172.21.15.61:0/528467333 wait complete. 2022-07-20T03:07:31.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 Processor -- start 2022-07-20T03:07:31.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.943+0000 7f6c824cb700 1 -- start start 2022-07-20T03:07:31.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c7c0addd0 0x7f6c7c140380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 0x7f6c7c13dd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c13e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c150f00 con 0x7f6c7c0b3770 2022-07-20T03:07:31.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c141da0 con 0x7f6c7c0b7bb0 2022-07-20T03:07:31.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c7c142d30 con 0x7f6c7c0addd0 2022-07-20T03:07:31.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 0x7f6c7c13dd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 0x7f6c7c13dd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42940/0 (socket says 172.21.15.61:42940) 2022-07-20T03:07:31.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 -- 172.21.15.61:0/1332657532 learned_addr learned my addr 172.21.15.61:0/1332657532 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:31.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c814c9700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c7c0addd0 0x7f6c7c140380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c81cca700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c13e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 -- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c7c0addd0 msgr2=0x7f6c7c140380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c7c0addd0 0x7f6c7c140380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 -- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 msgr2=0x7f6c7c13e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c13e210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 -- 172.21.15.61:0/1332657532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c74009470 con 0x7f6c7c0b3770 2022-07-20T03:07:31.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c80cc8700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 0x7f6c7c13dd20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6c7c05c8f0 tx=0x7f6c6c007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6c007030 con 0x7f6c7c0b3770 2022-07-20T03:07:31.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c007190 con 0x7f6c7c0b3770 2022-07-20T03:07:31.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c7c13e700 con 0x7f6c7c0b3770 2022-07-20T03:07:31.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6c006510 con 0x7f6c7c0b3770 2022-07-20T03:07:31.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.947+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c7c1475e0 con 0x7f6c7c0b3770 2022-07-20T03:07:31.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.951+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c7c150000 con 0x7f6c7c0b3770 2022-07-20T03:07:31.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.951+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6c6c010030 con 0x7f6c7c0b3770 2022-07-20T03:07:31.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.951+0000 7f6c727fc700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c64053a40 0x7f6c64055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.951+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6c6c06b990 con 0x7f6c7c0b3770 2022-07-20T03:07:31.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.951+0000 7f6c814c9700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c64053a40 0x7f6c64055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.951+0000 7f6c814c9700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c64053a40 0x7f6c64055f00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6c7401b000 tx=0x7f6c74016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:30.955+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6c7c150000 con 0x7f6c7c0b3770 2022-07-20T03:07:31.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.079+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6c7c150000 con 0x7f6c7c0b3770 2022-07-20T03:07:31.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.079+0000 7f6c727fc700 1 -- 172.21.15.61:0/1332657532 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6c7c150000 con 0x7f6c7c0b3770 2022-07-20T03:07:31.098 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:31.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c64053a40 msgr2=0x7f6c64055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c64053a40 0x7f6c64055f00 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f6c7401b000 tx=0x7f6c74016000).stop 2022-07-20T03:07:31.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 msgr2=0x7f6c7c13dd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 0x7f6c7c13dd20 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f6c7c05c8f0 tx=0x7f6c6c007c10).stop 2022-07-20T03:07:31.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 shutdown_connections 2022-07-20T03:07:31.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c64053a40 0x7f6c64055f00 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c7c0b7bb0 0x7f6c7c13e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c7c0b3770 0x7f6c7c13dd20 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 --2- 172.21.15.61:0/1332657532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c7c0addd0 0x7f6c7c140380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 >> 172.21.15.61:0/1332657532 conn(0x7f6c7c065720 msgr2=0x7f6c7c0b6520 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:31.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 shutdown_connections 2022-07-20T03:07:31.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.083+0000 7f6c824cb700 1 -- 172.21.15.61:0/1332657532 wait complete. 2022-07-20T03:07:31.131 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:31.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.251+0000 7f86fda6b700 1 Processor -- start 2022-07-20T03:07:31.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.251+0000 7f86fda6b700 1 -- start start 2022-07-20T03:07:31.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.251+0000 7f86fda6b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f86f80ac640 0x7f86f81410f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f80b2360 con 0x7f86f80ac260 2022-07-20T03:07:31.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86f8097a50 con 0x7f86f80b36b0 2022-07-20T03:07:31.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86f80b3b20 con 0x7f86f80ac640 2022-07-20T03:07:31.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f86f80ac260 0x7f86f8144060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39546/0 (socket says 172.21.15.61:39546) 2022-07-20T03:07:31.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f77fe700 1 -- 172.21.15.61:0/3398951386 learned_addr learned my addr 172.21.15.61:0/3398951386 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:31.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f6ffd700 1 --2- 172.21.15.61:0/3398951386 >> v2:172.21.15.117:3300/0 conn(0x7f86f80ac640 0x7f86f81410f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 687807428 0 0) 0x7f86f80b2360 con 0x7f86f80ac260 2022-07-20T03:07:31.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86e4003260 con 0x7f86f80ac260 2022-07-20T03:07:31.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 376135952 0 0) 0x7f86f8097a50 con 0x7f86f80b36b0 2022-07-20T03:07:31.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86f80b2360 con 0x7f86f80b36b0 2022-07-20T03:07:31.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1720606898 0 0) 0x7f86e4003260 con 0x7f86f80ac260 2022-07-20T03:07:31.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 >> v2:172.21.15.117:3300/0 conn(0x7f86f80ac640 msgr2=0x7f86f81410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:31.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 --2- 172.21.15.61:0/3398951386 >> v2:172.21.15.117:3300/0 conn(0x7f86f80ac640 0x7f86f81410f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 >> v1:172.21.15.117:6789/0 conn(0x7f86f80b36b0 legacy=0x7f86f8144770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8144e80 con 0x7f86f80ac260 2022-07-20T03:07:31.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f86ec001660 con 0x7f86f80ac260 2022-07-20T03:07:31.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 >> v1:172.21.15.61:6789/0 conn(0x7f86f80ac260 legacy=0x7f86f8144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86e40038e0 0x7f86e4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004500 0x7f86e4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86e4004e70 0x7f86e400fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80b3b20 con 0x7f86e4004e70 2022-07-20T03:07:31.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86e4010100 con 0x7f86e40038e0 2022-07-20T03:07:31.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86e40102c0 con 0x7f86e4004500 2022-07-20T03:07:31.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86e4004e70 0x7f86e400fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f6ffd700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004500 0x7f86e4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f77fe700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86e40038e0 0x7f86e4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 -- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004500 msgr2=0x7f86e4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:31.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004500 0x7f86e4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 -- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86e40038e0 msgr2=0x7f86e4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:31.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86e40038e0 0x7f86e4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 -- 172.21.15.61:0/3398951386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86e4003260 con 0x7f86e4004e70 2022-07-20T03:07:31.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f77fe700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86e40038e0 0x7f86e4003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:31.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 -- 172.21.15.61:0/3398951386 reap_dead start 2022-07-20T03:07:31.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86e4004e70 0x7f86e400fb60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f86f0001490 tx=0x7f86f0007d80).ready entity=mon.0 client_cookie=afff9083a6d5d23b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86f0006890 con 0x7f86e4004e70 2022-07-20T03:07:31.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f00069f0 con 0x7f86e4004e70 2022-07-20T03:07:31.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86f67fc700 1 -- 172.21.15.61:0/3398951386 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86f0006d50 con 0x7f86e4004e70 2022-07-20T03:07:31.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- 172.21.15.61:0/3398951386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86e4004e70 msgr2=0x7f86e400fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86e4004e70 0x7f86e400fb60 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f86f0001490 tx=0x7f86f0007d80).stop 2022-07-20T03:07:31.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- 172.21.15.61:0/3398951386 shutdown_connections 2022-07-20T03:07:31.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3398951386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86e4004e70 0x7f86e400fb60 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- 172.21.15.61:0/3398951386 >> 172.21.15.61:0/3398951386 conn(0x7f86f80b47c0 msgr2=0x7f86f805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- 172.21.15.61:0/3398951386 shutdown_connections 2022-07-20T03:07:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.255+0000 7f86fda6b700 1 -- 172.21.15.61:0/3398951386 wait complete. 2022-07-20T03:07:31.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 Processor -- start 2022-07-20T03:07:31.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 -- start start 2022-07-20T03:07:31.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004e70 0x7f86f8099030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86f8097f00 0x7f86f80b10a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 0x7f86f80afd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8144840 con 0x7f86f8144390 2022-07-20T03:07:31.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86f8144aa0 con 0x7f86f8097f00 2022-07-20T03:07:31.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86f80988f0 con 0x7f86e4004e70 2022-07-20T03:07:31.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 0x7f86f80afd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 0x7f86f80afd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42962/0 (socket says 172.21.15.61:42962) 2022-07-20T03:07:31.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 -- 172.21.15.61:0/3505926197 learned_addr learned my addr 172.21.15.61:0/3505926197 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:31.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f77fe700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004e70 0x7f86f8099030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f6ffd700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86f8097f00 0x7f86f80b10a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 -- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004e70 msgr2=0x7f86f8099030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004e70 0x7f86f8099030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 -- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86f8097f00 msgr2=0x7f86f80b10a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86f8097f00 0x7f86f80b10a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 -- 172.21.15.61:0/3505926197 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86f8098b90 con 0x7f86f8144390 2022-07-20T03:07:31.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f6ffd700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86f8097f00 0x7f86f80b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:31.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f7fff700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 0x7f86f80afd50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f86f0001490 tx=0x7f86f0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86f0006890 con 0x7f86f8144390 2022-07-20T03:07:31.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86f000c9e0 con 0x7f86f8144390 2022-07-20T03:07:31.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86f80b0240 con 0x7f86f8144390 2022-07-20T03:07:31.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86f0001780 con 0x7f86f8144390 2022-07-20T03:07:31.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.259+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86f80b07c0 con 0x7f86f8144390 2022-07-20T03:07:31.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.263+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86f8098d20 con 0x7f86f8144390 2022-07-20T03:07:31.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.263+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f86f0026030 con 0x7f86f8144390 2022-07-20T03:07:31.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.263+0000 7f86f4ff9700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86d80539b0 0x7f86d8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.263+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f86f006d5f0 con 0x7f86f8144390 2022-07-20T03:07:31.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.263+0000 7f86f77fe700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86d80539b0 0x7f86d8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.263+0000 7f86f77fe700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86d80539b0 0x7f86d8055e70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f86ec00e000 tx=0x7f86ec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.271+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86f0044440 con 0x7f86f8144390 2022-07-20T03:07:31.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f86f80574e0 con 0x7f86f8144390 2022-07-20T03:07:31.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86f4ff9700 1 -- 172.21.15.61:0/3505926197 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f86f0024070 con 0x7f86f8144390 2022-07-20T03:07:31.494 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:31.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86d80539b0 msgr2=0x7f86d8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86d80539b0 0x7f86d8055e70 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f86ec00e000 tx=0x7f86ec009040).stop 2022-07-20T03:07:31.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 msgr2=0x7f86f80afd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 0x7f86f80afd50 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f86f0001490 tx=0x7f86f0016000).stop 2022-07-20T03:07:31.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 shutdown_connections 2022-07-20T03:07:31.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86d80539b0 0x7f86d8055e70 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86f8097f00 0x7f86f80b10a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86f8144390 0x7f86f80afd50 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 --2- 172.21.15.61:0/3505926197 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86e4004e70 0x7f86f8099030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.395+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 >> 172.21.15.61:0/3505926197 conn(0x7f86f80b47c0 msgr2=0x7f86f8061260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:31.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.399+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 shutdown_connections 2022-07-20T03:07:31.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.399+0000 7f86fda6b700 1 -- 172.21.15.61:0/3505926197 wait complete. 2022-07-20T03:07:31.527 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:31.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.659+0000 7f196f18b700 1 Processor -- start 2022-07-20T03:07:31.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.659+0000 7f196f18b700 1 -- start start 2022-07-20T03:07:31.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f19680b9da0 0x7f19681468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f19680b3cf0 0x7f1968142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19680ba180 con 0x7f1968142f40 2022-07-20T03:07:31.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19680ad5b0 con 0x7f19680b9da0 2022-07-20T03:07:31.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f19681461b0 con 0x7f19680b3cf0 2022-07-20T03:07:31.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e189700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f19680b9da0 0x7f19681468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e98a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1968142f40 0x7f196805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39568/0 (socket says 172.21.15.61:39568) 2022-07-20T03:07:31.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e98a700 1 -- 172.21.15.61:0/1521275444 learned_addr learned my addr 172.21.15.61:0/1521275444 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:31.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196d988700 1 --2- 172.21.15.61:0/1521275444 >> v2:172.21.15.117:3300/0 conn(0x7f19680b3cf0 0x7f1968142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e189700 1 -- 172.21.15.61:0/1521275444 >> v2:172.21.15.117:3300/0 conn(0x7f19680b3cf0 msgr2=0x7f1968142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:31.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e189700 1 --2- 172.21.15.61:0/1521275444 >> v2:172.21.15.117:3300/0 conn(0x7f19680b3cf0 0x7f1968142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e189700 1 -- 172.21.15.61:0/1521275444 >> v1:172.21.15.61:6789/0 conn(0x7f1968142f40 legacy=0x7f196805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196d187700 1 -- 172.21.15.61:0/1521275444 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3518550002 0 0) 0x7f19680ba180 con 0x7f1968142f40 2022-07-20T03:07:31.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e189700 1 -- 172.21.15.61:0/1521275444 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1968146450 con 0x7f19680b9da0 2022-07-20T03:07:31.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196e189700 1 --2- 172.21.15.61:0/1521275444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b9da0 0x7f19681468f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f19600032f0 tx=0x7f19600091d0).ready entity=mon.0 client_cookie=acf45571865c85e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196d187700 1 -- 172.21.15.61:0/1521275444 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f196000b680 con 0x7f19680b9da0 2022-07-20T03:07:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196d187700 1 -- 172.21.15.61:0/1521275444 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f196000b7e0 con 0x7f19680b9da0 2022-07-20T03:07:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196d187700 1 -- 172.21.15.61:0/1521275444 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f196000b950 con 0x7f19680b9da0 2022-07-20T03:07:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- 172.21.15.61:0/1521275444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b9da0 msgr2=0x7f19681468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 --2- 172.21.15.61:0/1521275444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b9da0 0x7f19681468f0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f19600032f0 tx=0x7f19600091d0).stop 2022-07-20T03:07:31.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- 172.21.15.61:0/1521275444 shutdown_connections 2022-07-20T03:07:31.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 --2- 172.21.15.61:0/1521275444 >> v2:172.21.15.117:3300/0 conn(0x7f19680b3cf0 0x7f1968142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 --2- 172.21.15.61:0/1521275444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b9da0 0x7f19681468f0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- 172.21.15.61:0/1521275444 >> 172.21.15.61:0/1521275444 conn(0x7f19680678c0 msgr2=0x7f19680b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:31.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- 172.21.15.61:0/1521275444 shutdown_connections 2022-07-20T03:07:31.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.663+0000 7f196f18b700 1 -- 172.21.15.61:0/1521275444 wait complete. 2022-07-20T03:07:31.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 Processor -- start 2022-07-20T03:07:31.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 -- start start 2022-07-20T03:07:31.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 0x7f19680b18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19680b9da0 0x7f19681485a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1968142f40 0x7f1968155020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19681133f0 con 0x7f19680b3cf0 2022-07-20T03:07:31.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19680b0bb0 con 0x7f19680b9da0 2022-07-20T03:07:31.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196f18b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19680b0e50 con 0x7f1968142f40 2022-07-20T03:07:31.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196e189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 0x7f19680b18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196e189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 0x7f19680b18c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42972/0 (socket says 172.21.15.61:42972) 2022-07-20T03:07:31.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196e189700 1 -- 172.21.15.61:0/3076351330 learned_addr learned my addr 172.21.15.61:0/3076351330 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:31.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.667+0000 7f196d988700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19680b9da0 0x7f19681485a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e98a700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1968142f40 0x7f1968155020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e189700 1 -- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1968142f40 msgr2=0x7f1968155020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e189700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1968142f40 0x7f1968155020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e189700 1 -- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19680b9da0 msgr2=0x7f19681485a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e189700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19680b9da0 0x7f19681485a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e189700 1 -- 172.21.15.61:0/3076351330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1960009470 con 0x7f19680b3cf0 2022-07-20T03:07:31.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196e189700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 0x7f19680b18c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f19600032f0 tx=0x7f1960016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f196000b950 con 0x7f19680b3cf0 2022-07-20T03:07:31.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1960021630 con 0x7f19680b3cf0 2022-07-20T03:07:31.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19680b0830 con 0x7f19680b3cf0 2022-07-20T03:07:31.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f196000b680 con 0x7f19680b3cf0 2022-07-20T03:07:31.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19680b2ad0 con 0x7f19680b3cf0 2022-07-20T03:07:31.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19680574e0 con 0x7f19680b3cf0 2022-07-20T03:07:31.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.671+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1960027020 con 0x7f19680b3cf0 2022-07-20T03:07:31.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.675+0000 7f19677fe700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19480539b0 0x7f1948055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:31.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.675+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f196006d470 con 0x7f19680b3cf0 2022-07-20T03:07:31.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.675+0000 7f196d988700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19480539b0 0x7f1948055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:31.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.675+0000 7f196d988700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19480539b0 0x7f1948055e70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f195c00b000 tx=0x7f195c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:31.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.679+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19600442c0 con 0x7f19680b3cf0 2022-07-20T03:07:31.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.803+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f19680699b0 con 0x7f19680b3cf0 2022-07-20T03:07:31.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.803+0000 7f19677fe700 1 -- 172.21.15.61:0/3076351330 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f19600478e0 con 0x7f19680b3cf0 2022-07-20T03:07:31.840 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:31.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19480539b0 msgr2=0x7f1948055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19480539b0 0x7f1948055e70 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f195c00b000 tx=0x7f195c006df0).stop 2022-07-20T03:07:31.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 msgr2=0x7f19680b18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:31.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 0x7f19680b18c0 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f19600032f0 tx=0x7f1960016000).stop 2022-07-20T03:07:31.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 shutdown_connections 2022-07-20T03:07:31.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19480539b0 0x7f1948055e70 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19680b9da0 0x7f19681485a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1968142f40 0x7f1968155020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 --2- 172.21.15.61:0/3076351330 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19680b3cf0 0x7f19680b18c0 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:31.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 >> 172.21.15.61:0/3076351330 conn(0x7f19680678c0 msgr2=0x7f196805c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:31.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 shutdown_connections 2022-07-20T03:07:31.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:31.807+0000 7f196f18b700 1 -- 172.21.15.61:0/3076351330 wait complete. 2022-07-20T03:07:31.859 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:32.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 Processor -- start 2022-07-20T03:07:32.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 -- start start 2022-07-20T03:07:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd1480b23b0 0x7fd1480ab550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd1480aba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd148142fa0 0x7fd148143410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd148060010 con 0x7fd1480b23b0 2022-07-20T03:07:32.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1480602b0 con 0x7fd148142fa0 2022-07-20T03:07:32.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14ffe2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd1480ab0b0 con 0x7fd1480ac650 2022-07-20T03:07:32.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd1480b23b0 0x7fd1480ab550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd1480b23b0 0x7fd1480ab550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42980/0 (socket says 172.21.15.61:42980) 2022-07-20T03:07:32.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 -- 172.21.15.61:0/1346823802 learned_addr learned my addr 172.21.15.61:0/1346823802 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:32.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14e57f700 1 --2- 172.21.15.61:0/1346823802 >> v2:172.21.15.117:3300/0 conn(0x7fd148142fa0 0x7fd148143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14d57d700 1 --2- 172.21.15.61:0/1346823802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd1480aba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 -- 172.21.15.61:0/1346823802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 msgr2=0x7fd1480aba40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:32.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/1346823802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd1480aba40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 -- 172.21.15.61:0/1346823802 >> v2:172.21.15.117:3300/0 conn(0x7fd148142fa0 msgr2=0x7fd148143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:32.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/1346823802 >> v2:172.21.15.117:3300/0 conn(0x7fd148142fa0 0x7fd148143410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 -- 172.21.15.61:0/1346823802 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1480abf30 con 0x7fd1480b23b0 2022-07-20T03:07:32.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/1346823802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1480b23b0 0x7fd1480ab550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fd144002f70 tx=0x7fd144006eb0).ready entity=mon.0 client_cookie=9b5e329a9d225cb8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:32.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14cd7c700 1 -- 172.21.15.61:0/1346823802 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd14400f970 con 0x7fd1480b23b0 2022-07-20T03:07:32.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14cd7c700 1 -- 172.21.15.61:0/1346823802 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd144000e90 con 0x7fd1480b23b0 2022-07-20T03:07:32.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.007+0000 7fd14cd7c700 1 -- 172.21.15.61:0/1346823802 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd14400fd00 con 0x7fd1480b23b0 2022-07-20T03:07:32.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- 172.21.15.61:0/1346823802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1480b23b0 msgr2=0x7fd1480ab550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/1346823802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1480b23b0 0x7fd1480ab550 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7fd144002f70 tx=0x7fd144006eb0).stop 2022-07-20T03:07:32.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- 172.21.15.61:0/1346823802 shutdown_connections 2022-07-20T03:07:32.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/1346823802 >> v2:172.21.15.117:3300/0 conn(0x7fd148142fa0 0x7fd148143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/1346823802 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd1480aba40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/1346823802 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd1480b23b0 0x7fd1480ab550 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- 172.21.15.61:0/1346823802 >> 172.21.15.61:0/1346823802 conn(0x7fd1480b3600 msgr2=0x7fd1480b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:32.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- 172.21.15.61:0/1346823802 shutdown_connections 2022-07-20T03:07:32.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- 172.21.15.61:0/1346823802 wait complete. 2022-07-20T03:07:32.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 Processor -- start 2022-07-20T03:07:32.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- start start 2022-07-20T03:07:32.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd148155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1480b23b0 0x7fd148151f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 0x7fd148152440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd14814fa30 con 0x7fd148142fa0 2022-07-20T03:07:32.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1481556a0 con 0x7fd1480b23b0 2022-07-20T03:07:32.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14ffe2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd148155940 con 0x7fd1480ac650 2022-07-20T03:07:32.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14e57f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 0x7fd148152440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14e57f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 0x7fd148152440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42990/0 (socket says 172.21.15.61:42990) 2022-07-20T03:07:32.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.011+0000 7fd14e57f700 1 -- 172.21.15.61:0/727865946 learned_addr learned my addr 172.21.15.61:0/727865946 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:32.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd148155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14d57d700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1480b23b0 0x7fd148151f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14e57f700 1 -- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 msgr2=0x7fd148155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14e57f700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd148155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14e57f700 1 -- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1480b23b0 msgr2=0x7fd148151f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14e57f700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1480b23b0 0x7fd148151f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14e57f700 1 -- 172.21.15.61:0/727865946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd14400b1e0 con 0x7fd148142fa0 2022-07-20T03:07:32.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14d57d700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1480b23b0 0x7fd148151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:32.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd148155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:32.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14e57f700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 0x7fd148152440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd140006e00 tx=0x7fd1400091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:32.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd14000c7f0 con 0x7fd148142fa0 2022-07-20T03:07:32.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd14000c950 con 0x7fd148142fa0 2022-07-20T03:07:32.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd14000ccb0 con 0x7fd148142fa0 2022-07-20T03:07:32.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd148153080 con 0x7fd148142fa0 2022-07-20T03:07:32.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1480b0bc0 con 0x7fd148142fa0 2022-07-20T03:07:32.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd148155b00 con 0x7fd148142fa0 2022-07-20T03:07:32.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.015+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd140022070 con 0x7fd148142fa0 2022-07-20T03:07:32.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.019+0000 7fd136ffd700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd128053940 0x7fd128055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.019+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd14006c610 con 0x7fd148142fa0 2022-07-20T03:07:32.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.019+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd128053940 0x7fd128055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.019+0000 7fd14dd7e700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd128053940 0x7fd128055e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd14401a000 tx=0x7fd144017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:32.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.023+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd140043460 con 0x7fd148142fa0 2022-07-20T03:07:32.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.151+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd1480574e0 con 0x7fd148142fa0 2022-07-20T03:07:32.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.151+0000 7fd136ffd700 1 -- 172.21.15.61:0/727865946 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd140016020 con 0x7fd148142fa0 2022-07-20T03:07:32.230 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:32.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd128053940 msgr2=0x7fd128055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd128053940 0x7fd128055e00 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7fd14401a000 tx=0x7fd144017000).stop 2022-07-20T03:07:32.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 msgr2=0x7fd148152440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 0x7fd148152440 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fd140006e00 tx=0x7fd1400091d0).stop 2022-07-20T03:07:32.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 shutdown_connections 2022-07-20T03:07:32.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd128053940 0x7fd128055e00 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1480b23b0 0x7fd148151f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd148142fa0 0x7fd148152440 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 --2- 172.21.15.61:0/727865946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1480ac650 0x7fd148155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 >> 172.21.15.61:0/727865946 conn(0x7fd1480b3600 msgr2=0x7fd1480b3f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:32.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 shutdown_connections 2022-07-20T03:07:32.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.155+0000 7fd14ffe2700 1 -- 172.21.15.61:0/727865946 wait complete. 2022-07-20T03:07:32.244 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:07:32.245 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:32.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.355+0000 7ffae8d30700 1 Processor -- start 2022-07-20T03:07:32.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.355+0000 7ffae8d30700 1 -- start start 2022-07-20T03:07:32.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ffae40b1e00 0x7ffae414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffae405c3f0 0x7ffae40ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffae405d480 con 0x7ffae405bd90 2022-07-20T03:07:32.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae40b21e0 con 0x7ffae405c3f0 2022-07-20T03:07:32.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffae405cf40 con 0x7ffae40b1e00 2022-07-20T03:07:32.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffae405c3f0 0x7ffae40ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffae405c3f0 0x7ffae40ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:42996/0 (socket says 172.21.15.61:42996) 2022-07-20T03:07:32.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae2d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ffae405bd90 0x7ffae4146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39594/0 (socket says 172.21.15.61:39594) 2022-07-20T03:07:32.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 -- 172.21.15.61:0/2239053519 learned_addr learned my addr 172.21.15.61:0/2239053519 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:32.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae259c700 1 --2- 172.21.15.61:0/2239053519 >> v2:172.21.15.117:3300/0 conn(0x7ffae40b1e00 0x7ffae414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 -- 172.21.15.61:0/2239053519 >> v2:172.21.15.117:3300/0 conn(0x7ffae40b1e00 msgr2=0x7ffae414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:32.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 --2- 172.21.15.61:0/2239053519 >> v2:172.21.15.117:3300/0 conn(0x7ffae40b1e00 0x7ffae414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 -- 172.21.15.61:0/2239053519 >> v1:172.21.15.61:6789/0 conn(0x7ffae405bd90 legacy=0x7ffae4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 -- 172.21.15.61:0/2239053519 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffae405d1e0 con 0x7ffae405c3f0 2022-07-20T03:07:32.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae1d9b700 1 --2- 172.21.15.61:0/2239053519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae405c3f0 0x7ffae40ac7e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ffacc002f70 tx=0x7ffacc006eb0).ready entity=mon.0 client_cookie=1c1fac67d6616c90 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:32.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae159a700 1 -- 172.21.15.61:0/2239053519 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffacc00d610 con 0x7ffae405c3f0 2022-07-20T03:07:32.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae159a700 1 -- 172.21.15.61:0/2239053519 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc000e90 con 0x7ffae405c3f0 2022-07-20T03:07:32.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae159a700 1 -- 172.21.15.61:0/2239053519 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffacc00d9a0 con 0x7ffae405c3f0 2022-07-20T03:07:32.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- 172.21.15.61:0/2239053519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae405c3f0 msgr2=0x7ffae40ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 --2- 172.21.15.61:0/2239053519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae405c3f0 0x7ffae40ac7e0 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7ffacc002f70 tx=0x7ffacc006eb0).stop 2022-07-20T03:07:32.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- 172.21.15.61:0/2239053519 shutdown_connections 2022-07-20T03:07:32.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 --2- 172.21.15.61:0/2239053519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae405c3f0 0x7ffae40ac7e0 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 --2- 172.21.15.61:0/2239053519 >> v2:172.21.15.117:3300/0 conn(0x7ffae40b1e00 0x7ffae414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- 172.21.15.61:0/2239053519 >> 172.21.15.61:0/2239053519 conn(0x7ffae40611c0 msgr2=0x7ffae4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:32.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- 172.21.15.61:0/2239053519 shutdown_connections 2022-07-20T03:07:32.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- 172.21.15.61:0/2239053519 wait complete. 2022-07-20T03:07:32.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 Processor -- start 2022-07-20T03:07:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.359+0000 7ffae8d30700 1 -- start start 2022-07-20T03:07:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae40b1e00 0x7ffae409fb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 0x7ffae40a03f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffae40a08e0 0x7ffae4142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae409f140 con 0x7ffae4064570 2022-07-20T03:07:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4063b90 con 0x7ffae40b1e00 2022-07-20T03:07:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffae4063e30 con 0x7ffae40a08e0 2022-07-20T03:07:32.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 0x7ffae40a03f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 0x7ffae40a03f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43002/0 (socket says 172.21.15.61:43002) 2022-07-20T03:07:32.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 -- 172.21.15.61:0/744491131 learned_addr learned my addr 172.21.15.61:0/744491131 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:32.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae259c700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae40b1e00 0x7ffae409fb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae2d9d700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffae40a08e0 0x7ffae4142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 -- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffae40a08e0 msgr2=0x7ffae4142320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffae40a08e0 0x7ffae4142320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 -- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae40b1e00 msgr2=0x7ffae409fb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae40b1e00 0x7ffae409fb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 -- 172.21.15.61:0/744491131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffacc00b1e0 con 0x7ffae4064570 2022-07-20T03:07:32.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae1d9b700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 0x7ffae40a03f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ffacc000b60 tx=0x7ffacc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:32.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffacc00d990 con 0x7ffae4064570 2022-07-20T03:07:32.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffacc011c60 con 0x7ffae4064570 2022-07-20T03:07:32.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffae40640d0 con 0x7ffae4064570 2022-07-20T03:07:32.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffacc01e460 con 0x7ffae4064570 2022-07-20T03:07:32.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.363+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffae41429b0 con 0x7ffae4064570 2022-07-20T03:07:32.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.367+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffae4064260 con 0x7ffae4064570 2022-07-20T03:07:32.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.367+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffacc025030 con 0x7ffae4064570 2022-07-20T03:07:32.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.367+0000 7ffadb7fe700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffac8053af0 0x7ffac8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:32.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.367+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ffacc06d0b0 con 0x7ffae4064570 2022-07-20T03:07:32.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.367+0000 7ffae259c700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffac8053af0 0x7ffac8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:32.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.367+0000 7ffae259c700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffac8053af0 0x7ffac8055fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffad400c000 tx=0x7ffad4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:32.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.375+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffacc043f00 con 0x7ffae4064570 2022-07-20T03:07:32.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ffae40b04b0 con 0x7ffae4064570 2022-07-20T03:07:32.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffadb7fe700 1 -- 172.21.15.61:0/744491131 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2134 (secure 0 0 0) 0x7ffacc023020 con 0x7ffae4064570 2022-07-20T03:07:32.551 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:32.551 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:12.741818+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.117:6835/725875269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":725875269},{"type":"v1","addr":"172.21.15.117:6835","nonce":725875269}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:32.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffac8053af0 msgr2=0x7ffac8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffac8053af0 0x7ffac8055fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7ffad400c000 tx=0x7ffad4008040).stop 2022-07-20T03:07:32.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 msgr2=0x7ffae40a03f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:32.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 0x7ffae40a03f0 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7ffacc000b60 tx=0x7ffacc015000).stop 2022-07-20T03:07:32.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 shutdown_connections 2022-07-20T03:07:32.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffac8053af0 0x7ffac8055fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffae40a08e0 0x7ffae4142320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffae4064570 0x7ffae40a03f0 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 --2- 172.21.15.61:0/744491131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffae40b1e00 0x7ffae409fb20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:32.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 >> 172.21.15.61:0/744491131 conn(0x7ffae405bd90 msgr2=0x7ffae40af960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:32.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 shutdown_connections 2022-07-20T03:07:32.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:32.539+0000 7ffae8d30700 1 -- 172.21.15.61:0/744491131 wait complete. 2022-07-20T03:07:32.667 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2022-07-20T03:07:32.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:07:10.718708+0000', 'modified': '2022-07-20T03:07:12.741818+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.117:6835/725875269', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 725875269}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 725875269}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:07:32.885 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:07:32.898 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:07:33.048 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":4273,"want_state":"up:active","state":"up:active","rank_uptime":22.283489883000001,"mdsmap_epoch":7,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":25.825720658000002} 2022-07-20T03:07:33.049 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 4273, "mdsmap_epoch": 7, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 22.283489883, "state": "up:active", "uptime": 25.825720658, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:07:33.050 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:07:33.053 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:07:33.053 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:07:33.054 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:33.055 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:33.074 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:07:33.152 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:07:33.153 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:07:33.197 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:07:33.198 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:07:33.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:33.200 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:07:33.266 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:07:33.286 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:33.329 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:07:33.389 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:07:33.395 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:07:33.421 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:07:33.399+0000 7f4b7dab4080 -1 init, newargv = 0x5586fe4ea490 newargc=9 2022-07-20T03:07:33.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22236]: starting ceph client 2022-07-20T03:07:33.426 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22236]: starting fuse 2022-07-20T03:07:33.439 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:07:33.441 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:07:33.448 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:33.449 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:07:33.504 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:07:33.505 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:07:33.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:33.555 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:07:33.556 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:07:33.564 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:07:33.565 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:07:33.565 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:07:33.566 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:07:33.567 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:07:33.567 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:07:33.568 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:07:33.569 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:07:33.569 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:07:33.570 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:07:33.571 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:07:33.571 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:07:33.572 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:07:33.573 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:07:33.573 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:07:33.653 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:07:33.669 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:07:33.670 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:07:33.671 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:07:33.671 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:07:33.677 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:07:33.753 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22236.asok 2022-07-20T03:07:33.758 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22236.asok 2022-07-20T03:07:33.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:33.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22236.asok status 2022-07-20T03:07:33.931 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:07:33.932 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:07:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:07:33.933 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:07:33.938 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:07:33.939 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:07:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:07:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22236", 2022-07-20T03:07:33.940 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:07:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:07:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:07:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:07:33.941 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 4687, 2022-07-20T03:07:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:07:33.952 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:07:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:07:33.953 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 4687 2022-07-20T03:07:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:07:33.954 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:07:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:07:33.955 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:07:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2131031410 2022-07-20T03:07:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:07:33.956 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:07:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:07:33.957 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:07:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:07:33.958 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2131031410 2022-07-20T03:07:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:07:33.982 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.4687 172.21.15.163:0/2131031410", 2022-07-20T03:07:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/2131031410", 2022-07-20T03:07:33.983 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:07:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 7, 2022-07-20T03:07:33.984 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 23, 2022-07-20T03:07:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:07:33.985 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:07:33.997 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:07:33.998 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:07:34.000 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:34.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 Processor -- start 2022-07-20T03:07:34.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- start start 2022-07-20T03:07:34.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f25600b1620 0x7f2560142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25601426b0 con 0x7f256013ee40 2022-07-20T03:07:34.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25600b1a00 con 0x7f2560142fd0 2022-07-20T03:07:34.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2560060b50 con 0x7f25600b1620 2022-07-20T03:07:34.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2565df9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f25600b1620 0x7f2560142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f25665fa700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f256013ee40 0x7f25601499b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39608/0 (socket says 172.21.15.61:39608) 2022-07-20T03:07:34.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f25665fa700 1 -- 172.21.15.61:0/4237267768 learned_addr learned my addr 172.21.15.61:0/4237267768 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:34.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2565df9700 1 -- 172.21.15.61:0/4237267768 >> v1:172.21.15.117:6789/0 conn(0x7f2560142fd0 legacy=0x7f256013e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2565df9700 1 -- 172.21.15.61:0/4237267768 >> v1:172.21.15.61:6789/0 conn(0x7f256013ee40 legacy=0x7f25601499b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2564df7700 1 -- 172.21.15.61:0/4237267768 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2997222355 0 0) 0x7f25601426b0 con 0x7f256013ee40 2022-07-20T03:07:34.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2565df9700 1 -- 172.21.15.61:0/4237267768 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f256005f840 con 0x7f25600b1620 2022-07-20T03:07:34.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2565df9700 1 --2- 172.21.15.61:0/4237267768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560142ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f25500032f0 tx=0x7f255000a370).ready entity=mon.0 client_cookie=7629984570f3560a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2564df7700 1 -- 172.21.15.61:0/4237267768 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255000cc70 con 0x7f25600b1620 2022-07-20T03:07:34.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2564df7700 1 -- 172.21.15.61:0/4237267768 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255000cdd0 con 0x7f25600b1620 2022-07-20T03:07:34.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f2564df7700 1 -- 172.21.15.61:0/4237267768 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2550010600 con 0x7f25600b1620 2022-07-20T03:07:34.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- 172.21.15.61:0/4237267768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 msgr2=0x7f2560142ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 --2- 172.21.15.61:0/4237267768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560142ae0 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f25500032f0 tx=0x7f255000a370).stop 2022-07-20T03:07:34.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- 172.21.15.61:0/4237267768 shutdown_connections 2022-07-20T03:07:34.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 --2- 172.21.15.61:0/4237267768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560142ae0 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- 172.21.15.61:0/4237267768 >> 172.21.15.61:0/4237267768 conn(0x7f25600b23d0 msgr2=0x7f25600b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:34.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- 172.21.15.61:0/4237267768 shutdown_connections 2022-07-20T03:07:34.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.111+0000 7f256805d700 1 -- 172.21.15.61:0/4237267768 wait complete. 2022-07-20T03:07:34.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 Processor -- start 2022-07-20T03:07:34.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 -- start start 2022-07-20T03:07:34.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560146150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f256013ee40 0x7f2560152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2560142fd0 0x7f256009abc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25601558c0 con 0x7f25600b1620 2022-07-20T03:07:34.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2560092540 con 0x7f2560142fd0 2022-07-20T03:07:34.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f256805d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25600927e0 con 0x7f256013ee40 2022-07-20T03:07:34.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560146150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560146150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43014/0 (socket says 172.21.15.61:43014) 2022-07-20T03:07:34.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 -- 172.21.15.61:0/3864296621 learned_addr learned my addr 172.21.15.61:0/3864296621 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:34.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f25665fa700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2560142fd0 0x7f256009abc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f25655f8700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f256013ee40 0x7f2560152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 -- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f256013ee40 msgr2=0x7f2560152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f256013ee40 0x7f2560152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 -- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2560142fd0 msgr2=0x7f256009abc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2560142fd0 0x7f256009abc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 -- 172.21.15.61:0/3864296621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f255000a610 con 0x7f25600b1620 2022-07-20T03:07:34.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f25665fa700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2560142fd0 0x7f256009abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:34.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f25655f8700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f256013ee40 0x7f2560152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:34.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.115+0000 7f2565df9700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560146150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f25500032f0 tx=0x7f2560062d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255001a840 con 0x7f25600b1620 2022-07-20T03:07:34.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f255001a9a0 con 0x7f25600b1620 2022-07-20T03:07:34.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25600ae9e0 con 0x7f25600b1620 2022-07-20T03:07:34.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f255001ad00 con 0x7f25600b1620 2022-07-20T03:07:34.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25600aed50 con 0x7f25600b1620 2022-07-20T03:07:34.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f256005fae0 con 0x7f25600b1620 2022-07-20T03:07:34.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f255002a070 con 0x7f25600b1620 2022-07-20T03:07:34.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f255effd700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25400539b0 0x7f2540055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f255006ddd0 con 0x7f25600b1620 2022-07-20T03:07:34.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.119+0000 7f25655f8700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25400539b0 0x7f2540055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.123+0000 7f25655f8700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25400539b0 0x7f2540055e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2554008000 tx=0x7f2554009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.123+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2550044d30 con 0x7f25600b1620 2022-07-20T03:07:34.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.255+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f25600574e0 con 0x7f25600b1620 2022-07-20T03:07:34.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.259+0000 7f255effd700 1 -- 172.21.15.61:0/3864296621 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f2550048350 con 0x7f25600b1620 2022-07-20T03:07:34.265 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:34.274 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:34.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25400539b0 msgr2=0x7f2540055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25400539b0 0x7f2540055e70 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f2554008000 tx=0x7f2554009040).stop 2022-07-20T03:07:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 msgr2=0x7f2560146150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560146150 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f25500032f0 tx=0x7f2560062d60).stop 2022-07-20T03:07:34.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 shutdown_connections 2022-07-20T03:07:34.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25400539b0 0x7f2540055e70 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2560142fd0 0x7f256009abc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f256013ee40 0x7f2560152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 --2- 172.21.15.61:0/3864296621 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25600b1620 0x7f2560146150 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 >> 172.21.15.61:0/3864296621 conn(0x7f25600b23d0 msgr2=0x7f256013f220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:34.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 shutdown_connections 2022-07-20T03:07:34.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.263+0000 7f256805d700 1 -- 172.21.15.61:0/3864296621 wait complete. 2022-07-20T03:07:34.317 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:34.323 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 Processor -- start 2022-07-20T03:07:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 -- start start 2022-07-20T03:07:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d5c142f40 0x7f9d5c1419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d5c0ba180 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d5c0ad5b0 con 0x7f9d5c142f40 2022-07-20T03:07:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d626e0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c1461b0 con 0x7f9d5c0b3cf0 2022-07-20T03:07:34.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d616de700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9d5c0b9da0 0x7f9d5c1468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39620/0 (socket says 172.21.15.61:39620) 2022-07-20T03:07:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d616de700 1 -- 172.21.15.61:0/3933602075 learned_addr learned my addr 172.21.15.61:0/3933602075 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:34.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d61edf700 1 --2- 172.21.15.61:0/3933602075 >> v2:172.21.15.117:3300/0 conn(0x7f9d5c142f40 0x7f9d5c1419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.463+0000 7f9d60edd700 1 --2- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d53fff700 1 -- 172.21.15.61:0/3933602075 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3998363758 0 0) 0x7f9d5c0ba180 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d53fff700 1 -- 172.21.15.61:0/3933602075 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d40003260 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d61edf700 1 -- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 msgr2=0x7f9d5c142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d61edf700 1 --2- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d61edf700 1 -- 172.21.15.61:0/3933602075 >> v1:172.21.15.61:6789/0 conn(0x7f9d5c0b9da0 legacy=0x7f9d5c1468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d53fff700 1 -- 172.21.15.61:0/3933602075 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2942384401 0 0) 0x7f9d40003260 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d61edf700 1 -- 172.21.15.61:0/3933602075 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d5c146450 con 0x7f9d5c142f40 2022-07-20T03:07:34.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d61edf700 1 --2- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c1419a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9d4c001800 tx=0x7f9d540091d0).ready entity=mon.1 client_cookie=9a0285c74f322bb0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d53fff700 1 -- 172.21.15.61:0/3933602075 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d4c001660 con 0x7f9d5c142f40 2022-07-20T03:07:34.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d53fff700 1 -- 172.21.15.61:0/3933602075 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5400f510 con 0x7f9d5c142f40 2022-07-20T03:07:34.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d53fff700 1 -- 172.21.15.61:0/3933602075 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5400f680 con 0x7f9d5c142f40 2022-07-20T03:07:34.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 -- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 msgr2=0x7f9d5c1419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 --2- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c1419a0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f9d4c001800 tx=0x7f9d540091d0).stop 2022-07-20T03:07:34.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 -- 172.21.15.61:0/3933602075 shutdown_connections 2022-07-20T03:07:34.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 --2- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c1419a0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 --2- 172.21.15.61:0/3933602075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 -- 172.21.15.61:0/3933602075 >> 172.21.15.61:0/3933602075 conn(0x7f9d5c0678c0 msgr2=0x7f9d5c0b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:34.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 -- 172.21.15.61:0/3933602075 shutdown_connections 2022-07-20T03:07:34.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.467+0000 7f9d626e0700 1 -- 172.21.15.61:0/3933602075 wait complete. 2022-07-20T03:07:34.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 Processor -- start 2022-07-20T03:07:34.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 -- start start 2022-07-20T03:07:34.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c0b77e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 0x7f9d5c155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0af560 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c145890 con 0x7f9d5c142f40 2022-07-20T03:07:34.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d5c0b81c0 con 0x7f9d5c0b3cf0 2022-07-20T03:07:34.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 0x7f9d5c155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 0x7f9d5c155540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43030/0 (socket says 172.21.15.61:43030) 2022-07-20T03:07:34.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 -- 172.21.15.61:0/2529179767 learned_addr learned my addr 172.21.15.61:0/2529179767 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:34.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d616de700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c0b77e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d61edf700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 -- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 msgr2=0x7f9d5c0b77e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c0b77e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 -- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 msgr2=0x7f9d5c155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 -- 172.21.15.61:0/2529179767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d54009470 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d616de700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:34.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d60edd700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 0x7f9d5c155540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9d5c0af0f0 tx=0x7f9d58006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5800c640 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5800c7a0 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d5c0affe0 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5800cb00 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.471+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d5c145510 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.475+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d5c0667f0 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.475+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d5800cc60 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.475+0000 7f9d527fc700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053ad0 0x7f9d44055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.475+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9d5806bfe0 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.475+0000 7f9d616de700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053ad0 0x7f9d44055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.475+0000 7f9d616de700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053ad0 0x7f9d44055f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d4c00b000 tx=0x7f9d4c001140).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.483+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d58042e90 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.611+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f9d5c0574e0 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d527fc700 1 -- 172.21.15.61:0/2529179767 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9d580464b0 con 0x7f9d5c0b9da0 2022-07-20T03:07:34.623 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:34.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053ad0 msgr2=0x7f9d44055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053ad0 0x7f9d44055f90 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f9d4c00b000 tx=0x7f9d4c001140).stop 2022-07-20T03:07:34.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 msgr2=0x7f9d5c155540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 0x7f9d5c155540 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f9d5c0af0f0 tx=0x7f9d58006df0).stop 2022-07-20T03:07:34.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 shutdown_connections 2022-07-20T03:07:34.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d44053ad0 0x7f9d44055f90 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d5c142f40 0x7f9d5c155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d5c0b9da0 0x7f9d5c155540 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 --2- 172.21.15.61:0/2529179767 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d5c0b3cf0 0x7f9d5c0b77e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 >> 172.21.15.61:0/2529179767 conn(0x7f9d5c0678c0 msgr2=0x7f9d5c0b4dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:34.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 shutdown_connections 2022-07-20T03:07:34.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.615+0000 7f9d626e0700 1 -- 172.21.15.61:0/2529179767 wait complete. 2022-07-20T03:07:34.635 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 Processor -- start 2022-07-20T03:07:34.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- start start 2022-07-20T03:07:34.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2ea805cda0 0x7f2ea813a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2ea8143f00 0x7f2ea813aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:34.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2ea8065f20 con 0x7f2ea80b48d0 2022-07-20T03:07:34.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea80b29c0 con 0x7f2ea8143f00 2022-07-20T03:07:34.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2ea80661c0 con 0x7f2ea805cda0 2022-07-20T03:07:34.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2ea8143f00 0x7f2ea813aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea5d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2ea80b48d0 0x7f2ea80b38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39634/0 (socket says 172.21.15.61:39634) 2022-07-20T03:07:34.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2ea8143f00 0x7f2ea813aa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43040/0 (socket says 172.21.15.61:43040) 2022-07-20T03:07:34.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea5d9b700 1 -- 172.21.15.61:0/4243159131 learned_addr learned my addr 172.21.15.61:0/4243159131 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:34.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea659c700 1 --2- 172.21.15.61:0/4243159131 >> v2:172.21.15.117:3300/0 conn(0x7f2ea805cda0 0x7f2ea813a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:34.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/4243159131 >> v2:172.21.15.117:3300/0 conn(0x7f2ea805cda0 msgr2=0x7f2ea813a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:34.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 --2- 172.21.15.61:0/4243159131 >> v2:172.21.15.117:3300/0 conn(0x7f2ea805cda0 0x7f2ea813a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:34.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/4243159131 >> v1:172.21.15.61:6789/0 conn(0x7f2ea80b48d0 legacy=0x7f2ea80b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/4243159131 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea8066460 con 0x7f2ea8143f00 2022-07-20T03:07:34.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea6d9d700 1 --2- 172.21.15.61:0/4243159131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea8143f00 0x7f2ea813aa80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2ea00032f0 tx=0x7f2ea00091d0).ready entity=mon.0 client_cookie=3b42ec04e01f8888 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:34.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea559a700 1 -- 172.21.15.61:0/4243159131 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea0009ea0 con 0x7f2ea8143f00 2022-07-20T03:07:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea559a700 1 -- 172.21.15.61:0/4243159131 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea000b870 con 0x7f2ea8143f00 2022-07-20T03:07:34.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea559a700 1 -- 172.21.15.61:0/4243159131 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea000b9e0 con 0x7f2ea8143f00 2022-07-20T03:07:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- 172.21.15.61:0/4243159131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea8143f00 msgr2=0x7f2ea813aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 --2- 172.21.15.61:0/4243159131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea8143f00 0x7f2ea813aa80 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f2ea00032f0 tx=0x7f2ea00091d0).stop 2022-07-20T03:07:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- 172.21.15.61:0/4243159131 shutdown_connections 2022-07-20T03:07:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 --2- 172.21.15.61:0/4243159131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea8143f00 0x7f2ea813aa80 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 --2- 172.21.15.61:0/4243159131 >> v2:172.21.15.117:3300/0 conn(0x7f2ea805cda0 0x7f2ea813a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- 172.21.15.61:0/4243159131 >> 172.21.15.61:0/4243159131 conn(0x7f2ea8064b90 msgr2=0x7f2ea8064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:35.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- 172.21.15.61:0/4243159131 shutdown_connections 2022-07-20T03:07:35.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.771+0000 7f2ea759e700 1 -- 172.21.15.61:0/4243159131 wait complete. 2022-07-20T03:07:35.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 Processor -- start 2022-07-20T03:07:35.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 -- start start 2022-07-20T03:07:35.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ea805cda0 0x7f2ea805f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ea8143f00 0x7f2ea8142f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 0x7f2ea814f270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea80b06c0 con 0x7f2ea80b1170 2022-07-20T03:07:35.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea8061080 con 0x7f2ea805cda0 2022-07-20T03:07:35.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2ea81151b0 con 0x7f2ea8143f00 2022-07-20T03:07:35.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 0x7f2ea814f270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 0x7f2ea814f270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43046/0 (socket says 172.21.15.61:43046) 2022-07-20T03:07:35.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/508499085 learned_addr learned my addr 172.21.15.61:0/508499085 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:35.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea659c700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ea805cda0 0x7f2ea805f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea5d9b700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ea8143f00 0x7f2ea8142f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ea8143f00 msgr2=0x7f2ea8142f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ea8143f00 0x7f2ea8142f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ea805cda0 msgr2=0x7f2ea805f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ea805cda0 0x7f2ea805f840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 -- 172.21.15.61:0/508499085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2ea0009470 con 0x7f2ea80b1170 2022-07-20T03:07:35.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea6d9d700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 0x7f2ea814f270 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2ea00032f0 tx=0x7f2ea0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea0009ea0 con 0x7f2ea80b1170 2022-07-20T03:07:35.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2ea000b9e0 con 0x7f2ea80b1170 2022-07-20T03:07:35.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2ea8152320 con 0x7f2ea80b1170 2022-07-20T03:07:35.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2ea001f5e0 con 0x7f2ea80b1170 2022-07-20T03:07:35.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.775+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2ea8143420 con 0x7f2ea80b1170 2022-07-20T03:07:35.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.779+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2ea8060d80 con 0x7f2ea80b1170 2022-07-20T03:07:35.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.779+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2ea0026030 con 0x7f2ea80b1170 2022-07-20T03:07:35.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.779+0000 7f2e9f7fe700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2e80053a40 0x7f2e80055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.779+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2ea006c810 con 0x7f2ea80b1170 2022-07-20T03:07:35.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.779+0000 7f2ea659c700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2e80053a40 0x7f2e80055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.779+0000 7f2ea659c700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2e80053a40 0x7f2e80055f00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2e9800c000 tx=0x7f2e98008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.787+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2ea0043820 con 0x7f2ea80b1170 2022-07-20T03:07:35.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.911+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2ea81524b0 con 0x7f2ea80b1170 2022-07-20T03:07:35.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2e9f7fe700 1 -- 172.21.15.61:0/508499085 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2ea00150a0 con 0x7f2ea80b1170 2022-07-20T03:07:35.154 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2e80053a40 msgr2=0x7f2e80055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2e80053a40 0x7f2e80055f00 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f2e9800c000 tx=0x7f2e98008040).stop 2022-07-20T03:07:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 msgr2=0x7f2ea814f270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 0x7f2ea814f270 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f2ea00032f0 tx=0x7f2ea0016000).stop 2022-07-20T03:07:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 shutdown_connections 2022-07-20T03:07:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2e80053a40 0x7f2e80055f00 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2ea80b1170 0x7f2ea814f270 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2ea8143f00 0x7f2ea8142f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 --2- 172.21.15.61:0/508499085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2ea805cda0 0x7f2ea805f840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 >> 172.21.15.61:0/508499085 conn(0x7f2ea8064b90 msgr2=0x7f2ea813d0e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 shutdown_connections 2022-07-20T03:07:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:34.915+0000 7f2ea759e700 1 -- 172.21.15.61:0/508499085 wait complete. 2022-07-20T03:07:35.209 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:35.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 Processor -- start 2022-07-20T03:07:35.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- start start 2022-07-20T03:07:35.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f582409df40 0x7f5824145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58240b4d60 0x7f5824145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5824060ee0 0x7f5824146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5824060d20 con 0x7f58240b4d60 2022-07-20T03:07:35.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58240607e0 con 0x7f582409df40 2022-07-20T03:07:35.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5824060a80 con 0x7f5824060ee0 2022-07-20T03:07:35.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58240b4d60 0x7f5824145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58240b4d60 0x7f5824145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43052/0 (socket says 172.21.15.61:43052) 2022-07-20T03:07:35.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 -- 172.21.15.61:0/231462928 learned_addr learned my addr 172.21.15.61:0/231462928 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:35.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5822d9d700 1 --2- 172.21.15.61:0/231462928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5824060ee0 0x7f5824146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582259c700 1 --2- 172.21.15.61:0/231462928 >> v2:172.21.15.117:3300/0 conn(0x7f582409df40 0x7f5824145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 -- 172.21.15.61:0/231462928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5824060ee0 msgr2=0x7f5824146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:35.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 --2- 172.21.15.61:0/231462928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5824060ee0 0x7f5824146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 -- 172.21.15.61:0/231462928 >> v2:172.21.15.117:3300/0 conn(0x7f582409df40 msgr2=0x7f5824145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:35.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 --2- 172.21.15.61:0/231462928 >> v2:172.21.15.117:3300/0 conn(0x7f582409df40 0x7f5824145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 -- 172.21.15.61:0/231462928 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582409e320 con 0x7f58240b4d60 2022-07-20T03:07:35.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582259c700 1 --2- 172.21.15.61:0/231462928 >> v2:172.21.15.117:3300/0 conn(0x7f582409df40 0x7f5824145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:35.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f5821d9b700 1 --2- 172.21.15.61:0/231462928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58240b4d60 0x7f5824145b70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5818002f70 tx=0x7f5818006eb0).ready entity=mon.0 client_cookie=30da554ed8c4cb3f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582159a700 1 -- 172.21.15.61:0/231462928 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f581800d570 con 0x7f58240b4d60 2022-07-20T03:07:35.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582159a700 1 -- 172.21.15.61:0/231462928 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5818000e90 con 0x7f58240b4d60 2022-07-20T03:07:35.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582159a700 1 -- 172.21.15.61:0/231462928 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f581800d900 con 0x7f58240b4d60 2022-07-20T03:07:35.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- 172.21.15.61:0/231462928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58240b4d60 msgr2=0x7f5824145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- 172.21.15.61:0/231462928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58240b4d60 0x7f5824145b70 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f5818002f70 tx=0x7f5818006eb0).stop 2022-07-20T03:07:35.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- 172.21.15.61:0/231462928 shutdown_connections 2022-07-20T03:07:35.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- 172.21.15.61:0/231462928 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5824060ee0 0x7f5824146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- 172.21.15.61:0/231462928 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58240b4d60 0x7f5824145b70 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 --2- 172.21.15.61:0/231462928 >> v2:172.21.15.117:3300/0 conn(0x7f582409df40 0x7f5824145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- 172.21.15.61:0/231462928 >> 172.21.15.61:0/231462928 conn(0x7f58240b5f30 msgr2=0x7f58240b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:35.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.331+0000 7f582882a700 1 -- 172.21.15.61:0/231462928 shutdown_connections 2022-07-20T03:07:35.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- 172.21.15.61:0/231462928 wait complete. 2022-07-20T03:07:35.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 Processor -- start 2022-07-20T03:07:35.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- start start 2022-07-20T03:07:35.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5824060ee0 0x7f5824090e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 0x7f582409adb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58240b4d60 0x7f582409b2a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f582408f760 con 0x7f582409df40 2022-07-20T03:07:35.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5824152c10 con 0x7f5824060ee0 2022-07-20T03:07:35.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5824091710 con 0x7f58240b4d60 2022-07-20T03:07:35.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 0x7f582409adb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 0x7f582409adb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43058/0 (socket says 172.21.15.61:43058) 2022-07-20T03:07:35.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 -- 172.21.15.61:0/2192215341 learned_addr learned my addr 172.21.15.61:0/2192215341 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:35.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582259c700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5824060ee0 0x7f5824090e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5822d9d700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58240b4d60 0x7f582409b2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 -- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58240b4d60 msgr2=0x7f582409b2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58240b4d60 0x7f582409b2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 -- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5824060ee0 msgr2=0x7f5824090e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5824060ee0 0x7f5824090e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 -- 172.21.15.61:0/2192215341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f581800b1e0 con 0x7f582409df40 2022-07-20T03:07:35.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f5821d9b700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 0x7f582409adb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5818002f70 tx=0x7f5818015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5818006be0 con 0x7f582409df40 2022-07-20T03:07:35.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f58180068b0 con 0x7f582409df40 2022-07-20T03:07:35.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5824091470 con 0x7f582409df40 2022-07-20T03:07:35.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f581801e460 con 0x7f582409df40 2022-07-20T03:07:35.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.335+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f58240b0750 con 0x7f582409df40 2022-07-20T03:07:35.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.339+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f582409b930 con 0x7f582409df40 2022-07-20T03:07:35.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.339+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5818025030 con 0x7f582409df40 2022-07-20T03:07:35.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.339+0000 7f58137fe700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5800053a40 0x7f5800055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.339+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f581806d480 con 0x7f582409df40 2022-07-20T03:07:35.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.339+0000 7f582259c700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5800053a40 0x7f5800055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.339+0000 7f582259c700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5800053a40 0x7f5800055f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f581400c000 tx=0x7f5814008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.347+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58180442d0 con 0x7f582409df40 2022-07-20T03:07:35.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.475+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5824152d80 con 0x7f582409df40 2022-07-20T03:07:35.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.475+0000 7f58137fe700 1 -- 172.21.15.61:0/2192215341 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f58180478f0 con 0x7f582409df40 2022-07-20T03:07:35.494 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:35.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5800053a40 msgr2=0x7f5800055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5800053a40 0x7f5800055f00 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f581400c000 tx=0x7f5814008040).stop 2022-07-20T03:07:35.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 msgr2=0x7f582409adb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 0x7f582409adb0 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f5818002f70 tx=0x7f5818015000).stop 2022-07-20T03:07:35.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 shutdown_connections 2022-07-20T03:07:35.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5800053a40 0x7f5800055f00 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58240b4d60 0x7f582409b2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f582409df40 0x7f582409adb0 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 --2- 172.21.15.61:0/2192215341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5824060ee0 0x7f5824090e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 >> 172.21.15.61:0/2192215341 conn(0x7f58240b5f30 msgr2=0x7f58240b2d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:35.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 shutdown_connections 2022-07-20T03:07:35.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.479+0000 7f582882a700 1 -- 172.21.15.61:0/2192215341 wait complete. 2022-07-20T03:07:35.505 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:35.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 Processor -- start 2022-07-20T03:07:35.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 -- start start 2022-07-20T03:07:35.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f03b00ab0c0 0x7f03b0142ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b00ac840 con 0x7f03b00ac460 2022-07-20T03:07:35.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b0142fb0 con 0x7f03b00b4d70 2022-07-20T03:07:35.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b0143240 con 0x7f03b00ab0c0 2022-07-20T03:07:35.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03a6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f03b00ab0c0 0x7f03b0142ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03af7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f03b00ac460 0x7f03b013c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39662/0 (socket says 172.21.15.61:39662) 2022-07-20T03:07:35.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03af7fe700 1 -- 172.21.15.61:0/1735703292 learned_addr learned my addr 172.21.15.61:0/1735703292 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:35.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03aeffd700 1 -- 172.21.15.61:0/1735703292 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1460603752 0 0) 0x7f03b00ac840 con 0x7f03b00ac460 2022-07-20T03:07:35.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03aeffd700 1 -- 172.21.15.61:0/1735703292 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0390003260 con 0x7f03b00ac460 2022-07-20T03:07:35.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1735703292 >> v1:172.21.15.117:6789/0 conn(0x7f03b00b4d70 legacy=0x7f03b00b51c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1735703292 >> v1:172.21.15.61:6789/0 conn(0x7f03b00ac460 legacy=0x7f03b013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1735703292 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b01434e0 con 0x7f03b00ab0c0 2022-07-20T03:07:35.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03a6ffd700 1 --2- 172.21.15.61:0/1735703292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ab0c0 0x7f03b0142ac0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0398002f70 tx=0x7f0398006df0).ready entity=mon.0 client_cookie=6c6ef56b9faaab0b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03aeffd700 1 -- 172.21.15.61:0/1735703292 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039800d610 con 0x7f03b00ab0c0 2022-07-20T03:07:35.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03aeffd700 1 -- 172.21.15.61:0/1735703292 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0398000e90 con 0x7f03b00ab0c0 2022-07-20T03:07:35.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03aeffd700 1 -- 172.21.15.61:0/1735703292 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039800d960 con 0x7f03b00ab0c0 2022-07-20T03:07:35.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 -- 172.21.15.61:0/1735703292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ab0c0 msgr2=0x7f03b0142ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.655+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1735703292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ab0c0 0x7f03b0142ac0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0398002f70 tx=0x7f0398006df0).stop 2022-07-20T03:07:35.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- 172.21.15.61:0/1735703292 shutdown_connections 2022-07-20T03:07:35.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1735703292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ab0c0 0x7f03b0142ac0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- 172.21.15.61:0/1735703292 >> 172.21.15.61:0/1735703292 conn(0x7f03b00632b0 msgr2=0x7f03b0063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:35.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- 172.21.15.61:0/1735703292 shutdown_connections 2022-07-20T03:07:35.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- 172.21.15.61:0/1735703292 wait complete. 2022-07-20T03:07:35.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 Processor -- start 2022-07-20T03:07:35.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- start start 2022-07-20T03:07:35.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b00ab0c0 0x7f03b00acbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 0x7f03b0155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b00b4d70 0x7f03b00651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0156350 con 0x7f03b00ac460 2022-07-20T03:07:35.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b01560b0 con 0x7f03b00ab0c0 2022-07-20T03:07:35.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03b4c77700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b0064d00 con 0x7f03b00b4d70 2022-07-20T03:07:35.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03a6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 0x7f03b0155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03a6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 0x7f03b0155780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43072/0 (socket says 172.21.15.61:43072) 2022-07-20T03:07:35.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1450473152 learned_addr learned my addr 172.21.15.61:0/1450473152 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:35.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03af7fe700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b00ab0c0 0x7f03b00acbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.659+0000 7f03affff700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b00b4d70 0x7f03b00651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b00b4d70 msgr2=0x7f03b00651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03a6ffd700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b00b4d70 0x7f03b00651a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b00ab0c0 msgr2=0x7f03b00acbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03a6ffd700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b00ab0c0 0x7f03b00acbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03a6ffd700 1 -- 172.21.15.61:0/1450473152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f039800b180 con 0x7f03b00ac460 2022-07-20T03:07:35.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03a6ffd700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 0x7f03b0155780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0398002f70 tx=0x7f0398014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0398013de0 con 0x7f03b00ac460 2022-07-20T03:07:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f039800d290 con 0x7f03b00ac460 2022-07-20T03:07:35.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f039801e5f0 con 0x7f03b00ac460 2022-07-20T03:07:35.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b0065c10 con 0x7f03b00ac460 2022-07-20T03:07:35.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b00ad260 con 0x7f03b00ac460 2022-07-20T03:07:35.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b0143780 con 0x7f03b00ac460 2022-07-20T03:07:35.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0398025030 con 0x7f03b00ac460 2022-07-20T03:07:35.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03ad7fa700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0394053ad0 0x7f0394055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:35.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f039806d050 con 0x7f03b00ac460 2022-07-20T03:07:35.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.663+0000 7f03af7fe700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0394053ad0 0x7f0394055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:35.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.667+0000 7f03af7fe700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0394053ad0 0x7f0394055f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f03a000d000 tx=0x7f03a0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:35.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.671+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0398043fb0 con 0x7f03b00ac460 2022-07-20T03:07:35.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.795+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f03b00574e0 con 0x7f03b00ac460 2022-07-20T03:07:35.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03ad7fa700 1 -- 172.21.15.61:0/1450473152 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0398023030 con 0x7f03b00ac460 2022-07-20T03:07:35.890 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0394053ad0 msgr2=0x7f0394055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0394053ad0 0x7f0394055f90 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f03a000d000 tx=0x7f03a0009040).stop 2022-07-20T03:07:35.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 msgr2=0x7f03b0155780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:35.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 0x7f03b0155780 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0398002f70 tx=0x7f0398014000).stop 2022-07-20T03:07:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 shutdown_connections 2022-07-20T03:07:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0394053ad0 0x7f0394055f90 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b00b4d70 0x7f03b00651a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b00ac460 0x7f03b0155780 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 --2- 172.21.15.61:0/1450473152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b00ab0c0 0x7f03b00acbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 >> 172.21.15.61:0/1450473152 conn(0x7f03b00632b0 msgr2=0x7f03b00b6050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.799+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 shutdown_connections 2022-07-20T03:07:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:35.803+0000 7f03b4c77700 1 -- 172.21.15.61:0/1450473152 wait complete. 2022-07-20T03:07:35.913 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:07:35.916 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:36.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.043+0000 7f88c34c7700 1 Processor -- start 2022-07-20T03:07:36.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.043+0000 7f88c34c7700 1 -- start start 2022-07-20T03:07:36.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f88bc0acef0 0x7f88bc14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:36.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc05d480 con 0x7f88bc05c3f0 2022-07-20T03:07:36.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f88bc0b21e0 con 0x7f88bc0b1e00 2022-07-20T03:07:36.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88bc05cf40 con 0x7f88bc0acef0 2022-07-20T03:07:36.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c0a62700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f88bc05c3f0 0x7f88bc146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39676/0 (socket says 172.21.15.61:39676) 2022-07-20T03:07:36.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c1a64700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f88bc0acef0 0x7f88bc14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:36.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c0a62700 1 -- 172.21.15.61:0/3212511765 learned_addr learned my addr 172.21.15.61:0/3212511765 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:36.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88b3fff700 1 -- 172.21.15.61:0/3212511765 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709843850 0 0) 0x7f88bc05d480 con 0x7f88bc05c3f0 2022-07-20T03:07:36.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88b3fff700 1 -- 172.21.15.61:0/3212511765 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f88a0003260 con 0x7f88bc05c3f0 2022-07-20T03:07:36.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c1a64700 1 -- 172.21.15.61:0/3212511765 >> v1:172.21.15.117:6789/0 conn(0x7f88bc0b1e00 legacy=0x7f88bc0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c1a64700 1 -- 172.21.15.61:0/3212511765 >> v1:172.21.15.61:6789/0 conn(0x7f88bc05c3f0 legacy=0x7f88bc146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c1a64700 1 -- 172.21.15.61:0/3212511765 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88bc05d1e0 con 0x7f88bc0acef0 2022-07-20T03:07:36.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c1a64700 1 --2- 172.21.15.61:0/3212511765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0acef0 0x7f88bc14a6d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f88b40032f0 tx=0x7f88b40091d0).ready entity=mon.0 client_cookie=459cd7fde1de75c7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:36.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88b3fff700 1 -- 172.21.15.61:0/3212511765 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400b6c0 con 0x7f88bc0acef0 2022-07-20T03:07:36.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88b3fff700 1 -- 172.21.15.61:0/3212511765 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b400b820 con 0x7f88bc0acef0 2022-07-20T03:07:36.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88b3fff700 1 -- 172.21.15.61:0/3212511765 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b400b990 con 0x7f88bc0acef0 2022-07-20T03:07:36.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- 172.21.15.61:0/3212511765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0acef0 msgr2=0x7f88bc14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 --2- 172.21.15.61:0/3212511765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0acef0 0x7f88bc14a6d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f88b40032f0 tx=0x7f88b40091d0).stop 2022-07-20T03:07:36.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- 172.21.15.61:0/3212511765 shutdown_connections 2022-07-20T03:07:36.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 --2- 172.21.15.61:0/3212511765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0acef0 0x7f88bc14a6d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- 172.21.15.61:0/3212511765 >> 172.21.15.61:0/3212511765 conn(0x7f88bc0611c0 msgr2=0x7f88bc062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:36.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- 172.21.15.61:0/3212511765 shutdown_connections 2022-07-20T03:07:36.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.047+0000 7f88c34c7700 1 -- 172.21.15.61:0/3212511765 wait complete. 2022-07-20T03:07:36.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 Processor -- start 2022-07-20T03:07:36.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 -- start start 2022-07-20T03:07:36.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88bc0acef0 0x7f88bc146d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:36.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 0x7f88bc09ef40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:36.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88bc14a080 0x7f88bc148910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:36.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc149770 con 0x7f88bc0b1e00 2022-07-20T03:07:36.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc149ea0 con 0x7f88bc0acef0 2022-07-20T03:07:36.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f88bc149960 con 0x7f88bc14a080 2022-07-20T03:07:36.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 0x7f88bc09ef40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 0x7f88bc09ef40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43086/0 (socket says 172.21.15.61:43086) 2022-07-20T03:07:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 -- 172.21.15.61:0/2487805503 learned_addr learned my addr 172.21.15.61:0/2487805503 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c1263700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88bc0acef0 0x7f88bc146d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c1a64700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88bc14a080 0x7f88bc148910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 -- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88bc14a080 msgr2=0x7f88bc148910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88bc14a080 0x7f88bc148910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 -- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88bc0acef0 msgr2=0x7f88bc146d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88bc0acef0 0x7f88bc146d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 -- 172.21.15.61:0/2487805503 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88b4009470 con 0x7f88bc0b1e00 2022-07-20T03:07:36.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c0a62700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 0x7f88bc09ef40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f88b8001110 tx=0x7f88b8007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:36.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b80070d0 con 0x7f88bc0b1e00 2022-07-20T03:07:36.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88b8007230 con 0x7f88bc0b1e00 2022-07-20T03:07:36.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88bc149c00 con 0x7f88bc0b1e00 2022-07-20T03:07:36.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f88b8005610 con 0x7f88bc0b1e00 2022-07-20T03:07:36.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.051+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88bc0a0860 con 0x7f88bc0b1e00 2022-07-20T03:07:36.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.055+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88bc0b04b0 con 0x7f88bc0b1e00 2022-07-20T03:07:36.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.055+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f88b801a060 con 0x7f88bc0b1e00 2022-07-20T03:07:36.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.055+0000 7f88b27fc700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f88a40539b0 0x7f88a4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:36.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.055+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f88b806b240 con 0x7f88bc0b1e00 2022-07-20T03:07:36.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.055+0000 7f88c1263700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f88a40539b0 0x7f88a4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:36.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.055+0000 7f88c1263700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f88a40539b0 0x7f88a4055e70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f88ac00d000 tx=0x7f88ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:36.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.063+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f88b8042090 con 0x7f88bc0b1e00 2022-07-20T03:07:36.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.231+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f88bc05c8e0 con 0x7f88bc0b1e00 2022-07-20T03:07:36.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.231+0000 7f88b27fc700 1 -- 172.21.15.61:0/2487805503 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2134 (secure 0 0 0) 0x7f88b8018020 con 0x7f88bc0b1e00 2022-07-20T03:07:36.247 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:36.248 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:12.741818+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.117:6835/725875269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":725875269},{"type":"v1","addr":"172.21.15.117:6835","nonce":725875269}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f88a40539b0 msgr2=0x7f88a4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f88a40539b0 0x7f88a4055e70 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f88ac00d000 tx=0x7f88ac009040).stop 2022-07-20T03:07:36.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 msgr2=0x7f88bc09ef40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:36.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 0x7f88bc09ef40 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f88b8001110 tx=0x7f88b8007aa0).stop 2022-07-20T03:07:36.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 shutdown_connections 2022-07-20T03:07:36.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f88a40539b0 0x7f88a4055e70 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88bc14a080 0x7f88bc148910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88bc0b1e00 0x7f88bc09ef40 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 --2- 172.21.15.61:0/2487805503 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88bc0acef0 0x7f88bc146d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:36.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 >> 172.21.15.61:0/2487805503 conn(0x7f88bc05c3f0 msgr2=0x7f88bc143a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:36.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 shutdown_connections 2022-07-20T03:07:36.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:36.235+0000 7f88c34c7700 1 -- 172.21.15.61:0/2487805503 wait complete. 2022-07-20T03:07:36.281 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2022-07-20T03:07:36.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 7, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:07:10.718708+0000', 'modified': '2022-07-20T03:07:12.741818+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4273}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4273': {'gid': 4273, 'name': 'b', 'rank': 0, 'incarnation': 4, 'state': 'up:active', 'state_seq': 2, 'addr': '172.21.15.117:6835/725875269', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 725875269}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 725875269}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [4], 'metadata_pool': 3, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:07:36.504 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:07:36.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:07:36.645 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":4273,"want_state":"up:active","state":"up:active","rank_uptime":25.895886004000001,"mdsmap_epoch":7,"osdmap_epoch":23,"osdmap_epoch_barrier":23,"uptime":29.438117199000001} 2022-07-20T03:07:36.646 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 4273, "mdsmap_epoch": 7, "osdmap_epoch": 23, "osdmap_epoch_barrier": 23, "rank_uptime": 25.895886004, "state": "up:active", "uptime": 29.438117199, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:07:36.646 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:07:36.646 INFO:tasks.cephfs.fuse_mount:Client client.2 config is {} 2022-07-20T03:07:36.647 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.2 at ubuntu@smithi179.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.2... 2022-07-20T03:07:36.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:36.648 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:36.740 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe fuse 2022-07-20T03:07:36.815 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:07:36.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:07:36.875 INFO:teuthology.orchestra.run.smithi179.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:07:36.877 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:07:36.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:36.880 DEBUG:teuthology.orchestra.run.smithi179:> ls /sys/fs/fuse/connections 2022-07-20T03:07:36.909 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:07:36.910 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.2 /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:36.992 DEBUG:teuthology.orchestra.run.smithi179:> sudo modprobe fuse 2022-07-20T03:07:37.044 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:07:37.045 DEBUG:teuthology.orchestra.run.smithi179:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:07:37.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi179.stderr:2022-07-20T03:07:37.072+0000 7fa5735a6080 -1 init, newargv = 0x560857be7270 newargc=9 2022-07-20T03:07:37.076 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi179.stderr:ceph-fuse[22199]: starting ceph client 2022-07-20T03:07:37.085 INFO:tasks.cephfs.fuse_mount.ceph-fuse.2.smithi179.stderr:ceph-fuse[22199]: starting fuse 2022-07-20T03:07:37.100 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:07:37.104 INFO:teuthology.orchestra.run.smithi179.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:07:37.110 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.131 DEBUG:teuthology.orchestra.run.smithi179:> ls /sys/fs/fuse/connections 2022-07-20T03:07:37.168 INFO:teuthology.orchestra.run.smithi179.stdout:52 2022-07-20T03:07:37.169 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:07:37.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.176 DEBUG:teuthology.orchestra.run.smithi179:> sudo python3 -c ' 2022-07-20T03:07:37.179 DEBUG:teuthology.orchestra.run.smithi179:> import glob 2022-07-20T03:07:37.180 DEBUG:teuthology.orchestra.run.smithi179:> import re 2022-07-20T03:07:37.181 DEBUG:teuthology.orchestra.run.smithi179:> import os 2022-07-20T03:07:37.181 DEBUG:teuthology.orchestra.run.smithi179:> import subprocess 2022-07-20T03:07:37.182 DEBUG:teuthology.orchestra.run.smithi179:> 2022-07-20T03:07:37.182 DEBUG:teuthology.orchestra.run.smithi179:> def find_socket(client_name): 2022-07-20T03:07:37.183 DEBUG:teuthology.orchestra.run.smithi179:> asok_path = "/var/run/ceph/ceph-client.2.*.asok" 2022-07-20T03:07:37.184 DEBUG:teuthology.orchestra.run.smithi179:> files = glob.glob(asok_path) 2022-07-20T03:07:37.185 DEBUG:teuthology.orchestra.run.smithi179:> 2022-07-20T03:07:37.187 DEBUG:teuthology.orchestra.run.smithi179:> # Given a non-glob path, it better be there 2022-07-20T03:07:37.187 DEBUG:teuthology.orchestra.run.smithi179:> if "*" not in asok_path: 2022-07-20T03:07:37.189 DEBUG:teuthology.orchestra.run.smithi179:> assert(len(files) == 1) 2022-07-20T03:07:37.190 DEBUG:teuthology.orchestra.run.smithi179:> return files[0] 2022-07-20T03:07:37.197 DEBUG:teuthology.orchestra.run.smithi179:> 2022-07-20T03:07:37.198 DEBUG:teuthology.orchestra.run.smithi179:> for f in files: 2022-07-20T03:07:37.198 DEBUG:teuthology.orchestra.run.smithi179:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:07:37.198 DEBUG:teuthology.orchestra.run.smithi179:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:07:37.199 DEBUG:teuthology.orchestra.run.smithi179:> return f 2022-07-20T03:07:37.199 DEBUG:teuthology.orchestra.run.smithi179:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:07:37.199 DEBUG:teuthology.orchestra.run.smithi179:> 2022-07-20T03:07:37.200 DEBUG:teuthology.orchestra.run.smithi179:> print(find_socket("client.2")) 2022-07-20T03:07:37.200 DEBUG:teuthology.orchestra.run.smithi179:> ' 2022-07-20T03:07:37.285 INFO:teuthology.orchestra.run.smithi179.stdout:/var/run/ceph/ceph-client.2.22199.asok 2022-07-20T03:07:37.285 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.2.22199.asok 2022-07-20T03:07:37.286 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.287 DEBUG:teuthology.orchestra.run.smithi179:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.2.22199.asok status 2022-07-20T03:07:37.437 INFO:teuthology.orchestra.run.smithi179.stdout:{ 2022-07-20T03:07:37.439 INFO:teuthology.orchestra.run.smithi179.stdout: "metadata": { 2022-07-20T03:07:37.439 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:07:37.440 INFO:teuthology.orchestra.run.smithi179.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:07:37.440 INFO:teuthology.orchestra.run.smithi179.stdout: "entity_id": "2", 2022-07-20T03:07:37.441 INFO:teuthology.orchestra.run.smithi179.stdout: "hostname": "smithi179", 2022-07-20T03:07:37.441 INFO:teuthology.orchestra.run.smithi179.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.2", 2022-07-20T03:07:37.442 INFO:teuthology.orchestra.run.smithi179.stdout: "pid": "22199", 2022-07-20T03:07:37.443 INFO:teuthology.orchestra.run.smithi179.stdout: "root": "/" 2022-07-20T03:07:37.443 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2022-07-20T03:07:37.444 INFO:teuthology.orchestra.run.smithi179.stdout: "dentry_count": 0, 2022-07-20T03:07:37.445 INFO:teuthology.orchestra.run.smithi179.stdout: "dentry_pinned_count": 0, 2022-07-20T03:07:37.445 INFO:teuthology.orchestra.run.smithi179.stdout: "id": 4698, 2022-07-20T03:07:37.446 INFO:teuthology.orchestra.run.smithi179.stdout: "inst": { 2022-07-20T03:07:37.468 INFO:teuthology.orchestra.run.smithi179.stdout: "name": { 2022-07-20T03:07:37.469 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "client", 2022-07-20T03:07:37.470 INFO:teuthology.orchestra.run.smithi179.stdout: "num": 4698 2022-07-20T03:07:37.470 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2022-07-20T03:07:37.471 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": { 2022-07-20T03:07:37.485 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2022-07-20T03:07:37.485 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:0", 2022-07-20T03:07:37.485 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 799522106 2022-07-20T03:07:37.485 INFO:teuthology.orchestra.run.smithi179.stdout: } 2022-07-20T03:07:37.486 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2022-07-20T03:07:37.486 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": { 2022-07-20T03:07:37.486 INFO:teuthology.orchestra.run.smithi179.stdout: "type": "v1", 2022-07-20T03:07:37.486 INFO:teuthology.orchestra.run.smithi179.stdout: "addr": "172.21.15.179:0", 2022-07-20T03:07:37.487 INFO:teuthology.orchestra.run.smithi179.stdout: "nonce": 799522106 2022-07-20T03:07:37.487 INFO:teuthology.orchestra.run.smithi179.stdout: }, 2022-07-20T03:07:37.487 INFO:teuthology.orchestra.run.smithi179.stdout: "inst_str": "client.4698 172.21.15.179:0/799522106", 2022-07-20T03:07:37.487 INFO:teuthology.orchestra.run.smithi179.stdout: "addr_str": "172.21.15.179:0/799522106", 2022-07-20T03:07:37.488 INFO:teuthology.orchestra.run.smithi179.stdout: "inode_count": 1, 2022-07-20T03:07:37.488 INFO:teuthology.orchestra.run.smithi179.stdout: "mds_epoch": 7, 2022-07-20T03:07:37.488 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_epoch": 23, 2022-07-20T03:07:37.488 INFO:teuthology.orchestra.run.smithi179.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:07:37.489 INFO:teuthology.orchestra.run.smithi179.stdout: "blacklisted": false, 2022-07-20T03:07:37.489 INFO:teuthology.orchestra.run.smithi179.stdout: "fs_name": "cephfs" 2022-07-20T03:07:37.489 INFO:teuthology.orchestra.run.smithi179.stdout:} 2022-07-20T03:07:37.495 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.496 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:37.496 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:37.533 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:07:37.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:07:37.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.541 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:37.601 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.602 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:37.602 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:37.634 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:07:37.637 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:07:37.638 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.639 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:37.704 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.704 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:37.705 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:37.719 INFO:teuthology.orchestra.run.smithi179.stdout:fuseblk 2022-07-20T03:07:37.719 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-07-20T03:07:37.720 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:37.720 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:37.807 INFO:teuthology.run_tasks:Running task cephfs_test_runner... 2022-07-20T03:07:37.904 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:38.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.015+0000 7effb3162700 1 Processor -- start 2022-07-20T03:07:38.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.015+0000 7effb3162700 1 -- start start 2022-07-20T03:07:38.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7effac0b3de0 0x7effac14d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac0b42a0 con 0x7effac0afa10 2022-07-20T03:07:38.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7effac060c90 con 0x7effac0608b0 2022-07-20T03:07:38.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7effac0af410 con 0x7effac0b3de0 2022-07-20T03:07:38.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effabfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7effac0afa10 0x7effac0b46d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39692/0 (socket says 172.21.15.61:39692) 2022-07-20T03:07:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effabfff700 1 -- 172.21.15.61:0/3407885889 learned_addr learned my addr 172.21.15.61:0/3407885889 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb16ff700 1 --2- 172.21.15.61:0/3407885889 >> v2:172.21.15.117:3300/0 conn(0x7effac0b3de0 0x7effac14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 809758997 0 0) 0x7effac0b42a0 con 0x7effac0afa10 2022-07-20T03:07:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff8c003260 con 0x7effac0afa10 2022-07-20T03:07:38.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 216388081 0 0) 0x7effac060c90 con 0x7effac0608b0 2022-07-20T03:07:38.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7effac0b42a0 con 0x7effac0608b0 2022-07-20T03:07:38.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb16ff700 1 -- 172.21.15.61:0/3407885889 >> v1:172.21.15.117:6789/0 conn(0x7effac0608b0 legacy=0x7effac05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb16ff700 1 -- 172.21.15.61:0/3407885889 >> v1:172.21.15.61:6789/0 conn(0x7effac0afa10 legacy=0x7effac0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb16ff700 1 -- 172.21.15.61:0/3407885889 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effac0af6b0 con 0x7effac0b3de0 2022-07-20T03:07:38.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb16ff700 1 --2- 172.21.15.61:0/3407885889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac14d920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7effa40032f0 tx=0x7effa40091d0).ready entity=mon.1 client_cookie=31c074715d3873a4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effa400b600 con 0x7effac0b3de0 2022-07-20T03:07:38.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7effa400b760 con 0x7effac0b3de0 2022-07-20T03:07:38.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effab7fe700 1 -- 172.21.15.61:0/3407885889 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7effa400b8d0 con 0x7effac0b3de0 2022-07-20T03:07:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- 172.21.15.61:0/3407885889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 msgr2=0x7effac14d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 --2- 172.21.15.61:0/3407885889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac14d920 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7effa40032f0 tx=0x7effa40091d0).stop 2022-07-20T03:07:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- 172.21.15.61:0/3407885889 shutdown_connections 2022-07-20T03:07:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 --2- 172.21.15.61:0/3407885889 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac14d920 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- 172.21.15.61:0/3407885889 >> 172.21.15.61:0/3407885889 conn(0x7effac0b5ee0 msgr2=0x7effac0b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- 172.21.15.61:0/3407885889 shutdown_connections 2022-07-20T03:07:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.019+0000 7effb3162700 1 -- 172.21.15.61:0/3407885889 wait complete. 2022-07-20T03:07:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 Processor -- start 2022-07-20T03:07:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 -- start start 2022-07-20T03:07:38.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 0x7effac143780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7effac0afa10 0x7effac09cd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac08fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac1481b0 con 0x7effac0608b0 2022-07-20T03:07:38.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7effac149500 con 0x7effac0b3de0 2022-07-20T03:07:38.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7effac1483e0 con 0x7effac0afa10 2022-07-20T03:07:38.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 0x7effac143780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 0x7effac143780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43098/0 (socket says 172.21.15.61:43098) 2022-07-20T03:07:38.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 -- 172.21.15.61:0/154762902 learned_addr learned my addr 172.21.15.61:0/154762902 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effabfff700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7effac0afa10 0x7effac09cd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb16ff700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac08fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 -- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7effac0afa10 msgr2=0x7effac09cd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7effac0afa10 0x7effac09cd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 -- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 msgr2=0x7effac08fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac08fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 -- 172.21.15.61:0/154762902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7effa4009470 con 0x7effac0608b0 2022-07-20T03:07:38.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effabfff700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7effac0afa10 0x7effac09cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:38.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb0efe700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 0x7effac143780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7eff9c001110 tx=0x7eff9c007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff9c012400 con 0x7effac0608b0 2022-07-20T03:07:38.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff9c012560 con 0x7effac0608b0 2022-07-20T03:07:38.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7effac08f790 con 0x7effac0608b0 2022-07-20T03:07:38.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff9c0128c0 con 0x7effac0608b0 2022-07-20T03:07:38.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.023+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7effac1476e0 con 0x7effac0608b0 2022-07-20T03:07:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.027+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7effac09e4d0 con 0x7effac0608b0 2022-07-20T03:07:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.027+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7eff9c019030 con 0x7effac0608b0 2022-07-20T03:07:38.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.027+0000 7effa9ffb700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff90053ad0 0x7eff90055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.027+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7eff9c06b300 con 0x7effac0608b0 2022-07-20T03:07:38.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.027+0000 7effabfff700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff90053ad0 0x7eff90055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.027+0000 7effabfff700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff90053ad0 0x7eff90055f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7eff9400d000 tx=0x7eff94009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.035+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7effac09e4d0 con 0x7effac0608b0 2022-07-20T03:07:38.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.167+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7effac09e4d0 con 0x7effac0608b0 2022-07-20T03:07:38.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.167+0000 7effa9ffb700 1 -- 172.21.15.61:0/154762902 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7effac09e4d0 con 0x7effac0608b0 2022-07-20T03:07:38.175 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:38.175 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:38.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff90053ad0 msgr2=0x7eff90055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff90053ad0 0x7eff90055f90 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7eff9400d000 tx=0x7eff94009040).stop 2022-07-20T03:07:38.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 msgr2=0x7effac143780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 0x7effac143780 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7eff9c001110 tx=0x7eff9c007ae0).stop 2022-07-20T03:07:38.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 shutdown_connections 2022-07-20T03:07:38.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff90053ad0 0x7eff90055f90 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7effac0b3de0 0x7effac08fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7effac0afa10 0x7effac09cd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 --2- 172.21.15.61:0/154762902 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7effac0608b0 0x7effac143780 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 >> 172.21.15.61:0/154762902 conn(0x7effac0b5ee0 msgr2=0x7effac145f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 shutdown_connections 2022-07-20T03:07:38.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.171+0000 7effb3162700 1 -- 172.21.15.61:0/154762902 wait complete. 2022-07-20T03:07:38.192 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:38.193 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:38.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.323+0000 7fa8cce3e700 1 Processor -- start 2022-07-20T03:07:38.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- start start 2022-07-20T03:07:38.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c814b590 0x7fa8c814d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c80ba180 con 0x7fa8c80b3cf0 2022-07-20T03:07:38.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8c81461b0 con 0x7fa8c80b9da0 2022-07-20T03:07:38.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c80ad5b0 con 0x7fa8c814b590 2022-07-20T03:07:38.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa8c80b3cf0 0x7fa8c8142a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39706/0 (socket says 172.21.15.61:39706) 2022-07-20T03:07:38.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c6ffd700 1 -- 172.21.15.61:0/2832545068 learned_addr learned my addr 172.21.15.61:0/2832545068 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:38.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c7fff700 1 --2- 172.21.15.61:0/2832545068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c814b590 0x7fa8c814d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c67fc700 1 -- 172.21.15.61:0/2832545068 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4128516748 0 0) 0x7fa8c80ba180 con 0x7fa8c80b3cf0 2022-07-20T03:07:38.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c67fc700 1 -- 172.21.15.61:0/2832545068 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa8a8003260 con 0x7fa8c80b3cf0 2022-07-20T03:07:38.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c7fff700 1 -- 172.21.15.61:0/2832545068 >> v1:172.21.15.117:6789/0 conn(0x7fa8c80b9da0 legacy=0x7fa8c81468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c7fff700 1 -- 172.21.15.61:0/2832545068 >> v1:172.21.15.61:6789/0 conn(0x7fa8c80b3cf0 legacy=0x7fa8c8142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c7fff700 1 -- 172.21.15.61:0/2832545068 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c8146450 con 0x7fa8c814b590 2022-07-20T03:07:38.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c7fff700 1 --2- 172.21.15.61:0/2832545068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c814b590 0x7fa8c814d980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa8c00032f0 tx=0x7fa8c00091d0).ready entity=mon.2 client_cookie=d7a3fa69be8e0e53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c67fc700 1 -- 172.21.15.61:0/2832545068 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c000b6c0 con 0x7fa8c814b590 2022-07-20T03:07:38.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c67fc700 1 -- 172.21.15.61:0/2832545068 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c000b820 con 0x7fa8c814b590 2022-07-20T03:07:38.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8c67fc700 1 -- 172.21.15.61:0/2832545068 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c000b990 con 0x7fa8c814b590 2022-07-20T03:07:38.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- 172.21.15.61:0/2832545068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c814b590 msgr2=0x7fa8c814d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/2832545068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c814b590 0x7fa8c814d980 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fa8c00032f0 tx=0x7fa8c00091d0).stop 2022-07-20T03:07:38.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- 172.21.15.61:0/2832545068 shutdown_connections 2022-07-20T03:07:38.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/2832545068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c814b590 0x7fa8c814d980 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.327+0000 7fa8cce3e700 1 -- 172.21.15.61:0/2832545068 >> 172.21.15.61:0/2832545068 conn(0x7fa8c80678c0 msgr2=0x7fa8c80b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- 172.21.15.61:0/2832545068 shutdown_connections 2022-07-20T03:07:38.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- 172.21.15.61:0/2832545068 wait complete. 2022-07-20T03:07:38.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 Processor -- start 2022-07-20T03:07:38.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- start start 2022-07-20T03:07:38.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c80b3cf0 0x7fa8c813f5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c80b9da0 0x7fa8c8144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 0x7fa8c8144760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8137890 con 0x7fa8c814b590 2022-07-20T03:07:38.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c8137350 con 0x7fa8c80b3cf0 2022-07-20T03:07:38.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa8c81375f0 con 0x7fa8c80b9da0 2022-07-20T03:07:38.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 0x7fa8c8144760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 0x7fa8c8144760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43116/0 (socket says 172.21.15.61:43116) 2022-07-20T03:07:38.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 -- 172.21.15.61:0/867610172 learned_addr learned my addr 172.21.15.61:0/867610172 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:38.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c77fe700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c80b3cf0 0x7fa8c813f5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c6ffd700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c80b9da0 0x7fa8c8144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 -- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c80b9da0 msgr2=0x7fa8c8144270 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c80b9da0 0x7fa8c8144270 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 -- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c80b3cf0 msgr2=0x7fa8c813f5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c80b3cf0 0x7fa8c813f5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 -- 172.21.15.61:0/867610172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa8c0009470 con 0x7fa8c814b590 2022-07-20T03:07:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c7fff700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 0x7fa8c8144760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa8c00032f0 tx=0x7fa8c0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c000b990 con 0x7fa8c814b590 2022-07-20T03:07:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8c000b030 con 0x7fa8c814b590 2022-07-20T03:07:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8c813de70 con 0x7fa8c814b590 2022-07-20T03:07:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.331+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa8c000b6c0 con 0x7fa8c814b590 2022-07-20T03:07:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa8c81359a0 con 0x7fa8c814b590 2022-07-20T03:07:38.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8c813cab0 con 0x7fa8c814b590 2022-07-20T03:07:38.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa8c001e070 con 0x7fa8c814b590 2022-07-20T03:07:38.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8c4ff9700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8ac053ad0 0x7fa8ac055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa8c006ce60 con 0x7fa8c814b590 2022-07-20T03:07:38.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8c77fe700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8ac053ad0 0x7fa8ac055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.335+0000 7fa8c77fe700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8ac053ad0 0x7fa8ac055f90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa8b000b000 tx=0x7fa8b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.347+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa8c0043cc0 con 0x7fa8c814b590 2022-07-20T03:07:38.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa8c80574e0 con 0x7fa8c814b590 2022-07-20T03:07:38.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8c4ff9700 1 -- 172.21.15.61:0/867610172 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa8c0015020 con 0x7fa8c814b590 2022-07-20T03:07:38.493 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:38.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8ac053ad0 msgr2=0x7fa8ac055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8ac053ad0 0x7fa8ac055f90 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa8b000b000 tx=0x7fa8b0009040).stop 2022-07-20T03:07:38.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 msgr2=0x7fa8c8144760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 0x7fa8c8144760 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fa8c00032f0 tx=0x7fa8c0016000).stop 2022-07-20T03:07:38.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 shutdown_connections 2022-07-20T03:07:38.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8ac053ad0 0x7fa8ac055f90 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa8c814b590 0x7fa8c8144760 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8c80b9da0 0x7fa8c8144270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 --2- 172.21.15.61:0/867610172 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8c80b3cf0 0x7fa8c813f5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 >> 172.21.15.61:0/867610172 conn(0x7fa8c80678c0 msgr2=0x7fa8c80b5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 shutdown_connections 2022-07-20T03:07:38.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.475+0000 7fa8cce3e700 1 -- 172.21.15.61:0/867610172 wait complete. 2022-07-20T03:07:38.501 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:38.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.623+0000 7f43a695f700 1 Processor -- start 2022-07-20T03:07:38.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.623+0000 7f43a695f700 1 -- start start 2022-07-20T03:07:38.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43a01462a0 0x7f43a0146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a00afdb0 con 0x7f43a00988b0 2022-07-20T03:07:38.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f43a00b0050 con 0x7f43a0146b70 2022-07-20T03:07:38.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43a005fcd0 con 0x7f43a01462a0 2022-07-20T03:07:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43a01462a0 0x7f43a0146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f43a00988b0 0x7f43a01422c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39720/0 (socket says 172.21.15.61:39720) 2022-07-20T03:07:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439f7fe700 1 -- 172.21.15.61:0/1707898717 learned_addr learned my addr 172.21.15.61:0/1707898717 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439ffff700 1 -- 172.21.15.61:0/1707898717 >> v1:172.21.15.117:6789/0 conn(0x7f43a0146b70 legacy=0x7f43a0060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439ffff700 1 -- 172.21.15.61:0/1707898717 >> v1:172.21.15.61:6789/0 conn(0x7f43a00988b0 legacy=0x7f43a01422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439ffff700 1 -- 172.21.15.61:0/1707898717 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43a00afae0 con 0x7f43a01462a0 2022-07-20T03:07:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439ffff700 1 --2- 172.21.15.61:0/1707898717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a0146680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4394002f70 tx=0x7f4394006df0).ready entity=mon.0 client_cookie=940c22508113802c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439effd700 1 -- 172.21.15.61:0/1707898717 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f439400d610 con 0x7f43a01462a0 2022-07-20T03:07:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439effd700 1 -- 172.21.15.61:0/1707898717 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4394000e90 con 0x7f43a01462a0 2022-07-20T03:07:38.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f439effd700 1 -- 172.21.15.61:0/1707898717 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f439400d960 con 0x7f43a01462a0 2022-07-20T03:07:38.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- 172.21.15.61:0/1707898717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 msgr2=0x7f43a0146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 --2- 172.21.15.61:0/1707898717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a0146680 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4394002f70 tx=0x7f4394006df0).stop 2022-07-20T03:07:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- 172.21.15.61:0/1707898717 shutdown_connections 2022-07-20T03:07:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 --2- 172.21.15.61:0/1707898717 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a0146680 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- 172.21.15.61:0/1707898717 >> 172.21.15.61:0/1707898717 conn(0x7f43a00b5f00 msgr2=0x7f43a00b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- 172.21.15.61:0/1707898717 shutdown_connections 2022-07-20T03:07:38.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.627+0000 7f43a695f700 1 -- 172.21.15.61:0/1707898717 wait complete. 2022-07-20T03:07:38.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 Processor -- start 2022-07-20T03:07:38.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 -- start start 2022-07-20T03:07:38.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43a00988b0 0x7f43a009d840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a009dd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a0146b70 0x7f43a009b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a009d150 con 0x7f43a01462a0 2022-07-20T03:07:38.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43a00b3200 con 0x7f43a0146b70 2022-07-20T03:07:38.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.631+0000 7f43a695f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43a00b2530 con 0x7f43a00988b0 2022-07-20T03:07:38.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a009dd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a009dd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43128/0 (socket says 172.21.15.61:43128) 2022-07-20T03:07:38.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 -- 172.21.15.61:0/1196668655 learned_addr learned my addr 172.21.15.61:0/1196668655 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:38.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439ffff700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43a00988b0 0x7f43a009d840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f43a4efc700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a0146b70 0x7f43a009b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 -- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43a00988b0 msgr2=0x7f43a009d840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43a00988b0 0x7f43a009d840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 -- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a0146b70 msgr2=0x7f43a009b300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a0146b70 0x7f43a009b300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 -- 172.21.15.61:0/1196668655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f439400b180 con 0x7f43a01462a0 2022-07-20T03:07:38.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439f7fe700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a009dd30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f43a0066650 tx=0x7f4390007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f43900070d0 con 0x7f43a01462a0 2022-07-20T03:07:38.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4390007230 con 0x7f43a01462a0 2022-07-20T03:07:38.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43a009c490 con 0x7f43a01462a0 2022-07-20T03:07:38.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4390005610 con 0x7f43a01462a0 2022-07-20T03:07:38.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43a0099b90 con 0x7f43a01462a0 2022-07-20T03:07:38.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.635+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43a00574e0 con 0x7f43a01462a0 2022-07-20T03:07:38.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.639+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4390010070 con 0x7f43a01462a0 2022-07-20T03:07:38.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.639+0000 7f439d7fa700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4380053a40 0x7f4380055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.639+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f439006b390 con 0x7f43a01462a0 2022-07-20T03:07:38.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.639+0000 7f439ffff700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4380053a40 0x7f4380055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.639+0000 7f439ffff700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4380053a40 0x7f4380055f00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f439401c000 tx=0x7f4394015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.643+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43900422f0 con 0x7f43a01462a0 2022-07-20T03:07:38.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.771+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f43a0067ea0 con 0x7f43a01462a0 2022-07-20T03:07:38.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.771+0000 7f439d7fa700 1 -- 172.21.15.61:0/1196668655 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4390045910 con 0x7f43a01462a0 2022-07-20T03:07:38.776 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:38.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4380053a40 msgr2=0x7f4380055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4380053a40 0x7f4380055f00 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f439401c000 tx=0x7f4394015000).stop 2022-07-20T03:07:38.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 msgr2=0x7f43a009dd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a009dd30 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f43a0066650 tx=0x7f4390007c40).stop 2022-07-20T03:07:38.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 shutdown_connections 2022-07-20T03:07:38.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4380053a40 0x7f4380055f00 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43a0146b70 0x7f43a009b300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43a01462a0 0x7f43a009dd30 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 --2- 172.21.15.61:0/1196668655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43a00988b0 0x7f43a009d840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 >> 172.21.15.61:0/1196668655 conn(0x7f43a00b5f00 msgr2=0x7f43a00b8210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 shutdown_connections 2022-07-20T03:07:38.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.775+0000 7f43a695f700 1 -- 172.21.15.61:0/1196668655 wait complete. 2022-07-20T03:07:38.805 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:38.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.915+0000 7fd195d20700 1 Processor -- start 2022-07-20T03:07:38.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.915+0000 7fd195d20700 1 -- start start 2022-07-20T03:07:38.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd19009df40 0x7fd190145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd1900b4d60 0x7fd190145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1900607e0 con 0x7fd190060ee0 2022-07-20T03:07:38.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd190060d20 con 0x7fd1900b4d60 2022-07-20T03:07:38.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd190060a80 con 0x7fd19009df40 2022-07-20T03:07:38.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd19009df40 0x7fd190145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18effd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd1900b4d60 0x7fd190145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18ffff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd190060ee0 0x7fd190146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:48240/0 (socket says 172.21.15.61:48240) 2022-07-20T03:07:38.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18ffff700 1 -- 172.21.15.61:0/1838790849 learned_addr learned my addr 172.21.15.61:0/1838790849 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:38.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18effd700 1 -- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd19009df40 msgr2=0x7fd190145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:38.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18effd700 1 --2- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd19009df40 0x7fd190145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18effd700 1 -- 172.21.15.61:0/1838790849 >> v1:172.21.15.117:6789/0 conn(0x7fd190060ee0 legacy=0x7fd190146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18effd700 1 -- 172.21.15.61:0/1838790849 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd19009e320 con 0x7fd1900b4d60 2022-07-20T03:07:38.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18f7fe700 1 --2- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd19009df40 0x7fd190145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:38.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18effd700 1 --2- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1900b4d60 0x7fd190145b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd180002f70 tx=0x7fd180006df0).ready entity=mon.1 client_cookie=25b7d5cff98de81d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:38.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18e7fc700 1 -- 172.21.15.61:0/1838790849 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18000d570 con 0x7fd1900b4d60 2022-07-20T03:07:38.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18e7fc700 1 -- 172.21.15.61:0/1838790849 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd180000e90 con 0x7fd1900b4d60 2022-07-20T03:07:38.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd18e7fc700 1 -- 172.21.15.61:0/1838790849 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18000d8c0 con 0x7fd1900b4d60 2022-07-20T03:07:38.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1900b4d60 msgr2=0x7fd190145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:38.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 --2- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1900b4d60 0x7fd190145b70 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7fd180002f70 tx=0x7fd180006df0).stop 2022-07-20T03:07:38.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- 172.21.15.61:0/1838790849 shutdown_connections 2022-07-20T03:07:38.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 --2- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1900b4d60 0x7fd190145b70 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 --2- 172.21.15.61:0/1838790849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd19009df40 0x7fd190145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:38.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- 172.21.15.61:0/1838790849 >> 172.21.15.61:0/1838790849 conn(0x7fd1900b5f30 msgr2=0x7fd1900b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:38.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- 172.21.15.61:0/1838790849 shutdown_connections 2022-07-20T03:07:38.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.919+0000 7fd195d20700 1 -- 172.21.15.61:0/1838790849 wait complete. 2022-07-20T03:07:38.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 Processor -- start 2022-07-20T03:07:38.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 -- start start 2022-07-20T03:07:38.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd190060ee0 0x7fd190091660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 0x7fd1900b2600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1900b4d60 0x7fd1900b2af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:38.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd1901532b0 con 0x7fd19009df40 2022-07-20T03:07:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd190152da0 con 0x7fd190060ee0 2022-07-20T03:07:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd195d20700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd190153010 con 0x7fd1900b4d60 2022-07-20T03:07:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 0x7fd1900b2600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 0x7fd1900b2600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43142/0 (socket says 172.21.15.61:43142) 2022-07-20T03:07:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 -- 172.21.15.61:0/3476102662 learned_addr learned my addr 172.21.15.61:0/3476102662 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18f7fe700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd190060ee0 0x7fd190091660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18ffff700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1900b4d60 0x7fd1900b2af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 -- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1900b4d60 msgr2=0x7fd1900b2af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1900b4d60 0x7fd1900b2af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 -- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd190060ee0 msgr2=0x7fd190091660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd190060ee0 0x7fd190091660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 -- 172.21.15.61:0/3476102662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd18400b120 con 0x7fd19009df40 2022-07-20T03:07:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.923+0000 7fd18effd700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 0x7fd1900b2600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fd180002f70 tx=0x7fd180014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18000dea0 con 0x7fd19009df40 2022-07-20T03:07:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd180013de0 con 0x7fd19009df40 2022-07-20T03:07:39.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd18000b180 con 0x7fd19009df40 2022-07-20T03:07:39.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd18001e5f0 con 0x7fd19009df40 2022-07-20T03:07:39.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd1900901d0 con 0x7fd19009df40 2022-07-20T03:07:39.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd180025030 con 0x7fd19009df40 2022-07-20T03:07:39.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd190146060 con 0x7fd19009df40 2022-07-20T03:07:39.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18cff9700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd170053a40 0x7fd170055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd18006cb70 con 0x7fd19009df40 2022-07-20T03:07:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18f7fe700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd170053a40 0x7fd170055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.927+0000 7fd18f7fe700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd170053a40 0x7fd170055f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd184008000 tx=0x7fd184008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:38.931+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd1800439c0 con 0x7fd19009df40 2022-07-20T03:07:39.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fd1900574e0 con 0x7fd19009df40 2022-07-20T03:07:39.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd18cff9700 1 -- 172.21.15.61:0/3476102662 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7fd180023020 con 0x7fd19009df40 2022-07-20T03:07:39.069 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:39.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd170053a40 msgr2=0x7fd170055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd170053a40 0x7fd170055f00 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7fd184008000 tx=0x7fd184008040).stop 2022-07-20T03:07:39.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 msgr2=0x7fd1900b2600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 0x7fd1900b2600 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fd180002f70 tx=0x7fd180014000).stop 2022-07-20T03:07:39.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 shutdown_connections 2022-07-20T03:07:39.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd170053a40 0x7fd170055f00 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd1900b4d60 0x7fd1900b2af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd19009df40 0x7fd1900b2600 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 --2- 172.21.15.61:0/3476102662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd190060ee0 0x7fd190091660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.059+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 >> 172.21.15.61:0/3476102662 conn(0x7fd1900b5f30 msgr2=0x7fd190143e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:39.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.063+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 shutdown_connections 2022-07-20T03:07:39.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.063+0000 7fd195d20700 1 -- 172.21.15.61:0/3476102662 wait complete. 2022-07-20T03:07:39.089 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:39.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 Processor -- start 2022-07-20T03:07:39.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 -- start start 2022-07-20T03:07:39.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980b2210 0x7f2998140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29980ac680 0x7f29980b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f299813fe80 con 0x7f29980967e0 2022-07-20T03:07:39.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2998140120 con 0x7f29980ac680 2022-07-20T03:07:39.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299fe16700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29981403c0 con 0x7f29980b2210 2022-07-20T03:07:39.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299e3b3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29980ac680 0x7f29980b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299e3b3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29980ac680 0x7f29980b25f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43152/0 (socket says 172.21.15.61:43152) 2022-07-20T03:07:39.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.211+0000 7f299e3b3700 1 -- 172.21.15.61:0/1562656690 learned_addr learned my addr 172.21.15.61:0/1562656690 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:39.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299dbb2700 1 --2- 172.21.15.61:0/1562656690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980b2210 0x7f2998140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299e3b3700 1 -- 172.21.15.61:0/1562656690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980b2210 msgr2=0x7f2998140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:39.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299e3b3700 1 --2- 172.21.15.61:0/1562656690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980b2210 0x7f2998140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299e3b3700 1 -- 172.21.15.61:0/1562656690 >> v1:172.21.15.117:6789/0 conn(0x7f29980967e0 legacy=0x7f29980ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299e3b3700 1 -- 172.21.15.61:0/1562656690 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29980ab7c0 con 0x7f29980ac680 2022-07-20T03:07:39.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299e3b3700 1 --2- 172.21.15.61:0/1562656690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29980b25f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f29900032f0 tx=0x7f299000b530).ready entity=mon.0 client_cookie=1603634418c6d5e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299cbb0700 1 -- 172.21.15.61:0/1562656690 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2990010660 con 0x7f29980ac680 2022-07-20T03:07:39.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299cbb0700 1 -- 172.21.15.61:0/1562656690 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29900107c0 con 0x7f29980ac680 2022-07-20T03:07:39.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299cbb0700 1 -- 172.21.15.61:0/1562656690 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2990010b20 con 0x7f29980ac680 2022-07-20T03:07:39.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 -- 172.21.15.61:0/1562656690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 msgr2=0x7f29980b25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 --2- 172.21.15.61:0/1562656690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29980b25f0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f29900032f0 tx=0x7f299000b530).stop 2022-07-20T03:07:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 -- 172.21.15.61:0/1562656690 shutdown_connections 2022-07-20T03:07:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 --2- 172.21.15.61:0/1562656690 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29980b25f0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 --2- 172.21.15.61:0/1562656690 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980b2210 0x7f2998140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 -- 172.21.15.61:0/1562656690 >> 172.21.15.61:0/1562656690 conn(0x7f29980b36c0 msgr2=0x7f29980b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:39.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 -- 172.21.15.61:0/1562656690 shutdown_connections 2022-07-20T03:07:39.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 -- 172.21.15.61:0/1562656690 wait complete. 2022-07-20T03:07:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 Processor -- start 2022-07-20T03:07:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.215+0000 7f299fe16700 1 -- start start 2022-07-20T03:07:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980967e0 0x7f299805fbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29981523b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29980b2210 0x7f2998152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299813b010 con 0x7f29980ac680 2022-07-20T03:07:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299809a180 con 0x7f29980b2210 2022-07-20T03:07:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f299809a420 con 0x7f29980967e0 2022-07-20T03:07:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29981523b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29981523b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43156/0 (socket says 172.21.15.61:43156) 2022-07-20T03:07:39.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 -- 172.21.15.61:0/2564186156 learned_addr learned my addr 172.21.15.61:0/2564186156 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:39.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299dbb2700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980967e0 0x7f299805fbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299e3b3700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29980b2210 0x7f2998152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 -- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980967e0 msgr2=0x7f299805fbf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980967e0 0x7f299805fbf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 -- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29980b2210 msgr2=0x7f2998152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29980b2210 0x7f2998152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 -- 172.21.15.61:0/2564186156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f299000b7d0 con 0x7f29980ac680 2022-07-20T03:07:39.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299dbb2700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980967e0 0x7f299805fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:39.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299d3b1700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29981523b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f29980632d0 tx=0x7f2988007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f298800ce30 con 0x7f29980ac680 2022-07-20T03:07:39.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2988007240 con 0x7f29980ac680 2022-07-20T03:07:39.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2998099d90 con 0x7f29980ac680 2022-07-20T03:07:39.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2988006700 con 0x7f29980ac680 2022-07-20T03:07:39.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299809b400 con 0x7f29980ac680 2022-07-20T03:07:39.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.219+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29980994a0 con 0x7f29980ac680 2022-07-20T03:07:39.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.223+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2988010030 con 0x7f29980ac680 2022-07-20T03:07:39.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.223+0000 7f2986ffd700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2978053920 0x7f2978055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.223+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f298806bbf0 con 0x7f29980ac680 2022-07-20T03:07:39.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.223+0000 7f299dbb2700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2978053920 0x7f2978055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.223+0000 7f299dbb2700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2978053920 0x7f2978055de0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f299400c000 tx=0x7f2994008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.227+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29980994a0 con 0x7f29980ac680 2022-07-20T03:07:39.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f29980994a0 con 0x7f29980ac680 2022-07-20T03:07:39.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f2986ffd700 1 -- 172.21.15.61:0/2564186156 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7f29980994a0 con 0x7f29980ac680 2022-07-20T03:07:39.360 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2978053920 msgr2=0x7f2978055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2978053920 0x7f2978055de0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f299400c000 tx=0x7f2994008040).stop 2022-07-20T03:07:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 msgr2=0x7f29981523b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29981523b0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f29980632d0 tx=0x7f2988007c10).stop 2022-07-20T03:07:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 shutdown_connections 2022-07-20T03:07:39.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2978053920 0x7f2978055de0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29980b2210 0x7f2998152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29980ac680 0x7f29981523b0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 --2- 172.21.15.61:0/2564186156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29980967e0 0x7f299805fbf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 >> 172.21.15.61:0/2564186156 conn(0x7f29980b36c0 msgr2=0x7f299813bd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:39.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 shutdown_connections 2022-07-20T03:07:39.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.355+0000 7f299fe16700 1 -- 172.21.15.61:0/2564186156 wait complete. 2022-07-20T03:07:39.416 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:39.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 Processor -- start 2022-07-20T03:07:39.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 -- start start 2022-07-20T03:07:39.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b900b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b900af8e0 0x7f2b9005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b90147010 con 0x7f2b900b5650 2022-07-20T03:07:39.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b901472b0 con 0x7f2b900af8e0 2022-07-20T03:07:39.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b9484b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90146d40 con 0x7f2b900adff0 2022-07-20T03:07:39.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2b900b5650 0x7f2b90147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39762/0 (socket says 172.21.15.61:39762) 2022-07-20T03:07:39.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8ed9d700 1 -- 172.21.15.61:0/1495982911 learned_addr learned my addr 172.21.15.61:0/1495982911 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:39.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8e59c700 1 --2- 172.21.15.61:0/1495982911 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b900b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8f59e700 1 --2- 172.21.15.61:0/1495982911 >> v2:172.21.15.117:3300/0 conn(0x7f2b900af8e0 0x7f2b9005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8dd9b700 1 -- 172.21.15.61:0/1495982911 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1537467884 0 0) 0x7f2b90147010 con 0x7f2b900b5650 2022-07-20T03:07:39.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8dd9b700 1 -- 172.21.15.61:0/1495982911 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2b70003260 con 0x7f2b900b5650 2022-07-20T03:07:39.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8e59c700 1 -- 172.21.15.61:0/1495982911 >> v2:172.21.15.117:3300/0 conn(0x7f2b900af8e0 msgr2=0x7f2b9005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8e59c700 1 --2- 172.21.15.61:0/1495982911 >> v2:172.21.15.117:3300/0 conn(0x7f2b900af8e0 0x7f2b9005bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8e59c700 1 -- 172.21.15.61:0/1495982911 >> v1:172.21.15.61:6789/0 conn(0x7f2b900b5650 legacy=0x7f2b90147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8e59c700 1 -- 172.21.15.61:0/1495982911 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b900acae0 con 0x7f2b900adff0 2022-07-20T03:07:39.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8e59c700 1 --2- 172.21.15.61:0/1495982911 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b900b5a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2b780032f0 tx=0x7f2b780091a0).ready entity=mon.2 client_cookie=2ffb3d846f1b4a51 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8dd9b700 1 -- 172.21.15.61:0/1495982911 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b7800b600 con 0x7f2b900adff0 2022-07-20T03:07:39.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8dd9b700 1 -- 172.21.15.61:0/1495982911 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7800b760 con 0x7f2b900adff0 2022-07-20T03:07:39.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.531+0000 7f2b8dd9b700 1 -- 172.21.15.61:0/1495982911 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b7800b8d0 con 0x7f2b900adff0 2022-07-20T03:07:39.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- 172.21.15.61:0/1495982911 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 msgr2=0x7f2b900b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 --2- 172.21.15.61:0/1495982911 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b900b5a30 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f2b780032f0 tx=0x7f2b780091a0).stop 2022-07-20T03:07:39.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- 172.21.15.61:0/1495982911 shutdown_connections 2022-07-20T03:07:39.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 --2- 172.21.15.61:0/1495982911 >> v2:172.21.15.117:3300/0 conn(0x7f2b900af8e0 0x7f2b9005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 --2- 172.21.15.61:0/1495982911 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b900b5a30 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- 172.21.15.61:0/1495982911 >> 172.21.15.61:0/1495982911 conn(0x7f2b900634e0 msgr2=0x7f2b900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:39.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- 172.21.15.61:0/1495982911 shutdown_connections 2022-07-20T03:07:39.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- 172.21.15.61:0/1495982911 wait complete. 2022-07-20T03:07:39.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 Processor -- start 2022-07-20T03:07:39.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- start start 2022-07-20T03:07:39.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b90064c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900af8e0 0x7f2b9015a200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 0x7f2b9010f8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90110c90 con 0x7f2b900b5650 2022-07-20T03:07:39.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b90065110 con 0x7f2b900af8e0 2022-07-20T03:07:39.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b9484b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b9015a6f0 con 0x7f2b900adff0 2022-07-20T03:07:39.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b8f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 0x7f2b9010f8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b8ed9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b90064c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b8f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 0x7f2b9010f8c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43172/0 (socket says 172.21.15.61:43172) 2022-07-20T03:07:39.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b8f59e700 1 -- 172.21.15.61:0/2245084277 learned_addr learned my addr 172.21.15.61:0/2245084277 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:39.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.535+0000 7f2b8e59c700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900af8e0 0x7f2b9015a200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8f59e700 1 -- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 msgr2=0x7f2b90064c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8f59e700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b90064c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8f59e700 1 -- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900af8e0 msgr2=0x7f2b9015a200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8f59e700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900af8e0 0x7f2b9015a200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8f59e700 1 -- 172.21.15.61:0/2245084277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b78009420 con 0x7f2b900b5650 2022-07-20T03:07:39.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8f59e700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 0x7f2b9010f8c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2b8800c000 tx=0x7f2b88006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b88010780 con 0x7f2b900b5650 2022-07-20T03:07:39.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b880108e0 con 0x7f2b900b5650 2022-07-20T03:07:39.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b9010f3f0 con 0x7f2b900b5650 2022-07-20T03:07:39.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b88010c40 con 0x7f2b900b5650 2022-07-20T03:07:39.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b9010fdb0 con 0x7f2b900b5650 2022-07-20T03:07:39.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b9005c280 con 0x7f2b900b5650 2022-07-20T03:07:39.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b8801c460 con 0x7f2b900b5650 2022-07-20T03:07:39.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b87fff700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b74053a40 0x7f2b74055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2b8806c2e0 con 0x7f2b900b5650 2022-07-20T03:07:39.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.539+0000 7f2b8ed9d700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b74053a40 0x7f2b74055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.543+0000 7f2b8ed9d700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b74053a40 0x7f2b74055f00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2b8000d000 tx=0x7f2b80009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.547+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b880431e0 con 0x7f2b900b5650 2022-07-20T03:07:39.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.670+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2b900574e0 con 0x7f2b900b5650 2022-07-20T03:07:39.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b87fff700 1 -- 172.21.15.61:0/2245084277 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f2b88046800 con 0x7f2b900b5650 2022-07-20T03:07:39.745 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:39.768 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b74053a40 msgr2=0x7f2b74055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b74053a40 0x7f2b74055f00 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f2b8000d000 tx=0x7f2b80009040).stop 2022-07-20T03:07:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 msgr2=0x7f2b9010f8c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 0x7f2b9010f8c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f2b8800c000 tx=0x7f2b88006df0).stop 2022-07-20T03:07:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 shutdown_connections 2022-07-20T03:07:39.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b74053a40 0x7f2b74055f00 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b900af8e0 0x7f2b9015a200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b900b5650 0x7f2b9010f8c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 --2- 172.21.15.61:0/2245084277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b900adff0 0x7f2b90064c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:39.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 >> 172.21.15.61:0/2245084277 conn(0x7f2b900634e0 msgr2=0x7f2b900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:39.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 shutdown_connections 2022-07-20T03:07:39.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.674+0000 7f2b9484b700 1 -- 172.21.15.61:0/2245084277 wait complete. 2022-07-20T03:07:39.805 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:39.816 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:39.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.954+0000 7f5499208700 1 Processor -- start 2022-07-20T03:07:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.954+0000 7f5499208700 1 -- start start 2022-07-20T03:07:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f54940b3cf0 0x7f5494142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54940ba180 con 0x7f54940b9da0 2022-07-20T03:07:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54941461b0 con 0x7f5494142f40 2022-07-20T03:07:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54940ad5b0 con 0x7f54940b3cf0 2022-07-20T03:07:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5493fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f54940b9da0 0x7f54941468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39774/0 (socket says 172.21.15.61:39774) 2022-07-20T03:07:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5493fff700 1 -- 172.21.15.61:0/314905432 learned_addr learned my addr 172.21.15.61:0/314905432 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f548b7fe700 1 --2- 172.21.15.61:0/314905432 >> v2:172.21.15.117:3300/0 conn(0x7f54940b3cf0 0x7f5494142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2746876029 0 0) 0x7f54940ba180 con 0x7f54940b9da0 2022-07-20T03:07:39.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5474003260 con 0x7f54940b9da0 2022-07-20T03:07:39.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2926898438 0 0) 0x7f54941461b0 con 0x7f5494142f40 2022-07-20T03:07:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54940ba180 con 0x7f5494142f40 2022-07-20T03:07:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f548b7fe700 1 -- 172.21.15.61:0/314905432 >> v1:172.21.15.117:6789/0 conn(0x7f5494142f40 legacy=0x7f54941419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 497736172 0 0) 0x7f5474003260 con 0x7f54940b9da0 2022-07-20T03:07:39.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f548b7fe700 1 -- 172.21.15.61:0/314905432 >> v1:172.21.15.61:6789/0 conn(0x7f54940b9da0 legacy=0x7f54941468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:39.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f548b7fe700 1 -- 172.21.15.61:0/314905432 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5494146450 con 0x7f54940b3cf0 2022-07-20T03:07:39.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f548b7fe700 1 --2- 172.21.15.61:0/314905432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54940b3cf0 0x7f5494142a50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5484001110 tx=0x7f547c006df0).ready entity=mon.1 client_cookie=50e2032688f2f631 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:39.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5484001660 con 0x7f54940b3cf0 2022-07-20T03:07:39.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f547c000e90 con 0x7f54940b3cf0 2022-07-20T03:07:39.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f54937fe700 1 -- 172.21.15.61:0/314905432 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f547c012460 con 0x7f54940b3cf0 2022-07-20T03:07:40.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- 172.21.15.61:0/314905432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54940b3cf0 msgr2=0x7f5494142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 --2- 172.21.15.61:0/314905432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54940b3cf0 0x7f5494142a50 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f5484001110 tx=0x7f547c006df0).stop 2022-07-20T03:07:40.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- 172.21.15.61:0/314905432 shutdown_connections 2022-07-20T03:07:40.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 --2- 172.21.15.61:0/314905432 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54940b3cf0 0x7f5494142a50 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- 172.21.15.61:0/314905432 >> 172.21.15.61:0/314905432 conn(0x7f54940678c0 msgr2=0x7f54940b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:40.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- 172.21.15.61:0/314905432 shutdown_connections 2022-07-20T03:07:40.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.958+0000 7f5499208700 1 -- 172.21.15.61:0/314905432 wait complete. 2022-07-20T03:07:40.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 Processor -- start 2022-07-20T03:07:40.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 -- start start 2022-07-20T03:07:40.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 0x7f5494112880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54940b9da0 0x7f54941400c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5494142f40 0x7f54941405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494113e00 con 0x7f54940b3cf0 2022-07-20T03:07:40.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5494114030 con 0x7f5494142f40 2022-07-20T03:07:40.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54941142d0 con 0x7f54940b9da0 2022-07-20T03:07:40.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 0x7f5494112880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 0x7f5494112880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43182/0 (socket says 172.21.15.61:43182) 2022-07-20T03:07:40.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 -- 172.21.15.61:0/3423055407 learned_addr learned my addr 172.21.15.61:0/3423055407 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:40.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f548b7fe700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54940b9da0 0x7f54941400c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5498a07700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5494142f40 0x7f54941405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 -- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54940b9da0 msgr2=0x7f54941400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54940b9da0 0x7f54941400c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 -- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5494142f40 msgr2=0x7f54941405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5494142f40 0x7f54941405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 -- 172.21.15.61:0/3423055407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f547c00b180 con 0x7f54940b3cf0 2022-07-20T03:07:40.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5493fff700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 0x7f5494112880 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5484001840 tx=0x7f5484001720).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5484007170 con 0x7f54940b3cf0 2022-07-20T03:07:40.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54840072d0 con 0x7f54940b3cf0 2022-07-20T03:07:40.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.962+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5494114570 con 0x7f54940b3cf0 2022-07-20T03:07:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5484004920 con 0x7f54940b3cf0 2022-07-20T03:07:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54940b1270 con 0x7f54940b3cf0 2022-07-20T03:07:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5494114700 con 0x7f54940b3cf0 2022-07-20T03:07:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f548401b070 con 0x7f54940b3cf0 2022-07-20T03:07:40.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f5491ffb700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5478053920 0x7f5478055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f548406bcd0 con 0x7f54940b3cf0 2022-07-20T03:07:40.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f548b7fe700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5478053920 0x7f5478055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.966+0000 7f548b7fe700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5478053920 0x7f5478055de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f547c018000 tx=0x7f547c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:39.974+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5484042b20 con 0x7f54940b3cf0 2022-07-20T03:07:40.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.102+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f54940667f0 con 0x7f54940b3cf0 2022-07-20T03:07:40.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.102+0000 7f5491ffb700 1 -- 172.21.15.61:0/3423055407 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f54840220b0 con 0x7f54940b3cf0 2022-07-20T03:07:40.109 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:40.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5478053920 msgr2=0x7f5478055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5478053920 0x7f5478055de0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f547c018000 tx=0x7f547c014000).stop 2022-07-20T03:07:40.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 msgr2=0x7f5494112880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 0x7f5494112880 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f5484001840 tx=0x7f5484001720).stop 2022-07-20T03:07:40.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 shutdown_connections 2022-07-20T03:07:40.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5478053920 0x7f5478055de0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5494142f40 0x7f54941405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54940b9da0 0x7f54941400c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 --2- 172.21.15.61:0/3423055407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54940b3cf0 0x7f5494112880 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 >> 172.21.15.61:0/3423055407 conn(0x7f54940678c0 msgr2=0x7f54940b5cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:40.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 shutdown_connections 2022-07-20T03:07:40.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.106+0000 7f5499208700 1 -- 172.21.15.61:0/3423055407 wait complete. 2022-07-20T03:07:40.152 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:40.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.282+0000 7f116cd42700 1 Processor -- start 2022-07-20T03:07:40.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.282+0000 7f116cd42700 1 -- start start 2022-07-20T03:07:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f11680620e0 0x7f1168142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11680b2930 con 0x7f11680b2550 2022-07-20T03:07:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f116805fe30 con 0x7f11680ab650 2022-07-20T03:07:40.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f116805c5b0 con 0x7f11680620e0 2022-07-20T03:07:40.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f11680620e0 0x7f1168142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f11680620e0 0x7f1168142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43192/0 (socket says 172.21.15.61:43192) 2022-07-20T03:07:40.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 -- 172.21.15.61:0/572291991 learned_addr learned my addr 172.21.15.61:0/572291991 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:40.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116659c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f11680b2550 0x7f116805ca50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39788/0 (socket says 172.21.15.61:39788) 2022-07-20T03:07:40.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 -- 172.21.15.61:0/572291991 >> v1:172.21.15.117:6789/0 conn(0x7f11680ab650 legacy=0x7f1168142700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 -- 172.21.15.61:0/572291991 >> v1:172.21.15.61:6789/0 conn(0x7f11680b2550 legacy=0x7f116805ca50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 -- 172.21.15.61:0/572291991 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f116805d160 con 0x7f11680620e0 2022-07-20T03:07:40.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f115ffff700 1 --2- 172.21.15.61:0/572291991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680620e0 0x7f1168142210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1168063790 tx=0x7f1150006eb0).ready entity=mon.0 client_cookie=ed750296842988b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f1165d9b700 1 -- 172.21.15.61:0/572291991 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f115000d660 con 0x7f11680620e0 2022-07-20T03:07:40.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f1165d9b700 1 -- 172.21.15.61:0/572291991 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150000e90 con 0x7f11680620e0 2022-07-20T03:07:40.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f1165d9b700 1 -- 172.21.15.61:0/572291991 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f115000d9f0 con 0x7f11680620e0 2022-07-20T03:07:40.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- 172.21.15.61:0/572291991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680620e0 msgr2=0x7f1168142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 --2- 172.21.15.61:0/572291991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680620e0 0x7f1168142210 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f1168063790 tx=0x7f1150006eb0).stop 2022-07-20T03:07:40.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- 172.21.15.61:0/572291991 shutdown_connections 2022-07-20T03:07:40.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 --2- 172.21.15.61:0/572291991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680620e0 0x7f1168142210 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- 172.21.15.61:0/572291991 >> 172.21.15.61:0/572291991 conn(0x7f11680b36c0 msgr2=0x7f11680b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:40.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- 172.21.15.61:0/572291991 shutdown_connections 2022-07-20T03:07:40.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.286+0000 7f116cd42700 1 -- 172.21.15.61:0/572291991 wait complete. 2022-07-20T03:07:40.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 Processor -- start 2022-07-20T03:07:40.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- start start 2022-07-20T03:07:40.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11680620e0 0x7f1168096170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 0x7f1168094380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11680b2550 0x7f1168094870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11680923c0 con 0x7f11680ab650 2022-07-20T03:07:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1168093920 con 0x7f11680620e0 2022-07-20T03:07:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f116808f610 con 0x7f11680b2550 2022-07-20T03:07:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 0x7f1168094380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 0x7f1168094380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43196/0 (socket says 172.21.15.61:43196) 2022-07-20T03:07:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 -- 172.21.15.61:0/1059969444 learned_addr learned my addr 172.21.15.61:0/1059969444 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f1166d9d700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11680b2550 0x7f1168094870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116659c700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11680620e0 0x7f1168096170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 -- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11680b2550 msgr2=0x7f1168094870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11680b2550 0x7f1168094870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 -- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11680620e0 msgr2=0x7f1168096170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11680620e0 0x7f1168096170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 -- 172.21.15.61:0/1059969444 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f115000b1e0 con 0x7f11680ab650 2022-07-20T03:07:40.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116659c700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11680620e0 0x7f1168096170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:40.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f1166d9d700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11680b2550 0x7f1168094870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:40.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115ffff700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 0x7f1168094380 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1168063790 tx=0x7f1150015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1150002f70 con 0x7f11680ab650 2022-07-20T03:07:40.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1150006780 con 0x7f11680ab650 2022-07-20T03:07:40.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1168093b50 con 0x7f11680ab650 2022-07-20T03:07:40.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f115001d450 con 0x7f11680ab650 2022-07-20T03:07:40.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1168093ef0 con 0x7f11680ab650 2022-07-20T03:07:40.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.290+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1168093d10 con 0x7f11680ab650 2022-07-20T03:07:40.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.294+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1150024030 con 0x7f11680ab650 2022-07-20T03:07:40.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.294+0000 7f115f7fe700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f114c053920 0x7f114c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.294+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f115006d010 con 0x7f11680ab650 2022-07-20T03:07:40.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.294+0000 7f116659c700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f114c053920 0x7f114c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.294+0000 7f116659c700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f114c053920 0x7f114c055de0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1158008000 tx=0x7f1158009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.298+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1150047e60 con 0x7f11680ab650 2022-07-20T03:07:40.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.422+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f11680574e0 con 0x7f11680ab650 2022-07-20T03:07:40.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.422+0000 7f115f7fe700 1 -- 172.21.15.61:0/1059969444 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1150022070 con 0x7f11680ab650 2022-07-20T03:07:40.450 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:40.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f114c053920 msgr2=0x7f114c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f114c053920 0x7f114c055de0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f1158008000 tx=0x7f1158009040).stop 2022-07-20T03:07:40.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 msgr2=0x7f1168094380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 0x7f1168094380 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f1168063790 tx=0x7f1150015000).stop 2022-07-20T03:07:40.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 shutdown_connections 2022-07-20T03:07:40.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f114c053920 0x7f114c055de0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11680b2550 0x7f1168094870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11680ab650 0x7f1168094380 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 --2- 172.21.15.61:0/1059969444 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f11680620e0 0x7f1168096170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 >> 172.21.15.61:0/1059969444 conn(0x7f11680b36c0 msgr2=0x7f11680afe90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:40.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 shutdown_connections 2022-07-20T03:07:40.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.430+0000 7f116cd42700 1 -- 172.21.15.61:0/1059969444 wait complete. 2022-07-20T03:07:40.474 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:40.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.602+0000 7f1c562eb700 1 Processor -- start 2022-07-20T03:07:40.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.602+0000 7f1c562eb700 1 -- start start 2022-07-20T03:07:40.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1c50060ee0 0x7f1c50145d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c500607e0 con 0x7f1c5009df40 2022-07-20T03:07:40.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c50060a80 con 0x7f1c500b4d60 2022-07-20T03:07:40.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c50060d20 con 0x7f1c50060ee0 2022-07-20T03:07:40.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c54888700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1c50060ee0 0x7f1c50145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c4ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1c5009df40 0x7f1c50145680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39802/0 (socket says 172.21.15.61:39802) 2022-07-20T03:07:40.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c4ffff700 1 -- 172.21.15.61:0/1311698835 learned_addr learned my addr 172.21.15.61:0/1311698835 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:40.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c54888700 1 -- 172.21.15.61:0/1311698835 >> v1:172.21.15.117:6789/0 conn(0x7f1c500b4d60 legacy=0x7f1c5009cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c54888700 1 -- 172.21.15.61:0/1311698835 >> v1:172.21.15.61:6789/0 conn(0x7f1c5009df40 legacy=0x7f1c50145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c54888700 1 -- 172.21.15.61:0/1311698835 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5009e320 con 0x7f1c50060ee0 2022-07-20T03:07:40.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c54888700 1 --2- 172.21.15.61:0/1311698835 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50145d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1c480032f0 tx=0x7f1c480091d0).ready entity=mon.0 client_cookie=e8eac3eb7621626 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c4effd700 1 -- 172.21.15.61:0/1311698835 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c48009ea0 con 0x7f1c50060ee0 2022-07-20T03:07:40.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c4effd700 1 -- 172.21.15.61:0/1311698835 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c4800b8a0 con 0x7f1c50060ee0 2022-07-20T03:07:40.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c4effd700 1 -- 172.21.15.61:0/1311698835 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c4800ba10 con 0x7f1c50060ee0 2022-07-20T03:07:40.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- 172.21.15.61:0/1311698835 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 msgr2=0x7f1c50145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 --2- 172.21.15.61:0/1311698835 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50145d90 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f1c480032f0 tx=0x7f1c480091d0).stop 2022-07-20T03:07:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- 172.21.15.61:0/1311698835 shutdown_connections 2022-07-20T03:07:40.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 --2- 172.21.15.61:0/1311698835 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50145d90 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- 172.21.15.61:0/1311698835 >> 172.21.15.61:0/1311698835 conn(0x7f1c500b5f30 msgr2=0x7f1c500b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:40.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- 172.21.15.61:0/1311698835 shutdown_connections 2022-07-20T03:07:40.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.606+0000 7f1c562eb700 1 -- 172.21.15.61:0/1311698835 wait complete. 2022-07-20T03:07:40.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 Processor -- start 2022-07-20T03:07:40.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 -- start start 2022-07-20T03:07:40.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c5009df40 0x7f1c500b21d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c500b4d60 0x7f1c500b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c500b1680 con 0x7f1c50060ee0 2022-07-20T03:07:40.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c50152f30 con 0x7f1c500b4d60 2022-07-20T03:07:40.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c501531d0 con 0x7f1c5009df40 2022-07-20T03:07:40.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50143710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43210/0 (socket says 172.21.15.61:43210) 2022-07-20T03:07:40.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 -- 172.21.15.61:0/3336733194 learned_addr learned my addr 172.21.15.61:0/3336733194 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:40.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4f7fe700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c5009df40 0x7f1c500b21d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c54888700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c500b4d60 0x7f1c500b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 -- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c5009df40 msgr2=0x7f1c500b21d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c5009df40 0x7f1c500b21d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 -- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c500b4d60 msgr2=0x7f1c500b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c500b4d60 0x7f1c500b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 -- 172.21.15.61:0/3336733194 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c48009470 con 0x7f1c50060ee0 2022-07-20T03:07:40.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4ffff700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50143710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f1c50067ed0 tx=0x7f1c44007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c440070d0 con 0x7f1c50060ee0 2022-07-20T03:07:40.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c44007230 con 0x7f1c50060ee0 2022-07-20T03:07:40.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c50153540 con 0x7f1c50060ee0 2022-07-20T03:07:40.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c440073a0 con 0x7f1c50060ee0 2022-07-20T03:07:40.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.610+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c500b1d00 con 0x7f1c50060ee0 2022-07-20T03:07:40.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.614+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c500b14a0 con 0x7f1c50060ee0 2022-07-20T03:07:40.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.614+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1c44010070 con 0x7f1c50060ee0 2022-07-20T03:07:40.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.614+0000 7f1c4d7fa700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1c300539b0 0x7f1c30055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:40.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.614+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1c4406b4a0 con 0x7f1c50060ee0 2022-07-20T03:07:40.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.614+0000 7f1c4f7fe700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1c300539b0 0x7f1c30055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:40.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.614+0000 7f1c4f7fe700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1c300539b0 0x7f1c30055e70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f1c4000d000 tx=0x7f1c40007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:40.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.618+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1c44042400 con 0x7f1c50060ee0 2022-07-20T03:07:40.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.742+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1c500574e0 con 0x7f1c50060ee0 2022-07-20T03:07:40.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.742+0000 7f1c4d7fa700 1 -- 172.21.15.61:0/3336733194 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1c44045a20 con 0x7f1c50060ee0 2022-07-20T03:07:40.807 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:40.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1c300539b0 msgr2=0x7f1c30055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1c300539b0 0x7f1c30055e70 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f1c4000d000 tx=0x7f1c40007b90).stop 2022-07-20T03:07:40.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 msgr2=0x7f1c50143710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:40.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50143710 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f1c50067ed0 tx=0x7f1c44007ab0).stop 2022-07-20T03:07:40.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 shutdown_connections 2022-07-20T03:07:40.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1c300539b0 0x7f1c30055e70 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c500b4d60 0x7f1c500b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c5009df40 0x7f1c500b21d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 --2- 172.21.15.61:0/3336733194 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c50060ee0 0x7f1c50143710 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:40.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 >> 172.21.15.61:0/3336733194 conn(0x7f1c500b5f30 msgr2=0x7f1c500b40c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:40.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 shutdown_connections 2022-07-20T03:07:40.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.746+0000 7f1c562eb700 1 -- 172.21.15.61:0/3336733194 wait complete. 2022-07-20T03:07:40.846 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:40.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.970+0000 7fa74546f700 1 Processor -- start 2022-07-20T03:07:40.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.970+0000 7fa74546f700 1 -- start start 2022-07-20T03:07:41.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa7400b4b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa740143f60 0x7fa7400abfb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa7401443f0 con 0x7fa7400b58f0 2022-07-20T03:07:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa7400aae10 con 0x7fa740143f60 2022-07-20T03:07:41.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa7400b4150 con 0x7fa7400aa9d0 2022-07-20T03:07:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa740143f60 0x7fa7400abfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa740143f60 0x7fa7400abfb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43220/0 (socket says 172.21.15.61:43220) 2022-07-20T03:07:41.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa73ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa7400b58f0 0x7fa7400b01f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39816/0 (socket says 172.21.15.61:39816) 2022-07-20T03:07:41.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 -- 172.21.15.61:0/744913614 learned_addr learned my addr 172.21.15.61:0/744913614 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:41.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa73f7fe700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa7400b4b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 -- 172.21.15.61:0/744913614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 msgr2=0x7fa7400b4b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:41.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa7400b4b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 -- 172.21.15.61:0/744913614 >> v1:172.21.15.61:6789/0 conn(0x7fa7400b58f0 legacy=0x7fa7400b01f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 -- 172.21.15.61:0/744913614 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7400b43f0 con 0x7fa740143f60 2022-07-20T03:07:41.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa73f7fe700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa7400b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:41.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa744c6e700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa740143f60 0x7fa7400abfb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa7380032f0 tx=0x7fa738009290).ready entity=mon.0 client_cookie=c1a12d4072c22b9a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa73effd700 1 -- 172.21.15.61:0/744913614 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa73800b870 con 0x7fa740143f60 2022-07-20T03:07:41.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa73effd700 1 -- 172.21.15.61:0/744913614 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa73800b9d0 con 0x7fa740143f60 2022-07-20T03:07:41.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa73effd700 1 -- 172.21.15.61:0/744913614 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa73800bb80 con 0x7fa740143f60 2022-07-20T03:07:41.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- 172.21.15.61:0/744913614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa740143f60 msgr2=0x7fa7400abfb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa740143f60 0x7fa7400abfb0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fa7380032f0 tx=0x7fa738009290).stop 2022-07-20T03:07:41.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- 172.21.15.61:0/744913614 shutdown_connections 2022-07-20T03:07:41.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa740143f60 0x7fa7400abfb0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 --2- 172.21.15.61:0/744913614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa7400b4b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- 172.21.15.61:0/744913614 >> 172.21.15.61:0/744913614 conn(0x7fa7400632b0 msgr2=0x7fa740063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:41.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- 172.21.15.61:0/744913614 shutdown_connections 2022-07-20T03:07:41.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.974+0000 7fa74546f700 1 -- 172.21.15.61:0/744913614 wait complete. 2022-07-20T03:07:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 Processor -- start 2022-07-20T03:07:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 -- start start 2022-07-20T03:07:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa740147a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 0x7fa74013d970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa740143f60 0x7fa74013de60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740145fb0 con 0x7fa7400b58f0 2022-07-20T03:07:41.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa740146220 con 0x7fa740143f60 2022-07-20T03:07:41.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa740144710 con 0x7fa7400aa9d0 2022-07-20T03:07:41.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 0x7fa74013d970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 0x7fa74013d970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43224/0 (socket says 172.21.15.61:43224) 2022-07-20T03:07:41.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 -- 172.21.15.61:0/101190833 learned_addr learned my addr 172.21.15.61:0/101190833 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:41.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa744c6e700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa740143f60 0x7fa74013de60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73ffff700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa740147a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 -- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 msgr2=0x7fa740147a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa740147a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 -- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa740143f60 msgr2=0x7fa74013de60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa740143f60 0x7fa74013de60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 -- 172.21.15.61:0/101190833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa738009530 con 0x7fa7400b58f0 2022-07-20T03:07:41.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73f7fe700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 0x7fa74013d970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa730006940 tx=0x7fa730008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa730011780 con 0x7fa7400b58f0 2022-07-20T03:07:41.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa7300118e0 con 0x7fa7400b58f0 2022-07-20T03:07:41.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa740147f50 con 0x7fa7400b58f0 2022-07-20T03:07:41.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa730011c40 con 0x7fa7400b58f0 2022-07-20T03:07:41.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.978+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa740066e50 con 0x7fa7400b58f0 2022-07-20T03:07:41.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.982+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa730012420 con 0x7fa7400b58f0 2022-07-20T03:07:41.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.982+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7400b4690 con 0x7fa7400b58f0 2022-07-20T03:07:41.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.982+0000 7fa73d7fa700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa7200539b0 0x7fa720055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.982+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa73006bef0 con 0x7fa7400b58f0 2022-07-20T03:07:41.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.982+0000 7fa73ffff700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa7200539b0 0x7fa720055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.982+0000 7fa73ffff700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa7200539b0 0x7fa720055e70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa73400b000 tx=0x7fa734007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:40.986+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa730042ce0 con 0x7fa7400b58f0 2022-07-20T03:07:41.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.110+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fa7400b2a50 con 0x7fa7400b58f0 2022-07-20T03:07:41.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa73d7fa700 1 -- 172.21.15.61:0/101190833 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fa730046300 con 0x7fa7400b58f0 2022-07-20T03:07:41.200 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa7200539b0 msgr2=0x7fa720055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa7200539b0 0x7fa720055e70 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fa73400b000 tx=0x7fa734007160).stop 2022-07-20T03:07:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 msgr2=0x7fa74013d970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 0x7fa74013d970 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fa730006940 tx=0x7fa730008040).stop 2022-07-20T03:07:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 shutdown_connections 2022-07-20T03:07:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa7200539b0 0x7fa720055e70 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa740143f60 0x7fa74013de60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa7400b58f0 0x7fa74013d970 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 --2- 172.21.15.61:0/101190833 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa7400aa9d0 0x7fa740147a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 >> 172.21.15.61:0/101190833 conn(0x7fa7400632b0 msgr2=0x7fa7400b15d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 shutdown_connections 2022-07-20T03:07:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.114+0000 7fa74546f700 1 -- 172.21.15.61:0/101190833 wait complete. 2022-07-20T03:07:41.207 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 Processor -- start 2022-07-20T03:07:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- start start 2022-07-20T03:07:41.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780b61d0 0x7fb7780b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780b20f0 con 0x7fb7780b13e0 2022-07-20T03:07:41.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7780b65b0 con 0x7fb7780af120 2022-07-20T03:07:41.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77813efd0 con 0x7fb7780b61d0 2022-07-20T03:07:41.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb7777fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb7780b13e0 0x7fb7780aea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39832/0 (socket says 172.21.15.61:39832) 2022-07-20T03:07:41.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb7777fe700 1 -- 172.21.15.61:0/1044862 learned_addr learned my addr 172.21.15.61:0/1044862 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:41.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb777fff700 1 --2- 172.21.15.61:0/1044862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780b61d0 0x7fb7780b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb776ffd700 1 -- 172.21.15.61:0/1044862 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 780109441 0 0) 0x7fb7780b20f0 con 0x7fb7780b13e0 2022-07-20T03:07:41.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb776ffd700 1 -- 172.21.15.61:0/1044862 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb764003260 con 0x7fb7780b13e0 2022-07-20T03:07:41.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb777fff700 1 -- 172.21.15.61:0/1044862 >> v1:172.21.15.117:6789/0 conn(0x7fb7780af120 legacy=0x7fb778067260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb777fff700 1 -- 172.21.15.61:0/1044862 >> v1:172.21.15.61:6789/0 conn(0x7fb7780b13e0 legacy=0x7fb7780aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb777fff700 1 -- 172.21.15.61:0/1044862 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7780b1ad0 con 0x7fb7780b61d0 2022-07-20T03:07:41.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb777fff700 1 --2- 172.21.15.61:0/1044862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780b61d0 0x7fb7780b0460 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb768002f70 tx=0x7fb768006df0).ready entity=mon.2 client_cookie=ba20d6709b3c9bc7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb776ffd700 1 -- 172.21.15.61:0/1044862 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb76800d570 con 0x7fb7780b61d0 2022-07-20T03:07:41.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb776ffd700 1 -- 172.21.15.61:0/1044862 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb768000e90 con 0x7fb7780b61d0 2022-07-20T03:07:41.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb776ffd700 1 -- 172.21.15.61:0/1044862 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb76800d8c0 con 0x7fb7780b61d0 2022-07-20T03:07:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- 172.21.15.61:0/1044862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780b61d0 msgr2=0x7fb7780b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 --2- 172.21.15.61:0/1044862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780b61d0 0x7fb7780b0460 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fb768002f70 tx=0x7fb768006df0).stop 2022-07-20T03:07:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- 172.21.15.61:0/1044862 shutdown_connections 2022-07-20T03:07:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 --2- 172.21.15.61:0/1044862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780b61d0 0x7fb7780b0460 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- 172.21.15.61:0/1044862 >> 172.21.15.61:0/1044862 conn(0x7fb778063cf0 msgr2=0x7fb7780b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- 172.21.15.61:0/1044862 shutdown_connections 2022-07-20T03:07:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.334+0000 7fb77d189700 1 -- 172.21.15.61:0/1044862 wait complete. 2022-07-20T03:07:41.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 Processor -- start 2022-07-20T03:07:41.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 -- start start 2022-07-20T03:07:41.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780af120 0x7fb7780b40c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 0x7fb7781396b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7780b61d0 0x7fb778139ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb778065500 con 0x7fb7780b13e0 2022-07-20T03:07:41.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb778065730 con 0x7fb7780b61d0 2022-07-20T03:07:41.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77d189700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb77813a090 con 0x7fb7780af120 2022-07-20T03:07:41.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb7777fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 0x7fb7781396b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb7777fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 0x7fb7781396b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43240/0 (socket says 172.21.15.61:43240) 2022-07-20T03:07:41.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb777fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780af120 0x7fb7780b40c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb7777fe700 1 -- 172.21.15.61:0/3767818940 learned_addr learned my addr 172.21.15.61:0/3767818940 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:41.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb77c988700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7780b61d0 0x7fb778139ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb777fff700 1 -- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7780b61d0 msgr2=0x7fb778139ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb777fff700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7780b61d0 0x7fb778139ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb777fff700 1 -- 172.21.15.61:0/3767818940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 msgr2=0x7fb7781396b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb777fff700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 0x7fb7781396b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.338+0000 7fb777fff700 1 -- 172.21.15.61:0/3767818940 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76800b180 con 0x7fb7780af120 2022-07-20T03:07:41.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7777fe700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 0x7fb7781396b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:41.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb777fff700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780af120 0x7fb7780b40c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb768002f70 tx=0x7fb768014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb768013de0 con 0x7fb7780af120 2022-07-20T03:07:41.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76800de60 con 0x7fb7780af120 2022-07-20T03:07:41.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7781555c0 con 0x7fb7780af120 2022-07-20T03:07:41.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb76801e5f0 con 0x7fb7780af120 2022-07-20T03:07:41.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb778065f10 con 0x7fb7780af120 2022-07-20T03:07:41.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb768025030 con 0x7fb7780af120 2022-07-20T03:07:41.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb778065ab0 con 0x7fb7780af120 2022-07-20T03:07:41.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7757fa700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb758053ad0 0x7fb758055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.342+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb76806cb70 con 0x7fb7780af120 2022-07-20T03:07:41.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.346+0000 7fb7777fe700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb758053ad0 0x7fb758055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.346+0000 7fb7777fe700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb758053ad0 0x7fb758055f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb76c00d000 tx=0x7fb76c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.350+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7680439c0 con 0x7fb7780af120 2022-07-20T03:07:41.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.482+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb778155750 con 0x7fb7780af120 2022-07-20T03:07:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.482+0000 7fb7757fa700 1 -- 172.21.15.61:0/3767818940 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fb768023020 con 0x7fb7780af120 2022-07-20T03:07:41.500 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:41.501 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:41.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb758053ad0 msgr2=0x7fb758055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb758053ad0 0x7fb758055f90 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fb76c00d000 tx=0x7fb76c009040).stop 2022-07-20T03:07:41.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780af120 msgr2=0x7fb7780b40c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780af120 0x7fb7780b40c0 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fb768002f70 tx=0x7fb768014000).stop 2022-07-20T03:07:41.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 shutdown_connections 2022-07-20T03:07:41.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb758053ad0 0x7fb758055f90 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7780b61d0 0x7fb778139ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7780b13e0 0x7fb7781396b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 --2- 172.21.15.61:0/3767818940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7780af120 0x7fb7780b40c0 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 >> 172.21.15.61:0/3767818940 conn(0x7fb778063cf0 msgr2=0x7fb7780b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:41.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 shutdown_connections 2022-07-20T03:07:41.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.486+0000 7fb77d189700 1 -- 172.21.15.61:0/3767818940 wait complete. 2022-07-20T03:07:41.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:41.536 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:41.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.661+0000 7fe02f727700 1 Processor -- start 2022-07-20T03:07:41.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.661+0000 7fe02f727700 1 -- start start 2022-07-20T03:07:41.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0280666a0 0x7fe0281461a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe028062cf0 0x7fe0281420d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0280637b0 con 0x7fe02805cf40 2022-07-20T03:07:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0280b2fe0 con 0x7fe028062cf0 2022-07-20T03:07:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0280b1df0 con 0x7fe0280666a0 2022-07-20T03:07:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe028062cf0 0x7fe0281420d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe028062cf0 0x7fe0281420d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43250/0 (socket says 172.21.15.61:43250) 2022-07-20T03:07:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 -- 172.21.15.61:0/2994751730 learned_addr learned my addr 172.21.15.61:0/2994751730 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02ccc2700 1 --2- 172.21.15.61:0/2994751730 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0280666a0 0x7fe0281461a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 -- 172.21.15.61:0/2994751730 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0280666a0 msgr2=0x7fe0281461a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 --2- 172.21.15.61:0/2994751730 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0280666a0 0x7fe0281461a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 -- 172.21.15.61:0/2994751730 >> v1:172.21.15.117:6789/0 conn(0x7fe02805cf40 legacy=0x7fe028065f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 -- 172.21.15.61:0/2994751730 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe028146690 con 0x7fe028062cf0 2022-07-20T03:07:41.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02dcc4700 1 --2- 172.21.15.61:0/2994751730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe028062cf0 0x7fe0281420d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe0200032f0 tx=0x7fe020009290).ready entity=mon.0 client_cookie=764519638bf902d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe01ffff700 1 -- 172.21.15.61:0/2994751730 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe02000b8a0 con 0x7fe028062cf0 2022-07-20T03:07:41.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe01ffff700 1 -- 172.21.15.61:0/2994751730 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02000ba00 con 0x7fe028062cf0 2022-07-20T03:07:41.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe01ffff700 1 -- 172.21.15.61:0/2994751730 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe02000bb70 con 0x7fe028062cf0 2022-07-20T03:07:41.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- 172.21.15.61:0/2994751730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe028062cf0 msgr2=0x7fe0281420d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 --2- 172.21.15.61:0/2994751730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe028062cf0 0x7fe0281420d0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe0200032f0 tx=0x7fe020009290).stop 2022-07-20T03:07:41.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- 172.21.15.61:0/2994751730 shutdown_connections 2022-07-20T03:07:41.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 --2- 172.21.15.61:0/2994751730 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe028062cf0 0x7fe0281420d0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 --2- 172.21.15.61:0/2994751730 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0280666a0 0x7fe0281461a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- 172.21.15.61:0/2994751730 >> 172.21.15.61:0/2994751730 conn(0x7fe0280b7d20 msgr2=0x7fe0280b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- 172.21.15.61:0/2994751730 shutdown_connections 2022-07-20T03:07:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.665+0000 7fe02f727700 1 -- 172.21.15.61:0/2994751730 wait complete. 2022-07-20T03:07:41.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 Processor -- start 2022-07-20T03:07:41.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 -- start start 2022-07-20T03:07:41.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 0x7fe028099a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 0x7fe0281523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0280666a0 0x7fe028152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02809c490 con 0x7fe02805cf40 2022-07-20T03:07:41.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02809a500 con 0x7fe0280666a0 2022-07-20T03:07:41.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02809a9b0 con 0x7fe028062cf0 2022-07-20T03:07:41.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02d4c3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 0x7fe028099a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 0x7fe0281523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02dcc4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0280666a0 0x7fe028152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 0x7fe0281523a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58158/0 (socket says 172.21.15.61:58158) 2022-07-20T03:07:41.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 -- 172.21.15.61:0/3760454329 learned_addr learned my addr 172.21.15.61:0/3760454329 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:41.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02d4c3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 0x7fe028099a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43252/0 (socket says 172.21.15.61:43252) 2022-07-20T03:07:41.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 -- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0280666a0 msgr2=0x7fe028152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0280666a0 0x7fe028152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 -- 172.21.15.61:0/3760454329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 msgr2=0x7fe028099a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 0x7fe028099a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 -- 172.21.15.61:0/3760454329 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe020009530 con 0x7fe028062cf0 2022-07-20T03:07:41.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02d4c3700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 0x7fe028099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:41.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02ccc2700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 0x7fe0281523a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe01800c000 tx=0x7fe018006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe018011750 con 0x7fe028062cf0 2022-07-20T03:07:41.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0180118b0 con 0x7fe028062cf0 2022-07-20T03:07:41.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe018011c10 con 0x7fe028062cf0 2022-07-20T03:07:41.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe028152d80 con 0x7fe028062cf0 2022-07-20T03:07:41.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.669+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0281462c0 con 0x7fe028062cf0 2022-07-20T03:07:41.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.673+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02809a700 con 0x7fe028062cf0 2022-07-20T03:07:41.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.673+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe018011d70 con 0x7fe028062cf0 2022-07-20T03:07:41.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.673+0000 7fe01e7fc700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053af0 0x7fe010055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:41.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.673+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe01806c380 con 0x7fe028062cf0 2022-07-20T03:07:41.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.673+0000 7fe02d4c3700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053af0 0x7fe010055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:41.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.673+0000 7fe02d4c3700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053af0 0x7fe010055fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe02400d000 tx=0x7fe024009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:41.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.681+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe0180431d0 con 0x7fe028062cf0 2022-07-20T03:07:41.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.809+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fe02809c660 con 0x7fe028062cf0 2022-07-20T03:07:41.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.809+0000 7fe01e7fc700 1 -- 172.21.15.61:0/3760454329 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7fe0180467f0 con 0x7fe028062cf0 2022-07-20T03:07:41.815 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:41.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053af0 msgr2=0x7fe010055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053af0 0x7fe010055fb0 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe02400d000 tx=0x7fe024009040).stop 2022-07-20T03:07:41.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 msgr2=0x7fe0281523a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:41.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 0x7fe0281523a0 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fe01800c000 tx=0x7fe018006df0).stop 2022-07-20T03:07:41.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 shutdown_connections 2022-07-20T03:07:41.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053af0 0x7fe010055fb0 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0280666a0 0x7fe028152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe028062cf0 0x7fe0281523a0 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 --2- 172.21.15.61:0/3760454329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02805cf40 0x7fe028099a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:41.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 >> 172.21.15.61:0/3760454329 conn(0x7fe0280b7d20 msgr2=0x7fe028143420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:41.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 shutdown_connections 2022-07-20T03:07:41.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.813+0000 7fe02f727700 1 -- 172.21.15.61:0/3760454329 wait complete. 2022-07-20T03:07:41.880 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:42.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 Processor -- start 2022-07-20T03:07:42.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 -- start start 2022-07-20T03:07:42.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d40b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f10d405d0c0 0x7f10d40aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f10d40adf70 con 0x7f10d40af920 2022-07-20T03:07:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10d40b5db0 con 0x7f10d405d0c0 2022-07-20T03:07:42.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d97f7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d4146300 con 0x7f10d40b59d0 2022-07-20T03:07:42.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d40b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d40b0040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58164/0 (socket says 172.21.15.61:58164) 2022-07-20T03:07:42.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 -- 172.21.15.61:0/3226824726 learned_addr learned my addr 172.21.15.61:0/3226824726 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:42.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d37fe700 1 --1- 172.21.15.61:0/3226824726 >> v1:172.21.15.117:6789/0 conn(0x7f10d40af920 0x7f10d40ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:48364/0 (socket says 172.21.15.61:48364) 2022-07-20T03:07:42.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d8ff6700 1 --2- 172.21.15.61:0/3226824726 >> v2:172.21.15.117:3300/0 conn(0x7f10d405d0c0 0x7f10d40aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 -- 172.21.15.61:0/3226824726 >> v2:172.21.15.117:3300/0 conn(0x7f10d405d0c0 msgr2=0x7f10d40aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:42.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 --2- 172.21.15.61:0/3226824726 >> v2:172.21.15.117:3300/0 conn(0x7f10d405d0c0 0x7f10d40aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 -- 172.21.15.61:0/3226824726 >> v1:172.21.15.117:6789/0 conn(0x7f10d40af920 legacy=0x7f10d40ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 -- 172.21.15.61:0/3226824726 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10d4146060 con 0x7f10d40b59d0 2022-07-20T03:07:42.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d3fff700 1 --2- 172.21.15.61:0/3226824726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d40b0040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f10d40add70 tx=0x7f10c4006eb0).ready entity=mon.2 client_cookie=1de0e199b67364d1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:41.997+0000 7f10d2ffd700 1 -- 172.21.15.61:0/3226824726 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10c400d5c0 con 0x7f10d40b59d0 2022-07-20T03:07:42.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d2ffd700 1 -- 172.21.15.61:0/3226824726 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10c4000e90 con 0x7f10d40b59d0 2022-07-20T03:07:42.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d2ffd700 1 -- 172.21.15.61:0/3226824726 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10c400d950 con 0x7f10d40b59d0 2022-07-20T03:07:42.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 -- 172.21.15.61:0/3226824726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 msgr2=0x7f10d40b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 --2- 172.21.15.61:0/3226824726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d40b0040 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f10d40add70 tx=0x7f10c4006eb0).stop 2022-07-20T03:07:42.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 -- 172.21.15.61:0/3226824726 shutdown_connections 2022-07-20T03:07:42.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 --2- 172.21.15.61:0/3226824726 >> v2:172.21.15.117:3300/0 conn(0x7f10d405d0c0 0x7f10d40aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 --2- 172.21.15.61:0/3226824726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d40b0040 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 -- 172.21.15.61:0/3226824726 >> 172.21.15.61:0/3226824726 conn(0x7f10d40634c0 msgr2=0x7f10d40b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:42.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 -- 172.21.15.61:0/3226824726 shutdown_connections 2022-07-20T03:07:42.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 -- 172.21.15.61:0/3226824726 wait complete. 2022-07-20T03:07:42.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 Processor -- start 2022-07-20T03:07:42.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 -- start start 2022-07-20T03:07:42.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.001+0000 7f10d97f7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10d405d0c0 0x7f10d4066480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 0x7f10d41554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d4147190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d4156280 con 0x7f10d40af920 2022-07-20T03:07:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10d4155d40 con 0x7f10d405d0c0 2022-07-20T03:07:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10d4155fe0 con 0x7f10d40b59d0 2022-07-20T03:07:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 0x7f10d41554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d8ff6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d4147190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 0x7f10d41554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43268/0 (socket says 172.21.15.61:43268) 2022-07-20T03:07:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 -- 172.21.15.61:0/1431179013 learned_addr learned my addr 172.21.15.61:0/1431179013 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d3fff700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10d405d0c0 0x7f10d4066480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 -- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 msgr2=0x7f10d4147190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d4147190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 -- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10d405d0c0 msgr2=0x7f10d4066480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10d405d0c0 0x7f10d4066480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 -- 172.21.15.61:0/1431179013 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f10c400b1e0 con 0x7f10d40af920 2022-07-20T03:07:42.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d3fff700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10d405d0c0 0x7f10d4066480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:42.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d8ff6700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d4147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d37fe700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 0x7f10d41554e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f10d405ce80 tx=0x7f10bc007ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10bc007170 con 0x7f10d40af920 2022-07-20T03:07:42.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10bc0072d0 con 0x7f10d40af920 2022-07-20T03:07:42.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f10bc004710 con 0x7f10d40af920 2022-07-20T03:07:42.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10d4066970 con 0x7f10d40af920 2022-07-20T03:07:42.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f10d41559d0 con 0x7f10d40af920 2022-07-20T03:07:42.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10d40af2d0 con 0x7f10d40af920 2022-07-20T03:07:42.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.005+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f10bc004870 con 0x7f10d40af920 2022-07-20T03:07:42.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.009+0000 7f10d17fa700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f10b8053920 0x7f10b8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.009+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f10bc06b5e0 con 0x7f10d40af920 2022-07-20T03:07:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.009+0000 7f10d3fff700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f10b8053920 0x7f10b8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.009+0000 7f10d3fff700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f10b8053920 0x7f10b8055de0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f10c401b000 tx=0x7f10c4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.013+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f10bc042540 con 0x7f10d40af920 2022-07-20T03:07:42.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.137+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f10d40574e0 con 0x7f10d40af920 2022-07-20T03:07:42.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.137+0000 7f10d17fa700 1 -- 172.21.15.61:0/1431179013 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7f10bc045b60 con 0x7f10d40af920 2022-07-20T03:07:42.276 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:42.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f10b8053920 msgr2=0x7f10b8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f10b8053920 0x7f10b8055de0 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f10c401b000 tx=0x7f10c4015000).stop 2022-07-20T03:07:42.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 msgr2=0x7f10d41554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 0x7f10d41554e0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f10d405ce80 tx=0x7f10bc007ba0).stop 2022-07-20T03:07:42.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 shutdown_connections 2022-07-20T03:07:42.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f10b8053920 0x7f10b8055de0 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10d40b59d0 0x7f10d4147190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10d40af920 0x7f10d41554e0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 --2- 172.21.15.61:0/1431179013 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10d405d0c0 0x7f10d4066480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 >> 172.21.15.61:0/1431179013 conn(0x7f10d40634c0 msgr2=0x7f10d41464d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:42.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 shutdown_connections 2022-07-20T03:07:42.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.141+0000 7f10d97f7700 1 -- 172.21.15.61:0/1431179013 wait complete. 2022-07-20T03:07:42.306 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:42.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cdbb03700 1 Processor -- start 2022-07-20T03:07:42.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cdbb03700 1 -- start start 2022-07-20T03:07:42.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cdbb03700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cd4145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cdbb03700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd40607e0 con 0x7f1cd4060ee0 2022-07-20T03:07:42.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cdbb03700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1cd4060a80 con 0x7f1cd40b4d60 2022-07-20T03:07:42.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cdbb03700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4060d20 con 0x7f1cd409df40 2022-07-20T03:07:42.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1cd4060ee0 0x7f1cd409cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39876/0 (socket says 172.21.15.61:39876) 2022-07-20T03:07:42.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 -- 172.21.15.61:0/193743942 learned_addr learned my addr 172.21.15.61:0/193743942 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:42.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd989f700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cd4145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1677628170 0 0) 0x7f1cd40607e0 con 0x7f1cd4060ee0 2022-07-20T03:07:42.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1cbc003260 con 0x7f1cd4060ee0 2022-07-20T03:07:42.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1896452524 0 0) 0x7f1cbc003260 con 0x7f1cd4060ee0 2022-07-20T03:07:42.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 msgr2=0x7f1cd4145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:42.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cd4145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 >> v1:172.21.15.117:6789/0 conn(0x7f1cd40b4d60 legacy=0x7f1cd4145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd409e320 con 0x7f1cd4060ee0 2022-07-20T03:07:42.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f1ccc0019e0 con 0x7f1cd4060ee0 2022-07-20T03:07:42.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 >> v1:172.21.15.61:6789/0 conn(0x7f1cd4060ee0 legacy=0x7f1cd409cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cbc003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cbc003c70 0x7f1cbc004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cbc0049f0 0x7f1cbc004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd4060d20 con 0x7f1cbc0049f0 2022-07-20T03:07:42.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc00b750 con 0x7f1cbc003c70 2022-07-20T03:07:42.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cbc00b910 con 0x7f1cd409df40 2022-07-20T03:07:42.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cbc0049f0 0x7f1cbc004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd909e700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cbc003c70 0x7f1cbc004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cd989f700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cbc003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 -- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 msgr2=0x7f1cbc003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:42.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cbc003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 -- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cbc003c70 msgr2=0x7f1cbc004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:42.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cbc003c70 0x7f1cbc004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.429+0000 7f1cda0a0700 1 -- 172.21.15.61:0/193743942 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cbc003260 con 0x7f1cbc0049f0 2022-07-20T03:07:42.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cd989f700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd409df40 0x7f1cbc003780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:42.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cbc0049f0 0x7f1cbc004e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1ccc0016f0 tx=0x7f1cd409ca60).ready entity=mon.0 client_cookie=b146d8d67f733489 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ccc00cd70 con 0x7f1cbc0049f0 2022-07-20T03:07:42.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc005990 con 0x7f1cbc0049f0 2022-07-20T03:07:42.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cd889d700 1 -- 172.21.15.61:0/193743942 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ccc005cf0 con 0x7f1cbc0049f0 2022-07-20T03:07:42.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- 172.21.15.61:0/193743942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cbc0049f0 msgr2=0x7f1cbc004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/193743942 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cbc0049f0 0x7f1cbc004e70 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f1ccc0016f0 tx=0x7f1cd409ca60).stop 2022-07-20T03:07:42.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 -- 172.21.15.61:0/193743942 reap_dead start 2022-07-20T03:07:42.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- 172.21.15.61:0/193743942 shutdown_connections 2022-07-20T03:07:42.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- 172.21.15.61:0/193743942 >> 172.21.15.61:0/193743942 conn(0x7f1cd40b5f30 msgr2=0x7f1cd40b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:42.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- 172.21.15.61:0/193743942 shutdown_connections 2022-07-20T03:07:42.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- 172.21.15.61:0/193743942 wait complete. 2022-07-20T03:07:42.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 Processor -- start 2022-07-20T03:07:42.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- start start 2022-07-20T03:07:42.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd4149b90 0x7f1cd40b21a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd40b0e90 0x7f1cd4155d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 0x7f1cd40af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd408ffe0 con 0x7f1cd4156220 2022-07-20T03:07:42.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd41566a0 con 0x7f1cd40b0e90 2022-07-20T03:07:42.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cdbb03700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1cd40b18b0 con 0x7f1cd4149b90 2022-07-20T03:07:42.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 0x7f1cd40af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 0x7f1cd40af5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43290/0 (socket says 172.21.15.61:43290) 2022-07-20T03:07:42.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 -- 172.21.15.61:0/614056243 learned_addr learned my addr 172.21.15.61:0/614056243 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:42.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cd989f700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd4149b90 0x7f1cd40b21a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cd909e700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd40b0e90 0x7f1cd4155d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 -- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd4149b90 msgr2=0x7f1cd40b21a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd4149b90 0x7f1cd40b21a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 -- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd40b0e90 msgr2=0x7f1cd4155d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.433+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd40b0e90 0x7f1cd4155d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cda0a0700 1 -- 172.21.15.61:0/614056243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cd40b2690 con 0x7f1cd4156220 2022-07-20T03:07:42.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cda0a0700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 0x7f1cd40af5d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1cd4155c10 tx=0x7f1ccc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ccc001f70 con 0x7f1cd4156220 2022-07-20T03:07:42.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ccc0074d0 con 0x7f1cd4156220 2022-07-20T03:07:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cd4149f70 con 0x7f1cd4156220 2022-07-20T03:07:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ccc007830 con 0x7f1cd4156220 2022-07-20T03:07:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cd40b1310 con 0x7f1cd4156220 2022-07-20T03:07:42.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cd40574e0 con 0x7f1cd4156220 2022-07-20T03:07:42.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1ccc010070 con 0x7f1cd4156220 2022-07-20T03:07:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cc2ffd700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1cb40539b0 0x7f1cb4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.437+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1ccc06ca30 con 0x7f1cd4156220 2022-07-20T03:07:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.441+0000 7f1cd989f700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1cb40539b0 0x7f1cb4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.441+0000 7f1cd989f700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1cb40539b0 0x7f1cb4055e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1cd0006000 tx=0x7f1cd0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.445+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ccc043990 con 0x7f1cd4156220 2022-07-20T03:07:42.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.569+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f1cd40b2690 con 0x7f1cd4156220 2022-07-20T03:07:42.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.569+0000 7f1cc2ffd700 1 -- 172.21.15.61:0/614056243 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f1cd40b2690 con 0x7f1cd4156220 2022-07-20T03:07:42.652 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:42.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1cb40539b0 msgr2=0x7f1cb4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1cb40539b0 0x7f1cb4055e70 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f1cd0006000 tx=0x7f1cd0008040).stop 2022-07-20T03:07:42.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 msgr2=0x7f1cd40af5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 0x7f1cd40af5d0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f1cd4155c10 tx=0x7f1ccc015000).stop 2022-07-20T03:07:42.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 shutdown_connections 2022-07-20T03:07:42.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1cb40539b0 0x7f1cb4055e70 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1cd40b0e90 0x7f1cd4155d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1cd4156220 0x7f1cd40af5d0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 --2- 172.21.15.61:0/614056243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1cd4149b90 0x7f1cd40b21a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 >> 172.21.15.61:0/614056243 conn(0x7f1cd40b5f30 msgr2=0x7f1cd405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:42.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 shutdown_connections 2022-07-20T03:07:42.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.573+0000 7f1cdbb03700 1 -- 172.21.15.61:0/614056243 wait complete. 2022-07-20T03:07:42.664 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:42.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.773+0000 7fb0ab00c700 1 Processor -- start 2022-07-20T03:07:42.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.773+0000 7fb0ab00c700 1 -- start start 2022-07-20T03:07:42.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0a40b59d0 0x7fb0a40b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0a40b14f0 0x7fb0a405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a4148080 con 0x7fb0a40b59d0 2022-07-20T03:07:42.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0a40afbc0 con 0x7fb0a40b14f0 2022-07-20T03:07:42.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a40b00e0 con 0x7fb0a414ada0 2022-07-20T03:07:42.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0a40b59d0 0x7fb0a40b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0a40b59d0 0x7fb0a40b07e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43296/0 (socket says 172.21.15.61:43296) 2022-07-20T03:07:42.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 -- 172.21.15.61:0/416044413 learned_addr learned my addr 172.21.15.61:0/416044413 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:42.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0a9809700 1 --2- 172.21.15.61:0/416044413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa80b700 1 --2- 172.21.15.61:0/416044413 >> v2:172.21.15.117:3300/0 conn(0x7fb0a40b14f0 0x7fb0a405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 -- 172.21.15.61:0/416044413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 msgr2=0x7fb0a40b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:42.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 --2- 172.21.15.61:0/416044413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 -- 172.21.15.61:0/416044413 >> v2:172.21.15.117:3300/0 conn(0x7fb0a40b14f0 msgr2=0x7fb0a405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:42.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 --2- 172.21.15.61:0/416044413 >> v2:172.21.15.117:3300/0 conn(0x7fb0a40b14f0 0x7fb0a405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 -- 172.21.15.61:0/416044413 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0a40b5db0 con 0x7fb0a40b59d0 2022-07-20T03:07:42.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0aa00a700 1 --2- 172.21.15.61:0/416044413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a40b07e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fb094002f70 tx=0x7fb094006eb0).ready entity=mon.0 client_cookie=1e418581c6427e44 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0a9008700 1 -- 172.21.15.61:0/416044413 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09400d570 con 0x7fb0a40b59d0 2022-07-20T03:07:42.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0a9008700 1 -- 172.21.15.61:0/416044413 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb094000e90 con 0x7fb0a40b59d0 2022-07-20T03:07:42.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0a9008700 1 -- 172.21.15.61:0/416044413 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb09400d900 con 0x7fb0a40b59d0 2022-07-20T03:07:42.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 -- 172.21.15.61:0/416044413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 msgr2=0x7fb0a40b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.777+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/416044413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a40b07e0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fb094002f70 tx=0x7fb094006eb0).stop 2022-07-20T03:07:42.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- 172.21.15.61:0/416044413 shutdown_connections 2022-07-20T03:07:42.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/416044413 >> v2:172.21.15.117:3300/0 conn(0x7fb0a40b14f0 0x7fb0a405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/416044413 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/416044413 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a40b07e0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- 172.21.15.61:0/416044413 >> 172.21.15.61:0/416044413 conn(0x7fb0a40634c0 msgr2=0x7fb0a40b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:42.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- 172.21.15.61:0/416044413 shutdown_connections 2022-07-20T03:07:42.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- 172.21.15.61:0/416044413 wait complete. 2022-07-20T03:07:42.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 Processor -- start 2022-07-20T03:07:42.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- start start 2022-07-20T03:07:42.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a40b14f0 0x7fb0a40669b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a410ffb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b3fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4065c50 con 0x7fb0a40b59d0 2022-07-20T03:07:42.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4065ef0 con 0x7fb0a40b14f0 2022-07-20T03:07:42.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0ab00c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0a4155f00 con 0x7fb0a414ada0 2022-07-20T03:07:42.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0a9809700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a410ffb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0a9809700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a410ffb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43302/0 (socket says 172.21.15.61:43302) 2022-07-20T03:07:42.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0a9809700 1 -- 172.21.15.61:0/4244937701 learned_addr learned my addr 172.21.15.61:0/4244937701 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:42.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.781+0000 7fb0aa00a700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a40b14f0 0x7fb0a40669b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0aa80b700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b3fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0a9809700 1 -- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 msgr2=0x7fb0a40b3fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0a9809700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b3fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0a9809700 1 -- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a40b14f0 msgr2=0x7fb0a40669b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0a9809700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a40b14f0 0x7fb0a40669b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0a9809700 1 -- 172.21.15.61:0/4244937701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb09400b1e0 con 0x7fb0a40b59d0 2022-07-20T03:07:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0a9809700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a410ffb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb0a0006e00 tx=0x7fb0a00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a000ee60 con 0x7fb0a40b59d0 2022-07-20T03:07:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0a000c850 con 0x7fb0a40b59d0 2022-07-20T03:07:42.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0a40b44a0 con 0x7fb0a40b59d0 2022-07-20T03:07:42.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0a000cbb0 con 0x7fb0a40b59d0 2022-07-20T03:07:42.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0a40658d0 con 0x7fb0a40b59d0 2022-07-20T03:07:42.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0a4066060 con 0x7fb0a40b59d0 2022-07-20T03:07:42.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.785+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb0a0022070 con 0x7fb0a40b59d0 2022-07-20T03:07:42.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.789+0000 7fb0937fe700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0840539b0 0x7fb084055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:42.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.789+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb0a006c5a0 con 0x7fb0a40b59d0 2022-07-20T03:07:42.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.789+0000 7fb0aa00a700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0840539b0 0x7fb084055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:42.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.789+0000 7fb0aa00a700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0840539b0 0x7fb084055e70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb09401c000 tx=0x7fb094015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:42.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.793+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0a4066060 con 0x7fb0a40b59d0 2022-07-20T03:07:42.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.921+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb0a4066060 con 0x7fb0a40b59d0 2022-07-20T03:07:42.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.921+0000 7fb0937fe700 1 -- 172.21.15.61:0/4244937701 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb0a4066060 con 0x7fb0a40b59d0 2022-07-20T03:07:42.929 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:42.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.921+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0840539b0 msgr2=0x7fb084055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.921+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0840539b0 0x7fb084055e70 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fb09401c000 tx=0x7fb094015000).stop 2022-07-20T03:07:42.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.921+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 msgr2=0x7fb0a410ffb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:42.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.921+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a410ffb0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fb0a0006e00 tx=0x7fb0a00091d0).stop 2022-07-20T03:07:42.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 shutdown_connections 2022-07-20T03:07:42.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0840539b0 0x7fb084055e70 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0a414ada0 0x7fb0a40b3fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0a40b59d0 0x7fb0a410ffb0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 --2- 172.21.15.61:0/4244937701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0a40b14f0 0x7fb0a40669b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:42.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 >> 172.21.15.61:0/4244937701 conn(0x7fb0a40634c0 msgr2=0x7fb0a4146160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:42.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 shutdown_connections 2022-07-20T03:07:42.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:42.925+0000 7fb0ab00c700 1 -- 172.21.15.61:0/4244937701 wait complete. 2022-07-20T03:07:42.964 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:07:43.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.085+0000 7f79e8a44700 1 Processor -- start 2022-07-20T03:07:43.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.085+0000 7f79e8a44700 1 -- start start 2022-07-20T03:07:43.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e8a44700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e41424e0 0x7f79e40b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e8a44700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f79e40aea10 0x7f79e40ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e8a44700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e41433d0 con 0x7f79e40b0570 2022-07-20T03:07:43.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e8a44700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e4143b10 con 0x7f79e41424e0 2022-07-20T03:07:43.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e8a44700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e4143670 con 0x7f79e40aea10 2022-07-20T03:07:43.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79dbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e41424e0 0x7f79e40b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f79e40b0570 0x7f79e4141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39906/0 (socket says 172.21.15.61:39906) 2022-07-20T03:07:43.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e37fe700 1 -- 172.21.15.61:0/3160384295 learned_addr learned my addr 172.21.15.61:0/3160384295 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:43.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e3fff700 1 --2- 172.21.15.61:0/3160384295 >> v2:172.21.15.117:3300/0 conn(0x7f79e40aea10 0x7f79e40ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79dbfff700 1 -- 172.21.15.61:0/3160384295 >> v2:172.21.15.117:3300/0 conn(0x7f79e40aea10 msgr2=0x7f79e40ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:43.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79dbfff700 1 --2- 172.21.15.61:0/3160384295 >> v2:172.21.15.117:3300/0 conn(0x7f79e40aea10 0x7f79e40ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79dbfff700 1 -- 172.21.15.61:0/3160384295 >> v1:172.21.15.61:6789/0 conn(0x7f79e40b0570 legacy=0x7f79e4141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79dbfff700 1 -- 172.21.15.61:0/3160384295 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e40b1010 con 0x7f79e41424e0 2022-07-20T03:07:43.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e2ffd700 1 -- 172.21.15.61:0/3160384295 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 647532474 0 0) 0x7f79e41433d0 con 0x7f79e40b0570 2022-07-20T03:07:43.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79dbfff700 1 --2- 172.21.15.61:0/3160384295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b0950 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f79cc002f70 tx=0x7f79cc006df0).ready entity=mon.0 client_cookie=627d940cbb10de62 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:43.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e2ffd700 1 -- 172.21.15.61:0/3160384295 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79cc00d570 con 0x7f79e41424e0 2022-07-20T03:07:43.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e2ffd700 1 -- 172.21.15.61:0/3160384295 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79cc000e90 con 0x7f79e41424e0 2022-07-20T03:07:43.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.089+0000 7f79e2ffd700 1 -- 172.21.15.61:0/3160384295 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79cc00d8c0 con 0x7f79e41424e0 2022-07-20T03:07:43.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 -- 172.21.15.61:0/3160384295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 msgr2=0x7f79e40b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3160384295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b0950 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f79cc002f70 tx=0x7f79cc006df0).stop 2022-07-20T03:07:43.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 -- 172.21.15.61:0/3160384295 shutdown_connections 2022-07-20T03:07:43.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3160384295 >> v2:172.21.15.117:3300/0 conn(0x7f79e40aea10 0x7f79e40ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3160384295 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b0950 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 -- 172.21.15.61:0/3160384295 >> 172.21.15.61:0/3160384295 conn(0x7f79e40b9360 msgr2=0x7f79e40b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:43.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 -- 172.21.15.61:0/3160384295 shutdown_connections 2022-07-20T03:07:43.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 -- 172.21.15.61:0/3160384295 wait complete. 2022-07-20T03:07:43.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 Processor -- start 2022-07-20T03:07:43.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.093+0000 7f79e8a44700 1 -- start start 2022-07-20T03:07:43.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e40aea10 0x7f79e405c130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e40b0570 0x7f79e40b31c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b36b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e40b1e80 con 0x7f79e41424e0 2022-07-20T03:07:43.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e4146830 con 0x7f79e40aea10 2022-07-20T03:07:43.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e4145430 con 0x7f79e40b0570 2022-07-20T03:07:43.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b36b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b36b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43318/0 (socket says 172.21.15.61:43318) 2022-07-20T03:07:43.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 -- 172.21.15.61:0/3186179084 learned_addr learned my addr 172.21.15.61:0/3186179084 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:43.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e37fe700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e40aea10 0x7f79e405c130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79dbfff700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e40b0570 0x7f79e40b31c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 -- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e40b0570 msgr2=0x7f79e40b31c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e40b0570 0x7f79e40b31c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 -- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e40aea10 msgr2=0x7f79e405c130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e40aea10 0x7f79e405c130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 -- 172.21.15.61:0/3186179084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79cc00b180 con 0x7f79e41424e0 2022-07-20T03:07:43.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e3fff700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b36b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f79dc006e00 tx=0x7f79dc0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:43.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79dc00c640 con 0x7f79e41424e0 2022-07-20T03:07:43.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc00c7a0 con 0x7f79e41424e0 2022-07-20T03:07:43.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e41458d0 con 0x7f79e41424e0 2022-07-20T03:07:43.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79dc00cb00 con 0x7f79e41424e0 2022-07-20T03:07:43.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.097+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e4145bc0 con 0x7f79e41424e0 2022-07-20T03:07:43.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.101+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e41455f0 con 0x7f79e41424e0 2022-07-20T03:07:43.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.101+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f79dc022070 con 0x7f79e41424e0 2022-07-20T03:07:43.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.101+0000 7f79e17fa700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c8053ad0 0x7f79c8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.101+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f79dc06c460 con 0x7f79e41424e0 2022-07-20T03:07:43.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.101+0000 7f79e37fe700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c8053ad0 0x7f79c8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.101+0000 7f79e37fe700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c8053ad0 0x7f79c8055f90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f79d400d000 tx=0x7f79d4007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:43.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.109+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79dc0433c0 con 0x7f79e41424e0 2022-07-20T03:07:43.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.229+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f79e40ace30 con 0x7f79e41424e0 2022-07-20T03:07:43.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e17fa700 1 -- 172.21.15.61:0/3186179084 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v23) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f79dc0469e0 con 0x7f79e41424e0 2022-07-20T03:07:43.259 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:43.260 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:12.740931+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":3,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:07:08.880443+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"21","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":4,"pool_name":"cephfs_data","create_time":"2022-07-20T03:07:09.560512+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"23","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:07:43.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c8053ad0 msgr2=0x7f79c8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c8053ad0 0x7f79c8055f90 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f79d400d000 tx=0x7f79d4007a00).stop 2022-07-20T03:07:43.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 msgr2=0x7f79e40b36b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b36b0 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f79dc006e00 tx=0x7f79dc0091d0).stop 2022-07-20T03:07:43.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 shutdown_connections 2022-07-20T03:07:43.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c8053ad0 0x7f79c8055f90 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e41424e0 0x7f79e40b36b0 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e40b0570 0x7f79e40b31c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 --2- 172.21.15.61:0/3186179084 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e40aea10 0x7f79e405c130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 >> 172.21.15.61:0/3186179084 conn(0x7f79e40b9360 msgr2=0x7f79e41410b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:43.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 shutdown_connections 2022-07-20T03:07:43.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.233+0000 7f79e8a44700 1 -- 172.21.15.61:0/3186179084 wait complete. 2022-07-20T03:07:43.337 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 3, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:07:08.880443+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '21', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 4, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:07:09.560512+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '23', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:07:43.340 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:07:43.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.497+0000 7f4938826700 1 Processor -- start 2022-07-20T03:07:43.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.497+0000 7f4938826700 1 -- start start 2022-07-20T03:07:43.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49341468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f49340b3cf0 0x7f4934142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4934142f40 0x7f493405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49340ad5b0 con 0x7f4934142f40 2022-07-20T03:07:43.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49340ba180 con 0x7f49340b3cf0 2022-07-20T03:07:43.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49341461b0 con 0x7f49340b9da0 2022-07-20T03:07:43.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4934142f40 0x7f493405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4934142f40 0x7f493405bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43326/0 (socket says 172.21.15.61:43326) 2022-07-20T03:07:43.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 -- 172.21.15.61:0/3343655667 learned_addr learned my addr 172.21.15.61:0/3343655667 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:43.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4932ffd700 1 --2- 172.21.15.61:0/3343655667 >> v2:172.21.15.117:3300/0 conn(0x7f49340b3cf0 0x7f4934142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f49337fe700 1 --2- 172.21.15.61:0/3343655667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49341468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 -- 172.21.15.61:0/3343655667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 msgr2=0x7f49341468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:43.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 --2- 172.21.15.61:0/3343655667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49341468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 -- 172.21.15.61:0/3343655667 >> v2:172.21.15.117:3300/0 conn(0x7f49340b3cf0 msgr2=0x7f4934142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:43.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 --2- 172.21.15.61:0/3343655667 >> v2:172.21.15.117:3300/0 conn(0x7f49340b3cf0 0x7f4934142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 -- 172.21.15.61:0/3343655667 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4934146450 con 0x7f4934142f40 2022-07-20T03:07:43.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4932ffd700 1 --2- 172.21.15.61:0/3343655667 >> v2:172.21.15.117:3300/0 conn(0x7f49340b3cf0 0x7f4934142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:43.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4933fff700 1 --2- 172.21.15.61:0/3343655667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f493405bd90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4924002f70 tx=0x7f4924006eb0).ready entity=mon.0 client_cookie=e5372294f8a5fb53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:43.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f49327fc700 1 -- 172.21.15.61:0/3343655667 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f492400d570 con 0x7f4934142f40 2022-07-20T03:07:43.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f49327fc700 1 -- 172.21.15.61:0/3343655667 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924000e90 con 0x7f4934142f40 2022-07-20T03:07:43.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f49327fc700 1 -- 172.21.15.61:0/3343655667 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f492400d900 con 0x7f4934142f40 2022-07-20T03:07:43.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- 172.21.15.61:0/3343655667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 msgr2=0x7f493405bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- 172.21.15.61:0/3343655667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f493405bd90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f4924002f70 tx=0x7f4924006eb0).stop 2022-07-20T03:07:43.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- 172.21.15.61:0/3343655667 shutdown_connections 2022-07-20T03:07:43.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- 172.21.15.61:0/3343655667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f493405bd90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- 172.21.15.61:0/3343655667 >> v2:172.21.15.117:3300/0 conn(0x7f49340b3cf0 0x7f4934142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 --2- 172.21.15.61:0/3343655667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49341468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- 172.21.15.61:0/3343655667 >> 172.21.15.61:0/3343655667 conn(0x7f49340678c0 msgr2=0x7f49340b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:43.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- 172.21.15.61:0/3343655667 shutdown_connections 2022-07-20T03:07:43.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.501+0000 7f4938826700 1 -- 172.21.15.61:0/3343655667 wait complete. 2022-07-20T03:07:43.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 Processor -- start 2022-07-20T03:07:43.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 -- start start 2022-07-20T03:07:43.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49340b3cf0 0x7f4934113de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49340b7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f49340af060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49340af550 con 0x7f4934142f40 2022-07-20T03:07:43.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f49340b2860 con 0x7f49340b3cf0 2022-07-20T03:07:43.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4938826700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f49340b2b00 con 0x7f49340b9da0 2022-07-20T03:07:43.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4933fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f49340af060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4932ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49340b7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4932ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49340b7f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58234/0 (socket says 172.21.15.61:58234) 2022-07-20T03:07:43.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f4932ffd700 1 -- 172.21.15.61:0/608911426 learned_addr learned my addr 172.21.15.61:0/608911426 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:43.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.505+0000 7f49337fe700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49340b3cf0 0x7f4934113de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4933fff700 1 -- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 msgr2=0x7f49340b7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4933fff700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49340b7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4933fff700 1 -- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49340b3cf0 msgr2=0x7f4934113de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4933fff700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49340b3cf0 0x7f4934113de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4933fff700 1 -- 172.21.15.61:0/608911426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c00b180 con 0x7f4934142f40 2022-07-20T03:07:43.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4933fff700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f49340af060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4924002f70 tx=0x7f4924015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:43.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49240116e0 con 0x7f4934142f40 2022-07-20T03:07:43.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4924011840 con 0x7f4934142f40 2022-07-20T03:07:43.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f492400b1e0 con 0x7f4934142f40 2022-07-20T03:07:43.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4924011ba0 con 0x7f4934142f40 2022-07-20T03:07:43.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f49340b1db0 con 0x7f4934142f40 2022-07-20T03:07:43.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.509+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f49340574e0 con 0x7f4934142f40 2022-07-20T03:07:43.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.513+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f492401a040 con 0x7f4934142f40 2022-07-20T03:07:43.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.513+0000 7f4930ff9700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4918053af0 0x7f4918055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.513+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f492406ca60 con 0x7f4934142f40 2022-07-20T03:07:43.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.513+0000 7f49337fe700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4918053af0 0x7f4918055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.513+0000 7f49337fe700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4918053af0 0x7f4918055fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f492c00b000 tx=0x7f492c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:43.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.521+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f49240439c0 con 0x7f4934142f40 2022-07-20T03:07:43.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.640+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f49340699b0 con 0x7f4934142f40 2022-07-20T03:07:43.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.640+0000 7f4930ff9700 1 -- 172.21.15.61:0/608911426 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v23) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4924046fe0 con 0x7f4934142f40 2022-07-20T03:07:43.644 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:07:43.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4918053af0 msgr2=0x7f4918055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4918053af0 0x7f4918055fb0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f492c00b000 tx=0x7f492c0091d0).stop 2022-07-20T03:07:43.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 msgr2=0x7f49340af060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:43.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f49340af060 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f4924002f70 tx=0x7f4924015000).stop 2022-07-20T03:07:43.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 shutdown_connections 2022-07-20T03:07:43.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4918053af0 0x7f4918055fb0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4934142f40 0x7f49340af060 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f49340b9da0 0x7f49340b7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 --2- 172.21.15.61:0/608911426 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49340b3cf0 0x7f4934113de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:43.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 >> 172.21.15.61:0/608911426 conn(0x7f49340678c0 msgr2=0x7f49341478d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:43.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 shutdown_connections 2022-07-20T03:07:43.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.644+0000 7f4938826700 1 -- 172.21.15.61:0/608911426 wait complete. 2022-07-20T03:07:43.805 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:07:43.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.916+0000 7fc923587700 1 Processor -- start 2022-07-20T03:07:43.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.916+0000 7fc923587700 1 -- start start 2022-07-20T03:07:43.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc91c0978f0 0x7fc91c1400e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:43.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91c0ac980 con 0x7fc91c0ac5a0 2022-07-20T03:07:43.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc91c0b2210 con 0x7fc91c0b29a0 2022-07-20T03:07:43.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc91c063660 con 0x7fc91c0978f0 2022-07-20T03:07:43.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc921323700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc91c0ac5a0 0x7fc91c0b3520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39934/0 (socket says 172.21.15.61:39934) 2022-07-20T03:07:43.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc921323700 1 -- 172.21.15.61:0/3875152641 learned_addr learned my addr 172.21.15.61:0/3875152641 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:43.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc920b22700 1 --2- 172.21.15.61:0/3875152641 >> v2:172.21.15.117:3300/0 conn(0x7fc91c0978f0 0x7fc91c1400e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:43.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 209354064 0 0) 0x7fc91c0ac980 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc900003260 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4261893779 0 0) 0x7fc91c0b2210 con 0x7fc91c0b29a0 2022-07-20T03:07:44.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc91c0ac980 con 0x7fc91c0b29a0 2022-07-20T03:07:44.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc920b22700 1 -- 172.21.15.61:0/3875152641 >> v1:172.21.15.117:6789/0 conn(0x7fc91c0b29a0 legacy=0x7fc91c1405d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc920b22700 1 -- 172.21.15.61:0/3875152641 >> v1:172.21.15.61:6789/0 conn(0x7fc91c0ac5a0 legacy=0x7fc91c0b3520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc920b22700 1 -- 172.21.15.61:0/3875152641 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91c0b3080 con 0x7fc91c0978f0 2022-07-20T03:07:44.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc920b22700 1 --2- 172.21.15.61:0/3875152641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c0978f0 0x7fc91c1400e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc918002f70 tx=0x7fc918006df0).ready entity=mon.1 client_cookie=8165de76884b4207 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc91800d540 con 0x7fc91c0978f0 2022-07-20T03:07:44.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc918000e90 con 0x7fc91c0978f0 2022-07-20T03:07:44.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc913fff700 1 -- 172.21.15.61:0/3875152641 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc91800d890 con 0x7fc91c0978f0 2022-07-20T03:07:44.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- 172.21.15.61:0/3875152641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c0978f0 msgr2=0x7fc91c1400e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 --2- 172.21.15.61:0/3875152641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c0978f0 0x7fc91c1400e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fc918002f70 tx=0x7fc918006df0).stop 2022-07-20T03:07:44.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- 172.21.15.61:0/3875152641 shutdown_connections 2022-07-20T03:07:44.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 --2- 172.21.15.61:0/3875152641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c0978f0 0x7fc91c1400e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- 172.21.15.61:0/3875152641 >> 172.21.15.61:0/3875152641 conn(0x7fc91c0b47a0 msgr2=0x7fc91c05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:44.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- 172.21.15.61:0/3875152641 shutdown_connections 2022-07-20T03:07:44.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.920+0000 7fc923587700 1 -- 172.21.15.61:0/3875152641 wait complete. 2022-07-20T03:07:44.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 Processor -- start 2022-07-20T03:07:44.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 -- start start 2022-07-20T03:07:44.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 0x7fc91c0661f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc91c0b29a0 0x7fc91c09a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c09d050 0x7fc91c144020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c144d40 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c09c830 con 0x7fc91c09d050 2022-07-20T03:07:44.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc91c1424b0 con 0x7fc91c0b29a0 2022-07-20T03:07:44.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 0x7fc91c0661f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 0x7fc91c0661f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43342/0 (socket says 172.21.15.61:43342) 2022-07-20T03:07:44.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 -- 172.21.15.61:0/3845015918 learned_addr learned my addr 172.21.15.61:0/3845015918 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:44.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc920b22700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc91c0b29a0 0x7fc91c09a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921b24700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c09d050 0x7fc91c144020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 -- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc91c0b29a0 msgr2=0x7fc91c09a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc91c0b29a0 0x7fc91c09a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 -- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c09d050 msgr2=0x7fc91c144020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c09d050 0x7fc91c144020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 -- 172.21.15.61:0/3845015918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc91800b180 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921b24700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c09d050 0x7fc91c144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:44.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc920b22700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc91c0b29a0 0x7fc91c09a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:44.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc921323700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 0x7fc91c0661f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fc90c0011d0 tx=0x7fc90c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc90c00ce70 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc90c0072e0 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc91c143000 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc90c005800 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.924+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc91c144730 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.928+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc91c09c9f0 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.928+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc90c010070 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.928+0000 7fc9127fc700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9040539b0 0x7fc904055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.928+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc90c06b650 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.928+0000 7fc920b22700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9040539b0 0x7fc904055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.928+0000 7fc920b22700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9040539b0 0x7fc904055e70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc918018000 tx=0x7fc918014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:43.936+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc90c0425b0 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.056+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc91c0b26f0 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.056+0000 7fc9127fc700 1 -- 172.21.15.61:0/3845015918 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v23) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc90c045bd0 con 0x7fc91c0ac5a0 2022-07-20T03:07:44.064 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:44.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9040539b0 msgr2=0x7fc904055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9040539b0 0x7fc904055e70 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fc918018000 tx=0x7fc918014000).stop 2022-07-20T03:07:44.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 msgr2=0x7fc91c0661f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 0x7fc91c0661f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fc90c0011d0 tx=0x7fc90c007b50).stop 2022-07-20T03:07:44.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 shutdown_connections 2022-07-20T03:07:44.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9040539b0 0x7fc904055e70 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc91c09d050 0x7fc91c144020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc91c0b29a0 0x7fc91c09a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 --2- 172.21.15.61:0/3845015918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc91c0ac5a0 0x7fc91c0661f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 >> 172.21.15.61:0/3845015918 conn(0x7fc91c0b47a0 msgr2=0x7fc91c097e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:44.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 shutdown_connections 2022-07-20T03:07:44.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.060+0000 7fc923587700 1 -- 172.21.15.61:0/3845015918 wait complete. 2022-07-20T03:07:44.138 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:07:44.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 Processor -- start 2022-07-20T03:07:44.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- start start 2022-07-20T03:07:44.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f05300b1a00 0x7f05300b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0530095880 0x7f05300ab190 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f053005f840 con 0x7f05300ab680 2022-07-20T03:07:44.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05300acab0 con 0x7f0530095880 2022-07-20T03:07:44.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0530098a80 con 0x7f05300b1a00 2022-07-20T03:07:44.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0530095880 0x7f05300ab190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0534e42700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f05300ab680 0x7f053013eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39952/0 (socket says 172.21.15.61:39952) 2022-07-20T03:07:44.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0534e42700 1 -- 172.21.15.61:0/196515963 learned_addr learned my addr 172.21.15.61:0/196515963 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:44.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052ffff700 1 --2- 172.21.15.61:0/196515963 >> v2:172.21.15.117:3300/0 conn(0x7f05300b1a00 0x7f05300b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052f7fe700 1 -- 172.21.15.61:0/196515963 >> v2:172.21.15.117:3300/0 conn(0x7f05300b1a00 msgr2=0x7f05300b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:44.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052f7fe700 1 --2- 172.21.15.61:0/196515963 >> v2:172.21.15.117:3300/0 conn(0x7f05300b1a00 0x7f05300b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052f7fe700 1 -- 172.21.15.61:0/196515963 >> v1:172.21.15.61:6789/0 conn(0x7f05300ab680 legacy=0x7f053013eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052f7fe700 1 -- 172.21.15.61:0/196515963 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0530143870 con 0x7f0530095880 2022-07-20T03:07:44.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052f7fe700 1 --2- 172.21.15.61:0/196515963 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0530095880 0x7f05300ab190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f05300625b0 tx=0x7f0524006df0).ready entity=mon.0 client_cookie=1690cde11f35fed6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052effd700 1 -- 172.21.15.61:0/196515963 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f052400d5c0 con 0x7f0530095880 2022-07-20T03:07:44.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052effd700 1 -- 172.21.15.61:0/196515963 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0524000e90 con 0x7f0530095880 2022-07-20T03:07:44.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f052effd700 1 -- 172.21.15.61:0/196515963 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f052400d910 con 0x7f0530095880 2022-07-20T03:07:44.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- 172.21.15.61:0/196515963 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0530095880 msgr2=0x7f05300ab190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 --2- 172.21.15.61:0/196515963 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0530095880 0x7f05300ab190 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f05300625b0 tx=0x7f0524006df0).stop 2022-07-20T03:07:44.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- 172.21.15.61:0/196515963 shutdown_connections 2022-07-20T03:07:44.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 --2- 172.21.15.61:0/196515963 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0530095880 0x7f05300ab190 secure :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0x7f05300625b0 tx=0x7f0524006df0).stop 2022-07-20T03:07:44.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 --2- 172.21.15.61:0/196515963 >> v2:172.21.15.117:3300/0 conn(0x7f05300b1a00 0x7f05300b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.252+0000 7f0535643700 1 -- 172.21.15.61:0/196515963 >> 172.21.15.61:0/196515963 conn(0x7f05300b2af0 msgr2=0x7f05300b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:44.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.256+0000 7f0535643700 1 -- 172.21.15.61:0/196515963 shutdown_connections 2022-07-20T03:07:44.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.256+0000 7f0535643700 1 -- 172.21.15.61:0/196515963 wait complete. 2022-07-20T03:07:44.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.256+0000 7f0535643700 1 Processor -- start 2022-07-20T03:07:44.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.256+0000 7f0535643700 1 -- start start 2022-07-20T03:07:44.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05300ab680 0x7f05300ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f05300b1a00 0x7f0530065610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 0x7f0530153970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300af670 con 0x7f053009a1c0 2022-07-20T03:07:44.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05300af130 con 0x7f05300ab680 2022-07-20T03:07:44.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05300af3d0 con 0x7f05300b1a00 2022-07-20T03:07:44.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 0x7f0530153970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 0x7f0530153970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43360/0 (socket says 172.21.15.61:43360) 2022-07-20T03:07:44.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 -- 172.21.15.61:0/2111038727 learned_addr learned my addr 172.21.15.61:0/2111038727 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:44.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f052ffff700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05300ab680 0x7f05300ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f052f7fe700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f05300b1a00 0x7f0530065610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 -- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f05300b1a00 msgr2=0x7f0530065610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f05300b1a00 0x7f0530065610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 -- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05300ab680 msgr2=0x7f05300ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05300ab680 0x7f05300ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 -- 172.21.15.61:0/2111038727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052400b180 con 0x7f053009a1c0 2022-07-20T03:07:44.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0534e42700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 0x7f0530153970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f05300ac2c0 tx=0x7f0528007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0528005ba0 con 0x7f053009a1c0 2022-07-20T03:07:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0528005d00 con 0x7f053009a1c0 2022-07-20T03:07:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f05280072c0 con 0x7f053009a1c0 2022-07-20T03:07:44.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0530140380 con 0x7f053009a1c0 2022-07-20T03:07:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0530153e60 con 0x7f053009a1c0 2022-07-20T03:07:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.260+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05301536b0 con 0x7f053009a1c0 2022-07-20T03:07:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.264+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0528010070 con 0x7f053009a1c0 2022-07-20T03:07:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.264+0000 7f052d7fa700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0510053a40 0x7f0510055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.264+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f052806b700 con 0x7f053009a1c0 2022-07-20T03:07:44.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.264+0000 7f052ffff700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0510053a40 0x7f0510055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.264+0000 7f052ffff700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0510053a40 0x7f0510055f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f052000c000 tx=0x7f0520008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.272+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f05301536b0 con 0x7f053009a1c0 2022-07-20T03:07:44.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.396+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f05301536b0 con 0x7f053009a1c0 2022-07-20T03:07:44.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.396+0000 7f052d7fa700 1 -- 172.21.15.61:0/2111038727 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v23) v1 ==== 104+0+10 (secure 0 0 0) 0x7f05301536b0 con 0x7f053009a1c0 2022-07-20T03:07:44.443 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0510053a40 msgr2=0x7f0510055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0510053a40 0x7f0510055f00 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f052000c000 tx=0x7f0520008040).stop 2022-07-20T03:07:44.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 msgr2=0x7f0530153970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 0x7f0530153970 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7f05300ac2c0 tx=0x7f0528007f40).stop 2022-07-20T03:07:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 shutdown_connections 2022-07-20T03:07:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0510053a40 0x7f0510055f00 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f053009a1c0 0x7f0530153970 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f05300b1a00 0x7f0530065610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 --2- 172.21.15.61:0/2111038727 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f05300ab680 0x7f05300ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 >> 172.21.15.61:0/2111038727 conn(0x7f05300b2af0 msgr2=0x7f05300b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:44.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 shutdown_connections 2022-07-20T03:07:44.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.400+0000 7f0535643700 1 -- 172.21.15.61:0/2111038727 wait complete. 2022-07-20T03:07:44.449 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:07:44.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.564+0000 7fc6c17b4700 1 Processor -- start 2022-07-20T03:07:44.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.564+0000 7fc6c17b4700 1 -- start start 2022-07-20T03:07:44.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc6bc142ac0 0x7fc6bc0b5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc0ac840 con 0x7fc6bc0ac460 2022-07-20T03:07:44.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc6bc143240 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc6bc1434e0 con 0x7fc6bc142ac0 2022-07-20T03:07:44.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6bbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc6bc0ac460 0x7fc6bc13c830 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39962/0 (socket says 172.21.15.61:39962) 2022-07-20T03:07:44.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6bbfff700 1 -- 172.21.15.61:0/3286391528 learned_addr learned my addr 172.21.15.61:0/3286391528 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:44.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c0fb3700 1 --2- 172.21.15.61:0/3286391528 >> v2:172.21.15.117:3300/0 conn(0x7fc6bc142ac0 0x7fc6bc0b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6baffd700 1 -- 172.21.15.61:0/3286391528 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2788158371 0 0) 0x7fc6bc0ac840 con 0x7fc6bc0ac460 2022-07-20T03:07:44.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6baffd700 1 -- 172.21.15.61:0/3286391528 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc6a8003260 con 0x7fc6bc0ac460 2022-07-20T03:07:44.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c0fb3700 1 -- 172.21.15.61:0/3286391528 >> v1:172.21.15.117:6789/0 conn(0x7fc6bc0ab0c0 legacy=0x7fc6bc0b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6baffd700 1 -- 172.21.15.61:0/3286391528 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1041244771 0 0) 0x7fc6bc143240 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c0fb3700 1 -- 172.21.15.61:0/3286391528 >> v1:172.21.15.61:6789/0 conn(0x7fc6bc0ac460 legacy=0x7fc6bc13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c0fb3700 1 -- 172.21.15.61:0/3286391528 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6bc143780 con 0x7fc6bc142ac0 2022-07-20T03:07:44.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c0fb3700 1 --2- 172.21.15.61:0/3286391528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc142ac0 0x7fc6bc0b5480 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc6b40032f0 tx=0x7fc6b40091d0).ready entity=mon.1 client_cookie=80585d3751406617 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6baffd700 1 -- 172.21.15.61:0/3286391528 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b400b6c0 con 0x7fc6bc142ac0 2022-07-20T03:07:44.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6baffd700 1 -- 172.21.15.61:0/3286391528 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b400b820 con 0x7fc6bc142ac0 2022-07-20T03:07:44.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6baffd700 1 -- 172.21.15.61:0/3286391528 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b400b990 con 0x7fc6bc142ac0 2022-07-20T03:07:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- 172.21.15.61:0/3286391528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc142ac0 msgr2=0x7fc6bc0b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/3286391528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc142ac0 0x7fc6bc0b5480 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc6b40032f0 tx=0x7fc6b40091d0).stop 2022-07-20T03:07:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- 172.21.15.61:0/3286391528 shutdown_connections 2022-07-20T03:07:44.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/3286391528 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc142ac0 0x7fc6bc0b5480 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- 172.21.15.61:0/3286391528 >> 172.21.15.61:0/3286391528 conn(0x7fc6bc0632b0 msgr2=0x7fc6bc063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- 172.21.15.61:0/3286391528 shutdown_connections 2022-07-20T03:07:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.568+0000 7fc6c17b4700 1 -- 172.21.15.61:0/3286391528 wait complete. 2022-07-20T03:07:44.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 Processor -- start 2022-07-20T03:07:44.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 -- start start 2022-07-20T03:07:44.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 0x7fc6bc13bf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6bc0ac460 0x7fc6bc146a20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6bc142ac0 0x7fc6bc145460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc1446f0 con 0x7fc6bc0ac460 2022-07-20T03:07:44.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc065c30 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc6bc146580 con 0x7fc6bc142ac0 2022-07-20T03:07:44.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6bc0ac460 0x7fc6bc146a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 0x7fc6bc13bf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 0x7fc6bc13bf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:53050/0 (socket says 172.21.15.61:53050) 2022-07-20T03:07:44.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c0fb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6bc142ac0 0x7fc6bc145460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 -- 172.21.15.61:0/776923283 learned_addr learned my addr 172.21.15.61:0/776923283 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:44.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6bc0ac460 0x7fc6bc146a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43372/0 (socket says 172.21.15.61:43372) 2022-07-20T03:07:44.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 -- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6bc142ac0 msgr2=0x7fc6bc145460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6bc142ac0 0x7fc6bc145460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 -- 172.21.15.61:0/776923283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6bc0ac460 msgr2=0x7fc6bc146a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6bc0ac460 0x7fc6bc146a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 -- 172.21.15.61:0/776923283 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc6b4009470 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6bbfff700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 0x7fc6bc13bf90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc6b0001110 tx=0x7fc6b0007aa0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b000ce80 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc6bc146f10 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc6b00070d0 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc6bc064c10 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.572+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc6b0007240 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.576+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc6bc065df0 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.576+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc6b0010070 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.576+0000 7fc6b97fa700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc69c053ad0 0x7fc69c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.576+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc6b006b320 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.576+0000 7fc6bb7fe700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc69c053ad0 0x7fc69c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.576+0000 7fc6bb7fe700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc69c053ad0 0x7fc69c055f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc6ac00d000 tx=0x7fc6ac007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:44.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.588+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc6b00421e0 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.716+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fc6bc13cf40 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.716+0000 7fc6b97fa700 1 -- 172.21.15.61:0/776923283 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v23) v1 ==== 100+0+10 (secure 0 0 0) 0x7fc6b0045800 con 0x7fc6bc0ab0c0 2022-07-20T03:07:44.741 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:07:44.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc69c053ad0 msgr2=0x7fc69c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc69c053ad0 0x7fc69c055f90 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fc6ac00d000 tx=0x7fc6ac007a00).stop 2022-07-20T03:07:44.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 msgr2=0x7fc6bc13bf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:44.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 0x7fc6bc13bf90 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc6b0001110 tx=0x7fc6b0007aa0).stop 2022-07-20T03:07:44.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 shutdown_connections 2022-07-20T03:07:44.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc69c053ad0 0x7fc69c055f90 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc6bc142ac0 0x7fc6bc145460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc6bc0ac460 0x7fc6bc146a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 --2- 172.21.15.61:0/776923283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc6bc0ab0c0 0x7fc6bc13bf90 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:44.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 >> 172.21.15.61:0/776923283 conn(0x7fc6bc0632b0 msgr2=0x7fc6bc0606f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:44.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 shutdown_connections 2022-07-20T03:07:44.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:44.720+0000 7fc6c17b4700 1 -- 172.21.15.61:0/776923283 wait complete. 2022-07-20T03:07:44.868 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T03:07:44.915 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-20T03:07:45.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.048+0000 7fa2af242700 1 Processor -- start 2022-07-20T03:07:45.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.048+0000 7fa2af242700 1 -- start start 2022-07-20T03:07:45.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2a80b9ef0 0x7fa2a80b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:45.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa2a805d160 0x7fa2a80b2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:45.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa2a80ba2d0 con 0x7fa2a80b3e40 2022-07-20T03:07:45.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a80b4560 con 0x7fa2a80b9ef0 2022-07-20T03:07:45.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2a80b4800 con 0x7fa2a805d160 2022-07-20T03:07:45.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2acfde700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2a80b9ef0 0x7fa2a80b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:45.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2ad7df700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa2a80b3e40 0x7fa2a8142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39980/0 (socket says 172.21.15.61:39980) 2022-07-20T03:07:45.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2ad7df700 1 -- 172.21.15.61:0/2048513666 learned_addr learned my addr 172.21.15.61:0/2048513666 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:45.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2a7fff700 1 --2- 172.21.15.61:0/2048513666 >> v2:172.21.15.117:3300/0 conn(0x7fa2a805d160 0x7fa2a80b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:45.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2acfde700 1 -- 172.21.15.61:0/2048513666 >> v2:172.21.15.117:3300/0 conn(0x7fa2a805d160 msgr2=0x7fa2a80b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:45.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2acfde700 1 --2- 172.21.15.61:0/2048513666 >> v2:172.21.15.117:3300/0 conn(0x7fa2a805d160 0x7fa2a80b2250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:45.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2acfde700 1 -- 172.21.15.61:0/2048513666 >> v1:172.21.15.61:6789/0 conn(0x7fa2a80b3e40 legacy=0x7fa2a8142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:45.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2a77fe700 1 -- 172.21.15.61:0/2048513666 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3729286828 0 0) 0x7fa2a80ba2d0 con 0x7fa2a80b3e40 2022-07-20T03:07:45.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2acfde700 1 -- 172.21.15.61:0/2048513666 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2a80b1db0 con 0x7fa2a80b9ef0 2022-07-20T03:07:45.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2acfde700 1 --2- 172.21.15.61:0/2048513666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b9ef0 0x7fa2a80b4e50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa29c002f70 tx=0x7fa29c006df0).ready entity=mon.0 client_cookie=77fd802167dbfec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:45.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2a77fe700 1 -- 172.21.15.61:0/2048513666 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa29c00d570 con 0x7fa2a80b9ef0 2022-07-20T03:07:45.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2a77fe700 1 -- 172.21.15.61:0/2048513666 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa29c000e90 con 0x7fa2a80b9ef0 2022-07-20T03:07:45.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2a77fe700 1 -- 172.21.15.61:0/2048513666 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa29c00d8c0 con 0x7fa2a80b9ef0 2022-07-20T03:07:45.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- 172.21.15.61:0/2048513666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b9ef0 msgr2=0x7fa2a80b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:45.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 --2- 172.21.15.61:0/2048513666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b9ef0 0x7fa2a80b4e50 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7fa29c002f70 tx=0x7fa29c006df0).stop 2022-07-20T03:07:45.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- 172.21.15.61:0/2048513666 shutdown_connections 2022-07-20T03:07:45.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 --2- 172.21.15.61:0/2048513666 >> v2:172.21.15.117:3300/0 conn(0x7fa2a805d160 0x7fa2a80b2250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:45.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 --2- 172.21.15.61:0/2048513666 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b9ef0 0x7fa2a80b4e50 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:45.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- 172.21.15.61:0/2048513666 >> 172.21.15.61:0/2048513666 conn(0x7fa2a80679d0 msgr2=0x7fa2a80b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:45.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- 172.21.15.61:0/2048513666 shutdown_connections 2022-07-20T03:07:45.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.052+0000 7fa2af242700 1 -- 172.21.15.61:0/2048513666 wait complete. 2022-07-20T03:07:45.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 Processor -- start 2022-07-20T03:07:45.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 -- start start 2022-07-20T03:07:45.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2a805d160 0x7fa2a8142420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:45.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 0x7fa2a8144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2a80b9ef0 0x7fa2a813d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80ad590 con 0x7fa2a80b3e40 2022-07-20T03:07:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a80ad830 con 0x7fa2a805d160 2022-07-20T03:07:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2a8142910 con 0x7fa2a80b9ef0 2022-07-20T03:07:45.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 0x7fa2a8144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:45.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 0x7fa2a8144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43386/0 (socket says 172.21.15.61:43386) 2022-07-20T03:07:45.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 -- 172.21.15.61:0/2855340284 learned_addr learned my addr 172.21.15.61:0/2855340284 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:45.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2acfde700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2a805d160 0x7fa2a8142420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:45.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2ad7df700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2a80b9ef0 0x7fa2a813d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:45.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 -- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2a80b9ef0 msgr2=0x7fa2a813d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:45.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2a80b9ef0 0x7fa2a813d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:45.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 -- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2a805d160 msgr2=0x7fa2a8142420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:45.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2a805d160 0x7fa2a8142420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:45.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 -- 172.21.15.61:0/2855340284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa29c00b180 con 0x7fa2a80b3e40 2022-07-20T03:07:45.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a7fff700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 0x7fa2a8144220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa29800c000 tx=0x7fa298006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:45.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa298010780 con 0x7fa2a80b3e40 2022-07-20T03:07:45.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2a80af4e0 con 0x7fa2a80b3e40 2022-07-20T03:07:45.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2980108e0 con 0x7fa2a80b3e40 2022-07-20T03:07:45.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa298010c40 con 0x7fa2a80b3e40 2022-07-20T03:07:45.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.056+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2a80af320 con 0x7fa2a80b3e40 2022-07-20T03:07:45.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.060+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa2a80ada00 con 0x7fa2a80b3e40 2022-07-20T03:07:45.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.060+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa29801c460 con 0x7fa2a80b3e40 2022-07-20T03:07:45.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.060+0000 7fa2a5ffb700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa288053a40 0x7fa288055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:45.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.060+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa29806c2a0 con 0x7fa2a80b3e40 2022-07-20T03:07:45.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.060+0000 7fa2acfde700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa288053a40 0x7fa288055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:45.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.060+0000 7fa2acfde700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa288053a40 0x7fa288055f00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa29c01a000 tx=0x7fa29c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:45.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.064+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa298043010 con 0x7fa2a80b3e40 2022-07-20T03:07:45.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:45.236+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7fa2a80af320 con 0x7fa2a80b3e40 2022-07-20T03:07:46.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.219+0000 7fa2a5ffb700 1 -- 172.21.15.61:0/2855340284 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v52) v1 ==== 142+0+0 (secure 0 0 0) 0x7fa2a80af320 con 0x7fa2a80b3e40 2022-07-20T03:07:46.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa288053a40 msgr2=0x7fa288055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa288053a40 0x7fa288055f00 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fa29c01a000 tx=0x7fa29c014000).stop 2022-07-20T03:07:46.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 msgr2=0x7fa2a8144220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 0x7fa2a8144220 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fa29800c000 tx=0x7fa298006df0).stop 2022-07-20T03:07:46.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 shutdown_connections 2022-07-20T03:07:46.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa288053a40 0x7fa288055f00 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2a80b9ef0 0x7fa2a813d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2a80b3e40 0x7fa2a8144220 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 --2- 172.21.15.61:0/2855340284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2a805d160 0x7fa2a8142420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 >> 172.21.15.61:0/2855340284 conn(0x7fa2a80679d0 msgr2=0x7fa2a80b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:46.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 shutdown_connections 2022-07-20T03:07:46.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.223+0000 7fa2af242700 1 -- 172.21.15.61:0/2855340284 wait complete. 2022-07-20T03:07:46.248 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:07:46.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 Processor -- start 2022-07-20T03:07:46.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 -- start start 2022-07-20T03:07:46.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f680b2490 0x7f7f6805ff80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:46.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7f68142be0 0x7f7f68143050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:46.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f68140650 con 0x7f7f680630b0 2022-07-20T03:07:46.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f680ae220 con 0x7f7f680b2490 2022-07-20T03:07:46.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6ea1f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f680ac970 con 0x7f7f68142be0 2022-07-20T03:07:46.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f677fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f680b2490 0x7f7f6805ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:46.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f67fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7f680630b0 0x7f7f680ac260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:39990/0 (socket says 172.21.15.61:39990) 2022-07-20T03:07:46.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f67fff700 1 -- 172.21.15.61:0/959747505 learned_addr learned my addr 172.21.15.61:0/959747505 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:46.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6cfbc700 1 --2- 172.21.15.61:0/959747505 >> v2:172.21.15.117:3300/0 conn(0x7f7f68142be0 0x7f7f68143050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:46.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f677fe700 1 -- 172.21.15.61:0/959747505 >> v2:172.21.15.117:3300/0 conn(0x7f7f68142be0 msgr2=0x7f7f68143050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:46.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f677fe700 1 --2- 172.21.15.61:0/959747505 >> v2:172.21.15.117:3300/0 conn(0x7f7f68142be0 0x7f7f68143050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f677fe700 1 -- 172.21.15.61:0/959747505 >> v1:172.21.15.61:6789/0 conn(0x7f7f680630b0 legacy=0x7f7f680ac260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f677fe700 1 -- 172.21.15.61:0/959747505 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f680b37d0 con 0x7f7f680b2490 2022-07-20T03:07:46.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f66ffd700 1 -- 172.21.15.61:0/959747505 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1938814396 0 0) 0x7f7f50001110 con 0x7f7f680630b0 2022-07-20T03:07:46.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f6cfbc700 1 --2- 172.21.15.61:0/959747505 >> v2:172.21.15.117:3300/0 conn(0x7f7f68142be0 0x7f7f68143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:46.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.359+0000 7f7f677fe700 1 --2- 172.21.15.61:0/959747505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680b2490 0x7f7f6805ff80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7f58002f70 tx=0x7f7f58006df0).ready entity=mon.0 client_cookie=23abc29cb43cfea0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:46.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f66ffd700 1 -- 172.21.15.61:0/959747505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5800d5c0 con 0x7f7f680b2490 2022-07-20T03:07:46.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f66ffd700 1 -- 172.21.15.61:0/959747505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f58000e90 con 0x7f7f680b2490 2022-07-20T03:07:46.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f66ffd700 1 -- 172.21.15.61:0/959747505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5800d910 con 0x7f7f680b2490 2022-07-20T03:07:46.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/959747505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680b2490 msgr2=0x7f7f6805ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/959747505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680b2490 0x7f7f6805ff80 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f7f58002f70 tx=0x7f7f58006df0).stop 2022-07-20T03:07:46.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/959747505 shutdown_connections 2022-07-20T03:07:46.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/959747505 >> v2:172.21.15.117:3300/0 conn(0x7f7f68142be0 0x7f7f68143050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/959747505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680b2490 0x7f7f6805ff80 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/959747505 >> 172.21.15.61:0/959747505 conn(0x7f7f680b47a0 msgr2=0x7f7f6805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:46.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/959747505 shutdown_connections 2022-07-20T03:07:46.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/959747505 wait complete. 2022-07-20T03:07:46.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.363+0000 7f7f6ea1f700 1 Processor -- start 2022-07-20T03:07:46.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 -- start start 2022-07-20T03:07:46.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 0x7f7f68065350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:46.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f680b2490 0x7f7f68141800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:46.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f68142be0 0x7f7f68141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:46.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f680b0f90 con 0x7f7f680630b0 2022-07-20T03:07:46.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f681533c0 con 0x7f7f68142be0 2022-07-20T03:07:46.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f68065cd0 con 0x7f7f680b2490 2022-07-20T03:07:46.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 0x7f7f68065350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:46.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 0x7f7f68065350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43398/0 (socket says 172.21.15.61:43398) 2022-07-20T03:07:46.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 -- 172.21.15.61:0/1126139777 learned_addr learned my addr 172.21.15.61:0/1126139777 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:46.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6cfbc700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f68142be0 0x7f7f68141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:46.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f677fe700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f680b2490 0x7f7f68141800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:46.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 -- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f680b2490 msgr2=0x7f7f68141800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f680b2490 0x7f7f68141800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 -- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f68142be0 msgr2=0x7f7f68141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f68142be0 0x7f7f68141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 -- 172.21.15.61:0/1126139777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5800b180 con 0x7f7f680630b0 2022-07-20T03:07:46.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f677fe700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f680b2490 0x7f7f68141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:46.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f67fff700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 0x7f7f68065350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f7f50000ed0 tx=0x7f7f50007920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f5000ce40 con 0x7f7f680630b0 2022-07-20T03:07:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f500072e0 con 0x7f7f680630b0 2022-07-20T03:07:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f68152f20 con 0x7f7f680630b0 2022-07-20T03:07:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f50005800 con 0x7f7f680630b0 2022-07-20T03:07:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.367+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f68065840 con 0x7f7f680630b0 2022-07-20T03:07:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.371+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f680574e0 con 0x7f7f680630b0 2022-07-20T03:07:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.371+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7f5001a060 con 0x7f7f680630b0 2022-07-20T03:07:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.371+0000 7f7f657fa700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f4c053920 0x7f7f4c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.371+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7f5006b4d0 con 0x7f7f680630b0 2022-07-20T03:07:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.371+0000 7f7f677fe700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f4c053920 0x7f7f4c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.371+0000 7f7f677fe700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f4c053920 0x7f7f4c055de0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7f58018000 tx=0x7f7f58014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:46.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.375+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7f50042430 con 0x7f7f680630b0 2022-07-20T03:07:46.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.503+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7f68140650 con 0x7f7f680630b0 2022-07-20T03:07:46.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f657fa700 1 -- 172.21.15.61:0/1126139777 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v2)=0 v2) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7f68140650 con 0x7f7f680630b0 2022-07-20T03:07:46.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f4c053920 msgr2=0x7f7f4c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f4c053920 0x7f7f4c055de0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f7f58018000 tx=0x7f7f58014000).stop 2022-07-20T03:07:46.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 msgr2=0x7f7f68065350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:46.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 0x7f7f68065350 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f7f50000ed0 tx=0x7f7f50007920).stop 2022-07-20T03:07:46.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 shutdown_connections 2022-07-20T03:07:46.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f4c053920 0x7f7f4c055de0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f68142be0 0x7f7f68141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f680b2490 0x7f7f68141800 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 --2- 172.21.15.61:0/1126139777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f680630b0 0x7f7f68065350 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:46.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 >> 172.21.15.61:0/1126139777 conn(0x7f7f680b47a0 msgr2=0x7f7f6805ccb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:46.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 shutdown_connections 2022-07-20T03:07:46.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:46.511+0000 7f7f6ea1f700 1 -- 172.21.15.61:0/1126139777 wait complete. 2022-07-20T03:07:46.581 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:07:46.583 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:07:46.620 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:07:46.652 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:07:46.687 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:07:46.732 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:07:46.771 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:07:46.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:46.773 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:46.774 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:46.787 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:07:46.790 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:07:46.791 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:46.792 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:46.794 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:46.841 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:07:46.843 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:07:46.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:46.844 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:46.844 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:46.894 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:07:46.901 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T03:07:46.902 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T03:07:46.903 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:07:46.903 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:07:46.904 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:47.005 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.050 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.051 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:47.059 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:07:47.061 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:07:47.061 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:07:47.062 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:07:47.063 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:07:47.119 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.121 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.140 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:47.159 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:07:47.160 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:07:47.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.161 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.161 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:47.216 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:07:47.216 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:07:47.217 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.218 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.221 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:47.299 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:07:47.309 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': Transport endpoint is not connected 2022-07-20T03:07:47.330 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.1 2022-07-20T03:07:47.331 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:07:47.352 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:07:47.360 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:47.414 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.422 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.423 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:47.481 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:07:47.482 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:07:47.482 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:07:47.483 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:07:47.511 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:07:47.587 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.587 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.588 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:47.668 INFO:teuthology.orchestra.run.smithi179.stdout:fuseblk 2022-07-20T03:07:47.668 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-07-20T03:07:47.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.669 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.782 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:47.857 INFO:teuthology.orchestra.run.smithi179.stdout:fuseblk 2022-07-20T03:07:47.878 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.2 2022-07-20T03:07:47.879 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:47.880 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:47.881 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:47.972 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:07:47.978 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': Transport endpoint is not connected 2022-07-20T03:07:47.995 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.2 2022-07-20T03:07:47.996 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi179.front.sepia.ceph.com... 2022-07-20T03:07:47.997 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:07:48.017 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:48.071 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:07:48.072 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:07:48.072 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:48.107 INFO:teuthology.orchestra.run.smithi179.stdout:ext2/ext3 2022-07-20T03:07:48.108 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:07:48.161 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:07:48.161 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:07:48.162 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:07:48.225 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:07:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 Processor -- start 2022-07-20T03:07:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- start start 2022-07-20T03:07:48.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f158405c3f0 0x7f158414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:48.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f15840acef0 0x7f1584146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:48.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f158405d480 con 0x7f15840b1e00 2022-07-20T03:07:48.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f15840b21e0 con 0x7f15840acef0 2022-07-20T03:07:48.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f158405cf40 con 0x7f158405c3f0 2022-07-20T03:07:48.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f15840acef0 0x7f1584146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:48.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f157b7fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f158405c3f0 0x7f158414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:48.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f15840acef0 0x7f1584146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43410/0 (socket says 172.21.15.61:43410) 2022-07-20T03:07:48.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 -- 172.21.15.61:0/919480319 learned_addr learned my addr 172.21.15.61:0/919480319 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:48.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 -- 172.21.15.61:0/919480319 >> v2:172.21.15.117:3300/0 conn(0x7f158405c3f0 msgr2=0x7f158414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:48.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 --2- 172.21.15.61:0/919480319 >> v2:172.21.15.117:3300/0 conn(0x7f158405c3f0 0x7f158414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:48.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 -- 172.21.15.61:0/919480319 >> v1:172.21.15.117:6789/0 conn(0x7f15840b1e00 legacy=0x7f15840ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:48.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 -- 172.21.15.61:0/919480319 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f158405d1e0 con 0x7f15840acef0 2022-07-20T03:07:48.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f1588d16700 1 --2- 172.21.15.61:0/919480319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15840acef0 0x7f1584146410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f157c0032f0 tx=0x7f157c009290).ready entity=mon.0 client_cookie=9ccba9388a794035 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:48.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f15837fe700 1 -- 172.21.15.61:0/919480319 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f157c00b870 con 0x7f15840acef0 2022-07-20T03:07:48.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f15837fe700 1 -- 172.21.15.61:0/919480319 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c00b9d0 con 0x7f15840acef0 2022-07-20T03:07:48.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f15837fe700 1 -- 172.21.15.61:0/919480319 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f157c00bb40 con 0x7f15840acef0 2022-07-20T03:07:48.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- 172.21.15.61:0/919480319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15840acef0 msgr2=0x7f1584146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:48.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 --2- 172.21.15.61:0/919480319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15840acef0 0x7f1584146410 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f157c0032f0 tx=0x7f157c009290).stop 2022-07-20T03:07:48.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- 172.21.15.61:0/919480319 shutdown_connections 2022-07-20T03:07:48.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 --2- 172.21.15.61:0/919480319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f15840acef0 0x7f1584146410 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:48.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 --2- 172.21.15.61:0/919480319 >> v2:172.21.15.117:3300/0 conn(0x7f158405c3f0 0x7f158414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:48.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- 172.21.15.61:0/919480319 >> 172.21.15.61:0/919480319 conn(0x7f15840611c0 msgr2=0x7f1584062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:48.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- 172.21.15.61:0/919480319 shutdown_connections 2022-07-20T03:07:48.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.351+0000 7f158a779700 1 -- 172.21.15.61:0/919480319 wait complete. 2022-07-20T03:07:48.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 Processor -- start 2022-07-20T03:07:48.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 -- start start 2022-07-20T03:07:48.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f15840acef0 0x7f1584143e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:48.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f15840b1e00 0x7f15841485d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:48.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 0x7f1584148ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:48.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584143460 con 0x7f158409e080 2022-07-20T03:07:48.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1584143100 con 0x7f15840acef0 2022-07-20T03:07:48.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1584149770 con 0x7f15840b1e00 2022-07-20T03:07:48.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 0x7f1584148ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:48.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f157b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f15840b1e00 0x7f15841485d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:48.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 0x7f1584148ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43416/0 (socket says 172.21.15.61:43416) 2022-07-20T03:07:48.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 -- 172.21.15.61:0/4009604823 learned_addr learned my addr 172.21.15.61:0/4009604823 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:48.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1583fff700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f15840acef0 0x7f1584143e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:48.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 -- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f15840b1e00 msgr2=0x7f15841485d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:48.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f15840b1e00 0x7f15841485d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:48.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 -- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f15840acef0 msgr2=0x7f1584143e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:48.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f15840acef0 0x7f1584143e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:48.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 -- 172.21.15.61:0/4009604823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f157c009530 con 0x7f158409e080 2022-07-20T03:07:48.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1588d16700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 0x7f1584148ac0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f158414a840 tx=0x7f157c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:48.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f157c00b870 con 0x7f158409e080 2022-07-20T03:07:48.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f157c021440 con 0x7f158409e080 2022-07-20T03:07:48.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f158414a570 con 0x7f158409e080 2022-07-20T03:07:48.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f157c0217a0 con 0x7f158409e080 2022-07-20T03:07:48.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.355+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f158409f110 con 0x7f158409e080 2022-07-20T03:07:48.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.359+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15840b04b0 con 0x7f158409e080 2022-07-20T03:07:48.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.359+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f157c027020 con 0x7f158409e080 2022-07-20T03:07:48.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.359+0000 7f1581ffb700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1568053af0 0x7f1568055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:48.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.359+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f157c06d130 con 0x7f158409e080 2022-07-20T03:07:48.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.359+0000 7f1583fff700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1568053af0 0x7f1568055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:48.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.359+0000 7f1583fff700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1568053af0 0x7f1568055fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f157400d000 tx=0x7f1574007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:48.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.363+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f157c044090 con 0x7f158409e080 2022-07-20T03:07:48.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.487+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f15840574e0 con 0x7f158409e080 2022-07-20T03:07:48.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.491+0000 7f1581ffb700 1 -- 172.21.15.61:0/4009604823 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v23) v1 ==== 81+0+33693 (secure 0 0 0) 0x7f157c015090 con 0x7f158409e080 2022-07-20T03:07:48.585 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:48.600 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:07:48.601 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 23, 2022-07-20T03:07:48.601 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:07:48.605 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:07:48.605 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:07:12.740931+0000", 2022-07-20T03:07:48.606 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:07:48.606 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:07:48.606 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:07:48.607 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:07:48.607 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:07:48.608 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:07:48.608 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:07:48.609 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:07:48.609 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:07:48.610 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:07:48.610 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:07:48.611 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:07:48.611 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:07:48.612 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:07:48.612 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:07:48.613 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 4, 2022-07-20T03:07:48.613 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:07:48.614 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:07:48.614 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:07:48.615 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:07:48.616 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:07:48.617 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:48.617 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:07:48.618 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:07:48.618 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:07:48.618 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:07:48.619 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:07:48.619 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:07:48.620 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:07:48.620 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:07:48.620 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:07:48.621 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:07:48.621 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:07:48.645 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:07:48.645 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:07:48.646 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:07:48.646 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:07:48.647 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:07:48.647 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:07:48.649 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:07:48.649 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:07:48.650 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:07:48.650 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:07:48.651 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:07:48.651 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:07:48.652 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.652 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:07:48.655 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:07:48.655 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:07:48.656 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:07:48.656 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:07:48.657 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:07:48.657 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:07:48.657 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:07:48.674 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:07:48.715 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:07:48.715 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:07:48.716 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:07:48.716 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:07:48.716 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:07:48.716 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:07:48.717 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:07:48.717 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:07:48.717 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:07:48.717 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:07:48.718 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:07:48.718 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:07:48.718 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:07:48.718 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:07:48.774 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:07:48.775 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:07:48.775 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:07:48.776 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:07:48.776 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.777 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:07:48.777 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:07:48.792 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:07:48.792 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:07:48.793 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:07:48.793 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:07:48.793 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:07:48.794 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:07:48.794 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:07:48.795 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:07:48.795 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:07:48.796 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:07:48.796 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:07:48.796 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.797 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:07:48.797 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:07:48.798 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:48.798 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.798 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:48.799 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:07:48.799 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:07:48.800 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:07:48.800 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:07:48.810 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:07:48.811 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:07:48.811 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:07:48.818 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:07:48.819 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:07:48.819 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:07:48.820 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:07:48.820 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:07:48.820 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:07:48.821 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:07:48.827 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:07:48.827 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:07:48.828 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:07:48.837 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:07:48.837 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:07:48.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:07:48.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:07:48.839 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:07:48.839 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:07:48.840 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.840 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:07:48.840 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:07:48.841 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:07:48.841 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:07:48.843 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:07:48.843 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:07:48.844 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:07:48.844 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:07:48.845 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:07:48.845 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:07:48.845 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:07:48.846 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:07:48.846 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:07:48.847 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:07:48.847 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:07:48.848 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:07:48.848 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:07:48.849 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:07:48.859 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:07:48.860 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:07:48.860 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:07:48.861 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:07:48.861 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:07:48.877 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:07:48.877 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:07:48.888 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:07:48.888 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:07:48.888 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.889 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:07:48.889 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:07:48.894 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:07:48.895 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:07:48.896 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:07:48.896 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:07:48.897 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:07:48.897 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:07:48.899 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:07:48.900 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:07:48.900 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:07:48.901 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:07:48.901 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:07:48.916 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:07:48.916 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:48.917 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:48.918 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 3, 2022-07-20T03:07:48.918 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:07:48.927 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:08.880443+0000", 2022-07-20T03:07:48.927 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:07:48.928 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:07:48.928 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:07:48.928 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:07:48.929 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:07:48.930 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:07:48.930 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:07:48.931 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:07:48.931 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:07:48.931 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:07:48.931 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:07:48.932 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:07:48.932 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:07:48.932 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:07:48.932 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:07:48.933 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:07:48.933 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:07:48.933 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:07:48.933 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:07:48.934 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:07:48.934 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.935 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "21", 2022-07-20T03:07:48.935 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:07:48.935 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:07:48.936 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:07:48.936 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:07:48.936 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:07:48.937 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:07:48.938 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:07:48.939 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:07:48.939 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:07:48.949 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:07:48.950 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:07:48.950 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:07:48.951 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:07:48.952 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:07:48.952 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:07:48.952 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:07:48.953 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:07:48.953 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:07:48.954 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:07:48.954 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:07:48.954 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:07:48.970 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:07:48.988 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:07:48.989 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:07:48.990 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:07:48.990 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:07:48.991 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:48.991 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:07:48.992 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:07:48.992 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:07:48.993 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:07:48.993 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:07:48.993 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:07:48.995 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:07:48.995 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:07:48.996 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:07:48.996 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:07:48.996 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:07:48.997 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:07:48.997 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:07:48.998 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:07:48.999 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:07:48.999 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.046 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:07:49.046 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:07:49.046 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:07:49.047 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.047 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.051 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 4, 2022-07-20T03:07:49.051 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:07:49.051 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:09.560512+0000", 2022-07-20T03:07:49.052 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:07:49.052 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:07:49.053 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:07:49.053 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:07:49.062 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:07:49.062 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:07:49.063 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:07:49.063 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:07:49.063 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:07:49.064 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:07:49.064 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:07:49.065 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:07:49.065 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:07:49.065 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:07:49.066 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:07:49.066 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:07:49.067 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:07:49.067 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:07:49.067 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:07:49.068 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:07:49.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.069 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "23", 2022-07-20T03:07:49.069 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:07:49.069 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:07:49.070 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:07:49.070 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:07:49.071 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:07:49.071 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:07:49.072 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:07:49.072 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:07:49.073 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:07:49.073 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:07:49.073 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:07:49.075 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:07:49.075 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:07:49.123 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:07:49.126 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:07:49.127 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:07:49.127 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:07:49.128 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:07:49.128 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:07:49.129 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:07:49.129 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:07:49.129 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:07:49.133 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:07:49.134 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:07:49.134 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:07:49.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:07:49.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.136 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:07:49.136 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:07:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:07:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:07:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:07:49.155 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:07:49.156 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:07:49.156 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:07:49.156 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:07:49.167 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:07:49.167 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:07:49.168 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:07:49.168 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:07:49.169 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:07:49.187 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:07:49.188 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.188 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.189 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.189 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:07:49.199 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:07:49.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.200 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:07:49.201 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:07:49.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.202 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.202 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.203 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.204 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.207 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.209 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.209 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-20T03:07:49.209 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.210 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.211 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.212 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:07:49.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.213 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.213 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:07:49.213 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.214 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.214 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.214 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.214 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.216 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:07:49.216 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:07:49.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.217 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.218 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.220 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:07:49.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.221 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:07:49.221 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.222 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.222 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.222 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.223 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.223 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:07:49.224 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.224 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.225 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:07:49.225 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:07:49.225 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.225 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.226 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:07:49.226 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:07:49.227 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:07:49.227 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:07:49.227 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.228 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.228 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.228 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.229 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:07:49.237 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:07:49.237 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.237 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.238 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.238 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.238 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.238 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.240 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.240 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-20T03:07:49.241 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.241 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.241 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.241 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:07:49.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:07:49.243 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.244 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.244 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.245 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:07:49.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.246 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:07:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.249 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:07:49.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:07:49.252 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.252 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.252 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.253 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:07:49.254 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.255 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.255 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.255 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.255 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:07:49.256 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:07:49.256 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.256 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.257 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:07:49.257 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:07:49.257 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:07:49.258 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:07:49.258 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.258 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.258 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.259 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.259 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:07:49.260 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:07:49.260 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.260 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.260 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.261 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.262 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.262 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 16, 2022-07-20T03:07:49.271 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.272 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.273 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.273 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.273 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:07:49.281 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.283 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:07:49.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.283 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.284 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.284 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.286 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:07:49.286 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:07:49.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.287 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.308 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.308 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.309 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.310 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:07:49.310 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.310 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.311 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.311 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:07:49.311 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.312 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.312 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.312 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.313 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.313 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:07:49.314 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.314 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.316 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:07:49.316 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:07:49.316 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.317 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.317 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.317 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:07:49.317 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:07:49.318 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:07:49.318 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:07:49.318 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.319 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.319 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.319 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.319 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.320 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.320 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:07:49.320 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:07:49.321 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.321 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.330 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.330 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.330 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.331 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.331 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.331 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-20T03:07:49.331 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.332 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.332 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.332 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.332 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.333 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:07:49.333 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.333 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.343 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.343 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:07:49.344 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.344 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.344 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.345 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.345 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.345 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.348 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:07:49.348 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.349 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.349 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:07:49.366 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.366 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.366 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.367 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.367 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.367 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.367 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.368 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.368 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:07:49.368 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.368 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.369 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.369 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:07:49.369 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.387 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.388 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.388 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.388 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.389 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.389 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.389 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:07:49.390 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.390 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.390 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.390 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:07:49.391 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:07:49.391 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.391 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.392 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.392 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:07:49.392 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:07:49.392 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:07:49.393 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:07:49.393 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.394 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.394 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.394 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.395 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.395 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:07:49.396 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:07:49.396 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.396 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.397 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.397 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.397 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.397 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.398 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.398 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 20, 2022-07-20T03:07:49.398 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.398 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.399 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:07:49.400 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.401 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.401 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:07:49.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.401 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.402 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.411 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.412 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.412 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:07:49.413 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:07:49.415 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.415 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.415 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.424 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.424 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.425 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:07:49.426 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.426 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.426 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:07:49.427 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.427 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.427 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.428 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.428 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.429 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.429 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:07:49.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.429 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.430 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.430 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:07:49.430 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:07:49.431 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.431 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.431 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.431 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:07:49.432 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:07:49.432 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:07:49.432 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:07:49.432 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.432 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.433 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.440 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.440 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.440 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.441 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:07:49.441 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:07:49.441 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.441 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.442 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.442 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.454 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.492 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 19, 2022-07-20T03:07:49.493 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.493 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.493 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.494 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.494 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.495 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.495 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:07:49.495 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.496 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.496 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.496 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.497 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:07:49.497 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.498 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.499 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.499 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.500 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.500 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.500 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.501 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.501 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:07:49.502 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.502 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.502 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.503 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.503 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:07:49.504 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.504 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.505 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.505 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.506 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.507 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.507 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.508 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:07:49.508 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.509 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.510 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.510 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.527 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:07:49.528 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.528 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.529 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.529 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.531 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.531 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.532 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.533 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:07:49.533 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.534 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.534 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.535 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:07:49.541 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:07:49.550 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.551 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.553 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:07:49.554 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:07:49.555 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:07:49.555 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:07:49.556 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.556 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.557 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.557 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.558 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:07:49.559 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:07:49.559 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.560 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.560 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.561 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.561 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.562 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.563 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.563 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 21, 2022-07-20T03:07:49.564 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.564 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.565 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.565 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.567 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.567 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.568 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:07:49.585 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.585 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.602 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.603 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.603 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:07:49.604 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.604 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.615 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.615 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.617 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.617 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.617 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.618 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.618 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:07:49.619 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.619 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.620 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.620 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.621 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:07:49.621 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.623 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.623 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.624 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.624 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.625 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.625 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.626 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.627 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:07:49.627 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.628 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.628 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.629 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.629 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:07:49.630 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.630 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.631 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.631 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.631 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.632 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.632 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.633 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.633 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:07:49.634 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.634 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.635 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.635 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.636 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:07:49.636 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:07:49.637 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.638 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.638 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.639 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:07:49.640 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:07:49.640 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:07:49.640 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:07:49.641 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.641 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.642 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.643 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.643 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.644 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.644 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:07:49.645 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:07:49.646 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:07:49.671 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:07:49.671 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:07:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:07:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:07:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:07:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:07:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 16, 2022-07-20T03:07:49.681 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:07:49.682 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:07:49.682 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:07:49.682 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.683 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.683 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.684 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:07:49.684 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.685 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.685 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.686 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.686 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:07:49.687 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.687 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.687 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.688 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.688 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:07:49.689 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.689 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.698 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.710 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:07:49.710 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.711 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.711 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.711 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.711 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:07:49.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.712 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.712 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.712 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.713 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:07:49.713 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.713 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.713 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.720 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:07:49.720 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.721 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.722 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.722 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:07:49.723 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.723 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.724 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.724 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.725 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:07:49.725 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:07:49.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.728 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:07:49.728 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:07:49.728 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.729 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.730 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.746 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:07:49.746 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:07:49.747 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:07:49.766 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.786 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.787 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:07:49.787 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:07:49.788 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:07:49.788 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:07:49.789 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:07:49.789 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:07:49.811 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:07:49.812 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:49.815 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:49.815 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:07:49.816 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:07:49.835 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.835 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:07:49.835 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.836 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.836 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:49.837 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:49.837 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:49.844 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:07:49.856 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:49.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.857 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.879 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:07:49.880 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.880 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.881 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:49.881 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:49.882 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:49.882 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:07:49.883 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:49.883 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.884 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.929 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:07:49.930 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.941 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.942 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:49.942 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:49.943 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:49.943 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:07:49.963 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:49.963 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.963 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.964 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:07:49.964 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.965 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.965 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:49.965 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:49.966 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:49.966 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:07:49.966 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:49.967 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.973 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.974 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:07:49.974 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.974 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.975 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:49.975 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:49.976 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:49.976 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:07:49.976 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:49.977 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.977 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.977 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:07:49.988 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.988 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.989 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:49.989 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:49.990 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:49.990 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:07:49.991 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:49.991 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:49.992 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:49.992 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:07:49.992 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:49.993 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:49.993 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:50.004 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:50.005 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:50.006 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:07:50.006 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:50.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:50.007 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:50.008 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:07:50.008 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:07:50.008 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:07:50.009 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:07:50.016 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:07:50.017 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:07:50.018 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:07:50.018 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:07:50.019 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:50.019 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:07:50.020 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:07:50.020 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:07:50.021 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:07:50.021 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:07:50.022 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:07:50.030 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:07:50.030 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:07:50.031 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:07:50.031 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:07:50.031 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:07:50.032 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:07:50.032 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:07:50.033 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:07:50.033 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:50.034 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:50.034 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:07:50.035 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:07:50.035 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:07:50.035 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:07:50.036 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:07:50.036 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:07:50.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.491+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1568053af0 msgr2=0x7f1568055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.491+0000 7f158a779700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1568053af0 0x7f1568055fb0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f157400d000 tx=0x7f1574007b70).stop 2022-07-20T03:07:50.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.491+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 msgr2=0x7f1584148ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.491+0000 7f158a779700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 0x7f1584148ac0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f158414a840 tx=0x7f157c016000).stop 2022-07-20T03:07:50.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.491+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 shutdown_connections 2022-07-20T03:07:50.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1568053af0 0x7f1568055fb0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f158409e080 0x7f1584148ac0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f15840b1e00 0x7f15841485d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 --2- 172.21.15.61:0/4009604823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f15840acef0 0x7f1584143e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 >> 172.21.15.61:0/4009604823 conn(0x7f158405c3f0 msgr2=0x7f1584069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:50.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 shutdown_connections 2022-07-20T03:07:50.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:48.495+0000 7f158a779700 1 -- 172.21.15.61:0/4009604823 wait complete. 2022-07-20T03:07:50.176 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:50.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.299+0000 7fe3b5012700 1 Processor -- start 2022-07-20T03:07:50.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.299+0000 7fe3b5012700 1 -- start start 2022-07-20T03:07:50.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe3b013c790 0x7fe3b00b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b00b3ca0 0x7fe3b00b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3b0142ba0 con 0x7fe3b00b9f00 2022-07-20T03:07:50.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3b01429b0 con 0x7fe3b013c790 2022-07-20T03:07:50.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b0146330 con 0x7fe3b00b3ca0 2022-07-20T03:07:50.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b4811700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe3b00b9f00 0x7fe3b0143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40022/0 (socket says 172.21.15.61:40022) 2022-07-20T03:07:50.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b4811700 1 -- 172.21.15.61:0/3364312579 learned_addr learned my addr 172.21.15.61:0/3364312579 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:50.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3affff700 1 --2- 172.21.15.61:0/3364312579 >> v2:172.21.15.117:3300/0 conn(0x7fe3b013c790 0x7fe3b00b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3af7fe700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b00b3ca0 0x7fe3b00b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3aeffd700 1 -- 172.21.15.61:0/3364312579 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 770167227 0 0) 0x7fe3b0142ba0 con 0x7fe3b00b9f00 2022-07-20T03:07:50.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3aeffd700 1 -- 172.21.15.61:0/3364312579 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe394003260 con 0x7fe3b00b9f00 2022-07-20T03:07:50.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3affff700 1 -- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b00b3ca0 msgr2=0x7fe3b00b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:50.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3affff700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b00b3ca0 0x7fe3b00b9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3affff700 1 -- 172.21.15.61:0/3364312579 >> v1:172.21.15.61:6789/0 conn(0x7fe3b00b9f00 legacy=0x7fe3b0143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3affff700 1 -- 172.21.15.61:0/3364312579 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b00b4c00 con 0x7fe3b013c790 2022-07-20T03:07:50.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3af7fe700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b00b3ca0 0x7fe3b00b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:50.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3affff700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b013c790 0x7fe3b00b4710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fe3a0002f70 tx=0x7fe3a0006df0).ready entity=mon.1 client_cookie=e3aa063f505bc6df server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:50.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3aeffd700 1 -- 172.21.15.61:0/3364312579 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a000d4a0 con 0x7fe3b013c790 2022-07-20T03:07:50.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3aeffd700 1 -- 172.21.15.61:0/3364312579 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3a0000e90 con 0x7fe3b013c790 2022-07-20T03:07:50.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3aeffd700 1 -- 172.21.15.61:0/3364312579 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3a000d7f0 con 0x7fe3b013c790 2022-07-20T03:07:50.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b013c790 msgr2=0x7fe3b00b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b013c790 0x7fe3b00b4710 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fe3a0002f70 tx=0x7fe3a0006df0).stop 2022-07-20T03:07:50.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- 172.21.15.61:0/3364312579 shutdown_connections 2022-07-20T03:07:50.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b00b3ca0 0x7fe3b00b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 --2- 172.21.15.61:0/3364312579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b013c790 0x7fe3b00b4710 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- 172.21.15.61:0/3364312579 >> 172.21.15.61:0/3364312579 conn(0x7fe3b00678b0 msgr2=0x7fe3b00b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:50.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- 172.21.15.61:0/3364312579 shutdown_connections 2022-07-20T03:07:50.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.303+0000 7fe3b5012700 1 -- 172.21.15.61:0/3364312579 wait complete. 2022-07-20T03:07:50.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 Processor -- start 2022-07-20T03:07:50.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 -- start start 2022-07-20T03:07:50.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b00b3ca0 0x7fe3b00af970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 0x7fe3b00b0240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b013c790 0x7fe3b013aca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00af000 con 0x7fe3b00b9f00 2022-07-20T03:07:50.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00b1fe0 con 0x7fe3b00b3ca0 2022-07-20T03:07:50.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3b00b2280 con 0x7fe3b013c790 2022-07-20T03:07:50.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 0x7fe3b00b0240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 0x7fe3b00b0240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43426/0 (socket says 172.21.15.61:43426) 2022-07-20T03:07:50.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 -- 172.21.15.61:0/2387214707 learned_addr learned my addr 172.21.15.61:0/2387214707 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:50.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3affff700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b00b3ca0 0x7fe3b00af970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b4811700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b013c790 0x7fe3b013aca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 -- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b013c790 msgr2=0x7fe3b013aca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b013c790 0x7fe3b013aca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 -- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b00b3ca0 msgr2=0x7fe3b00af970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b00b3ca0 0x7fe3b00af970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 -- 172.21.15.61:0/2387214707 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe39800b180 con 0x7fe3b00b9f00 2022-07-20T03:07:50.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3af7fe700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 0x7fe3b00b0240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe39800c000 tx=0x7fe398006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:50.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe39800c890 con 0x7fe3b00b9f00 2022-07-20T03:07:50.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe39800c9f0 con 0x7fe3b00b9f00 2022-07-20T03:07:50.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3a000b180 con 0x7fe3b00b9f00 2022-07-20T03:07:50.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe39800cd50 con 0x7fe3b00b9f00 2022-07-20T03:07:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.307+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3b00b0730 con 0x7fe3b00b9f00 2022-07-20T03:07:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.311+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3b00b1e10 con 0x7fe3b00b9f00 2022-07-20T03:07:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.311+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe39801c460 con 0x7fe3b00b9f00 2022-07-20T03:07:50.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.311+0000 7fe3ad7fa700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe388053a40 0x7fe388055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.311+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe39806c660 con 0x7fe3b00b9f00 2022-07-20T03:07:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.311+0000 7fe3affff700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe388053a40 0x7fe388055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.311+0000 7fe3affff700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe388053a40 0x7fe388055f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe3a0018000 tx=0x7fe3a0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:50.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.315+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3980433d0 con 0x7fe3b00b9f00 2022-07-20T03:07:50.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.479+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe3b0146090 con 0x7fe3b00b9f00 2022-07-20T03:07:50.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3ad7fa700 1 -- 172.21.15.61:0/2387214707 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 7 v7) v1 ==== 93+0+2134 (secure 0 0 0) 0x7fe3980469f0 con 0x7fe3b00b9f00 2022-07-20T03:07:50.494 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:50.494 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":7,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":7,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:12.741818+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4273},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4273":{"gid":4273,"name":"b","rank":0,"incarnation":4,"state":"up:active","state_seq":2,"addr":"172.21.15.117:6835/725875269","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":725875269},{"type":"v1","addr":"172.21.15.117:6835","nonce":725875269}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:50.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe388053a40 msgr2=0x7fe388055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe388053a40 0x7fe388055f00 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe3a0018000 tx=0x7fe3a0014000).stop 2022-07-20T03:07:50.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 msgr2=0x7fe3b00b0240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 0x7fe3b00b0240 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fe39800c000 tx=0x7fe398006df0).stop 2022-07-20T03:07:50.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 shutdown_connections 2022-07-20T03:07:50.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe388053a40 0x7fe388055f00 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3b013c790 0x7fe3b013aca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3b00b9f00 0x7fe3b00b0240 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 --2- 172.21.15.61:0/2387214707 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3b00b3ca0 0x7fe3b00af970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 >> 172.21.15.61:0/2387214707 conn(0x7fe3b00678b0 msgr2=0x7fe3b013fcd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:50.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 shutdown_connections 2022-07-20T03:07:50.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.483+0000 7fe3b5012700 1 -- 172.21.15.61:0/2387214707 wait complete. 2022-07-20T03:07:50.500 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 7 2022-07-20T03:07:50.501 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:07:50.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31911700 1 Processor -- start 2022-07-20T03:07:50.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31911700 1 -- start start 2022-07-20T03:07:50.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31911700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5f2c0b4fd0 0x7f5f2c146a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31911700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2c0b48d0 con 0x7f5f2c0b13f0 2022-07-20T03:07:50.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31911700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f2c0b65e0 con 0x7f5f2c0b6200 2022-07-20T03:07:50.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31911700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f2c1481a0 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f2bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5f2c0b13f0 0x7f5f2c14a520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40034/0 (socket says 172.21.15.61:40034) 2022-07-20T03:07:50.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f2bfff700 1 -- 172.21.15.61:0/1283059561 learned_addr learned my addr 172.21.15.61:0/1283059561 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:50.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31110700 1 --2- 172.21.15.61:0/1283059561 >> v2:172.21.15.117:3300/0 conn(0x7f5f2c0b4fd0 0x7f5f2c146a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/1283059561 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2089562822 0 0) 0x7f5f2c0b48d0 con 0x7f5f2c0b13f0 2022-07-20T03:07:50.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/1283059561 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f0c003260 con 0x7f5f2c0b13f0 2022-07-20T03:07:50.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31110700 1 -- 172.21.15.61:0/1283059561 >> v1:172.21.15.117:6789/0 conn(0x7f5f2c0b6200 legacy=0x7f5f2c146320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31110700 1 -- 172.21.15.61:0/1283059561 >> v1:172.21.15.61:6789/0 conn(0x7f5f2c0b13f0 legacy=0x7f5f2c14a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.611+0000 7f5f31110700 1 -- 172.21.15.61:0/1283059561 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2c0b4d00 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31110700 1 --2- 172.21.15.61:0/1283059561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c146a30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5f1c002f70 tx=0x7f5f1c006df0).ready entity=mon.1 client_cookie=7ce4cf403033b280 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:50.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/1283059561 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f1c00d610 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/1283059561 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f1c000e90 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f2b7fe700 1 -- 172.21.15.61:0/1283059561 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f1c00d960 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 -- 172.21.15.61:0/1283059561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b4fd0 msgr2=0x7f5f2c146a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 --2- 172.21.15.61:0/1283059561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c146a30 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f5f1c002f70 tx=0x7f5f1c006df0).stop 2022-07-20T03:07:50.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 -- 172.21.15.61:0/1283059561 shutdown_connections 2022-07-20T03:07:50.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 --2- 172.21.15.61:0/1283059561 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c146a30 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 -- 172.21.15.61:0/1283059561 >> 172.21.15.61:0/1283059561 conn(0x7f5f2c063d10 msgr2=0x7f5f2c0b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:50.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 -- 172.21.15.61:0/1283059561 shutdown_connections 2022-07-20T03:07:50.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 -- 172.21.15.61:0/1283059561 wait complete. 2022-07-20T03:07:50.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 Processor -- start 2022-07-20T03:07:50.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.615+0000 7f5f31911700 1 -- start start 2022-07-20T03:07:50.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f2c0b13f0 0x7f5f2c14a290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c0ab210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b6200 0x7f5f2c142de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c149180 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c1486c0 con 0x7f5f2c0b6200 2022-07-20T03:07:50.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f2c148420 con 0x7f5f2c0b13f0 2022-07-20T03:07:50.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c0ab210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c0ab210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43442/0 (socket says 172.21.15.61:43442) 2022-07-20T03:07:50.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 -- 172.21.15.61:0/3593093079 learned_addr learned my addr 172.21.15.61:0/3593093079 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:50.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f3090f700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f2c0b13f0 0x7f5f2c14a290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31110700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b6200 0x7f5f2c142de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 -- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f2c0b13f0 msgr2=0x7f5f2c14a290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f2c0b13f0 0x7f5f2c14a290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 -- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b6200 msgr2=0x7f5f2c142de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b6200 0x7f5f2c142de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 -- 172.21.15.61:0/3593093079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f1c00b180 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f2bfff700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c0ab210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5f14001110 tx=0x7f5f14007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:50.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f140070d0 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f14007230 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f2c148ee0 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f14005610 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f2c1432d0 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.619+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f2c0b4360 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.623+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5f14010070 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.623+0000 7f5f29ffb700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f100539b0 0x7f5f10055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:50.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.623+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(23..23 src has 1..23) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f1406b5c0 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.623+0000 7f5f3090f700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f100539b0 0x7f5f10055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.623+0000 7f5f3090f700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f100539b0 0x7f5f10055e70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5f2400d000 tx=0x7f5f24007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:50.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.627+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f14042410 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.787+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f5f2c0574e0 con 0x7f5f2c0b4fd0 2022-07-20T03:07:50.935 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:07:50.927+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:07:51.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.935+0000 7f5f29ffb700 1 -- 172.21.15.61:0/3593093079 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v8) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5f14045a30 con 0x7f5f2c0b4fd0 2022-07-20T03:07:51.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f100539b0 msgr2=0x7f5f10055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f100539b0 0x7f5f10055e70 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5f2400d000 tx=0x7f5f24007f10).stop 2022-07-20T03:07:51.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 msgr2=0x7f5f2c0ab210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c0ab210 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f5f14001110 tx=0x7f5f14007ad0).stop 2022-07-20T03:07:51.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 shutdown_connections 2022-07-20T03:07:51.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f100539b0 0x7f5f10055e70 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f2c0b6200 0x7f5f2c142de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f2c0b4fd0 0x7f5f2c0ab210 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 --2- 172.21.15.61:0/3593093079 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f2c0b13f0 0x7f5f2c14a290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 >> 172.21.15.61:0/3593093079 conn(0x7f5f2c063d10 msgr2=0x7f5f2c14c750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:51.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 shutdown_connections 2022-07-20T03:07:51.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:50.939+0000 7f5f31911700 1 -- 172.21.15.61:0/3593093079 wait complete. 2022-07-20T03:07:51.054 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:07:51.055 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:07:51.148 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:07:51.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 Processor -- start 2022-07-20T03:07:51.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 -- start start 2022-07-20T03:07:51.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80adff0 0x7f79d80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:51.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d8147010 con 0x7f79d80af8e0 2022-07-20T03:07:51.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79d81472b0 con 0x7f79d80b5650 2022-07-20T03:07:51.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8146d40 con 0x7f79d80adff0 2022-07-20T03:07:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79de6cb700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f79d80af8e0 0x7f79d805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40050/0 (socket says 172.21.15.61:40050) 2022-07-20T03:07:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79de6cb700 1 -- 172.21.15.61:0/1934307781 learned_addr learned my addr 172.21.15.61:0/1934307781 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dd6c9700 1 --2- 172.21.15.61:0/1934307781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80adff0 0x7f79d80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dcec8700 1 -- 172.21.15.61:0/1934307781 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2166599418 0 0) 0x7f79d8147010 con 0x7f79d80af8e0 2022-07-20T03:07:51.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dcec8700 1 -- 172.21.15.61:0/1934307781 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f79c4003260 con 0x7f79d80af8e0 2022-07-20T03:07:51.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dd6c9700 1 -- 172.21.15.61:0/1934307781 >> v1:172.21.15.117:6789/0 conn(0x7f79d80b5650 legacy=0x7f79d8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dcec8700 1 -- 172.21.15.61:0/1934307781 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 314093362 0 0) 0x7f79d81472b0 con 0x7f79d80b5650 2022-07-20T03:07:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dd6c9700 1 -- 172.21.15.61:0/1934307781 >> v1:172.21.15.61:6789/0 conn(0x7f79d80af8e0 legacy=0x7f79d805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dd6c9700 1 -- 172.21.15.61:0/1934307781 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d805c4a0 con 0x7f79d80adff0 2022-07-20T03:07:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dd6c9700 1 --2- 172.21.15.61:0/1934307781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80adff0 0x7f79d80b5a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f79d00032f0 tx=0x7f79d00091a0).ready entity=mon.2 client_cookie=c7026e9d4551ad01 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dcec8700 1 -- 172.21.15.61:0/1934307781 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79d000b6c0 con 0x7f79d80adff0 2022-07-20T03:07:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dcec8700 1 -- 172.21.15.61:0/1934307781 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d000b820 con 0x7f79d80adff0 2022-07-20T03:07:51.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79dcec8700 1 -- 172.21.15.61:0/1934307781 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79d000b990 con 0x7f79d80adff0 2022-07-20T03:07:51.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 -- 172.21.15.61:0/1934307781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80adff0 msgr2=0x7f79d80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 --2- 172.21.15.61:0/1934307781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80adff0 0x7f79d80b5a30 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f79d00032f0 tx=0x7f79d00091a0).stop 2022-07-20T03:07:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 -- 172.21.15.61:0/1934307781 shutdown_connections 2022-07-20T03:07:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.267+0000 7f79deecc700 1 --2- 172.21.15.61:0/1934307781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80adff0 0x7f79d80b5a30 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- 172.21.15.61:0/1934307781 >> 172.21.15.61:0/1934307781 conn(0x7f79d80634e0 msgr2=0x7f79d80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:51.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- 172.21.15.61:0/1934307781 shutdown_connections 2022-07-20T03:07:51.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- 172.21.15.61:0/1934307781 wait complete. 2022-07-20T03:07:51.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 Processor -- start 2022-07-20T03:07:51.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- start start 2022-07-20T03:07:51.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79d80adff0 0x7f79d810e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:51.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80af8e0 0x7f79d8066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:51.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 0x7f79d8159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:51.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d810f0a0 con 0x7f79d80b5650 2022-07-20T03:07:51.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79d81112a0 con 0x7f79d80adff0 2022-07-20T03:07:51.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79d8065f10 con 0x7f79d80af8e0 2022-07-20T03:07:51.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 0x7f79d8159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:51.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 0x7f79d8159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43458/0 (socket says 172.21.15.61:43458) 2022-07-20T03:07:51.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 -- 172.21.15.61:0/3812226357 learned_addr learned my addr 172.21.15.61:0/3812226357 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:51.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79dd6c9700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80af8e0 0x7f79d8066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:51.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79ddeca700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79d80adff0 0x7f79d810e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:51.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 -- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80af8e0 msgr2=0x7f79d8066650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80af8e0 0x7f79d8066650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 -- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79d80adff0 msgr2=0x7f79d810e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79d80adff0 0x7f79d810e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 -- 172.21.15.61:0/3812226357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d0009420 con 0x7f79d80b5650 2022-07-20T03:07:51.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79ddeca700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79d80adff0 0x7f79d810e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:51.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79de6cb700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 0x7f79d8159af0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f79c8001110 tx=0x7f79c8007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:51.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79c80070d0 con 0x7f79d80b5650 2022-07-20T03:07:51.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79c8007230 con 0x7f79d80b5650 2022-07-20T03:07:51.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79d80661b0 con 0x7f79d80b5650 2022-07-20T03:07:51.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79c8005610 con 0x7f79d80b5650 2022-07-20T03:07:51.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.271+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79d8066b40 con 0x7f79d80b5650 2022-07-20T03:07:51.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.275+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79d8066340 con 0x7f79d80b5650 2022-07-20T03:07:51.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.275+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f79c801a060 con 0x7f79d80b5650 2022-07-20T03:07:51.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.275+0000 7f79ceffd700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79b8053ad0 0x7f79b8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:51.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.275+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f79c806b780 con 0x7f79d80b5650 2022-07-20T03:07:51.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.275+0000 7f79ddeca700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79b8053ad0 0x7f79b8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:51.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.275+0000 7f79ddeca700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79b8053ad0 0x7f79b8055f90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f79d400b000 tx=0x7f79d4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:51.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.283+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79c8042580 con 0x7f79d80b5650 2022-07-20T03:07:51.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.443+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79d80574e0 con 0x7f79d80b5650 2022-07-20T03:07:51.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.443+0000 7f79ceffd700 1 -- 172.21.15.61:0/3812226357 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 8 v8) v1 ==== 93+0+1774 (secure 0 0 0) 0x7f79c8018070 con 0x7f79d80b5650 2022-07-20T03:07:51.753 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:51.761 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":8,"default_fscid":1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4257,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1159050961","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1159050961},{"type":"v1","addr":"172.21.15.61:6833","nonce":1159050961}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":2}],"filesystems":[{"mdsmap":{"epoch":8,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:07:10.718708+0000","modified":"2022-07-20T03:07:50.934009+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":24,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[4],"metadata_pool":3,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":1}]} 2022-07-20T03:07:51.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79b8053ad0 msgr2=0x7f79b8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79b8053ad0 0x7f79b8055f90 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f79d400b000 tx=0x7f79d4009040).stop 2022-07-20T03:07:51.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 msgr2=0x7f79d8159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:51.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 0x7f79d8159af0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f79c8001110 tx=0x7f79c8007a80).stop 2022-07-20T03:07:51.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 shutdown_connections 2022-07-20T03:07:51.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79b8053ad0 0x7f79b8055f90 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79d80b5650 0x7f79d8159af0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79d80af8e0 0x7f79d8066650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 --2- 172.21.15.61:0/3812226357 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79d80adff0 0x7f79d810e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:51.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 >> 172.21.15.61:0/3812226357 conn(0x7f79d80634e0 msgr2=0x7f79d8144030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:51.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 shutdown_connections 2022-07-20T03:07:51.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:51.447+0000 7f79deecc700 1 -- 172.21.15.61:0/3812226357 wait complete. 2022-07-20T03:07:51.897 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 8 2022-07-20T03:07:51.897 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:07:52.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.007+0000 7fee262dc700 1 Processor -- start 2022-07-20T03:07:52.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.007+0000 7fee262dc700 1 -- start start 2022-07-20T03:07:52.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fee20099f80 0x7fee200b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:52.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee2009ae60 con 0x7fee2005bd90 2022-07-20T03:07:52.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee200b02a0 con 0x7fee200afec0 2022-07-20T03:07:52.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee200610d0 con 0x7fee20099f80 2022-07-20T03:07:52.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee24879700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fee2005bd90 0x7fee20141070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40064/0 (socket says 172.21.15.61:40064) 2022-07-20T03:07:52.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee24879700 1 -- 172.21.15.61:0/2884949190 learned_addr learned my addr 172.21.15.61:0/2884949190 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:52.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee17fff700 1 --2- 172.21.15.61:0/2884949190 >> v2:172.21.15.117:3300/0 conn(0x7fee20099f80 0x7fee200b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:52.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee17fff700 1 -- 172.21.15.61:0/2884949190 >> v1:172.21.15.117:6789/0 conn(0x7fee200afec0 legacy=0x7fee20099870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:52.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee1f7fe700 1 -- 172.21.15.61:0/2884949190 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1791678713 0 0) 0x7fee2009ae60 con 0x7fee2005bd90 2022-07-20T03:07:52.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee17fff700 1 -- 172.21.15.61:0/2884949190 >> v1:172.21.15.61:6789/0 conn(0x7fee2005bd90 legacy=0x7fee20141070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:52.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee17fff700 1 -- 172.21.15.61:0/2884949190 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee20061370 con 0x7fee20099f80 2022-07-20T03:07:52.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee17fff700 1 --2- 172.21.15.61:0/2884949190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee20099f80 0x7fee200b26c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fee10002f70 tx=0x7fee10006df0).ready entity=mon.1 client_cookie=1dde5b786b157a9f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:52.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee1f7fe700 1 -- 172.21.15.61:0/2884949190 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1000d610 con 0x7fee20099f80 2022-07-20T03:07:52.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee1f7fe700 1 -- 172.21.15.61:0/2884949190 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee10000e90 con 0x7fee20099f80 2022-07-20T03:07:52.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee1f7fe700 1 -- 172.21.15.61:0/2884949190 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1000d960 con 0x7fee20099f80 2022-07-20T03:07:52.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- 172.21.15.61:0/2884949190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee20099f80 msgr2=0x7fee200b26c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:52.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 --2- 172.21.15.61:0/2884949190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee20099f80 0x7fee200b26c0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fee10002f70 tx=0x7fee10006df0).stop 2022-07-20T03:07:52.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- 172.21.15.61:0/2884949190 shutdown_connections 2022-07-20T03:07:52.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 --2- 172.21.15.61:0/2884949190 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee20099f80 0x7fee200b26c0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:52.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- 172.21.15.61:0/2884949190 >> 172.21.15.61:0/2884949190 conn(0x7fee200b5cb0 msgr2=0x7fee200b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:52.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- 172.21.15.61:0/2884949190 shutdown_connections 2022-07-20T03:07:52.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.011+0000 7fee262dc700 1 -- 172.21.15.61:0/2884949190 wait complete. 2022-07-20T03:07:52.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 Processor -- start 2022-07-20T03:07:52.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 -- start start 2022-07-20T03:07:52.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee2005bd90 0x7fee201445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:52.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 0x7fee200ab8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:52.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee200afec0 0x7fee200abda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:52.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee201438a0 con 0x7fee20099f80 2022-07-20T03:07:52.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee200ac830 con 0x7fee200afec0 2022-07-20T03:07:52.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee20093a20 con 0x7fee2005bd90 2022-07-20T03:07:52.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 0x7fee200ab8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:52.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 0x7fee200ab8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43470/0 (socket says 172.21.15.61:43470) 2022-07-20T03:07:52.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 -- 172.21.15.61:0/1964070490 learned_addr learned my addr 172.21.15.61:0/1964070490 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:52.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee24879700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee200afec0 0x7fee200abda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:52.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee1ffff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee2005bd90 0x7fee201445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:52.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 -- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee2005bd90 msgr2=0x7fee201445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:52.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee2005bd90 0x7fee201445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:52.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 -- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee200afec0 msgr2=0x7fee200abda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:52.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee200afec0 0x7fee200abda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:52.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 -- 172.21.15.61:0/1964070490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee1000b180 con 0x7fee20099f80 2022-07-20T03:07:52.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee24879700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee200afec0 0x7fee200abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:52.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee1ffff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee2005bd90 0x7fee201445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:52.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee17fff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 0x7fee200ab8b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fee10002f70 tx=0x7fee10015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:52.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1000de70 con 0x7fee20099f80 2022-07-20T03:07:52.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee10011c60 con 0x7fee20099f80 2022-07-20T03:07:52.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee200aca60 con 0x7fee20099f80 2022-07-20T03:07:52.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee1001e460 con 0x7fee20099f80 2022-07-20T03:07:52.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.015+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee20144ee0 con 0x7fee20099f80 2022-07-20T03:07:52.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.019+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee2005c190 con 0x7fee20099f80 2022-07-20T03:07:52.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.019+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fee10025030 con 0x7fee20099f80 2022-07-20T03:07:52.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.019+0000 7fee1dffb700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee040539b0 0x7fee04055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:52.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.019+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fee1006d530 con 0x7fee20099f80 2022-07-20T03:07:52.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.019+0000 7fee1ffff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee040539b0 0x7fee04055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:52.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.023+0000 7fee1ffff700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee040539b0 0x7fee04055e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fee0800b000 tx=0x7fee08009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:52.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.027+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee100442b0 con 0x7fee20099f80 2022-07-20T03:07:52.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:52.191+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fee2005c8a0 con 0x7fee20099f80 2022-07-20T03:07:53.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.003+0000 7fee1dffb700 1 -- 172.21.15.61:0/1964070490 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v10) v1 ==== 104+0+0 (secure 0 0 0) 0x7fee100478d0 con 0x7fee20099f80 2022-07-20T03:07:53.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee040539b0 msgr2=0x7fee04055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:53.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee040539b0 0x7fee04055e70 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fee0800b000 tx=0x7fee08009040).stop 2022-07-20T03:07:53.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 msgr2=0x7fee200ab8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:53.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 0x7fee200ab8b0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7fee10002f70 tx=0x7fee10015000).stop 2022-07-20T03:07:53.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 shutdown_connections 2022-07-20T03:07:53.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee040539b0 0x7fee04055e70 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee200afec0 0x7fee200abda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee20099f80 0x7fee200ab8b0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 --2- 172.21.15.61:0/1964070490 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee2005bd90 0x7fee201445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.007+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 >> 172.21.15.61:0/1964070490 conn(0x7fee200b5cb0 msgr2=0x7fee2009a3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:53.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.011+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 shutdown_connections 2022-07-20T03:07:53.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.011+0000 7fee262dc700 1 -- 172.21.15.61:0/1964070490 wait complete. 2022-07-20T03:07:53.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:07:53.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.227+0000 7f8643aa4700 1 Processor -- start 2022-07-20T03:07:53.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.227+0000 7f8643aa4700 1 -- start start 2022-07-20T03:07:53.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c13f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:53.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f863c13fe10 0x7f863c099300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:53.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f863c0b3a50 con 0x7f863c0b3670 2022-07-20T03:07:53.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f863c13d3b0 con 0x7f863c13fe10 2022-07-20T03:07:53.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863c1430a0 con 0x7f863c0ac180 2022-07-20T03:07:53.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8641840700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f863c0b3670 0x7f863c1437e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40074/0 (socket says 172.21.15.61:40074) 2022-07-20T03:07:53.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8641840700 1 -- 172.21.15.61:0/3016229062 learned_addr learned my addr 172.21.15.61:0/3016229062 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:53.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8642041700 1 --2- 172.21.15.61:0/3016229062 >> v2:172.21.15.117:3300/0 conn(0x7f863c13fe10 0x7f863c099300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:53.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f864103f700 1 --2- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c13f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:53.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f864083e700 1 -- 172.21.15.61:0/3016229062 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 891663333 0 0) 0x7f863c0b3a50 con 0x7f863c0b3670 2022-07-20T03:07:53.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f864083e700 1 -- 172.21.15.61:0/3016229062 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8624003260 con 0x7f863c0b3670 2022-07-20T03:07:53.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8642041700 1 -- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 msgr2=0x7f863c13f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:53.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8642041700 1 --2- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c13f920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8642041700 1 -- 172.21.15.61:0/3016229062 >> v1:172.21.15.61:6789/0 conn(0x7f863c0b3670 legacy=0x7f863c1437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:53.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8642041700 1 -- 172.21.15.61:0/3016229062 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f863c143340 con 0x7f863c13fe10 2022-07-20T03:07:53.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8642041700 1 --2- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c13fe10 0x7f863c099300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f86340032f0 tx=0x7f86340091d0).ready entity=mon.1 client_cookie=41b1d456fd113022 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:53.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f864083e700 1 -- 172.21.15.61:0/3016229062 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863400b600 con 0x7f863c13fe10 2022-07-20T03:07:53.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f864083e700 1 -- 172.21.15.61:0/3016229062 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f863400b760 con 0x7f863c13fe10 2022-07-20T03:07:53.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f864083e700 1 -- 172.21.15.61:0/3016229062 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863400b8d0 con 0x7f863c13fe10 2022-07-20T03:07:53.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 -- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c13fe10 msgr2=0x7f863c099300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:53.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 --2- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c13fe10 0x7f863c099300 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f86340032f0 tx=0x7f86340091d0).stop 2022-07-20T03:07:53.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 -- 172.21.15.61:0/3016229062 shutdown_connections 2022-07-20T03:07:53.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 --2- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c13fe10 0x7f863c099300 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 --2- 172.21.15.61:0/3016229062 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c13f920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.231+0000 7f8643aa4700 1 -- 172.21.15.61:0/3016229062 >> 172.21.15.61:0/3016229062 conn(0x7f863c0b6c50 msgr2=0x7f863c0b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:53.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.235+0000 7f8643aa4700 1 -- 172.21.15.61:0/3016229062 shutdown_connections 2022-07-20T03:07:53.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.235+0000 7f8643aa4700 1 -- 172.21.15.61:0/3016229062 wait complete. 2022-07-20T03:07:53.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.235+0000 7f8643aa4700 1 Processor -- start 2022-07-20T03:07:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.235+0000 7f8643aa4700 1 -- start start 2022-07-20T03:07:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c0afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:53.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c0b3670 0x7f863c14f3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:53.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 0x7f863c14f8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:53.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c13f270 con 0x7f863c13fe10 2022-07-20T03:07:53.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f863c13f510 con 0x7f863c0b3670 2022-07-20T03:07:53.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f863c13f7b0 con 0x7f863c0ac180 2022-07-20T03:07:53.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 0x7f863c14f8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:53.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 0x7f863c14f8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43486/0 (socket says 172.21.15.61:43486) 2022-07-20T03:07:53.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 -- 172.21.15.61:0/4069211593 learned_addr learned my addr 172.21.15.61:0/4069211593 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:53.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f864103f700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c0b3670 0x7f863c14f3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:53.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8641840700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c0afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:53.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 -- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 msgr2=0x7f863c0afb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:53.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c0afb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 -- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c0b3670 msgr2=0x7f863c14f3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:53.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c0b3670 0x7f863c14f3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:53.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 -- 172.21.15.61:0/4069211593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8634009470 con 0x7f863c13fe10 2022-07-20T03:07:53.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8642041700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 0x7f863c14f8d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f86340032f0 tx=0x7f8634016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:53.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863400b600 con 0x7f863c13fe10 2022-07-20T03:07:53.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8634011c10 con 0x7f863c13fe10 2022-07-20T03:07:53.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f863c14fdc0 con 0x7f863c13fe10 2022-07-20T03:07:53.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f863401f3f0 con 0x7f863c13fe10 2022-07-20T03:07:53.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.239+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f863c13eef0 con 0x7f863c13fe10 2022-07-20T03:07:53.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.243+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f863c0574e0 con 0x7f863c13fe10 2022-07-20T03:07:53.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.243+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8634026030 con 0x7f863c13fe10 2022-07-20T03:07:53.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.243+0000 7f862affd700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f861c053a40 0x7f861c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:53.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.243+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(24..24 src has 1..24) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f863406ca60 con 0x7f863c13fe10 2022-07-20T03:07:53.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.243+0000 7f8641840700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f861c053a40 0x7f861c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:53.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.243+0000 7f8641840700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f861c053a40 0x7f861c055f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f863800d000 tx=0x7f8638009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:53.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.247+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86340473a0 con 0x7f863c13fe10 2022-07-20T03:07:53.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:53.379+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f863c09c210 con 0x7f863c13fe10 2022-07-20T03:07:54.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.015+0000 7f862affd700 1 -- 172.21.15.61:0/4069211593 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v25) v1 ==== 185+0+0 (secure 0 0 0) 0x7f8634024020 con 0x7f863c13fe10 2022-07-20T03:07:54.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f861c053a40 msgr2=0x7f861c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:54.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f861c053a40 0x7f861c055f00 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f863800d000 tx=0x7f8638009040).stop 2022-07-20T03:07:54.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 msgr2=0x7f863c14f8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:54.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 0x7f863c14f8d0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f86340032f0 tx=0x7f8634016000).stop 2022-07-20T03:07:54.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 shutdown_connections 2022-07-20T03:07:54.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f861c053a40 0x7f861c055f00 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f863c0b3670 0x7f863c14f3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f863c13fe10 0x7f863c14f8d0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 --2- 172.21.15.61:0/4069211593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f863c0ac180 0x7f863c0afb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 >> 172.21.15.61:0/4069211593 conn(0x7f863c0b6c50 msgr2=0x7f863c13c900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:54.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 shutdown_connections 2022-07-20T03:07:54.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.023+0000 7f8643aa4700 1 -- 172.21.15.61:0/4069211593 wait complete. 2022-07-20T03:07:54.118 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:07:54.119 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:07:54.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.251+0000 7ffb962a3700 1 Processor -- start 2022-07-20T03:07:54.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.251+0000 7ffb962a3700 1 -- start start 2022-07-20T03:07:54.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb900b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:54.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffb900b3cf0 0x7ffb90146040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:54.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffb90066b10 con 0x7ffb90146530 2022-07-20T03:07:54.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffb900b6c20 con 0x7ffb900b3cf0 2022-07-20T03:07:54.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb900aead0 con 0x7ffb900b6840 2022-07-20T03:07:54.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb94aa0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffb900b3cf0 0x7ffb90146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:54.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb95aa2700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ffb90146530 0x7ffb9005bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40090/0 (socket says 172.21.15.61:40090) 2022-07-20T03:07:54.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb95aa2700 1 -- 172.21.15.61:0/2028632515 learned_addr learned my addr 172.21.15.61:0/2028632515 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:54.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb952a1700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb900b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:54.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb94aa0700 1 -- 172.21.15.61:0/2028632515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 msgr2=0x7ffb900b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:54.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb94aa0700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb900b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb94aa0700 1 -- 172.21.15.61:0/2028632515 >> v1:172.21.15.61:6789/0 conn(0x7ffb90146530 legacy=0x7ffb9005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:54.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb7bfff700 1 -- 172.21.15.61:0/2028632515 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1773370726 0 0) 0x7ffb90066b10 con 0x7ffb90146530 2022-07-20T03:07:54.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb94aa0700 1 -- 172.21.15.61:0/2028632515 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb900aed70 con 0x7ffb900b3cf0 2022-07-20T03:07:54.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb952a1700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb900b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:54.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb94aa0700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb900b3cf0 0x7ffb90146040 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ffb8c002f70 tx=0x7ffb8c006df0).ready entity=mon.0 client_cookie=dc020e3f25e60f05 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:54.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb7bfff700 1 -- 172.21.15.61:0/2028632515 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb8c00d570 con 0x7ffb900b3cf0 2022-07-20T03:07:54.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb7bfff700 1 -- 172.21.15.61:0/2028632515 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb8c000e90 con 0x7ffb900b3cf0 2022-07-20T03:07:54.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb7bfff700 1 -- 172.21.15.61:0/2028632515 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb8c00d8c0 con 0x7ffb900b3cf0 2022-07-20T03:07:54.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- 172.21.15.61:0/2028632515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb900b3cf0 msgr2=0x7ffb90146040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:54.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb900b3cf0 0x7ffb90146040 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7ffb8c002f70 tx=0x7ffb8c006df0).stop 2022-07-20T03:07:54.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- 172.21.15.61:0/2028632515 shutdown_connections 2022-07-20T03:07:54.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb900b3cf0 0x7ffb90146040 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 --2- 172.21.15.61:0/2028632515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb900b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- 172.21.15.61:0/2028632515 >> 172.21.15.61:0/2028632515 conn(0x7ffb900b9e10 msgr2=0x7ffb900b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:54.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- 172.21.15.61:0/2028632515 shutdown_connections 2022-07-20T03:07:54.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.255+0000 7ffb962a3700 1 -- 172.21.15.61:0/2028632515 wait complete. 2022-07-20T03:07:54.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 Processor -- start 2022-07-20T03:07:54.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 -- start start 2022-07-20T03:07:54.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb900b3cf0 0x7ffb900b2c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:54.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb901435a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:54.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 0x7ffb90112540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:54.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb9013ed30 con 0x7ffb90146530 2022-07-20T03:07:54.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb90114b70 con 0x7ffb900b3cf0 2022-07-20T03:07:54.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffb900b0270 con 0x7ffb900b6840 2022-07-20T03:07:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 0x7ffb90112540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 0x7ffb90112540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43500/0 (socket says 172.21.15.61:43500) 2022-07-20T03:07:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 -- 172.21.15.61:0/4038582085 learned_addr learned my addr 172.21.15.61:0/4038582085 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb94aa0700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb901435a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb952a1700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb900b3cf0 0x7ffb900b2c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 -- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 msgr2=0x7ffb901435a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb901435a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 -- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb900b3cf0 msgr2=0x7ffb900b2c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb900b3cf0 0x7ffb900b2c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 -- 172.21.15.61:0/4038582085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb8c00b180 con 0x7ffb90146530 2022-07-20T03:07:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb952a1700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb900b3cf0 0x7ffb900b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:07:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb95aa2700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 0x7ffb90112540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ffb88001250 tx=0x7ffb88001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:54.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb88005ba0 con 0x7ffb90146530 2022-07-20T03:07:54.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffb88005d00 con 0x7ffb90146530 2022-07-20T03:07:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb900b1ee0 con 0x7ffb90146530 2022-07-20T03:07:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffb88004a90 con 0x7ffb90146530 2022-07-20T03:07:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.259+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb900b08b0 con 0x7ffb90146530 2022-07-20T03:07:54.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.263+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb900b0430 con 0x7ffb90146530 2022-07-20T03:07:54.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.263+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffb88004bf0 con 0x7ffb90146530 2022-07-20T03:07:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.263+0000 7ffb82ffd700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb6c053a40 0x7ffb6c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.263+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(25..25 src has 1..25) v4 ==== 5721+0+0 (secure 0 0 0) 0x7ffb8806b960 con 0x7ffb90146530 2022-07-20T03:07:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.263+0000 7ffb952a1700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb6c053a40 0x7ffb6c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:54.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.263+0000 7ffb952a1700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb6c053a40 0x7ffb6c055f00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ffb84008000 tx=0x7ffb84008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:54.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.275+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffb880429c0 con 0x7ffb90146530 2022-07-20T03:07:54.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:54.403+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ffb900b50e0 con 0x7ffb90146530 2022-07-20T03:07:55.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.027+0000 7ffb82ffd700 1 -- 172.21.15.61:0/4038582085 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v26) v1 ==== 173+0+0 (secure 0 0 0) 0x7ffb88045fe0 con 0x7ffb90146530 2022-07-20T03:07:55.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb6c053a40 msgr2=0x7ffb6c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:55.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb6c053a40 0x7ffb6c055f00 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7ffb84008000 tx=0x7ffb84008040).stop 2022-07-20T03:07:55.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 msgr2=0x7ffb90112540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:55.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 0x7ffb90112540 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7ffb88001250 tx=0x7ffb88001360).stop 2022-07-20T03:07:55.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 shutdown_connections 2022-07-20T03:07:55.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffb6c053a40 0x7ffb6c055f00 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffb90146530 0x7ffb90112540 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffb900b6840 0x7ffb901435a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 --2- 172.21.15.61:0/4038582085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffb900b3cf0 0x7ffb900b2c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 >> 172.21.15.61:0/4038582085 conn(0x7ffb900b9e10 msgr2=0x7ffb9005c9a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 shutdown_connections 2022-07-20T03:07:55.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.031+0000 7ffb962a3700 1 -- 172.21.15.61:0/4038582085 wait complete. 2022-07-20T03:07:55.075 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:07:55.079 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:07:55.088 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:07:55.089 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:07:55.089 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:07:55.090 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:07:55.090 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:07:55.091 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:07:55.091 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:07:55.117 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:07:55.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:07:55.120 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:07:55.121 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:07:55.180 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:07:55.180 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:07:55.181 DEBUG:teuthology.parallel:result is None 2022-07-20T03:07:55.181 DEBUG:teuthology.parallel:result is None 2022-07-20T03:07:55.185 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:07:55.204 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:07:55.204 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:07:55.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.299+0000 7f9104b12700 1 Processor -- start 2022-07-20T03:07:55.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.299+0000 7f9104b12700 1 -- start start 2022-07-20T03:07:55.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f91000b3560 0x7f91000ab960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:55.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f910005cd60 0x7f9100143700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:55.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f91000acbc0 con 0x7f9100143bf0 2022-07-20T03:07:55.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f91000b3940 con 0x7f910005cd60 2022-07-20T03:07:55.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f91001472a0 con 0x7f91000b3560 2022-07-20T03:07:55.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90fffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9100143bf0 0x7f91000b1a20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40116/0 (socket says 172.21.15.61:40116) 2022-07-20T03:07:55.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90fffff700 1 -- 172.21.15.61:0/489911824 learned_addr learned my addr 172.21.15.61:0/489911824 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:55.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90ff7fe700 1 --2- 172.21.15.61:0/489911824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f91000b3560 0x7f91000ab960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:55.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90f7fff700 1 --2- 172.21.15.61:0/489911824 >> v2:172.21.15.117:3300/0 conn(0x7f910005cd60 0x7f9100143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:55.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90feffd700 1 -- 172.21.15.61:0/489911824 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2122509449 0 0) 0x7f91000acbc0 con 0x7f9100143bf0 2022-07-20T03:07:55.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90feffd700 1 -- 172.21.15.61:0/489911824 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f90e0003260 con 0x7f9100143bf0 2022-07-20T03:07:55.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90ff7fe700 1 -- 172.21.15.61:0/489911824 >> v2:172.21.15.117:3300/0 conn(0x7f910005cd60 msgr2=0x7f9100143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:55.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90ff7fe700 1 --2- 172.21.15.61:0/489911824 >> v2:172.21.15.117:3300/0 conn(0x7f910005cd60 0x7f9100143700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90ff7fe700 1 -- 172.21.15.61:0/489911824 >> v1:172.21.15.61:6789/0 conn(0x7f9100143bf0 legacy=0x7f91000b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:55.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90ff7fe700 1 -- 172.21.15.61:0/489911824 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9100143260 con 0x7f91000b3560 2022-07-20T03:07:55.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90f7fff700 1 --2- 172.21.15.61:0/489911824 >> v2:172.21.15.117:3300/0 conn(0x7f910005cd60 0x7f9100143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:55.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90ff7fe700 1 --2- 172.21.15.61:0/489911824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f91000b3560 0x7f91000ab960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f90f0002f70 tx=0x7f90f0006df0).ready entity=mon.2 client_cookie=cda347b8fee9c9e6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:55.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90feffd700 1 -- 172.21.15.61:0/489911824 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90f000d4a0 con 0x7f91000b3560 2022-07-20T03:07:55.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90feffd700 1 -- 172.21.15.61:0/489911824 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f0000e90 con 0x7f91000b3560 2022-07-20T03:07:55.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f90feffd700 1 -- 172.21.15.61:0/489911824 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90f000d7f0 con 0x7f91000b3560 2022-07-20T03:07:55.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- 172.21.15.61:0/489911824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f91000b3560 msgr2=0x7f91000ab960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:55.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 --2- 172.21.15.61:0/489911824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f91000b3560 0x7f91000ab960 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f90f0002f70 tx=0x7f90f0006df0).stop 2022-07-20T03:07:55.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- 172.21.15.61:0/489911824 shutdown_connections 2022-07-20T03:07:55.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 --2- 172.21.15.61:0/489911824 >> v2:172.21.15.117:3300/0 conn(0x7f910005cd60 0x7f9100143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 --2- 172.21.15.61:0/489911824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f91000b3560 0x7f91000ab960 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- 172.21.15.61:0/489911824 >> 172.21.15.61:0/489911824 conn(0x7f91000b6b20 msgr2=0x7f91000b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:55.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- 172.21.15.61:0/489911824 shutdown_connections 2022-07-20T03:07:55.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.303+0000 7f9104b12700 1 -- 172.21.15.61:0/489911824 wait complete. 2022-07-20T03:07:55.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 Processor -- start 2022-07-20T03:07:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 -- start start 2022-07-20T03:07:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 0x7f9100141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91000b3560 0x7f91000ad670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:55.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9100143bf0 0x7f91000adb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:55.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91000ae050 con 0x7f910005cd60 2022-07-20T03:07:55.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9100155800 con 0x7f91000b3560 2022-07-20T03:07:55.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f9104b12700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9100155aa0 con 0x7f9100143bf0 2022-07-20T03:07:55.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f90ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 0x7f9100141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:55.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f90ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 0x7f9100141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43522/0 (socket says 172.21.15.61:43522) 2022-07-20T03:07:55.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f90ff7fe700 1 -- 172.21.15.61:0/2334410256 learned_addr learned my addr 172.21.15.61:0/2334410256 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:55.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f90f7fff700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91000b3560 0x7f91000ad670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:55.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.307+0000 7f90fffff700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9100143bf0 0x7f91000adb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:55.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90ff7fe700 1 -- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9100143bf0 msgr2=0x7f91000adb60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:55.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90ff7fe700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9100143bf0 0x7f91000adb60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90ff7fe700 1 -- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91000b3560 msgr2=0x7f91000ad670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90ff7fe700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91000b3560 0x7f91000ad670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90ff7fe700 1 -- 172.21.15.61:0/2334410256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e800b180 con 0x7f910005cd60 2022-07-20T03:07:55.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90f7fff700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91000b3560 0x7f91000ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:55.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90ff7fe700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 0x7f9100141600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f90f0002f70 tx=0x7f90f0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:55.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90f0012930 con 0x7f910005cd60 2022-07-20T03:07:55.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90f0012a90 con 0x7f910005cd60 2022-07-20T03:07:55.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90f000b180 con 0x7f910005cd60 2022-07-20T03:07:55.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90f0012df0 con 0x7f910005cd60 2022-07-20T03:07:55.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9100141af0 con 0x7f910005cd60 2022-07-20T03:07:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9100155620 con 0x7f910005cd60 2022-07-20T03:07:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f90f001c070 con 0x7f910005cd60 2022-07-20T03:07:55.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90fd7fa700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90e4053a60 0x7f90e4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:55.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.311+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(28..28 src has 1..28) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f90f006c6a0 con 0x7f910005cd60 2022-07-20T03:07:55.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.315+0000 7f90f7fff700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90e4053a60 0x7f90e4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:55.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.315+0000 7f90f7fff700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90e4053a60 0x7f90e4055f20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f90e800b000 tx=0x7f90e8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:55.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.319+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90f0043840 con 0x7f910005cd60 2022-07-20T03:07:55.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:55.459+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f9100142f90 con 0x7f910005cd60 2022-07-20T03:07:56.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.243+0000 7f90fd7fa700 1 -- 172.21.15.61:0/2334410256 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v29) v1 ==== 97+0+0 (secure 0 0 0) 0x7f90f0019030 con 0x7f910005cd60 2022-07-20T03:07:56.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90e4053a60 msgr2=0x7f90e4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90e4053a60 0x7f90e4055f20 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f90e800b000 tx=0x7f90e8006df0).stop 2022-07-20T03:07:56.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 msgr2=0x7f9100141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 0x7f9100141600 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f90f0002f70 tx=0x7f90f0014000).stop 2022-07-20T03:07:56.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 shutdown_connections 2022-07-20T03:07:56.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90e4053a60 0x7f90e4055f20 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f91000b3560 0x7f91000ad670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9100143bf0 0x7f91000adb60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 --2- 172.21.15.61:0/2334410256 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f910005cd60 0x7f9100141600 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 >> 172.21.15.61:0/2334410256 conn(0x7f91000b6b20 msgr2=0x7f91000b4de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:56.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 shutdown_connections 2022-07-20T03:07:56.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.247+0000 7f9104b12700 1 -- 172.21.15.61:0/2334410256 wait complete. 2022-07-20T03:07:56.342 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:07:56.348 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:07:56.354 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:07:56.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa21493700 1 Processor -- start 2022-07-20T03:07:56.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa21493700 1 -- start start 2022-07-20T03:07:56.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa21493700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 0x7ffa1c14a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa21493700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c1426b0 con 0x7ffa1c1431f0 2022-07-20T03:07:56.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa21493700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa1c0b1a00 con 0x7ffa1c0b1620 2022-07-20T03:07:56.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa21493700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c060b50 con 0x7ffa1c13ee40 2022-07-20T03:07:56.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ffa1c1431f0 0x7ffa1c13e730 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40132/0 (socket says 172.21.15.61:40132) 2022-07-20T03:07:56.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 -- 172.21.15.61:0/3289450655 learned_addr learned my addr 172.21.15.61:0/3289450655 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:56.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1b7fe700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 0x7ffa1c14a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4227263304 0 0) 0x7ffa1c1426b0 con 0x7ffa1c1431f0 2022-07-20T03:07:56.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa00003260 con 0x7ffa1c1431f0 2022-07-20T03:07:56.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1978442012 0 0) 0x7ffa1c0b1a00 con 0x7ffa1c0b1620 2022-07-20T03:07:56.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ffa1c1426b0 con 0x7ffa1c0b1620 2022-07-20T03:07:56.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2834110603 0 0) 0x7ffa00003260 con 0x7ffa1c1431f0 2022-07-20T03:07:56.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 msgr2=0x7ffa1c14a9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:56.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 0x7ffa1c14a9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 >> v1:172.21.15.117:6789/0 conn(0x7ffa1c0b1620 legacy=0x7ffa1c142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c05f840 con 0x7ffa1c1431f0 2022-07-20T03:07:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7ffa04001660 con 0x7ffa1c1431f0 2022-07-20T03:07:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 >> v1:172.21.15.61:6789/0 conn(0x7ffa1c1431f0 legacy=0x7ffa1c13e730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 0x7ffa00003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa00003e20 0x7ffa00004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa000049f0 0x7ffa00004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c060b50 con 0x7ffa000049f0 2022-07-20T03:07:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000b800 con 0x7ffa00003e20 2022-07-20T03:07:56.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa0000b9c0 con 0x7ffa1c13ee40 2022-07-20T03:07:56.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa00003e20 0x7ffa00004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1b7fe700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa000049f0 0x7ffa00004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1affd700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 0x7ffa00003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 -- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 msgr2=0x7ffa00003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:56.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c13ee40 0x7ffa00003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 -- 172.21.15.61:0/3289450655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa000049f0 msgr2=0x7ffa00004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:56.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa000049f0 0x7ffa00004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 -- 172.21.15.61:0/3289450655 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa00003260 con 0x7ffa00003e20 2022-07-20T03:07:56.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa127fc700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa00003e20 0x7ffa00004500 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ffa04001430 tx=0x7ffa04001dd0).ready entity=mon.1 client_cookie=6ad11c6de200379c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:56.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa040070d0 con 0x7ffa00003e20 2022-07-20T03:07:56.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa04007230 con 0x7ffa00003e20 2022-07-20T03:07:56.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.491+0000 7ffa1a7fc700 1 -- 172.21.15.61:0/3289450655 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa0400c8f0 con 0x7ffa00003e20 2022-07-20T03:07:56.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 -- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa00003e20 msgr2=0x7ffa00004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 --2- 172.21.15.61:0/3289450655 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa00003e20 0x7ffa00004500 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7ffa04001430 tx=0x7ffa04001dd0).stop 2022-07-20T03:07:56.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa1b7fe700 1 -- 172.21.15.61:0/3289450655 reap_dead start 2022-07-20T03:07:56.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 -- 172.21.15.61:0/3289450655 shutdown_connections 2022-07-20T03:07:56.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 -- 172.21.15.61:0/3289450655 >> 172.21.15.61:0/3289450655 conn(0x7ffa1c0b23d0 msgr2=0x7ffa1c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:56.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 -- 172.21.15.61:0/3289450655 shutdown_connections 2022-07-20T03:07:56.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 -- 172.21.15.61:0/3289450655 wait complete. 2022-07-20T03:07:56.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 Processor -- start 2022-07-20T03:07:56.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.495+0000 7ffa21493700 1 -- start start 2022-07-20T03:07:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 0x7ffa1c0b44e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c065400 0x7ffa1c155820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa1c155d10 0x7ffa1c156190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c0673c0 con 0x7ffa00003e20 2022-07-20T03:07:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c065880 con 0x7ffa1c155d10 2022-07-20T03:07:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa1c066270 con 0x7ffa1c065400 2022-07-20T03:07:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 0x7ffa1c0b44e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 0x7ffa1c0b44e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43544/0 (socket says 172.21.15.61:43544) 2022-07-20T03:07:56.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 -- 172.21.15.61:0/1083893935 learned_addr learned my addr 172.21.15.61:0/1083893935 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:56.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1b7fe700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa1c155d10 0x7ffa1c156190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa127fc700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c065400 0x7ffa1c155820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:56.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 -- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c065400 msgr2=0x7ffa1c155820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c065400 0x7ffa1c155820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 -- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa1c155d10 msgr2=0x7ffa1c156190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:56.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa1c155d10 0x7ffa1c156190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:56.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 -- 172.21.15.61:0/1083893935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa1c152130 con 0x7ffa00003e20 2022-07-20T03:07:56.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1b7fe700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa1c155d10 0x7ffa1c156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:07:56.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa1affd700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 0x7ffa1c0b44e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7ffa0c0011f0 tx=0x7ffa0c007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:56.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa0c011780 con 0x7ffa00003e20 2022-07-20T03:07:56.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa0c0118e0 con 0x7ffa00003e20 2022-07-20T03:07:56.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa1c066510 con 0x7ffa00003e20 2022-07-20T03:07:56.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa0c011c40 con 0x7ffa00003e20 2022-07-20T03:07:56.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.499+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa1c0650c0 con 0x7ffa00003e20 2022-07-20T03:07:56.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.503+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa1c05fae0 con 0x7ffa00003e20 2022-07-20T03:07:56.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.503+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffa0c01b030 con 0x7ffa00003e20 2022-07-20T03:07:56.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.503+0000 7ffa18ff9700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff9f4053940 0x7ff9f4055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.503+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ffa0c06bb90 con 0x7ffa00003e20 2022-07-20T03:07:57.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.503+0000 7ffa127fc700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff9f4053940 0x7ff9f4055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.503+0000 7ffa127fc700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff9f4053940 0x7ff9f4055e00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ffa0401a000 tx=0x7ffa04015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:57.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.507+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffa0c042e20 con 0x7ffa00003e20 2022-07-20T03:07:57.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7ffa1c0574e0 con 0x7ffa00003e20 2022-07-20T03:07:57.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa18ff9700 1 -- 172.21.15.61:0/1083893935 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v6) v1 ==== 80+0+5627 (secure 0 0 0) 0x7ffa0c046440 con 0x7ffa00003e20 2022-07-20T03:07:57.023 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:07:57.023 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:07:57.024 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:07:57.024 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:07:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:07:57.025 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.026 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:07:57.026 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.027 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.027 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.028 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.028 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.029 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.029 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:07:57.030 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:07:57.030 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.031 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:07:57.031 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.031 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.032 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.032 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.033 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.033 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:07:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:07:57.034 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.035 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.035 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.036 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.036 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.038 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:07:57.038 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:07:57.039 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.039 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.039 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.040 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.041 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.042 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:07:57.043 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:07:57.044 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.044 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:07:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:07:57.047 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.048 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.049 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.049 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.051 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:07:57.051 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:07:57.052 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.052 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.052 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.053 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.053 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.085 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.096 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:07:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:07:57.097 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.098 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.098 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.098 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.099 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.106 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:07:57.106 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:07:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.107 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.108 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.138 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:07:57.139 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:07:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.140 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:07:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:07:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:07:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow r", 2022-07-20T03:07:57.142 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow rw", 2022-07-20T03:07:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rwx" 2022-07-20T03:07:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.143 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:07:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:07:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.144 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:07:57.170 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow r", 2022-07-20T03:07:57.171 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow rw", 2022-07-20T03:07:57.171 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rwx" 2022-07-20T03:07:57.171 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.172 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.175 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:07:57.176 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:07:57.176 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.177 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:07:57.177 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow r", 2022-07-20T03:07:57.177 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow rw", 2022-07-20T03:07:57.177 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rwx" 2022-07-20T03:07:57.178 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.179 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:07:57.179 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:07:57.179 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.180 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:07:57.180 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:07:57.194 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:07:57.213 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.242 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.300 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-mds", 2022-07-20T03:07:57.301 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBKcddiCleAChAAwLLVz+YkLvnp3gFZd7CJlA==", 2022-07-20T03:07:57.301 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.302 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-mds" 2022-07-20T03:07:57.302 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.303 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.304 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-mgr", 2022-07-20T03:07:57.304 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBKcddi42GAChAADTrDzWMTty2mchZOLIyF5A==", 2022-07-20T03:07:57.305 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.305 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-mgr" 2022-07-20T03:07:57.305 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.341 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.341 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-osd", 2022-07-20T03:07:57.341 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBKcddiJ2uAChAA0DrqFoJ/Wwik7QIgOSkMNw==", 2022-07-20T03:07:57.341 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.342 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-osd" 2022-07-20T03:07:57.342 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.342 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-rbd", 2022-07-20T03:07:57.343 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBKcddiv3SAChAA5xsvxIK0GG19UcwMCo6jMw==", 2022-07-20T03:07:57.343 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.343 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-rbd" 2022-07-20T03:07:57.343 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.344 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.344 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-rbd-mirror", 2022-07-20T03:07:57.344 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBKcddiIX6AChAAw902qfKaI8qAu2vfjHkhSA==", 2022-07-20T03:07:57.344 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.371 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-rbd-mirror" 2022-07-20T03:07:57.372 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.372 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.373 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.bootstrap-rgw", 2022-07-20T03:07:57.382 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBKcddivomAChAA4IuP5H124Frd+pHXho595Q==", 2022-07-20T03:07:57.383 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.383 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile bootstrap-rgw" 2022-07-20T03:07:57.384 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.385 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.385 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.394 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:07:57.394 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:07:57.394 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.395 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:07:57.395 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:07:57.395 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.395 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.396 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:07:57.396 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:07:57.396 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:07:57.396 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:07:57.397 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:07:57.397 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:07:57.397 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:07:57.499 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:07:57.500 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.500 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:07:57.500 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:07:57.501 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:07:57.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff9f4053940 msgr2=0x7ff9f4055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff9f4053940 0x7ff9f4055e00 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7ffa0401a000 tx=0x7ffa04015000).stop 2022-07-20T03:07:57.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 msgr2=0x7ffa1c0b44e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 0x7ffa1c0b44e0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7ffa0c0011f0 tx=0x7ffa0c007a00).stop 2022-07-20T03:07:57.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 shutdown_connections 2022-07-20T03:07:57.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff9f4053940 0x7ff9f4055e00 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa1c155d10 0x7ffa1c156190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa1c065400 0x7ffa1c155820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 --2- 172.21.15.61:0/1083893935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa00003e20 0x7ffa1c0b44e0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 >> 172.21.15.61:0/1083893935 conn(0x7ffa1c0b23d0 msgr2=0x7ffa1c0b3630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:57.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 shutdown_connections 2022-07-20T03:07:57.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:56.675+0000 7ffa21493700 1 -- 172.21.15.61:0/1083893935 wait complete. 2022-07-20T03:07:57.582 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mds 2022-07-20T03:07:57.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.711+0000 7f3c815f0700 1 Processor -- start 2022-07-20T03:07:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.711+0000 7f3c815f0700 1 -- start start 2022-07-20T03:07:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c7c0ad0e0 0x7f3c7c1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c7c0ae280 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c7c0b6be0 con 0x7f3c7c0ad0e0 2022-07-20T03:07:57.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c0ae4b0 con 0x7f3c7c0b6800 2022-07-20T03:07:57.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c7c0ad0e0 0x7f3c7c1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c7c0ad0e0 0x7f3c7c1418f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43556/0 (socket says 172.21.15.61:43556) 2022-07-20T03:07:57.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 -- 172.21.15.61:0/1095342695 learned_addr learned my addr 172.21.15.61:0/1095342695 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:57.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c7bfff700 1 --2- 172.21.15.61:0/1095342695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 -- 172.21.15.61:0/1095342695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 msgr2=0x7f3c7c0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:57.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 --2- 172.21.15.61:0/1095342695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c0aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 -- 172.21.15.61:0/1095342695 >> v1:172.21.15.117:6789/0 conn(0x7f3c7c0b3cb0 legacy=0x7f3c7c145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 -- 172.21.15.61:0/1095342695 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c7c0ae750 con 0x7f3c7c0ad0e0 2022-07-20T03:07:57.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c80def700 1 --2- 172.21.15.61:0/1095342695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c1418f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f3c6c002f70 tx=0x7f3c6c006eb0).ready entity=mon.0 client_cookie=5ffc5b176190c459 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:57.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c7affd700 1 -- 172.21.15.61:0/1095342695 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c00d570 con 0x7f3c7c0ad0e0 2022-07-20T03:07:57.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c7affd700 1 -- 172.21.15.61:0/1095342695 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c6c000e90 con 0x7f3c7c0ad0e0 2022-07-20T03:07:57.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c7affd700 1 -- 172.21.15.61:0/1095342695 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c6c00d900 con 0x7f3c7c0ad0e0 2022-07-20T03:07:57.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- 172.21.15.61:0/1095342695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0ad0e0 msgr2=0x7f3c7c1418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 --2- 172.21.15.61:0/1095342695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c1418f0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f3c6c002f70 tx=0x7f3c6c006eb0).stop 2022-07-20T03:07:57.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- 172.21.15.61:0/1095342695 shutdown_connections 2022-07-20T03:07:57.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 --2- 172.21.15.61:0/1095342695 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c1418f0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 --2- 172.21.15.61:0/1095342695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- 172.21.15.61:0/1095342695 >> 172.21.15.61:0/1095342695 conn(0x7f3c7c0b9dd0 msgr2=0x7f3c7c0b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:57.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- 172.21.15.61:0/1095342695 shutdown_connections 2022-07-20T03:07:57.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.715+0000 7f3c815f0700 1 -- 172.21.15.61:0/1095342695 wait complete. 2022-07-20T03:07:57.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 Processor -- start 2022-07-20T03:07:57.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 -- start start 2022-07-20T03:07:57.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 0x7f3c7c0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c13f8f0 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c1151d0 con 0x7f3c7c0ad0e0 2022-07-20T03:07:57.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c7c152040 con 0x7f3c7c0b6800 2022-07-20T03:07:57.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 0x7f3c7c0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 0x7f3c7c0af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43560/0 (socket says 172.21.15.61:43560) 2022-07-20T03:07:57.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c155450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:53238/0 (socket says 172.21.15.61:53238) 2022-07-20T03:07:57.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 -- 172.21.15.61:0/2510415072 learned_addr learned my addr 172.21.15.61:0/2510415072 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:57.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c80def700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 -- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 msgr2=0x7f3c7c1514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c1514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 -- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c7c0ad0e0 msgr2=0x7f3c7c155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 -- 172.21.15.61:0/2510415072 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c6c00b1e0 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c7b7fe700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 0x7f3c7c0af9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3c7c066600 tx=0x7f3c70007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c70012400 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c70012560 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c7c0afeb0 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c700128c0 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.719+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c7c0b23e0 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.723+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c7c115390 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.723+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3c70019030 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.723+0000 7f3c797fa700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3c5c053ad0 0x7f3c5c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:57.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.723+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3c7006af50 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.723+0000 7f3c7bfff700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3c5c053ad0 0x7f3c5c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:57.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.723+0000 7f3c7bfff700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3c5c053ad0 0x7f3c5c055f90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3c7400b000 tx=0x7f3c740091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:57.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.731+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c700421e0 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.903+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mds"} v 0) v1 -- 0x7f3c7c152200 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.911+0000 7f3c797fa700 1 -- 172.21.15.61:0/2510415072 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mds"}]=0 updated v7) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3c70045800 con 0x7f3c7c0b3cb0 2022-07-20T03:07:57.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3c5c053ad0 msgr2=0x7f3c5c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3c5c053ad0 0x7f3c5c055f90 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f3c7400b000 tx=0x7f3c740091d0).stop 2022-07-20T03:07:57.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 msgr2=0x7f3c7c0af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:57.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 0x7f3c7c0af9c0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f3c7c066600 tx=0x7f3c70007c50).stop 2022-07-20T03:07:57.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 shutdown_connections 2022-07-20T03:07:57.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3c5c053ad0 0x7f3c5c055f90 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c7c0b6800 0x7f3c7c1514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c7c0b3cb0 0x7f3c7c0af9c0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 --2- 172.21.15.61:0/2510415072 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c7c0ad0e0 0x7f3c7c155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 >> 172.21.15.61:0/2510415072 conn(0x7f3c7c0b9dd0 msgr2=0x7f3c7c140300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 shutdown_connections 2022-07-20T03:07:57.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:57.915+0000 7f3c815f0700 1 -- 172.21.15.61:0/2510415072 wait complete. 2022-07-20T03:07:57.945 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-20T03:07:57.980 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-mgr 2022-07-20T03:07:58.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 Processor -- start 2022-07-20T03:07:58.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 -- start start 2022-07-20T03:07:58.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0188097a80 0x7f0188061080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f01880afb80 0x7f018814b290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f018814a750 con 0x7f0188060ca0 2022-07-20T03:07:58.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0188148430 con 0x7f01880afb80 2022-07-20T03:07:58.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f018d5ea700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01880b2840 con 0x7f0188097a80 2022-07-20T03:07:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f01877fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f01880afb80 0x7f018814b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f0186ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0188060ca0 0x7f018814ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40164/0 (socket says 172.21.15.61:40164) 2022-07-20T03:07:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.091+0000 7f0186ffd700 1 -- 172.21.15.61:0/387549551 learned_addr learned my addr 172.21.15.61:0/387549551 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f01867fc700 1 --2- 172.21.15.61:0/387549551 >> v2:172.21.15.117:3300/0 conn(0x7f0188097a80 0x7f0188061080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f01877fe700 1 -- 172.21.15.61:0/387549551 >> v2:172.21.15.117:3300/0 conn(0x7f0188097a80 msgr2=0x7f0188061080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:58.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f01877fe700 1 --2- 172.21.15.61:0/387549551 >> v2:172.21.15.117:3300/0 conn(0x7f0188097a80 0x7f0188061080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f01877fe700 1 -- 172.21.15.61:0/387549551 >> v1:172.21.15.61:6789/0 conn(0x7f0188060ca0 legacy=0x7f018814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f01877fe700 1 -- 172.21.15.61:0/387549551 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01880ac110 con 0x7f01880afb80 2022-07-20T03:07:58.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f01877fe700 1 --2- 172.21.15.61:0/387549551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f018814b290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0180002f70 tx=0x7f0180006df0).ready entity=mon.0 client_cookie=d0801e0e546d820d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:58.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f0185ffb700 1 -- 172.21.15.61:0/387549551 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f018000d570 con 0x7f01880afb80 2022-07-20T03:07:58.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f0185ffb700 1 -- 172.21.15.61:0/387549551 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0180000e90 con 0x7f01880afb80 2022-07-20T03:07:58.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f0185ffb700 1 -- 172.21.15.61:0/387549551 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f018000d8c0 con 0x7f01880afb80 2022-07-20T03:07:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 -- 172.21.15.61:0/387549551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 msgr2=0x7f018814b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 --2- 172.21.15.61:0/387549551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f018814b290 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f0180002f70 tx=0x7f0180006df0).stop 2022-07-20T03:07:58.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 -- 172.21.15.61:0/387549551 shutdown_connections 2022-07-20T03:07:58.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 --2- 172.21.15.61:0/387549551 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f018814b290 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 --2- 172.21.15.61:0/387549551 >> v2:172.21.15.117:3300/0 conn(0x7f0188097a80 0x7f0188061080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 -- 172.21.15.61:0/387549551 >> 172.21.15.61:0/387549551 conn(0x7f01880b5cb0 msgr2=0x7f01880b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:58.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 -- 172.21.15.61:0/387549551 shutdown_connections 2022-07-20T03:07:58.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.095+0000 7f018d5ea700 1 -- 172.21.15.61:0/387549551 wait complete. 2022-07-20T03:07:58.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 Processor -- start 2022-07-20T03:07:58.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 -- start start 2022-07-20T03:07:58.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0188060ca0 0x7f018814b920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0188097a80 0x7f0188141110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f0188141600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01880af5c0 con 0x7f01880afb80 2022-07-20T03:07:58.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0188141af0 con 0x7f0188060ca0 2022-07-20T03:07:58.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01880af320 con 0x7f0188097a80 2022-07-20T03:07:58.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f0188141600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f0188141600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43576/0 (socket says 172.21.15.61:43576) 2022-07-20T03:07:58.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 -- 172.21.15.61:0/3375562667 learned_addr learned my addr 172.21.15.61:0/3375562667 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:58.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f0186ffd700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0188060ca0 0x7f018814b920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01867fc700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0188097a80 0x7f0188141110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 -- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0188097a80 msgr2=0x7f0188141110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0188097a80 0x7f0188141110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 -- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0188060ca0 msgr2=0x7f018814b920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0188060ca0 0x7f018814b920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 -- 172.21.15.61:0/3375562667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f018000b180 con 0x7f01880afb80 2022-07-20T03:07:58.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01867fc700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0188097a80 0x7f0188141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:07:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f01877fe700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f0188141600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0180002f70 tx=0x7f0180014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:58.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0180013ca0 con 0x7f01880afb80 2022-07-20T03:07:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0180013e00 con 0x7f01880afb80 2022-07-20T03:07:58.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0188156620 con 0x7f01880afb80 2022-07-20T03:07:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f018001e5f0 con 0x7f01880afb80 2022-07-20T03:07:58.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.099+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f018814a1d0 con 0x7f01880afb80 2022-07-20T03:07:58.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.103+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f01881567b0 con 0x7f01880afb80 2022-07-20T03:07:58.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.103+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0180025030 con 0x7f01880afb80 2022-07-20T03:07:58.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.103+0000 7f0177fff700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0164053a40 0x7f0164055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.103+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f018006c7f0 con 0x7f01880afb80 2022-07-20T03:07:58.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.103+0000 7f0186ffd700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0164053a40 0x7f0164055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.103+0000 7f0186ffd700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0164053a40 0x7f0164055f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f017c00d000 tx=0x7f017c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:58.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.107+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0180043a80 con 0x7f01880afb80 2022-07-20T03:07:58.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.271+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-mgr"} v 0) v1 -- 0x7f01880abe40 con 0x7f01880afb80 2022-07-20T03:07:58.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.279+0000 7f0177fff700 1 -- 172.21.15.61:0/3375562667 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-mgr"}]=0 updated v8) v1 ==== 97+0+0 (secure 0 0 0) 0x7f0180023070 con 0x7f01880afb80 2022-07-20T03:07:58.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.279+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0164053a40 msgr2=0x7f0164055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.279+0000 7f018d5ea700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0164053a40 0x7f0164055f00 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f017c00d000 tx=0x7f017c009040).stop 2022-07-20T03:07:58.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.279+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 msgr2=0x7f0188141600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.279+0000 7f018d5ea700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f0188141600 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0180002f70 tx=0x7f0180014000).stop 2022-07-20T03:07:58.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 shutdown_connections 2022-07-20T03:07:58.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0164053a40 0x7f0164055f00 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01880afb80 0x7f0188141600 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0188097a80 0x7f0188141110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 --2- 172.21.15.61:0/3375562667 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0188060ca0 0x7f018814b920 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 >> 172.21.15.61:0/3375562667 conn(0x7f01880b5cb0 msgr2=0x7f01880b49c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:58.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 shutdown_connections 2022-07-20T03:07:58.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.283+0000 7f018d5ea700 1 -- 172.21.15.61:0/3375562667 wait complete. 2022-07-20T03:07:58.447 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-20T03:07:58.447 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-osd 2022-07-20T03:07:58.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.627+0000 7ff321e4d700 1 Processor -- start 2022-07-20T03:07:58.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.627+0000 7ff321e4d700 1 -- start start 2022-07-20T03:07:58.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff31c0b02a0 0x7ff31c146a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c146f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff31c146340 con 0x7ff31c060fe0 2022-07-20T03:07:58.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff31c0613c0 con 0x7ff31c0b02a0 2022-07-20T03:07:58.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c0ac110 con 0x7ff31c0afb80 2022-07-20T03:07:58.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff31c0b02a0 0x7ff31c146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff31c0b02a0 0x7ff31c146a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43582/0 (socket says 172.21.15.61:43582) 2022-07-20T03:07:58.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 -- 172.21.15.61:0/2192393025 learned_addr learned my addr 172.21.15.61:0/2192393025 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:58.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff31bfff700 1 --2- 172.21.15.61:0/2192393025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c146f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 -- 172.21.15.61:0/2192393025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 msgr2=0x7ff31c146f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:07:58.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 --2- 172.21.15.61:0/2192393025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c146f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 -- 172.21.15.61:0/2192393025 >> v1:172.21.15.117:6789/0 conn(0x7ff31c060fe0 legacy=0x7ff31c0b08b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 -- 172.21.15.61:0/2192393025 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff31c146570 con 0x7ff31c0b02a0 2022-07-20T03:07:58.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff313fff700 1 --2- 172.21.15.61:0/2192393025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c0b02a0 0x7ff31c146a10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff30c002f70 tx=0x7ff30c006eb0).ready entity=mon.0 client_cookie=3f26c9f71a90cc1f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:58.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff31affd700 1 -- 172.21.15.61:0/2192393025 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff30c00d570 con 0x7ff31c0b02a0 2022-07-20T03:07:58.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff31affd700 1 -- 172.21.15.61:0/2192393025 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff30c000e90 con 0x7ff31c0b02a0 2022-07-20T03:07:58.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff31affd700 1 -- 172.21.15.61:0/2192393025 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff30c00d900 con 0x7ff31c0b02a0 2022-07-20T03:07:58.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- 172.21.15.61:0/2192393025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c0b02a0 msgr2=0x7ff31c146a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 --2- 172.21.15.61:0/2192393025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c0b02a0 0x7ff31c146a10 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7ff30c002f70 tx=0x7ff30c006eb0).stop 2022-07-20T03:07:58.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- 172.21.15.61:0/2192393025 shutdown_connections 2022-07-20T03:07:58.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 --2- 172.21.15.61:0/2192393025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c146f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 --2- 172.21.15.61:0/2192393025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c0b02a0 0x7ff31c146a10 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- 172.21.15.61:0/2192393025 >> 172.21.15.61:0/2192393025 conn(0x7ff31c0b5cb0 msgr2=0x7ff31c0b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:58.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- 172.21.15.61:0/2192393025 shutdown_connections 2022-07-20T03:07:58.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.631+0000 7ff321e4d700 1 -- 172.21.15.61:0/2192393025 wait complete. 2022-07-20T03:07:58.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 Processor -- start 2022-07-20T03:07:58.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 -- start start 2022-07-20T03:07:58.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 0x7ff31c063ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c0643e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff31c0b02a0 0x7ff31c13deb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c157190 con 0x7ff31c060fe0 2022-07-20T03:07:58.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c13eb30 con 0x7ff31c0b02a0 2022-07-20T03:07:58.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff321e4d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff31c13edd0 con 0x7ff31c0afb80 2022-07-20T03:07:58.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 0x7ff31c063ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 0x7ff31c063ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43586/0 (socket says 172.21.15.61:43586) 2022-07-20T03:07:58.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 -- 172.21.15.61:0/976662736 learned_addr learned my addr 172.21.15.61:0/976662736 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:58.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff313fff700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c0643e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31bfff700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff31c0b02a0 0x7ff31c13deb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 -- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 msgr2=0x7ff31c0643e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c0643e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 -- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff31c0b02a0 msgr2=0x7ff31c13deb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff31c0b02a0 0x7ff31c13deb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 -- 172.21.15.61:0/976662736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff30c00b1e0 con 0x7ff31c060fe0 2022-07-20T03:07:58.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.635+0000 7ff31b7fe700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 0x7ff31c063ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ff31c05f930 tx=0x7ff304007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:58.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff304012400 con 0x7ff31c060fe0 2022-07-20T03:07:58.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff304012560 con 0x7ff31c060fe0 2022-07-20T03:07:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff31c097c60 con 0x7ff31c060fe0 2022-07-20T03:07:58.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3040128c0 con 0x7ff31c060fe0 2022-07-20T03:07:58.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff31c09ad50 con 0x7ff31c060fe0 2022-07-20T03:07:58.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff31c157940 con 0x7ff31c060fe0 2022-07-20T03:07:58.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff304019030 con 0x7ff31c060fe0 2022-07-20T03:07:58.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff3197fa700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff300053a40 0x7ff300055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:58.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7ff30406af70 con 0x7ff31c060fe0 2022-07-20T03:07:58.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.639+0000 7ff313fff700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff300053a40 0x7ff300055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:58.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.643+0000 7ff313fff700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff300053a40 0x7ff300055f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7ff30c01c000 tx=0x7ff30c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:58.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.643+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3040420f0 con 0x7ff31c060fe0 2022-07-20T03:07:58.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.807+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-osd"} v 0) v1 -- 0x7ff31c157360 con 0x7ff31c060fe0 2022-07-20T03:07:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.811+0000 7ff3197fa700 1 -- 172.21.15.61:0/976662736 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-osd"}]=0 updated v9) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff31c157360 con 0x7ff31c060fe0 2022-07-20T03:07:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff300053a40 msgr2=0x7ff300055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff300053a40 0x7ff300055f00 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7ff30c01c000 tx=0x7ff30c015000).stop 2022-07-20T03:07:58.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 msgr2=0x7ff31c063ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:58.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 0x7ff31c063ef0 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7ff31c05f930 tx=0x7ff304007c50).stop 2022-07-20T03:07:58.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 shutdown_connections 2022-07-20T03:07:58.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff300053a40 0x7ff300055f00 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff31c0b02a0 0x7ff31c13deb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff31c0afb80 0x7ff31c0643e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:58.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 --2- 172.21.15.61:0/976662736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff31c060fe0 0x7ff31c063ef0 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 >> 172.21.15.61:0/976662736 conn(0x7ff31c0b5cb0 msgr2=0x7ff31c13f490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:59.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 shutdown_connections 2022-07-20T03:07:59.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:58.815+0000 7ff321e4d700 1 -- 172.21.15.61:0/976662736 wait complete. 2022-07-20T03:07:59.002 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-20T03:07:59.003 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd 2022-07-20T03:07:59.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 Processor -- start 2022-07-20T03:07:59.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- start start 2022-07-20T03:07:59.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93240b6860 0x7f932405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9324063c00 con 0x7f932405c990 2022-07-20T03:07:59.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9324146c90 con 0x7f9324061ac0 2022-07-20T03:07:59.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93240b6c40 con 0x7f93240b6860 2022-07-20T03:07:59.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9329ddc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93240b6860 0x7f932405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932a5dd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f932405c990 0x7f93240b4800 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40196/0 (socket says 172.21.15.61:40196) 2022-07-20T03:07:59.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932a5dd700 1 -- 172.21.15.61:0/348302734 learned_addr learned my addr 172.21.15.61:0/348302734 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:59.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9329ddc700 1 -- 172.21.15.61:0/348302734 >> v1:172.21.15.117:6789/0 conn(0x7f9324061ac0 legacy=0x7f932405c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9329ddc700 1 -- 172.21.15.61:0/348302734 >> v1:172.21.15.61:6789/0 conn(0x7f932405c990 legacy=0x7f93240b4800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9329ddc700 1 -- 172.21.15.61:0/348302734 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9324063e30 con 0x7f93240b6860 2022-07-20T03:07:59.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9329ddc700 1 --2- 172.21.15.61:0/348302734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93240b6860 0x7f932405bd90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9324066600 tx=0x7f9314006df0).ready entity=mon.0 client_cookie=3e0e31b4b38ac046 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:59.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9328dda700 1 -- 172.21.15.61:0/348302734 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f931400d610 con 0x7f93240b6860 2022-07-20T03:07:59.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9328dda700 1 -- 172.21.15.61:0/348302734 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9314000e90 con 0x7f93240b6860 2022-07-20T03:07:59.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f9328dda700 1 -- 172.21.15.61:0/348302734 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f931400d960 con 0x7f93240b6860 2022-07-20T03:07:59.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- 172.21.15.61:0/348302734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93240b6860 msgr2=0x7f932405bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 --2- 172.21.15.61:0/348302734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93240b6860 0x7f932405bd90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f9324066600 tx=0x7f9314006df0).stop 2022-07-20T03:07:59.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- 172.21.15.61:0/348302734 shutdown_connections 2022-07-20T03:07:59.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 --2- 172.21.15.61:0/348302734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93240b6860 0x7f932405bd90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.111+0000 7f932adde700 1 -- 172.21.15.61:0/348302734 >> 172.21.15.61:0/348302734 conn(0x7f93240b9dd0 msgr2=0x7f93240b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:59.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 -- 172.21.15.61:0/348302734 shutdown_connections 2022-07-20T03:07:59.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 -- 172.21.15.61:0/348302734 wait complete. 2022-07-20T03:07:59.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 Processor -- start 2022-07-20T03:07:59.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 -- start start 2022-07-20T03:07:59.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f932405c990 0x7f93240b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9324061ac0 0x7f9324149b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93240b6860 0x7f93241407f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9324156090 con 0x7f9324061ac0 2022-07-20T03:07:59.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93241450b0 con 0x7f93240b6860 2022-07-20T03:07:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932adde700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9324145350 con 0x7f932405c990 2022-07-20T03:07:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932a5dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93240b6860 0x7f93241407f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f9329ddc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f932405c990 0x7f93240b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f93295db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9324061ac0 0x7f9324149b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f9329ddc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f932405c990 0x7f93240b84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58504/0 (socket says 172.21.15.61:58504) 2022-07-20T03:07:59.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f9329ddc700 1 -- 172.21.15.61:0/3114520321 learned_addr learned my addr 172.21.15.61:0/3114520321 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:59.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f932a5dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93240b6860 0x7f93241407f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:53282/0 (socket says 172.21.15.61:53282) 2022-07-20T03:07:59.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f93295db700 1 -- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f932405c990 msgr2=0x7f93240b84e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f93295db700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f932405c990 0x7f93240b84e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f93295db700 1 -- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93240b6860 msgr2=0x7f93241407f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f93295db700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93240b6860 0x7f93241407f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.115+0000 7f93295db700 1 -- 172.21.15.61:0/3114520321 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f931400b180 con 0x7f9324061ac0 2022-07-20T03:07:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f93295db700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9324061ac0 0x7f9324149b70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f93240b41b0 tx=0x7f931c007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f931c0014a0 con 0x7f9324061ac0 2022-07-20T03:07:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f931c00f470 con 0x7f9324061ac0 2022-07-20T03:07:59.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93241455f0 con 0x7f9324061ac0 2022-07-20T03:07:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f931c00f7d0 con 0x7f9324061ac0 2022-07-20T03:07:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93240acd00 con 0x7f9324061ac0 2022-07-20T03:07:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f932414a060 con 0x7f9324061ac0 2022-07-20T03:07:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f931c019030 con 0x7f9324061ac0 2022-07-20T03:07:59.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9312ffd700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9304053a40 0x7f9304055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f931c06b770 con 0x7f9324061ac0 2022-07-20T03:07:59.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.119+0000 7f9329ddc700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9304053a40 0x7f9304055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.123+0000 7f9329ddc700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9304053a40 0x7f9304055f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f931401c000 tx=0x7f9314015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:59.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.123+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f931c0428f0 con 0x7f9324061ac0 2022-07-20T03:07:59.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.295+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd"} v 0) v1 -- 0x7f93240b3e30 con 0x7f9324061ac0 2022-07-20T03:07:59.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.299+0000 7f9312ffd700 1 -- 172.21.15.61:0/3114520321 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd"}]=0 updated v10) v1 ==== 97+0+0 (secure 0 0 0) 0x7f931c045f10 con 0x7f9324061ac0 2022-07-20T03:07:59.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9304053a40 msgr2=0x7f9304055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9304053a40 0x7f9304055f00 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f931401c000 tx=0x7f9314015000).stop 2022-07-20T03:07:59.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9324061ac0 msgr2=0x7f9324149b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9324061ac0 0x7f9324149b70 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f93240b41b0 tx=0x7f931c007f10).stop 2022-07-20T03:07:59.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 shutdown_connections 2022-07-20T03:07:59.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9304053a40 0x7f9304055f00 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93240b6860 0x7f93241407f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9324061ac0 0x7f9324149b70 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 --2- 172.21.15.61:0/3114520321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f932405c990 0x7f93240b84e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 >> 172.21.15.61:0/3114520321 conn(0x7f93240b9dd0 msgr2=0x7f9324066d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:59.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 shutdown_connections 2022-07-20T03:07:59.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.303+0000 7f932adde700 1 -- 172.21.15.61:0/3114520321 wait complete. 2022-07-20T03:07:59.349 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-20T03:07:59.351 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rbd-mirror 2022-07-20T03:07:59.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.539+0000 7f134ccc3700 1 Processor -- start 2022-07-20T03:07:59.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.539+0000 7f134ccc3700 1 -- start start 2022-07-20T03:07:59.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13480b1480 0x7f13480ac040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13480ac6e0 con 0x7f134805fc80 2022-07-20T03:07:59.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1348140080 con 0x7f134805f840 2022-07-20T03:07:59.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f134813f230 con 0x7f13480b1480 2022-07-20T03:07:59.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13480b1480 0x7f13480ac040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f1346d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f134805fc80 0x7f13480acd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40210/0 (socket says 172.21.15.61:40210) 2022-07-20T03:07:59.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f1346d9d700 1 -- 172.21.15.61:0/4060101858 learned_addr learned my addr 172.21.15.61:0/4060101858 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:07:59.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 370548248 0 0) 0x7f13480ac6e0 con 0x7f134805fc80 2022-07-20T03:07:59.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f132c003260 con 0x7f134805fc80 2022-07-20T03:07:59.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 518837606 0 0) 0x7f1348140080 con 0x7f134805f840 2022-07-20T03:07:59.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13480ac6e0 con 0x7f134805f840 2022-07-20T03:07:59.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134659c700 1 -- 172.21.15.61:0/4060101858 >> v1:172.21.15.117:6789/0 conn(0x7f134805f840 legacy=0x7f134813f970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134659c700 1 -- 172.21.15.61:0/4060101858 >> v1:172.21.15.61:6789/0 conn(0x7f134805fc80 legacy=0x7f13480acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134659c700 1 -- 172.21.15.61:0/4060101858 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f134813f4d0 con 0x7f13480b1480 2022-07-20T03:07:59.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134659c700 1 --2- 172.21.15.61:0/4060101858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13480b1480 0x7f13480ac040 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f133c002f70 tx=0x7f133c006e80).ready entity=mon.0 client_cookie=93e2f8d7e4f7289 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:59.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f133c014070 con 0x7f13480b1480 2022-07-20T03:07:59.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133c000e90 con 0x7f13480b1480 2022-07-20T03:07:59.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134559a700 1 -- 172.21.15.61:0/4060101858 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f133c0107f0 con 0x7f13480b1480 2022-07-20T03:07:59.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- 172.21.15.61:0/4060101858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13480b1480 msgr2=0x7f13480ac040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 --2- 172.21.15.61:0/4060101858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13480b1480 0x7f13480ac040 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f133c002f70 tx=0x7f133c006e80).stop 2022-07-20T03:07:59.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- 172.21.15.61:0/4060101858 shutdown_connections 2022-07-20T03:07:59.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 --2- 172.21.15.61:0/4060101858 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13480b1480 0x7f13480ac040 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- 172.21.15.61:0/4060101858 >> 172.21.15.61:0/4060101858 conn(0x7f13480b23d0 msgr2=0x7f13480b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:07:59.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- 172.21.15.61:0/4060101858 shutdown_connections 2022-07-20T03:07:59.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.543+0000 7f134ccc3700 1 -- 172.21.15.61:0/4060101858 wait complete. 2022-07-20T03:07:59.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.547+0000 7f134ccc3700 1 Processor -- start 2022-07-20T03:07:59.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.547+0000 7f134ccc3700 1 -- start start 2022-07-20T03:07:59.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f134805f840 0x7f13480666c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 0x7f1348064400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13480b1480 0x7f1348150020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348065620 con 0x7f134805fc80 2022-07-20T03:07:59.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1348142c30 con 0x7f13480b1480 2022-07-20T03:07:59.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1348142ed0 con 0x7f134805f840 2022-07-20T03:07:59.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 0x7f1348064400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 0x7f1348064400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43616/0 (socket says 172.21.15.61:43616) 2022-07-20T03:07:59.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 -- 172.21.15.61:0/2629944226 learned_addr learned my addr 172.21.15.61:0/2629944226 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:07:59.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134659c700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f134805f840 0x7f13480666c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1346d9d700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13480b1480 0x7f1348150020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 -- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f134805f840 msgr2=0x7f13480666c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f134805f840 0x7f13480666c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 -- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13480b1480 msgr2=0x7f1348150020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:07:59.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13480b1480 0x7f1348150020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:07:59.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 -- 172.21.15.61:0/2629944226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c00c310 con 0x7f134805fc80 2022-07-20T03:07:59.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f1345d9b700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 0x7f1348064400 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1338001110 tx=0x7f1338007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:59.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1338012400 con 0x7f134805fc80 2022-07-20T03:07:59.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1338012560 con 0x7f134805fc80 2022-07-20T03:07:59.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1348140790 con 0x7f134805fc80 2022-07-20T03:07:59.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13380128c0 con 0x7f134805fc80 2022-07-20T03:07:59.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1348066bb0 con 0x7f134805fc80 2022-07-20T03:07:59.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1348140920 con 0x7f134805fc80 2022-07-20T03:07:59.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.551+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1338019030 con 0x7f134805fc80 2022-07-20T03:07:59.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.555+0000 7f13377fe700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1324053a40 0x7f1324055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:07:59.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.555+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f133806aea0 con 0x7f134805fc80 2022-07-20T03:07:59.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.555+0000 7f134659c700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1324053a40 0x7f1324055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:07:59.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.555+0000 7f134659c700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1324053a40 0x7f1324055f00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f133c01a000 tx=0x7f13480613f0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:07:59.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.563+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1338042130 con 0x7f134805fc80 2022-07-20T03:08:00.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.727+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"} v 0) v1 -- 0x7f13480574e0 con 0x7f134805fc80 2022-07-20T03:08:00.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.735+0000 7f13377fe700 1 -- 172.21.15.61:0/2629944226 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rbd-mirror"}]=0 updated v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f1338045750 con 0x7f134805fc80 2022-07-20T03:08:00.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1324053a40 msgr2=0x7f1324055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1324053a40 0x7f1324055f00 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f133c01a000 tx=0x7f13480613f0).stop 2022-07-20T03:08:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 msgr2=0x7f1348064400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 0x7f1348064400 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f1338001110 tx=0x7f1338007a90).stop 2022-07-20T03:08:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 shutdown_connections 2022-07-20T03:08:00.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1324053a40 0x7f1324055f00 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13480b1480 0x7f1348150020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f134805fc80 0x7f1348064400 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 --2- 172.21.15.61:0/2629944226 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f134805f840 0x7f13480666c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 >> 172.21.15.61:0/2629944226 conn(0x7f13480b23d0 msgr2=0x7f13480b27b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:00.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 shutdown_connections 2022-07-20T03:08:00.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:07:59.739+0000 7f134ccc3700 1 -- 172.21.15.61:0/2629944226 wait complete. 2022-07-20T03:08:00.138 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-20T03:08:00.139 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth del client.bootstrap-rgw 2022-07-20T03:08:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 Processor -- start 2022-07-20T03:08:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 -- start start 2022-07-20T03:08:00.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3cfc0b5320 0x7f3cfc05c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3cfc0b0c20 con 0x7f3cfc05c770 2022-07-20T03:08:00.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3cfc05cbe0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d02a9c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc0ba290 con 0x7f3cfc0b9eb0 2022-07-20T03:08:00.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d01039700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3cfc05c770 0x7f3cfc141d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40222/0 (socket says 172.21.15.61:40222) 2022-07-20T03:08:00.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d01039700 1 -- 172.21.15.61:0/4273597348 learned_addr learned my addr 172.21.15.61:0/4273597348 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:00.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3cfbfff700 1 --2- 172.21.15.61:0/4273597348 >> v2:172.21.15.117:3300/0 conn(0x7f3cfc0b5320 0x7f3cfc05c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3d00838700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3cfb7fe700 1 -- 172.21.15.61:0/4273597348 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 593194290 0 0) 0x7f3cfc0b0c20 con 0x7f3cfc05c770 2022-07-20T03:08:00.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.247+0000 7f3cfb7fe700 1 -- 172.21.15.61:0/4273597348 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3cdc003260 con 0x7f3cfc05c770 2022-07-20T03:08:00.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfbfff700 1 -- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 msgr2=0x7f3cfc05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:00.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfbfff700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc05bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfbfff700 1 -- 172.21.15.61:0/4273597348 >> v1:172.21.15.61:6789/0 conn(0x7f3cfc05c770 legacy=0x7f3cfc141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfbfff700 1 -- 172.21.15.61:0/4273597348 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cfc142e00 con 0x7f3cfc0b5320 2022-07-20T03:08:00.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d00838700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc05bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:00.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfbfff700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc05c280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4006df0).ready entity=mon.1 client_cookie=a5d595eebb7919bc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:00.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfb7fe700 1 -- 172.21.15.61:0/4273597348 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce400d4a0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfb7fe700 1 -- 172.21.15.61:0/4273597348 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4000e90 con 0x7f3cfc0b5320 2022-07-20T03:08:00.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3cfb7fe700 1 -- 172.21.15.61:0/4273597348 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce400d7f0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 -- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc0b5320 msgr2=0x7f3cfc05c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc05c280 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4006df0).stop 2022-07-20T03:08:00.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 -- 172.21.15.61:0/4273597348 shutdown_connections 2022-07-20T03:08:00.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc05c280 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/4273597348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc05bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 -- 172.21.15.61:0/4273597348 >> 172.21.15.61:0/4273597348 conn(0x7f3cfc067980 msgr2=0x7f3cfc0b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:00.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 -- 172.21.15.61:0/4273597348 shutdown_connections 2022-07-20T03:08:00.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 -- 172.21.15.61:0/4273597348 wait complete. 2022-07-20T03:08:00.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 Processor -- start 2022-07-20T03:08:00.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.251+0000 7f3d02a9c700 1 -- start start 2022-07-20T03:08:00.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc05c770 0x7f3cfc0b7810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc13b580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc13ba70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc1477f0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc147a20 con 0x7f3cfc05c770 2022-07-20T03:08:00.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3cfc13de90 con 0x7f3cfc0b9eb0 2022-07-20T03:08:00.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc13b580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc13b580 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43630/0 (socket says 172.21.15.61:43630) 2022-07-20T03:08:00.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 -- 172.21.15.61:0/3770382135 learned_addr learned my addr 172.21.15.61:0/3770382135 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:00.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d00838700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc05c770 0x7f3cfc0b7810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d01039700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc13ba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 -- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 msgr2=0x7f3cfc13ba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc13ba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 -- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc05c770 msgr2=0x7f3cfc0b7810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc05c770 0x7f3cfc0b7810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 -- 172.21.15.61:0/3770382135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cec00b180 con 0x7f3cfc0b5320 2022-07-20T03:08:00.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cfbfff700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc13b580 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:00.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce4012930 con 0x7f3cfc0b5320 2022-07-20T03:08:00.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ce400b180 con 0x7f3cfc0b5320 2022-07-20T03:08:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ce4012a90 con 0x7f3cfc0b5320 2022-07-20T03:08:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cfc13e150 con 0x7f3cfc0b5320 2022-07-20T03:08:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ce4012c00 con 0x7f3cfc0b5320 2022-07-20T03:08:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.255+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cfc0afed0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.259+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3ce401c070 con 0x7f3cfc0b5320 2022-07-20T03:08:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.259+0000 7f3cf9ffb700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ce0053ad0 0x7f3ce0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.259+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f3ce406c530 con 0x7f3cfc0b5320 2022-07-20T03:08:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.259+0000 7f3d00838700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ce0053ad0 0x7f3ce0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.259+0000 7f3d00838700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ce0053ad0 0x7f3ce0055f90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3cec00b000 tx=0x7f3cec006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:00.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.267+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ce40437d0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.431+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth del", "entity": "client.bootstrap-rgw"} v 0) v1 -- 0x7f3cfc0574e0 con 0x7f3cfc0b5320 2022-07-20T03:08:00.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.439+0000 7f3cf9ffb700 1 -- 172.21.15.61:0/3770382135 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth del", "entity": "client.bootstrap-rgw"}]=0 updated v12) v1 ==== 97+0+0 (secure 0 0 0) 0x7f3ce4019070 con 0x7f3cfc0b5320 2022-07-20T03:08:00.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ce0053ad0 msgr2=0x7f3ce0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ce0053ad0 0x7f3ce0055f90 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f3cec00b000 tx=0x7f3cec006df0).stop 2022-07-20T03:08:00.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 msgr2=0x7f3cfc13b580 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc13b580 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f3ce4002f70 tx=0x7f3ce4014000).stop 2022-07-20T03:08:00.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 shutdown_connections 2022-07-20T03:08:00.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ce0053ad0 0x7f3ce0055f90 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3cfc0b9eb0 0x7f3cfc13ba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3cfc0b5320 0x7f3cfc13b580 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 --2- 172.21.15.61:0/3770382135 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3cfc05c770 0x7f3cfc0b7810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 >> 172.21.15.61:0/3770382135 conn(0x7f3cfc067980 msgr2=0x7f3cfc142600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:00.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 shutdown_connections 2022-07-20T03:08:00.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.443+0000 7f3d02a9c700 1 -- 172.21.15.61:0/3770382135 wait complete. 2022-07-20T03:08:00.601 INFO:teuthology.orchestra.run.smithi061.stderr:updated 2022-07-20T03:08:00.603 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:08:00.655 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:00.693 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:00.749 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:00.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 Processor -- start 2022-07-20T03:08:00.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 -- start start 2022-07-20T03:08:00.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc4d80b1200 0x7fc4d805c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4d80aea30 con 0x7fc4d805bd90 2022-07-20T03:08:00.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4d8142970 con 0x7fc4d80b1200 2022-07-20T03:08:00.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4dd132700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80aec60 con 0x7fc4d80b5650 2022-07-20T03:08:00.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d80b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58540/0 (socket says 172.21.15.61:58540) 2022-07-20T03:08:00.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 -- 172.21.15.61:0/2818915818 learned_addr learned my addr 172.21.15.61:0/2818915818 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:00.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d77fe700 1 --2- 172.21.15.61:0/2818915818 >> v2:172.21.15.117:3300/0 conn(0x7fc4d80b1200 0x7fc4d805c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 -- 172.21.15.61:0/2818915818 >> v2:172.21.15.117:3300/0 conn(0x7fc4d80b1200 msgr2=0x7fc4d805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:00.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 --2- 172.21.15.61:0/2818915818 >> v2:172.21.15.117:3300/0 conn(0x7fc4d80b1200 0x7fc4d805c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 -- 172.21.15.61:0/2818915818 >> v1:172.21.15.117:6789/0 conn(0x7fc4d805bd90 legacy=0x7fc4d8142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 -- 172.21.15.61:0/2818915818 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4d80af180 con 0x7fc4d80b5650 2022-07-20T03:08:00.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d6ffd700 1 -- 172.21.15.61:0/2818915818 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2682904257 0 0) 0x7fc4d80aea30 con 0x7fc4d805bd90 2022-07-20T03:08:00.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.723+0000 7fc4d7fff700 1 --2- 172.21.15.61:0/2818915818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d80b5a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc4c80032f0 tx=0x7fc4c8009290).ready entity=mon.2 client_cookie=b6e73dc73377baf9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:00.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4d6ffd700 1 -- 172.21.15.61:0/2818915818 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4c800b8a0 con 0x7fc4d80b5650 2022-07-20T03:08:00.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4d6ffd700 1 -- 172.21.15.61:0/2818915818 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c800ba00 con 0x7fc4d80b5650 2022-07-20T03:08:00.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4d6ffd700 1 -- 172.21.15.61:0/2818915818 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4c800bb70 con 0x7fc4d80b5650 2022-07-20T03:08:00.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 -- 172.21.15.61:0/2818915818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 msgr2=0x7fc4d80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 --2- 172.21.15.61:0/2818915818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d80b5a30 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fc4c80032f0 tx=0x7fc4c8009290).stop 2022-07-20T03:08:00.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 -- 172.21.15.61:0/2818915818 shutdown_connections 2022-07-20T03:08:00.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 --2- 172.21.15.61:0/2818915818 >> v2:172.21.15.117:3300/0 conn(0x7fc4d80b1200 0x7fc4d805c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 --2- 172.21.15.61:0/2818915818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d80b5a30 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 -- 172.21.15.61:0/2818915818 >> 172.21.15.61:0/2818915818 conn(0x7fc4d80634e0 msgr2=0x7fc4d80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:00.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 -- 172.21.15.61:0/2818915818 shutdown_connections 2022-07-20T03:08:00.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.727+0000 7fc4dd132700 1 -- 172.21.15.61:0/2818915818 wait complete. 2022-07-20T03:08:00.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 Processor -- start 2022-07-20T03:08:00.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 -- start start 2022-07-20T03:08:00.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 0x7fc4d80b3440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d80b1200 0x7fc4d8067160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d8155210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d813f9b0 con 0x7fc4d805bd90 2022-07-20T03:08:00.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 0x7fc4d80b3440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d81427c0 con 0x7fc4d80b1200 2022-07-20T03:08:00.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dd132700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc4d80644b0 con 0x7fc4d80b5650 2022-07-20T03:08:00.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 0x7fc4d80b3440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43642/0 (socket says 172.21.15.61:43642) 2022-07-20T03:08:00.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d77fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d80b1200 0x7fc4d8067160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 -- 172.21.15.61:0/3412152019 learned_addr learned my addr 172.21.15.61:0/3412152019 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:00.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4dc931700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d8155210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 -- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 msgr2=0x7fc4d8155210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d8155210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 -- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d80b1200 msgr2=0x7fc4d8067160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d80b1200 0x7fc4d8067160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 -- 172.21.15.61:0/3412152019 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4c8009530 con 0x7fc4d805bd90 2022-07-20T03:08:00.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.731+0000 7fc4d7fff700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 0x7fc4d80b3440 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc4d8151cd0 tx=0x7fc4c8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:00.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4c800bb70 con 0x7fc4d805bd90 2022-07-20T03:08:00.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4c8021670 con 0x7fc4d805bd90 2022-07-20T03:08:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4d8064750 con 0x7fc4d805bd90 2022-07-20T03:08:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc4c800b8a0 con 0x7fc4d805bd90 2022-07-20T03:08:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4d8067650 con 0x7fc4d805bd90 2022-07-20T03:08:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4d8066ec0 con 0x7fc4d805bd90 2022-07-20T03:08:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.735+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc4c800f070 con 0x7fc4d805bd90 2022-07-20T03:08:00.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.739+0000 7fc4d57fa700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4bc053a40 0x7fc4bc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:00.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.739+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc4c806cd80 con 0x7fc4d805bd90 2022-07-20T03:08:00.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.739+0000 7fc4d77fe700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4bc053a40 0x7fc4bc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:00.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.739+0000 7fc4d77fe700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4bc053a40 0x7fc4bc055f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc4c0006000 tx=0x7fc4c0006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:00.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.743+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4d8066ec0 con 0x7fc4d805bd90 2022-07-20T03:08:00.811 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:00.816 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:00.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.867+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc4d8066ec0 con 0x7fc4d805bd90 2022-07-20T03:08:00.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.867+0000 7fc4d57fa700 1 -- 172.21.15.61:0/3412152019 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v29) v1 ==== 74+0+13229 (secure 0 0 0) 0x7fc4d8066ec0 con 0x7fc4d805bd90 2022-07-20T03:08:00.876 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:00.876 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:07:56.239265+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":4,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":20,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":19,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":21,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":16,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:08:00.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4bc053a40 msgr2=0x7fc4bc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4bc053a40 0x7fc4bc055f00 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fc4c0006000 tx=0x7fc4c0006d90).stop 2022-07-20T03:08:00.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 msgr2=0x7fc4d80b3440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:00.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 0x7fc4d80b3440 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fc4d8151cd0 tx=0x7fc4c8016000).stop 2022-07-20T03:08:00.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 shutdown_connections 2022-07-20T03:08:00.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4bc053a40 0x7fc4bc055f00 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4d80b1200 0x7fc4d8067160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4d80b5650 0x7fc4d8155210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 --2- 172.21.15.61:0/3412152019 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4d805bd90 0x7fc4d80b3440 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:00.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 >> 172.21.15.61:0/3412152019 conn(0x7fc4d80634e0 msgr2=0x7fc4d80afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:00.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 shutdown_connections 2022-07-20T03:08:00.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:00.871+0000 7fc4dd132700 1 -- 172.21.15.61:0/3412152019 wait complete. 2022-07-20T03:08:00.918 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:08:00.919 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:08:01.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 Processor -- start 2022-07-20T03:08:01.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 -- start start 2022-07-20T03:08:01.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48d4142ac0 0x7f48d4142ea0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f48d40accf0 0x7f48d40b03b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40b08a0 0x7f48d4145ee0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d4143430 con 0x7f48d4142ac0 2022-07-20T03:08:01.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f48d40b0d10 con 0x7f48d40accf0 2022-07-20T03:08:01.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d915f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d40aef20 con 0x7f48d40b08a0 2022-07-20T03:08:01.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48d4142ac0 0x7f48d4142ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f48d4142ac0 0x7f48d4142ea0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43650/0 (socket says 172.21.15.61:43650) 2022-07-20T03:08:01.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d3fff700 1 -- 172.21.15.61:0/856070887 learned_addr learned my addr 172.21.15.61:0/856070887 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:01.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48cbfff700 1 --2- 172.21.15.61:0/856070887 >> v2:172.21.15.117:3300/0 conn(0x7f48d40accf0 0x7f48d40b03b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.051+0000 7f48d895e700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40b08a0 0x7f48d4145ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d3fff700 1 -- 172.21.15.61:0/856070887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40b08a0 msgr2=0x7f48d4145ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:01.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d3fff700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40b08a0 0x7f48d4145ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d3fff700 1 -- 172.21.15.61:0/856070887 >> v2:172.21.15.117:3300/0 conn(0x7f48d40accf0 msgr2=0x7f48d40b03b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d3fff700 1 --2- 172.21.15.61:0/856070887 >> v2:172.21.15.117:3300/0 conn(0x7f48d40accf0 0x7f48d40b03b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d3fff700 1 -- 172.21.15.61:0/856070887 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48d4143660 con 0x7f48d4142ac0 2022-07-20T03:08:01.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d895e700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40b08a0 0x7f48d4145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:08:01.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d3fff700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d4142ac0 0x7f48d4142ea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f48bc002f70 tx=0x7f48bc006eb0).ready entity=mon.0 client_cookie=4df6c5e5d09b3d0e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:01.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d37fe700 1 -- 172.21.15.61:0/856070887 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48bc00d570 con 0x7f48d4142ac0 2022-07-20T03:08:01.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d37fe700 1 -- 172.21.15.61:0/856070887 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48bc000e90 con 0x7f48d4142ac0 2022-07-20T03:08:01.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d37fe700 1 -- 172.21.15.61:0/856070887 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48bc00d900 con 0x7f48d4142ac0 2022-07-20T03:08:01.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 -- 172.21.15.61:0/856070887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d4142ac0 msgr2=0x7f48d4142ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d4142ac0 0x7f48d4142ea0 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f48bc002f70 tx=0x7f48bc006eb0).stop 2022-07-20T03:08:01.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 -- 172.21.15.61:0/856070887 shutdown_connections 2022-07-20T03:08:01.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40b08a0 0x7f48d4145ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 --2- 172.21.15.61:0/856070887 >> v2:172.21.15.117:3300/0 conn(0x7f48d40accf0 0x7f48d40b03b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 --2- 172.21.15.61:0/856070887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d4142ac0 0x7f48d4142ea0 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 -- 172.21.15.61:0/856070887 >> 172.21.15.61:0/856070887 conn(0x7f48d40b93c0 msgr2=0x7f48d40b97d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 -- 172.21.15.61:0/856070887 shutdown_connections 2022-07-20T03:08:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.055+0000 7f48d915f700 1 -- 172.21.15.61:0/856070887 wait complete. 2022-07-20T03:08:01.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 Processor -- start 2022-07-20T03:08:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 -- start start 2022-07-20T03:08:01.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40accf0 0x7f48d4141830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 0x7f48d413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48d4142ac0 0x7f48d413e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d413d530 con 0x7f48d40b08a0 2022-07-20T03:08:01.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48d413d7d0 con 0x7f48d4142ac0 2022-07-20T03:08:01.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48d413f730 con 0x7f48d40accf0 2022-07-20T03:08:01.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 0x7f48d413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 0x7f48d413fc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43658/0 (socket says 172.21.15.61:43658) 2022-07-20T03:08:01.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 -- 172.21.15.61:0/2323071228 learned_addr learned my addr 172.21.15.61:0/2323071228 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:01.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d3fff700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40accf0 0x7f48d4141830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d895e700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48d4142ac0 0x7f48d413e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 -- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40accf0 msgr2=0x7f48d4141830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40accf0 0x7f48d4141830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 -- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48d4142ac0 msgr2=0x7f48d413e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48d4142ac0 0x7f48d413e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 -- 172.21.15.61:0/2323071228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48bc00b1e0 con 0x7f48d40b08a0 2022-07-20T03:08:01.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48cbfff700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 0x7f48d413fc30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f48c400a000 tx=0x7f48c4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:01.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48c400dea0 con 0x7f48d40b08a0 2022-07-20T03:08:01.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f48c4011780 con 0x7f48d40b08a0 2022-07-20T03:08:01.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f48c4011ae0 con 0x7f48d40b08a0 2022-07-20T03:08:01.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48d413b6d0 con 0x7f48d40b08a0 2022-07-20T03:08:01.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.059+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48d4140340 con 0x7f48d40b08a0 2022-07-20T03:08:01.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.063+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48d413f950 con 0x7f48d40b08a0 2022-07-20T03:08:01.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.063+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f48c4011c40 con 0x7f48d40b08a0 2022-07-20T03:08:01.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.063+0000 7f48d1ffb700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48ac053a40 0x7f48ac055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.063+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f48c406bde0 con 0x7f48d40b08a0 2022-07-20T03:08:01.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.063+0000 7f48d3fff700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48ac053a40 0x7f48ac055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.063+0000 7f48d3fff700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48ac053a40 0x7f48ac055f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f48bc01c000 tx=0x7f48bc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:01.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.067+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f48c4043010 con 0x7f48d40b08a0 2022-07-20T03:08:01.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f48d40b0f40 con 0x7f48d40b08a0 2022-07-20T03:08:01.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d1ffb700 1 -- 172.21.15.61:0/2323071228 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v29) v1 ==== 110+0+10 (secure 0 0 0) 0x7f48c4046630 con 0x7f48d40b08a0 2022-07-20T03:08:01.255 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:08:01.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48ac053a40 msgr2=0x7f48ac055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48ac053a40 0x7f48ac055f00 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f48bc01c000 tx=0x7f48bc015000).stop 2022-07-20T03:08:01.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 msgr2=0x7f48d413fc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 0x7f48d413fc30 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f48c400a000 tx=0x7f48c4008040).stop 2022-07-20T03:08:01.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 shutdown_connections 2022-07-20T03:08:01.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48ac053a40 0x7f48ac055f00 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f48d4142ac0 0x7f48d413e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f48d40b08a0 0x7f48d413fc30 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 --2- 172.21.15.61:0/2323071228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f48d40accf0 0x7f48d4141830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 >> 172.21.15.61:0/2323071228 conn(0x7f48d40b93c0 msgr2=0x7f48d413beb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:01.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 shutdown_connections 2022-07-20T03:08:01.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.195+0000 7f48d915f700 1 -- 172.21.15.61:0/2323071228 wait complete. 2022-07-20T03:08:01.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:08:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d680c700 1 Processor -- start 2022-07-20T03:08:01.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d680c700 1 -- start start 2022-07-20T03:08:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d680c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f73d00b59d0 0x7f73d005d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d680c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d00b5db0 con 0x7f73d0128d70 2022-07-20T03:08:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d680c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d00ad800 con 0x7f73d00af180 2022-07-20T03:08:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d680c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73d01291b0 con 0x7f73d00b59d0 2022-07-20T03:08:01.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d600b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f73d0128d70 0x7f73d013fc00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40264/0 (socket says 172.21.15.61:40264) 2022-07-20T03:08:01.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d600b700 1 -- 172.21.15.61:0/3110738003 learned_addr learned my addr 172.21.15.61:0/3110738003 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:01.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d580a700 1 --2- 172.21.15.61:0/3110738003 >> v2:172.21.15.117:3300/0 conn(0x7f73d00b59d0 0x7f73d005d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2343391461 0 0) 0x7f73d00b5db0 con 0x7f73d0128d70 2022-07-20T03:08:01.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b8003260 con 0x7f73d0128d70 2022-07-20T03:08:01.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1952827363 0 0) 0x7f73d00ad800 con 0x7f73d00af180 2022-07-20T03:08:01.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73d00b5db0 con 0x7f73d00af180 2022-07-20T03:08:01.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d580a700 1 -- 172.21.15.61:0/3110738003 >> v1:172.21.15.117:6789/0 conn(0x7f73d00af180 legacy=0x7f73d0128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 401310876 0 0) 0x7f73b8003260 con 0x7f73d0128d70 2022-07-20T03:08:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d580a700 1 -- 172.21.15.61:0/3110738003 >> v1:172.21.15.61:6789/0 conn(0x7f73d0128d70 legacy=0x7f73d013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.431+0000 7f73d580a700 1 -- 172.21.15.61:0/3110738003 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00adaa0 con 0x7f73d00b59d0 2022-07-20T03:08:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d580a700 1 --2- 172.21.15.61:0/3110738003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d005d1c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f73c8001490 tx=0x7f73cc006df0).ready entity=mon.1 client_cookie=b839b93d4a7cce70 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c80019e0 con 0x7f73d00b59d0 2022-07-20T03:08:01.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73cc000e90 con 0x7f73d00b59d0 2022-07-20T03:08:01.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d4808700 1 -- 172.21.15.61:0/3110738003 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73cc012460 con 0x7f73d00b59d0 2022-07-20T03:08:01.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 -- 172.21.15.61:0/3110738003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 msgr2=0x7f73d005d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 --2- 172.21.15.61:0/3110738003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d005d1c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f73c8001490 tx=0x7f73cc006df0).stop 2022-07-20T03:08:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 -- 172.21.15.61:0/3110738003 shutdown_connections 2022-07-20T03:08:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 --2- 172.21.15.61:0/3110738003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d005d1c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 -- 172.21.15.61:0/3110738003 >> 172.21.15.61:0/3110738003 conn(0x7f73d00634c0 msgr2=0x7f73d00b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 -- 172.21.15.61:0/3110738003 shutdown_connections 2022-07-20T03:08:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 -- 172.21.15.61:0/3110738003 wait complete. 2022-07-20T03:08:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 Processor -- start 2022-07-20T03:08:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.435+0000 7f73d680c700 1 -- start start 2022-07-20T03:08:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d00af180 0x7f73d0127010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d0151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 0x7f73d0152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0124360 con 0x7f73d0128d70 2022-07-20T03:08:01.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0124600 con 0x7f73d00b59d0 2022-07-20T03:08:01.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d01249e0 con 0x7f73d00af180 2022-07-20T03:08:01.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 0x7f73d0152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 0x7f73d0152330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43674/0 (socket says 172.21.15.61:43674) 2022-07-20T03:08:01.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 -- 172.21.15.61:0/4244086241 learned_addr learned my addr 172.21.15.61:0/4244086241 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:01.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d580a700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d00af180 0x7f73d0127010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d5009700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d0151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 -- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d00af180 msgr2=0x7f73d0127010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d00af180 0x7f73d0127010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 -- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 msgr2=0x7f73d0151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d0151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 -- 172.21.15.61:0/4244086241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73cc00b180 con 0x7f73d0128d70 2022-07-20T03:08:01.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d600b700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 0x7f73d0152330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f73c8001bc0 tx=0x7f73c8001a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:01.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c8004ab0 con 0x7f73d0128d70 2022-07-20T03:08:01.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c8004c10 con 0x7f73d0128d70 2022-07-20T03:08:01.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d0123250 con 0x7f73d0128d70 2022-07-20T03:08:01.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c8007170 con 0x7f73d0128d70 2022-07-20T03:08:01.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.439+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d01237c0 con 0x7f73d0128d70 2022-07-20T03:08:01.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.443+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d0124ba0 con 0x7f73d0128d70 2022-07-20T03:08:01.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.443+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f73c801b070 con 0x7f73d0128d70 2022-07-20T03:08:01.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.443+0000 7f73beffd700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b0053a40 0x7f73b0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:01.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.443+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f73c806b420 con 0x7f73d0128d70 2022-07-20T03:08:01.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.443+0000 7f73d580a700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b0053a40 0x7f73b0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:01.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.443+0000 7f73d580a700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b0053a40 0x7f73b0055f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f73cc01a000 tx=0x7f73cc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:01.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.451+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73c8042610 con 0x7f73d0128d70 2022-07-20T03:08:01.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.575+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f73d00ad4f0 con 0x7f73d0128d70 2022-07-20T03:08:01.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73beffd700 1 -- 172.21.15.61:0/4244086241 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v29) v1 ==== 92+0+10 (secure 0 0 0) 0x7f73c8045c30 con 0x7f73d0128d70 2022-07-20T03:08:01.794 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:08:01.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b0053a40 msgr2=0x7f73b0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b0053a40 0x7f73b0055f00 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f73cc01a000 tx=0x7f73cc014000).stop 2022-07-20T03:08:01.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 msgr2=0x7f73d0152330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:01.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 0x7f73d0152330 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f73c8001bc0 tx=0x7f73c8001a40).stop 2022-07-20T03:08:01.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 shutdown_connections 2022-07-20T03:08:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b0053a40 0x7f73b0055f00 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b59d0 0x7f73d0151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d0128d70 0x7f73d0152330 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 --2- 172.21.15.61:0/4244086241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d00af180 0x7f73d0127010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:01.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 >> 172.21.15.61:0/4244086241 conn(0x7f73d00634c0 msgr2=0x7f73d00b00e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:01.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 shutdown_connections 2022-07-20T03:08:01.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.579+0000 7f73d680c700 1 -- 172.21.15.61:0/4244086241 wait complete. 2022-07-20T03:08:01.836 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:08:02.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867d84a700 1 Processor -- start 2022-07-20T03:08:02.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867d84a700 1 -- start start 2022-07-20T03:08:02.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867d84a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b1200 0x7f86780b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867d84a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86780aea30 con 0x7f86780b5650 2022-07-20T03:08:02.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867d84a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86780aec60 con 0x7f867805bd90 2022-07-20T03:08:02.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867d84a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8678142970 con 0x7f86780b1200 2022-07-20T03:08:02.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867c848700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f86780b5650 0x7f867805c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40278/0 (socket says 172.21.15.61:40278) 2022-07-20T03:08:02.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f867c848700 1 -- 172.21.15.61:0/340705622 learned_addr learned my addr 172.21.15.61:0/340705622 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:02.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.959+0000 7f8677fff700 1 --2- 172.21.15.61:0/340705622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b1200 0x7f86780b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3514828148 0 0) 0x7f86780aea30 con 0x7f86780b5650 2022-07-20T03:08:02.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f865c003260 con 0x7f86780b5650 2022-07-20T03:08:02.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2535186466 0 0) 0x7f86780aec60 con 0x7f867805bd90 2022-07-20T03:08:02.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f86780aea30 con 0x7f867805bd90 2022-07-20T03:08:02.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f8677fff700 1 -- 172.21.15.61:0/340705622 >> v1:172.21.15.117:6789/0 conn(0x7f867805bd90 legacy=0x7f8678142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f8677fff700 1 -- 172.21.15.61:0/340705622 >> v1:172.21.15.61:6789/0 conn(0x7f86780b5650 legacy=0x7f867805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f8677fff700 1 -- 172.21.15.61:0/340705622 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86780af180 con 0x7f86780b1200 2022-07-20T03:08:02.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f8677fff700 1 --2- 172.21.15.61:0/340705622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b1200 0x7f86780b5a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8664002f70 tx=0x7f8664006d90).ready entity=mon.2 client_cookie=e4685d94ee11c6d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:02.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f866400d4a0 con 0x7f86780b1200 2022-07-20T03:08:02.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664000e90 con 0x7f86780b1200 2022-07-20T03:08:02.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f86777fe700 1 -- 172.21.15.61:0/340705622 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f866400d7f0 con 0x7f86780b1200 2022-07-20T03:08:02.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 -- 172.21.15.61:0/340705622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b1200 msgr2=0x7f86780b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 --2- 172.21.15.61:0/340705622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b1200 0x7f86780b5a30 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f8664002f70 tx=0x7f8664006d90).stop 2022-07-20T03:08:02.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 -- 172.21.15.61:0/340705622 shutdown_connections 2022-07-20T03:08:02.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 --2- 172.21.15.61:0/340705622 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b1200 0x7f86780b5a30 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 -- 172.21.15.61:0/340705622 >> 172.21.15.61:0/340705622 conn(0x7f86780634e0 msgr2=0x7f86780b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:02.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 -- 172.21.15.61:0/340705622 shutdown_connections 2022-07-20T03:08:02.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 -- 172.21.15.61:0/340705622 wait complete. 2022-07-20T03:08:02.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 Processor -- start 2022-07-20T03:08:02.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.963+0000 7f867d84a700 1 -- start start 2022-07-20T03:08:02.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867805bd90 0x7f8678149f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 0x7f8678151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b5650 0x7f8678140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8678152290 con 0x7f86780b1200 2022-07-20T03:08:02.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8678141f10 con 0x7f867805bd90 2022-07-20T03:08:02.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86781405e0 con 0x7f86780b5650 2022-07-20T03:08:02.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 0x7f8678151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 0x7f8678151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43686/0 (socket says 172.21.15.61:43686) 2022-07-20T03:08:02.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 -- 172.21.15.61:0/4237652382 learned_addr learned my addr 172.21.15.61:0/4237652382 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:02.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867c848700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867805bd90 0x7f8678149f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d049700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b5650 0x7f8678140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 -- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b5650 msgr2=0x7f8678140a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b5650 0x7f8678140a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 -- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867805bd90 msgr2=0x7f8678149f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867805bd90 0x7f8678149f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 -- 172.21.15.61:0/4237652382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f866400b120 con 0x7f86780b1200 2022-07-20T03:08:02.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d049700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b5650 0x7f8678140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:02.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8677fff700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 0x7f8678151da0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8664002f70 tx=0x7f8664014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:02.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8664011b90 con 0x7f86780b1200 2022-07-20T03:08:02.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8664011cf0 con 0x7f86780b1200 2022-07-20T03:08:02.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8664006220 con 0x7f86780b1200 2022-07-20T03:08:02.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f867813edc0 con 0x7f86780b1200 2022-07-20T03:08:02.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f867813cf50 con 0x7f86780b1200 2022-07-20T03:08:02.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86780574e0 con 0x7f86780b1200 2022-07-20T03:08:02.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.967+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8664023030 con 0x7f86780b1200 2022-07-20T03:08:02.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.971+0000 7f8675ffb700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8658053ad0 0x7f8658055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.971+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f866406c4a0 con 0x7f86780b1200 2022-07-20T03:08:02.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.971+0000 7f867c848700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8658053ad0 0x7f8658055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.971+0000 7f867c848700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8658053ad0 0x7f8658055f90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f866c00d000 tx=0x7f866c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:02.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:01.975+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8664043730 con 0x7f86780b1200 2022-07-20T03:08:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f8678067ea0 con 0x7f86780b1200 2022-07-20T03:08:02.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f8675ffb700 1 -- 172.21.15.61:0/4237652382 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v29) v1 ==== 84+0+143 (secure 0 0 0) 0x7f8664021090 con 0x7f86780b1200 2022-07-20T03:08:02.246 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:02.247 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:08:02.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:08:02.248 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:08:02.248 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:08:02.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:08:02.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:08:02.315 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:08:02.322 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:08:02.323 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:08:02.323 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:08:02.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8658053ad0 msgr2=0x7f8658055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8658053ad0 0x7f8658055f90 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f866c00d000 tx=0x7f866c009040).stop 2022-07-20T03:08:02.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 msgr2=0x7f8678151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 0x7f8678151da0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f8664002f70 tx=0x7f8664014000).stop 2022-07-20T03:08:02.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 shutdown_connections 2022-07-20T03:08:02.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8658053ad0 0x7f8658055f90 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86780b5650 0x7f8678140a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86780b1200 0x7f8678151da0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 --2- 172.21.15.61:0/4237652382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f867805bd90 0x7f8678149f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 >> 172.21.15.61:0/4237652382 conn(0x7f86780634e0 msgr2=0x7f86780b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 shutdown_connections 2022-07-20T03:08:02.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.107+0000 7f867d84a700 1 -- 172.21.15.61:0/4237652382 wait complete. 2022-07-20T03:08:02.410 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:08:02.412 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:08:02.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f85a5b44700 1 Processor -- start 2022-07-20T03:08:02.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f85a5b44700 1 -- start start 2022-07-20T03:08:02.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f85a5b44700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f85a0145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f85a5b44700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a00607e0 con 0x7f85a0060ee0 2022-07-20T03:08:02.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f85a5b44700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85a0060a80 con 0x7f85a00b4d60 2022-07-20T03:08:02.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f85a5b44700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0060d20 con 0x7f85a009df40 2022-07-20T03:08:02.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f85a0060ee0 0x7f85a009cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40292/0 (socket says 172.21.15.61:40292) 2022-07-20T03:08:02.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 -- 172.21.15.61:0/2733413179 learned_addr learned my addr 172.21.15.61:0/2733413179 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:02.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859f7fe700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f85a0145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1134934941 0 0) 0x7f85a00607e0 con 0x7f85a0060ee0 2022-07-20T03:08:02.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8580003260 con 0x7f85a0060ee0 2022-07-20T03:08:02.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3610642909 0 0) 0x7f85a0060a80 con 0x7f85a00b4d60 2022-07-20T03:08:02.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a00607e0 con 0x7f85a00b4d60 2022-07-20T03:08:02.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 781731976 0 0) 0x7f8580003260 con 0x7f85a0060ee0 2022-07-20T03:08:02.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 msgr2=0x7f85a0145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:02.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f85a0145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 >> v1:172.21.15.117:6789/0 conn(0x7f85a00b4d60 legacy=0x7f85a0145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a009e320 con 0x7f85a0060ee0 2022-07-20T03:08:02.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f85980019e0 con 0x7f85a0060ee0 2022-07-20T03:08:02.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 >> v1:172.21.15.61:6789/0 conn(0x7f85a0060ee0 legacy=0x7f85a009cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f8580003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8580003e70 0x7f8580004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85800049f0 0x7f8580004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0060d20 con 0x7f85800049f0 2022-07-20T03:08:02.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f858000b830 con 0x7f8580003e70 2022-07-20T03:08:02.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f858000b9f0 con 0x7f85a009df40 2022-07-20T03:08:02.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85800049f0 0x7f8580004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859f7fe700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f8580003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859effd700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8580003e70 0x7f8580004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 -- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 msgr2=0x7f8580003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:02.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f8580003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 -- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8580003e70 msgr2=0x7f8580004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:02.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8580003e70 0x7f8580004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 -- 172.21.15.61:0/2733413179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8580003260 con 0x7f85800049f0 2022-07-20T03:08:02.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859f7fe700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85a009df40 0x7f8580003910 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:02.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859ffff700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85800049f0 0x7f8580004e70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f85980017b0 tx=0x7f85a009ca60).ready entity=mon.0 client_cookie=40f42111b37eccc8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:02.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f859800caf0 con 0x7f85800049f0 2022-07-20T03:08:02.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f859800cc50 con 0x7f85800049f0 2022-07-20T03:08:02.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.571+0000 7f859e7fc700 1 -- 172.21.15.61:0/2733413179 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8598005990 con 0x7f85800049f0 2022-07-20T03:08:02.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 -- 172.21.15.61:0/2733413179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85800049f0 msgr2=0x7f8580004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2733413179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85800049f0 0x7f8580004e70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f85980017b0 tx=0x7f85a009ca60).stop 2022-07-20T03:08:02.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f859ffff700 1 -- 172.21.15.61:0/2733413179 reap_dead start 2022-07-20T03:08:02.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 -- 172.21.15.61:0/2733413179 shutdown_connections 2022-07-20T03:08:02.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 -- 172.21.15.61:0/2733413179 >> 172.21.15.61:0/2733413179 conn(0x7f85a00b5f30 msgr2=0x7f85a00b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:02.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 -- 172.21.15.61:0/2733413179 shutdown_connections 2022-07-20T03:08:02.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 -- 172.21.15.61:0/2733413179 wait complete. 2022-07-20T03:08:02.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 Processor -- start 2022-07-20T03:08:02.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.575+0000 7f85a5b44700 1 -- start start 2022-07-20T03:08:02.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85800049f0 0x7f85a008fd90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85a0091cc0 0x7f85a00b2400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a0090c20 0x7f85a00b28f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a00926f0 con 0x7f85a0090c20 2022-07-20T03:08:02.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0093440 con 0x7f85a0091cc0 2022-07-20T03:08:02.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85a0093670 con 0x7f85800049f0 2022-07-20T03:08:02.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a0090c20 0x7f85a00b28f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859effd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85a0091cc0 0x7f85a00b2400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85800049f0 0x7f85a008fd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859effd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85a0091cc0 0x7f85a00b2400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:53386/0 (socket says 172.21.15.61:53386) 2022-07-20T03:08:02.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859effd700 1 -- 172.21.15.61:0/2182510683 learned_addr learned my addr 172.21.15.61:0/2182510683 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:02.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 -- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85800049f0 msgr2=0x7f85a008fd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85800049f0 0x7f85a008fd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 -- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85a0091cc0 msgr2=0x7f85a00b2400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:02.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85a0091cc0 0x7f85a00b2400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:02.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 -- 172.21.15.61:0/2182510683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85a0093910 con 0x7f85a0090c20 2022-07-20T03:08:02.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859f7fe700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85800049f0 0x7f85a008fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:02.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859ffff700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a0090c20 0x7f85a00b28f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8598001bc0 tx=0x7f8598015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:02.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8598004ca0 con 0x7f85a0090c20 2022-07-20T03:08:02.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8598004e00 con 0x7f85a0090c20 2022-07-20T03:08:02.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85a0093bb0 con 0x7f85a0090c20 2022-07-20T03:08:02.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85980072e0 con 0x7f85a0090c20 2022-07-20T03:08:02.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.579+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85a0094050 con 0x7f85a0090c20 2022-07-20T03:08:02.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.583+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85a0093d40 con 0x7f85a0090c20 2022-07-20T03:08:02.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.583+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8598010070 con 0x7f85a0090c20 2022-07-20T03:08:02.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.583+0000 7f859cff9700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8584053a60 0x7f8584055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:02.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.583+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(29..29 src has 1..29) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8598019020 con 0x7f85a0090c20 2022-07-20T03:08:02.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.583+0000 7f859f7fe700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8584053a60 0x7f8584055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:02.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.583+0000 7f859f7fe700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8584053a60 0x7f8584055f20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8590006000 tx=0x7f8590008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:02.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.591+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8598043dc0 con 0x7f85a0090c20 2022-07-20T03:08:02.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:02.719+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f85a00574e0 con 0x7f85a0090c20 2022-07-20T03:08:03.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.071+0000 7f859cff9700 1 -- 172.21.15.61:0/2182510683 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v30) v1 ==== 133+0+0 (secure 0 0 0) 0x7f85980473e0 con 0x7f85a0090c20 2022-07-20T03:08:03.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8584053a60 msgr2=0x7f8584055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:03.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8584053a60 0x7f8584055f20 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8590006000 tx=0x7f8590008040).stop 2022-07-20T03:08:03.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a0090c20 msgr2=0x7f85a00b28f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:03.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a0090c20 0x7f85a00b28f0 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f8598001bc0 tx=0x7f8598015000).stop 2022-07-20T03:08:03.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 shutdown_connections 2022-07-20T03:08:03.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8584053a60 0x7f8584055f20 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85a0091cc0 0x7f85a00b2400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85a0090c20 0x7f85a00b28f0 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 --2- 172.21.15.61:0/2182510683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85800049f0 0x7f85a008fd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 >> 172.21.15.61:0/2182510683 conn(0x7f85a00b5f30 msgr2=0x7f85a0143ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:03.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 shutdown_connections 2022-07-20T03:08:03.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.075+0000 7f85a5b44700 1 -- 172.21.15.61:0/2182510683 wait complete. 2022-07-20T03:08:03.098 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:08:03.133 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:08:03.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 Processor -- start 2022-07-20T03:08:03.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 -- start start 2022-07-20T03:08:03.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c00aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:03.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47c00b0b90 0x7f47c005fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47c00aced0 con 0x7f47c00ac9f0 2022-07-20T03:08:03.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f47c0143c80 con 0x7f47c00b0b90 2022-07-20T03:08:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47c48e3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c0143f20 con 0x7f47c00ac610 2022-07-20T03:08:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47be59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47c00b0b90 0x7f47c005fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:03.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47be59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f47c00b0b90 0x7f47c005fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43714/0 (socket says 172.21.15.61:43714) 2022-07-20T03:08:03.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47be59c700 1 -- 172.21.15.61:0/1784088444 learned_addr learned my addr 172.21.15.61:0/1784088444 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:03.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.271+0000 7f47bed9d700 1 --2- 172.21.15.61:0/1784088444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c00aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:03.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47be59c700 1 -- 172.21.15.61:0/1784088444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 msgr2=0x7f47c00aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:03.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47be59c700 1 --2- 172.21.15.61:0/1784088444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c00aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47be59c700 1 -- 172.21.15.61:0/1784088444 >> v1:172.21.15.117:6789/0 conn(0x7f47c00ac9f0 legacy=0x7f47c0140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:03.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47be59c700 1 -- 172.21.15.61:0/1784088444 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47c005ffc0 con 0x7f47c00b0b90 2022-07-20T03:08:03.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47be59c700 1 --2- 172.21.15.61:0/1784088444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c005fad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f47b4002f70 tx=0x7f47b4006eb0).ready entity=mon.0 client_cookie=eb4870bc94a4cf77 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:03.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47bdd9b700 1 -- 172.21.15.61:0/1784088444 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47b400d570 con 0x7f47c00b0b90 2022-07-20T03:08:03.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47bdd9b700 1 -- 172.21.15.61:0/1784088444 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b4000e90 con 0x7f47c00b0b90 2022-07-20T03:08:03.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47bdd9b700 1 -- 172.21.15.61:0/1784088444 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47b400d900 con 0x7f47c00b0b90 2022-07-20T03:08:03.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 -- 172.21.15.61:0/1784088444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 msgr2=0x7f47c005fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:03.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 --2- 172.21.15.61:0/1784088444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c005fad0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f47b4002f70 tx=0x7f47b4006eb0).stop 2022-07-20T03:08:03.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 -- 172.21.15.61:0/1784088444 shutdown_connections 2022-07-20T03:08:03.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 --2- 172.21.15.61:0/1784088444 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c005fad0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 --2- 172.21.15.61:0/1784088444 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c00aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 -- 172.21.15.61:0/1784088444 >> 172.21.15.61:0/1784088444 conn(0x7f47c00b61f0 msgr2=0x7f47c00b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:03.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 -- 172.21.15.61:0/1784088444 shutdown_connections 2022-07-20T03:08:03.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.275+0000 7f47c48e3700 1 -- 172.21.15.61:0/1784088444 wait complete. 2022-07-20T03:08:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 Processor -- start 2022-07-20T03:08:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 -- start start 2022-07-20T03:08:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c0152830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:03.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47c00ac9f0 0x7f47c0142140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c0142630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c0153440 con 0x7f47c00b0b90 2022-07-20T03:08:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47c0066360 con 0x7f47c00ac9f0 2022-07-20T03:08:03.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47c0153120 con 0x7f47c00ac610 2022-07-20T03:08:03.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c0142630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:03.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47be59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47c00ac9f0 0x7f47c0142140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:03.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bed9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c0152830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:03.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c0142630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43722/0 (socket says 172.21.15.61:43722) 2022-07-20T03:08:03.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 -- 172.21.15.61:0/518396005 learned_addr learned my addr 172.21.15.61:0/518396005 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:03.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 -- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 msgr2=0x7f47c0152830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:03.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c0152830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 -- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47c00ac9f0 msgr2=0x7f47c0142140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:03.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47c00ac9f0 0x7f47c0142140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:03.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 -- 172.21.15.61:0/518396005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47b400b1e0 con 0x7f47c00b0b90 2022-07-20T03:08:03.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47bf59e700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c0142630 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f47c01532b0 tx=0x7f47b8007f10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:03.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47b800f470 con 0x7f47c00b0b90 2022-07-20T03:08:03.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47b800f5d0 con 0x7f47c00b0b90 2022-07-20T03:08:03.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f47c0152e80 con 0x7f47c00b0b90 2022-07-20T03:08:03.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47b800f930 con 0x7f47c00b0b90 2022-07-20T03:08:03.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.279+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f47c0065020 con 0x7f47c00b0b90 2022-07-20T03:08:03.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.283+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47c00574e0 con 0x7f47c00b0b90 2022-07-20T03:08:03.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.283+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f47b8019030 con 0x7f47c00b0b90 2022-07-20T03:08:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.283+0000 7f47affff700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f479c0539b0 0x7f479c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.283+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(30..30 src has 1..30) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f47b806b540 con 0x7f47c00b0b90 2022-07-20T03:08:03.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.283+0000 7f47bed9d700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f479c0539b0 0x7f479c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.283+0000 7f47bed9d700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f479c0539b0 0x7f479c055e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f47b000b000 tx=0x7f47b0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:03.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.291+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f47b8042670 con 0x7f47c00b0b90 2022-07-20T03:08:03.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:03.415+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f47c00447e0 con 0x7f47c00b0b90 2022-07-20T03:08:04.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.075+0000 7f47affff700 1 -- 172.21.15.61:0/518396005 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v31) v1 ==== 125+0+0 (secure 0 0 0) 0x7f47b8045c90 con 0x7f47c00b0b90 2022-07-20T03:08:04.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.083+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f479c0539b0 msgr2=0x7f479c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.083+0000 7f47c48e3700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f479c0539b0 0x7f479c055e70 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f47b000b000 tx=0x7f47b0006df0).stop 2022-07-20T03:08:04.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.083+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 msgr2=0x7f47c0142630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.083+0000 7f47c48e3700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c0142630 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f47c01532b0 tx=0x7f47b8007f10).stop 2022-07-20T03:08:04.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 shutdown_connections 2022-07-20T03:08:04.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f479c0539b0 0x7f479c055e70 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47c00ac9f0 0x7f47c0142140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47c00b0b90 0x7f47c0142630 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 --2- 172.21.15.61:0/518396005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47c00ac610 0x7f47c0152830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 >> 172.21.15.61:0/518396005 conn(0x7f47c00b61f0 msgr2=0x7f47c00b0fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:04.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 shutdown_connections 2022-07-20T03:08:04.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.087+0000 7f47c48e3700 1 -- 172.21.15.61:0/518396005 wait complete. 2022-07-20T03:08:04.142 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:08:04.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:08:04.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 Processor -- start 2022-07-20T03:08:04.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 -- start start 2022-07-20T03:08:04.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a980b6690 0x7f7a980b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7a980ae670 0x7f7a980aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a980acae0 con 0x7f7a980b59a0 2022-07-20T03:08:04.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a98147100 con 0x7f7a980b6690 2022-07-20T03:08:04.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a9de92700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a980acd10 con 0x7f7a980ae670 2022-07-20T03:08:04.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.291+0000 7f7a977fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a980b6690 0x7f7a980b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a980b6690 0x7f7a980b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43726/0 (socket says 172.21.15.61:43726) 2022-07-20T03:08:04.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 -- 172.21.15.61:0/2680899368 learned_addr learned my addr 172.21.15.61:0/2680899368 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:04.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a97fff700 1 --2- 172.21.15.61:0/2680899368 >> v2:172.21.15.117:3300/0 conn(0x7f7a980ae670 0x7f7a980aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 -- 172.21.15.61:0/2680899368 >> v2:172.21.15.117:3300/0 conn(0x7f7a980ae670 msgr2=0x7f7a980aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:04.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 --2- 172.21.15.61:0/2680899368 >> v2:172.21.15.117:3300/0 conn(0x7f7a980ae670 0x7f7a980aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 -- 172.21.15.61:0/2680899368 >> v1:172.21.15.117:6789/0 conn(0x7f7a980b59a0 legacy=0x7f7a9813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 -- 172.21.15.61:0/2680899368 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a980b5df0 con 0x7f7a980b6690 2022-07-20T03:08:04.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a97fff700 1 --2- 172.21.15.61:0/2680899368 >> v2:172.21.15.117:3300/0 conn(0x7f7a980ae670 0x7f7a980aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:04.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a977fe700 1 --2- 172.21.15.61:0/2680899368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980b6690 0x7f7a980b6a70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7a80002f70 tx=0x7f7a80006eb0).ready entity=mon.0 client_cookie=39500e678883de81 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a967fc700 1 -- 172.21.15.61:0/2680899368 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a8000d610 con 0x7f7a980b6690 2022-07-20T03:08:04.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a967fc700 1 -- 172.21.15.61:0/2680899368 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a80000e90 con 0x7f7a980b6690 2022-07-20T03:08:04.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a967fc700 1 -- 172.21.15.61:0/2680899368 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a8000d9a0 con 0x7f7a980b6690 2022-07-20T03:08:04.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 -- 172.21.15.61:0/2680899368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980b6690 msgr2=0x7f7a980b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 --2- 172.21.15.61:0/2680899368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980b6690 0x7f7a980b6a70 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f7a80002f70 tx=0x7f7a80006eb0).stop 2022-07-20T03:08:04.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 -- 172.21.15.61:0/2680899368 shutdown_connections 2022-07-20T03:08:04.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 --2- 172.21.15.61:0/2680899368 >> v2:172.21.15.117:3300/0 conn(0x7f7a980ae670 0x7f7a980aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 --2- 172.21.15.61:0/2680899368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980b6690 0x7f7a980b6a70 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 -- 172.21.15.61:0/2680899368 >> 172.21.15.61:0/2680899368 conn(0x7f7a980b9f70 msgr2=0x7f7a980b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:04.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 -- 172.21.15.61:0/2680899368 shutdown_connections 2022-07-20T03:08:04.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 -- 172.21.15.61:0/2680899368 wait complete. 2022-07-20T03:08:04.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 Processor -- start 2022-07-20T03:08:04.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.295+0000 7f7a9de92700 1 -- start start 2022-07-20T03:08:04.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 0x7f7a981403d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a980b59a0 0x7f7a980b3170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a980b6690 0x7f7a98112820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a98112ec0 con 0x7f7a980ae670 2022-07-20T03:08:04.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a98113160 con 0x7f7a980b59a0 2022-07-20T03:08:04.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9813ee50 con 0x7f7a980b6690 2022-07-20T03:08:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 0x7f7a981403d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 0x7f7a981403d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43732/0 (socket says 172.21.15.61:43732) 2022-07-20T03:08:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 -- 172.21.15.61:0/898598229 learned_addr learned my addr 172.21.15.61:0/898598229 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a97fff700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a980b6690 0x7f7a98112820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a96ffd700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a980b59a0 0x7f7a980b3170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 -- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a980b6690 msgr2=0x7f7a98112820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a980b6690 0x7f7a98112820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 -- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a980b59a0 msgr2=0x7f7a980b3170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a980b59a0 0x7f7a980b3170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 -- 172.21.15.61:0/898598229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a8000b1e0 con 0x7f7a980ae670 2022-07-20T03:08:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a97fff700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a980b6690 0x7f7a98112820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a977fe700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 0x7f7a981403d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f7a80002f70 tx=0x7f7a80015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a800068b0 con 0x7f7a980ae670 2022-07-20T03:08:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a8000d970 con 0x7f7a980ae670 2022-07-20T03:08:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a80011ce0 con 0x7f7a980ae670 2022-07-20T03:08:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a98152500 con 0x7f7a980ae670 2022-07-20T03:08:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a98152920 con 0x7f7a980ae670 2022-07-20T03:08:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7a8001a040 con 0x7f7a980ae670 2022-07-20T03:08:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a98112d10 con 0x7f7a980ae670 2022-07-20T03:08:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a94ff9700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a7c053ad0 0x7f7a7c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.299+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(31..31 src has 1..31) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f7a8006cd40 con 0x7f7a980ae670 2022-07-20T03:08:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.303+0000 7f7a96ffd700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a7c053ad0 0x7f7a7c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.303+0000 7f7a96ffd700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a7c053ad0 0x7f7a7c055f90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f7a8800b000 tx=0x7f7a88009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.311+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7a80043c70 con 0x7f7a980ae670 2022-07-20T03:08:04.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.475+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f7a980acfb0 con 0x7f7a980ae670 2022-07-20T03:08:04.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.483+0000 7f7a94ff9700 1 -- 172.21.15.61:0/898598229 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 5 and data pool 6 v12) v1 ==== 187+0+0 (secure 0 0 0) 0x7f7a80023070 con 0x7f7a980ae670 2022-07-20T03:08:04.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a7c053ad0 msgr2=0x7f7a7c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a7c053ad0 0x7f7a7c055f90 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f7a8800b000 tx=0x7f7a88009040).stop 2022-07-20T03:08:04.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 msgr2=0x7f7a981403d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 0x7f7a981403d0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f7a80002f70 tx=0x7f7a80015000).stop 2022-07-20T03:08:04.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 shutdown_connections 2022-07-20T03:08:04.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a7c053ad0 0x7f7a7c055f90 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a980b59a0 0x7f7a980b3170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a980b6690 0x7f7a98112820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 --2- 172.21.15.61:0/898598229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a980ae670 0x7f7a981403d0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 >> 172.21.15.61:0/898598229 conn(0x7f7a980b9f70 msgr2=0x7f7a980b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:04.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 shutdown_connections 2022-07-20T03:08:04.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.491+0000 7f7a9de92700 1 -- 172.21.15.61:0/898598229 wait complete. 2022-07-20T03:08:04.516 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 5 and data pool 6 2022-07-20T03:08:04.517 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:08:04.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5c55f700 1 Processor -- start 2022-07-20T03:08:04.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5c55f700 1 -- start start 2022-07-20T03:08:04.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5c55f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 0x7fed540b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5c55f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed540624a0 con 0x7fed540618e0 2022-07-20T03:08:04.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5c55f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed5405f970 con 0x7fed540b21f0 2022-07-20T03:08:04.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5c55f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed54060b90 con 0x7fed540ab8b0 2022-07-20T03:08:04.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fed540618e0 0x7fed54061d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40342/0 (socket says 172.21.15.61:40342) 2022-07-20T03:08:04.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 -- 172.21.15.61:0/3123249230 learned_addr learned my addr 172.21.15.61:0/3123249230 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:04.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed59afa700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 0x7fed540b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389915428 0 0) 0x7fed540624a0 con 0x7fed540618e0 2022-07-20T03:08:04.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed40003260 con 0x7fed540618e0 2022-07-20T03:08:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1156670380 0 0) 0x7fed5405f970 con 0x7fed540b21f0 2022-07-20T03:08:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed540624a0 con 0x7fed540b21f0 2022-07-20T03:08:04.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3361404500 0 0) 0x7fed40003260 con 0x7fed540618e0 2022-07-20T03:08:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 msgr2=0x7fed540b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 0x7fed540b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 >> v1:172.21.15.117:6789/0 conn(0x7fed540b21f0 legacy=0x7fed54146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed5405fc10 con 0x7fed540618e0 2022-07-20T03:08:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fed44001660 con 0x7fed540618e0 2022-07-20T03:08:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 >> v1:172.21.15.61:6789/0 conn(0x7fed540618e0 legacy=0x7fed54061d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed40003910 0x7fed40003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 0x7fed40004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed400049f0 0x7fed40004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed54060b90 con 0x7fed400049f0 2022-07-20T03:08:04.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000b780 con 0x7fed40003910 2022-07-20T03:08:04.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed4000b940 con 0x7fed540ab8b0 2022-07-20T03:08:04.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5a2fb700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed40003910 0x7fed40003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed400049f0 0x7fed40004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed59afa700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 0x7fed40004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 -- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 msgr2=0x7fed40004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:04.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540ab8b0 0x7fed40004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 -- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed40003910 msgr2=0x7fed40003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:04.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed40003910 0x7fed40003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5aafc700 1 -- 172.21.15.61:0/3123249230 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed40003260 con 0x7fed400049f0 2022-07-20T03:08:04.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.607+0000 7fed5a2fb700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed40003910 0x7fed40003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:04.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5aafc700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed400049f0 0x7fed40004e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fed44001430 tx=0x7fed540614b0).ready entity=mon.0 client_cookie=5229ee6c35218a18 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed440070d0 con 0x7fed400049f0 2022-07-20T03:08:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed44007230 con 0x7fed400049f0 2022-07-20T03:08:04.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed592f9700 1 -- 172.21.15.61:0/3123249230 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed4400c8f0 con 0x7fed400049f0 2022-07-20T03:08:04.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5c55f700 1 -- 172.21.15.61:0/3123249230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed400049f0 msgr2=0x7fed40004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5c55f700 1 --2- 172.21.15.61:0/3123249230 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed400049f0 0x7fed40004e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fed44001430 tx=0x7fed540614b0).stop 2022-07-20T03:08:04.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5aafc700 1 -- 172.21.15.61:0/3123249230 reap_dead start 2022-07-20T03:08:04.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5c55f700 1 -- 172.21.15.61:0/3123249230 shutdown_connections 2022-07-20T03:08:04.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5c55f700 1 -- 172.21.15.61:0/3123249230 >> 172.21.15.61:0/3123249230 conn(0x7fed540b3690 msgr2=0x7fed540b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:04.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5c55f700 1 -- 172.21.15.61:0/3123249230 shutdown_connections 2022-07-20T03:08:04.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.611+0000 7fed5c55f700 1 -- 172.21.15.61:0/3123249230 wait complete. 2022-07-20T03:08:04.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 Processor -- start 2022-07-20T03:08:04.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 -- start start 2022-07-20T03:08:04.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 0x7fed540ad4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540af3b0 0x7fed54156c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed54157130 0x7fed54097e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed541575b0 con 0x7fed540b58b0 2022-07-20T03:08:04.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed540afb60 con 0x7fed54157130 2022-07-20T03:08:04.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed540ad990 con 0x7fed540af3b0 2022-07-20T03:08:04.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 0x7fed540ad4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 0x7fed540ad4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43752/0 (socket says 172.21.15.61:43752) 2022-07-20T03:08:04.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 -- 172.21.15.61:0/2616341168 learned_addr learned my addr 172.21.15.61:0/2616341168 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:04.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed59afa700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540af3b0 0x7fed54156c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5aafc700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed54157130 0x7fed54097e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 -- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540af3b0 msgr2=0x7fed54156c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540af3b0 0x7fed54156c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 -- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed54157130 msgr2=0x7fed54097e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed54157130 0x7fed54097e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 -- 172.21.15.61:0/2616341168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed540ae9b0 con 0x7fed540b58b0 2022-07-20T03:08:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5a2fb700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 0x7fed540ad4a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fed541576f0 tx=0x7fed48007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed480116b0 con 0x7fed540b58b0 2022-07-20T03:08:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed48011810 con 0x7fed540b58b0 2022-07-20T03:08:04.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed48011b70 con 0x7fed540b58b0 2022-07-20T03:08:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed540af8c0 con 0x7fed540b58b0 2022-07-20T03:08:04.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.615+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed540b5c90 con 0x7fed540b58b0 2022-07-20T03:08:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.619+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed5405feb0 con 0x7fed540b58b0 2022-07-20T03:08:04.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.619+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fed4801b030 con 0x7fed540b58b0 2022-07-20T03:08:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.619+0000 7fed537fe700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed34053ad0 0x7fed34055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.619+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fed4806bcf0 con 0x7fed540b58b0 2022-07-20T03:08:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.619+0000 7fed59afa700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed34053ad0 0x7fed34055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.619+0000 7fed59afa700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed34053ad0 0x7fed34055f90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fed4c00b000 tx=0x7fed4c0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.627+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed48042bb0 con 0x7fed540b58b0 2022-07-20T03:08:04.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.755+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fed540574e0 con 0x7fed540b58b0 2022-07-20T03:08:04.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed537fe700 1 -- 172.21.15.61:0/2616341168 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16099 (secure 0 0 0) 0x7fed480461d0 con 0x7fed540b58b0 2022-07-20T03:08:04.764 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:04.766 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":32,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:08:04.482072+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:08:02.776339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T03:08:03.472472+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:08:04.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed34053ad0 msgr2=0x7fed34055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed34053ad0 0x7fed34055f90 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7fed4c00b000 tx=0x7fed4c0091d0).stop 2022-07-20T03:08:04.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 msgr2=0x7fed540ad4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 0x7fed540ad4a0 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fed541576f0 tx=0x7fed48007a00).stop 2022-07-20T03:08:04.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 shutdown_connections 2022-07-20T03:08:04.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed34053ad0 0x7fed34055f90 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed54157130 0x7fed54097e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed540af3b0 0x7fed54156c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 --2- 172.21.15.61:0/2616341168 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed540b58b0 0x7fed540ad4a0 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 >> 172.21.15.61:0/2616341168 conn(0x7fed540b3690 msgr2=0x7fed540ac2c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:04.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 shutdown_connections 2022-07-20T03:08:04.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.759+0000 7fed5c55f700 1 -- 172.21.15.61:0/2616341168 wait complete. 2022-07-20T03:08:04.785 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:08:04.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 Processor -- start 2022-07-20T03:08:04.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 -- start start 2022-07-20T03:08:04.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f404005c3f0 0x7f404014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f40400acef0 0x7f4040146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f404005d480 con 0x7f40400b1e00 2022-07-20T03:08:04.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40400b21e0 con 0x7f404005c3f0 2022-07-20T03:08:04.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f404481f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f404005cf40 con 0x7f40400acef0 2022-07-20T03:08:04.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f404005c3f0 0x7f404014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f404005c3f0 0x7f404014a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43760/0 (socket says 172.21.15.61:43760) 2022-07-20T03:08:04.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 -- 172.21.15.61:0/2342756770 learned_addr learned my addr 172.21.15.61:0/2342756770 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:04.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403ed9d700 1 --2- 172.21.15.61:0/2342756770 >> v2:172.21.15.117:3300/0 conn(0x7f40400acef0 0x7f4040146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 -- 172.21.15.61:0/2342756770 >> v2:172.21.15.117:3300/0 conn(0x7f40400acef0 msgr2=0x7f4040146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:04.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 --2- 172.21.15.61:0/2342756770 >> v2:172.21.15.117:3300/0 conn(0x7f40400acef0 0x7f4040146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 -- 172.21.15.61:0/2342756770 >> v1:172.21.15.117:6789/0 conn(0x7f40400b1e00 legacy=0x7f40400ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.911+0000 7f403dd9b700 1 -- 172.21.15.61:0/2342756770 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f404005d1e0 con 0x7f404005c3f0 2022-07-20T03:08:04.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f403dd9b700 1 --2- 172.21.15.61:0/2342756770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f404005c3f0 0x7f404014a6d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f40300032f0 tx=0x7f4030009290).ready entity=mon.0 client_cookie=a8e39934c932e11a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:04.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f403d59a700 1 -- 172.21.15.61:0/2342756770 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f403000b870 con 0x7f404005c3f0 2022-07-20T03:08:04.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f403d59a700 1 -- 172.21.15.61:0/2342756770 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403000b9d0 con 0x7f404005c3f0 2022-07-20T03:08:04.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f403d59a700 1 -- 172.21.15.61:0/2342756770 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f403000bb40 con 0x7f404005c3f0 2022-07-20T03:08:04.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 -- 172.21.15.61:0/2342756770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f404005c3f0 msgr2=0x7f404014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:04.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 --2- 172.21.15.61:0/2342756770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f404005c3f0 0x7f404014a6d0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f40300032f0 tx=0x7f4030009290).stop 2022-07-20T03:08:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 -- 172.21.15.61:0/2342756770 shutdown_connections 2022-07-20T03:08:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 --2- 172.21.15.61:0/2342756770 >> v2:172.21.15.117:3300/0 conn(0x7f40400acef0 0x7f4040146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 --2- 172.21.15.61:0/2342756770 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f404005c3f0 0x7f404014a6d0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 -- 172.21.15.61:0/2342756770 >> 172.21.15.61:0/2342756770 conn(0x7f40400611c0 msgr2=0x7f4040062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:04.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 -- 172.21.15.61:0/2342756770 shutdown_connections 2022-07-20T03:08:04.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 -- 172.21.15.61:0/2342756770 wait complete. 2022-07-20T03:08:04.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 Processor -- start 2022-07-20T03:08:04.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 -- start start 2022-07-20T03:08:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 0x7f40400666f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40400b1e00 0x7f40400b5170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.915+0000 7f404481f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40400639e0 0x7f4040155bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f404481f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40400672f0 con 0x7f40400acef0 2022-07-20T03:08:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f404481f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4040066db0 con 0x7f40400b1e00 2022-07-20T03:08:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f404481f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4040067050 con 0x7f40400639e0 2022-07-20T03:08:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 0x7f40400666f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 0x7f40400666f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43766/0 (socket says 172.21.15.61:43766) 2022-07-20T03:08:05.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 -- 172.21.15.61:0/997274523 learned_addr learned my addr 172.21.15.61:0/997274523 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:05.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403dd9b700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40400b1e00 0x7f40400b5170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403ed9d700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40400639e0 0x7f4040155bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 -- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40400639e0 msgr2=0x7f4040155bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40400639e0 0x7f4040155bb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 -- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40400b1e00 msgr2=0x7f40400b5170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40400b1e00 0x7f40400b5170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 -- 172.21.15.61:0/997274523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4030009530 con 0x7f40400acef0 2022-07-20T03:08:05.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403ed9d700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40400639e0 0x7f4040155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:08:05.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f403e59c700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 0x7f40400666f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f404005f3a0 tx=0x7f4034007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:05.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4034007e70 con 0x7f40400acef0 2022-07-20T03:08:05.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034007030 con 0x7f40400acef0 2022-07-20T03:08:05.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40400b5660 con 0x7f40400acef0 2022-07-20T03:08:05.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4034007390 con 0x7f40400acef0 2022-07-20T03:08:05.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4040066310 con 0x7f40400acef0 2022-07-20T03:08:05.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4040066be0 con 0x7f40400acef0 2022-07-20T03:08:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.919+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4034010030 con 0x7f40400acef0 2022-07-20T03:08:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.923+0000 7f402f7fe700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f401c053ad0 0x7f401c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.923+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f403406ba10 con 0x7f40400acef0 2022-07-20T03:08:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.923+0000 7f403dd9b700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f401c053ad0 0x7f401c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.923+0000 7f403dd9b700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f401c053ad0 0x7f401c055f90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f403001b000 tx=0x7f4030016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:05.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:04.927+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4034042970 con 0x7f40400acef0 2022-07-20T03:08:05.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.051+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f40400574e0 con 0x7f40400acef0 2022-07-20T03:08:05.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.051+0000 7f402f7fe700 1 -- 172.21.15.61:0/997274523 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v32) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f4034045f90 con 0x7f40400acef0 2022-07-20T03:08:05.067 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:05.068 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":32,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:08:04.482072+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:08:02.776339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T03:08:03.472472+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:08:05.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f401c053ad0 msgr2=0x7f401c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f401c053ad0 0x7f401c055f90 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f403001b000 tx=0x7f4030016000).stop 2022-07-20T03:08:05.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 msgr2=0x7f40400666f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 0x7f40400666f0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f404005f3a0 tx=0x7f4034007c00).stop 2022-07-20T03:08:05.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 shutdown_connections 2022-07-20T03:08:05.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f401c053ad0 0x7f401c055f90 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40400b1e00 0x7f40400b5170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40400639e0 0x7f4040155bb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 --2- 172.21.15.61:0/997274523 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40400acef0 0x7f40400666f0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 >> 172.21.15.61:0/997274523 conn(0x7f404005c3f0 msgr2=0x7f40400afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:05.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 shutdown_connections 2022-07-20T03:08:05.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.055+0000 7f404481f700 1 -- 172.21.15.61:0/997274523 wait complete. 2022-07-20T03:08:05.189 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:08:05.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.323+0000 7fdf5359e700 1 Processor -- start 2022-07-20T03:08:05.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5359e700 1 -- start start 2022-07-20T03:08:05.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdf540b3ee0 0x7fdf54128a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdf540b1b10 0x7fdf54128f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdf540b42c0 con 0x7fdf5405d130 2022-07-20T03:08:05.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf5412a310 con 0x7fdf540b3ee0 2022-07-20T03:08:05.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5359e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdf54067c90 con 0x7fdf540b1b10 2022-07-20T03:08:05.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdf540b3ee0 0x7fdf54128a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf52d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdf5405d130 0x7fdf5405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40372/0 (socket says 172.21.15.61:40372) 2022-07-20T03:08:05.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdf540b3ee0 0x7fdf54128a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43774/0 (socket says 172.21.15.61:43774) 2022-07-20T03:08:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf52d9d700 1 -- 172.21.15.61:0/3470961908 learned_addr learned my addr 172.21.15.61:0/3470961908 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:05.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf51d9b700 1 --2- 172.21.15.61:0/3470961908 >> v2:172.21.15.117:3300/0 conn(0x7fdf540b1b10 0x7fdf54128f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5159a700 1 -- 172.21.15.61:0/3470961908 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963729407 0 0) 0x7fdf540b42c0 con 0x7fdf5405d130 2022-07-20T03:08:05.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5159a700 1 -- 172.21.15.61:0/3470961908 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdf34003260 con 0x7fdf5405d130 2022-07-20T03:08:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 -- 172.21.15.61:0/3470961908 >> v2:172.21.15.117:3300/0 conn(0x7fdf540b1b10 msgr2=0x7fdf54128f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 --2- 172.21.15.61:0/3470961908 >> v2:172.21.15.117:3300/0 conn(0x7fdf540b1b10 0x7fdf54128f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 -- 172.21.15.61:0/3470961908 >> v1:172.21.15.61:6789/0 conn(0x7fdf5405d130 legacy=0x7fdf5405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 -- 172.21.15.61:0/3470961908 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf540672b0 con 0x7fdf540b3ee0 2022-07-20T03:08:05.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5259c700 1 --2- 172.21.15.61:0/3470961908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf540b3ee0 0x7fdf54128a70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdf540b4450 tx=0x7fdf44006df0).ready entity=mon.0 client_cookie=968218bd1446438f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:05.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5159a700 1 -- 172.21.15.61:0/3470961908 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf4400d4f0 con 0x7fdf540b3ee0 2022-07-20T03:08:05.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5159a700 1 -- 172.21.15.61:0/3470961908 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf44000e90 con 0x7fdf540b3ee0 2022-07-20T03:08:05.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.327+0000 7fdf5159a700 1 -- 172.21.15.61:0/3470961908 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf4400d840 con 0x7fdf540b3ee0 2022-07-20T03:08:05.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 -- 172.21.15.61:0/3470961908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf540b3ee0 msgr2=0x7fdf54128a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 --2- 172.21.15.61:0/3470961908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf540b3ee0 0x7fdf54128a70 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fdf540b4450 tx=0x7fdf44006df0).stop 2022-07-20T03:08:05.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 -- 172.21.15.61:0/3470961908 shutdown_connections 2022-07-20T03:08:05.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 --2- 172.21.15.61:0/3470961908 >> v2:172.21.15.117:3300/0 conn(0x7fdf540b1b10 0x7fdf54128f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 --2- 172.21.15.61:0/3470961908 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf540b3ee0 0x7fdf54128a70 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 -- 172.21.15.61:0/3470961908 >> 172.21.15.61:0/3470961908 conn(0x7fdf54064c50 msgr2=0x7fdf54065030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:05.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 -- 172.21.15.61:0/3470961908 shutdown_connections 2022-07-20T03:08:05.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 -- 172.21.15.61:0/3470961908 wait complete. 2022-07-20T03:08:05.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 Processor -- start 2022-07-20T03:08:05.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.331+0000 7fdf5359e700 1 -- start start 2022-07-20T03:08:05.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 0x7fdf54145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdf540b1b10 0x7fdf5412ab30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf540b3ee0 0x7fdf5412b020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5412c9b0 con 0x7fdf5405d130 2022-07-20T03:08:05.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf5412a460 con 0x7fdf540b3ee0 2022-07-20T03:08:05.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdf54145e40 con 0x7fdf540b1b10 2022-07-20T03:08:05.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 0x7fdf54145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 0x7fdf54145490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43780/0 (socket says 172.21.15.61:43780) 2022-07-20T03:08:05.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 -- 172.21.15.61:0/2716511107 learned_addr learned my addr 172.21.15.61:0/2716511107 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:05.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf51d9b700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdf540b1b10 0x7fdf5412ab30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf52d9d700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf540b3ee0 0x7fdf5412b020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 -- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdf540b1b10 msgr2=0x7fdf5412ab30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdf540b1b10 0x7fdf5412ab30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 -- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf540b3ee0 msgr2=0x7fdf5412b020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:05.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf540b3ee0 0x7fdf5412b020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:05.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 -- 172.21.15.61:0/2716511107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf4400b180 con 0x7fdf5405d130 2022-07-20T03:08:05.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf52d9d700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf540b3ee0 0x7fdf5412b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:05.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5259c700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 0x7fdf54145490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fdf540b4450 tx=0x7fdf44014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:05.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf44016070 con 0x7fdf5405d130 2022-07-20T03:08:05.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdf440139c0 con 0x7fdf5405d130 2022-07-20T03:08:05.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf5412b510 con 0x7fdf5405d130 2022-07-20T03:08:05.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdf44013d20 con 0x7fdf5405d130 2022-07-20T03:08:05.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.335+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf5412a690 con 0x7fdf5405d130 2022-07-20T03:08:05.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.339+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf5412a850 con 0x7fdf5405d130 2022-07-20T03:08:05.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.339+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdf44025030 con 0x7fdf5405d130 2022-07-20T03:08:05.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.339+0000 7fdf4b7fe700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdf38053ad0 0x7fdf38055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:05.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.339+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(32..32 src has 1..32) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdf4406ce40 con 0x7fdf5405d130 2022-07-20T03:08:05.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.339+0000 7fdf51d9b700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdf38053ad0 0x7fdf38055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:05.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.339+0000 7fdf51d9b700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdf38053ad0 0x7fdf38055f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fdf3c00b000 tx=0x7fdf3c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:05.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.343+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdf44043da0 con 0x7fdf5405d130 2022-07-20T03:08:05.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:05.507+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fdf540574e0 con 0x7fdf5405d130 2022-07-20T03:08:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.491+0000 7fdf4b7fe700 1 -- 172.21.15.61:0/2716511107 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v14) v1 ==== 118+0+0 (secure 0 0 0) 0x7fdf440230b0 con 0x7fdf5405d130 2022-07-20T03:08:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.491+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdf38053ad0 msgr2=0x7fdf38055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdf38053ad0 0x7fdf38055f90 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7fdf3c00b000 tx=0x7fdf3c006df0).stop 2022-07-20T03:08:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 msgr2=0x7fdf54145490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 0x7fdf54145490 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fdf540b4450 tx=0x7fdf44014000).stop 2022-07-20T03:08:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 shutdown_connections 2022-07-20T03:08:06.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdf38053ad0 0x7fdf38055f90 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdf540b3ee0 0x7fdf5412b020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdf540b1b10 0x7fdf5412ab30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 --2- 172.21.15.61:0/2716511107 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdf5405d130 0x7fdf54145490 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 >> 172.21.15.61:0/2716511107 conn(0x7fdf54064c50 msgr2=0x7fdf5405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:06.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 shutdown_connections 2022-07-20T03:08:06.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.495+0000 7fdf5359e700 1 -- 172.21.15.61:0/2716511107 wait complete. 2022-07-20T03:08:06.536 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:08:06.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aed052700 1 Processor -- start 2022-07-20T03:08:06.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aed052700 1 -- start start 2022-07-20T03:08:06.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aed052700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80adff0 0x7f5ae80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:06.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aed052700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae8147010 con 0x7f5ae80af8e0 2022-07-20T03:08:06.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aed052700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae81472b0 con 0x7f5ae80b5650 2022-07-20T03:08:06.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aed052700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8146d40 con 0x7f5ae80adff0 2022-07-20T03:08:06.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aec851700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5ae80af8e0 0x7f5ae805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40390/0 (socket says 172.21.15.61:40390) 2022-07-20T03:08:06.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5aec851700 1 -- 172.21.15.61:0/59250291 learned_addr learned my addr 172.21.15.61:0/59250291 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:06.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/59250291 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80adff0 0x7f5ae80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:06.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/59250291 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2070557937 0 0) 0x7f5ae8147010 con 0x7f5ae80af8e0 2022-07-20T03:08:06.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/59250291 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac8003260 con 0x7f5ae80af8e0 2022-07-20T03:08:06.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/59250291 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 879882263 0 0) 0x7f5ae81472b0 con 0x7f5ae80b5650 2022-07-20T03:08:06.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae77fe700 1 -- 172.21.15.61:0/59250291 >> v1:172.21.15.117:6789/0 conn(0x7f5ae80b5650 legacy=0x7f5ae8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae77fe700 1 -- 172.21.15.61:0/59250291 >> v1:172.21.15.61:6789/0 conn(0x7f5ae80af8e0 legacy=0x7f5ae805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae77fe700 1 -- 172.21.15.61:0/59250291 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae805c4a0 con 0x7f5ae80adff0 2022-07-20T03:08:06.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/59250291 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80adff0 0x7f5ae80b5a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5ae0001780 tx=0x7f5ad80091a0).ready entity=mon.2 client_cookie=ce34b05b917d30f4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:06.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/59250291 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ae00015e0 con 0x7f5ae80adff0 2022-07-20T03:08:06.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/59250291 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad800b580 con 0x7f5ae80adff0 2022-07-20T03:08:06.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.651+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/59250291 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad800b6f0 con 0x7f5ae80adff0 2022-07-20T03:08:06.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/59250291 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80adff0 msgr2=0x7f5ae80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 --2- 172.21.15.61:0/59250291 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80adff0 0x7f5ae80b5a30 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f5ae0001780 tx=0x7f5ad80091a0).stop 2022-07-20T03:08:06.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/59250291 shutdown_connections 2022-07-20T03:08:06.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 --2- 172.21.15.61:0/59250291 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80adff0 0x7f5ae80b5a30 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/59250291 >> 172.21.15.61:0/59250291 conn(0x7f5ae80634e0 msgr2=0x7f5ae80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:06.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/59250291 shutdown_connections 2022-07-20T03:08:06.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/59250291 wait complete. 2022-07-20T03:08:06.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 Processor -- start 2022-07-20T03:08:06.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- start start 2022-07-20T03:08:06.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80adff0 0x7f5ae8067070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:06.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 0x7f5ae8067560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:06.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 0x7f5ae8065d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:06.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8110070 con 0x7f5ae80adff0 2022-07-20T03:08:06.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae8064710 con 0x7f5ae80af8e0 2022-07-20T03:08:06.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae80644b0 con 0x7f5ae80b5650 2022-07-20T03:08:06.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80adff0 0x7f5ae8067070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:06.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 0x7f5ae8067560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:06.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aec851700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 0x7f5ae8065d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:06.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 0x7f5ae8067560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:53474/0 (socket says 172.21.15.61:53474) 2022-07-20T03:08:06.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 -- 172.21.15.61:0/3554802037 learned_addr learned my addr 172.21.15.61:0/3554802037 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:06.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aec851700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 0x7f5ae8065d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58702/0 (socket says 172.21.15.61:58702) 2022-07-20T03:08:06.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 -- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 msgr2=0x7f5ae8065d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 0x7f5ae8065d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 -- 172.21.15.61:0/3554802037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80adff0 msgr2=0x7f5ae8067070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:06.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80adff0 0x7f5ae8067070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:06.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 -- 172.21.15.61:0/3554802037 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad8009420 con 0x7f5ae80af8e0 2022-07-20T03:08:06.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aec851700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 0x7f5ae8065d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:06.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 0x7f5ae8067560 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f5ae0001780 tx=0x7f5ad80091a0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:06.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad8006dc0 con 0x7f5ae80af8e0 2022-07-20T03:08:06.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad80032d0 con 0x7f5ae80af8e0 2022-07-20T03:08:06.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae8111f60 con 0x7f5ae80af8e0 2022-07-20T03:08:06.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad8011d70 con 0x7f5ae80af8e0 2022-07-20T03:08:06.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.655+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae8111c20 con 0x7f5ae80af8e0 2022-07-20T03:08:06.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.659+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae80574e0 con 0x7f5ae80af8e0 2022-07-20T03:08:06.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.659+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5ad8017070 con 0x7f5ae80af8e0 2022-07-20T03:08:06.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.659+0000 7f5ae57fa700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053940 0x7f5acc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:06.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.659+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5ad806ce30 con 0x7f5ae80af8e0 2022-07-20T03:08:06.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.659+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053940 0x7f5acc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:06.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.659+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053940 0x7f5acc055e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f5ad000d000 tx=0x7f5ad0007ab0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:06.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.671+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ad8043d90 con 0x7f5ae80af8e0 2022-07-20T03:08:07.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.835+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5ae805c280 con 0x7f5ae80af8e0 2022-07-20T03:08:07.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.835+0000 7f5ae57fa700 1 -- 172.21.15.61:0/3554802037 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f5ad80473b0 con 0x7f5ae80af8e0 2022-07-20T03:08:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:07.095 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:08:07.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053940 msgr2=0x7f5acc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053940 0x7f5acc055e00 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f5ad000d000 tx=0x7f5ad0007ab0).stop 2022-07-20T03:08:07.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 msgr2=0x7f5ae8067560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 0x7f5ae8067560 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f5ae0001780 tx=0x7f5ad80091a0).stop 2022-07-20T03:08:07.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 shutdown_connections 2022-07-20T03:08:07.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053940 0x7f5acc055e00 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80af8e0 0x7f5ae8067560 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80b5650 0x7f5ae8065d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 --2- 172.21.15.61:0/3554802037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80adff0 0x7f5ae8067070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 >> 172.21.15.61:0/3554802037 conn(0x7f5ae80634e0 msgr2=0x7f5ae8144010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:07.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 shutdown_connections 2022-07-20T03:08:07.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:06.839+0000 7f5aed052700 1 -- 172.21.15.61:0/3554802037 wait complete. 2022-07-20T03:08:07.187 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:08:07.188 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:08:07.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.295+0000 7fb7f6bb9700 1 Processor -- start 2022-07-20T03:08:07.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.295+0000 7fb7f6bb9700 1 -- start start 2022-07-20T03:08:07.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.295+0000 7fb7f6bb9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb7f013c790 0x7fb7f00b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.295+0000 7fb7f6bb9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f00b3ca0 0x7fb7f00b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7f0142ba0 con 0x7fb7f00b9f00 2022-07-20T03:08:07.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb7f01429b0 con 0x7fb7f013c790 2022-07-20T03:08:07.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f0146330 con 0x7fb7f00b3ca0 2022-07-20T03:08:07.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb7f013c790 0x7fb7f00b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:07.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f63b8700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb7f00b9f00 0x7fb7f0143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:48908/0 (socket says 172.21.15.61:48908) 2022-07-20T03:08:07.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb7f013c790 0x7fb7f00b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43804/0 (socket says 172.21.15.61:43804) 2022-07-20T03:08:07.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f53b6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f00b3ca0 0x7fb7f00b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:07.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f63b8700 1 -- 172.21.15.61:0/1778734713 learned_addr learned my addr 172.21.15.61:0/1778734713 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:07.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f4bb5700 1 -- 172.21.15.61:0/1778734713 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2042836598 0 0) 0x7fb7f0142ba0 con 0x7fb7f00b9f00 2022-07-20T03:08:07.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 -- 172.21.15.61:0/1778734713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f00b3ca0 msgr2=0x7fb7f00b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:07.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 --2- 172.21.15.61:0/1778734713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f00b3ca0 0x7fb7f00b9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 -- 172.21.15.61:0/1778734713 >> v1:172.21.15.117:6789/0 conn(0x7fb7f00b9f00 legacy=0x7fb7f0143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 -- 172.21.15.61:0/1778734713 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f00b4c00 con 0x7fb7f013c790 2022-07-20T03:08:07.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f5bb7700 1 --2- 172.21.15.61:0/1778734713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f013c790 0x7fb7f00b4710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb7e0002f70 tx=0x7fb7e0006df0).ready entity=mon.0 client_cookie=e3dc8ab5b8beb4d4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:07.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f4bb5700 1 -- 172.21.15.61:0/1778734713 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7e000d570 con 0x7fb7f013c790 2022-07-20T03:08:07.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f4bb5700 1 -- 172.21.15.61:0/1778734713 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7e0000e90 con 0x7fb7f013c790 2022-07-20T03:08:07.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f4bb5700 1 -- 172.21.15.61:0/1778734713 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7e000d8c0 con 0x7fb7f013c790 2022-07-20T03:08:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/1778734713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f013c790 msgr2=0x7fb7f00b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/1778734713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f013c790 0x7fb7f00b4710 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7fb7e0002f70 tx=0x7fb7e0006df0).stop 2022-07-20T03:08:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/1778734713 shutdown_connections 2022-07-20T03:08:07.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/1778734713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f00b3ca0 0x7fb7f00b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/1778734713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f013c790 0x7fb7f00b4710 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/1778734713 >> 172.21.15.61:0/1778734713 conn(0x7fb7f00678b0 msgr2=0x7fb7f00b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:07.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/1778734713 shutdown_connections 2022-07-20T03:08:07.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.299+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/1778734713 wait complete. 2022-07-20T03:08:07.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 Processor -- start 2022-07-20T03:08:07.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 -- start start 2022-07-20T03:08:07.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7f00b3ca0 0x7fb7f013e030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 0x7fb7f00b1cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f013c790 0x7fb7f00b21b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00af540 con 0x7fb7f00b9f00 2022-07-20T03:08:07.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00ae260 con 0x7fb7f00b3ca0 2022-07-20T03:08:07.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7f00ae930 con 0x7fb7f013c790 2022-07-20T03:08:07.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 0x7fb7f00b1cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f5bb7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7f00b3ca0 0x7fb7f013e030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 0x7fb7f00b1cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43810/0 (socket says 172.21.15.61:43810) 2022-07-20T03:08:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 -- 172.21.15.61:0/3541284282 learned_addr learned my addr 172.21.15.61:0/3541284282 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f63b8700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f013c790 0x7fb7f00b21b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:07.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 -- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f013c790 msgr2=0x7fb7f00b21b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f013c790 0x7fb7f00b21b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 -- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7f00b3ca0 msgr2=0x7fb7f013e030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7f00b3ca0 0x7fb7f013e030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 -- 172.21.15.61:0/3541284282 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7e000b180 con 0x7fb7f00b9f00 2022-07-20T03:08:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f63b8700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f013c790 0x7fb7f00b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f53b6700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 0x7fb7f00b1cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fb7f00b0d60 tx=0x7fb7ec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:07.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7ec010780 con 0x7fb7f00b9f00 2022-07-20T03:08:07.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7ec00b3b0 con 0x7fb7f00b9f00 2022-07-20T03:08:07.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7f00af730 con 0x7fb7f00b9f00 2022-07-20T03:08:07.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7ec010ad0 con 0x7fb7f00b9f00 2022-07-20T03:08:07.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.303+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb7f00b2cd0 con 0x7fb7f00b9f00 2022-07-20T03:08:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.307+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7f013e6c0 con 0x7fb7f00b9f00 2022-07-20T03:08:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.307+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb7ec010c30 con 0x7fb7f00b9f00 2022-07-20T03:08:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.307+0000 7fb7deffd700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7d0053ad0 0x7fb7d0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.307+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb7ec06c310 con 0x7fb7f00b9f00 2022-07-20T03:08:07.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.307+0000 7fb7f5bb7700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7d0053ad0 0x7fb7d0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:07.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.307+0000 7fb7f5bb7700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7d0053ad0 0x7fb7d0055f90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb7e001a000 tx=0x7fb7e0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:07.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.311+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7ec043170 con 0x7fb7f00b9f00 2022-07-20T03:08:07.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.435+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7f0146090 con 0x7fb7f00b9f00 2022-07-20T03:08:07.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7deffd700 1 -- 172.21.15.61:0/3541284282 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fb7ec046790 con 0x7fb7f00b9f00 2022-07-20T03:08:07.639 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:07.640 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":33,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:08:05.485261+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:08:02.776339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T03:08:03.472472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:08:07.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7d0053ad0 msgr2=0x7fb7d0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7d0053ad0 0x7fb7d0055f90 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fb7e001a000 tx=0x7fb7e0014000).stop 2022-07-20T03:08:07.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 msgr2=0x7fb7f00b1cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:07.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 0x7fb7f00b1cc0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7fb7f00b0d60 tx=0x7fb7ec006df0).stop 2022-07-20T03:08:07.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 shutdown_connections 2022-07-20T03:08:07.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7d0053ad0 0x7fb7d0055f90 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7f013c790 0x7fb7f00b21b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7f00b9f00 0x7fb7f00b1cc0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 --2- 172.21.15.61:0/3541284282 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7f00b3ca0 0x7fb7f013e030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:07.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 >> 172.21.15.61:0/3541284282 conn(0x7fb7f00678b0 msgr2=0x7fb7f01466a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:07.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 shutdown_connections 2022-07-20T03:08:07.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.439+0000 7fb7f6bb9700 1 -- 172.21.15.61:0/3541284282 wait complete. 2022-07-20T03:08:07.725 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:08:07.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.883+0000 7f4d231e8700 1 Processor -- start 2022-07-20T03:08:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.883+0000 7f4d231e8700 1 -- start start 2022-07-20T03:08:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4d1c0af0d0 0x7f4d1c0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:07.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0b4290 con 0x7f4d1c0b4c20 2022-07-20T03:08:07.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4d1c0af4b0 con 0x7f4d1c0a9690 2022-07-20T03:08:07.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4d1c0a9340 con 0x7f4d1c0af0d0 2022-07-20T03:08:08.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d229e7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4d1c0b4c20 0x7f4d1c1409b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40418/0 (socket says 172.21.15.61:40418) 2022-07-20T03:08:08.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d229e7700 1 -- 172.21.15.61:0/3644862337 learned_addr learned my addr 172.21.15.61:0/3644862337 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:08.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d221e6700 1 --2- 172.21.15.61:0/3644862337 >> v2:172.21.15.117:3300/0 conn(0x7f4d1c0af0d0 0x7f4d1c0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1717925000 0 0) 0x7f4d1c0b4290 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d04003260 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4273541343 0 0) 0x7f4d1c0af4b0 con 0x7f4d1c0a9690 2022-07-20T03:08:08.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4d1c0b4290 con 0x7f4d1c0a9690 2022-07-20T03:08:08.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d221e6700 1 -- 172.21.15.61:0/3644862337 >> v1:172.21.15.117:6789/0 conn(0x7f4d1c0a9690 legacy=0x7f4d1c0ac2f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d221e6700 1 -- 172.21.15.61:0/3644862337 >> v1:172.21.15.61:6789/0 conn(0x7f4d1c0b4c20 legacy=0x7f4d1c1409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d221e6700 1 -- 172.21.15.61:0/3644862337 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1c0b44e0 con 0x7f4d1c0af0d0 2022-07-20T03:08:08.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d221e6700 1 --2- 172.21.15.61:0/3644862337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0af0d0 0x7f4d1c0abe00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4d10002f70 tx=0x7f4d10006df0).ready entity=mon.1 client_cookie=4ad37fe5db96fdd6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:08.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d1000d540 con 0x7f4d1c0af0d0 2022-07-20T03:08:08.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d10000e90 con 0x7f4d1c0af0d0 2022-07-20T03:08:08.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d211e4700 1 -- 172.21.15.61:0/3644862337 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d1000d890 con 0x7f4d1c0af0d0 2022-07-20T03:08:08.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- 172.21.15.61:0/3644862337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0af0d0 msgr2=0x7f4d1c0abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 --2- 172.21.15.61:0/3644862337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0af0d0 0x7f4d1c0abe00 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f4d10002f70 tx=0x7f4d10006df0).stop 2022-07-20T03:08:08.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- 172.21.15.61:0/3644862337 shutdown_connections 2022-07-20T03:08:08.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 --2- 172.21.15.61:0/3644862337 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0af0d0 0x7f4d1c0abe00 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- 172.21.15.61:0/3644862337 >> 172.21.15.61:0/3644862337 conn(0x7f4d1c063150 msgr2=0x7f4d1c063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:08.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- 172.21.15.61:0/3644862337 shutdown_connections 2022-07-20T03:08:08.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.887+0000 7f4d231e8700 1 -- 172.21.15.61:0/3644862337 wait complete. 2022-07-20T03:08:08.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 Processor -- start 2022-07-20T03:08:08.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 -- start start 2022-07-20T03:08:08.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0a9690 0x7f4d1c05fb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d1c0af0d0 0x7f4d1c153150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 0x7f4d1c066680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c153640 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c143b40 con 0x7f4d1c0a9690 2022-07-20T03:08:08.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d231e8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4d1c143de0 con 0x7f4d1c0af0d0 2022-07-20T03:08:08.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d229e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 0x7f4d1c066680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d221e6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0a9690 0x7f4d1c05fb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d229e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 0x7f4d1c066680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43826/0 (socket says 172.21.15.61:43826) 2022-07-20T03:08:08.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d219e5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d1c0af0d0 0x7f4d1c153150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.891+0000 7f4d229e7700 1 -- 172.21.15.61:0/416114123 learned_addr learned my addr 172.21.15.61:0/416114123 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:08.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d229e7700 1 -- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d1c0af0d0 msgr2=0x7f4d1c153150 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d229e7700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d1c0af0d0 0x7f4d1c153150 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d229e7700 1 -- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0a9690 msgr2=0x7f4d1c05fb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d229e7700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0a9690 0x7f4d1c05fb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d229e7700 1 -- 172.21.15.61:0/416114123 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d1000b180 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d221e6700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0a9690 0x7f4d1c05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:08.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d229e7700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 0x7f4d1c066680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f4d14001490 tx=0x7f4d14007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:08.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d14005ba0 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4d14005d00 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4d140072c0 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d1c144080 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d1c09e080 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d1c0b4780 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.895+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4d14010070 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.899+0000 7f4d0f7fe700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cfc053a40 0x7f4cfc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.899+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4d1406b7d0 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.899+0000 7f4d221e6700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cfc053a40 0x7f4cfc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.899+0000 7f4d221e6700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cfc053a40 0x7f4cfc055f00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4d10018000 tx=0x7f4d10014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:08.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:07.903+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4d14042620 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.067+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4d1c0574e0 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.075+0000 7f4d0f7fe700 1 -- 172.21.15.61:0/416114123 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v13) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4d14045c40 con 0x7f4d1c0b4c20 2022-07-20T03:08:08.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cfc053a40 msgr2=0x7f4cfc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cfc053a40 0x7f4cfc055f00 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4d10018000 tx=0x7f4d10014000).stop 2022-07-20T03:08:08.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 msgr2=0x7f4d1c066680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 0x7f4d1c066680 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f4d14001490 tx=0x7f4d14007de0).stop 2022-07-20T03:08:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 shutdown_connections 2022-07-20T03:08:08.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4cfc053a40 0x7f4cfc055f00 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4d1c0b4c20 0x7f4d1c066680 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4d1c0af0d0 0x7f4d1c153150 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 --2- 172.21.15.61:0/416114123 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4d1c0a9690 0x7f4d1c05fb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 >> 172.21.15.61:0/416114123 conn(0x7f4d1c063150 msgr2=0x7f4d1c0b2840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:08.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 shutdown_connections 2022-07-20T03:08:08.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.079+0000 7f4d231e8700 1 -- 172.21.15.61:0/416114123 wait complete. 2022-07-20T03:08:08.270 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:08:08.270 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:08:08.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 Processor -- start 2022-07-20T03:08:08.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 -- start start 2022-07-20T03:08:08.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e0146a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24e00b5830 0x7f24e0142c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24e00b0130 con 0x7f24e0143120 2022-07-20T03:08:08.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24e01462d0 con 0x7f24e00b5830 2022-07-20T03:08:08.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24e5425700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0146570 con 0x7f24e005cf40 2022-07-20T03:08:08.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24e00b5830 0x7f24e0142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24e00b5830 0x7f24e0142c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43832/0 (socket says 172.21.15.61:43832) 2022-07-20T03:08:08.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24df7fe700 1 --1- 172.21.15.61:0/151317746 >> v1:172.21.15.117:6789/0 conn(0x7f24e0143120 0x7f24e00af2c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:48936/0 (socket says 172.21.15.61:48936) 2022-07-20T03:08:08.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 -- 172.21.15.61:0/151317746 learned_addr learned my addr 172.21.15.61:0/151317746 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:08.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24deffd700 1 --2- 172.21.15.61:0/151317746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e0146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 -- 172.21.15.61:0/151317746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 msgr2=0x7f24e0146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:08.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24de7fc700 1 -- 172.21.15.61:0/151317746 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808627781 0 0) 0x7f24e00b0130 con 0x7f24e0143120 2022-07-20T03:08:08.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 --2- 172.21.15.61:0/151317746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e0146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 -- 172.21.15.61:0/151317746 >> v1:172.21.15.117:6789/0 conn(0x7f24e0143120 legacy=0x7f24e00af2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 -- 172.21.15.61:0/151317746 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24e0061280 con 0x7f24e00b5830 2022-07-20T03:08:08.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24d7fff700 1 --2- 172.21.15.61:0/151317746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e00b5830 0x7f24e0142c30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f24c8002f70 tx=0x7f24c8006eb0).ready entity=mon.0 client_cookie=aacfb4792966d260 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:08.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24de7fc700 1 -- 172.21.15.61:0/151317746 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24c800d570 con 0x7f24e00b5830 2022-07-20T03:08:08.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24de7fc700 1 -- 172.21.15.61:0/151317746 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c8000e90 con 0x7f24e00b5830 2022-07-20T03:08:08.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.395+0000 7f24de7fc700 1 -- 172.21.15.61:0/151317746 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24c800d8c0 con 0x7f24e00b5830 2022-07-20T03:08:08.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- 172.21.15.61:0/151317746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e00b5830 msgr2=0x7f24e0142c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 --2- 172.21.15.61:0/151317746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e00b5830 0x7f24e0142c30 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f24c8002f70 tx=0x7f24c8006eb0).stop 2022-07-20T03:08:08.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- 172.21.15.61:0/151317746 shutdown_connections 2022-07-20T03:08:08.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 --2- 172.21.15.61:0/151317746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e00b5830 0x7f24e0142c30 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 --2- 172.21.15.61:0/151317746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e0146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- 172.21.15.61:0/151317746 >> 172.21.15.61:0/151317746 conn(0x7f24e00b69e0 msgr2=0x7f24e0066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:08.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- 172.21.15.61:0/151317746 shutdown_connections 2022-07-20T03:08:08.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- 172.21.15.61:0/151317746 wait complete. 2022-07-20T03:08:08.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 Processor -- start 2022-07-20T03:08:08.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- start start 2022-07-20T03:08:08.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e00b2370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24e00b5830 0x7f24e009ea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 0x7f24e013eed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e013de00 con 0x7f24e0143120 2022-07-20T03:08:08.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24e009e600 con 0x7f24e00b5830 2022-07-20T03:08:08.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24e5425700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24e0140220 con 0x7f24e005cf40 2022-07-20T03:08:08.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 0x7f24e013eed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 0x7f24e013eed0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43840/0 (socket says 172.21.15.61:43840) 2022-07-20T03:08:08.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.399+0000 7f24df7fe700 1 -- 172.21.15.61:0/1474299648 learned_addr learned my addr 172.21.15.61:0/1474299648 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:08.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24d7fff700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24e00b5830 0x7f24e009ea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24deffd700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e00b2370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24df7fe700 1 -- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 msgr2=0x7f24e00b2370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24df7fe700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e00b2370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24df7fe700 1 -- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24e00b5830 msgr2=0x7f24e009ea60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24df7fe700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24e00b5830 0x7f24e009ea60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24df7fe700 1 -- 172.21.15.61:0/1474299648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24c800b1e0 con 0x7f24e0143120 2022-07-20T03:08:08.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24df7fe700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 0x7f24e013eed0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f24e013df70 tx=0x7f24d8001360).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:08.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24d8005ae0 con 0x7f24e0143120 2022-07-20T03:08:08.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24d8005c40 con 0x7f24e0143120 2022-07-20T03:08:08.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24d80070d0 con 0x7f24e0143120 2022-07-20T03:08:08.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24e013ffe0 con 0x7f24e0143120 2022-07-20T03:08:08.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24e009d8f0 con 0x7f24e0143120 2022-07-20T03:08:08.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24e009e7c0 con 0x7f24e0143120 2022-07-20T03:08:08.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.403+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f24d8010070 con 0x7f24e0143120 2022-07-20T03:08:08.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.407+0000 7f24dcff9700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c4053940 0x7f24c4055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.407+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f24d806b660 con 0x7f24e0143120 2022-07-20T03:08:08.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.407+0000 7f24deffd700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c4053940 0x7f24c4055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.407+0000 7f24deffd700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c4053940 0x7f24c4055e00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f24d000b000 tx=0x7f24d0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:08.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.415+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24e009e7c0 con 0x7f24e0143120 2022-07-20T03:08:08.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.579+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f24e009e7c0 con 0x7f24e0143120 2022-07-20T03:08:08.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.587+0000 7f24dcff9700 1 -- 172.21.15.61:0/1474299648 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v14) v1 ==== 184+0+0 (secure 0 0 0) 0x7f24e009e7c0 con 0x7f24e0143120 2022-07-20T03:08:08.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c4053940 msgr2=0x7f24c4055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c4053940 0x7f24c4055e00 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f24d000b000 tx=0x7f24d0008040).stop 2022-07-20T03:08:08.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 msgr2=0x7f24e013eed0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 0x7f24e013eed0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f24e013df70 tx=0x7f24d8001360).stop 2022-07-20T03:08:08.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 shutdown_connections 2022-07-20T03:08:08.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c4053940 0x7f24c4055e00 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24e00b5830 0x7f24e009ea60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24e0143120 0x7f24e013eed0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 --2- 172.21.15.61:0/1474299648 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24e005cf40 0x7f24e00b2370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 >> 172.21.15.61:0/1474299648 conn(0x7f24e00b69e0 msgr2=0x7f24e0060790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:08.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 shutdown_connections 2022-07-20T03:08:08.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.591+0000 7f24e5425700 1 -- 172.21.15.61:0/1474299648 wait complete. 2022-07-20T03:08:08.633 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:08:08.635 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:08:08.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 Processor -- start 2022-07-20T03:08:08.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 -- start start 2022-07-20T03:08:08.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ed80b3300 0x7f6ed80b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6ed813f6c0 0x7f6ed80618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:08.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ed80ac8a0 con 0x7f6ed8060220 2022-07-20T03:08:08.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed80acb40 con 0x7f6ed80b3300 2022-07-20T03:08:08.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edf472700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ed80acde0 con 0x7f6ed813f6c0 2022-07-20T03:08:08.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ed80b3300 0x7f6ed80b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ed80b3300 0x7f6ed80b36e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43844/0 (socket says 172.21.15.61:43844) 2022-07-20T03:08:08.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 -- 172.21.15.61:0/1604194493 learned_addr learned my addr 172.21.15.61:0/1604194493 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:08.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edda0f700 1 --2- 172.21.15.61:0/1604194493 >> v2:172.21.15.117:3300/0 conn(0x7f6ed813f6c0 0x7f6ed80618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:08.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 -- 172.21.15.61:0/1604194493 >> v2:172.21.15.117:3300/0 conn(0x7f6ed813f6c0 msgr2=0x7f6ed80618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:08.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 --2- 172.21.15.61:0/1604194493 >> v2:172.21.15.117:3300/0 conn(0x7f6ed813f6c0 0x7f6ed80618e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:08.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 -- 172.21.15.61:0/1604194493 >> v1:172.21.15.117:6789/0 conn(0x7f6ed8060220 legacy=0x7f6ed813efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:08.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 -- 172.21.15.61:0/1604194493 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed80ac590 con 0x7f6ed80b3300 2022-07-20T03:08:09.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.747+0000 7f6edd20e700 1 --2- 172.21.15.61:0/1604194493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed80b3300 0x7f6ed80b36e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f6ed40032f0 tx=0x7f6ed4009290).ready entity=mon.0 client_cookie=fb8ad9dd43a326af server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:09.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6ecbfff700 1 -- 172.21.15.61:0/1604194493 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ed400b870 con 0x7f6ed80b3300 2022-07-20T03:08:09.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6ecbfff700 1 -- 172.21.15.61:0/1604194493 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ed400b9d0 con 0x7f6ed80b3300 2022-07-20T03:08:09.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6ecbfff700 1 -- 172.21.15.61:0/1604194493 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ed400bb40 con 0x7f6ed80b3300 2022-07-20T03:08:09.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- 172.21.15.61:0/1604194493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed80b3300 msgr2=0x7f6ed80b36e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 --2- 172.21.15.61:0/1604194493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed80b3300 0x7f6ed80b36e0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f6ed40032f0 tx=0x7f6ed4009290).stop 2022-07-20T03:08:09.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- 172.21.15.61:0/1604194493 shutdown_connections 2022-07-20T03:08:09.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 --2- 172.21.15.61:0/1604194493 >> v2:172.21.15.117:3300/0 conn(0x7f6ed813f6c0 0x7f6ed80618e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 --2- 172.21.15.61:0/1604194493 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed80b3300 0x7f6ed80b36e0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- 172.21.15.61:0/1604194493 >> 172.21.15.61:0/1604194493 conn(0x7f6ed80b6c20 msgr2=0x7f6ed80b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:09.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- 172.21.15.61:0/1604194493 shutdown_connections 2022-07-20T03:08:09.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- 172.21.15.61:0/1604194493 wait complete. 2022-07-20T03:08:09.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 Processor -- start 2022-07-20T03:08:09.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- start start 2022-07-20T03:08:09.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ed8060220 0x7f6ed809b4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 0x7f6ed814f4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed813f6c0 0x7f6ed814f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed814fed0 con 0x7f6ed813f6c0 2022-07-20T03:08:09.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed80afa10 con 0x7f6ed8060220 2022-07-20T03:08:09.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.751+0000 7f6edf472700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ed80ae890 con 0x7f6ed80b3300 2022-07-20T03:08:09.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 0x7f6ed814f4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edd20e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ed8060220 0x7f6ed809b4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 0x7f6ed814f4f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58756/0 (socket says 172.21.15.61:58756) 2022-07-20T03:08:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 -- 172.21.15.61:0/2653328440 learned_addr learned my addr 172.21.15.61:0/2653328440 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edda0f700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed813f6c0 0x7f6ed814f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 -- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ed8060220 msgr2=0x7f6ed809b4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ed8060220 0x7f6ed809b4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 -- 172.21.15.61:0/2653328440 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed813f6c0 msgr2=0x7f6ed814f9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed813f6c0 0x7f6ed814f9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 -- 172.21.15.61:0/2653328440 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ed4009530 con 0x7f6ed80b3300 2022-07-20T03:08:09.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edd20e700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ed8060220 0x7f6ed809b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:09.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edda0f700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed813f6c0 0x7f6ed814f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:09.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edca0d700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 0x7f6ed814f4f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6ed8150010 tx=0x7f6ecc007c10).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:09.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ecc007030 con 0x7f6ed80b3300 2022-07-20T03:08:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ed80afec0 con 0x7f6ed80b3300 2022-07-20T03:08:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ed80af690 con 0x7f6ed80b3300 2022-07-20T03:08:09.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ecc007190 con 0x7f6ed80b3300 2022-07-20T03:08:09.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ecc006510 con 0x7f6ed80b3300 2022-07-20T03:08:09.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.755+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ed80b0240 con 0x7f6ed80b3300 2022-07-20T03:08:09.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.759+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6ecc010030 con 0x7f6ed80b3300 2022-07-20T03:08:09.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.759+0000 7f6eca7fc700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eb8055b30 0x7f6eb8057ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.759+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6ecc06bf60 con 0x7f6ed80b3300 2022-07-20T03:08:09.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.759+0000 7f6edd20e700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eb8055b30 0x7f6eb8057ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.759+0000 7f6edd20e700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eb8055b30 0x7f6eb8057ff0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6ed401b000 tx=0x7f6ed4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:09.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.767+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ecc042ec0 con 0x7f6ed80b3300 2022-07-20T03:08:09.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.931+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6ed8142fa0 con 0x7f6ed80b3300 2022-07-20T03:08:09.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6eca7fc700 1 -- 172.21.15.61:0/2653328440 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v15) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6ecc0464e0 con 0x7f6ed80b3300 2022-07-20T03:08:09.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eb8055b30 msgr2=0x7f6eb8057ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eb8055b30 0x7f6eb8057ff0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6ed401b000 tx=0x7f6ed4016000).stop 2022-07-20T03:08:09.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 msgr2=0x7f6ed814f4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 0x7f6ed814f4f0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f6ed8150010 tx=0x7f6ecc007c10).stop 2022-07-20T03:08:09.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 shutdown_connections 2022-07-20T03:08:09.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eb8055b30 0x7f6eb8057ff0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ed813f6c0 0x7f6ed814f9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ed80b3300 0x7f6ed814f4f0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 --2- 172.21.15.61:0/2653328440 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ed8060220 0x7f6ed809b4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 >> 172.21.15.61:0/2653328440 conn(0x7f6ed80b6c20 msgr2=0x7f6ed80b2b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:09.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 shutdown_connections 2022-07-20T03:08:09.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:08.943+0000 7f6edf472700 1 -- 172.21.15.61:0/2653328440 wait complete. 2022-07-20T03:08:09.177 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:08:09.180 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:08:09.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc948087700 1 Processor -- start 2022-07-20T03:08:09.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc948087700 1 -- start start 2022-07-20T03:08:09.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc948087700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400aedc0 0x7fc94013f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc948087700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9400acdb0 con 0x7fc9400b0080 2022-07-20T03:08:09.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc948087700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9400b1450 con 0x7fc9400ac9d0 2022-07-20T03:08:09.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc948087700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400af260 con 0x7fc9400aedc0 2022-07-20T03:08:09.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc946624700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc9400b0080 0x7fc94013f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40460/0 (socket says 172.21.15.61:40460) 2022-07-20T03:08:09.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc946624700 1 -- 172.21.15.61:0/2137695681 learned_addr learned my addr 172.21.15.61:0/2137695681 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:09.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc945622700 1 --2- 172.21.15.61:0/2137695681 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400aedc0 0x7fc94013f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc944e21700 1 -- 172.21.15.61:0/2137695681 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1045273503 0 0) 0x7fc9400acdb0 con 0x7fc9400b0080 2022-07-20T03:08:09.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.291+0000 7fc944e21700 1 -- 172.21.15.61:0/2137695681 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc928003260 con 0x7fc9400b0080 2022-07-20T03:08:09.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc945622700 1 -- 172.21.15.61:0/2137695681 >> v1:172.21.15.117:6789/0 conn(0x7fc9400ac9d0 legacy=0x7fc94013e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc944e21700 1 -- 172.21.15.61:0/2137695681 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2342550617 0 0) 0x7fc9400b1450 con 0x7fc9400ac9d0 2022-07-20T03:08:09.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc945622700 1 -- 172.21.15.61:0/2137695681 >> v1:172.21.15.61:6789/0 conn(0x7fc9400b0080 legacy=0x7fc94013f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc945622700 1 -- 172.21.15.61:0/2137695681 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9400af450 con 0x7fc9400aedc0 2022-07-20T03:08:09.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc945622700 1 --2- 172.21.15.61:0/2137695681 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400aedc0 0x7fc94013f2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc938001b80 tx=0x7fc930006df0).ready entity=mon.2 client_cookie=e9c98b0cb66d590e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:09.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc944e21700 1 -- 172.21.15.61:0/2137695681 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9380019e0 con 0x7fc9400aedc0 2022-07-20T03:08:09.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc944e21700 1 -- 172.21.15.61:0/2137695681 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc930000e90 con 0x7fc9400aedc0 2022-07-20T03:08:09.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc944e21700 1 -- 172.21.15.61:0/2137695681 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc93000d4a0 con 0x7fc9400aedc0 2022-07-20T03:08:09.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 -- 172.21.15.61:0/2137695681 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400aedc0 msgr2=0x7fc94013f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 --2- 172.21.15.61:0/2137695681 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400aedc0 0x7fc94013f2a0 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc938001b80 tx=0x7fc930006df0).stop 2022-07-20T03:08:09.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 -- 172.21.15.61:0/2137695681 shutdown_connections 2022-07-20T03:08:09.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 --2- 172.21.15.61:0/2137695681 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400aedc0 0x7fc94013f2a0 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 -- 172.21.15.61:0/2137695681 >> 172.21.15.61:0/2137695681 conn(0x7fc9400611c0 msgr2=0x7fc940062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 -- 172.21.15.61:0/2137695681 shutdown_connections 2022-07-20T03:08:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 -- 172.21.15.61:0/2137695681 wait complete. 2022-07-20T03:08:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 Processor -- start 2022-07-20T03:08:09.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.295+0000 7fc948087700 1 -- start start 2022-07-20T03:08:09.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9400aedc0 0x7fc9400a52e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 0x7fc940152bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400a71e0 0x7fc9400a8e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400a95f0 con 0x7fc9400b0080 2022-07-20T03:08:09.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9400a9890 con 0x7fc9400aedc0 2022-07-20T03:08:09.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9401530a0 con 0x7fc9400a71e0 2022-07-20T03:08:09.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 0x7fc940152bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 0x7fc940152bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43866/0 (socket says 172.21.15.61:43866) 2022-07-20T03:08:09.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 -- 172.21.15.61:0/565441125 learned_addr learned my addr 172.21.15.61:0/565441125 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:09.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945e23700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9400aedc0 0x7fc9400a52e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc946624700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400a71e0 0x7fc9400a8e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 -- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400a71e0 msgr2=0x7fc9400a8e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400a71e0 0x7fc9400a8e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 -- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9400aedc0 msgr2=0x7fc9400a52e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9400aedc0 0x7fc9400a52e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 -- 172.21.15.61:0/565441125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc93000b180 con 0x7fc9400b0080 2022-07-20T03:08:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc945622700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 0x7fc940152bb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc938001b80 tx=0x7fc930006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:09.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc930012890 con 0x7fc9400b0080 2022-07-20T03:08:09.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9300129f0 con 0x7fc9400b0080 2022-07-20T03:08:09.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9400a9300 con 0x7fc9400b0080 2022-07-20T03:08:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc930012d50 con 0x7fc9400b0080 2022-07-20T03:08:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9401519c0 con 0x7fc9400b0080 2022-07-20T03:08:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.299+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9400574e0 con 0x7fc9400b0080 2022-07-20T03:08:09.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.303+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc930024030 con 0x7fc9400b0080 2022-07-20T03:08:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.303+0000 7fc92effd700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc920053a40 0x7fc920055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:09.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.303+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc93006c700 con 0x7fc9400b0080 2022-07-20T03:08:09.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.303+0000 7fc945e23700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc920053a40 0x7fc920055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:09.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.303+0000 7fc945e23700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc920053a40 0x7fc920055f00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc93c00d000 tx=0x7fc93c007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:09.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.311+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc930043660 con 0x7fc9400b0080 2022-07-20T03:08:09.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.471+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc940062f80 con 0x7fc9400b0080 2022-07-20T03:08:09.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc92effd700 1 -- 172.21.15.61:0/565441125 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fc930022020 con 0x7fc9400b0080 2022-07-20T03:08:09.484 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:09.485 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:08:09.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc920053a40 msgr2=0x7fc920055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc920053a40 0x7fc920055f00 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fc93c00d000 tx=0x7fc93c007a00).stop 2022-07-20T03:08:09.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 msgr2=0x7fc940152bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:09.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 0x7fc940152bb0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fc938001b80 tx=0x7fc930006df0).stop 2022-07-20T03:08:09.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 shutdown_connections 2022-07-20T03:08:09.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc920053a40 0x7fc920055f00 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9400a71e0 0x7fc9400a8e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9400b0080 0x7fc940152bb0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 --2- 172.21.15.61:0/565441125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9400aedc0 0x7fc9400a52e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:09.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 >> 172.21.15.61:0/565441125 conn(0x7fc9400ac9d0 msgr2=0x7fc9400b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:09.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 shutdown_connections 2022-07-20T03:08:09.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.475+0000 7fc948087700 1 -- 172.21.15.61:0/565441125 wait complete. 2022-07-20T03:08:09.508 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:08:09.719 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:08:04.482490+0000', 'modified': '2022-07-20T03:08:06.488044+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4911': {'gid': 4911, 'name': 'b', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.117:6835/939788248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 939788248}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 939788248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:08:09.720 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:08:09.721 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:08:09.841 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":4911,"want_state":"up:active","state":"up:active","rank_uptime":5.3412692760000002,"mdsmap_epoch":14,"osdmap_epoch":33,"osdmap_epoch_barrier":33,"uptime":14.623623581} 2022-07-20T03:08:09.842 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 4911, "mdsmap_epoch": 14, "osdmap_epoch": 33, "osdmap_epoch_barrier": 33, "rank_uptime": 5.341269276, "state": "up:active", "uptime": 14.623623581, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:08:09.844 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:08:09.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 Processor -- start 2022-07-20T03:08:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 -- start start 2022-07-20T03:08:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f994c063d60 0x7f994c0b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f994c0b1440 0x7f994c0acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0af150 0x7f994c0acfd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c064140 con 0x7f994c063d60 2022-07-20T03:08:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f994c143800 con 0x7f994c0b1440 2022-07-20T03:08:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9951309700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c143aa0 con 0x7f994c0af150 2022-07-20T03:08:10.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f994c063d60 0x7f994c0b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f994c063d60 0x7f994c0b57e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43872/0 (socket says 172.21.15.61:43872) 2022-07-20T03:08:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 -- 172.21.15.61:0/2866450045 learned_addr learned my addr 172.21.15.61:0/2866450045 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9950b08700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0af150 0x7f994c0acfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f99437fe700 1 --2- 172.21.15.61:0/2866450045 >> v2:172.21.15.117:3300/0 conn(0x7f994c0b1440 0x7f994c0acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 -- 172.21.15.61:0/2866450045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0af150 msgr2=0x7f994c0acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0af150 0x7f994c0acfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 -- 172.21.15.61:0/2866450045 >> v2:172.21.15.117:3300/0 conn(0x7f994c0b1440 msgr2=0x7f994c0acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 --2- 172.21.15.61:0/2866450045 >> v2:172.21.15.117:3300/0 conn(0x7f994c0b1440 0x7f994c0acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 -- 172.21.15.61:0/2866450045 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f994c0b1050 con 0x7f994c063d60 2022-07-20T03:08:10.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f9950b08700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0af150 0x7f994c0acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:10.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994bfff700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c063d60 0x7f994c0b57e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f993c002f70 tx=0x7f993c006eb0).ready entity=mon.0 client_cookie=c8c1f016c5ee486b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.955+0000 7f994b7fe700 1 -- 172.21.15.61:0/2866450045 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993c00d570 con 0x7f994c063d60 2022-07-20T03:08:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f994b7fe700 1 -- 172.21.15.61:0/2866450045 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f993c000e90 con 0x7f994c063d60 2022-07-20T03:08:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f994b7fe700 1 -- 172.21.15.61:0/2866450045 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f993c00d900 con 0x7f994c063d60 2022-07-20T03:08:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 -- 172.21.15.61:0/2866450045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c063d60 msgr2=0x7f994c0b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c063d60 0x7f994c0b57e0 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f993c002f70 tx=0x7f993c006eb0).stop 2022-07-20T03:08:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 -- 172.21.15.61:0/2866450045 shutdown_connections 2022-07-20T03:08:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0af150 0x7f994c0acfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 --2- 172.21.15.61:0/2866450045 >> v2:172.21.15.117:3300/0 conn(0x7f994c0b1440 0x7f994c0acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 --2- 172.21.15.61:0/2866450045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c063d60 0x7f994c0b57e0 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 -- 172.21.15.61:0/2866450045 >> 172.21.15.61:0/2866450045 conn(0x7f994c0634e0 msgr2=0x7f994c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 -- 172.21.15.61:0/2866450045 shutdown_connections 2022-07-20T03:08:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 -- 172.21.15.61:0/2866450045 wait complete. 2022-07-20T03:08:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 Processor -- start 2022-07-20T03:08:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 -- start start 2022-07-20T03:08:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994c063d60 0x7f994c0b4330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.959+0000 7f9951309700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 0x7f994c066b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0b1440 0x7f994c067010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c066930 con 0x7f994c0af150 2022-07-20T03:08:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f994c13fcf0 con 0x7f994c063d60 2022-07-20T03:08:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f994c066690 con 0x7f994c0b1440 2022-07-20T03:08:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 0x7f994c066b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 0x7f994c066b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43880/0 (socket says 172.21.15.61:43880) 2022-07-20T03:08:10.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 -- 172.21.15.61:0/1114447892 learned_addr learned my addr 172.21.15.61:0/1114447892 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:10.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f994bfff700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994c063d60 0x7f994c0b4330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9950b08700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0b1440 0x7f994c067010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 -- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0b1440 msgr2=0x7f994c067010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0b1440 0x7f994c067010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 -- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994c063d60 msgr2=0x7f994c0b4330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994c063d60 0x7f994c0b4330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 -- 172.21.15.61:0/1114447892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993c00b1e0 con 0x7f994c0af150 2022-07-20T03:08:10.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f99437fe700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 0x7f994c066b20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f993400a000 tx=0x7f9934008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9934011590 con 0x7f994c0af150 2022-07-20T03:08:10.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99340116f0 con 0x7f994c0af150 2022-07-20T03:08:10.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f994c1456e0 con 0x7f994c0af150 2022-07-20T03:08:10.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9934011a50 con 0x7f994c0af150 2022-07-20T03:08:10.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f994c0658b0 con 0x7f994c0af150 2022-07-20T03:08:10.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f994c0b5cd0 con 0x7f994c0af150 2022-07-20T03:08:10.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.963+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9934011bb0 con 0x7f994c0af150 2022-07-20T03:08:10.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.967+0000 7f9949ffb700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9930053a40 0x7f9930055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.967+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f993406c280 con 0x7f994c0af150 2022-07-20T03:08:10.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.967+0000 7f994bfff700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9930053a40 0x7f9930055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.967+0000 7f994bfff700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9930053a40 0x7f9930055f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f993c01c000 tx=0x7f993c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:09.971+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99340430e0 con 0x7f994c0af150 2022-07-20T03:08:10.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.095+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f994c0574e0 con 0x7f994c0af150 2022-07-20T03:08:10.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9949ffb700 1 -- 172.21.15.61:0/1114447892 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v33) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f9934046700 con 0x7f994c0af150 2022-07-20T03:08:10.240 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:10.241 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":33,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:08:05.485261+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":5,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:08:02.776339+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"32","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":6,"pool_name":"cephfs_data","create_time":"2022-07-20T03:08:03.472472+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"33","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:08:10.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9930053a40 msgr2=0x7f9930055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9930053a40 0x7f9930055f00 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f993c01c000 tx=0x7f993c015000).stop 2022-07-20T03:08:10.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 msgr2=0x7f994c066b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 0x7f994c066b20 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f993400a000 tx=0x7f9934008040).stop 2022-07-20T03:08:10.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 shutdown_connections 2022-07-20T03:08:10.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9930053a40 0x7f9930055f00 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f994c0b1440 0x7f994c067010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f994c0af150 0x7f994c066b20 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 --2- 172.21.15.61:0/1114447892 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f994c063d60 0x7f994c0b4330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 >> 172.21.15.61:0/1114447892 conn(0x7f994c0634e0 msgr2=0x7f994c138350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:10.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 shutdown_connections 2022-07-20T03:08:10.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.099+0000 7f9951309700 1 -- 172.21.15.61:0/1114447892 wait complete. 2022-07-20T03:08:10.282 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 5, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:08:02.776339+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '32', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 6, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:08:03.472472+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '33', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:08:10.284 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:08:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 Processor -- start 2022-07-20T03:08:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 -- start start 2022-07-20T03:08:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f00d80b3a10 0x7f00d80b3df0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 0x7f00d81433f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d813c620 0x7f00d813ca90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d8143c90 con 0x7f00d8066930 2022-07-20T03:08:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00d80b42e0 con 0x7f00d80b3a10 2022-07-20T03:08:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00df1d4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d8142a10 con 0x7f00d813c620 2022-07-20T03:08:10.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.395+0000 7f00dd9d1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 0x7f00d81433f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f00d80b3a10 0x7f00d80b3df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de9d3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d813c620 0x7f00d813ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00dd9d1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 0x7f00d81433f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43886/0 (socket says 172.21.15.61:43886) 2022-07-20T03:08:10.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00dd9d1700 1 -- 172.21.15.61:0/3663924630 learned_addr learned my addr 172.21.15.61:0/3663924630 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:10.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 -- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d813c620 msgr2=0x7f00d813ca90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:10.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d813c620 0x7f00d813ca90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 -- 172.21.15.61:0/3663924630 >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 msgr2=0x7f00d81433f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:10.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3663924630 >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 0x7f00d81433f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 -- 172.21.15.61:0/3663924630 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00d8142cb0 con 0x7f00d80b3a10 2022-07-20T03:08:10.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00dd9d1700 1 --2- 172.21.15.61:0/3663924630 >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 0x7f00d81433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:10.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d80b3a10 0x7f00d80b3df0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f00d4002f70 tx=0x7f00d4006df0).ready entity=mon.1 client_cookie=2da65606d01b2526 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00dd1d0700 1 -- 172.21.15.61:0/3663924630 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00d400d4a0 con 0x7f00d80b3a10 2022-07-20T03:08:10.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00dd1d0700 1 -- 172.21.15.61:0/3663924630 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d4000e90 con 0x7f00d80b3a10 2022-07-20T03:08:10.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00dd1d0700 1 -- 172.21.15.61:0/3663924630 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00d400d7f0 con 0x7f00d80b3a10 2022-07-20T03:08:10.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 -- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d80b3a10 msgr2=0x7f00d80b3df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d80b3a10 0x7f00d80b3df0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f00d4002f70 tx=0x7f00d4006df0).stop 2022-07-20T03:08:10.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 -- 172.21.15.61:0/3663924630 shutdown_connections 2022-07-20T03:08:10.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d813c620 0x7f00d813ca90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3663924630 >> v2:172.21.15.61:3300/0 conn(0x7f00d8066930 0x7f00d81433f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3663924630 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d80b3a10 0x7f00d80b3df0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 -- 172.21.15.61:0/3663924630 >> 172.21.15.61:0/3663924630 conn(0x7f00d8064c50 msgr2=0x7f00d8065030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:10.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 -- 172.21.15.61:0/3663924630 shutdown_connections 2022-07-20T03:08:10.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 -- 172.21.15.61:0/3663924630 wait complete. 2022-07-20T03:08:10.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 Processor -- start 2022-07-20T03:08:10.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.399+0000 7f00df1d4700 1 -- start start 2022-07-20T03:08:10.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d8066930 0x7f00d8144e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 0x7f00d8155ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d813c620 0x7f00d8110d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d813e870 con 0x7f00d80b3a10 2022-07-20T03:08:10.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00d813eb10 con 0x7f00d813c620 2022-07-20T03:08:10.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00d813edb0 con 0x7f00d8066930 2022-07-20T03:08:10.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00dd9d1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 0x7f00d8155ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00dd9d1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 0x7f00d8155ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43894/0 (socket says 172.21.15.61:43894) 2022-07-20T03:08:10.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00dd9d1700 1 -- 172.21.15.61:0/3623295555 learned_addr learned my addr 172.21.15.61:0/3623295555 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:10.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d8066930 0x7f00d8144e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de9d3700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d813c620 0x7f00d8110d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 -- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d813c620 msgr2=0x7f00d8110d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d813c620 0x7f00d8110d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 -- 172.21.15.61:0/3623295555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 msgr2=0x7f00d8155ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 0x7f00d8155ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 -- 172.21.15.61:0/3623295555 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00c800b1e0 con 0x7f00d8066930 2022-07-20T03:08:10.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00dd9d1700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 0x7f00d8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:10.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00de1d2700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d8066930 0x7f00d8144e70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f00d4002f70 tx=0x7f00d4014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00d4013bf0 con 0x7f00d8066930 2022-07-20T03:08:10.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00d4013d50 con 0x7f00d8066930 2022-07-20T03:08:10.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00d400b180 con 0x7f00d8066930 2022-07-20T03:08:10.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00d401e5f0 con 0x7f00d8066930 2022-07-20T03:08:10.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00d813de60 con 0x7f00d8066930 2022-07-20T03:08:10.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00d8142f50 con 0x7f00d8066930 2022-07-20T03:08:10.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.403+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f00d4025030 con 0x7f00d8066930 2022-07-20T03:08:10.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.407+0000 7f00c77fe700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00b8053a40 0x7f00b8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.407+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f00d406cb40 con 0x7f00d8066930 2022-07-20T03:08:10.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.407+0000 7f00dd9d1700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00b8053a40 0x7f00b8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.407+0000 7f00dd9d1700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00b8053a40 0x7f00b8055f00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f00c800b000 tx=0x7f00c8006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.415+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f00d4043990 con 0x7f00d8066930 2022-07-20T03:08:10.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.543+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f00d80574e0 con 0x7f00d8066930 2022-07-20T03:08:10.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.543+0000 7f00c77fe700 1 -- 172.21.15.61:0/3623295555 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v33) v1 ==== 110+0+10 (secure 0 0 0) 0x7f00d4023070 con 0x7f00d8066930 2022-07-20T03:08:10.603 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:08:10.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00b8053a40 msgr2=0x7f00b8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00b8053a40 0x7f00b8055f00 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f00c800b000 tx=0x7f00c8006eb0).stop 2022-07-20T03:08:10.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d8066930 msgr2=0x7f00d8144e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d8066930 0x7f00d8144e70 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f00d4002f70 tx=0x7f00d4014000).stop 2022-07-20T03:08:10.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 shutdown_connections 2022-07-20T03:08:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00b8053a40 0x7f00b8055f00 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00d813c620 0x7f00d8110d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00d80b3a10 0x7f00d8155ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 --2- 172.21.15.61:0/3623295555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00d8066930 0x7f00d8144e70 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 >> 172.21.15.61:0/3623295555 conn(0x7f00d8064c50 msgr2=0x7f00d813bf10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 shutdown_connections 2022-07-20T03:08:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.547+0000 7f00df1d4700 1 -- 172.21.15.61:0/3623295555 wait complete. 2022-07-20T03:08:10.628 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:08:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7d0e7700 1 Processor -- start 2022-07-20T03:08:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7d0e7700 1 -- start start 2022-07-20T03:08:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7d0e7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b780b32a0 0x7f2b781468d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7d0e7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b780b3f10 con 0x7f2b780b2ec0 2022-07-20T03:08:10.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7d0e7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b780b2620 con 0x7f2b7805d160 2022-07-20T03:08:10.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7d0e7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b780b4ea0 con 0x7f2b780b32a0 2022-07-20T03:08:10.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7c8e6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b780b32a0 0x7f2b781468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b77fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2b780b2ec0 0x7f2b78142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40498/0 (socket says 172.21.15.61:40498) 2022-07-20T03:08:10.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b77fff700 1 -- 172.21.15.61:0/3584643788 learned_addr learned my addr 172.21.15.61:0/3584643788 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:10.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/3584643788 >> v1:172.21.15.117:6789/0 conn(0x7f2b7805d160 legacy=0x7f2b781461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/3584643788 >> v1:172.21.15.61:6789/0 conn(0x7f2b780b2ec0 legacy=0x7f2b78142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.739+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/3584643788 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b780b28c0 con 0x7f2b780b32a0 2022-07-20T03:08:10.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7c8e6700 1 --2- 172.21.15.61:0/3584643788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b781468d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f2b700032f0 tx=0x7f2b700091d0).ready entity=mon.0 client_cookie=5a0ff6e933f83efa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b76ffd700 1 -- 172.21.15.61:0/3584643788 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b70009ea0 con 0x7f2b780b32a0 2022-07-20T03:08:10.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b76ffd700 1 -- 172.21.15.61:0/3584643788 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7000b8a0 con 0x7f2b780b32a0 2022-07-20T03:08:10.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b76ffd700 1 -- 172.21.15.61:0/3584643788 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b7000ba10 con 0x7f2b780b32a0 2022-07-20T03:08:10.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/3584643788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 msgr2=0x7f2b781468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/3584643788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b781468d0 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7f2b700032f0 tx=0x7f2b700091d0).stop 2022-07-20T03:08:10.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/3584643788 shutdown_connections 2022-07-20T03:08:10.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/3584643788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b781468d0 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/3584643788 >> 172.21.15.61:0/3584643788 conn(0x7f2b78066e20 msgr2=0x7f2b78067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/3584643788 shutdown_connections 2022-07-20T03:08:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/3584643788 wait complete. 2022-07-20T03:08:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 Processor -- start 2022-07-20T03:08:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 -- start start 2022-07-20T03:08:10.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b7805d160 0x7f2b78112fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.743+0000 7f2b7d0e7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b780b2ec0 0x7f2b780ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b78155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b780ad930 con 0x7f2b780b32a0 2022-07-20T03:08:10.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b780ae790 con 0x7f2b780b2ec0 2022-07-20T03:08:10.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b78146c10 con 0x7f2b7805d160 2022-07-20T03:08:10.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b78155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b78155d10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43910/0 (socket says 172.21.15.61:43910) 2022-07-20T03:08:10.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/2531913246 learned_addr learned my addr 172.21.15.61:0/2531913246 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:10.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b777fe700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b780b2ec0 0x7f2b780ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b77fff700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b7805d160 0x7f2b78112fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b7805d160 msgr2=0x7f2b78112fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b7805d160 0x7f2b78112fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b780b2ec0 msgr2=0x7f2b780ae0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b780b2ec0 0x7f2b780ae0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 -- 172.21.15.61:0/2531913246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b70009470 con 0x7f2b780b32a0 2022-07-20T03:08:10.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7c8e6700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b78155d10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f2b700032f0 tx=0x7f2b70016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b70009ea0 con 0x7f2b780b32a0 2022-07-20T03:08:10.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b7000bec0 con 0x7f2b780b32a0 2022-07-20T03:08:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b780acff0 con 0x7f2b780b32a0 2022-07-20T03:08:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b70011ea0 con 0x7f2b780b32a0 2022-07-20T03:08:10.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b780acc80 con 0x7f2b780b32a0 2022-07-20T03:08:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b780ae5c0 con 0x7f2b780b32a0 2022-07-20T03:08:10.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.747+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b70026030 con 0x7f2b780b32a0 2022-07-20T03:08:10.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.751+0000 7f2b757fa700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b58053ad0 0x7f2b58055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:10.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.751+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2b7006cae0 con 0x7f2b780b32a0 2022-07-20T03:08:10.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.751+0000 7f2b77fff700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b58053ad0 0x7f2b58055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.751+0000 7f2b77fff700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b58053ad0 0x7f2b58055f90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f2b6c00d000 tx=0x7f2b6c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:10.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.755+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b70047470 con 0x7f2b780b32a0 2022-07-20T03:08:10.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f2b780b4a00 con 0x7f2b780b32a0 2022-07-20T03:08:10.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b757fa700 1 -- 172.21.15.61:0/2531913246 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v33) v1 ==== 92+0+10 (secure 0 0 0) 0x7f2b70015020 con 0x7f2b780b32a0 2022-07-20T03:08:10.886 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:08:10.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b58053ad0 msgr2=0x7f2b58055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b58053ad0 0x7f2b58055f90 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f2b6c00d000 tx=0x7f2b6c009040).stop 2022-07-20T03:08:10.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 msgr2=0x7f2b78155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:10.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b78155d10 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7f2b700032f0 tx=0x7f2b70016000).stop 2022-07-20T03:08:10.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 shutdown_connections 2022-07-20T03:08:10.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b58053ad0 0x7f2b58055f90 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b780b2ec0 0x7f2b780ae0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b780b32a0 0x7f2b78155d10 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 --2- 172.21.15.61:0/2531913246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b7805d160 0x7f2b78112fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:10.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 >> 172.21.15.61:0/2531913246 conn(0x7f2b78066e20 msgr2=0x7f2b781429c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:10.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 shutdown_connections 2022-07-20T03:08:10.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:10.879+0000 7f2b7d0e7700 1 -- 172.21.15.61:0/2531913246 wait complete. 2022-07-20T03:08:10.912 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:08:11.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.027+0000 7fb0e7fff700 1 Processor -- start 2022-07-20T03:08:11.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.027+0000 7fb0e7fff700 1 -- start start 2022-07-20T03:08:11.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e805c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0e8146160 0x7fb0e805cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0e80b1a00 0x7fb0e805d000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e805d4f0 con 0x7fb0e8146160 2022-07-20T03:08:11.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e8062180 con 0x7fb0e80b1a00 2022-07-20T03:08:11.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80ad100 con 0x7fb0e80acd20 2022-07-20T03:08:11.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0e8146160 0x7fb0e805cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0e8146160 0x7fb0e805cb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43916/0 (socket says 172.21.15.61:43916) 2022-07-20T03:08:11.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2360393610 learned_addr learned my addr 172.21.15.61:0/2360393610 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:11.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e6ffd700 1 --2- 172.21.15.61:0/2360393610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e805c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e77fe700 1 --2- 172.21.15.61:0/2360393610 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80b1a00 0x7fb0e805d000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2360393610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 msgr2=0x7fb0e805c620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:11.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/2360393610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e805c620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2360393610 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80b1a00 msgr2=0x7fb0e805d000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:11.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/2360393610 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80b1a00 0x7fb0e805d000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2360393610 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e81466e0 con 0x7fb0e8146160 2022-07-20T03:08:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/2360393610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e8146160 0x7fb0e805cb10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb0d0002f70 tx=0x7fb0d0006eb0).ready entity=mon.0 client_cookie=eab53629931ca762 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:11.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e67fc700 1 -- 172.21.15.61:0/2360393610 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0d000d570 con 0x7fb0e8146160 2022-07-20T03:08:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e67fc700 1 -- 172.21.15.61:0/2360393610 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0000e90 con 0x7fb0e8146160 2022-07-20T03:08:11.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e67fc700 1 -- 172.21.15.61:0/2360393610 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0d000d900 con 0x7fb0e8146160 2022-07-20T03:08:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- 172.21.15.61:0/2360393610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e8146160 msgr2=0x7fb0e805cb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/2360393610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e8146160 0x7fb0e805cb10 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb0d0002f70 tx=0x7fb0d0006eb0).stop 2022-07-20T03:08:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- 172.21.15.61:0/2360393610 shutdown_connections 2022-07-20T03:08:11.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/2360393610 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80b1a00 0x7fb0e805d000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/2360393610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e8146160 0x7fb0e805cb10 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/2360393610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e805c620 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- 172.21.15.61:0/2360393610 >> 172.21.15.61:0/2360393610 conn(0x7fb0e80b2af0 msgr2=0x7fb0e80b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:11.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- 172.21.15.61:0/2360393610 shutdown_connections 2022-07-20T03:08:11.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.031+0000 7fb0e7fff700 1 -- 172.21.15.61:0/2360393610 wait complete. 2022-07-20T03:08:11.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 Processor -- start 2022-07-20T03:08:11.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 -- start start 2022-07-20T03:08:11.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e8149dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 0x7fb0e8155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e8146160 0x7fb0e8152170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8099270 con 0x7fb0e80b1a00 2022-07-20T03:08:11.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8099510 con 0x7fb0e8146160 2022-07-20T03:08:11.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8151cd0 con 0x7fb0e80acd20 2022-07-20T03:08:11.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0de7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 0x7fb0e8155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0de7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 0x7fb0e8155d00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43920/0 (socket says 172.21.15.61:43920) 2022-07-20T03:08:11.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0de7fc700 1 -- 172.21.15.61:0/3988022683 learned_addr learned my addr 172.21.15.61:0/3988022683 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:11.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e6ffd700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e8149dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0e77fe700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e8146160 0x7fb0e8152170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0de7fc700 1 -- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 msgr2=0x7fb0e8149dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.035+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e8149dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0de7fc700 1 -- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e8146160 msgr2=0x7fb0e8152170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e8146160 0x7fb0e8152170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0de7fc700 1 -- 172.21.15.61:0/3988022683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0d000b1e0 con 0x7fb0e80b1a00 2022-07-20T03:08:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e77fe700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e8146160 0x7fb0e8152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 0x7fb0e8155d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb0d0002f70 tx=0x7fb0d0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0d0011950 con 0x7fb0e80b1a00 2022-07-20T03:08:11.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e809b0d0 con 0x7fb0e80b1a00 2022-07-20T03:08:11.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0d0011ab0 con 0x7fb0e80b1a00 2022-07-20T03:08:11.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e8152660 con 0x7fb0e80b1a00 2022-07-20T03:08:11.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0d0011e10 con 0x7fb0e80b1a00 2022-07-20T03:08:11.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e8152cb0 con 0x7fb0e80b1a00 2022-07-20T03:08:11.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb0d001a040 con 0x7fb0e80b1a00 2022-07-20T03:08:11.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e4ff9700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0cc053a40 0x7fb0cc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.039+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb0d006cd90 con 0x7fb0e80b1a00 2022-07-20T03:08:11.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.043+0000 7fb0e6ffd700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0cc053a40 0x7fb0cc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.043+0000 7fb0e6ffd700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0cc053a40 0x7fb0cc055f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb0d800b000 tx=0x7fb0d8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.043+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0d0043c50 con 0x7fb0e80b1a00 2022-07-20T03:08:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb0e8146980 con 0x7fb0e80b1a00 2022-07-20T03:08:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e4ff9700 1 -- 172.21.15.61:0/3988022683 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v33) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb0d0023020 con 0x7fb0e80b1a00 2022-07-20T03:08:11.606 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:08:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0cc053a40 msgr2=0x7fb0cc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0cc053a40 0x7fb0cc055f00 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb0d800b000 tx=0x7fb0d8006df0).stop 2022-07-20T03:08:11.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 msgr2=0x7fb0e8155d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 0x7fb0e8155d00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7fb0d0002f70 tx=0x7fb0d0015000).stop 2022-07-20T03:08:11.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 shutdown_connections 2022-07-20T03:08:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0cc053a40 0x7fb0cc055f00 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e8146160 0x7fb0e8152170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b1a00 0x7fb0e8155d00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 --2- 172.21.15.61:0/3988022683 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80acd20 0x7fb0e8149dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 >> 172.21.15.61:0/3988022683 conn(0x7fb0e80b2af0 msgr2=0x7fb0e80b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 shutdown_connections 2022-07-20T03:08:11.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.171+0000 7fb0e7fff700 1 -- 172.21.15.61:0/3988022683 wait complete. 2022-07-20T03:08:11.628 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:08:11.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 Processor -- start 2022-07-20T03:08:11.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- start start 2022-07-20T03:08:11.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0d1405cbb0 0x7f0d140b59d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d1414a6c0 0x7f0d14146530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d140b3f30 con 0x7f0d14098160 2022-07-20T03:08:11.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d140b4a10 con 0x7f0d1405cbb0 2022-07-20T03:08:11.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14098540 con 0x7f0d1414a6c0 2022-07-20T03:08:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0d1405cbb0 0x7f0d140b59d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0d1405cbb0 0x7f0d140b59d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43930/0 (socket says 172.21.15.61:43930) 2022-07-20T03:08:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 -- 172.21.15.61:0/1824264020 learned_addr learned my addr 172.21.15.61:0/1824264020 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d137fe700 1 --2- 172.21.15.61:0/1824264020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d1414a6c0 0x7f0d14146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 -- 172.21.15.61:0/1824264020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d1414a6c0 msgr2=0x7f0d14146530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 --2- 172.21.15.61:0/1824264020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d1414a6c0 0x7f0d14146530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 -- 172.21.15.61:0/1824264020 >> v1:172.21.15.117:6789/0 conn(0x7f0d14098160 legacy=0x7f0d1405c110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 -- 172.21.15.61:0/1824264020 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d140b4160 con 0x7f0d1405cbb0 2022-07-20T03:08:11.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d127fc700 1 --2- 172.21.15.61:0/1824264020 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b59d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f0cfc002f70 tx=0x7f0cfc006eb0).ready entity=mon.0 client_cookie=7f18e3a0f9aefa27 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:11.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d11ffb700 1 -- 172.21.15.61:0/1824264020 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cfc00d5c0 con 0x7f0d1405cbb0 2022-07-20T03:08:11.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d11ffb700 1 -- 172.21.15.61:0/1824264020 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cfc000e90 con 0x7f0d1405cbb0 2022-07-20T03:08:11.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d11ffb700 1 -- 172.21.15.61:0/1824264020 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cfc00d950 con 0x7f0d1405cbb0 2022-07-20T03:08:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- 172.21.15.61:0/1824264020 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1405cbb0 msgr2=0x7f0d140b59d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 --2- 172.21.15.61:0/1824264020 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b59d0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f0cfc002f70 tx=0x7f0cfc006eb0).stop 2022-07-20T03:08:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- 172.21.15.61:0/1824264020 shutdown_connections 2022-07-20T03:08:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 --2- 172.21.15.61:0/1824264020 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d1414a6c0 0x7f0d14146530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 --2- 172.21.15.61:0/1824264020 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b59d0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- 172.21.15.61:0/1824264020 >> 172.21.15.61:0/1824264020 conn(0x7f0d140b5f00 msgr2=0x7f0d140b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- 172.21.15.61:0/1824264020 shutdown_connections 2022-07-20T03:08:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.743+0000 7f0d1951b700 1 -- 172.21.15.61:0/1824264020 wait complete. 2022-07-20T03:08:11.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 Processor -- start 2022-07-20T03:08:11.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 -- start start 2022-07-20T03:08:11.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b08a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d14098160 0x7f0d14148f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 0x7f0d140b2d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140b0ed0 con 0x7f0d1414a6c0 2022-07-20T03:08:11.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d140b1170 con 0x7f0d1405cbb0 2022-07-20T03:08:11.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d1951b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d14159ac0 con 0x7f0d14098160 2022-07-20T03:08:11.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 0x7f0d140b2d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 0x7f0d140b2d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43938/0 (socket says 172.21.15.61:43938) 2022-07-20T03:08:11.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 -- 172.21.15.61:0/192620480 learned_addr learned my addr 172.21.15.61:0/192620480 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:11.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d12ffd700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b08a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d127fc700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d14098160 0x7f0d14148f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 -- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d14098160 msgr2=0x7f0d14148f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d14098160 0x7f0d14148f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 -- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d1405cbb0 msgr2=0x7f0d140b08a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:11.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b08a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:11.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 -- 172.21.15.61:0/192620480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cfc00b1e0 con 0x7f0d1414a6c0 2022-07-20T03:08:11.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.747+0000 7f0d137fe700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 0x7f0d140b2d20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f0d0c00a000 tx=0x7f0d0c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:11.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d0c013620 con 0x7f0d1414a6c0 2022-07-20T03:08:11.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d14156440 con 0x7f0d1414a6c0 2022-07-20T03:08:11.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d0c013780 con 0x7f0d1414a6c0 2022-07-20T03:08:11.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d14156910 con 0x7f0d1414a6c0 2022-07-20T03:08:11.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d0c013ae0 con 0x7f0d1414a6c0 2022-07-20T03:08:11.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d14159c80 con 0x7f0d1414a6c0 2022-07-20T03:08:11.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0d0c013c40 con 0x7f0d1414a6c0 2022-07-20T03:08:11.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d0bfff700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cf8053af0 0x7f0cf8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:11.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d0c06c090 con 0x7f0d1414a6c0 2022-07-20T03:08:11.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.751+0000 7f0d12ffd700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cf8053af0 0x7f0cf8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:11.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.755+0000 7f0d12ffd700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cf8053af0 0x7f0cf8055fb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0d0400d000 tx=0x7f0d04007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:11.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.759+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d0c042f50 con 0x7f0d1414a6c0 2022-07-20T03:08:11.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0d140b37f0 con 0x7f0d1414a6c0 2022-07-20T03:08:11.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d0bfff700 1 -- 172.21.15.61:0/192620480 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v33) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0d0c00f040 con 0x7f0d1414a6c0 2022-07-20T03:08:11.983 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:08:12.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cf8053af0 msgr2=0x7f0cf8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cf8053af0 0x7f0cf8055fb0 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f0d0400d000 tx=0x7f0d04007b90).stop 2022-07-20T03:08:12.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 msgr2=0x7f0d140b2d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 0x7f0d140b2d20 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f0d0c00a000 tx=0x7f0d0c008040).stop 2022-07-20T03:08:12.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 shutdown_connections 2022-07-20T03:08:12.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cf8053af0 0x7f0cf8055fb0 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d1414a6c0 0x7f0d140b2d20 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d14098160 0x7f0d14148f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 --2- 172.21.15.61:0/192620480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d1405cbb0 0x7f0d140b08a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 >> 172.21.15.61:0/192620480 conn(0x7f0d140b5f00 msgr2=0x7f0d140af2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:12.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 shutdown_connections 2022-07-20T03:08:12.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:11.887+0000 7f0d1951b700 1 -- 172.21.15.61:0/192620480 wait complete. 2022-07-20T03:08:12.064 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:08:12.064 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:08:12.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.183+0000 7f0447fff700 1 Processor -- start 2022-07-20T03:08:12.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.183+0000 7f0447fff700 1 -- start start 2022-07-20T03:08:12.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0447fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f04480b7b70 0x7f0448146860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:12.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0447fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f04480b1520 0x7f0448066870 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:12.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0447fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f04480b7f50 con 0x7f0448066d60 2022-07-20T03:08:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0447fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0448146120 con 0x7f04480b7b70 2022-07-20T03:08:12.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0447fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f04481463c0 con 0x7f04480b1520 2022-07-20T03:08:12.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04477fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0448066d60 0x7f044812aad0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40544/0 (socket says 172.21.15.61:40544) 2022-07-20T03:08:12.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04477fe700 1 -- 172.21.15.61:0/642235039 learned_addr learned my addr 172.21.15.61:0/642235039 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:12.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0446ffd700 1 --2- 172.21.15.61:0/642235039 >> v2:172.21.15.61:3300/0 conn(0x7f04480b7b70 0x7f0448146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:12.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f043ffff700 1 --2- 172.21.15.61:0/642235039 >> v2:172.21.15.117:3300/0 conn(0x7f04480b1520 0x7f0448066870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04467fc700 1 -- 172.21.15.61:0/642235039 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1119445623 0 0) 0x7f04480b7f50 con 0x7f0448066d60 2022-07-20T03:08:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04467fc700 1 -- 172.21.15.61:0/642235039 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0428003260 con 0x7f0448066d60 2022-07-20T03:08:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04467fc700 1 -- 172.21.15.61:0/642235039 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 319666053 0 0) 0x7f0428003260 con 0x7f0448066d60 2022-07-20T03:08:12.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0446ffd700 1 -- 172.21.15.61:0/642235039 >> v2:172.21.15.117:3300/0 conn(0x7f04480b1520 msgr2=0x7f0448066870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:12.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0446ffd700 1 --2- 172.21.15.61:0/642235039 >> v2:172.21.15.117:3300/0 conn(0x7f04480b1520 0x7f0448066870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0446ffd700 1 -- 172.21.15.61:0/642235039 >> v1:172.21.15.61:6789/0 conn(0x7f0448066d60 legacy=0x7f044812aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0446ffd700 1 -- 172.21.15.61:0/642235039 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0448146d50 con 0x7f04480b7b70 2022-07-20T03:08:12.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f0446ffd700 1 --2- 172.21.15.61:0/642235039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b7b70 0x7f0448146860 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f0440001b80 tx=0x7f0438006df0).ready entity=mon.0 client_cookie=4f8e748864d0e129 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:12.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04467fc700 1 -- 172.21.15.61:0/642235039 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f04400019e0 con 0x7f04480b7b70 2022-07-20T03:08:12.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04467fc700 1 -- 172.21.15.61:0/642235039 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0438000e90 con 0x7f04480b7b70 2022-07-20T03:08:12.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.187+0000 7f04467fc700 1 -- 172.21.15.61:0/642235039 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0438012460 con 0x7f04480b7b70 2022-07-20T03:08:12.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 -- 172.21.15.61:0/642235039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b7b70 msgr2=0x7f0448146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 --2- 172.21.15.61:0/642235039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b7b70 0x7f0448146860 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f0440001b80 tx=0x7f0438006df0).stop 2022-07-20T03:08:12.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 -- 172.21.15.61:0/642235039 shutdown_connections 2022-07-20T03:08:12.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 --2- 172.21.15.61:0/642235039 >> v2:172.21.15.117:3300/0 conn(0x7f04480b1520 0x7f0448066870 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 --2- 172.21.15.61:0/642235039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b7b70 0x7f0448146860 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 -- 172.21.15.61:0/642235039 >> 172.21.15.61:0/642235039 conn(0x7f04480656b0 msgr2=0x7f04480b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:12.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 -- 172.21.15.61:0/642235039 shutdown_connections 2022-07-20T03:08:12.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 -- 172.21.15.61:0/642235039 wait complete. 2022-07-20T03:08:12.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 Processor -- start 2022-07-20T03:08:12.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.191+0000 7f0447fff700 1 -- start start 2022-07-20T03:08:12.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0448066d60 0x7f04480b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:12.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 0x7f0448141b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:12.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f04480b7b70 0x7f0448142030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:12.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448143ea0 con 0x7f04480b1520 2022-07-20T03:08:12.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0448144140 con 0x7f0448066d60 2022-07-20T03:08:12.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f04481443e0 con 0x7f04480b7b70 2022-07-20T03:08:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 0x7f0448141b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 0x7f0448141b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43950/0 (socket says 172.21.15.61:43950) 2022-07-20T03:08:12.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 -- 172.21.15.61:0/123102538 learned_addr learned my addr 172.21.15.61:0/123102538 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:12.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0446ffd700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0448066d60 0x7f04480b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:12.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f04477fe700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f04480b7b70 0x7f0448142030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:12.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 -- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f04480b7b70 msgr2=0x7f0448142030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f04480b7b70 0x7f0448142030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 -- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0448066d60 msgr2=0x7f04480b5290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0448066d60 0x7f04480b5290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 -- 172.21.15.61:0/123102538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f043800b180 con 0x7f04480b1520 2022-07-20T03:08:12.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f04477fe700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f04480b7b70 0x7f0448142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:08:12.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0446ffd700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0448066d60 0x7f04480b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:08:12.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f043ffff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 0x7f0448141b40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f043000c000 tx=0x7f0430006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:12.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0430010590 con 0x7f04480b1520 2022-07-20T03:08:12.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f04300106f0 con 0x7f04480b1520 2022-07-20T03:08:12.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0448144680 con 0x7f04480b1520 2022-07-20T03:08:12.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0430010a50 con 0x7f04480b1520 2022-07-20T03:08:12.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0448144b20 con 0x7f04480b1520 2022-07-20T03:08:12.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.195+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f044812d5c0 con 0x7f04480b1520 2022-07-20T03:08:12.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.199+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0430010bb0 con 0x7f04480b1520 2022-07-20T03:08:12.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.199+0000 7f0444ff9700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f042c053ad0 0x7f042c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:12.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.199+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f043006c000 con 0x7f04480b1520 2022-07-20T03:08:12.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.199+0000 7f0446ffd700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f042c053ad0 0x7f042c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:12.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.199+0000 7f0446ffd700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f042c053ad0 0x7f042c055f90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0438015000 tx=0x7f0438014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:12.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.203+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0430042f60 con 0x7f04480b1520 2022-07-20T03:08:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.363+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0448144810 con 0x7f04480b1520 2022-07-20T03:08:12.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.363+0000 7f0444ff9700 1 -- 172.21.15.61:0/123102538 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f0430046580 con 0x7f04480b1520 2022-07-20T03:08:12.435 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:12.436 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:08:12.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f042c053ad0 msgr2=0x7f042c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f042c053ad0 0x7f042c055f90 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f0438015000 tx=0x7f0438014000).stop 2022-07-20T03:08:12.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 msgr2=0x7f0448141b40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:12.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 0x7f0448141b40 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f043000c000 tx=0x7f0430006df0).stop 2022-07-20T03:08:12.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 shutdown_connections 2022-07-20T03:08:12.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f042c053ad0 0x7f042c055f90 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f04480b7b70 0x7f0448142030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f04480b1520 0x7f0448141b40 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 --2- 172.21.15.61:0/123102538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0448066d60 0x7f04480b5290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:12.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 >> 172.21.15.61:0/123102538 conn(0x7f04480656b0 msgr2=0x7f0448142a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:12.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 shutdown_connections 2022-07-20T03:08:12.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:12.367+0000 7f0447fff700 1 -- 172.21.15.61:0/123102538 wait complete. 2022-07-20T03:08:12.464 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:08:12.682 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 14, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:08:04.482490+0000', 'modified': '2022-07-20T03:08:06.488044+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 4911}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_4911': {'gid': 4911, 'name': 'b', 'rank': 0, 'incarnation': 12, 'state': 'up:active', 'state_seq': 4, 'addr': '172.21.15.117:6835/939788248', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 939788248}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 939788248}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [6], 'metadata_pool': 5, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:08:12.683 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:08:12.684 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:08:12.823 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":4911,"want_state":"up:active","state":"up:active","rank_uptime":8.3122975910000001,"mdsmap_epoch":14,"osdmap_epoch":33,"osdmap_epoch_barrier":33,"uptime":17.594659676999999} 2022-07-20T03:08:12.824 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 4911, "mdsmap_epoch": 14, "osdmap_epoch": 33, "osdmap_epoch_barrier": 33, "rank_uptime": 8.312297591, "state": "up:active", "uptime": 17.594659677, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:08:12.825 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:08:12.825 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:08:12.828 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:08:12.829 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:12.831 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:08:12.864 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:08:12.925 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:08:12.925 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:08:12.951 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:08:12.952 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:08:12.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:12.968 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:08:13.045 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:08:13.059 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:08:13.083 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:08:13.159 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:08:13.152+0000 7f487b833080 -1 init, newargv = 0x55ff94b852a0 newargc=9ceph-fuse[22377]: starting ceph client 2022-07-20T03:08:13.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr: 2022-07-20T03:08:13.184 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:08:13.185 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:08:13.212 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22377]: starting fuse 2022-07-20T03:08:13.241 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:08:13.242 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:08:13.244 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:13.260 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:08:13.291 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:08:13.292 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:08:13.294 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:13.295 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:08:13.296 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:08:13.297 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:08:13.298 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:08:13.310 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:08:13.311 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.312 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:08:13.322 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:08:13.323 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:08:13.323 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.324 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:08:13.324 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:08:13.324 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:08:13.325 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:08:13.325 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.334 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:08:13.335 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:08:13.335 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:08:13.336 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:08:13.336 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:08:13.337 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.337 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:08:13.337 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:08:13.385 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22377.asok 2022-07-20T03:08:13.390 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22377.asok 2022-07-20T03:08:13.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:13.392 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22377.asok status 2022-07-20T03:08:13.489 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:08:13.490 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:08:13.490 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:08:13.490 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:08:13.491 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:08:13.491 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:08:13.492 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:08:13.492 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22377", 2022-07-20T03:08:13.492 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:08:13.493 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:08:13.493 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:08:13.494 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:08:13.499 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5115, 2022-07-20T03:08:13.499 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:08:13.502 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:08:13.503 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:08:13.503 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5115 2022-07-20T03:08:13.503 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:08:13.504 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:08:13.504 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:08:13.505 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:08:13.505 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2396892433 2022-07-20T03:08:13.506 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:08:13.506 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:08:13.507 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:08:13.507 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:08:13.507 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:08:13.508 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2396892433 2022-07-20T03:08:13.509 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:08:13.509 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5115 172.21.15.124:0/2396892433", 2022-07-20T03:08:13.515 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/2396892433", 2022-07-20T03:08:13.524 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:08:13.525 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 14, 2022-07-20T03:08:13.526 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 33, 2022-07-20T03:08:13.526 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:08:13.527 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:08:13.527 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:08:13.527 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:08:13.531 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:13.532 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:08:13.537 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:08:13.559 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:08:13.559 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:08:13.560 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:13.560 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:08:13.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mon.a.asok --format=json config get mds_min_caps_per_client 2022-07-20T03:08:13.738 INFO:teuthology.orchestra.run.smithi061.stdout:{"mds_min_caps_per_client":"100"} 2022-07-20T03:08:13.751 DEBUG:tasks.cephfs.filesystem:_json_asok output { "mds_min_caps_per_client": "100" } 2022-07-20T03:08:13.752 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:08:13.753 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:08:13.759 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:08:13.794 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:08:13.795 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:08:13.796 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:08:13.797 DEBUG:teuthology.orchestra.run.smithi124:> import sys 2022-07-20T03:08:13.798 DEBUG:teuthology.orchestra.run.smithi124:> import time 2022-07-20T03:08:13.800 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:08:13.801 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.801 DEBUG:teuthology.orchestra.run.smithi124:> n = 1000 2022-07-20T03:08:13.802 DEBUG:teuthology.orchestra.run.smithi124:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T03:08:13.803 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.804 DEBUG:teuthology.orchestra.run.smithi124:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T03:08:13.804 DEBUG:teuthology.orchestra.run.smithi124:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T03:08:13.805 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:08:13.806 DEBUG:teuthology.orchestra.run.smithi124:> for i in range(0, n): 2022-07-20T03:08:13.807 DEBUG:teuthology.orchestra.run.smithi124:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T03:08:13.826 DEBUG:teuthology.orchestra.run.smithi124:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T03:08:13.827 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content'"'"') 2022-07-20T03:08:13.828 DEBUG:teuthology.orchestra.run.smithi124:> if True: 2022-07-20T03:08:13.828 DEBUG:teuthology.orchestra.run.smithi124:> f.flush() 2022-07-20T03:08:13.829 DEBUG:teuthology.orchestra.run.smithi124:> os.fsync(f.fileno()) 2022-07-20T03:08:13.837 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:08:22.922 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:08:23.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 Processor -- start 2022-07-20T03:08:23.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 -- start start 2022-07-20T03:08:23.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f98cc13a5b0 0x7f98cc13a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f98cc0b20c0 0x7f98cc05d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f98cc0b4410 con 0x7f98cc140810 2022-07-20T03:08:23.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc141ac0 con 0x7f98cc0b20c0 2022-07-20T03:08:23.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d2393700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f98cc0b4170 con 0x7f98cc13a5b0 2022-07-20T03:08:23.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d1b92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f98cc0b20c0 0x7f98cc05d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d1b92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f98cc0b20c0 0x7f98cc05d0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43960/0 (socket says 172.21.15.61:43960) 2022-07-20T03:08:23.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d1b92700 1 -- 172.21.15.61:0/13387055 learned_addr learned my addr 172.21.15.61:0/13387055 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:23.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.027+0000 7f98d1391700 1 --2- 172.21.15.61:0/13387055 >> v2:172.21.15.117:3300/0 conn(0x7f98cc13a5b0 0x7f98cc13a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d1b92700 1 -- 172.21.15.61:0/13387055 >> v2:172.21.15.117:3300/0 conn(0x7f98cc13a5b0 msgr2=0x7f98cc13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:23.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d1b92700 1 --2- 172.21.15.61:0/13387055 >> v2:172.21.15.117:3300/0 conn(0x7f98cc13a5b0 0x7f98cc13a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d1b92700 1 -- 172.21.15.61:0/13387055 >> v1:172.21.15.117:6789/0 conn(0x7f98cc140810 legacy=0x7f98cc0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98c3fff700 1 -- 172.21.15.61:0/13387055 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 118679323 0 0) 0x7f98cc0b4410 con 0x7f98cc140810 2022-07-20T03:08:23.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d1b92700 1 -- 172.21.15.61:0/13387055 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98cc05bfc0 con 0x7f98cc0b20c0 2022-07-20T03:08:23.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d1b92700 1 --2- 172.21.15.61:0/13387055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc0b20c0 0x7f98cc05d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f98cc0ae0f0 tx=0x7f98c4009290).ready entity=mon.0 client_cookie=b95f1a9a401d8b7f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:23.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98c3fff700 1 -- 172.21.15.61:0/13387055 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98c400b860 con 0x7f98cc0b20c0 2022-07-20T03:08:23.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98c3fff700 1 -- 172.21.15.61:0/13387055 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c400b9c0 con 0x7f98cc0b20c0 2022-07-20T03:08:23.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98c3fff700 1 -- 172.21.15.61:0/13387055 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98c400bb30 con 0x7f98cc0b20c0 2022-07-20T03:08:23.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 -- 172.21.15.61:0/13387055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc0b20c0 msgr2=0x7f98cc05d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 --2- 172.21.15.61:0/13387055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc0b20c0 0x7f98cc05d0c0 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f98cc0ae0f0 tx=0x7f98c4009290).stop 2022-07-20T03:08:23.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 -- 172.21.15.61:0/13387055 shutdown_connections 2022-07-20T03:08:23.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 --2- 172.21.15.61:0/13387055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc0b20c0 0x7f98cc05d0c0 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 --2- 172.21.15.61:0/13387055 >> v2:172.21.15.117:3300/0 conn(0x7f98cc13a5b0 0x7f98cc13a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 -- 172.21.15.61:0/13387055 >> 172.21.15.61:0/13387055 conn(0x7f98cc064be0 msgr2=0x7f98cc064fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:23.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 -- 172.21.15.61:0/13387055 shutdown_connections 2022-07-20T03:08:23.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.031+0000 7f98d2393700 1 -- 172.21.15.61:0/13387055 wait complete. 2022-07-20T03:08:23.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 Processor -- start 2022-07-20T03:08:23.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 -- start start 2022-07-20T03:08:23.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98cc0b20c0 0x7f98cc13b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98cc13a5b0 0x7f98cc13be90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 0x7f98cc13d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0b1400 con 0x7f98cc140810 2022-07-20T03:08:23.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc150780 con 0x7f98cc0b20c0 2022-07-20T03:08:23.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d2393700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f98cc0b5580 con 0x7f98cc13a5b0 2022-07-20T03:08:23.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d1b92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 0x7f98cc13d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d1b92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 0x7f98cc13d900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43966/0 (socket says 172.21.15.61:43966) 2022-07-20T03:08:23.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d1b92700 1 -- 172.21.15.61:0/2686313729 learned_addr learned my addr 172.21.15.61:0/2686313729 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:23.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d1391700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98cc0b20c0 0x7f98cc13b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.035+0000 7f98d0b90700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98cc13a5b0 0x7f98cc13be90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d1b92700 1 -- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98cc13a5b0 msgr2=0x7f98cc13be90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d1b92700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98cc13a5b0 0x7f98cc13be90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d1b92700 1 -- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98cc0b20c0 msgr2=0x7f98cc13b9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d1b92700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98cc0b20c0 0x7f98cc13b9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d1b92700 1 -- 172.21.15.61:0/2686313729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c4009530 con 0x7f98cc140810 2022-07-20T03:08:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d1b92700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 0x7f98cc13d900 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f98cc0ae0f0 tx=0x7f98c4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98c4021630 con 0x7f98cc140810 2022-07-20T03:08:23.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f98c4021790 con 0x7f98cc140810 2022-07-20T03:08:23.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f98c4021af0 con 0x7f98cc140810 2022-07-20T03:08:23.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98cc0b5a20 con 0x7f98cc140810 2022-07-20T03:08:23.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98cc0b5df0 con 0x7f98cc140810 2022-07-20T03:08:23.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98cc1505f0 con 0x7f98cc140810 2022-07-20T03:08:23.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.039+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f98c400f070 con 0x7f98cc140810 2022-07-20T03:08:23.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.043+0000 7f98c27fc700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f98b4053ad0 0x7f98b4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.043+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f98c406d490 con 0x7f98cc140810 2022-07-20T03:08:23.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.043+0000 7f98d1391700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f98b4053ad0 0x7f98b4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.043+0000 7f98d1391700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f98b4053ad0 0x7f98b4055f90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f98bc00b000 tx=0x7f98bc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:23.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.047+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f98c40443f0 con 0x7f98cc140810 2022-07-20T03:08:23.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.211+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f98cc0b5740 con 0x7f98cc140810 2022-07-20T03:08:23.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.211+0000 7f98c27fc700 1 -- 172.21.15.61:0/2686313729 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f98c4015070 con 0x7f98cc140810 2022-07-20T03:08:23.266 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:23.267 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:08:23.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f98b4053ad0 msgr2=0x7f98b4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f98b4053ad0 0x7f98b4055f90 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f98bc00b000 tx=0x7f98bc006df0).stop 2022-07-20T03:08:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 msgr2=0x7f98cc13d900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 0x7f98cc13d900 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f98cc0ae0f0 tx=0x7f98c4016000).stop 2022-07-20T03:08:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 shutdown_connections 2022-07-20T03:08:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f98b4053ad0 0x7f98b4055f90 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f98cc140810 0x7f98cc13d900 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f98cc13a5b0 0x7f98cc13be90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 --2- 172.21.15.61:0/2686313729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f98cc0b20c0 0x7f98cc13b9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 >> 172.21.15.61:0/2686313729 conn(0x7f98cc064be0 msgr2=0x7f98cc05c4b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:23.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 shutdown_connections 2022-07-20T03:08:23.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.215+0000 7f98d2393700 1 -- 172.21.15.61:0/2686313729 wait complete. 2022-07-20T03:08:23.388 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:08:23.389 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_caps 20 2022-07-20T03:08:23.503 INFO:teuthology.orchestra.run.smithi117.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T03:08:23.503 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T03:08:23.504 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:08:23.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 Processor -- start 2022-07-20T03:08:23.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 -- start start 2022-07-20T03:08:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc5dc0b5650 0x7fc5dc0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5dc0af8e0 0x7fc5dc05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5dc147010 con 0x7fc5dc0adff0 2022-07-20T03:08:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5dc1472b0 con 0x7fc5dc0af8e0 2022-07-20T03:08:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5db59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5dc146d40 con 0x7fc5dc0b5650 2022-07-20T03:08:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5dad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc5dc0af8e0 0x7fc5dc05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5d9d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc5dc0adff0 0x7fc5dc147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40568/0 (socket says 172.21.15.61:40568) 2022-07-20T03:08:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5d9d9b700 1 -- 172.21.15.61:0/524661552 learned_addr learned my addr 172.21.15.61:0/524661552 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:23.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.639+0000 7fc5da59c700 1 --2- 172.21.15.61:0/524661552 >> v2:172.21.15.117:3300/0 conn(0x7fc5dc0b5650 0x7fc5dc0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5d959a700 1 -- 172.21.15.61:0/524661552 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 723991590 0 0) 0x7fc5dc147010 con 0x7fc5dc0adff0 2022-07-20T03:08:23.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5d959a700 1 -- 172.21.15.61:0/524661552 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5c0003260 con 0x7fc5dc0adff0 2022-07-20T03:08:23.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5dad9d700 1 -- 172.21.15.61:0/524661552 >> v2:172.21.15.117:3300/0 conn(0x7fc5dc0b5650 msgr2=0x7fc5dc0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:23.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5dad9d700 1 --2- 172.21.15.61:0/524661552 >> v2:172.21.15.117:3300/0 conn(0x7fc5dc0b5650 0x7fc5dc0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5dad9d700 1 -- 172.21.15.61:0/524661552 >> v1:172.21.15.61:6789/0 conn(0x7fc5dc0adff0 legacy=0x7fc5dc147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5dad9d700 1 -- 172.21.15.61:0/524661552 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5dc0acae0 con 0x7fc5dc0af8e0 2022-07-20T03:08:23.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5dad9d700 1 --2- 172.21.15.61:0/524661552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0af8e0 0x7fc5dc05bd90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc5d40032f0 tx=0x7fc5d40091d0).ready entity=mon.0 client_cookie=aa65ba4b4fbef6a7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:23.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5d959a700 1 -- 172.21.15.61:0/524661552 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d400b600 con 0x7fc5dc0af8e0 2022-07-20T03:08:23.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5d959a700 1 -- 172.21.15.61:0/524661552 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d400b760 con 0x7fc5dc0af8e0 2022-07-20T03:08:23.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5d959a700 1 -- 172.21.15.61:0/524661552 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d400b8d0 con 0x7fc5dc0af8e0 2022-07-20T03:08:23.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 -- 172.21.15.61:0/524661552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0af8e0 msgr2=0x7fc5dc05bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 --2- 172.21.15.61:0/524661552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0af8e0 0x7fc5dc05bd90 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc5d40032f0 tx=0x7fc5d40091d0).stop 2022-07-20T03:08:23.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 -- 172.21.15.61:0/524661552 shutdown_connections 2022-07-20T03:08:23.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 --2- 172.21.15.61:0/524661552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0af8e0 0x7fc5dc05bd90 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 --2- 172.21.15.61:0/524661552 >> v2:172.21.15.117:3300/0 conn(0x7fc5dc0b5650 0x7fc5dc0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 -- 172.21.15.61:0/524661552 >> 172.21.15.61:0/524661552 conn(0x7fc5dc0634e0 msgr2=0x7fc5dc0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:23.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 -- 172.21.15.61:0/524661552 shutdown_connections 2022-07-20T03:08:23.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 -- 172.21.15.61:0/524661552 wait complete. 2022-07-20T03:08:23.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 Processor -- start 2022-07-20T03:08:23.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.643+0000 7fc5db59e700 1 -- start start 2022-07-20T03:08:23.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5dc0adff0 0x7fc5dc0b3610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5dc0af8e0 0x7fc5dc146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 0x7fc5dc1464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc1105f0 con 0x7fc5dc0b5650 2022-07-20T03:08:23.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc110820 con 0x7fc5dc0adff0 2022-07-20T03:08:23.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5dc110ac0 con 0x7fc5dc0af8e0 2022-07-20T03:08:23.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 0x7fc5dc1464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 0x7fc5dc1464f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:43980/0 (socket says 172.21.15.61:43980) 2022-07-20T03:08:23.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 -- 172.21.15.61:0/1044220276 learned_addr learned my addr 172.21.15.61:0/1044220276 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:23.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5da59c700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5dc0adff0 0x7fc5dc0b3610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5d9d9b700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5dc0af8e0 0x7fc5dc146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 -- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5dc0af8e0 msgr2=0x7fc5dc146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5dc0af8e0 0x7fc5dc146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 -- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5dc0adff0 msgr2=0x7fc5dc0b3610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5dc0adff0 0x7fc5dc0b3610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 -- 172.21.15.61:0/1044220276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d4009470 con 0x7fc5dc0b5650 2022-07-20T03:08:23.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5dad9d700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 0x7fc5dc1464f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc5d40032f0 tx=0x7fc5d4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:23.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d400b8d0 con 0x7fc5dc0b5650 2022-07-20T03:08:23.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5d401f3f0 con 0x7fc5dc0b5650 2022-07-20T03:08:23.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5dc159820 con 0x7fc5dc0b5650 2022-07-20T03:08:23.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5d400b600 con 0x7fc5dc0b5650 2022-07-20T03:08:23.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5dc065180 con 0x7fc5dc0b5650 2022-07-20T03:08:23.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.647+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5dc110cc0 con 0x7fc5dc0b5650 2022-07-20T03:08:23.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.651+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc5d4026030 con 0x7fc5dc0b5650 2022-07-20T03:08:23.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.651+0000 7fc5cb7fe700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053a60 0x7fc5b8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:23.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.651+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc5d406c830 con 0x7fc5dc0b5650 2022-07-20T03:08:23.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.651+0000 7fc5da59c700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053a60 0x7fc5b8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:23.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.651+0000 7fc5da59c700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053a60 0x7fc5b8055f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fc5cc00b000 tx=0x7fc5cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:23.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.659+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5d4043730 con 0x7fc5dc0b5650 2022-07-20T03:08:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc5dc05c280 con 0x7fc5dc0b5650 2022-07-20T03:08:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5cb7fe700 1 -- 172.21.15.61:0/1044220276 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fc5d4024090 con 0x7fc5dc0b5650 2022-07-20T03:08:23.830 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:23.831 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:08:23.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053a60 msgr2=0x7fc5b8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053a60 0x7fc5b8055f20 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fc5cc00b000 tx=0x7fc5cc006df0).stop 2022-07-20T03:08:23.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 msgr2=0x7fc5dc1464f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:23.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 0x7fc5dc1464f0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fc5d40032f0 tx=0x7fc5d4016000).stop 2022-07-20T03:08:23.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 shutdown_connections 2022-07-20T03:08:23.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053a60 0x7fc5b8055f20 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5dc0b5650 0x7fc5dc1464f0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5dc0af8e0 0x7fc5dc146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 --2- 172.21.15.61:0/1044220276 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5dc0adff0 0x7fc5dc0b3610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:23.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 >> 172.21.15.61:0/1044220276 conn(0x7fc5dc0634e0 msgr2=0x7fc5dc069e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:23.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 shutdown_connections 2022-07-20T03:08:23.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:23.823+0000 7fc5db59e700 1 -- 172.21.15.61:0/1044220276 wait complete. 2022-07-20T03:08:23.908 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:08:23.948 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T03:08:24.065 INFO:teuthology.orchestra.run.smithi117.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T03:08:24.066 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T03:08:24.067 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:08:24.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.171+0000 7fad6fafd700 1 Processor -- start 2022-07-20T03:08:24.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.171+0000 7fad6fafd700 1 -- start start 2022-07-20T03:08:24.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6fafd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 0x7fad680af670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6fafd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad68067690 con 0x7fad680ae750 2022-07-20T03:08:24.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6fafd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fad68140c90 con 0x7fad680ad130 2022-07-20T03:08:24.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6fafd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad680ada00 con 0x7fad680ac830 2022-07-20T03:08:24.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fad680ae750 0x7fad681410f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40582/0 (socket says 172.21.15.61:40582) 2022-07-20T03:08:24.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 -- 172.21.15.61:0/4021613538 learned_addr learned my addr 172.21.15.61:0/4021613538 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:08:24.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6e09a700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 0x7fad680af670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1669967863 0 0) 0x7fad68067690 con 0x7fad680ae750 2022-07-20T03:08:24.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fad50003260 con 0x7fad680ae750 2022-07-20T03:08:24.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3330672669 0 0) 0x7fad50003260 con 0x7fad680ae750 2022-07-20T03:08:24.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 msgr2=0x7fad680af670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 0x7fad680af670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 >> v1:172.21.15.117:6789/0 conn(0x7fad680ad130 legacy=0x7fad680aef60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad680ad510 con 0x7fad680ae750 2022-07-20T03:08:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fad58001680 con 0x7fad680ae750 2022-07-20T03:08:24.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 >> v1:172.21.15.61:6789/0 conn(0x7fad680ae750 legacy=0x7fad681410f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 0x7fad50003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad50003c70 0x7fad50004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad500049f0 0x7fad50004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad680ada00 con 0x7fad50003c70 2022-07-20T03:08:24.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad5000b750 con 0x7fad500049f0 2022-07-20T03:08:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad5000b910 con 0x7fad680ac830 2022-07-20T03:08:24.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad50003c70 0x7fad50004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d899700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 0x7fad50003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6e09a700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad500049f0 0x7fad50004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 -- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 msgr2=0x7fad50003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad680ac830 0x7fad50003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 -- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad500049f0 msgr2=0x7fad50004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad500049f0 0x7fad50004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 -- 172.21.15.61:0/4021613538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad50003260 con 0x7fad50003c70 2022-07-20T03:08:24.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.175+0000 7fad6d098700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad50003c70 0x7fad50004500 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fad58001390 tx=0x7fad680628c0).ready entity=mon.0 client_cookie=fd70052695084860 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad58007170 con 0x7fad50003c70 2022-07-20T03:08:24.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad580072d0 con 0x7fad50003c70 2022-07-20T03:08:24.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6c897700 1 -- 172.21.15.61:0/4021613538 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad5800c8c0 con 0x7fad50003c70 2022-07-20T03:08:24.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 -- 172.21.15.61:0/4021613538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad50003c70 msgr2=0x7fad50004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 --2- 172.21.15.61:0/4021613538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad50003c70 0x7fad50004500 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7fad58001390 tx=0x7fad680628c0).stop 2022-07-20T03:08:24.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6e09a700 1 -- 172.21.15.61:0/4021613538 reap_dead start 2022-07-20T03:08:24.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 -- 172.21.15.61:0/4021613538 shutdown_connections 2022-07-20T03:08:24.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 -- 172.21.15.61:0/4021613538 >> 172.21.15.61:0/4021613538 conn(0x7fad680b23d0 msgr2=0x7fad680b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:24.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 -- 172.21.15.61:0/4021613538 shutdown_connections 2022-07-20T03:08:24.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 -- 172.21.15.61:0/4021613538 wait complete. 2022-07-20T03:08:24.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 Processor -- start 2022-07-20T03:08:24.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.179+0000 7fad6fafd700 1 -- start start 2022-07-20T03:08:24.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad50003c70 0x7fad680abca0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad681434e0 0x7fad68063690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 0x7fad681438e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68145db0 con 0x7fad68142410 2022-07-20T03:08:24.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fad68145b10 con 0x7fad681434e0 2022-07-20T03:08:24.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fad680631f0 con 0x7fad50003c70 2022-07-20T03:08:24.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 0x7fad681438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6d899700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad50003c70 0x7fad680abca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 0x7fad681438e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44000/0 (socket says 172.21.15.61:44000) 2022-07-20T03:08:24.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 -- 172.21.15.61:0/3844677930 learned_addr learned my addr 172.21.15.61:0/3844677930 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:24.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6d098700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad681434e0 0x7fad68063690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 -- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad50003c70 msgr2=0x7fad680abca0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad50003c70 0x7fad680abca0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 -- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad681434e0 msgr2=0x7fad68063690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad681434e0 0x7fad68063690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 -- 172.21.15.61:0/3844677930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fad6809b8e0 con 0x7fad68142410 2022-07-20T03:08:24.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6e09a700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 0x7fad681438e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fad6000b000 tx=0x7fad600091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad60019070 con 0x7fad68142410 2022-07-20T03:08:24.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fad60013750 con 0x7fad68142410 2022-07-20T03:08:24.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fad68143dd0 con 0x7fad68142410 2022-07-20T03:08:24.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fad60013ab0 con 0x7fad68142410 2022-07-20T03:08:24.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fad68144280 con 0x7fad68142410 2022-07-20T03:08:24.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.183+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fad680afca0 con 0x7fad68142410 2022-07-20T03:08:24.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.187+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fad60013c10 con 0x7fad68142410 2022-07-20T03:08:24.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.187+0000 7fad56ffd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fad48053a60 0x7fad48055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.187+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fad6006c7e0 con 0x7fad68142410 2022-07-20T03:08:24.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.187+0000 7fad6d899700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fad48053a60 0x7fad48055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.187+0000 7fad6d899700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fad48053a60 0x7fad48055f20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fad64003000 tx=0x7fad640079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.191+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fad600436e0 con 0x7fad68142410 2022-07-20T03:08:24.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.359+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fad680574e0 con 0x7fad68142410 2022-07-20T03:08:24.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad56ffd700 1 -- 172.21.15.61:0/3844677930 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fad60046d00 con 0x7fad68142410 2022-07-20T03:08:24.387 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:08:24.394 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:08:24.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fad48053a60 msgr2=0x7fad48055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fad48053a60 0x7fad48055f20 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fad64003000 tx=0x7fad640079e0).stop 2022-07-20T03:08:24.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 msgr2=0x7fad681438e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 0x7fad681438e0 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7fad6000b000 tx=0x7fad600091d0).stop 2022-07-20T03:08:24.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 shutdown_connections 2022-07-20T03:08:24.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fad48053a60 0x7fad48055f20 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fad681434e0 0x7fad68063690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fad68142410 0x7fad681438e0 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 --2- 172.21.15.61:0/3844677930 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fad50003c70 0x7fad680abca0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 >> 172.21.15.61:0/3844677930 conn(0x7fad680b23d0 msgr2=0x7fad68146010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:24.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 shutdown_connections 2022-07-20T03:08:24.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.363+0000 7fad6fafd700 1 -- 172.21.15.61:0/3844677930 wait complete. 2022-07-20T03:08:24.404 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:08:24.405 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b cache drop 2022-07-20T03:08:24.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 Processor -- start 2022-07-20T03:08:24.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 -- start start 2022-07-20T03:08:24.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f85680b5990 0x7f8568146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f856805d210 0x7f856814a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8568146c70 con 0x7f85680af3e0 2022-07-20T03:08:24.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85680ad050 con 0x7f856805d210 2022-07-20T03:08:24.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.511+0000 7f856dc69700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85680b17e0 con 0x7f85680b5990 2022-07-20T03:08:24.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f856805d210 0x7f856814a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f856805d210 0x7f856814a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44004/0 (socket says 172.21.15.61:44004) 2022-07-20T03:08:24.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 -- 172.21.15.61:0/3831289669 learned_addr learned my addr 172.21.15.61:0/3831289669 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:24.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f8567fff700 1 --1- 172.21.15.61:0/3831289669 >> v1:172.21.15.61:6789/0 conn(0x7f85680af3e0 0x7f8568146560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40606/0 (socket says 172.21.15.61:40606) 2022-07-20T03:08:24.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856cc67700 1 --2- 172.21.15.61:0/3831289669 >> v2:172.21.15.117:3300/0 conn(0x7f85680b5990 0x7f8568146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 -- 172.21.15.61:0/3831289669 >> v2:172.21.15.117:3300/0 conn(0x7f85680b5990 msgr2=0x7f8568146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 --2- 172.21.15.61:0/3831289669 >> v2:172.21.15.117:3300/0 conn(0x7f85680b5990 0x7f8568146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 -- 172.21.15.61:0/3831289669 >> v1:172.21.15.61:6789/0 conn(0x7f85680af3e0 legacy=0x7f8568146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f85677fe700 1 -- 172.21.15.61:0/3831289669 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3832256026 0 0) 0x7f8568146c70 con 0x7f85680af3e0 2022-07-20T03:08:24.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 -- 172.21.15.61:0/3831289669 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85680b5d70 con 0x7f856805d210 2022-07-20T03:08:24.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856d468700 1 --2- 172.21.15.61:0/3831289669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f856814a360 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f85600032f0 tx=0x7f8560009290).ready entity=mon.0 client_cookie=49ccb2cd14abe88e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f85677fe700 1 -- 172.21.15.61:0/3831289669 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856000b870 con 0x7f856805d210 2022-07-20T03:08:24.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f85677fe700 1 -- 172.21.15.61:0/3831289669 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856000b9d0 con 0x7f856805d210 2022-07-20T03:08:24.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f85677fe700 1 -- 172.21.15.61:0/3831289669 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856000bb40 con 0x7f856805d210 2022-07-20T03:08:24.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 -- 172.21.15.61:0/3831289669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 msgr2=0x7f856814a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 --2- 172.21.15.61:0/3831289669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f856814a360 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f85600032f0 tx=0x7f8560009290).stop 2022-07-20T03:08:24.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 -- 172.21.15.61:0/3831289669 shutdown_connections 2022-07-20T03:08:24.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 --2- 172.21.15.61:0/3831289669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f856814a360 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 --2- 172.21.15.61:0/3831289669 >> v2:172.21.15.117:3300/0 conn(0x7f85680b5990 0x7f8568146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 -- 172.21.15.61:0/3831289669 >> 172.21.15.61:0/3831289669 conn(0x7f8568063480 msgr2=0x7f85680b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:24.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 -- 172.21.15.61:0/3831289669 shutdown_connections 2022-07-20T03:08:24.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 -- 172.21.15.61:0/3831289669 wait complete. 2022-07-20T03:08:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 Processor -- start 2022-07-20T03:08:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.515+0000 7f856dc69700 1 -- start start 2022-07-20T03:08:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f8568140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85680af3e0 0x7f8568159680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85680b5990 0x7f8568159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8568143d70 con 0x7f856805d210 2022-07-20T03:08:24.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8568144440 con 0x7f85680b5990 2022-07-20T03:08:24.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85681113b0 con 0x7f85680af3e0 2022-07-20T03:08:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f8568140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f8568140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44010/0 (socket says 172.21.15.61:44010) 2022-07-20T03:08:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 -- 172.21.15.61:0/1026518559 learned_addr learned my addr 172.21.15.61:0/1026518559 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:24.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856d468700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85680b5990 0x7f8568159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f8567fff700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85680af3e0 0x7f8568159680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 -- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85680af3e0 msgr2=0x7f8568159680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85680af3e0 0x7f8568159680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 -- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85680b5990 msgr2=0x7f8568159b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85680b5990 0x7f8568159b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 -- 172.21.15.61:0/1026518559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8560009530 con 0x7f856805d210 2022-07-20T03:08:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856d468700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85680b5990 0x7f8568159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:24.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856cc67700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f8568140f80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f855800c000 tx=0x7f8558006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f8565ffb700 1 -- 172.21.15.61:0/1026518559 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8558011750 con 0x7f856805d210 2022-07-20T03:08:24.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f8565ffb700 1 -- 172.21.15.61:0/1026518559 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85580118b0 con 0x7f856805d210 2022-07-20T03:08:24.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8568111650 con 0x7f856805d210 2022-07-20T03:08:24.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f8565ffb700 1 -- 172.21.15.61:0/1026518559 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8558011c10 con 0x7f856805d210 2022-07-20T03:08:24.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.519+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8568144c40 con 0x7f856805d210 2022-07-20T03:08:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8565ffb700 1 -- 172.21.15.61:0/1026518559 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8558011d70 con 0x7f856805d210 2022-07-20T03:08:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8565ffb700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f854c053ad0 0x7f854c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8565ffb700 1 -- 172.21.15.61:0/1026518559 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f855806c140 con 0x7f856805d210 2022-07-20T03:08:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f854c053ad0 0x7f854c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 Processor -- start 2022-07-20T03:08:24.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 -- start start 2022-07-20T03:08:24.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f854c053ad0 0x7f854c055f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f855000d000 tx=0x7f85500079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xef5530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 0xef5a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeebbd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xe28010 con 0xeb4bd0 2022-07-20T03:08:24.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xe08250 con 0xeb6270 2022-07-20T03:08:24.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xeb37c0 con 0xef5f10 2022-07-20T03:08:24.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 0xef5a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 0xef5a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44020/0 (socket says 172.21.15.61:44020) 2022-07-20T03:08:24.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 -- 172.21.15.61:0/2531961481 learned_addr learned my addr 172.21.15.61:0/2531961481 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:24.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f856cc67700 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xef5530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f856d468700 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeebbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 -- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 msgr2=0xeebbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeebbd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 -- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 msgr2=0xef5530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xef5530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 -- 172.21.15.61:0/2531961481 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe60030 con 0xeb4bd0 2022-07-20T03:08:24.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f856cc67700 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xef5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:24.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f8567fff700 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 0xef5a20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8550000bc0 tx=0x7f855000a000).ready entity=mon.0 client_cookie=ca58b664e2fdc83e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f856dc69700 1 -- 172.21.15.61:0/2531961481 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f855001c070 con 0xeb4bd0 2022-07-20T03:08:24.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f856dc69700 1 -- 172.21.15.61:0/2531961481 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85500157e0 con 0xeb4bd0 2022-07-20T03:08:24.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.523+0000 7f856dc69700 1 -- 172.21.15.61:0/2531961481 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8550015b40 con 0xeb4bd0 2022-07-20T03:08:24.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- 172.21.15.61:0/2531961481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 msgr2=0xef5a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 0xef5a20 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f8550000bc0 tx=0x7f855000a000).stop 2022-07-20T03:08:24.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- 172.21.15.61:0/2531961481 shutdown_connections 2022-07-20T03:08:24.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeebbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb4bd0 0xef5a20 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- 172.21.15.61:0/2531961481 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xef5530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- 172.21.15.61:0/2531961481 >> 172.21.15.61:0/2531961481 conn(0xeb40e0 msgr2=0xeb44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:24.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- 172.21.15.61:0/2531961481 shutdown_connections 2022-07-20T03:08:24.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- 172.21.15.61:0/2531961481 wait complete. 2022-07-20T03:08:24.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- start start 2022-07-20T03:08:24.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 Processor -- start 2022-07-20T03:08:24.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xeeece0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xefee60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xef5f10 0xeff350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xef23c0 con 0xeb6270 2022-07-20T03:08:24.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xef00f0 con 0xef5f10 2022-07-20T03:08:24.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xef25f0 con 0xeb4bd0 2022-07-20T03:08:24.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8579122740 10 client.0 ll_register_callbacks cb 0xe6f610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:08:24.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8567fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xefee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8567fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xefee60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44026/0 (socket says 172.21.15.61:44026) 2022-07-20T03:08:24.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8567fff700 1 -- 172.21.15.61:0/268881837 learned_addr learned my addr 172.21.15.61:0/268881837 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:24.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xef5f10 0xeff350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856cc67700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xeeece0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 -- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 msgr2=0xeeece0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xeeece0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 -- 172.21.15.61:0/268881837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 msgr2=0xefee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xefee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 -- 172.21.15.61:0/268881837 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8550017400 con 0xef5f10 2022-07-20T03:08:24.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f8567fff700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xefee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:08:24.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f856d468700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xef5f10 0xeff350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0xef1850 tx=0x7f8560016000).ready entity=mon.1 client_cookie=9c058b26efd199db server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.527+0000 7f85477fe700 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2022-07-20T03:08:24.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 1 -- 172.21.15.61:0/268881837 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856000bb40 con 0xef5f10 2022-07-20T03:08:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 1 -- 172.21.15.61:0/268881837 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f856001d040 con 0xef5f10 2022-07-20T03:08:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xe1dfc0 con 0xef5f10 2022-07-20T03:08:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 1 -- 172.21.15.61:0/268881837 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f8560006da0 con 0xef5f10 2022-07-20T03:08:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 10 client.5161 fetch_fsmap learned FSMap version 14 2022-07-20T03:08:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0xdfa9b0 con 0xef5f10 2022-07-20T03:08:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 1 -- 172.21.15.61:0/268881837 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xe1dfc0 con 0xef5f10 2022-07-20T03:08:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 1 -- 172.21.15.61:0/268881837 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 14) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f856002b920 con 0xef5f10 2022-07-20T03:08:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 10 client.5161 fetch_fsmap finished waiting for FSMap version 14 2022-07-20T03:08:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 10 client.5161 resolve_mds: resolved ID 'b' to GID 4911 2022-07-20T03:08:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 20 client.5161 populate_metadata read hostname 'smithi061' 2022-07-20T03:08:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5ab0 0xee5e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 4 client.5161 mds_command: new command op to 4911 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T03:08:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 --> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xe16040 con 0xee5ab0 2022-07-20T03:08:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f856cc67700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5ab0 0xee5e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 10 client.5161 ms_handle_connect on v2:172.21.15.117:6834/939788248 2022-07-20T03:08:24.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f856cc67700 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5ab0 0xee5e90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=310c0342d324effa server_cookie=3b62ad6229404630 in_seq=0 out_seq=0 2022-07-20T03:08:24.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 1 -- 172.21.15.61:0/268881837 <== mds.0 v2:172.21.15.117:6834/939788248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0xe16040 con 0xee5ab0 2022-07-20T03:08:24.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 10 client.5161 handle_command_reply: tid=0 2022-07-20T03:08:24.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 client.5161 shutdown 2022-07-20T03:08:24.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xef5f10 msgr2=0xeff350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xef5f10 0xeff350 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0xef1850 tx=0x7f8560016000).stop 2022-07-20T03:08:24.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 shutdown_connections 2022-07-20T03:08:24.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xef5f10 0xeff350 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5ab0 0xee5e90 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xefee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 --2- 172.21.15.61:0/268881837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xeeece0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f85477fe700 0 client.5161 ms_handle_reset on v2:172.21.15.117:6834/939788248 2022-07-20T03:08:24.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 >> 172.21.15.61:0/268881837 conn(0xeb40e0 msgr2=0xeb44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:24.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 shutdown_connections 2022-07-20T03:08:24.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 1 -- 172.21.15.61:0/268881837 wait complete. 2022-07-20T03:08:24.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.531+0000 7f8579122740 20 client.5161 trim_cache size 0 max 16384 2022-07-20T03:08:24.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 Processor -- start 2022-07-20T03:08:24.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 -- start start 2022-07-20T03:08:24.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb4bd0 0xeeb480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xeb29e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeb2ed0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xeede70 con 0xeb6270 2022-07-20T03:08:24.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xef0580 con 0xeb4bd0 2022-07-20T03:08:24.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xe1dfc0 con 0xef5f10 2022-07-20T03:08:24.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xeb29e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xeb29e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44034/0 (socket says 172.21.15.61:44034) 2022-07-20T03:08:24.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 -- 172.21.15.61:0/549558722 learned_addr learned my addr 172.21.15.61:0/549558722 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:24.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f856cc67700 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb4bd0 0xeeb480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f856d468700 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeb2ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:24.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 -- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 msgr2=0xeb2ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeb2ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 -- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb4bd0 msgr2=0xeeb480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:24.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb4bd0 0xeeb480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f8567fff700 1 -- 172.21.15.61:0/549558722 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xe175b0 con 0xeb6270 2022-07-20T03:08:24.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.547+0000 7f856cc67700 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb4bd0 0xeeb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:08:24.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8567fff700 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xeb29e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8550000bc0 tx=0x7f8550016000).ready entity=mon.0 client_cookie=95b0ed394e8b6436 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:24.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f855dffb700 1 -- 172.21.15.61:0/549558722 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8550015c10 con 0xeb6270 2022-07-20T03:08:24.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f855dffb700 1 -- 172.21.15.61:0/549558722 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f855002a040 con 0xeb6270 2022-07-20T03:08:24.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f855dffb700 1 -- 172.21.15.61:0/549558722 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8550022420 con 0xeb6270 2022-07-20T03:08:24.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 -- 172.21.15.61:0/549558722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 msgr2=0xeb29e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:08:24.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xeb29e0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7f8550000bc0 tx=0x7f8550016000).stop 2022-07-20T03:08:24.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 -- 172.21.15.61:0/549558722 shutdown_connections 2022-07-20T03:08:24.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xef5f10 0xeb2ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xeb6270 0xeb29e0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 --2- 172.21.15.61:0/549558722 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb4bd0 0xeeb480 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:24.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 -- 172.21.15.61:0/549558722 >> 172.21.15.61:0/549558722 conn(0xeb40e0 msgr2=0xeef3e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:08:24.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 -- 172.21.15.61:0/549558722 shutdown_connections 2022-07-20T03:08:24.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 -- 172.21.15.61:0/549558722 wait complete. 2022-07-20T03:08:24.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 -- start start 2022-07-20T03:08:24.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.551+0000 7f8579122740 1 Processor -- start 2022-07-20T03:08:24.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xf4e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xf4eab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 0xf526f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xefa1f0 con 0xef5f10 2022-07-20T03:08:25.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xefa490 con 0xeb6270 2022-07-20T03:08:25.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xef88d0 con 0xeb4bd0 2022-07-20T03:08:25.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 10 client.0 ll_register_callbacks cb 0xef1710 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:08:25.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 0xf526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:25.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 0xf526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44042/0 (socket says 172.21.15.61:44042) 2022-07-20T03:08:25.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8567fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xf4eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:25.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 -- 172.21.15.61:0/3009596971 learned_addr learned my addr 172.21.15.61:0/3009596971 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:08:25.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856cc67700 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xf4e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:25.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 -- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 msgr2=0xf4e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:25.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xf4e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:25.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 -- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 msgr2=0xf4eab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:08:25.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xf4eab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:08:25.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 -- 172.21.15.61:0/3009596971 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0xe60030 con 0xef5f10 2022-07-20T03:08:25.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856d468700 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 0xf526f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f856002e000 tx=0x7f8560021000).ready entity=mon.0 client_cookie=be66997d8a36a8eb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:08:25.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-20T03:08:25.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 1 -- 172.21.15.61:0/3009596971 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f856000b870 con 0xef5f10 2022-07-20T03:08:25.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 1 -- 172.21.15.61:0/3009596971 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8560011e00 con 0xef5f10 2022-07-20T03:08:25.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f85500157e0 con 0xef5f10 2022-07-20T03:08:25.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 1 -- 172.21.15.61:0/3009596971 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=14) v2 ==== 24+0+0 (secure 0 0 0) 0x7f856000bb40 con 0xef5f10 2022-07-20T03:08:25.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 10 client.5157 fetch_fsmap learned FSMap version 14 2022-07-20T03:08:25.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=14,monmap=2+}) v3 -- 0xe1cb90 con 0xef5f10 2022-07-20T03:08:25.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 1 -- 172.21.15.61:0/3009596971 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f856003e020 con 0xef5f10 2022-07-20T03:08:25.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 1 -- 172.21.15.61:0/3009596971 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 14) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f85600103a0 con 0xef5f10 2022-07-20T03:08:25.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 10 client.5157 fetch_fsmap finished waiting for FSMap version 14 2022-07-20T03:08:25.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 10 client.5157 resolve_mds: resolved ID 'b' to GID 4911 2022-07-20T03:08:25.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 20 client.5157 populate_metadata read hostname 'smithi061' 2022-07-20T03:08:25.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5640 0xf52be0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:08:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 4 client.5157 mds_command: new command op to 4911 tid=0[{"prefix": "cache drop"}] 2022-07-20T03:08:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 --> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0xd0b590 con 0xee5640 2022-07-20T03:08:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856cc67700 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5640 0xf52be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:08:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f856cc67700 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5640 0xf52be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=1a7fb63b832b7c88 server_cookie=afbd2a0899334169 in_seq=0 out_seq=0 2022-07-20T03:08:25.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:08:24.555+0000 7f855cff9700 10 client.5157 ms_handle_connect on v2:172.21.15.117:6834/939788248 2022-07-20T03:08:30.972 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:30.992 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:31.016 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:31.042 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:08:31.064 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:01.148 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:01.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:01.200 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:01.211 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:01.227 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:08.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f855cff9700 1 -- 172.21.15.61:0/3009596971 <== mds.0 v2:172.21.15.117:6834/939788248 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+386 (crc 0 0 0) 0xd0b590 con 0xee5640 2022-07-20T03:09:08.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f855cff9700 10 client.5157 handle_command_reply: tid=0 2022-07-20T03:09:08.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 client.5157 shutdown 2022-07-20T03:09:08.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 msgr2=0xf526f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:08.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 0xf526f0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7f856002e000 tx=0x7f8560021000).stop 2022-07-20T03:09:08.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 shutdown_connections 2022-07-20T03:09:08.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xeb6270 0xf4eab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:08.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:6834/939788248,v1:172.21.15.117:6835/939788248] conn(0xee5640 0xf52be0 crc :-1 s=READY pgs=14 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:08.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xef5f10 0xf526f0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:08.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f855cff9700 0 client.5157 ms_handle_reset on v2:172.21.15.117:6834/939788248 2022-07-20T03:09:08.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 --2- 172.21.15.61:0/3009596971 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xeb4bd0 0xf4e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:08.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 >> 172.21.15.61:0/3009596971 conn(0xeb40e0 msgr2=0xeed3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:08.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 shutdown_connections 2022-07-20T03:09:08.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 1 -- 172.21.15.61:0/3009596971 wait complete. 2022-07-20T03:09:08.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f8579122740 20 client.5157 trim_cache size 0 max 16384 2022-07-20T03:09:08.937 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:09:08.938 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-20T03:09:08.938 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-20T03:09:08.939 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(0) Success", 2022-07-20T03:09:08.939 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 920 2022-07-20T03:09:08.940 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:08.940 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-20T03:09:08.941 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-20T03:09:08.941 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-20T03:09:08.941 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:08.942 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-20T03:09:08.942 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 900 2022-07-20T03:09:08.943 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:08.943 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-20T03:09:08.943 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-20T03:09:08.944 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 2503, 2022-07-20T03:09:08.944 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 431744 2022-07-20T03:09:08.945 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:08.945 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:08.946 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 44.356054888999999 2022-07-20T03:09:08.990 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:09:08.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f854c053ad0 msgr2=0x7f854c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:09.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f854c053ad0 0x7f854c055f90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f855000d000 tx=0x7f85500079e0).stop 2022-07-20T03:09:09.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 msgr2=0x7f8568140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:09.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f8568140f80 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f855800c000 tx=0x7f8558006d90).stop 2022-07-20T03:09:09.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 shutdown_connections 2022-07-20T03:09:09.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f854c053ad0 0x7f854c055f90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:09.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85680b5990 0x7f8568159b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:09.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85680af3e0 0x7f8568159680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:09.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 --2- 172.21.15.61:0/1026518559 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f856805d210 0x7f8568140f80 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:09.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 >> 172.21.15.61:0/1026518559 conn(0x7f8568063480 msgr2=0x7f85681453f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:09.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 shutdown_connections 2022-07-20T03:09:09.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:08.915+0000 7f856dc69700 1 -- 172.21.15.61:0/1026518559 wait complete. 2022-07-20T03:09:09.024 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:09:09.024 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:09.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:09.111 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:09.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:09.231 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:09.279 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:09:09.280 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:09.281 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:09.282 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:09.300 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:09:09.302 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:09:09.303 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:09:09.304 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:09:09.311 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:09.379 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22377]: fuse finished with error 0 and tester_r 0 2022-07-20T03:09:09.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:09.382 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:09.383 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:09.410 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:09:09.411 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:09:09.434 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:09:09.435 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:09.529 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:09.573 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:09.574 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:09:09.610 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:09.612 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:09:09.612 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:09:09.613 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:09:09.613 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:09:09.667 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:09.668 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:09:09.669 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:09.669 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:09.669 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:09:09.734 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:09.735 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:09:09.736 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:09:09.737 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:09:09.738 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:09:09.790 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:09.803 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:09:09.804 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:09:09.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8ffff700 1 Processor -- start 2022-07-20T03:09:09.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8ffff700 1 -- start start 2022-07-20T03:09:09.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8ffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5a900b5650 0x7f5a900b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:09.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a90147010 con 0x7f5a900af8e0 2022-07-20T03:09:09.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8ffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a901472b0 con 0x7f5a900adff0 2022-07-20T03:09:09.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8ffff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a90146d40 con 0x7f5a900b5650 2022-07-20T03:09:09.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5a900af8e0 0x7f5a9005bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40646/0 (socket says 172.21.15.61:40646) 2022-07-20T03:09:09.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8f7fe700 1 -- 172.21.15.61:0/3248666955 learned_addr learned my addr 172.21.15.61:0/3248666955 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:09.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.931+0000 7f5a8effd700 1 --2- 172.21.15.61:0/3248666955 >> v2:172.21.15.117:3300/0 conn(0x7f5a900b5650 0x7f5a900b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:10.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8dffb700 1 -- 172.21.15.61:0/3248666955 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3097021640 0 0) 0x7f5a90147010 con 0x7f5a900af8e0 2022-07-20T03:09:10.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8dffb700 1 -- 172.21.15.61:0/3248666955 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5a70003260 con 0x7f5a900af8e0 2022-07-20T03:09:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8effd700 1 -- 172.21.15.61:0/3248666955 >> v1:172.21.15.117:6789/0 conn(0x7f5a900adff0 legacy=0x7f5a90147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8dffb700 1 -- 172.21.15.61:0/3248666955 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3237197345 0 0) 0x7f5a70003260 con 0x7f5a900af8e0 2022-07-20T03:09:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8effd700 1 -- 172.21.15.61:0/3248666955 >> v1:172.21.15.61:6789/0 conn(0x7f5a900af8e0 legacy=0x7f5a9005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8effd700 1 -- 172.21.15.61:0/3248666955 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a9005c4a0 con 0x7f5a900b5650 2022-07-20T03:09:10.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8effd700 1 --2- 172.21.15.61:0/3248666955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a900b5a30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5a88001b80 tx=0x7f5a78006df0).ready entity=mon.1 client_cookie=8a62c86b8d1f4b11 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8dffb700 1 -- 172.21.15.61:0/3248666955 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a880019e0 con 0x7f5a900b5650 2022-07-20T03:09:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8dffb700 1 -- 172.21.15.61:0/3248666955 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a78000e90 con 0x7f5a900b5650 2022-07-20T03:09:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8dffb700 1 -- 172.21.15.61:0/3248666955 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a7800d4f0 con 0x7f5a900b5650 2022-07-20T03:09:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3248666955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 msgr2=0x7f5a900b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3248666955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a900b5a30 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f5a88001b80 tx=0x7f5a78006df0).stop 2022-07-20T03:09:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3248666955 shutdown_connections 2022-07-20T03:09:10.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3248666955 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a900b5a30 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3248666955 >> 172.21.15.61:0/3248666955 conn(0x7f5a900634e0 msgr2=0x7f5a900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3248666955 shutdown_connections 2022-07-20T03:09:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3248666955 wait complete. 2022-07-20T03:09:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 Processor -- start 2022-07-20T03:09:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.935+0000 7f5a8ffff700 1 -- start start 2022-07-20T03:09:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 0x7f5a9010ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a900af8e0 0x7f5a90155980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a90159890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a90110df0 con 0x7f5a900adff0 2022-07-20T03:09:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9010f990 con 0x7f5a900b5650 2022-07-20T03:09:10.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a9010e840 con 0x7f5a900af8e0 2022-07-20T03:09:10.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 0x7f5a9010ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:10.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 0x7f5a9010ffa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44050/0 (socket says 172.21.15.61:44050) 2022-07-20T03:09:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 -- 172.21.15.61:0/3561940937 learned_addr learned my addr 172.21.15.61:0/3561940937 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:10.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8f7fe700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a90159890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:10.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8e7fc700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a900af8e0 0x7f5a90155980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 -- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a900af8e0 msgr2=0x7f5a90155980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a900af8e0 0x7f5a90155980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 -- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 msgr2=0x7f5a90159890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a90159890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 -- 172.21.15.61:0/3561940937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7800b180 con 0x7f5a900adff0 2022-07-20T03:09:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8effd700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 0x7f5a9010ffa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f5a88001b80 tx=0x7f5a78014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a78016070 con 0x7f5a900adff0 2022-07-20T03:09:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a78013b30 con 0x7f5a900adff0 2022-07-20T03:09:10.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a9010eae0 con 0x7f5a900adff0 2022-07-20T03:09:10.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a7800d4f0 con 0x7f5a900adff0 2022-07-20T03:09:10.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a900643a0 con 0x7f5a900adff0 2022-07-20T03:09:10.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.939+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a9010ec70 con 0x7f5a900adff0 2022-07-20T03:09:10.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.943+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5a7800f030 con 0x7f5a900adff0 2022-07-20T03:09:10.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.943+0000 7f5a87fff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a74053ad0 0x7f5a74055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:10.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.943+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a7806c840 con 0x7f5a900adff0 2022-07-20T03:09:10.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.943+0000 7f5a8e7fc700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a74053ad0 0x7f5a74055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:10.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.943+0000 7f5a8e7fc700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a74053ad0 0x7f5a74055f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5a8000d000 tx=0x7f5a80007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:10.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:09.947+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a78043610 con 0x7f5a900adff0 2022-07-20T03:09:10.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.075+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5a900574e0 con 0x7f5a900adff0 2022-07-20T03:09:10.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.075+0000 7f5a87fff700 1 -- 172.21.15.61:0/3561940937 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v33) v1 ==== 81+0+33693 (secure 0 0 0) 0x7f5a78022020 con 0x7f5a900adff0 2022-07-20T03:09:10.141 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:10.141 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:09:10.141 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 33, 2022-07-20T03:09:10.142 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:09:10.152 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:09:10.152 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:08:05.485261+0000", 2022-07-20T03:09:10.152 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:09:10.153 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:09:10.153 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:09:10.154 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:09:10.154 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:09:10.154 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:09:10.155 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:09:10.155 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:09:10.156 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:09:10.156 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:10.156 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:09:10.157 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:09:10.157 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:09:10.158 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:09:10.158 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:09:10.160 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 6, 2022-07-20T03:09:10.160 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:09:10.161 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:09:10.161 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:09:10.161 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:09:10.162 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:09:10.162 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.168 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:09:10.168 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:09:10.168 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:09:10.183 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:09:10.183 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:09:10.184 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:09:10.185 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:09:10.202 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:09:10.203 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:09:10.203 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:09:10.204 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:09:10.204 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:09:10.205 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:09:10.205 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:09:10.219 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:09:10.219 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:09:10.219 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:09:10.220 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:09:10.220 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:09:10.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:09:10.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:09:10.221 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:09:10.254 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:09:10.255 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.257 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:09:10.258 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:09:10.259 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:09:10.259 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:09:10.259 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:09:10.260 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:09:10.260 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:09:10.261 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:09:10.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:09:10.262 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:09:10.262 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:09:10.263 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:09:10.263 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:09:10.263 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:09:10.264 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:09:10.268 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:09:10.269 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:09:10.269 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:09:10.269 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:09:10.270 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:09:10.270 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:09:10.271 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:09:10.271 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:09:10.296 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:09:10.296 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:09:10.296 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:09:10.297 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:09:10.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.298 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:09:10.298 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:09:10.298 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:09:10.300 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:09:10.301 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:09:10.301 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:09:10.301 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:09:10.304 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:09:10.304 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:09:10.305 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:09:10.305 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:09:10.306 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:09:10.306 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:09:10.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.307 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:09:10.307 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:09:10.308 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.309 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:09:10.310 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:09:10.310 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:09:10.311 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:09:10.311 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:09:10.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:09:10.312 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:09:10.312 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:09:10.313 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:09:10.313 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:09:10.314 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:09:10.314 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:09:10.316 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:09:10.316 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:09:10.317 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:09:10.317 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:09:10.318 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:09:10.318 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:09:10.318 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:09:10.319 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:09:10.319 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:09:10.320 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:09:10.321 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:09:10.322 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.322 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:09:10.344 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:09:10.344 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:09:10.345 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:09:10.345 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:09:10.345 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:09:10.353 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:09:10.354 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:09:10.354 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:09:10.355 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:09:10.356 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:09:10.357 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:09:10.357 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:09:10.358 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:09:10.358 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:09:10.359 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:09:10.359 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:09:10.360 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:09:10.360 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:09:10.370 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:09:10.371 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:09:10.371 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:09:10.372 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:09:10.372 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:09:10.373 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:09:10.374 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:09:10.383 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:09:10.384 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.384 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:09:10.384 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:09:10.385 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:09:10.386 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:09:10.386 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:09:10.387 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:09:10.388 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:09:10.390 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:09:10.391 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:09:10.391 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:09:10.392 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:09:10.393 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:09:10.394 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:09:10.394 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:09:10.395 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.395 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.396 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.396 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 5, 2022-07-20T03:09:10.397 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:09:10.397 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:08:02.776339+0000", 2022-07-20T03:09:10.397 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:09:10.398 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:09:10.405 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:09:10.406 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:09:10.406 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:09:10.407 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:09:10.407 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:09:10.408 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:09:10.408 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:09:10.409 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:09:10.410 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:09:10.410 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:09:10.411 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:09:10.411 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:09:10.411 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:09:10.412 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:09:10.412 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:09:10.413 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:09:10.413 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:09:10.415 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:09:10.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.416 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "32", 2022-07-20T03:09:10.417 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:09:10.417 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:09:10.418 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:09:10.418 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:09:10.419 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:09:10.419 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:09:10.420 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:09:10.420 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:09:10.495 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:09:10.495 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:09:10.496 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:09:10.498 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:09:10.499 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:09:10.499 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:09:10.499 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:09:10.500 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:09:10.500 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:09:10.501 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:09:10.501 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:09:10.504 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:09:10.505 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:09:10.505 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:09:10.505 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:09:10.506 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:09:10.507 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:09:10.507 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:09:10.508 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.508 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:09:10.509 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:09:10.509 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:09:10.510 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:09:10.510 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:09:10.511 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:09:10.511 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:09:10.512 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:09:10.513 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:09:10.514 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:09:10.514 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:09:10.515 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:09:10.515 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:09:10.516 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:09:10.516 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:09:10.516 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.517 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:09:10.517 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:09:10.517 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:09:10.518 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.518 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.519 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.519 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.519 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 6, 2022-07-20T03:09:10.520 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:09:10.520 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:08:03.472472+0000", 2022-07-20T03:09:10.521 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:09:10.521 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:09:10.522 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:09:10.523 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:09:10.523 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:09:10.530 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:09:10.553 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:09:10.564 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:09:10.565 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:09:10.565 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:09:10.565 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:09:10.566 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:09:10.566 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:09:10.567 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:09:10.567 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:09:10.568 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:09:10.568 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:09:10.569 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:09:10.569 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:09:10.570 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:09:10.585 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.585 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "33", 2022-07-20T03:09:10.586 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:09:10.586 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:09:10.613 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:09:10.614 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:09:10.614 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:09:10.615 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:09:10.615 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:09:10.616 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:09:10.616 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:09:10.616 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:09:10.617 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:09:10.617 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:09:10.617 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:09:10.618 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:09:10.618 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:09:10.619 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:09:10.620 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:09:10.621 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:09:10.621 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:09:10.622 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:09:10.622 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:09:10.622 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:09:10.623 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:09:10.624 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:09:10.624 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:09:10.624 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:09:10.625 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.625 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:09:10.626 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:09:10.626 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:09:10.627 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:09:10.627 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:09:10.628 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:09:10.628 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:09:10.629 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:09:10.629 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:09:10.672 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:09:10.672 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:09:10.673 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:09:10.673 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:09:10.673 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:09:10.673 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:09:10.674 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.674 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.675 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.675 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:10.675 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:09:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:09:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:09:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:10.676 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:10.677 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:10.677 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:10.677 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:10.677 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:10.678 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:10.678 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:10.678 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:10.678 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:10.678 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:10.679 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.679 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.679 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.679 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:09:10.680 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.680 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.680 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.680 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.680 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:09:10.688 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.689 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.689 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.690 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.690 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:10.690 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.691 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.691 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.692 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:09:10.692 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.693 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.696 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.696 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.697 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:09:10.697 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.698 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.698 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.698 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.699 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:10.699 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.700 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.700 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.701 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:09:10.701 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.704 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.705 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:09:10.706 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.706 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.707 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.707 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.707 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:10.708 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.709 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.709 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:09:10.710 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.710 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.710 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.711 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.712 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:09:10.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:10.714 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.714 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.715 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.715 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:09:10.717 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:09:10.717 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:09:10.717 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:09:10.718 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:10.718 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:10.719 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:10.719 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.719 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.720 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.720 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:09:10.721 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:09:10.721 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:10.731 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:10.732 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:10.732 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:10.733 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:10.733 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:10.739 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:10.740 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:10.740 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:10.741 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:10.750 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:10.750 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.751 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.751 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.751 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:09:10.752 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.765 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.766 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.766 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:09:10.766 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.767 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.767 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.768 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.768 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:10.769 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.769 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.786 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.787 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:09:10.787 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.788 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.802 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.802 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:09:10.803 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.803 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.817 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.822 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.823 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:10.826 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.827 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.827 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.827 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:09:10.827 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.828 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.828 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.828 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:09:10.829 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.829 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.829 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.829 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.829 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:10.830 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.832 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.833 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.833 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:09:10.834 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.834 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.835 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.835 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.836 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:09:10.836 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:10.837 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.837 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.837 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.864 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:09:10.864 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:09:10.865 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:09:10.865 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:09:10.865 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:10.865 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.866 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.867 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:09:10.867 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:09:10.867 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:10.867 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:10.868 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:10.868 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:10.868 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:10.868 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-20T03:09:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:10.869 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:10.870 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.870 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.870 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.870 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:09:10.871 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.871 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.871 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.876 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.876 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:09:10.876 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.876 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.877 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.877 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:10.877 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.878 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.878 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:09:10.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.879 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.879 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.880 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:09:10.880 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.880 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.880 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.880 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.881 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:10.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.881 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.881 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.882 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:09:10.882 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.882 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.882 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.883 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.883 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:09:10.883 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.883 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.884 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.884 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.884 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:10.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.885 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.885 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.885 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:09:10.885 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.886 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.886 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.886 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.886 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:09:10.887 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:10.887 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.887 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.900 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.900 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:09:10.900 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:09:10.901 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:09:10.901 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:09:10.901 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:10.901 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:10.902 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:10.902 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.902 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.903 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:09:10.903 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:09:10.903 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:10.903 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:10.904 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:10.904 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:10.904 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:10.904 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:10.905 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:10.905 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:10.905 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:10.905 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:10.905 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:10.950 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.950 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.950 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.951 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:09:10.966 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:10.966 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.966 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.966 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.967 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:09:10.967 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:10.967 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.967 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.968 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.968 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:10.968 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:10.968 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.969 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:10.969 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:09:10.969 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:10.969 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.994 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:10.994 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:10.995 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:09:10.995 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:10.996 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:10.996 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:10.996 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:10.997 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:10.997 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.015 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:09:11.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:11.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.017 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.018 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:09:11.040 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:11.040 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.041 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.042 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:11.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:09:11.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:11.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.044 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:09:11.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:11.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.045 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.046 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:09:11.055 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:09:11.055 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:09:11.055 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:09:11.056 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:11.056 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:11.056 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:11.056 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.056 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.057 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:09:11.058 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:09:11.058 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:11.068 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:11.068 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:11.068 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:11.068 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:11.069 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:11.069 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:11.069 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:11.069 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:11.069 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:11.070 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:11.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.071 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:09:11.072 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.072 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.073 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:09:11.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.073 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.073 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.074 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:11.074 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.075 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.075 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:09:11.075 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.075 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.075 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.076 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:09:11.076 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.076 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.077 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.077 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:11.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.078 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.078 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:09:11.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.086 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:09:11.087 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.110 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:11.110 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.111 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:09:11.111 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.112 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.112 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:09:11.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:11.113 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.113 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.114 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:09:11.114 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:09:11.114 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:09:11.114 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:09:11.115 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:11.115 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:11.115 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:11.115 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.122 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:09:11.122 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:09:11.122 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:11.122 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:11.123 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:11.123 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:11.123 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:11.123 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:11.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:11.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:11.125 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:11.125 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:11.125 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:11.126 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.127 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:09:11.128 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.128 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.129 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:09:11.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.135 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.135 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.136 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:11.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.137 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:09:11.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:09:11.138 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.139 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.139 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.139 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:11.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.140 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:09:11.141 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:09:11.143 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.143 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.144 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:11.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.173 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.173 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:09:11.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.174 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.174 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:09:11.175 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:11.175 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.175 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.176 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:09:11.176 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:09:11.176 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:09:11.177 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:09:11.177 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:11.177 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:11.177 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:11.205 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.214 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:09:11.214 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:09:11.214 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:11.215 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:11.215 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:11.215 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:11.215 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:11.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:11.216 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:11.216 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-20T03:09:11.216 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:11.217 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:11.217 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:11.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.217 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.234 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:09:11.235 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:09:11.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.291 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.291 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.295 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:11.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.307 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.308 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:09:11.323 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.323 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.324 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.324 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:09:11.325 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.334 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.336 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.336 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.337 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:11.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.338 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:09:11.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.354 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.354 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.355 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.355 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:09:11.355 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.356 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.356 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.357 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:11.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.360 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.360 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:09:11.361 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.404 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.404 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.405 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:09:11.415 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:11.415 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.416 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.417 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:09:11.417 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:09:11.418 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:09:11.418 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:09:11.419 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:11.419 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:11.424 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:11.424 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.425 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.425 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:09:11.426 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:09:11.426 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:11.427 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:11.427 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:11.428 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:11.448 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:11.448 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:11.449 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:11.449 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:11.449 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:11.450 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:11.450 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:11.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.451 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.484 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:09:11.485 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.488 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.489 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.489 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.490 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:09:11.490 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.490 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.491 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.491 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.492 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:11.492 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.493 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.493 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.519 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:09:11.519 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.520 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.559 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.560 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.593 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:09:11.593 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.594 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.594 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.595 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.595 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:11.595 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.596 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.597 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:09:11.597 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.597 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.611 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.611 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.612 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:09:11.612 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.612 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.612 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.613 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.613 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:11.622 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:11.625 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.625 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:11.625 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:09:11.625 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.626 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.626 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.626 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:11.626 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:09:11.627 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:11.627 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.627 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.627 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.628 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:09:11.628 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:09:11.628 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:09:11.628 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:09:11.629 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:11.629 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:11.629 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:11.629 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:11.630 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.697 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:11.698 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:09:11.699 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.699 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:09:11.699 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.700 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.700 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.701 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.701 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.722 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:09:11.723 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.723 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.724 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.724 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:09:11.724 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.725 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.725 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.751 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.751 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.752 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:09:11.752 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.753 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.753 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.753 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:09:11.759 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.760 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.767 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.767 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.768 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.768 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:09:11.768 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.769 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.769 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.769 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:09:11.788 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.789 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.789 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.790 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.790 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.791 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:09:11.791 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.792 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.792 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.804 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:09:11.805 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.806 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.807 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.807 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.808 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.808 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:09:11.809 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.809 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.810 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.810 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:09:11.811 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.812 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.812 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.813 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.814 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:09:11.815 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.815 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.816 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.816 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:09:11.817 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.817 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.818 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.819 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.819 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.820 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:09:11.820 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.820 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.821 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:11.821 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:09:11.822 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:11.822 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:11.823 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:11.824 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:11.824 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:11.824 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:09:11.825 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:11.825 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.838 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:11.839 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:09:11.839 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:09:11.840 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:09:11.840 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:09:11.851 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:09:11.852 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:09:11.861 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:09:11.861 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:09:11.862 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:09:11.863 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:09:11.863 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:09:11.864 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:09:11.864 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:09:11.864 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:11.865 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:11.865 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:09:11.866 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:09:11.867 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:09:11.875 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:09:11.898 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:09:11.899 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:09:11.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a74053ad0 msgr2=0x7f5a74055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:11.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a74053ad0 0x7f5a74055f90 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f5a8000d000 tx=0x7f5a80007b80).stop 2022-07-20T03:09:11.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 msgr2=0x7f5a9010ffa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:11.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 0x7f5a9010ffa0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f5a88001b80 tx=0x7f5a78014000).stop 2022-07-20T03:09:11.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 shutdown_connections 2022-07-20T03:09:11.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a74053ad0 0x7f5a74055f90 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:11.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a900b5650 0x7f5a90159890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:11.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a900af8e0 0x7f5a90155980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:11.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 --2- 172.21.15.61:0/3561940937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a900adff0 0x7f5a9010ffa0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:11.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 >> 172.21.15.61:0/3561940937 conn(0x7f5a900634e0 msgr2=0x7f5a90141790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:11.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 shutdown_connections 2022-07-20T03:09:11.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:10.079+0000 7f5a8ffff700 1 -- 172.21.15.61:0/3561940937 wait complete. 2022-07-20T03:09:11.983 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5afcd50700 1 Processor -- start 2022-07-20T03:09:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5afcd50700 1 -- start start 2022-07-20T03:09:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5afcd50700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5af805ce60 0x7f5af8061e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5afcd50700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af80acba0 con 0x7f5af809aa30 2022-07-20T03:09:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5afcd50700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5af80b1d30 con 0x7f5af8061a50 2022-07-20T03:09:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5afcd50700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5af80b1fd0 con 0x7f5af805ce60 2022-07-20T03:09:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5af809aa30 0x7f5af809b010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40658/0 (socket says 172.21.15.61:40658) 2022-07-20T03:09:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af6ffd700 1 -- 172.21.15.61:0/3184896157 learned_addr learned my addr 172.21.15.61:0/3184896157 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af7fff700 1 --2- 172.21.15.61:0/3184896157 >> v2:172.21.15.117:3300/0 conn(0x7f5af805ce60 0x7f5af8061e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1721468547 0 0) 0x7f5af80acba0 con 0x7f5af809aa30 2022-07-20T03:09:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ae4003260 con 0x7f5af809aa30 2022-07-20T03:09:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 81574707 0 0) 0x7f5af80b1d30 con 0x7f5af8061a50 2022-07-20T03:09:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5af80acba0 con 0x7f5af8061a50 2022-07-20T03:09:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2549200226 0 0) 0x7f5ae4003260 con 0x7f5af809aa30 2022-07-20T03:09:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 >> v2:172.21.15.117:3300/0 conn(0x7f5af805ce60 msgr2=0x7f5af8061e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 --2- 172.21.15.61:0/3184896157 >> v2:172.21.15.117:3300/0 conn(0x7f5af805ce60 0x7f5af8061e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 >> v1:172.21.15.117:6789/0 conn(0x7f5af8061a50 legacy=0x7f5af80b2470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af809b720 con 0x7f5af809aa30 2022-07-20T03:09:12.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f5aec001660 con 0x7f5af809aa30 2022-07-20T03:09:12.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 >> v1:172.21.15.61:6789/0 conn(0x7f5af809aa30 legacy=0x7f5af809b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae40038e0 0x7f5ae4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae4004500 0x7f5ae4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae4004e70 0x7f5ae400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af80b1fd0 con 0x7f5ae4004500 2022-07-20T03:09:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae400bcf0 con 0x7f5ae40038e0 2022-07-20T03:09:12.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.123+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae400beb0 con 0x7f5ae4004e70 2022-07-20T03:09:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af6ffd700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae4004500 0x7f5ae4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae4004e70 0x7f5ae400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af77fe700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae40038e0 0x7f5ae4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 -- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae40038e0 msgr2=0x7f5ae4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae40038e0 0x7f5ae4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 -- 172.21.15.61:0/3184896157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae4004500 msgr2=0x7f5ae4004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae4004500 0x7f5ae4004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 -- 172.21.15.61:0/3184896157 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae4003260 con 0x7f5ae4004e70 2022-07-20T03:09:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af6ffd700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae4004500 0x7f5ae4004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 -- 172.21.15.61:0/3184896157 reap_dead start 2022-07-20T03:09:12.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af7fff700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae4004e70 0x7f5ae400b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f5ae40054d0 tx=0x7f5af00091d0).ready entity=mon.2 client_cookie=c6848a25350439d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:12.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af000e710 con 0x7f5ae4004e70 2022-07-20T03:09:12.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af000e870 con 0x7f5ae4004e70 2022-07-20T03:09:12.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5af67fc700 1 -- 172.21.15.61:0/3184896157 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af000ebd0 con 0x7f5ae4004e70 2022-07-20T03:09:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 -- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae4004e70 msgr2=0x7f5ae400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae4004e70 0x7f5ae400b750 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f5ae40054d0 tx=0x7f5af00091d0).stop 2022-07-20T03:09:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 -- 172.21.15.61:0/3184896157 shutdown_connections 2022-07-20T03:09:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 --2- 172.21.15.61:0/3184896157 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae4004e70 0x7f5ae400b750 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 -- 172.21.15.61:0/3184896157 >> 172.21.15.61:0/3184896157 conn(0x7f5af80b2ca0 msgr2=0x7f5af80b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 -- 172.21.15.61:0/3184896157 shutdown_connections 2022-07-20T03:09:12.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.127+0000 7f5afcd50700 1 -- 172.21.15.61:0/3184896157 wait complete. 2022-07-20T03:09:12.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 Processor -- start 2022-07-20T03:09:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 -- start start 2022-07-20T03:09:12.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae4004e70 0x7f5af8063ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af8063fa0 0x7f5af8097e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5af80659b0 0x7f5af80ad770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af8064e20 con 0x7f5af8063fa0 2022-07-20T03:09:12.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5af813e780 con 0x7f5ae4004e70 2022-07-20T03:09:12.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5af8065270 con 0x7f5af80659b0 2022-07-20T03:09:12.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af8063fa0 0x7f5af8097e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af77fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae4004e70 0x7f5af8063ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5af80659b0 0x7f5af80ad770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af8063fa0 0x7f5af8097e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44072/0 (socket says 172.21.15.61:44072) 2022-07-20T03:09:12.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af6ffd700 1 -- 172.21.15.61:0/1145958050 learned_addr learned my addr 172.21.15.61:0/1145958050 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:12.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 -- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae4004e70 msgr2=0x7f5af8063ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae4004e70 0x7f5af8063ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 -- 172.21.15.61:0/1145958050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af8063fa0 msgr2=0x7f5af8097e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af8063fa0 0x7f5af8097e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 -- 172.21.15.61:0/1145958050 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5af8065510 con 0x7f5af80659b0 2022-07-20T03:09:12.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af7fff700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5af80659b0 0x7f5af80ad770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5af8065d90 tx=0x7f5af0019000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:12.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af0006ea0 con 0x7f5af80659b0 2022-07-20T03:09:12.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5af000eea0 con 0x7f5af80659b0 2022-07-20T03:09:12.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5af8064aa0 con 0x7f5af80659b0 2022-07-20T03:09:12.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5af000e710 con 0x7f5af80659b0 2022-07-20T03:09:12.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.131+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5af80b0490 con 0x7f5af80659b0 2022-07-20T03:09:12.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.135+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5af80656a0 con 0x7f5af80659b0 2022-07-20T03:09:12.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.135+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5af0018020 con 0x7f5af80659b0 2022-07-20T03:09:12.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.135+0000 7f5af4ff9700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ad8053a60 0x7f5ad8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.135+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5af006db40 con 0x7f5af80659b0 2022-07-20T03:09:12.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.135+0000 7f5af77fe700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ad8053a60 0x7f5ad8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.135+0000 7f5af77fe700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ad8053a60 0x7f5ad8055f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5ae8003000 tx=0x7f5ae8007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:12.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.143+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5af0044ae0 con 0x7f5af80659b0 2022-07-20T03:09:12.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.303+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5af80b0490 con 0x7f5af80659b0 2022-07-20T03:09:12.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5af4ff9700 1 -- 172.21.15.61:0/1145958050 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 14 v14) v1 ==== 94+0+2135 (secure 0 0 0) 0x7f5af80b0490 con 0x7f5af80659b0 2022-07-20T03:09:12.461 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:12.484 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":14,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":14,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:08:06.488044+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":4911},"failed":[],"damaged":[],"stopped":[],"info":{"gid_4911":{"gid":4911,"name":"b","rank":0,"incarnation":12,"state":"up:active","state_seq":4,"addr":"172.21.15.117:6835/939788248","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":939788248},{"type":"v1","addr":"172.21.15.117:6835","nonce":939788248}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:09:12.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ad8053a60 msgr2=0x7f5ad8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ad8053a60 0x7f5ad8055f20 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f5ae8003000 tx=0x7f5ae8007a00).stop 2022-07-20T03:09:12.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5af80659b0 msgr2=0x7f5af80ad770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5af80659b0 0x7f5af80ad770 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7f5af8065d90 tx=0x7f5af0019000).stop 2022-07-20T03:09:12.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 shutdown_connections 2022-07-20T03:09:12.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ad8053a60 0x7f5ad8055f20 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5af80659b0 0x7f5af80ad770 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5af8063fa0 0x7f5af8097e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 --2- 172.21.15.61:0/1145958050 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae4004e70 0x7f5af8063ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 >> 172.21.15.61:0/1145958050 conn(0x7f5af80b2ca0 msgr2=0x7f5af813f210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:12.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 shutdown_connections 2022-07-20T03:09:12.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.307+0000 7f5afcd50700 1 -- 172.21.15.61:0/1145958050 wait complete. 2022-07-20T03:09:12.498 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 14 2022-07-20T03:09:12.502 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:09:12.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 Processor -- start 2022-07-20T03:09:12.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- start start 2022-07-20T03:09:12.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f7790146780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7790146c70 con 0x7f77900b5990 2022-07-20T03:09:12.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77900b5d70 con 0x7f779005d210 2022-07-20T03:09:12.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77900ad050 con 0x7f77900af3e0 2022-07-20T03:09:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795811700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f77900b5990 0x7f7790146070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40676/0 (socket says 172.21.15.61:40676) 2022-07-20T03:09:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795811700 1 -- 172.21.15.61:0/866713549 learned_addr learned my addr 172.21.15.61:0/866713549 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:12.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795010700 1 --2- 172.21.15.61:0/866713549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f7790146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f779480f700 1 -- 172.21.15.61:0/866713549 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3997927137 0 0) 0x7f7790146c70 con 0x7f77900b5990 2022-07-20T03:09:12.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f779480f700 1 -- 172.21.15.61:0/866713549 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7774003260 con 0x7f77900b5990 2022-07-20T03:09:12.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795010700 1 -- 172.21.15.61:0/866713549 >> v1:172.21.15.117:6789/0 conn(0x7f779005d210 legacy=0x7f779014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f779480f700 1 -- 172.21.15.61:0/866713549 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2287778547 0 0) 0x7f77900b5d70 con 0x7f779005d210 2022-07-20T03:09:12.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795010700 1 -- 172.21.15.61:0/866713549 >> v1:172.21.15.61:6789/0 conn(0x7f77900b5990 legacy=0x7f7790146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795010700 1 -- 172.21.15.61:0/866713549 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77900b1010 con 0x7f77900af3e0 2022-07-20T03:09:12.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7795010700 1 --2- 172.21.15.61:0/866713549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f7790146780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f77880032f0 tx=0x7f77880091d0).ready entity=mon.2 client_cookie=fa831b2880cf664e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:12.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f779480f700 1 -- 172.21.15.61:0/866713549 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f778800b6c0 con 0x7f77900af3e0 2022-07-20T03:09:12.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f779480f700 1 -- 172.21.15.61:0/866713549 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778800b820 con 0x7f77900af3e0 2022-07-20T03:09:12.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f779480f700 1 -- 172.21.15.61:0/866713549 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f778800b990 con 0x7f77900af3e0 2022-07-20T03:09:12.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- 172.21.15.61:0/866713549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 msgr2=0x7f7790146780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 --2- 172.21.15.61:0/866713549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f7790146780 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f77880032f0 tx=0x7f77880091d0).stop 2022-07-20T03:09:12.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- 172.21.15.61:0/866713549 shutdown_connections 2022-07-20T03:09:12.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 --2- 172.21.15.61:0/866713549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f7790146780 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- 172.21.15.61:0/866713549 >> 172.21.15.61:0/866713549 conn(0x7f7790063480 msgr2=0x7f77900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:12.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- 172.21.15.61:0/866713549 shutdown_connections 2022-07-20T03:09:12.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.619+0000 7f7796813700 1 -- 172.21.15.61:0/866713549 wait complete. 2022-07-20T03:09:12.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 Processor -- start 2022-07-20T03:09:12.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 -- start start 2022-07-20T03:09:12.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f779005d210 0x7f7790111470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f779013f380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 0x7f779013f870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77901570c0 con 0x7f77900b5990 2022-07-20T03:09:12.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7790157320 con 0x7f779005d210 2022-07-20T03:09:12.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7790111cf0 con 0x7f77900af3e0 2022-07-20T03:09:12.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 0x7f779013f870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7795811700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f779005d210 0x7f7790111470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7795010700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f779013f380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 0x7f779013f870 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44088/0 (socket says 172.21.15.61:44088) 2022-07-20T03:09:12.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7795010700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f779013f380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:58990/0 (socket says 172.21.15.61:58990) 2022-07-20T03:09:12.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 -- 172.21.15.61:0/2381271366 learned_addr learned my addr 172.21.15.61:0/2381271366 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:12.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 -- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 msgr2=0x7f779013f380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f779013f380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 -- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f779005d210 msgr2=0x7f7790111470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:12.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f779005d210 0x7f7790111470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:12.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 -- 172.21.15.61:0/2381271366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7788009470 con 0x7f77900b5990 2022-07-20T03:09:12.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796012700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 0x7f779013f870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f778c000bc0 tx=0x7f778c0079b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:12.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f778c00cde0 con 0x7f77900b5990 2022-07-20T03:09:12.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f778c012400 con 0x7f77900b5990 2022-07-20T03:09:12.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f779013fd60 con 0x7f77900b5990 2022-07-20T03:09:12.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f778c012760 con 0x7f77900b5990 2022-07-20T03:09:12.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.623+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77901451c0 con 0x7f77900b5990 2022-07-20T03:09:12.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.627+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7790111b00 con 0x7f77900b5990 2022-07-20T03:09:12.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.627+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f778c019030 con 0x7f77900b5990 2022-07-20T03:09:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.627+0000 7f7782ffd700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77780539b0 0x7f7778055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.627+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(33..33 src has 1..33) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f778c06b110 con 0x7f77900b5990 2022-07-20T03:09:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.627+0000 7f7795811700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77780539b0 0x7f7778055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.631+0000 7f7795811700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77780539b0 0x7f7778055e70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f778400d000 tx=0x7f7784009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.635+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f778c042010 con 0x7f77900b5990 2022-07-20T03:09:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:12.799+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f77900574e0 con 0x7f77900b5990 2022-07-20T03:09:13.124 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:09:13.119+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:09:13.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.135+0000 7f7782ffd700 1 -- 172.21.15.61:0/2381271366 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v15) v1 ==== 161+0+0 (secure 0 0 0) 0x7f778c045630 con 0x7f77900b5990 2022-07-20T03:09:13.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77780539b0 msgr2=0x7f7778055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77780539b0 0x7f7778055e70 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f778400d000 tx=0x7f7784009040).stop 2022-07-20T03:09:13.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 msgr2=0x7f779013f870 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 0x7f779013f870 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f778c000bc0 tx=0x7f778c0079b0).stop 2022-07-20T03:09:13.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 shutdown_connections 2022-07-20T03:09:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77780539b0 0x7f7778055e70 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77900b5990 0x7f779013f870 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77900af3e0 0x7f779013f380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 --2- 172.21.15.61:0/2381271366 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f779005d210 0x7f7790111470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 >> 172.21.15.61:0/2381271366 conn(0x7f7790063480 msgr2=0x7f779014aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:13.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 shutdown_connections 2022-07-20T03:09:13.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.139+0000 7f7796813700 1 -- 172.21.15.61:0/2381271366 wait complete. 2022-07-20T03:09:13.198 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:09:13.199 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:09:13.201 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:13.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71ba2bf700 1 Processor -- start 2022-07-20T03:09:13.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71ba2bf700 1 -- start start 2022-07-20T03:09:13.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71ba2bf700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f71b40b3110 0x7f71b4061d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71ba2bf700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40acaf0 con 0x7f71b413f940 2022-07-20T03:09:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71ba2bf700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71b40b2190 con 0x7f71b40b34f0 2022-07-20T03:09:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71ba2bf700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71b40acd90 con 0x7f71b40b3110 2022-07-20T03:09:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b885c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f71b413f940 0x7f71b413fdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40694/0 (socket says 172.21.15.61:40694) 2022-07-20T03:09:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b885c700 1 -- 172.21.15.61:0/4075722121 learned_addr learned my addr 172.21.15.61:0/4075722121 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:13.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b3fff700 1 --2- 172.21.15.61:0/4075722121 >> v2:172.21.15.117:3300/0 conn(0x7f71b40b3110 0x7f71b4061d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3130805291 0 0) 0x7f71b40acaf0 con 0x7f71b413f940 2022-07-20T03:09:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7194003260 con 0x7f71b413f940 2022-07-20T03:09:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 657359633 0 0) 0x7f71b40b2190 con 0x7f71b40b34f0 2022-07-20T03:09:13.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f71b40acaf0 con 0x7f71b40b34f0 2022-07-20T03:09:13.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1248069433 0 0) 0x7f7194003260 con 0x7f71b413f940 2022-07-20T03:09:13.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b3fff700 1 -- 172.21.15.61:0/4075722121 >> v1:172.21.15.117:6789/0 conn(0x7f71b40b34f0 legacy=0x7f71b41414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b3fff700 1 -- 172.21.15.61:0/4075722121 >> v1:172.21.15.61:6789/0 conn(0x7f71b413f940 legacy=0x7f71b413fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b3fff700 1 -- 172.21.15.61:0/4075722121 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71b40b3940 con 0x7f71b40b3110 2022-07-20T03:09:13.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b3fff700 1 --2- 172.21.15.61:0/4075722121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b3110 0x7f71b4061d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f71ac001490 tx=0x7f71a4006df0).ready entity=mon.1 client_cookie=c59b677824e1abdf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:13.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71ac0019e0 con 0x7f71b40b3110 2022-07-20T03:09:13.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a4000e90 con 0x7f71b40b3110 2022-07-20T03:09:13.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.327+0000 7f71b37fe700 1 -- 172.21.15.61:0/4075722121 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71a4012460 con 0x7f71b40b3110 2022-07-20T03:09:13.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- 172.21.15.61:0/4075722121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b3110 msgr2=0x7f71b4061d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/4075722121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b3110 0x7f71b4061d80 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f71ac001490 tx=0x7f71a4006df0).stop 2022-07-20T03:09:13.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- 172.21.15.61:0/4075722121 shutdown_connections 2022-07-20T03:09:13.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/4075722121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b3110 0x7f71b4061d80 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- 172.21.15.61:0/4075722121 >> 172.21.15.61:0/4075722121 conn(0x7f71b40b47e0 msgr2=0x7f71b405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:13.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- 172.21.15.61:0/4075722121 shutdown_connections 2022-07-20T03:09:13.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- 172.21.15.61:0/4075722121 wait complete. 2022-07-20T03:09:13.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 Processor -- start 2022-07-20T03:09:13.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- start start 2022-07-20T03:09:13.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 0x7f71b409c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b34f0 0x7f71b4141f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71b413f940 0x7f71b40b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40ae800 con 0x7f71b40b3110 2022-07-20T03:09:13.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71b40676d0 con 0x7f71b40b34f0 2022-07-20T03:09:13.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ba2bf700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71b4066c50 con 0x7f71b413f940 2022-07-20T03:09:13.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 0x7f71b409c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 0x7f71b409c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44098/0 (socket says 172.21.15.61:44098) 2022-07-20T03:09:13.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71b3fff700 1 -- 172.21.15.61:0/271424641 learned_addr learned my addr 172.21.15.61:0/271424641 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:13.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71ab7fe700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b34f0 0x7f71b4141f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.331+0000 7f71b885c700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71b413f940 0x7f71b40b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b3fff700 1 -- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71b413f940 msgr2=0x7f71b40b0d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b3fff700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71b413f940 0x7f71b40b0d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b3fff700 1 -- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b34f0 msgr2=0x7f71b4141f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b3fff700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b34f0 0x7f71b4141f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b3fff700 1 -- 172.21.15.61:0/271424641 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71a400b180 con 0x7f71b40b3110 2022-07-20T03:09:13.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b3fff700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 0x7f71b409c620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f71b4066de0 tx=0x7f71a4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:13.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71a4016070 con 0x7f71b40b3110 2022-07-20T03:09:13.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71a400da90 con 0x7f71b40b3110 2022-07-20T03:09:13.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71b4066910 con 0x7f71b40b3110 2022-07-20T03:09:13.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71a4012460 con 0x7f71b40b3110 2022-07-20T03:09:13.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71b4144e80 con 0x7f71b40b3110 2022-07-20T03:09:13.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.335+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71b40b2630 con 0x7f71b40b3110 2022-07-20T03:09:13.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.339+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f71a4024030 con 0x7f71b40b3110 2022-07-20T03:09:13.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.339+0000 7f71b1ffb700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7198053a40 0x7f7198055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.339+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f71a406cb60 con 0x7f71b40b3110 2022-07-20T03:09:13.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.339+0000 7f71ab7fe700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7198053a40 0x7f7198055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.339+0000 7f71ab7fe700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7198053a40 0x7f7198055f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f719c00d000 tx=0x7f719c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:13.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.347+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71a4043a70 con 0x7f71b40b3110 2022-07-20T03:09:13.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71b40574e0 con 0x7f71b40b3110 2022-07-20T03:09:13.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71b1ffb700 1 -- 172.21.15.61:0/271424641 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 15 v15) v1 ==== 94+0+1774 (secure 0 0 0) 0x7f71a4022070 con 0x7f71b40b3110 2022-07-20T03:09:13.544 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:13.544 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":15,"default_fscid":2,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11}],"filesystems":[{"mdsmap":{"epoch":15,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:08:04.482490+0000","modified":"2022-07-20T03:09:13.123661+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":34,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[6],"metadata_pool":5,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":2}]} 2022-07-20T03:09:13.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7198053a40 msgr2=0x7f7198055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7198053a40 0x7f7198055f00 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f719c00d000 tx=0x7f719c007a20).stop 2022-07-20T03:09:13.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 msgr2=0x7f71b409c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 0x7f71b409c620 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f71b4066de0 tx=0x7f71a4014000).stop 2022-07-20T03:09:13.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 shutdown_connections 2022-07-20T03:09:13.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7198053a40 0x7f7198055f00 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71b40b34f0 0x7f71b4141f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71b413f940 0x7f71b40b0d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 --2- 172.21.15.61:0/271424641 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71b40b3110 0x7f71b409c620 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 >> 172.21.15.61:0/271424641 conn(0x7f71b40b47e0 msgr2=0x7f71b413ba50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:13.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.511+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 shutdown_connections 2022-07-20T03:09:13.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.515+0000 7f71ba2bf700 1 -- 172.21.15.61:0/271424641 wait complete. 2022-07-20T03:09:13.613 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 15 2022-07-20T03:09:13.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:09:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed95c96700 1 Processor -- start 2022-07-20T03:09:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed95c96700 1 -- start start 2022-07-20T03:09:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed95c96700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fed900b2640 0x7fed9005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed95c96700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed90146260 con 0x7fed900b4190 2022-07-20T03:09:13.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed95c96700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed900ad540 con 0x7fed90140480 2022-07-20T03:09:13.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed95c96700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed900acae0 con 0x7fed900b2640 2022-07-20T03:09:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fed900b4190 0x7fed9013fd70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40704/0 (socket says 172.21.15.61:40704) 2022-07-20T03:09:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8f7fe700 1 -- 172.21.15.61:0/1834801199 learned_addr learned my addr 172.21.15.61:0/1834801199 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8effd700 1 --2- 172.21.15.61:0/1834801199 >> v2:172.21.15.117:3300/0 conn(0x7fed900b2640 0x7fed9005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8e7fc700 1 -- 172.21.15.61:0/1834801199 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 257477596 0 0) 0x7fed90146260 con 0x7fed900b4190 2022-07-20T03:09:13.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8e7fc700 1 -- 172.21.15.61:0/1834801199 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fed70003260 con 0x7fed900b4190 2022-07-20T03:09:13.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8effd700 1 -- 172.21.15.61:0/1834801199 >> v1:172.21.15.117:6789/0 conn(0x7fed90140480 legacy=0x7fed9005c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8effd700 1 -- 172.21.15.61:0/1834801199 >> v1:172.21.15.61:6789/0 conn(0x7fed900b4190 legacy=0x7fed9013fd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.723+0000 7fed8effd700 1 -- 172.21.15.61:0/1834801199 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed900acd80 con 0x7fed900b2640 2022-07-20T03:09:13.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed8effd700 1 --2- 172.21.15.61:0/1834801199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b2640 0x7fed9005bd90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fed78002f70 tx=0x7fed78006df0).ready entity=mon.1 client_cookie=c87a75e69dbe6642 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:13.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed8e7fc700 1 -- 172.21.15.61:0/1834801199 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed7800d610 con 0x7fed900b2640 2022-07-20T03:09:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed8e7fc700 1 -- 172.21.15.61:0/1834801199 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed78000e90 con 0x7fed900b2640 2022-07-20T03:09:13.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed8e7fc700 1 -- 172.21.15.61:0/1834801199 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed7800d960 con 0x7fed900b2640 2022-07-20T03:09:13.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- 172.21.15.61:0/1834801199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b2640 msgr2=0x7fed9005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 --2- 172.21.15.61:0/1834801199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b2640 0x7fed9005bd90 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fed78002f70 tx=0x7fed78006df0).stop 2022-07-20T03:09:13.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- 172.21.15.61:0/1834801199 shutdown_connections 2022-07-20T03:09:13.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 --2- 172.21.15.61:0/1834801199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b2640 0x7fed9005bd90 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- 172.21.15.61:0/1834801199 >> 172.21.15.61:0/1834801199 conn(0x7fed90064d00 msgr2=0x7fed90065110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- 172.21.15.61:0/1834801199 shutdown_connections 2022-07-20T03:09:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- 172.21.15.61:0/1834801199 wait complete. 2022-07-20T03:09:13.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 Processor -- start 2022-07-20T03:09:13.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- start start 2022-07-20T03:09:13.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 0x7fed900ae860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b4190 0x7fed90060520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed90140480 0x7fed901444a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed90145860 con 0x7fed900b2640 2022-07-20T03:09:13.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed90152790 con 0x7fed900b4190 2022-07-20T03:09:13.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.727+0000 7fed95c96700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed901529b0 con 0x7fed90140480 2022-07-20T03:09:13.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 0x7fed900ae860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 0x7fed900ae860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44112/0 (socket says 172.21.15.61:44112) 2022-07-20T03:09:13.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 -- 172.21.15.61:0/813086618 learned_addr learned my addr 172.21.15.61:0/813086618 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:13.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8ffff700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed90140480 0x7fed901444a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8effd700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b4190 0x7fed90060520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 -- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed90140480 msgr2=0x7fed901444a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed90140480 0x7fed901444a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 -- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b4190 msgr2=0x7fed90060520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b4190 0x7fed90060520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:13.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 -- 172.21.15.61:0/813086618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed7800b180 con 0x7fed900b2640 2022-07-20T03:09:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8f7fe700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 0x7fed900ae860 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fed80001110 tx=0x7fed80007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed800070d0 con 0x7fed900b2640 2022-07-20T03:09:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed80007230 con 0x7fed900b2640 2022-07-20T03:09:13.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed80005610 con 0x7fed900b2640 2022-07-20T03:09:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed90060240 con 0x7fed900b2640 2022-07-20T03:09:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed90060a10 con 0x7fed900b2640 2022-07-20T03:09:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed900ad020 con 0x7fed900b2640 2022-07-20T03:09:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.731+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fed80010070 con 0x7fed900b2640 2022-07-20T03:09:13.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.735+0000 7fed8cff9700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed740539b0 0x7fed74055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.735+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fed8006b610 con 0x7fed900b2640 2022-07-20T03:09:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.735+0000 7fed8effd700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed740539b0 0x7fed74055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.735+0000 7fed8effd700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed740539b0 0x7fed74055e70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fed7801a000 tx=0x7fed78014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:13.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.743+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed80042410 con 0x7fed900b2640 2022-07-20T03:09:13.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:13.907+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fed900574e0 con 0x7fed900b2640 2022-07-20T03:09:14.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.811+0000 7fed8cff9700 1 -- 172.21.15.61:0/813086618 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v16) v1 ==== 104+0+0 (secure 0 0 0) 0x7fed80045a30 con 0x7fed900b2640 2022-07-20T03:09:14.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed740539b0 msgr2=0x7fed74055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed740539b0 0x7fed74055e70 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fed7801a000 tx=0x7fed78014000).stop 2022-07-20T03:09:14.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 msgr2=0x7fed900ae860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:14.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 0x7fed900ae860 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7fed80001110 tx=0x7fed80007ad0).stop 2022-07-20T03:09:14.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 shutdown_connections 2022-07-20T03:09:14.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed740539b0 0x7fed74055e70 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:14.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed900b4190 0x7fed90060520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:14.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed90140480 0x7fed901444a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:14.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 --2- 172.21.15.61:0/813086618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed900b2640 0x7fed900ae860 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:14.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 >> 172.21.15.61:0/813086618 conn(0x7fed90064d00 msgr2=0x7fed9013f360 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:14.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 shutdown_connections 2022-07-20T03:09:14.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.815+0000 7fed95c96700 1 -- 172.21.15.61:0/813086618 wait complete. 2022-07-20T03:09:14.861 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:09:15.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.975+0000 7f0ce06b7700 1 Processor -- start 2022-07-20T03:09:15.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.975+0000 7f0ce06b7700 1 -- start start 2022-07-20T03:09:15.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0cd80b5b30 0x7f0cd80aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 0x7f0cd80b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cd81473e0 con 0x7f0cd80b11c0 2022-07-20T03:09:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cd80b5f10 con 0x7f0cd80b5b30 2022-07-20T03:09:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80afc40 con 0x7f0cd80ace90 2022-07-20T03:09:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0cd80b11c0 0x7f0cd814a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40722/0 (socket says 172.21.15.61:40722) 2022-07-20T03:09:15.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 -- 172.21.15.61:0/2103078769 learned_addr learned my addr 172.21.15.61:0/2103078769 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:15.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cde453700 1 --2- 172.21.15.61:0/2103078769 >> v2:172.21.15.117:3300/0 conn(0x7f0cd80b5b30 0x7f0cd80aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cddc52700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 0x7f0cd80b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 779449683 0 0) 0x7f0cd0001110 con 0x7f0cd80b11c0 2022-07-20T03:09:15.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0cc4003260 con 0x7f0cd80b11c0 2022-07-20T03:09:15.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1989058285 0 0) 0x7f0cc4003260 con 0x7f0cd80b11c0 2022-07-20T03:09:15.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 msgr2=0x7f0cd80b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:15.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 0x7f0cd80b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 >> v2:172.21.15.117:3300/0 conn(0x7f0cd80b5b30 msgr2=0x7f0cd80aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:15.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 --2- 172.21.15.61:0/2103078769 >> v2:172.21.15.117:3300/0 conn(0x7f0cd80b5b30 0x7f0cd80aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd80afee0 con 0x7f0cd80b11c0 2022-07-20T03:09:15.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f0cd00017f0 con 0x7f0cd80b11c0 2022-07-20T03:09:15.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 >> v1:172.21.15.61:6789/0 conn(0x7f0cd80b11c0 legacy=0x7f0cd814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:15.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 0x7f0cc4003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc4003c80 0x7f0cc4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc40049f0 0x7f0cc4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80b5f10 con 0x7f0cc40049f0 2022-07-20T03:09:15.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80afc40 con 0x7f0cc4003c80 2022-07-20T03:09:15.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc400b750 con 0x7f0cd80ace90 2022-07-20T03:09:15.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc40049f0 0x7f0cc4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cde453700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 0x7f0cc4003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cddc52700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc4003c80 0x7f0cc4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 -- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 msgr2=0x7f0cc4003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:15.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd80ace90 0x7f0cc4003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 -- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc4003c80 msgr2=0x7f0cc4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:15.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc4003c80 0x7f0cc4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 -- 172.21.15.61:0/2103078769 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc4003260 con 0x7f0cc40049f0 2022-07-20T03:09:15.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cddc52700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc4003c80 0x7f0cc4004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:15.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc40049f0 0x7f0cc4004e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f0cc40054d0 tx=0x7f0cd80b0070).ready entity=mon.0 client_cookie=bffc31d30c6873c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:15.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd00070d0 con 0x7f0cc40049f0 2022-07-20T03:09:15.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0007230 con 0x7f0cc40049f0 2022-07-20T03:09:15.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdd451700 1 -- 172.21.15.61:0/2103078769 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd000ca90 con 0x7f0cc40049f0 2022-07-20T03:09:15.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 -- 172.21.15.61:0/2103078769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc40049f0 msgr2=0x7f0cc4004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:15.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/2103078769 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc40049f0 0x7f0cc4004e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f0cc40054d0 tx=0x7f0cd80b0070).stop 2022-07-20T03:09:15.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0cdec54700 1 -- 172.21.15.61:0/2103078769 reap_dead start 2022-07-20T03:09:15.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 -- 172.21.15.61:0/2103078769 shutdown_connections 2022-07-20T03:09:15.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.979+0000 7f0ce06b7700 1 -- 172.21.15.61:0/2103078769 >> 172.21.15.61:0/2103078769 conn(0x7f0cd80635b0 msgr2=0x7f0cd80b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:15.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.983+0000 7f0ce06b7700 1 -- 172.21.15.61:0/2103078769 shutdown_connections 2022-07-20T03:09:15.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.983+0000 7f0ce06b7700 1 -- 172.21.15.61:0/2103078769 wait complete. 2022-07-20T03:09:15.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.983+0000 7f0ce06b7700 1 Processor -- start 2022-07-20T03:09:15.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.983+0000 7f0ce06b7700 1 -- start start 2022-07-20T03:09:15.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc40049f0 0x7f0cd810e610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd81115c0 0x7f0cd8111a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 0x7f0cd8159ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80675e0 con 0x7f0cd8159a40 2022-07-20T03:09:15.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd80671e0 con 0x7f0cc40049f0 2022-07-20T03:09:15.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cd8111120 con 0x7f0cd81115c0 2022-07-20T03:09:15.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 0x7f0cd8159ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 0x7f0cd8159ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44136/0 (socket says 172.21.15.61:44136) 2022-07-20T03:09:15.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 -- 172.21.15.61:0/3307383189 learned_addr learned my addr 172.21.15.61:0/3307383189 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:15.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cde453700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc40049f0 0x7f0cd810e610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cddc52700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd81115c0 0x7f0cd8111a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 -- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd81115c0 msgr2=0x7f0cd8111a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:15.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd81115c0 0x7f0cd8111a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 -- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc40049f0 msgr2=0x7f0cd810e610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:15.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc40049f0 0x7f0cd810e610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 -- 172.21.15.61:0/3307383189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cd8066110 con 0x7f0cd8159a40 2022-07-20T03:09:15.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cddc52700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd81115c0 0x7f0cd8111a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:15.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cdec54700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 0x7f0cd8159ef0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0cd80ae470 tx=0x7f0cd0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd00070d0 con 0x7f0cd8159a40 2022-07-20T03:09:15.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cd0004920 con 0x7f0cd8159a40 2022-07-20T03:09:15.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cd810f7b0 con 0x7f0cd8159a40 2022-07-20T03:09:15.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cd0001da0 con 0x7f0cd8159a40 2022-07-20T03:09:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cd815a3e0 con 0x7f0cd8159a40 2022-07-20T03:09:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cd8067350 con 0x7f0cd8159a40 2022-07-20T03:09:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0cd0025020 con 0x7f0cd8159a40 2022-07-20T03:09:15.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cd77fe700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cb8053af0 0x7f0cb8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.987+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(34..34 src has 1..34) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0cd006c690 con 0x7f0cd8159a40 2022-07-20T03:09:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.991+0000 7f0cde453700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cb8053af0 0x7f0cb8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.991+0000 7f0cde453700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cb8053af0 0x7f0cb8055fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0ccc00b000 tx=0x7f0ccc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:15.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:14.995+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0cd8067350 con 0x7f0cd8159a40 2022-07-20T03:09:15.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.119+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f0cd81473e0 con 0x7f0cd8159a40 2022-07-20T03:09:15.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.963+0000 7f0cd77fe700 1 -- 172.21.15.61:0/3307383189 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v35) v1 ==== 185+0+0 (secure 0 0 0) 0x7f0cd81473e0 con 0x7f0cd8159a40 2022-07-20T03:09:15.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cb8053af0 msgr2=0x7f0cb8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:15.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cb8053af0 0x7f0cb8055fb0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f0ccc00b000 tx=0x7f0ccc006df0).stop 2022-07-20T03:09:15.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 msgr2=0x7f0cd8159ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:15.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 0x7f0cd8159ef0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f0cd80ae470 tx=0x7f0cd0015000).stop 2022-07-20T03:09:15.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 shutdown_connections 2022-07-20T03:09:15.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0cb8053af0 0x7f0cb8055fb0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cd8159a40 0x7f0cd8159ef0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cd81115c0 0x7f0cd8111a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 --2- 172.21.15.61:0/3307383189 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc40049f0 0x7f0cd810e610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:15.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 >> 172.21.15.61:0/3307383189 conn(0x7f0cd80635b0 msgr2=0x7f0cd8144960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:15.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 shutdown_connections 2022-07-20T03:09:15.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:15.967+0000 7f0ce06b7700 1 -- 172.21.15.61:0/3307383189 wait complete. 2022-07-20T03:09:15.998 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:09:16.000 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:09:16.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 Processor -- start 2022-07-20T03:09:16.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 -- start start 2022-07-20T03:09:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b1090 0x7fa46c13c610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa46c0b1b00 0x7fa46c13cb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa46c05d060 con 0x7fa46c142a10 2022-07-20T03:09:16.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa46c05ca80 con 0x7fa46c0b1b00 2022-07-20T03:09:16.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c067450 con 0x7fa46c0b1090 2022-07-20T03:09:16.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa463fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa46c142a10 0x7fa46c146280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40740/0 (socket says 172.21.15.61:40740) 2022-07-20T03:09:16.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa463fff700 1 -- 172.21.15.61:0/1494375535 learned_addr learned my addr 172.21.15.61:0/1494375535 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:16.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46b7fe700 1 --2- 172.21.15.61:0/1494375535 >> v2:172.21.15.61:3300/0 conn(0x7fa46c0b1b00 0x7fa46c13cb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:16.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46affd700 1 --2- 172.21.15.61:0/1494375535 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b1090 0x7fa46c13c610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:16.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46a7fc700 1 -- 172.21.15.61:0/1494375535 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 600715797 0 0) 0x7fa46c05d060 con 0x7fa46c142a10 2022-07-20T03:09:16.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46a7fc700 1 -- 172.21.15.61:0/1494375535 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa44c003260 con 0x7fa46c142a10 2022-07-20T03:09:16.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46b7fe700 1 -- 172.21.15.61:0/1494375535 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b1090 msgr2=0x7fa46c13c610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:16.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46b7fe700 1 --2- 172.21.15.61:0/1494375535 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b1090 0x7fa46c13c610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:16.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46b7fe700 1 -- 172.21.15.61:0/1494375535 >> v1:172.21.15.61:6789/0 conn(0x7fa46c142a10 legacy=0x7fa46c146280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:16.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46b7fe700 1 -- 172.21.15.61:0/1494375535 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa46c0676f0 con 0x7fa46c0b1b00 2022-07-20T03:09:16.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46b7fe700 1 --2- 172.21.15.61:0/1494375535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c13cb00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa4640032f0 tx=0x7fa4640091d0).ready entity=mon.0 client_cookie=8c6d5a078cb03841 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:16.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46a7fc700 1 -- 172.21.15.61:0/1494375535 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa46400b600 con 0x7fa46c0b1b00 2022-07-20T03:09:16.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46a7fc700 1 -- 172.21.15.61:0/1494375535 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa46400b760 con 0x7fa46c0b1b00 2022-07-20T03:09:16.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.135+0000 7fa46a7fc700 1 -- 172.21.15.61:0/1494375535 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa46400b8d0 con 0x7fa46c0b1b00 2022-07-20T03:09:16.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/1494375535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 msgr2=0x7fa46c13cb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:16.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 --2- 172.21.15.61:0/1494375535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c13cb00 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7fa4640032f0 tx=0x7fa4640091d0).stop 2022-07-20T03:09:16.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/1494375535 shutdown_connections 2022-07-20T03:09:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 --2- 172.21.15.61:0/1494375535 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c13cb00 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 --2- 172.21.15.61:0/1494375535 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b1090 0x7fa46c13c610 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/1494375535 >> 172.21.15.61:0/1494375535 conn(0x7fa46c064c00 msgr2=0x7fa46c064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:16.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/1494375535 shutdown_connections 2022-07-20T03:09:16.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/1494375535 wait complete. 2022-07-20T03:09:16.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 Processor -- start 2022-07-20T03:09:16.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- start start 2022-07-20T03:09:16.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa46c0b1090 0x7fa46c1445a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c137cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b4f90 0x7fa46c1381a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c139370 con 0x7fa46c0b1b00 2022-07-20T03:09:16.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c141b30 con 0x7fa46c0b1090 2022-07-20T03:09:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c141dd0 con 0x7fa46c0b4f90 2022-07-20T03:09:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c137cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c137cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44148/0 (socket says 172.21.15.61:44148) 2022-07-20T03:09:16.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 -- 172.21.15.61:0/519958805 learned_addr learned my addr 172.21.15.61:0/519958805 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:16.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46b7fe700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b4f90 0x7fa46c1381a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:16.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46affd700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa46c0b1090 0x7fa46c1445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:16.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 -- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b4f90 msgr2=0x7fa46c1381a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:16.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b4f90 0x7fa46c1381a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:16.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 -- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa46c0b1090 msgr2=0x7fa46c1445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:16.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa46c0b1090 0x7fa46c1445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:16.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 -- 172.21.15.61:0/519958805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa464009470 con 0x7fa46c0b1b00 2022-07-20T03:09:16.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa463fff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c137cb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fa454001110 tx=0x7fa454007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:16.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4540070d0 con 0x7fa46c0b1b00 2022-07-20T03:09:16.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa454007230 con 0x7fa46c0b1b00 2022-07-20T03:09:16.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c142070 con 0x7fa46c0b1b00 2022-07-20T03:09:16.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa454005610 con 0x7fa46c0b1b00 2022-07-20T03:09:16.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.139+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c13ddd0 con 0x7fa46c0b1b00 2022-07-20T03:09:16.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.143+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46c142200 con 0x7fa46c0b1b00 2022-07-20T03:09:16.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.143+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa454010070 con 0x7fa46c0b1b00 2022-07-20T03:09:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.143+0000 7fa468ff9700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa450053a40 0x7fa450055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.143+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(35..35 src has 1..35) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fa45406b360 con 0x7fa46c0b1b00 2022-07-20T03:09:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.143+0000 7fa46affd700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa450053a40 0x7fa450055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.147+0000 7fa46affd700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa450053a40 0x7fa450055f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa45c00b000 tx=0x7fa45c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:16.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.151+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa4540423c0 con 0x7fa46c0b1b00 2022-07-20T03:09:16.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:16.275+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa46c143b40 con 0x7fa46c0b1b00 2022-07-20T03:09:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.143+0000 7fa468ff9700 1 -- 172.21.15.61:0/519958805 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v36) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa454018020 con 0x7fa46c0b1b00 2022-07-20T03:09:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa450053a40 msgr2=0x7fa450055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa450053a40 0x7fa450055f00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fa45c00b000 tx=0x7fa45c006df0).stop 2022-07-20T03:09:17.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 msgr2=0x7fa46c137cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c137cb0 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fa454001110 tx=0x7fa454007aa0).stop 2022-07-20T03:09:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 shutdown_connections 2022-07-20T03:09:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa450053a40 0x7fa450055f00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa46c0b4f90 0x7fa46c1381a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa46c0b1b00 0x7fa46c137cb0 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 --2- 172.21.15.61:0/519958805 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa46c0b1090 0x7fa46c1445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 >> 172.21.15.61:0/519958805 conn(0x7fa46c064c00 msgr2=0x7fa46c13f8f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 shutdown_connections 2022-07-20T03:09:17.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.147+0000 7fa46bfff700 1 -- 172.21.15.61:0/519958805 wait complete. 2022-07-20T03:09:17.160 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:09:17.167 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:09:17.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.275+0000 7febf58a8700 1 Processor -- start 2022-07-20T03:09:17.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.275+0000 7febf58a8700 1 -- start start 2022-07-20T03:09:17.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7febf00b5990 0x7febf0146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf0146560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf00b5d70 con 0x7febf005d210 2022-07-20T03:09:17.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7febf00ad050 con 0x7febf00b5990 2022-07-20T03:09:17.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf00b17e0 con 0x7febf00af3e0 2022-07-20T03:09:17.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf50a7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7febf005d210 0x7febf014a360 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40756/0 (socket says 172.21.15.61:40756) 2022-07-20T03:09:17.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf50a7700 1 -- 172.21.15.61:0/1535603361 learned_addr learned my addr 172.21.15.61:0/1535603361 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:17.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf48a6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7febf00b5990 0x7febf0146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febeffff700 1 --2- 172.21.15.61:0/1535603361 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf0146560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febef7fe700 1 -- 172.21.15.61:0/1535603361 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1722256238 0 0) 0x7febf00b5d70 con 0x7febf005d210 2022-07-20T03:09:17.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febef7fe700 1 -- 172.21.15.61:0/1535603361 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd0003260 con 0x7febf005d210 2022-07-20T03:09:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf48a6700 1 -- 172.21.15.61:0/1535603361 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 msgr2=0x7febf0146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf48a6700 1 --2- 172.21.15.61:0/1535603361 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf0146560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf48a6700 1 -- 172.21.15.61:0/1535603361 >> v1:172.21.15.61:6789/0 conn(0x7febf005d210 legacy=0x7febf014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf48a6700 1 -- 172.21.15.61:0/1535603361 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf00b1010 con 0x7febf00b5990 2022-07-20T03:09:17.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf48a6700 1 --2- 172.21.15.61:0/1535603361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf0146070 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7febd8002f70 tx=0x7febd8006df0).ready entity=mon.0 client_cookie=935c702d6622185a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:17.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febef7fe700 1 -- 172.21.15.61:0/1535603361 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd800e470 con 0x7febf00b5990 2022-07-20T03:09:17.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febef7fe700 1 -- 172.21.15.61:0/1535603361 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febd8000e90 con 0x7febf00b5990 2022-07-20T03:09:17.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febef7fe700 1 -- 172.21.15.61:0/1535603361 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febd800e7c0 con 0x7febf00b5990 2022-07-20T03:09:17.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- 172.21.15.61:0/1535603361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 msgr2=0x7febf0146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 --2- 172.21.15.61:0/1535603361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf0146070 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7febd8002f70 tx=0x7febd8006df0).stop 2022-07-20T03:09:17.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- 172.21.15.61:0/1535603361 shutdown_connections 2022-07-20T03:09:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 --2- 172.21.15.61:0/1535603361 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf0146560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 --2- 172.21.15.61:0/1535603361 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf0146070 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- 172.21.15.61:0/1535603361 >> 172.21.15.61:0/1535603361 conn(0x7febf0063480 msgr2=0x7febf00b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:17.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- 172.21.15.61:0/1535603361 shutdown_connections 2022-07-20T03:09:17.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.279+0000 7febf58a8700 1 -- 172.21.15.61:0/1535603361 wait complete. 2022-07-20T03:09:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 Processor -- start 2022-07-20T03:09:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- start start 2022-07-20T03:09:17.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf005d210 0x7febf010ed10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf01596b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf010e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf0157360 con 0x7febf00b5990 2022-07-20T03:09:17.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf010f200 con 0x7febf005d210 2022-07-20T03:09:17.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf0111970 con 0x7febf00af3e0 2022-07-20T03:09:17.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf010e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf010e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44164/0 (socket says 172.21.15.61:44164) 2022-07-20T03:09:17.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 -- 172.21.15.61:0/3857154865 learned_addr learned my addr 172.21.15.61:0/3857154865 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:17.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf48a6700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf005d210 0x7febf010ed10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febeffff700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf01596b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 -- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 msgr2=0x7febf01596b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf01596b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 -- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf005d210 msgr2=0x7febf010ed10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf005d210 0x7febf010ed10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 -- 172.21.15.61:0/3857154865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd800b120 con 0x7febf00b5990 2022-07-20T03:09:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf50a7700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf010e130 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7febf01447c0 tx=0x7febe8007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:17.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febe8005ba0 con 0x7febf00b5990 2022-07-20T03:09:17.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe8005d00 con 0x7febf00b5990 2022-07-20T03:09:17.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf010f440 con 0x7febf00b5990 2022-07-20T03:09:17.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febe80072c0 con 0x7febf00b5990 2022-07-20T03:09:17.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf0159ba0 con 0x7febf00b5990 2022-07-20T03:09:17.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.283+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf0144fa0 con 0x7febf00b5990 2022-07-20T03:09:17.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.287+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7febe8010070 con 0x7febf00b5990 2022-07-20T03:09:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.287+0000 7febedffb700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd4053ad0 0x7febd4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.287+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7febe806b4f0 con 0x7febf00b5990 2022-07-20T03:09:17.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.287+0000 7febf48a6700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd4053ad0 0x7febd4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.287+0000 7febf48a6700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd4053ad0 0x7febd4055f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7febd8018000 tx=0x7febd8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:17.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.291+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febe8042730 con 0x7febf00b5990 2022-07-20T03:09:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.419+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7febf00afa60 con 0x7febf00b5990 2022-07-20T03:09:17.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febedffb700 1 -- 172.21.15.61:0/3857154865 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v3) v1 ==== 104+0+0 (secure 0 0 0) 0x7febe8018020 con 0x7febf00b5990 2022-07-20T03:09:17.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd4053ad0 msgr2=0x7febd4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd4053ad0 0x7febd4055f90 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7febd8018000 tx=0x7febd8014000).stop 2022-07-20T03:09:17.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 msgr2=0x7febf010e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf010e130 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7febf01447c0 tx=0x7febe8007d60).stop 2022-07-20T03:09:17.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 shutdown_connections 2022-07-20T03:09:17.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd4053ad0 0x7febd4055f90 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf00b5990 0x7febf010e130 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf00af3e0 0x7febf01596b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 --2- 172.21.15.61:0/3857154865 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf005d210 0x7febf010ed10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 >> 172.21.15.61:0/3857154865 conn(0x7febf0063480 msgr2=0x7febf014a9c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 shutdown_connections 2022-07-20T03:09:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.427+0000 7febf58a8700 1 -- 172.21.15.61:0/3857154865 wait complete. 2022-07-20T03:09:17.542 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command' 2022-07-20T03:09:17.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd484939700 1 Processor -- start 2022-07-20T03:09:17.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd484939700 1 -- start start 2022-07-20T03:09:17.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd484939700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd480143330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd484939700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4800623f0 con 0x7fd4800ac540 2022-07-20T03:09:17.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd484939700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd480143ae0 con 0x7fd4800b34d0 2022-07-20T03:09:17.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd484939700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd480143d80 con 0x7fd480142ee0 2022-07-20T03:09:17.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47dd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd4800ac540 0x7fd48013f910 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40768/0 (socket says 172.21.15.61:40768) 2022-07-20T03:09:17.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47dd9b700 1 -- 172.21.15.61:0/62002278 learned_addr learned my addr 172.21.15.61:0/62002278 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:17.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47ed9d700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd480143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2557521037 0 0) 0x7fd478001490 con 0x7fd4800ac540 2022-07-20T03:09:17.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd460003260 con 0x7fd4800ac540 2022-07-20T03:09:17.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 602925657 0 0) 0x7fd480143ae0 con 0x7fd4800b34d0 2022-07-20T03:09:17.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd478001490 con 0x7fd4800b34d0 2022-07-20T03:09:17.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1382521651 0 0) 0x7fd460003260 con 0x7fd4800ac540 2022-07-20T03:09:17.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 msgr2=0x7fd480143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:17.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd480143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 >> v1:172.21.15.117:6789/0 conn(0x7fd4800b34d0 legacy=0x7fd4800997a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd480099300 con 0x7fd4800ac540 2022-07-20T03:09:17.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fd478001b90 con 0x7fd4800ac540 2022-07-20T03:09:17.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47ed9d700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd480143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:17.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 >> v1:172.21.15.61:6789/0 conn(0x7fd4800ac540 legacy=0x7fd48013f910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd460003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd460004500 0x7fd460004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd460004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd480143d80 con 0x7fd460003910 2022-07-20T03:09:17.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd46000b750 con 0x7fd460004500 2022-07-20T03:09:17.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd46000b910 con 0x7fd480142ee0 2022-07-20T03:09:17.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd460003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47ed9d700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd460004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47dd9b700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd460004500 0x7fd460004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 -- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 msgr2=0x7fd460004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480142ee0 0x7fd460004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 -- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd460004500 msgr2=0x7fd460004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:17.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd460004500 0x7fd460004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 -- 172.21.15.61:0/62002278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd47000b180 con 0x7fd460003910 2022-07-20T03:09:17.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47dd9b700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd460004500 0x7fd460004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:17.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47e59c700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd460003d10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd468001110 tx=0x7fd468007a00).ready entity=mon.0 client_cookie=1f354e534c7dc04d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:17.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd468007d10 con 0x7fd460003910 2022-07-20T03:09:17.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.643+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd468007030 con 0x7fd460003910 2022-07-20T03:09:17.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47d59a700 1 -- 172.21.15.61:0/62002278 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd468007390 con 0x7fd460003910 2022-07-20T03:09:17.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- 172.21.15.61:0/62002278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 msgr2=0x7fd460003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 --2- 172.21.15.61:0/62002278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd460003d10 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7fd468001110 tx=0x7fd468007a00).stop 2022-07-20T03:09:17.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47ed9d700 1 -- 172.21.15.61:0/62002278 reap_dead start 2022-07-20T03:09:17.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- 172.21.15.61:0/62002278 shutdown_connections 2022-07-20T03:09:17.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- 172.21.15.61:0/62002278 >> 172.21.15.61:0/62002278 conn(0x7fd4800b6c50 msgr2=0x7fd4800b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:17.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- 172.21.15.61:0/62002278 shutdown_connections 2022-07-20T03:09:17.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- 172.21.15.61:0/62002278 wait complete. 2022-07-20T03:09:17.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 Processor -- start 2022-07-20T03:09:17.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- start start 2022-07-20T03:09:17.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd48013ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480063a40 0x7fd48009a000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4800adbf0 0x7fd4800ae5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48013c150 con 0x7fd460003910 2022-07-20T03:09:17.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd48009be90 con 0x7fd4800adbf0 2022-07-20T03:09:17.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd484939700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd480063fc0 con 0x7fd480063a40 2022-07-20T03:09:17.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd48013ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd48013ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44180/0 (socket says 172.21.15.61:44180) 2022-07-20T03:09:17.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47e59c700 1 -- 172.21.15.61:0/1051001478 learned_addr learned my addr 172.21.15.61:0/1051001478 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:17.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47dd9b700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480063a40 0x7fd48009a000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.647+0000 7fd47ed9d700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4800adbf0 0x7fd4800ae5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47e59c700 1 -- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480063a40 msgr2=0x7fd48009a000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47e59c700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480063a40 0x7fd48009a000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47e59c700 1 -- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4800adbf0 msgr2=0x7fd4800ae5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47e59c700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4800adbf0 0x7fd4800ae5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47e59c700 1 -- 172.21.15.61:0/1051001478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd480064260 con 0x7fd460003910 2022-07-20T03:09:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47e59c700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd48013ed40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fd468001110 tx=0x7fd468016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd46801f620 con 0x7fd460003910 2022-07-20T03:09:17.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd46801f780 con 0x7fd460003910 2022-07-20T03:09:17.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd48009a4f0 con 0x7fd460003910 2022-07-20T03:09:17.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd46801fae0 con 0x7fd460003910 2022-07-20T03:09:17.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4800aecc0 con 0x7fd460003910 2022-07-20T03:09:17.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd480063e20 con 0x7fd460003910 2022-07-20T03:09:17.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd468025020 con 0x7fd460003910 2022-07-20T03:09:17.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd4777fe700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd464053a60 0x7fd464055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:17.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd46806cb10 con 0x7fd460003910 2022-07-20T03:09:17.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.651+0000 7fd47dd9b700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd464053a60 0x7fd464055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:17.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.655+0000 7fd47dd9b700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd464053a60 0x7fd464055f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fd47800d000 tx=0x7fd4800aba40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:17.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.655+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd480063e20 con 0x7fd460003910 2022-07-20T03:09:17.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:17.827+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]} v 0) v1 -- 0x7fd480063e20 con 0x7fd460003910 2022-07-20T03:09:18.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.147+0000 7fd4777fe700 1 -- 172.21.15.61:0/1051001478 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command"]}]=0 v115) v1 ==== 139+0+0 (secure 0 0 0) 0x7fd480063e20 con 0x7fd460003910 2022-07-20T03:09:18.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd464053a60 msgr2=0x7fd464055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd464053a60 0x7fd464055f20 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7fd47800d000 tx=0x7fd4800aba40).stop 2022-07-20T03:09:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 msgr2=0x7fd48013ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd48013ed40 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7fd468001110 tx=0x7fd468016000).stop 2022-07-20T03:09:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 shutdown_connections 2022-07-20T03:09:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd464053a60 0x7fd464055f20 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4800adbf0 0x7fd4800ae5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd480063a40 0x7fd48009a000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 --2- 172.21.15.61:0/1051001478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd460003910 0x7fd48013ed40 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 >> 172.21.15.61:0/1051001478 conn(0x7fd4800b6c50 msgr2=0x7fd4800b2a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:18.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 shutdown_connections 2022-07-20T03:09:18.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.151+0000 7fd484939700 1 -- 172.21.15.61:0/1051001478 wait complete. 2022-07-20T03:09:18.175 INFO:tasks.cephfs_test_runner:test_drop_cache_command (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T03:09:18.176 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T03:09:18.177 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-20T03:09:18.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.315+0000 7fce5f390700 1 Processor -- start 2022-07-20T03:09:18.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.315+0000 7fce5f390700 1 -- start start 2022-07-20T03:09:18.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce5805c0d0 0x7fce580b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:18.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce5805c4b0 con 0x7fce5805d200 2022-07-20T03:09:18.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce580b3da0 con 0x7fce581439c0 2022-07-20T03:09:18.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce580b4040 con 0x7fce5805c0d0 2022-07-20T03:09:18.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5e38e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce5805c0d0 0x7fce580b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:18.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5db8d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fce5805d200 0x7fce581432b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40786/0 (socket says 172.21.15.61:40786) 2022-07-20T03:09:18.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5db8d700 1 -- 172.21.15.61:0/2699428215 learned_addr learned my addr 172.21.15.61:0/2699428215 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:18.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5d38c700 1 -- 172.21.15.61:0/2699428215 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1170235344 0 0) 0x7fce5805c4b0 con 0x7fce5805d200 2022-07-20T03:09:18.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5d38c700 1 -- 172.21.15.61:0/2699428215 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fce40003260 con 0x7fce5805d200 2022-07-20T03:09:18.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5e38e700 1 -- 172.21.15.61:0/2699428215 >> v1:172.21.15.117:6789/0 conn(0x7fce581439c0 legacy=0x7fce58146440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5e38e700 1 -- 172.21.15.61:0/2699428215 >> v1:172.21.15.61:6789/0 conn(0x7fce5805d200 legacy=0x7fce581432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5e38e700 1 -- 172.21.15.61:0/2699428215 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce58142ad0 con 0x7fce5805c0d0 2022-07-20T03:09:18.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5e38e700 1 --2- 172.21.15.61:0/2699428215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce580b3770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fce54002f70 tx=0x7fce54006df0).ready entity=mon.0 client_cookie=6f5b877b06cad506 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:18.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5d38c700 1 -- 172.21.15.61:0/2699428215 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce5400d610 con 0x7fce5805c0d0 2022-07-20T03:09:18.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5d38c700 1 -- 172.21.15.61:0/2699428215 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce54000e90 con 0x7fce5805c0d0 2022-07-20T03:09:18.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5d38c700 1 -- 172.21.15.61:0/2699428215 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce5400d960 con 0x7fce5805c0d0 2022-07-20T03:09:18.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- 172.21.15.61:0/2699428215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 msgr2=0x7fce580b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 --2- 172.21.15.61:0/2699428215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce580b3770 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fce54002f70 tx=0x7fce54006df0).stop 2022-07-20T03:09:18.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- 172.21.15.61:0/2699428215 shutdown_connections 2022-07-20T03:09:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 --2- 172.21.15.61:0/2699428215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce580b3770 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- 172.21.15.61:0/2699428215 >> 172.21.15.61:0/2699428215 conn(0x7fce58064b90 msgr2=0x7fce58064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:18.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- 172.21.15.61:0/2699428215 shutdown_connections 2022-07-20T03:09:18.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.319+0000 7fce5f390700 1 -- 172.21.15.61:0/2699428215 wait complete. 2022-07-20T03:09:18.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 Processor -- start 2022-07-20T03:09:18.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 -- start start 2022-07-20T03:09:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce58148fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:18.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce5805d200 0x7fce58149b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce581439c0 0x7fce580ae500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:18.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce58147fe0 con 0x7fce5805c0d0 2022-07-20T03:09:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce58155e80 con 0x7fce581439c0 2022-07-20T03:09:18.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce58156120 con 0x7fce5805d200 2022-07-20T03:09:18.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce58148fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:18.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce58148fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44196/0 (socket says 172.21.15.61:44196) 2022-07-20T03:09:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 -- 172.21.15.61:0/1773348803 learned_addr learned my addr 172.21.15.61:0/1773348803 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5eb8f700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce581439c0 0x7fce580ae500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5db8d700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce5805d200 0x7fce58149b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 -- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce5805d200 msgr2=0x7fce58149b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce5805d200 0x7fce58149b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 -- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce581439c0 msgr2=0x7fce580ae500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce581439c0 0x7fce580ae500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 -- 172.21.15.61:0/1773348803 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce5400b180 con 0x7fce5805c0d0 2022-07-20T03:09:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5db8d700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce5805d200 0x7fce58149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5e38e700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce58148fa0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fce54002f70 tx=0x7fce54014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:18.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce5400d290 con 0x7fce5805c0d0 2022-07-20T03:09:18.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce54013e90 con 0x7fce5805c0d0 2022-07-20T03:09:18.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce5401e5f0 con 0x7fce5805c0d0 2022-07-20T03:09:18.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce580ae9f0 con 0x7fce5805c0d0 2022-07-20T03:09:18.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.323+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce581483e0 con 0x7fce5805c0d0 2022-07-20T03:09:18.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.327+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce581562e0 con 0x7fce5805c0d0 2022-07-20T03:09:18.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.327+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fce54025030 con 0x7fce5805c0d0 2022-07-20T03:09:18.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.327+0000 7fce477fe700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce380539b0 0x7fce38055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:18.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.327+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fce5406cab0 con 0x7fce5805c0d0 2022-07-20T03:09:18.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.327+0000 7fce5db8d700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce380539b0 0x7fce38055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:18.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.327+0000 7fce5db8d700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce380539b0 0x7fce38055e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fce4800b000 tx=0x7fce48009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:18.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.331+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fce54043be0 con 0x7fce5805c0d0 2022-07-20T03:09:18.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:18.499+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7fce580699b0 con 0x7fce5805c0d0 2022-07-20T03:09:19.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.151+0000 7fce477fe700 1 -- 172.21.15.61:0/1773348803 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v116) v1 ==== 147+0+0 (secure 0 0 0) 0x7fce54023070 con 0x7fce5805c0d0 2022-07-20T03:09:19.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce380539b0 msgr2=0x7fce38055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce380539b0 0x7fce38055e70 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fce4800b000 tx=0x7fce48009040).stop 2022-07-20T03:09:19.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 msgr2=0x7fce58148fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce58148fa0 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fce54002f70 tx=0x7fce54014000).stop 2022-07-20T03:09:19.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 shutdown_connections 2022-07-20T03:09:19.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce380539b0 0x7fce38055e70 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce581439c0 0x7fce580ae500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce5805d200 0x7fce58149b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 --2- 172.21.15.61:0/1773348803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce5805c0d0 0x7fce58148fa0 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 >> 172.21.15.61:0/1773348803 conn(0x7fce58064b90 msgr2=0x7fce58066750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:19.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 shutdown_connections 2022-07-20T03:09:19.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.155+0000 7fce5f390700 1 -- 172.21.15.61:0/1773348803 wait complete. 2022-07-20T03:09:19.287 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:09:19.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 Processor -- start 2022-07-20T03:09:19.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 -- start start 2022-07-20T03:09:19.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fded005d0e0 0x7fded0146920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fded00b5960 0x7fded0142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded0143060 0x7fded009b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded005d4c0 con 0x7fded005d0e0 2022-07-20T03:09:19.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fded01461e0 con 0x7fded00b5960 2022-07-20T03:09:19.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.415+0000 7fded81a9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded0146480 con 0x7fded0143060 2022-07-20T03:09:19.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fded005d0e0 0x7fded0146920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fded005d0e0 0x7fded0146920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44202/0 (socket says 172.21.15.61:44202) 2022-07-20T03:09:19.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 -- 172.21.15.61:0/2794866553 learned_addr learned my addr 172.21.15.61:0/2794866553 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:19.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded6746700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded0143060 0x7fded009b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5744700 1 --2- 172.21.15.61:0/2794866553 >> v2:172.21.15.117:3300/0 conn(0x7fded00b5960 0x7fded0142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 -- 172.21.15.61:0/2794866553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded0143060 msgr2=0x7fded009b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:19.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded0143060 0x7fded009b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 -- 172.21.15.61:0/2794866553 >> v2:172.21.15.117:3300/0 conn(0x7fded00b5960 msgr2=0x7fded0142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:19.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 --2- 172.21.15.61:0/2794866553 >> v2:172.21.15.117:3300/0 conn(0x7fded00b5960 0x7fded0142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 -- 172.21.15.61:0/2794866553 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fded00b01d0 con 0x7fded005d0e0 2022-07-20T03:09:19.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded6746700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded0143060 0x7fded009b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:19.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded5f45700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded005d0e0 0x7fded0146920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fdec0002f70 tx=0x7fdec0006eb0).ready entity=mon.0 client_cookie=af2c66b7db689a10 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:19.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded4f43700 1 -- 172.21.15.61:0/2794866553 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec000d570 con 0x7fded005d0e0 2022-07-20T03:09:19.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded4f43700 1 -- 172.21.15.61:0/2794866553 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec0000e90 con 0x7fded005d0e0 2022-07-20T03:09:19.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded4f43700 1 -- 172.21.15.61:0/2794866553 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec000d900 con 0x7fded005d0e0 2022-07-20T03:09:19.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- 172.21.15.61:0/2794866553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded005d0e0 msgr2=0x7fded0146920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded005d0e0 0x7fded0146920 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fdec0002f70 tx=0x7fdec0006eb0).stop 2022-07-20T03:09:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- 172.21.15.61:0/2794866553 shutdown_connections 2022-07-20T03:09:19.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded0143060 0x7fded009b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- 172.21.15.61:0/2794866553 >> v2:172.21.15.117:3300/0 conn(0x7fded00b5960 0x7fded0142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- 172.21.15.61:0/2794866553 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded005d0e0 0x7fded0146920 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- 172.21.15.61:0/2794866553 >> 172.21.15.61:0/2794866553 conn(0x7fded00b6a20 msgr2=0x7fded0066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:19.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- 172.21.15.61:0/2794866553 shutdown_connections 2022-07-20T03:09:19.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- 172.21.15.61:0/2794866553 wait complete. 2022-07-20T03:09:19.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 Processor -- start 2022-07-20T03:09:19.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- start start 2022-07-20T03:09:19.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded005d0e0 0x7fded01408e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fded00b5960 0x7fded0155a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 0x7fded009e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded00b3080 con 0x7fded0143060 2022-07-20T03:09:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fded009ddf0 con 0x7fded00b5960 2022-07-20T03:09:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded81a9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fded0155310 con 0x7fded005d0e0 2022-07-20T03:09:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded6746700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 0x7fded009e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded6746700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 0x7fded009e050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44210/0 (socket says 172.21.15.61:44210) 2022-07-20T03:09:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.419+0000 7fded6746700 1 -- 172.21.15.61:0/2109794572 learned_addr learned my addr 172.21.15.61:0/2109794572 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded5f45700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded005d0e0 0x7fded01408e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded5744700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fded00b5960 0x7fded0155a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded6746700 1 -- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded005d0e0 msgr2=0x7fded01408e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded6746700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded005d0e0 0x7fded01408e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded6746700 1 -- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fded00b5960 msgr2=0x7fded0155a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded6746700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fded00b5960 0x7fded0155a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded6746700 1 -- 172.21.15.61:0/2109794572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdec000b1e0 con 0x7fded0143060 2022-07-20T03:09:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded6746700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 0x7fded009e050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fdec800a000 tx=0x7fdec8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec800f710 con 0x7fded0143060 2022-07-20T03:09:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fded01555b0 con 0x7fded0143060 2022-07-20T03:09:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdec800f870 con 0x7fded0143060 2022-07-20T03:09:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fded00b0eb0 con 0x7fded0143060 2022-07-20T03:09:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdec8005c60 con 0x7fded0143060 2022-07-20T03:09:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeac003410 con 0x7fded0143060 2022-07-20T03:09:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdec801a030 con 0x7fded0143060 2022-07-20T03:09:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fdeceffd700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdeb00539b0 0x7fdeb0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.423+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fdec806bb70 con 0x7fded0143060 2022-07-20T03:09:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.427+0000 7fded5f45700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdeb00539b0 0x7fdeb0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:19.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.427+0000 7fded5f45700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdeb00539b0 0x7fdeb0055e70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdec001c000 tx=0x7fdec0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:19.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.431+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdec8042d10 con 0x7fded0143060 2022-07-20T03:09:19.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.563+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fdeac003c40 con 0x7fded0143060 2022-07-20T03:09:19.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fdeceffd700 1 -- 172.21.15.61:0/2109794572 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v4)=0 v4) v1 ==== 122+0+0 (secure 0 0 0) 0x7fdec8046330 con 0x7fded0143060 2022-07-20T03:09:19.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdeb00539b0 msgr2=0x7fdeb0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdeb00539b0 0x7fdeb0055e70 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7fdec001c000 tx=0x7fdec0015000).stop 2022-07-20T03:09:19.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 msgr2=0x7fded009e050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:19.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 0x7fded009e050 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fdec800a000 tx=0x7fdec8008040).stop 2022-07-20T03:09:19.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 shutdown_connections 2022-07-20T03:09:19.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdeb00539b0 0x7fdeb0055e70 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fded00b5960 0x7fded0155a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fded0143060 0x7fded009e050 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 --2- 172.21.15.61:0/2109794572 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fded005d0e0 0x7fded01408e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:19.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 >> 172.21.15.61:0/2109794572 conn(0x7fded00b6a20 msgr2=0x7fded00b79c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 shutdown_connections 2022-07-20T03:09:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:19.571+0000 7fded81a9700 1 -- 172.21.15.61:0/2109794572 wait complete. 2022-07-20T03:09:19.621 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:09:19.622 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:19.705 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:19.823 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:19.880 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:19.925 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:09:19.948 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:09:19.949 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:19.950 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:19.952 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:19.965 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:19.966 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:09:19.966 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:09:19.967 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:19.967 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:19.979 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:09:19.997 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:19.999 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:09:20.007 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:09:20.009 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:20.010 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:20.011 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:09:20.043 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:20.044 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:09:20.045 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:09:20.045 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:09:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 Processor -- start 2022-07-20T03:09:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- start start 2022-07-20T03:09:20.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00b1e00 0x7fe0d014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:20.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d005d480 con 0x7fe0d00acef0 2022-07-20T03:09:20.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0d00b21e0 con 0x7fe0d005c3f0 2022-07-20T03:09:20.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d005cf40 con 0x7fe0d00b1e00 2022-07-20T03:09:20.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ced9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe0d00acef0 0x7fe0d0146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40818/0 (socket says 172.21.15.61:40818) 2022-07-20T03:09:20.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ced9d700 1 -- 172.21.15.61:0/4215391605 learned_addr learned my addr 172.21.15.61:0/4215391605 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:20.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ce59c700 1 --2- 172.21.15.61:0/4215391605 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00b1e00 0x7fe0d014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:20.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648232611 0 0) 0x7fe0d00b21e0 con 0x7fe0d005c3f0 2022-07-20T03:09:20.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0b0003260 con 0x7fe0d005c3f0 2022-07-20T03:09:20.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3900252640 0 0) 0x7fe0d005d480 con 0x7fe0d00acef0 2022-07-20T03:09:20.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe0d00b21e0 con 0x7fe0d00acef0 2022-07-20T03:09:20.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ce59c700 1 -- 172.21.15.61:0/4215391605 >> v1:172.21.15.117:6789/0 conn(0x7fe0d005c3f0 legacy=0x7fe0d00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:20.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ce59c700 1 -- 172.21.15.61:0/4215391605 >> v1:172.21.15.61:6789/0 conn(0x7fe0d00acef0 legacy=0x7fe0d0146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:20.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ce59c700 1 -- 172.21.15.61:0/4215391605 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0d005d1e0 con 0x7fe0d00b1e00 2022-07-20T03:09:20.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0ce59c700 1 --2- 172.21.15.61:0/4215391605 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00b1e00 0x7fe0d014a6d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe0c0002f70 tx=0x7fe0c0006df0).ready entity=mon.2 client_cookie=4f72d776bb75024 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:20.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c000d5c0 con 0x7fe0d00b1e00 2022-07-20T03:09:20.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c0000e90 con 0x7fe0d00b1e00 2022-07-20T03:09:20.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0cdd9b700 1 -- 172.21.15.61:0/4215391605 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c000d910 con 0x7fe0d00b1e00 2022-07-20T03:09:20.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/4215391605 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00b1e00 msgr2=0x7fe0d014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:20.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/4215391605 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00b1e00 0x7fe0d014a6d0 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7fe0c0002f70 tx=0x7fe0c0006df0).stop 2022-07-20T03:09:20.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/4215391605 shutdown_connections 2022-07-20T03:09:20.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/4215391605 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00b1e00 0x7fe0d014a6d0 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:20.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/4215391605 >> 172.21.15.61:0/4215391605 conn(0x7fe0d00611c0 msgr2=0x7fe0d0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:20.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/4215391605 shutdown_connections 2022-07-20T03:09:20.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.191+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/4215391605 wait complete. 2022-07-20T03:09:20.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 Processor -- start 2022-07-20T03:09:20.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 -- start start 2022-07-20T03:09:20.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00acef0 0x7fe0d00ad8c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:20.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0d00b1e00 0x7fe0d01432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:20.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 0x7fe0d0064250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:20.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d0065fa0 con 0x7fe0d0063da0 2022-07-20T03:09:20.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d00ae130 con 0x7fe0d00b1e00 2022-07-20T03:09:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0d014a490 con 0x7fe0d00acef0 2022-07-20T03:09:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 0x7fe0d0064250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 0x7fe0d0064250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44226/0 (socket says 172.21.15.61:44226) 2022-07-20T03:09:20.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 -- 172.21.15.61:0/3403817389 learned_addr learned my addr 172.21.15.61:0/3403817389 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:20.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ce59c700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00acef0 0x7fe0d00ad8c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:20.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0c7fff700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0d00b1e00 0x7fe0d01432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:20.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 -- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00acef0 msgr2=0x7fe0d00ad8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:20.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00acef0 0x7fe0d00ad8c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:20.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 -- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0d00b1e00 msgr2=0x7fe0d01432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:20.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0d00b1e00 0x7fe0d01432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:20.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 -- 172.21.15.61:0/3403817389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0c000b180 con 0x7fe0d0063da0 2022-07-20T03:09:20.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0c7fff700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0d00b1e00 0x7fe0d01432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:20.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0ced9d700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 0x7fe0d0064250 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fe0c80013c0 tx=0x7fe0c8007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:20.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c8005ba0 con 0x7fe0d0063da0 2022-07-20T03:09:20.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0c8005d00 con 0x7fe0d0063da0 2022-07-20T03:09:20.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0d014a6f0 con 0x7fe0d0063da0 2022-07-20T03:09:20.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0c80072c0 con 0x7fe0d0063da0 2022-07-20T03:09:20.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.195+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0d01499a0 con 0x7fe0d0063da0 2022-07-20T03:09:20.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.199+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0d00b04b0 con 0x7fe0d0063da0 2022-07-20T03:09:20.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.199+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe0c801a060 con 0x7fe0d0063da0 2022-07-20T03:09:20.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.199+0000 7fe0c77fe700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0b4053a40 0x7fe0b4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:20.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.199+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe0c806b2c0 con 0x7fe0d0063da0 2022-07-20T03:09:20.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.199+0000 7fe0ce59c700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0b4053a40 0x7fe0b4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:20.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.199+0000 7fe0ce59c700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0b4053a40 0x7fe0b4055f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe0c001a000 tx=0x7fe0c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:20.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.207+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe0c80424a0 con 0x7fe0d0063da0 2022-07-20T03:09:20.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.331+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fe0d00574e0 con 0x7fe0d0063da0 2022-07-20T03:09:20.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0c77fe700 1 -- 172.21.15.61:0/3403817389 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v36) v1 ==== 81+0+28875 (secure 0 0 0) 0x7fe0c8018090 con 0x7fe0d0063da0 2022-07-20T03:09:20.350 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:20.386 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:09:20.387 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 36, 2022-07-20T03:09:20.387 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:09:20.388 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:09:20.388 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:09:17.135654+0000", 2022-07-20T03:09:20.389 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:09:20.389 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:09:20.390 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:09:20.390 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:09:20.391 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:09:20.392 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:09:20.392 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:09:20.393 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:09:20.393 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:09:20.394 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:20.395 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:09:20.396 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:09:20.404 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:09:20.406 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:09:20.406 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:09:20.407 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 6, 2022-07-20T03:09:20.407 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:09:20.408 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:09:20.408 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:09:20.409 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:09:20.409 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:09:20.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.410 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:09:20.411 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:09:20.411 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:09:20.412 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:09:20.412 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:09:20.413 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:09:20.413 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:09:20.414 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:09:20.414 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:09:20.415 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:09:20.415 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:09:20.416 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:09:20.416 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:09:20.417 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:09:20.417 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:09:20.424 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:09:20.425 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:09:20.426 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:09:20.426 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:09:20.427 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:09:20.427 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:09:20.428 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:09:20.428 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:09:20.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.430 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:09:20.430 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:09:20.431 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:09:20.431 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:09:20.432 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:09:20.433 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:09:20.434 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:09:20.434 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:09:20.452 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:09:20.453 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:09:20.453 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:09:20.454 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:09:20.454 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:09:20.455 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:09:20.455 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:09:20.456 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:09:20.456 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:09:20.457 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:09:20.457 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:09:20.458 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:09:20.465 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:09:20.466 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:09:20.467 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:09:20.468 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:09:20.468 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:09:20.469 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:09:20.469 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:09:20.486 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.486 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:09:20.487 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:09:20.487 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:09:20.487 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:09:20.488 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:09:20.488 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:09:20.488 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:09:20.488 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:09:20.488 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:09:20.489 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:09:20.489 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:09:20.489 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:09:20.489 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:09:20.490 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.490 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:09:20.490 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:09:20.490 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.491 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.491 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.491 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:09:20.491 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:09:20.492 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:09:20.492 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:09:20.493 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:09:20.493 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:09:20.498 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:09:20.498 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:09:20.508 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:09:20.511 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:09:20.512 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:09:20.512 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:09:20.513 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:09:20.513 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:09:20.538 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:09:20.539 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:09:20.539 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:09:20.540 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:09:20.540 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:09:20.541 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:09:20.541 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:09:20.561 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:09:20.566 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:09:20.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.568 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:09:20.569 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:09:20.569 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:09:20.569 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:09:20.569 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:09:20.570 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:09:20.570 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:09:20.570 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:09:20.570 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:09:20.571 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:09:20.571 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:09:20.571 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:09:20.571 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:09:20.572 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:09:20.572 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:09:20.572 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:09:20.572 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:09:20.573 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:09:20.573 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:09:20.573 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:09:20.573 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:09:20.573 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:09:20.583 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:09:20.583 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:09:20.586 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:09:20.587 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:09:20.587 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:09:20.588 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.588 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:09:20.588 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:09:20.589 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:09:20.589 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:09:20.590 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:09:20.590 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:09:20.591 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:09:20.591 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:09:20.592 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:09:20.592 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:09:20.593 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:09:20.594 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:09:20.594 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:09:20.594 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:09:20.595 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.595 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.596 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:20.597 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:09:20.597 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.598 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:09:20.599 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:09:20.599 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:20.599 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:20.600 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:20.600 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:20.601 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:20.601 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:20.620 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:20.621 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:20.621 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:20.622 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:20.622 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:20.623 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.623 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.625 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.626 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:09:20.626 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.627 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.627 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.627 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.628 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:09:20.628 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.629 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.629 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.630 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.631 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:20.631 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.631 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.632 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.632 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:09:20.633 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.633 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.633 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.634 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.634 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:09:20.634 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.635 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.636 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.636 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.637 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:20.637 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.638 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.638 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.639 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:09:20.640 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.640 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.640 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.641 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.641 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:09:20.642 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.642 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.643 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.643 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.643 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:20.644 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.644 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.645 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.645 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:09:20.647 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.647 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.657 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.657 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.658 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:09:20.658 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:09:20.659 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.659 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.660 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.660 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:09:20.661 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:09:20.661 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:09:20.669 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:09:20.670 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:20.670 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:20.671 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:20.671 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.672 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.672 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.673 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:09:20.673 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:09:20.674 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:20.674 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:20.675 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:20.675 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:20.676 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:20.676 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:20.677 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:20.677 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:20.678 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:20.679 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:20.679 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:20.694 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.695 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.718 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.718 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:09:20.718 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.719 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.719 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.720 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.721 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:09:20.736 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.737 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.737 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.738 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.738 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:20.776 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.776 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.776 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.777 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:09:20.777 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.784 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.785 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.786 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:09:20.786 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.787 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.787 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.788 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.788 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:20.838 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.839 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.839 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.840 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:09:20.840 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.841 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.841 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.874 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.875 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:09:20.875 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.876 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.876 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.881 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:20.882 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.883 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.883 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.883 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:09:20.884 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.884 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.885 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.885 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.885 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:09:20.886 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:09:20.886 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.887 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.887 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.887 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:09:20.888 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:09:20.888 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:09:20.889 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:09:20.889 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:20.889 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:20.895 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:20.895 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.897 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:09:20.897 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:09:20.898 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:20.898 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:20.899 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:20.899 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:20.900 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:20.900 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:20.909 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:20.910 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-20T03:09:20.910 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:20.910 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:20.911 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:20.912 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.912 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.913 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.913 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:09:20.914 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.914 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.914 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.915 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.915 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:09:20.916 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.916 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.917 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.917 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.917 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:20.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.919 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.919 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.920 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:09:20.920 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.921 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.921 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.931 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.932 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:09:20.932 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.933 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.933 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.934 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.934 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:20.935 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.935 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.936 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.936 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:09:20.936 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.937 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.937 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.938 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.939 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:09:20.939 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.940 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.940 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.940 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.941 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:20.941 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:20.941 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.949 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:20.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:09:20.958 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.959 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.959 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.960 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:20.960 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:09:20.961 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:09:20.962 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:20.962 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.963 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.963 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:09:20.964 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:09:20.964 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:09:20.965 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:09:20.965 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:20.966 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:20.978 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:20.985 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:20.987 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:20.987 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:20.988 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:09:20.988 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:09:20.988 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:20.989 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:20.989 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:20.990 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:20.990 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:20.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:20.998 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:20.999 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:20.999 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:20.999 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:21.004 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:21.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.004 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.005 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:09:21.005 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.005 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.005 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.006 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:09:21.006 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.007 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.007 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.007 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:21.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.015 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.015 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:09:21.015 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.016 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:09:21.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.017 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.017 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.018 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.018 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:21.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.018 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.021 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.021 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:09:21.021 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.022 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.022 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:09:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.023 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.024 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.024 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.025 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:21.025 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.025 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.026 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:09:21.026 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.026 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.027 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.027 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:09:21.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:09:21.028 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.028 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.028 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.028 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:09:21.028 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:09:21.029 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:09:21.029 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:09:21.029 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:21.029 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:21.032 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:21.032 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.032 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.033 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:09:21.033 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:09:21.033 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:21.033 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:21.034 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:21.034 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:21.034 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:21.035 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:21.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:21.036 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:21.036 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:21.036 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:21.036 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:21.037 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.037 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.037 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:09:21.038 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.038 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.038 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.038 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:09:21.039 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.039 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.039 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.040 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:21.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:09:21.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.043 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:09:21.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.044 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.045 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.046 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:21.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:09:21.047 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.049 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:09:21.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.049 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.049 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.050 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:21.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:09:21.053 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.053 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.054 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.054 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:09:21.054 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:09:21.054 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.055 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.055 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.055 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:09:21.056 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:09:21.056 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:09:21.057 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:09:21.057 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:21.057 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:21.057 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:21.058 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.058 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.058 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:09:21.059 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:09:21.059 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:21.059 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:21.059 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:21.060 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:21.060 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:21.060 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:21.060 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:21.061 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:21.061 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:21.063 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:21.063 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:21.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.064 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.064 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:09:21.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.065 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:09:21.065 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.066 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.066 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.067 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.067 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:21.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.068 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:09:21.068 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:09:21.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.069 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.070 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.070 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:21.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.071 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:09:21.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.073 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.074 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.074 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:09:21.074 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.075 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.075 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.075 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.075 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:21.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.076 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.076 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:09:21.077 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.077 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.077 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:09:21.078 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:09:21.079 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.079 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.079 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:09:21.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:09:21.080 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:09:21.080 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:09:21.080 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:21.081 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:21.081 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:21.093 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.094 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.094 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:09:21.095 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:09:21.095 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:21.096 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:21.096 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:21.103 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:21.104 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:21.104 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:21.105 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:21.105 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-20T03:09:21.106 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:21.107 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:21.107 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:21.108 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:09:21.112 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.126 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:09:21.127 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.127 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.127 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.128 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:21.128 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.128 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.128 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.129 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:09:21.129 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.130 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:09:21.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.131 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.132 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.132 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.132 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:21.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.133 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.133 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:09:21.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.134 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.134 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:09:21.135 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.135 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.135 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.135 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:21.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:09:21.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.137 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:09:21.138 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:09:21.138 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.138 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.138 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.139 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:09:21.139 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:09:21.139 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:09:21.139 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:09:21.139 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:21.140 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:21.140 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:21.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.141 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:09:21.141 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:09:21.141 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:09:21.141 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:09:21.142 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:09:21.142 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:09:21.142 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:09:21.143 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:09:21.143 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:09:21.143 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 31, 2022-07-20T03:09:21.143 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:09:21.144 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:09:21.144 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:09:21.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.145 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:09:21.145 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.146 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:09:21.147 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.147 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.147 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.147 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.148 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:09:21.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.148 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:09:21.149 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.149 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.170 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:09:21.171 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.171 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.171 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.172 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:09:21.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.172 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.172 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:09:21.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.173 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.173 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:09:21.174 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.174 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.175 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.175 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:09:21.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:09:21.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.176 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:09:21.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:09:21.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.176 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:09:21.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:09:21.177 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:09:21.177 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.178 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.178 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:09:21.178 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:09:21.179 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:09:21.179 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:09:21.197 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:09:21.198 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:09:21.198 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:09:21.198 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:21.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.199 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:21.199 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:09:21.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.200 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:09:21.209 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.210 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.210 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.211 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.211 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:09:21.212 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.214 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.215 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:09:21.215 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.216 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.216 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.217 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.217 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.217 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:09:21.223 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.225 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:09:21.225 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.226 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.226 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.227 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.227 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.228 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:09:21.228 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.229 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:09:21.239 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.240 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.240 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.241 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.241 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.242 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:09:21.243 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.245 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:09:21.245 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.245 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.246 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.246 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.247 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.247 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:09:21.248 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.249 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:09:21.249 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.250 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.250 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.251 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.251 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.252 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:09:21.252 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.253 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:09:21.254 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.255 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.255 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.256 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.256 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.257 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:09:21.257 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.258 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:21.259 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:09:21.259 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:09:21.259 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:09:21.260 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:09:21.260 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:09:21.261 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:09:21.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:09:21.262 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:09:21.262 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.263 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:09:21.263 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:09:21.263 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:09:21.264 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:09:21.264 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:09:21.265 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:09:21.265 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6835/939788248": "2022-07-21T03:09:12.803824+0000", 2022-07-20T03:09:21.266 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6834/939788248": "2022-07-21T03:09:12.803824+0000" 2022-07-20T03:09:21.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.267 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:09:21.267 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:09:21.267 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:09:21.268 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:09:21.268 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:09:21.269 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:09:21.269 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:09:21.269 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:09:21.270 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:21.270 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:21.271 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:09:21.271 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:09:21.272 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:09:21.272 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:09:21.272 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:09:21.273 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:09:21.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0b4053a40 msgr2=0x7fe0b4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0b4053a40 0x7fe0b4055f00 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7fe0c001a000 tx=0x7fe0c0014000).stop 2022-07-20T03:09:21.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 msgr2=0x7fe0d0064250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 0x7fe0d0064250 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7fe0c80013c0 tx=0x7fe0c8007e20).stop 2022-07-20T03:09:21.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 shutdown_connections 2022-07-20T03:09:21.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0b4053a40 0x7fe0b4055f00 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0d00b1e00 0x7fe0d01432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0d0063da0 0x7fe0d0064250 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 --2- 172.21.15.61:0/3403817389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0d00acef0 0x7fe0d00ad8c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 >> 172.21.15.61:0/3403817389 conn(0x7fe0d005c3f0 msgr2=0x7fe0d014b810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:21.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 shutdown_connections 2022-07-20T03:09:21.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:20.335+0000 7fe0d4bc9700 1 -- 172.21.15.61:0/3403817389 wait complete. 2022-07-20T03:09:21.284 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:21.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.403+0000 7f027099d700 1 Processor -- start 2022-07-20T03:09:21.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.403+0000 7f027099d700 1 -- start start 2022-07-20T03:09:21.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f026c0b3ca0 0x7f026c0b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f026c142ba0 con 0x7f026c13c790 2022-07-20T03:09:21.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f026c1429b0 con 0x7f026c0b3ca0 2022-07-20T03:09:21.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c146330 con 0x7f026c13cb70 2022-07-20T03:09:21.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f026c0b3ca0 0x7f026c0b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f026c13c790 0x7f026c0b9a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40828/0 (socket says 172.21.15.61:40828) 2022-07-20T03:09:21.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026b7fe700 1 -- 172.21.15.61:0/2908058208 learned_addr learned my addr 172.21.15.61:0/2908058208 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:21.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026bfff700 1 --2- 172.21.15.61:0/2908058208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026affd700 1 -- 172.21.15.61:0/2908058208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 msgr2=0x7f026c143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:21.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026affd700 1 --2- 172.21.15.61:0/2908058208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026affd700 1 -- 172.21.15.61:0/2908058208 >> v1:172.21.15.61:6789/0 conn(0x7f026c13c790 legacy=0x7f026c0b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026affd700 1 -- 172.21.15.61:0/2908058208 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f026c0ba120 con 0x7f026c0b3ca0 2022-07-20T03:09:21.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026a7fc700 1 -- 172.21.15.61:0/2908058208 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 825190570 0 0) 0x7f026c142ba0 con 0x7f026c13c790 2022-07-20T03:09:21.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026affd700 1 --2- 172.21.15.61:0/2908058208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b4710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f025c002f70 tx=0x7f025c006df0).ready entity=mon.0 client_cookie=48e5168c4cc379d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:21.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026a7fc700 1 -- 172.21.15.61:0/2908058208 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f025c00d570 con 0x7f026c0b3ca0 2022-07-20T03:09:21.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026a7fc700 1 -- 172.21.15.61:0/2908058208 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f025c000e90 con 0x7f026c0b3ca0 2022-07-20T03:09:21.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f026a7fc700 1 -- 172.21.15.61:0/2908058208 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f025c00d8c0 con 0x7f026c0b3ca0 2022-07-20T03:09:21.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- 172.21.15.61:0/2908058208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 msgr2=0x7f026c0b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 --2- 172.21.15.61:0/2908058208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b4710 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f025c002f70 tx=0x7f025c006df0).stop 2022-07-20T03:09:21.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- 172.21.15.61:0/2908058208 shutdown_connections 2022-07-20T03:09:21.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 --2- 172.21.15.61:0/2908058208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 --2- 172.21.15.61:0/2908058208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b4710 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- 172.21.15.61:0/2908058208 >> 172.21.15.61:0/2908058208 conn(0x7f026c0678b0 msgr2=0x7f026c0b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:21.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- 172.21.15.61:0/2908058208 shutdown_connections 2022-07-20T03:09:21.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.407+0000 7f027099d700 1 -- 172.21.15.61:0/2908058208 wait complete. 2022-07-20T03:09:21.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 Processor -- start 2022-07-20T03:09:21.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 -- start start 2022-07-20T03:09:21.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b27b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c13c790 0x7f026c145490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c152530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0b0d00 con 0x7f026c0b3ca0 2022-07-20T03:09:21.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f026c155a80 con 0x7f026c13c790 2022-07-20T03:09:21.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f027099d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f026c0b21d0 con 0x7f026c13cb70 2022-07-20T03:09:21.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b27b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b27b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44236/0 (socket says 172.21.15.61:44236) 2022-07-20T03:09:21.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 -- 172.21.15.61:0/4189600983 learned_addr learned my addr 172.21.15.61:0/4189600983 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:21.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026affd700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c13c790 0x7f026c145490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026bfff700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c152530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 -- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 msgr2=0x7f026c152530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c152530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 -- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c13c790 msgr2=0x7f026c145490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c13c790 0x7f026c145490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 -- 172.21.15.61:0/4189600983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f025c00b180 con 0x7f026c0b3ca0 2022-07-20T03:09:21.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.411+0000 7f026b7fe700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b27b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f026c0b0b00 tx=0x7f0260007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:21.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f02600070d0 con 0x7f026c0b3ca0 2022-07-20T03:09:21.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0260007230 con 0x7f026c0b3ca0 2022-07-20T03:09:21.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f026c0b2470 con 0x7f026c0b3ca0 2022-07-20T03:09:21.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0260005610 con 0x7f026c0b3ca0 2022-07-20T03:09:21.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f026c145980 con 0x7f026c0b3ca0 2022-07-20T03:09:21.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0260010070 con 0x7f026c0b3ca0 2022-07-20T03:09:21.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f026c0574e0 con 0x7f026c0b3ca0 2022-07-20T03:09:21.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f0268ff9700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f024c053ad0 0x7f024c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f026006b260 con 0x7f026c0b3ca0 2022-07-20T03:09:21.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f026affd700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f024c053ad0 0x7f024c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.415+0000 7f026affd700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f024c053ad0 0x7f024c055f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f025c01a000 tx=0x7f025c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:21.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.423+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02600424a0 con 0x7f026c0b3ca0 2022-07-20T03:09:21.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.583+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f026c0b5390 con 0x7f026c0b3ca0 2022-07-20T03:09:21.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.583+0000 7f0268ff9700 1 -- 172.21.15.61:0/4189600983 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f0260045ac0 con 0x7f026c0b3ca0 2022-07-20T03:09:21.591 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:21.591 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":5200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2574241743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2574241743},{"type":"v1","addr":"172.21.15.117:6835","nonce":2574241743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":17}],"filesystems":[]} 2022-07-20T03:09:21.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f024c053ad0 msgr2=0x7f024c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f024c053ad0 0x7f024c055f90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f025c01a000 tx=0x7f025c014000).stop 2022-07-20T03:09:21.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 msgr2=0x7f026c0b27b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b27b0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f026c0b0b00 tx=0x7f0260007ac0).stop 2022-07-20T03:09:21.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 shutdown_connections 2022-07-20T03:09:21.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f024c053ad0 0x7f024c055f90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f026c13c790 0x7f026c145490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f026c13cb70 0x7f026c152530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 --2- 172.21.15.61:0/4189600983 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f026c0b3ca0 0x7f026c0b27b0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 >> 172.21.15.61:0/4189600983 conn(0x7f026c0678b0 msgr2=0x7f026c0b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:21.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 shutdown_connections 2022-07-20T03:09:21.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.587+0000 7f027099d700 1 -- 172.21.15.61:0/4189600983 wait complete. 2022-07-20T03:09:21.601 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-20T03:09:21.603 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:21.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.703+0000 7f2914af8700 1 Processor -- start 2022-07-20T03:09:21.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.703+0000 7f2914af8700 1 -- start start 2022-07-20T03:09:21.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.703+0000 7f2914af8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2910060d80 0x7f2910145b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.703+0000 7f2914af8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29100b1d10 con 0x7f29100b61d0 2022-07-20T03:09:21.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.703+0000 7f2914af8700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29100b65b0 con 0x7f29100b08c0 2022-07-20T03:09:21.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.703+0000 7f2914af8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29100ad480 con 0x7f2910060d80 2022-07-20T03:09:21.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f29100b61d0 0x7f2910145400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40844/0 (socket says 172.21.15.61:40844) 2022-07-20T03:09:21.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290f7fe700 1 -- 172.21.15.61:0/86862302 learned_addr learned my addr 172.21.15.61:0/86862302 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:21.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290ffff700 1 --2- 172.21.15.61:0/86862302 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2910060d80 0x7f2910145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290e7fc700 1 -- 172.21.15.61:0/86862302 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 829479755 0 0) 0x7f29100b65b0 con 0x7f29100b08c0 2022-07-20T03:09:21.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290ffff700 1 -- 172.21.15.61:0/86862302 >> v1:172.21.15.117:6789/0 conn(0x7f29100b08c0 legacy=0x7f291014b4c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290ffff700 1 -- 172.21.15.61:0/86862302 >> v1:172.21.15.61:6789/0 conn(0x7f29100b61d0 legacy=0x7f2910145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290ffff700 1 -- 172.21.15.61:0/86862302 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29100b1f70 con 0x7f2910060d80 2022-07-20T03:09:21.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290ffff700 1 --2- 172.21.15.61:0/86862302 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2910060d80 0x7f2910145b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2900001110 tx=0x7f29080091d0).ready entity=mon.2 client_cookie=c70921cd34f308e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:21.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290e7fc700 1 -- 172.21.15.61:0/86862302 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290800b760 con 0x7f2910060d80 2022-07-20T03:09:21.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290e7fc700 1 -- 172.21.15.61:0/86862302 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f290800b8c0 con 0x7f2910060d80 2022-07-20T03:09:21.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f290e7fc700 1 -- 172.21.15.61:0/86862302 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290800ba30 con 0x7f2910060d80 2022-07-20T03:09:21.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 -- 172.21.15.61:0/86862302 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2910060d80 msgr2=0x7f2910145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 --2- 172.21.15.61:0/86862302 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2910060d80 0x7f2910145b10 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f2900001110 tx=0x7f29080091d0).stop 2022-07-20T03:09:21.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 -- 172.21.15.61:0/86862302 shutdown_connections 2022-07-20T03:09:21.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 --2- 172.21.15.61:0/86862302 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2910060d80 0x7f2910145b10 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 -- 172.21.15.61:0/86862302 >> 172.21.15.61:0/86862302 conn(0x7f2910063cf0 msgr2=0x7f29100b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 -- 172.21.15.61:0/86862302 shutdown_connections 2022-07-20T03:09:21.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.707+0000 7f2914af8700 1 -- 172.21.15.61:0/86862302 wait complete. 2022-07-20T03:09:21.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 Processor -- start 2022-07-20T03:09:21.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 -- start start 2022-07-20T03:09:21.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2910060d80 0x7f2910144130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29100b08c0 0x7f2910064650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 0x7f2910155430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2910143340 con 0x7f29100b61d0 2022-07-20T03:09:21.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2910066d90 con 0x7f2910060d80 2022-07-20T03:09:21.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2910067030 con 0x7f29100b08c0 2022-07-20T03:09:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 0x7f2910155430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 0x7f2910155430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44254/0 (socket says 172.21.15.61:44254) 2022-07-20T03:09:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290ffff700 1 -- 172.21.15.61:0/334321006 learned_addr learned my addr 172.21.15.61:0/334321006 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:21.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29100b08c0 0x7f2910064650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290f7fe700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2910060d80 0x7f2910144130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 -- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2910060d80 msgr2=0x7f2910144130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2910060d80 0x7f2910144130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 -- 172.21.15.61:0/334321006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 msgr2=0x7f2910155430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 0x7f2910155430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 -- 172.21.15.61:0/334321006 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2908009470 con 0x7f29100b08c0 2022-07-20T03:09:21.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290ffff700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 0x7f2910155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:21.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290effd700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29100b08c0 0x7f2910064650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2910143480 tx=0x7f29040079b0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:21.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f290400cdc0 con 0x7f29100b08c0 2022-07-20T03:09:21.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2904012400 con 0x7f29100b08c0 2022-07-20T03:09:21.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2910143030 con 0x7f29100b08c0 2022-07-20T03:09:21.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2904012760 con 0x7f29100b08c0 2022-07-20T03:09:21.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.711+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2910155920 con 0x7f29100b08c0 2022-07-20T03:09:21.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.715+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29100671f0 con 0x7f29100b08c0 2022-07-20T03:09:21.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.715+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2904019030 con 0x7f29100b08c0 2022-07-20T03:09:21.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.715+0000 7f290cff9700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f28f00539b0 0x7f28f0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:21.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.715+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f290406afd0 con 0x7f29100b08c0 2022-07-20T03:09:21.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.715+0000 7f290f7fe700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f28f00539b0 0x7f28f0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:21.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.715+0000 7f290f7fe700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f28f00539b0 0x7f28f0055e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f290000d000 tx=0x7f2900009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:21.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.727+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f29040420a0 con 0x7f29100b08c0 2022-07-20T03:09:21.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.887+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2910066bc0 con 0x7f29100b08c0 2022-07-20T03:09:21.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.887+0000 7f290cff9700 1 -- 172.21.15.61:0/334321006 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 17 v17) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f29040456c0 con 0x7f29100b08c0 2022-07-20T03:09:21.925 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:21.925 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":17,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":4908,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/724836284","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":724836284},{"type":"v1","addr":"172.21.15.61:6833","nonce":724836284}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":11},{"gid":5200,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2574241743","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2574241743},{"type":"v1","addr":"172.21.15.117:6835","nonce":2574241743}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":17}],"filesystems":[]} 2022-07-20T03:09:21.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.887+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f28f00539b0 msgr2=0x7f28f0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.887+0000 7f2914af8700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f28f00539b0 0x7f28f0055e70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f290000d000 tx=0x7f2900009040).stop 2022-07-20T03:09:21.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29100b08c0 msgr2=0x7f2910064650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:21.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29100b08c0 0x7f2910064650 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f2910143480 tx=0x7f29040079b0).stop 2022-07-20T03:09:21.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 shutdown_connections 2022-07-20T03:09:21.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f28f00539b0 0x7f28f0055e70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29100b61d0 0x7f2910155430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29100b08c0 0x7f2910064650 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 --2- 172.21.15.61:0/334321006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2910060d80 0x7f2910144130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:21.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 >> 172.21.15.61:0/334321006 conn(0x7f2910063cf0 msgr2=0x7f29101451d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 shutdown_connections 2022-07-20T03:09:21.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:21.891+0000 7f2914af8700 1 -- 172.21.15.61:0/334321006 wait complete. 2022-07-20T03:09:21.949 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 17 2022-07-20T03:09:21.954 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:09:21.954 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:09:21.955 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:09:21.956 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:09:21.956 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:09:21.957 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:09:21.957 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:09:21.958 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:09:21.979 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:09:21.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:09:21.980 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:09:21.980 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:09:21.993 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:09:21.993 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:09:21.994 DEBUG:teuthology.parallel:result is None 2022-07-20T03:09:21.994 DEBUG:teuthology.parallel:result is None 2022-07-20T03:09:21.995 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:09:22.056 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:09:22.061 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:09:22.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 Processor -- start 2022-07-20T03:09:22.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 -- start start 2022-07-20T03:09:22.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8146040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:22.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44e8146530 0x7f44e8141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:22.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f44e8066b10 con 0x7f44e80b6840 2022-07-20T03:09:22.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44e80b5c90 con 0x7f44e8146530 2022-07-20T03:09:22.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44eddeb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80b6c20 con 0x7f44e80b3cf0 2022-07-20T03:09:22.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ecde9700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f44e80b6840 0x7f44e80b5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40868/0 (socket says 172.21.15.61:40868) 2022-07-20T03:09:22.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ecde9700 1 -- 172.21.15.61:0/2374267566 learned_addr learned my addr 172.21.15.61:0/2374267566 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:22.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ed5ea700 1 --2- 172.21.15.61:0/2374267566 >> v2:172.21.15.61:3300/0 conn(0x7f44e8146530 0x7f44e8141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:22.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e7fff700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:22.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e77fe700 1 -- 172.21.15.61:0/2374267566 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1459034846 0 0) 0x7f44e8066b10 con 0x7f44e80b6840 2022-07-20T03:09:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e77fe700 1 -- 172.21.15.61:0/2374267566 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f44c8003260 con 0x7f44e80b6840 2022-07-20T03:09:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ed5ea700 1 -- 172.21.15.61:0/2374267566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 msgr2=0x7f44e8146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:22.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ed5ea700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:22.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ed5ea700 1 -- 172.21.15.61:0/2374267566 >> v1:172.21.15.61:6789/0 conn(0x7f44e80b6840 legacy=0x7f44e80b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:22.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ed5ea700 1 -- 172.21.15.61:0/2374267566 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e80aead0 con 0x7f44e8146530 2022-07-20T03:09:22.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e7fff700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:22.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44ed5ea700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e8146530 0x7f44e8141a90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f44e00032f0 tx=0x7f44e00091d0).ready entity=mon.0 client_cookie=ad5fc797a0d4f752 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:22.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e77fe700 1 -- 172.21.15.61:0/2374267566 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e000b6c0 con 0x7f44e8146530 2022-07-20T03:09:22.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e77fe700 1 -- 172.21.15.61:0/2374267566 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44e000b820 con 0x7f44e8146530 2022-07-20T03:09:22.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.095+0000 7f44e77fe700 1 -- 172.21.15.61:0/2374267566 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44e000b990 con 0x7f44e8146530 2022-07-20T03:09:22.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- 172.21.15.61:0/2374267566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e8146530 msgr2=0x7f44e8141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:22.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e8146530 0x7f44e8141a90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f44e00032f0 tx=0x7f44e00091d0).stop 2022-07-20T03:09:22.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- 172.21.15.61:0/2374267566 shutdown_connections 2022-07-20T03:09:22.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e8146530 0x7f44e8141a90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:22.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 --2- 172.21.15.61:0/2374267566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:22.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- 172.21.15.61:0/2374267566 >> 172.21.15.61:0/2374267566 conn(0x7f44e80b9e10 msgr2=0x7f44e80b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:22.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- 172.21.15.61:0/2374267566 shutdown_connections 2022-07-20T03:09:22.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- 172.21.15.61:0/2374267566 wait complete. 2022-07-20T03:09:22.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 Processor -- start 2022-07-20T03:09:22.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- start start 2022-07-20T03:09:22.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8115c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:22.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 0x7f44e8151a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:22.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44e8146530 0x7f44e8141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:22.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e8112640 con 0x7f44e80b6840 2022-07-20T03:09:22.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80b2e10 con 0x7f44e8146530 2022-07-20T03:09:22.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44eddeb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f44e80b30b0 con 0x7f44e80b3cf0 2022-07-20T03:09:22.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 0x7f44e8151a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:22.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 0x7f44e8151a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44278/0 (socket says 172.21.15.61:44278) 2022-07-20T03:09:22.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 -- 172.21.15.61:0/1874729128 learned_addr learned my addr 172.21.15.61:0/1874729128 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:22.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44ecde9700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8115c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:22.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44ed5ea700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44e8146530 0x7f44e8141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:22.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 -- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 msgr2=0x7f44e8115c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:22.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8115c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:22.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 -- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44e8146530 msgr2=0x7f44e8141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:22.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44e8146530 0x7f44e8141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:22.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44e7fff700 1 -- 172.21.15.61:0/1874729128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e0009470 con 0x7f44e80b6840 2022-07-20T03:09:22.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.099+0000 7f44ed5ea700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44e8146530 0x7f44e8141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:22.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e7fff700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 0x7f44e8151a90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f44d000a000 tx=0x7f44d0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:22.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44d0011780 con 0x7f44e80b6840 2022-07-20T03:09:22.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f44d00118e0 con 0x7f44e80b6840 2022-07-20T03:09:22.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44e80b2b00 con 0x7f44e80b6840 2022-07-20T03:09:22.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f44d0011c40 con 0x7f44e80b6840 2022-07-20T03:09:22.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44e8112ac0 con 0x7f44e80b6840 2022-07-20T03:09:22.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44e81127b0 con 0x7f44e80b6840 2022-07-20T03:09:22.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f44d0012420 con 0x7f44e80b6840 2022-07-20T03:09:22.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e5ffb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44cc053ad0 0x7f44cc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:22.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(36..36 src has 1..36) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f44d006c740 con 0x7f44e80b6840 2022-07-20T03:09:22.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.103+0000 7f44ecde9700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44cc053ad0 0x7f44cc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:22.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.107+0000 7f44ecde9700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44cc053ad0 0x7f44cc055f90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f44d800d000 tx=0x7f44d8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:22.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.111+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f44d0043920 con 0x7f44e80b6840 2022-07-20T03:09:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:22.251+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f44e80574e0 con 0x7f44e80b6840 2022-07-20T03:09:23.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.179+0000 7f44e5ffb700 1 -- 172.21.15.61:0/1874729128 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v37) v1 ==== 97+0+0 (secure 0 0 0) 0x7f44d0046f40 con 0x7f44e80b6840 2022-07-20T03:09:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44cc053ad0 msgr2=0x7f44cc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44cc053ad0 0x7f44cc055f90 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f44d800d000 tx=0x7f44d8009040).stop 2022-07-20T03:09:23.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 msgr2=0x7f44e8151a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 0x7f44e8151a90 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7f44d000a000 tx=0x7f44d0008040).stop 2022-07-20T03:09:23.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 shutdown_connections 2022-07-20T03:09:23.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f44cc053ad0 0x7f44cc055f90 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44e8146530 0x7f44e8141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44e80b6840 0x7f44e8151a90 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 --2- 172.21.15.61:0/1874729128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44e80b3cf0 0x7f44e8115c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 >> 172.21.15.61:0/1874729128 conn(0x7f44e80b9e10 msgr2=0x7f44e8143260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:23.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 shutdown_connections 2022-07-20T03:09:23.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.183+0000 7f44eddeb700 1 -- 172.21.15.61:0/1874729128 wait complete. 2022-07-20T03:09:23.213 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:09:23.214 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:09:23.215 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:09:23.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 Processor -- start 2022-07-20T03:09:23.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- start start 2022-07-20T03:09:23.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21fc0b1200 0x7f21fc0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc05bd90 0x7f21fc142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21fc0aea30 con 0x7f21fc0b5650 2022-07-20T03:09:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21fc142970 con 0x7f21fc0b1200 2022-07-20T03:09:23.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc0aec60 con 0x7f21fc05bd90 2022-07-20T03:09:23.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21fc0b1200 0x7f21fc0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:23.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21fc0b1200 0x7f21fc0b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44284/0 (socket says 172.21.15.61:44284) 2022-07-20T03:09:23.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 -- 172.21.15.61:0/3877363677 learned_addr learned my addr 172.21.15.61:0/3877363677 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:23.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 -- 172.21.15.61:0/3877363677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc05bd90 msgr2=0x7f21fc142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:23.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fad9d700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc05bd90 0x7f21fc142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:23.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc05bd90 0x7f21fc142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 -- 172.21.15.61:0/3877363677 >> v1:172.21.15.61:6789/0 conn(0x7f21fc0b5650 legacy=0x7f21fc05c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 -- 172.21.15.61:0/3877363677 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21fc0af180 con 0x7f21fc0b1200 2022-07-20T03:09:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fad9d700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc05bd90 0x7f21fc142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:09:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f1d9b700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc0b1200 0x7f21fc0b5a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f21e4002f70 tx=0x7f21e4006e50).ready entity=mon.0 client_cookie=9f7e131e9e3cd31b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:23.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f9d9b700 1 -- 172.21.15.61:0/3877363677 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21e400d5c0 con 0x7f21fc0b1200 2022-07-20T03:09:23.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f9d9b700 1 -- 172.21.15.61:0/3877363677 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21e4000e90 con 0x7f21fc0b1200 2022-07-20T03:09:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21f9d9b700 1 -- 172.21.15.61:0/3877363677 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21e400d950 con 0x7f21fc0b1200 2022-07-20T03:09:23.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- 172.21.15.61:0/3877363677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc0b1200 msgr2=0x7f21fc0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc0b1200 0x7f21fc0b5a30 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f21e4002f70 tx=0x7f21e4006e50).stop 2022-07-20T03:09:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- 172.21.15.61:0/3877363677 shutdown_connections 2022-07-20T03:09:23.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc05bd90 0x7f21fc142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 --2- 172.21.15.61:0/3877363677 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc0b1200 0x7f21fc0b5a30 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- 172.21.15.61:0/3877363677 >> 172.21.15.61:0/3877363677 conn(0x7f21fc0634e0 msgr2=0x7f21fc0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:23.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- 172.21.15.61:0/3877363677 shutdown_connections 2022-07-20T03:09:23.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.375+0000 7f21fb59e700 1 -- 172.21.15.61:0/3877363677 wait complete. 2022-07-20T03:09:23.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 Processor -- start 2022-07-20T03:09:23.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 -- start start 2022-07-20T03:09:23.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 0x7f21fc145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21fc0b1200 0x7f21fc13f980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc0b5650 0x7f21fc151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc13fe70 con 0x7f21fc05bd90 2022-07-20T03:09:23.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc142580 con 0x7f21fc0b1200 2022-07-20T03:09:23.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f21fc13f4e0 con 0x7f21fc0b5650 2022-07-20T03:09:23.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 0x7f21fc145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:23.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 0x7f21fc145110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44290/0 (socket says 172.21.15.61:44290) 2022-07-20T03:09:23.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 -- 172.21.15.61:0/2474880424 learned_addr learned my addr 172.21.15.61:0/2474880424 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:23.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21f1d9b700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21fc0b1200 0x7f21fc13f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:23.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fad9d700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc0b5650 0x7f21fc151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:23.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 -- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc0b5650 msgr2=0x7f21fc151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc0b5650 0x7f21fc151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 -- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21fc0b1200 msgr2=0x7f21fc13f980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21fc0b1200 0x7f21fc13f980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 -- 172.21.15.61:0/2474880424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21e400b1e0 con 0x7f21fc05bd90 2022-07-20T03:09:23.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21f1d9b700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21fc0b1200 0x7f21fc13f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:23.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fa59c700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 0x7f21fc145110 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f21fc0608e0 tx=0x7f21ec007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:23.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21ec00ce30 con 0x7f21fc05bd90 2022-07-20T03:09:23.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f21ec00f470 con 0x7f21fc05bd90 2022-07-20T03:09:23.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f21ec00f7d0 con 0x7f21fc05bd90 2022-07-20T03:09:23.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21fc14a4a0 con 0x7f21fc05bd90 2022-07-20T03:09:23.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.379+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f21fc13cf50 con 0x7f21fc05bd90 2022-07-20T03:09:23.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.383+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f21fc0574e0 con 0x7f21fc05bd90 2022-07-20T03:09:23.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.383+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f21ec019030 con 0x7f21fc05bd90 2022-07-20T03:09:23.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.383+0000 7f21f3fff700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f21e0053af0 0x7f21e0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.383+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f21ec06afe0 con 0x7f21fc05bd90 2022-07-20T03:09:23.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.383+0000 7f21f1d9b700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f21e0053af0 0x7f21e0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:23.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.383+0000 7f21f1d9b700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f21e0053af0 0x7f21e0055fb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f21e401a000 tx=0x7f21e4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:23.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.391+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f21ec042190 con 0x7f21fc05bd90 2022-07-20T03:09:23.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.559+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f21fc067ea0 con 0x7f21fc05bd90 2022-07-20T03:09:23.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.559+0000 7f21f3fff700 1 -- 172.21.15.61:0/2474880424 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v15) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f21ec0457b0 con 0x7f21fc05bd90 2022-07-20T03:09:23.572 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:23.572 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:09:23.573 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:09:23.573 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.573 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:09:23.574 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:09:23.574 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.574 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:09:23.574 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.575 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.575 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.576 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.577 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.577 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.577 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:09:23.577 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:09:23.578 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:09:23.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.579 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.579 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.579 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.579 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.580 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:09:23.580 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:09:23.580 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.580 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.583 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.583 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.584 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.584 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.585 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.589 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:09:23.590 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:09:23.590 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.591 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.603 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.604 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.604 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.605 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.605 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.606 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:09:23.606 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:09:23.607 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.607 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.608 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.608 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.609 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.609 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.620 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.621 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:09:23.622 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:09:23.622 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.623 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.623 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.624 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.624 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.625 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.625 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.625 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:09:23.626 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:09:23.635 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.636 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.636 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.637 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.638 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.638 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.639 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.694 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:09:23.695 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:09:23.695 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.695 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.696 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.696 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.697 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.697 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.706 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.707 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:09:23.707 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:09:23.708 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.709 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.709 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.710 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.710 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.711 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.711 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.712 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:09:23.712 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:09:23.713 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.713 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.713 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.714 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.726 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.726 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.727 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:09:23.728 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:09:23.728 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.741 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:09:23.743 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:09:23.744 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:09:23.744 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.773 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.774 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.774 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:09:23.774 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:09:23.775 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.775 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:09:23.776 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:09:23.776 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:09:23.777 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.777 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.777 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.806 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:09:23.807 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:09:23.807 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.807 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:09:23.808 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:09:23.808 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:09:23.809 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.809 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.810 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:09:23.810 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:09:23.811 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:09:23.812 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:09:23.828 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:09:23.829 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.829 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.830 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.830 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.831 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:09:23.831 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:09:23.831 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.832 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:09:23.832 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:09:23.833 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.833 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.834 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:23.834 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:23.835 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:09:23.835 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:09:23.836 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:09:23.871 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:09:23.872 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:09:23.872 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:09:23.872 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.873 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:23.873 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:09:23.873 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:09:23.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f21e0053af0 msgr2=0x7f21e0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f21e0053af0 0x7f21e0055fb0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f21e401a000 tx=0x7f21e4015000).stop 2022-07-20T03:09:23.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 msgr2=0x7f21fc145110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:23.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 0x7f21fc145110 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f21fc0608e0 tx=0x7f21ec007c50).stop 2022-07-20T03:09:23.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 shutdown_connections 2022-07-20T03:09:23.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f21e0053af0 0x7f21e0055fb0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21fc0b1200 0x7f21fc13f980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f21fc0b5650 0x7f21fc151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 --2- 172.21.15.61:0/2474880424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21fc05bd90 0x7f21fc145110 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:23.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 >> 172.21.15.61:0/2474880424 conn(0x7f21fc0634e0 msgr2=0x7f21fc0b2120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:23.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 shutdown_connections 2022-07-20T03:09:23.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.563+0000 7f21fb59e700 1 -- 172.21.15.61:0/2474880424 wait complete. 2022-07-20T03:09:23.881 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:09:23.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 Processor -- start 2022-07-20T03:09:23.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- start start 2022-07-20T03:09:23.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f689c0afa10 0x7f689c0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c1452d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:23.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f689c0b42a0 con 0x7f689c0608b0 2022-07-20T03:09:23.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f689c060c90 con 0x7f689c0afa10 2022-07-20T03:09:23.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689c0af410 con 0x7f689c0b3de0 2022-07-20T03:09:23.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f689c0608b0 0x7f689c05bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40896/0 (socket says 172.21.15.61:40896) 2022-07-20T03:09:23.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689a59c700 1 -- 172.21.15.61:0/3413805323 learned_addr learned my addr 172.21.15.61:0/3413805323 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f6899d9b700 1 --2- 172.21.15.61:0/3413805323 >> v2:172.21.15.117:3300/0 conn(0x7f689c0afa10 0x7f689c0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689ad9d700 1 --2- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c1452d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689959a700 1 -- 172.21.15.61:0/3413805323 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 132477018 0 0) 0x7f689c0b42a0 con 0x7f689c0608b0 2022-07-20T03:09:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689959a700 1 -- 172.21.15.61:0/3413805323 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6880003260 con 0x7f689c0608b0 2022-07-20T03:09:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f6899d9b700 1 -- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 msgr2=0x7f689c1452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f6899d9b700 1 --2- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c1452d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f6899d9b700 1 -- 172.21.15.61:0/3413805323 >> v1:172.21.15.61:6789/0 conn(0x7f689c0608b0 legacy=0x7f689c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f6899d9b700 1 -- 172.21.15.61:0/3413805323 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f689c0af6b0 con 0x7f689c0afa10 2022-07-20T03:09:24.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f6899d9b700 1 --2- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0afa10 0x7f689c0b46d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f68940032f0 tx=0x7f68940091d0).ready entity=mon.1 client_cookie=45c33c359240161f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689959a700 1 -- 172.21.15.61:0/3413805323 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f689400b600 con 0x7f689c0afa10 2022-07-20T03:09:24.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689959a700 1 -- 172.21.15.61:0/3413805323 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f689400b760 con 0x7f689c0afa10 2022-07-20T03:09:24.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f689959a700 1 -- 172.21.15.61:0/3413805323 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f689400b8d0 con 0x7f689c0afa10 2022-07-20T03:09:24.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0afa10 msgr2=0x7f689c0b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0afa10 0x7f689c0b46d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f68940032f0 tx=0x7f68940091d0).stop 2022-07-20T03:09:24.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- 172.21.15.61:0/3413805323 shutdown_connections 2022-07-20T03:09:24.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c1452d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/3413805323 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0afa10 0x7f689c0b46d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.991+0000 7f68a0e2d700 1 -- 172.21.15.61:0/3413805323 >> 172.21.15.61:0/3413805323 conn(0x7f689c0b5ee0 msgr2=0x7f689c0b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:24.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 -- 172.21.15.61:0/3413805323 shutdown_connections 2022-07-20T03:09:24.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 -- 172.21.15.61:0/3413805323 wait complete. 2022-07-20T03:09:24.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 Processor -- start 2022-07-20T03:09:24.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 -- start start 2022-07-20T03:09:24.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0608b0 0x7f689c090390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 0x7f689c092010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c092500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c0933a0 con 0x7f689c0afa10 2022-07-20T03:09:24.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f689c090df0 con 0x7f689c0608b0 2022-07-20T03:09:24.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f68a0e2d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f689c094910 con 0x7f689c0b3de0 2022-07-20T03:09:24.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f6899d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 0x7f689c092010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f689a59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0608b0 0x7f689c090390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f6899d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 0x7f689c092010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44306/0 (socket says 172.21.15.61:44306) 2022-07-20T03:09:24.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f6899d9b700 1 -- 172.21.15.61:0/4149225224 learned_addr learned my addr 172.21.15.61:0/4149225224 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:24.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.995+0000 7f689ad9d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c092500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f6899d9b700 1 -- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 msgr2=0x7f689c092500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f6899d9b700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c092500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f6899d9b700 1 -- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0608b0 msgr2=0x7f689c090390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f6899d9b700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0608b0 0x7f689c090390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f6899d9b700 1 -- 172.21.15.61:0/4149225224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6894009470 con 0x7f689c0afa10 2022-07-20T03:09:24.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f6899d9b700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 0x7f689c092010 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f68940032f0 tx=0x7f6894016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f689400b8d0 con 0x7f689c0afa10 2022-07-20T03:09:24.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6894021630 con 0x7f689c0afa10 2022-07-20T03:09:24.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f689400b600 con 0x7f689c0afa10 2022-07-20T03:09:24.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f689c094bb0 con 0x7f689c0afa10 2022-07-20T03:09:24.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f689c090880 con 0x7f689c0afa10 2022-07-20T03:09:24.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f689c0574e0 con 0x7f689c0afa10 2022-07-20T03:09:24.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6894027020 con 0x7f689c0afa10 2022-07-20T03:09:24.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:23.999+0000 7f68937fe700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6874053ad0 0x7f6874055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.003+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f689406d0f0 con 0x7f689c0afa10 2022-07-20T03:09:24.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.003+0000 7f689a59c700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6874053ad0 0x7f6874055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.003+0000 7f689a59c700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6874053ad0 0x7f6874055f90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f688400d000 tx=0x7f6884009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.007+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6894044320 con 0x7f689c0afa10 2022-07-20T03:09:24.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.139+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f689c0b0090 con 0x7f689c0afa10 2022-07-20T03:09:24.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.139+0000 7f68937fe700 1 -- 172.21.15.61:0/4149225224 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v37) v1 ==== 74+0+13229 (secure 0 0 0) 0x7f6894047900 con 0x7f689c0afa10 2022-07-20T03:09:24.177 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:24.177 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":37,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:09:23.167781+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":6,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":31,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:09:24.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.139+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6874053ad0 msgr2=0x7f6874055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.139+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6874053ad0 0x7f6874055f90 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f688400d000 tx=0x7f6884009040).stop 2022-07-20T03:09:24.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.139+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 msgr2=0x7f689c092010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.139+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 0x7f689c092010 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f68940032f0 tx=0x7f6894016000).stop 2022-07-20T03:09:24.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 shutdown_connections 2022-07-20T03:09:24.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6874053ad0 0x7f6874055f90 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f689c0b3de0 0x7f689c092500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f689c0afa10 0x7f689c092010 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 --2- 172.21.15.61:0/4149225224 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f689c0608b0 0x7f689c090390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 >> 172.21.15.61:0/4149225224 conn(0x7f689c0b5ee0 msgr2=0x7f689c145ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:24.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 shutdown_connections 2022-07-20T03:09:24.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.143+0000 7f68a0e2d700 1 -- 172.21.15.61:0/4149225224 wait complete. 2022-07-20T03:09:24.213 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:09:24.217 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:09:24.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114bfff700 1 Processor -- start 2022-07-20T03:09:24.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114bfff700 1 -- start start 2022-07-20T03:09:24.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f114c0b3cf0 0x7f114c142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c0ba180 con 0x7f114c142f40 2022-07-20T03:09:24.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f114c1461b0 con 0x7f114c0b9da0 2022-07-20T03:09:24.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f114c0ad5b0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114a7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f114c0b3cf0 0x7f114c142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f114c142f40 0x7f114c1419a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40914/0 (socket says 172.21.15.61:40914) 2022-07-20T03:09:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.331+0000 7f114b7fe700 1 -- 172.21.15.61:0/1271479188 learned_addr learned my addr 172.21.15.61:0/1271479188 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3415848123 0 0) 0x7f114c0ba180 con 0x7f114c142f40 2022-07-20T03:09:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1130003260 con 0x7f114c142f40 2022-07-20T03:09:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 83050411 0 0) 0x7f114c1461b0 con 0x7f114c0b9da0 2022-07-20T03:09:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f114c0ba180 con 0x7f114c0b9da0 2022-07-20T03:09:24.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114a7fc700 1 -- 172.21.15.61:0/1271479188 >> v1:172.21.15.117:6789/0 conn(0x7f114c0b9da0 legacy=0x7f114c1468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114a7fc700 1 -- 172.21.15.61:0/1271479188 >> v1:172.21.15.61:6789/0 conn(0x7f114c142f40 legacy=0x7f114c1419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114a7fc700 1 -- 172.21.15.61:0/1271479188 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f114c146450 con 0x7f114c0b3cf0 2022-07-20T03:09:24.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114a7fc700 1 --2- 172.21.15.61:0/1271479188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c142a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f113c002f70 tx=0x7f113c006df0).ready entity=mon.0 client_cookie=4561ca9918f228fe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f113c00d540 con 0x7f114c0b3cf0 2022-07-20T03:09:24.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f113c000e90 con 0x7f114c0b3cf0 2022-07-20T03:09:24.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f1149ffb700 1 -- 172.21.15.61:0/1271479188 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f113c00d890 con 0x7f114c0b3cf0 2022-07-20T03:09:24.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 -- 172.21.15.61:0/1271479188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 msgr2=0x7f114c142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 --2- 172.21.15.61:0/1271479188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c142a50 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f113c002f70 tx=0x7f113c006df0).stop 2022-07-20T03:09:24.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 -- 172.21.15.61:0/1271479188 shutdown_connections 2022-07-20T03:09:24.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 --2- 172.21.15.61:0/1271479188 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c142a50 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 -- 172.21.15.61:0/1271479188 >> 172.21.15.61:0/1271479188 conn(0x7f114c0678c0 msgr2=0x7f114c0b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:24.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 -- 172.21.15.61:0/1271479188 shutdown_connections 2022-07-20T03:09:24.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 -- 172.21.15.61:0/1271479188 wait complete. 2022-07-20T03:09:24.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 Processor -- start 2022-07-20T03:09:24.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.335+0000 7f114bfff700 1 -- start start 2022-07-20T03:09:24.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c0b8690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f114c0b9da0 0x7f114c0affe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f114c142f40 0x7f114c0b04d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0b1330 con 0x7f114c0b3cf0 2022-07-20T03:09:24.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0b09c0 con 0x7f114c0b9da0 2022-07-20T03:09:24.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f114c0b2790 con 0x7f114c142f40 2022-07-20T03:09:24.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c0b8690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c0b8690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44318/0 (socket says 172.21.15.61:44318) 2022-07-20T03:09:24.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 -- 172.21.15.61:0/2103218208 learned_addr learned my addr 172.21.15.61:0/2103218208 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:24.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114a7fc700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f114c0b9da0 0x7f114c0affe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114b7fe700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f114c142f40 0x7f114c0b04d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 -- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f114c142f40 msgr2=0x7f114c0b04d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f114c142f40 0x7f114c0b04d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 -- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f114c0b9da0 msgr2=0x7f114c0affe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f114c0b9da0 0x7f114c0affe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 -- 172.21.15.61:0/2103218208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f113c00b180 con 0x7f114c0b3cf0 2022-07-20T03:09:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114a7fc700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f114c0b9da0 0x7f114c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:09:24.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114b7fe700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f114c142f40 0x7f114c0b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:09:24.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114affd700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c0b8690 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f114c14a400 tx=0x7f1140007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114000f470 con 0x7f114c0b3cf0 2022-07-20T03:09:24.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f114000f5d0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f114c145bb0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f114000f930 con 0x7f114c0b3cf0 2022-07-20T03:09:24.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f114c0af840 con 0x7f114c0b3cf0 2022-07-20T03:09:24.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f114c0667f0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1140019030 con 0x7f114c0b3cf0 2022-07-20T03:09:24.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f113bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11280539d0 0x7f1128055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f114006b130 con 0x7f114c0b3cf0 2022-07-20T03:09:24.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.339+0000 7f114a7fc700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11280539d0 0x7f1128055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.343+0000 7f114a7fc700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11280539d0 0x7f1128055e90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f113c015000 tx=0x7f113c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.347+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f11400422b0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.471+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f114c0574e0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.471+0000 7f113bfff700 1 -- 172.21.15.61:0/2103218208 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v37) v1 ==== 110+0+10 (secure 0 0 0) 0x7f11400458d0 con 0x7f114c0b3cf0 2022-07-20T03:09:24.488 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:09:24.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11280539d0 msgr2=0x7f1128055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11280539d0 0x7f1128055e90 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f113c015000 tx=0x7f113c014000).stop 2022-07-20T03:09:24.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 msgr2=0x7f114c0b8690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c0b8690 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f114c14a400 tx=0x7f1140007ae0).stop 2022-07-20T03:09:24.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 shutdown_connections 2022-07-20T03:09:24.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11280539d0 0x7f1128055e90 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f114c0b9da0 0x7f114c0affe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f114c142f40 0x7f114c0b04d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 --2- 172.21.15.61:0/2103218208 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f114c0b3cf0 0x7f114c0b8690 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 >> 172.21.15.61:0/2103218208 conn(0x7f114c0678c0 msgr2=0x7f114c05bf60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:24.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 shutdown_connections 2022-07-20T03:09:24.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.475+0000 7f114bfff700 1 -- 172.21.15.61:0/2103218208 wait complete. 2022-07-20T03:09:24.495 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:09:24.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 Processor -- start 2022-07-20T03:09:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 -- start start 2022-07-20T03:09:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cb80988b0 0x7f9cb81422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427b0 0x7f9cb8060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb80afdb0 con 0x7f9cb81462a0 2022-07-20T03:09:24.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb80b0050 con 0x7f9cb80988b0 2022-07-20T03:09:24.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbecef700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb805fcd0 con 0x7f9cb81427b0 2022-07-20T03:09:24.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cb80988b0 0x7f9cb81422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cb80988b0 0x7f9cb81422c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44328/0 (socket says 172.21.15.61:44328) 2022-07-20T03:09:24.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2184242500 learned_addr learned my addr 172.21.15.61:0/2184242500 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:24.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cbd28c700 1 --2- 172.21.15.61:0/2184242500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427b0 0x7f9cb8060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2184242500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427b0 msgr2=0x7f9cb8060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:24.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/2184242500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427b0 0x7f9cb8060170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2184242500 >> v1:172.21.15.117:6789/0 conn(0x7f9cb81462a0 legacy=0x7f9cb8146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2184242500 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb80afae0 con 0x7f9cb80988b0 2022-07-20T03:09:24.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/2184242500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80988b0 0x7f9cb81422c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f9cac002f70 tx=0x7f9cac006eb0).ready entity=mon.0 client_cookie=e3b3b4bcc7726fb0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb77fe700 1 -- 172.21.15.61:0/2184242500 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cac00d570 con 0x7f9cb80988b0 2022-07-20T03:09:24.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb77fe700 1 -- 172.21.15.61:0/2184242500 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac000e90 con 0x7f9cb80988b0 2022-07-20T03:09:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.615+0000 7f9cb77fe700 1 -- 172.21.15.61:0/2184242500 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cac00d900 con 0x7f9cb80988b0 2022-07-20T03:09:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- 172.21.15.61:0/2184242500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80988b0 msgr2=0x7f9cb81422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 --2- 172.21.15.61:0/2184242500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80988b0 0x7f9cb81422c0 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f9cac002f70 tx=0x7f9cac006eb0).stop 2022-07-20T03:09:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- 172.21.15.61:0/2184242500 shutdown_connections 2022-07-20T03:09:24.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 --2- 172.21.15.61:0/2184242500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427b0 0x7f9cb8060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 --2- 172.21.15.61:0/2184242500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80988b0 0x7f9cb81422c0 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- 172.21.15.61:0/2184242500 >> 172.21.15.61:0/2184242500 conn(0x7f9cb80b5f00 msgr2=0x7f9cb80b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:24.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- 172.21.15.61:0/2184242500 shutdown_connections 2022-07-20T03:09:24.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- 172.21.15.61:0/2184242500 wait complete. 2022-07-20T03:09:24.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 Processor -- start 2022-07-20T03:09:24.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- start start 2022-07-20T03:09:24.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb80988b0 0x7f9cb80b8350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 0x7f9cb80b3040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb81462a0 0x7f9cb81555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb80b2520 con 0x7f9cb81427b0 2022-07-20T03:09:24.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8152d30 con 0x7f9cb81462a0 2022-07-20T03:09:24.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbecef700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8152fd0 con 0x7f9cb80988b0 2022-07-20T03:09:24.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cb7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 0x7f9cb80b3040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cb7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 0x7f9cb80b3040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44334/0 (socket says 172.21.15.61:44334) 2022-07-20T03:09:24.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cbca8b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb80988b0 0x7f9cb80b8350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.619+0000 7f9cb7fff700 1 -- 172.21.15.61:0/1068836016 learned_addr learned my addr 172.21.15.61:0/1068836016 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:24.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cbd28c700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb81462a0 0x7f9cb81555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb7fff700 1 -- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb80988b0 msgr2=0x7f9cb80b8350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb80988b0 0x7f9cb80b8350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb7fff700 1 -- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb81462a0 msgr2=0x7f9cb81555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb81462a0 0x7f9cb81555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb7fff700 1 -- 172.21.15.61:0/1068836016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac00b1e0 con 0x7f9cb81427b0 2022-07-20T03:09:24.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 0x7f9cb80b3040 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f9cac002f70 tx=0x7f9cac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cac0116e0 con 0x7f9cb81427b0 2022-07-20T03:09:24.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cac011840 con 0x7f9cb81427b0 2022-07-20T03:09:24.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb8152a30 con 0x7f9cb81427b0 2022-07-20T03:09:24.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cac011ba0 con 0x7f9cb81427b0 2022-07-20T03:09:24.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb80b3530 con 0x7f9cb81427b0 2022-07-20T03:09:24.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb8146d90 con 0x7f9cb81427b0 2022-07-20T03:09:24.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.623+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9cac025030 con 0x7f9cb81427b0 2022-07-20T03:09:24.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.627+0000 7f9cb5ffb700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c98053a40 0x7f9c98055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:24.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.627+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f9cac06cc40 con 0x7f9cb81427b0 2022-07-20T03:09:24.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.627+0000 7f9cbca8b700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c98053a40 0x7f9c98055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.627+0000 7f9cbca8b700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c98053a40 0x7f9c98055f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9ca800d000 tx=0x7f9ca8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.635+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9cac043ed0 con 0x7f9cb81427b0 2022-07-20T03:09:24.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.759+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9cb809c7d0 con 0x7f9cb81427b0 2022-07-20T03:09:24.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.759+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/1068836016 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v37) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9cac023020 con 0x7f9cb81427b0 2022-07-20T03:09:24.916 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:24.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c98053a40 msgr2=0x7f9c98055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c98053a40 0x7f9c98055f00 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f9ca800d000 tx=0x7f9ca8007b90).stop 2022-07-20T03:09:24.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 msgr2=0x7f9cb80b3040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:24.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 0x7f9cb80b3040 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f9cac002f70 tx=0x7f9cac015000).stop 2022-07-20T03:09:24.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 shutdown_connections 2022-07-20T03:09:24.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c98053a40 0x7f9c98055f00 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb81462a0 0x7f9cb81555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb81427b0 0x7f9cb80b3040 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 --2- 172.21.15.61:0/1068836016 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb80988b0 0x7f9cb80b8350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:24.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 >> 172.21.15.61:0/1068836016 conn(0x7f9cb80b5f00 msgr2=0x7f9cb805cdc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:24.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 shutdown_connections 2022-07-20T03:09:24.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:24.763+0000 7f9cbecef700 1 -- 172.21.15.61:0/1068836016 wait complete. 2022-07-20T03:09:24.961 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:09:25.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.067+0000 7f03bd0b0700 1 Processor -- start 2022-07-20T03:09:25.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.067+0000 7f03bd0b0700 1 -- start start 2022-07-20T03:09:25.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f03b80b0570 0x7f03b80b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b8141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f03b8142750 con 0x7f03b81422c0 2022-07-20T03:09:25.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f03b8143b10 con 0x7f03b80b0570 2022-07-20T03:09:25.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b81433d0 con 0x7f03b80aea10 2022-07-20T03:09:25.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f03b80b0570 0x7f03b80b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f03b80b0570 0x7f03b80b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44340/0 (socket says 172.21.15.61:44340) 2022-07-20T03:09:25.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 -- 172.21.15.61:0/2154035282 learned_addr learned my addr 172.21.15.61:0/2154035282 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:25.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03af7fe700 1 --2- 172.21.15.61:0/2154035282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b8141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 -- 172.21.15.61:0/2154035282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 msgr2=0x7f03b8141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:25.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b77fe700 1 -- 172.21.15.61:0/2154035282 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 598048195 0 0) 0x7f03b8142750 con 0x7f03b81422c0 2022-07-20T03:09:25.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 --2- 172.21.15.61:0/2154035282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b8141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 -- 172.21.15.61:0/2154035282 >> v1:172.21.15.117:6789/0 conn(0x7f03b81422c0 legacy=0x7f03b80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 -- 172.21.15.61:0/2154035282 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03b8143670 con 0x7f03b80b0570 2022-07-20T03:09:25.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b7fff700 1 --2- 172.21.15.61:0/2154035282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b80b0950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f03a8002f70 tx=0x7f03a8006eb0).ready entity=mon.0 client_cookie=1d36797d51eab665 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:25.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b77fe700 1 -- 172.21.15.61:0/2154035282 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03a800d570 con 0x7f03b80b0570 2022-07-20T03:09:25.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b77fe700 1 -- 172.21.15.61:0/2154035282 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a8000e90 con 0x7f03b80b0570 2022-07-20T03:09:25.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03b77fe700 1 -- 172.21.15.61:0/2154035282 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03a800d8c0 con 0x7f03b80b0570 2022-07-20T03:09:25.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- 172.21.15.61:0/2154035282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 msgr2=0x7f03b80b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/2154035282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b80b0950 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f03a8002f70 tx=0x7f03a8006eb0).stop 2022-07-20T03:09:25.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- 172.21.15.61:0/2154035282 shutdown_connections 2022-07-20T03:09:25.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/2154035282 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b8141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/2154035282 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b80b0950 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- 172.21.15.61:0/2154035282 >> 172.21.15.61:0/2154035282 conn(0x7f03b80b9360 msgr2=0x7f03b80b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:25.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- 172.21.15.61:0/2154035282 shutdown_connections 2022-07-20T03:09:25.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.071+0000 7f03bd0b0700 1 -- 172.21.15.61:0/2154035282 wait complete. 2022-07-20T03:09:25.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 Processor -- start 2022-07-20T03:09:25.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 -- start start 2022-07-20T03:09:25.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b813fe80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b8140370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b81422c0 0x7f03b8112b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b80b2170 con 0x7f03b80b0570 2022-07-20T03:09:25.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f03b8152850 con 0x7f03b81422c0 2022-07-20T03:09:25.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bd0b0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f03b8152af0 con 0x7f03b80aea10 2022-07-20T03:09:25.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b8140370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b8140370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44348/0 (socket says 172.21.15.61:44348) 2022-07-20T03:09:25.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 -- 172.21.15.61:0/1592246008 learned_addr learned my addr 172.21.15.61:0/1592246008 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:25.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03b7fff700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b813fe80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03bc8af700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b81422c0 0x7f03b8112b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 -- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 msgr2=0x7f03b813fe80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b813fe80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 -- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b81422c0 msgr2=0x7f03b8112b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b81422c0 0x7f03b8112b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 -- 172.21.15.61:0/1592246008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f03a800b1e0 con 0x7f03b80b0570 2022-07-20T03:09:25.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03af7fe700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b8140370 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f03a000c000 tx=0x7f03a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:25.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03a0010780 con 0x7f03b80b0570 2022-07-20T03:09:25.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.075+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f03a00108e0 con 0x7f03b80b0570 2022-07-20T03:09:25.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f03b80b1f00 con 0x7f03b80b0570 2022-07-20T03:09:25.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f03a0010c40 con 0x7f03b80b0570 2022-07-20T03:09:25.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f03b80b14c0 con 0x7f03b80b0570 2022-07-20T03:09:25.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f03b80b1010 con 0x7f03b80b0570 2022-07-20T03:09:25.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f03a001c460 con 0x7f03b80b0570 2022-07-20T03:09:25.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03b5ffb700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f039c053a40 0x7f039c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f03a006be50 con 0x7f03b80b0570 2022-07-20T03:09:25.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.079+0000 7f03b7fff700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f039c053a40 0x7f039c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.083+0000 7f03b7fff700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f039c053a40 0x7f039c055f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f03a801c000 tx=0x7f03a8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:25.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.087+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f03a0042fe0 con 0x7f03b80b0570 2022-07-20T03:09:25.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f03b80b14c0 con 0x7f03b80b0570 2022-07-20T03:09:25.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03b5ffb700 1 -- 172.21.15.61:0/1592246008 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v37) v1 ==== 84+0+143 (secure 0 0 0) 0x7f03b80b14c0 con 0x7f03b80b0570 2022-07-20T03:09:25.221 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:25.222 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:09:25.222 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:25.223 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:09:25.223 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:09:25.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:25.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:09:25.224 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:09:25.225 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:09:25.236 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:25.242 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:09:25.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f039c053a40 msgr2=0x7f039c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f039c053a40 0x7f039c055f00 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7f03a801c000 tx=0x7f03a8015000).stop 2022-07-20T03:09:25.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 msgr2=0x7f03b8140370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b8140370 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f03a000c000 tx=0x7f03a0006df0).stop 2022-07-20T03:09:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 shutdown_connections 2022-07-20T03:09:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f039c053a40 0x7f039c055f00 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f03b81422c0 0x7f03b8112b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f03b80b0570 0x7f03b8140370 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 --2- 172.21.15.61:0/1592246008 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f03b80aea10 0x7f03b813fe80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 >> 172.21.15.61:0/1592246008 conn(0x7f03b80b9360 msgr2=0x7f03b8141010 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 shutdown_connections 2022-07-20T03:09:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.215+0000 7f03bd0b0700 1 -- 172.21.15.61:0/1592246008 wait complete. 2022-07-20T03:09:25.268 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:09:25.268 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:09:25.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b6a76700 1 Processor -- start 2022-07-20T03:09:25.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b6a76700 1 -- start start 2022-07-20T03:09:25.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b6a76700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f60b00af940 0x7f60b00b0060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b6a76700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b005cc70 con 0x7f60b00b59f0 2022-07-20T03:09:25.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b6a76700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60b00b5dd0 con 0x7f60b0147ab0 2022-07-20T03:09:25.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b6a76700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60b005c760 con 0x7f60b00af940 2022-07-20T03:09:25.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5a74700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f60b00b59f0 0x7f60b01473a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40952/0 (socket says 172.21.15.61:40952) 2022-07-20T03:09:25.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5a74700 1 -- 172.21.15.61:0/4103609583 learned_addr learned my addr 172.21.15.61:0/4103609583 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:25.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5273700 1 --2- 172.21.15.61:0/4103609583 >> v2:172.21.15.117:3300/0 conn(0x7f60b00af940 0x7f60b00b0060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b4a72700 1 -- 172.21.15.61:0/4103609583 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1290119960 0 0) 0x7f60b005cc70 con 0x7f60b00b59f0 2022-07-20T03:09:25.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b4a72700 1 -- 172.21.15.61:0/4103609583 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6090003260 con 0x7f60b00b59f0 2022-07-20T03:09:25.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5273700 1 -- 172.21.15.61:0/4103609583 >> v1:172.21.15.117:6789/0 conn(0x7f60b0147ab0 legacy=0x7f60b00b0550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5273700 1 -- 172.21.15.61:0/4103609583 >> v1:172.21.15.61:6789/0 conn(0x7f60b00b59f0 legacy=0x7f60b01473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5273700 1 -- 172.21.15.61:0/4103609583 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60b005c9d0 con 0x7f60b00af940 2022-07-20T03:09:25.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.375+0000 7f60b5273700 1 --2- 172.21.15.61:0/4103609583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b00af940 0x7f60b00b0060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f60a80032f0 tx=0x7f60a80091d0).ready entity=mon.1 client_cookie=b3df4503beddc4da server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:25.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b4a72700 1 -- 172.21.15.61:0/4103609583 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60a800b710 con 0x7f60b00af940 2022-07-20T03:09:25.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b4a72700 1 -- 172.21.15.61:0/4103609583 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a800b870 con 0x7f60b00af940 2022-07-20T03:09:25.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b4a72700 1 -- 172.21.15.61:0/4103609583 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60a800b9e0 con 0x7f60b00af940 2022-07-20T03:09:25.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 -- 172.21.15.61:0/4103609583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b00af940 msgr2=0x7f60b00b0060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 --2- 172.21.15.61:0/4103609583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b00af940 0x7f60b00b0060 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f60a80032f0 tx=0x7f60a80091d0).stop 2022-07-20T03:09:25.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 -- 172.21.15.61:0/4103609583 shutdown_connections 2022-07-20T03:09:25.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 --2- 172.21.15.61:0/4103609583 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b00af940 0x7f60b00b0060 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 -- 172.21.15.61:0/4103609583 >> 172.21.15.61:0/4103609583 conn(0x7f60b0063480 msgr2=0x7f60b00b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:25.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 -- 172.21.15.61:0/4103609583 shutdown_connections 2022-07-20T03:09:25.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 -- 172.21.15.61:0/4103609583 wait complete. 2022-07-20T03:09:25.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.379+0000 7f60b6a76700 1 Processor -- start 2022-07-20T03:09:25.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 -- start start 2022-07-20T03:09:25.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60b00af940 0x7f60b014a970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 0x7f60b0156c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b0147ab0 0x7f60b01492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0157180 con 0x7f60b00b59f0 2022-07-20T03:09:25.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60b0144820 con 0x7f60b0147ab0 2022-07-20T03:09:25.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60b010e310 con 0x7f60b00af940 2022-07-20T03:09:25.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 0x7f60b0156c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 0x7f60b0156c90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44362/0 (socket says 172.21.15.61:44362) 2022-07-20T03:09:25.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 -- 172.21.15.61:0/2320352579 learned_addr learned my addr 172.21.15.61:0/2320352579 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:25.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6275700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b0147ab0 0x7f60b01492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5a74700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60b00af940 0x7f60b014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 -- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60b00af940 msgr2=0x7f60b014a970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60b00af940 0x7f60b014a970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 -- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b0147ab0 msgr2=0x7f60b01492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:25.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b0147ab0 0x7f60b01492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:25.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 -- 172.21.15.61:0/2320352579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60a8009470 con 0x7f60b00b59f0 2022-07-20T03:09:25.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b5273700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 0x7f60b0156c90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f60b0144960 tx=0x7f60a8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:25.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60a800b9e0 con 0x7f60b00b59f0 2022-07-20T03:09:25.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f60a80107a0 con 0x7f60b00b59f0 2022-07-20T03:09:25.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60b010e5b0 con 0x7f60b00b59f0 2022-07-20T03:09:25.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f60a800b710 con 0x7f60b00b59f0 2022-07-20T03:09:25.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.383+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60b0144480 con 0x7f60b00b59f0 2022-07-20T03:09:25.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.387+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60b01572d0 con 0x7f60b00b59f0 2022-07-20T03:09:25.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.387+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f60a8027020 con 0x7f60b00b59f0 2022-07-20T03:09:25.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.387+0000 7f60aeffd700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6094053ad0 0x7f6094055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:25.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.387+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(37..37 src has 1..37) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f60a806d1c0 con 0x7f60b00b59f0 2022-07-20T03:09:25.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.387+0000 7f60b5a74700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6094053ad0 0x7f6094055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:25.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.387+0000 7f60b5a74700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6094053ad0 0x7f6094055f90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f609800d000 tx=0x7f6098009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:25.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.395+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f60a8044450 con 0x7f60b00b59f0 2022-07-20T03:09:25.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:25.519+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f60b010e740 con 0x7f60b00b59f0 2022-07-20T03:09:26.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.099+0000 7f60aeffd700 1 -- 172.21.15.61:0/2320352579 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v38) v1 ==== 133+0+0 (secure 0 0 0) 0x7f60a80150a0 con 0x7f60b00b59f0 2022-07-20T03:09:26.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6094053ad0 msgr2=0x7f6094055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6094053ad0 0x7f6094055f90 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f609800d000 tx=0x7f6098009040).stop 2022-07-20T03:09:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 msgr2=0x7f60b0156c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:26.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 0x7f60b0156c90 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f60b0144960 tx=0x7f60a8016000).stop 2022-07-20T03:09:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 shutdown_connections 2022-07-20T03:09:26.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6094053ad0 0x7f6094055f90 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60b0147ab0 0x7f60b01492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60b00b59f0 0x7f60b0156c90 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 --2- 172.21.15.61:0/2320352579 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60b00af940 0x7f60b014a970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 >> 172.21.15.61:0/2320352579 conn(0x7f60b0063480 msgr2=0x7f60b0140570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:26.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 shutdown_connections 2022-07-20T03:09:26.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.103+0000 7f60b6a76700 1 -- 172.21.15.61:0/2320352579 wait complete. 2022-07-20T03:09:26.116 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:09:26.128 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:09:26.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 Processor -- start 2022-07-20T03:09:26.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 -- start start 2022-07-20T03:09:26.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc0b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:26.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1edc0af920 0x7f1edc0ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:26.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1edc0adf70 con 0x7f1edc05d0c0 2022-07-20T03:09:26.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1edc0b5db0 con 0x7f1edc0af920 2022-07-20T03:09:26.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee16f9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc146300 con 0x7f1edc0b59d0 2022-07-20T03:09:26.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc0b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:26.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1edb7fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1edc0af920 0x7f1edc0ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:26.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee0ef8700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1edc05d0c0 0x7f1edc0ae8c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:49472/0 (socket says 172.21.15.61:49472) 2022-07-20T03:09:26.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1ee0ef8700 1 -- 172.21.15.61:0/1360200810 learned_addr learned my addr 172.21.15.61:0/1360200810 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:26.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1edb7fe700 1 -- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 msgr2=0x7f1edc0b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:26.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1edb7fe700 1 --2- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc0b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.235+0000 7f1edb7fe700 1 -- 172.21.15.61:0/1360200810 >> v1:172.21.15.117:6789/0 conn(0x7f1edc05d0c0 legacy=0x7f1edc0ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:26.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edb7fe700 1 -- 172.21.15.61:0/1360200810 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1edc146060 con 0x7f1edc0af920 2022-07-20T03:09:26.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edbfff700 1 --2- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc0b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:26.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edb7fe700 1 --2- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc0ae3d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1edc0add70 tx=0x7f1ecc006df0).ready entity=mon.1 client_cookie=356b45af55abad53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:26.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edaffd700 1 -- 172.21.15.61:0/1360200810 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ecc00d570 con 0x7f1edc0af920 2022-07-20T03:09:26.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edaffd700 1 -- 172.21.15.61:0/1360200810 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ecc000e90 con 0x7f1edc0af920 2022-07-20T03:09:26.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edaffd700 1 -- 172.21.15.61:0/1360200810 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ecc00d8c0 con 0x7f1edc0af920 2022-07-20T03:09:26.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 msgr2=0x7f1edc0ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:26.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc0ae3d0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f1edc0add70 tx=0x7f1ecc006df0).stop 2022-07-20T03:09:26.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1360200810 shutdown_connections 2022-07-20T03:09:26.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc0ae3d0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1360200810 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc0b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1360200810 >> 172.21.15.61:0/1360200810 conn(0x7f1edc0634c0 msgr2=0x7f1edc0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:26.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1360200810 shutdown_connections 2022-07-20T03:09:26.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1360200810 wait complete. 2022-07-20T03:09:26.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 Processor -- start 2022-07-20T03:09:26.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- start start 2022-07-20T03:09:26.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 0x7f1edc065da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:26.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc066290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:26.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc115630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:26.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc1126d0 con 0x7f1edc05d0c0 2022-07-20T03:09:26.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc067360 con 0x7f1edc0af920 2022-07-20T03:09:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee16f9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1edc067600 con 0x7f1edc0b59d0 2022-07-20T03:09:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 0x7f1edc065da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edb7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc066290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1ee0ef8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc115630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:26.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edb7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc066290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54056/0 (socket says 172.21.15.61:54056) 2022-07-20T03:09:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edb7fe700 1 -- 172.21.15.61:0/1624277700 learned_addr learned my addr 172.21.15.61:0/1624277700 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.239+0000 7f1edbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 0x7f1edc065da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44374/0 (socket says 172.21.15.61:44374) 2022-07-20T03:09:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edbfff700 1 -- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 msgr2=0x7f1edc115630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:26.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edbfff700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc115630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edbfff700 1 -- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 msgr2=0x7f1edc066290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edbfff700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc066290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edbfff700 1 -- 172.21.15.61:0/1624277700 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ed000b180 con 0x7f1edc05d0c0 2022-07-20T03:09:26.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edb7fe700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc066290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:26.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edbfff700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 0x7f1edc065da0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f1ed000c000 tx=0x7f1ed0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:26.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed0010780 con 0x7f1edc05d0c0 2022-07-20T03:09:26.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ed00108e0 con 0x7f1edc05d0c0 2022-07-20T03:09:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ecc00b180 con 0x7f1edc05d0c0 2022-07-20T03:09:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ed0010c40 con 0x7f1edc05d0c0 2022-07-20T03:09:26.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1edc152b20 con 0x7f1edc05d0c0 2022-07-20T03:09:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1edc067800 con 0x7f1edc05d0c0 2022-07-20T03:09:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1ed0026430 con 0x7f1edc05d0c0 2022-07-20T03:09:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ed97fa700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1ebc053a40 0x7f1ebc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(39..39 src has 1..39) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f1ed006ca30 con 0x7f1edc05d0c0 2022-07-20T03:09:26.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.243+0000 7f1edb7fe700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1ebc053a40 0x7f1ebc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.247+0000 7f1edb7fe700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1ebc053a40 0x7f1ebc055f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1ecc01a000 tx=0x7f1ecc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:26.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.251+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1ed0043b00 con 0x7f1edc05d0c0 2022-07-20T03:09:26.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:26.375+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f1edc067800 con 0x7f1edc05d0c0 2022-07-20T03:09:27.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.123+0000 7f1ed97fa700 1 -- 172.21.15.61:0/1624277700 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v40) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1edc067800 con 0x7f1edc05d0c0 2022-07-20T03:09:27.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1ebc053a40 msgr2=0x7f1ebc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1ebc053a40 0x7f1ebc055f00 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f1ecc01a000 tx=0x7f1ecc014000).stop 2022-07-20T03:09:27.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 msgr2=0x7f1edc065da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 0x7f1edc065da0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f1ed000c000 tx=0x7f1ed0006df0).stop 2022-07-20T03:09:27.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 shutdown_connections 2022-07-20T03:09:27.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1ebc053a40 0x7f1ebc055f00 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1edc0af920 0x7f1edc066290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1edc0b59d0 0x7f1edc115630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 --2- 172.21.15.61:0/1624277700 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1edc05d0c0 0x7f1edc065da0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 >> 172.21.15.61:0/1624277700 conn(0x7f1edc0634c0 msgr2=0x7f1edc1402f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:27.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 shutdown_connections 2022-07-20T03:09:27.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.131+0000 7f1ee16f9700 1 -- 172.21.15.61:0/1624277700 wait complete. 2022-07-20T03:09:27.142 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:09:27.179 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:09:27.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 Processor -- start 2022-07-20T03:09:27.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 -- start start 2022-07-20T03:09:27.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7feb6c0b1700 0x7feb6c0abef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb6c13eed0 0x7feb6c13f2e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb6c0ad070 con 0x7feb6c05bd90 2022-07-20T03:09:27.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6c0acaf0 con 0x7feb6c13eed0 2022-07-20T03:09:27.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.311+0000 7feb73d5a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb6c0603e0 con 0x7feb6c0b1700 2022-07-20T03:09:27.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb722f7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb6c13eed0 0x7feb6c13f2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb712f5700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feb6c05bd90 0x7feb6c0ac3e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40980/0 (socket says 172.21.15.61:40980) 2022-07-20T03:09:27.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb712f5700 1 -- 172.21.15.61:0/901024442 learned_addr learned my addr 172.21.15.61:0/901024442 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:27.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb71af6700 1 --2- 172.21.15.61:0/901024442 >> v2:172.21.15.117:3300/0 conn(0x7feb6c0b1700 0x7feb6c0abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb70af4700 1 -- 172.21.15.61:0/901024442 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3497029572 0 0) 0x7feb6c0ad070 con 0x7feb6c05bd90 2022-07-20T03:09:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb70af4700 1 -- 172.21.15.61:0/901024442 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb58003260 con 0x7feb6c05bd90 2022-07-20T03:09:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb722f7700 1 -- 172.21.15.61:0/901024442 >> v2:172.21.15.117:3300/0 conn(0x7feb6c0b1700 msgr2=0x7feb6c0abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb722f7700 1 --2- 172.21.15.61:0/901024442 >> v2:172.21.15.117:3300/0 conn(0x7feb6c0b1700 0x7feb6c0abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb722f7700 1 -- 172.21.15.61:0/901024442 >> v1:172.21.15.61:6789/0 conn(0x7feb6c05bd90 legacy=0x7feb6c0ac3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb722f7700 1 -- 172.21.15.61:0/901024442 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c0ace10 con 0x7feb6c13eed0 2022-07-20T03:09:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb722f7700 1 --2- 172.21.15.61:0/901024442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c13eed0 0x7feb6c13f2e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7feb5c002f70 tx=0x7feb5c006df0).ready entity=mon.0 client_cookie=e444633f7a91eb23 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb70af4700 1 -- 172.21.15.61:0/901024442 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb5c00d4f0 con 0x7feb6c13eed0 2022-07-20T03:09:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb70af4700 1 -- 172.21.15.61:0/901024442 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb5c000e90 con 0x7feb6c13eed0 2022-07-20T03:09:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb70af4700 1 -- 172.21.15.61:0/901024442 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb5c00d840 con 0x7feb6c13eed0 2022-07-20T03:09:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 -- 172.21.15.61:0/901024442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c13eed0 msgr2=0x7feb6c13f2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 --2- 172.21.15.61:0/901024442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c13eed0 0x7feb6c13f2e0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7feb5c002f70 tx=0x7feb5c006df0).stop 2022-07-20T03:09:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 -- 172.21.15.61:0/901024442 shutdown_connections 2022-07-20T03:09:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 --2- 172.21.15.61:0/901024442 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c13eed0 0x7feb6c13f2e0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 --2- 172.21.15.61:0/901024442 >> v2:172.21.15.117:3300/0 conn(0x7feb6c0b1700 0x7feb6c0abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 -- 172.21.15.61:0/901024442 >> 172.21.15.61:0/901024442 conn(0x7feb6c0b5910 msgr2=0x7feb6c0b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:27.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 -- 172.21.15.61:0/901024442 shutdown_connections 2022-07-20T03:09:27.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.315+0000 7feb73d5a700 1 -- 172.21.15.61:0/901024442 wait complete. 2022-07-20T03:09:27.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 Processor -- start 2022-07-20T03:09:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 -- start start 2022-07-20T03:09:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 0x7feb6c1459f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb6c0b1700 0x7feb6c0b4a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb6c13eed0 0x7feb6c1464a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c064730 con 0x7feb6c05bd90 2022-07-20T03:09:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c147050 con 0x7feb6c13eed0 2022-07-20T03:09:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb6c066f50 con 0x7feb6c0b1700 2022-07-20T03:09:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 0x7feb6c1459f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 0x7feb6c1459f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44388/0 (socket says 172.21.15.61:44388) 2022-07-20T03:09:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 -- 172.21.15.61:0/2028817333 learned_addr learned my addr 172.21.15.61:0/2028817333 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb712f5700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb6c0b1700 0x7feb6c0b4a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb722f7700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb6c13eed0 0x7feb6c1464a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 -- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb6c0b1700 msgr2=0x7feb6c0b4a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb6c0b1700 0x7feb6c0b4a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 -- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb6c13eed0 msgr2=0x7feb6c1464a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb6c13eed0 0x7feb6c1464a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 -- 172.21.15.61:0/2028817333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb5c00b180 con 0x7feb6c05bd90 2022-07-20T03:09:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb71af6700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 0x7feb6c1459f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7feb6400c000 tx=0x7feb64006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb64010780 con 0x7feb6c05bd90 2022-07-20T03:09:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb640108e0 con 0x7feb6c05bd90 2022-07-20T03:09:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 -- 172.21.15.61:0/2028817333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb6c147d40 con 0x7feb6c05bd90 2022-07-20T03:09:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb64010c40 con 0x7feb6c05bd90 2022-07-20T03:09:27.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.319+0000 7feb73d5a700 1 -- 172.21.15.61:0/2028817333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb6c147500 con 0x7feb6c05bd90 2022-07-20T03:09:27.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.323+0000 7feb73d5a700 1 -- 172.21.15.61:0/2028817333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb6c0617c0 con 0x7feb6c05bd90 2022-07-20T03:09:27.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.323+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feb6401c460 con 0x7feb6c05bd90 2022-07-20T03:09:27.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.323+0000 7feb6affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb4c053a40 0x7feb4c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.323+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(40..40 src has 1..40) v4 ==== 5957+0+0 (secure 0 0 0) 0x7feb6406c240 con 0x7feb6c05bd90 2022-07-20T03:09:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.323+0000 7feb712f5700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb4c053a40 0x7feb4c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.327+0000 7feb712f5700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb4c053a40 0x7feb4c055f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7feb6000d000 tx=0x7feb60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.331+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feb640430a0 con 0x7feb6c05bd90 2022-07-20T03:09:27.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.495+0000 7feb73d5a700 1 -- 172.21.15.61:0/2028817333 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7feb6c0574e0 con 0x7feb6c05bd90 2022-07-20T03:09:27.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.499+0000 7feb6affd700 1 -- 172.21.15.61:0/2028817333 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 7 and data pool 8 v19) v1 ==== 187+0+0 (secure 0 0 0) 0x7feb640466c0 con 0x7feb6c05bd90 2022-07-20T03:09:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 -- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb4c053a40 msgr2=0x7feb4c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb4c053a40 0x7feb4c055f00 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7feb6000d000 tx=0x7feb60009040).stop 2022-07-20T03:09:27.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 -- 172.21.15.61:0/2028817333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 msgr2=0x7feb6c1459f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 0x7feb6c1459f0 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7feb6400c000 tx=0x7feb64006df0).stop 2022-07-20T03:09:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 -- 172.21.15.61:0/2028817333 shutdown_connections 2022-07-20T03:09:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb4c053a40 0x7feb4c055f00 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb6c13eed0 0x7feb6c1464a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb6c0b1700 0x7feb6c0b4a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 --2- 172.21.15.61:0/2028817333 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb6c05bd90 0x7feb6c1459f0 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.503+0000 7feb4affd700 1 -- 172.21.15.61:0/2028817333 >> 172.21.15.61:0/2028817333 conn(0x7feb6c0b5910 msgr2=0x7feb6c140510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:27.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.507+0000 7feb4affd700 1 -- 172.21.15.61:0/2028817333 shutdown_connections 2022-07-20T03:09:27.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.507+0000 7feb4affd700 1 -- 172.21.15.61:0/2028817333 wait complete. 2022-07-20T03:09:27.513 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 7 and data pool 8 2022-07-20T03:09:27.526 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:09:27.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 Processor -- start 2022-07-20T03:09:27.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 -- start start 2022-07-20T03:09:27.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fde680608b0 0x7fde6805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680b42a0 con 0x7fde680b3de0 2022-07-20T03:09:27.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde68060c90 con 0x7fde680608b0 2022-07-20T03:09:27.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde6dff8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde680af410 con 0x7fde680afa10 2022-07-20T03:09:27.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde67fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fde680b3de0 0x7fde68146350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:40998/0 (socket says 172.21.15.61:40998) 2022-07-20T03:09:27.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde67fff700 1 -- 172.21.15.61:0/239905007 learned_addr learned my addr 172.21.15.61:0/239905007 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:27.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde66ffd700 1 --2- 172.21.15.61:0/239905007 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde677fe700 1 --2- 172.21.15.61:0/239905007 >> v2:172.21.15.117:3300/0 conn(0x7fde680608b0 0x7fde6805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde667fc700 1 -- 172.21.15.61:0/239905007 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1525688858 0 0) 0x7fde680b42a0 con 0x7fde680b3de0 2022-07-20T03:09:27.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde667fc700 1 -- 172.21.15.61:0/239905007 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fde54003260 con 0x7fde680b3de0 2022-07-20T03:09:27.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde667fc700 1 -- 172.21.15.61:0/239905007 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3933710255 0 0) 0x7fde54003260 con 0x7fde680b3de0 2022-07-20T03:09:27.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde66ffd700 1 -- 172.21.15.61:0/239905007 >> v2:172.21.15.117:3300/0 conn(0x7fde680608b0 msgr2=0x7fde6805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:27.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde66ffd700 1 --2- 172.21.15.61:0/239905007 >> v2:172.21.15.117:3300/0 conn(0x7fde680608b0 0x7fde6805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde66ffd700 1 -- 172.21.15.61:0/239905007 >> v1:172.21.15.61:6789/0 conn(0x7fde680b3de0 legacy=0x7fde68146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde66ffd700 1 -- 172.21.15.61:0/239905007 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680af6b0 con 0x7fde680afa10 2022-07-20T03:09:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde677fe700 1 --2- 172.21.15.61:0/239905007 >> v2:172.21.15.117:3300/0 conn(0x7fde680608b0 0x7fde6805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:27.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde66ffd700 1 --2- 172.21.15.61:0/239905007 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b46d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fde60001800 tx=0x7fde580091d0).ready entity=mon.2 client_cookie=5a3fa486a7980732 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:27.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde667fc700 1 -- 172.21.15.61:0/239905007 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde60001660 con 0x7fde680afa10 2022-07-20T03:09:27.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde667fc700 1 -- 172.21.15.61:0/239905007 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde5800f510 con 0x7fde680afa10 2022-07-20T03:09:27.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.611+0000 7fde667fc700 1 -- 172.21.15.61:0/239905007 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde5800f680 con 0x7fde680afa10 2022-07-20T03:09:27.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- 172.21.15.61:0/239905007 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 msgr2=0x7fde680b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 --2- 172.21.15.61:0/239905007 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b46d0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fde60001800 tx=0x7fde580091d0).stop 2022-07-20T03:09:27.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- 172.21.15.61:0/239905007 shutdown_connections 2022-07-20T03:09:27.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 --2- 172.21.15.61:0/239905007 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b46d0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 --2- 172.21.15.61:0/239905007 >> v2:172.21.15.117:3300/0 conn(0x7fde680608b0 0x7fde6805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- 172.21.15.61:0/239905007 >> 172.21.15.61:0/239905007 conn(0x7fde680b5ee0 msgr2=0x7fde680b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:27.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- 172.21.15.61:0/239905007 shutdown_connections 2022-07-20T03:09:27.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- 172.21.15.61:0/239905007 wait complete. 2022-07-20T03:09:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 Processor -- start 2022-07-20T03:09:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- start start 2022-07-20T03:09:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680608b0 0x7fde68155ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b1930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 0x7fde680b28a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde681554f0 con 0x7fde680b3de0 2022-07-20T03:09:27.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68155790 con 0x7fde680608b0 2022-07-20T03:09:27.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde6dff8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde68155a30 con 0x7fde680afa10 2022-07-20T03:09:27.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 0x7fde680b28a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde66ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b1930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde66ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b1930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:59306/0 (socket says 172.21.15.61:59306) 2022-07-20T03:09:27.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 0x7fde680b28a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44406/0 (socket says 172.21.15.61:44406) 2022-07-20T03:09:27.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde66ffd700 1 -- 172.21.15.61:0/1286828170 learned_addr learned my addr 172.21.15.61:0/1286828170 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:27.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.615+0000 7fde677fe700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680608b0 0x7fde68155ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde67fff700 1 -- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 msgr2=0x7fde680b1930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde67fff700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b1930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde67fff700 1 -- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680608b0 msgr2=0x7fde68155ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde67fff700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680608b0 0x7fde68155ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde67fff700 1 -- 172.21.15.61:0/1286828170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5c00b180 con 0x7fde680b3de0 2022-07-20T03:09:27.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde66ffd700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:27.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde67fff700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 0x7fde680b28a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fde600018c0 tx=0x7fde60001950).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:27.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde6000ce70 con 0x7fde680b3de0 2022-07-20T03:09:27.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde60007170 con 0x7fde680b3de0 2022-07-20T03:09:27.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde58009470 con 0x7fde680b3de0 2022-07-20T03:09:27.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde60004730 con 0x7fde680b3de0 2022-07-20T03:09:27.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde680b0aa0 con 0x7fde680b3de0 2022-07-20T03:09:27.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde40003410 con 0x7fde680b3de0 2022-07-20T03:09:27.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.619+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fde60004890 con 0x7fde680b3de0 2022-07-20T03:09:27.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.623+0000 7fde64ff9700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.623+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fde6006b930 con 0x7fde680b3de0 2022-07-20T03:09:27.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.623+0000 7fde677fe700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.623+0000 7fde677fe700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fde5c00b000 tx=0x7fde5c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:27.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.627+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde60042830 con 0x7fde680b3de0 2022-07-20T03:09:27.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.755+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fde400036c0 con 0x7fde680b3de0 2022-07-20T03:09:27.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.755+0000 7fde64ff9700 1 -- 172.21.15.61:0/1286828170 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16099 (secure 0 0 0) 0x7fde60045e50 con 0x7fde680b3de0 2022-07-20T03:09:27.798 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:27.799 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":41,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:09:27.499210+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:09:25.577917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T03:09:26.436916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:09:27.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 msgr2=0x7fde48055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fde5c00b000 tx=0x7fde5c006df0).stop 2022-07-20T03:09:27.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 msgr2=0x7fde680b28a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 0x7fde680b28a0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fde600018c0 tx=0x7fde60001950).stop 2022-07-20T03:09:27.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 shutdown_connections 2022-07-20T03:09:27.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b3de0 0x7fde680b28a0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680afa10 0x7fde680b1930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 --2- 172.21.15.61:0/1286828170 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680608b0 0x7fde68155ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 >> 172.21.15.61:0/1286828170 conn(0x7fde680b5ee0 msgr2=0x7fde680b8b20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:27.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 shutdown_connections 2022-07-20T03:09:27.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.759+0000 7fde6dff8700 1 -- 172.21.15.61:0/1286828170 wait complete. 2022-07-20T03:09:27.805 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:09:27.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.927+0000 7f1a457f3700 1 Processor -- start 2022-07-20T03:09:27.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.927+0000 7f1a457f3700 1 -- start start 2022-07-20T03:09:27.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a457f3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1a400ab3b0 0x7f1a40142690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a457f3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a400b22d0 con 0x7f1a400b1ef0 2022-07-20T03:09:27.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a457f3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1a400ac1c0 con 0x7f1a40061020 2022-07-20T03:09:27.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a457f3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1a40142b80 con 0x7f1a400ab3b0 2022-07-20T03:09:27.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1a400b1ef0 0x7f1a40141f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41008/0 (socket says 172.21.15.61:41008) 2022-07-20T03:09:27.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 -- 172.21.15.61:0/2568816867 learned_addr learned my addr 172.21.15.61:0/2568816867 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:27.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3f7fe700 1 --2- 172.21.15.61:0/2568816867 >> v2:172.21.15.117:3300/0 conn(0x7f1a400ab3b0 0x7f1a40142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:27.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1064385609 0 0) 0x7f1a400b22d0 con 0x7f1a400b1ef0 2022-07-20T03:09:27.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1a2c003260 con 0x7f1a400b1ef0 2022-07-20T03:09:27.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 293020522 0 0) 0x7f1a2c003260 con 0x7f1a400b1ef0 2022-07-20T03:09:27.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 >> v2:172.21.15.117:3300/0 conn(0x7f1a400ab3b0 msgr2=0x7f1a40142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:27.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 --2- 172.21.15.61:0/2568816867 >> v2:172.21.15.117:3300/0 conn(0x7f1a400ab3b0 0x7f1a40142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:27.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 >> v1:172.21.15.117:6789/0 conn(0x7f1a40061020 legacy=0x7f1a40061420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a400ac460 con 0x7f1a400b1ef0 2022-07-20T03:09:27.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f1a30001660 con 0x7f1a400b1ef0 2022-07-20T03:09:27.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 >> v1:172.21.15.61:6789/0 conn(0x7f1a400b1ef0 legacy=0x7f1a40141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:27.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a2c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a2c004500 0x7f1a2c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a2c004ec0 0x7f1a2c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:27.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40142b80 con 0x7f1a2c004ec0 2022-07-20T03:09:27.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c00be50 con 0x7f1a2c003750 2022-07-20T03:09:27.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a2c00c010 con 0x7f1a2c004500 2022-07-20T03:09:28.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a2c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3f7fe700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a2c004500 0x7f1a2c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a44ff2700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a2c004ec0 0x7f1a2c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 -- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a2c004500 msgr2=0x7f1a2c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:28.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a2c004500 0x7f1a2c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 -- 172.21.15.61:0/2568816867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a2c004ec0 msgr2=0x7f1a2c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:28.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a2c004ec0 0x7f1a2c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 -- 172.21.15.61:0/2568816867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a2c003260 con 0x7f1a2c003750 2022-07-20T03:09:28.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a44ff2700 1 -- 172.21.15.61:0/2568816867 reap_dead start 2022-07-20T03:09:28.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a2c003b70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1a30001370 tx=0x7f1a400b3cb0).ready entity=mon.1 client_cookie=84f773fc8fe4106c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a300070d0 con 0x7f1a2c003750 2022-07-20T03:09:28.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30007230 con 0x7f1a2c003750 2022-07-20T03:09:28.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.931+0000 7f1a3effd700 1 -- 172.21.15.61:0/2568816867 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3000c8f0 con 0x7f1a2c003750 2022-07-20T03:09:28.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 msgr2=0x7f1a2c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a2c003b70 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f1a30001370 tx=0x7f1a400b3cb0).stop 2022-07-20T03:09:28.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- 172.21.15.61:0/2568816867 shutdown_connections 2022-07-20T03:09:28.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 --2- 172.21.15.61:0/2568816867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a2c003b70 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- 172.21.15.61:0/2568816867 >> 172.21.15.61:0/2568816867 conn(0x7f1a400b2ca0 msgr2=0x7f1a400b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:28.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- 172.21.15.61:0/2568816867 shutdown_connections 2022-07-20T03:09:28.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- 172.21.15.61:0/2568816867 wait complete. 2022-07-20T03:09:28.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 Processor -- start 2022-07-20T03:09:28.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- start start 2022-07-20T03:09:28.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a4013d8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ad640 0x7f1a40155340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a401524f0 0x7f1a40152970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a401471d0 con 0x7f1a400ad640 2022-07-20T03:09:28.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a4009b4c0 con 0x7f1a2c003750 2022-07-20T03:09:28.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a457f3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1a40145550 con 0x7f1a401524f0 2022-07-20T03:09:28.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a4013d8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a3f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ad640 0x7f1a40155340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a3ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a4013d8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54102/0 (socket says 172.21.15.61:54102) 2022-07-20T03:09:28.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a3ffff700 1 -- 172.21.15.61:0/772848588 learned_addr learned my addr 172.21.15.61:0/772848588 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:28.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.935+0000 7f1a44ff2700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a401524f0 0x7f1a40152970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3ffff700 1 -- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a401524f0 msgr2=0x7f1a40152970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a401524f0 0x7f1a40152970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3ffff700 1 -- 172.21.15.61:0/772848588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ad640 msgr2=0x7f1a40155340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ad640 0x7f1a40155340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3ffff700 1 -- 172.21.15.61:0/772848588 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a40151db0 con 0x7f1a2c003750 2022-07-20T03:09:28.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3f7fe700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ad640 0x7f1a40155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:28.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3ffff700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a4013d8b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1a4009c330 tx=0x7f1a30013000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a3000c6d0 con 0x7f1a2c003750 2022-07-20T03:09:28.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1a30000e90 con 0x7f1a2c003750 2022-07-20T03:09:28.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a40152050 con 0x7f1a2c003750 2022-07-20T03:09:28.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1a30004710 con 0x7f1a2c003750 2022-07-20T03:09:28.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a400adce0 con 0x7f1a2c003750 2022-07-20T03:09:28.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a401521e0 con 0x7f1a2c003750 2022-07-20T03:09:28.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1a3001c070 con 0x7f1a2c003750 2022-07-20T03:09:28.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3d7fa700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a20053ad0 0x7f1a20055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.939+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1a3006bd30 con 0x7f1a2c003750 2022-07-20T03:09:28.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.943+0000 7f1a3f7fe700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a20053ad0 0x7f1a20055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.943+0000 7f1a3f7fe700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a20053ad0 0x7f1a20055f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1a3400c000 tx=0x7f1a34008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:27.947+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1a30042bf0 con 0x7f1a2c003750 2022-07-20T03:09:28.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.075+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1a400ac700 con 0x7f1a2c003750 2022-07-20T03:09:28.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.079+0000 7f1a3d7fa700 1 -- 172.21.15.61:0/772848588 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v41) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f1a30046210 con 0x7f1a2c003750 2022-07-20T03:09:28.085 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:28.086 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":41,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:09:27.499210+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:09:25.577917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T03:09:26.436916+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:09:28.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a20053ad0 msgr2=0x7f1a20055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a20053ad0 0x7f1a20055f90 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f1a3400c000 tx=0x7f1a34008040).stop 2022-07-20T03:09:28.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 msgr2=0x7f1a4013d8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a4013d8b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f1a4009c330 tx=0x7f1a30013000).stop 2022-07-20T03:09:28.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 shutdown_connections 2022-07-20T03:09:28.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1a20053ad0 0x7f1a20055f90 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1a401524f0 0x7f1a40152970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1a400ad640 0x7f1a40155340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 --2- 172.21.15.61:0/772848588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1a2c003750 0x7f1a4013d8b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 >> 172.21.15.61:0/772848588 conn(0x7f1a400b2ca0 msgr2=0x7f1a40061400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:28.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 shutdown_connections 2022-07-20T03:09:28.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.083+0000 7f1a457f3700 1 -- 172.21.15.61:0/772848588 wait complete. 2022-07-20T03:09:28.116 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:09:28.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.223+0000 7ff6c3041700 1 Processor -- start 2022-07-20T03:09:28.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.223+0000 7ff6c3041700 1 -- start start 2022-07-20T03:09:28.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff6bc0b0580 0x7ff6bc05c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff6bc1468a0 0x7ff6bc0b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6bc0b16a0 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6bc067f70 con 0x7ff6bc1468a0 2022-07-20T03:09:28.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6bc0ad0e0 con 0x7ff6bc0b0580 2022-07-20T03:09:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff6bc1468a0 0x7ff6bc0b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff6bc1468a0 0x7ff6bc0b1bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44434/0 (socket says 172.21.15.61:44434) 2022-07-20T03:09:28.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 -- 172.21.15.61:0/1801611483 learned_addr learned my addr 172.21.15.61:0/1801611483 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:28.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c183e700 1 --2- 172.21.15.61:0/1801611483 >> v2:172.21.15.117:3300/0 conn(0x7ff6bc0b0580 0x7ff6bc05c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 -- 172.21.15.61:0/1801611483 >> v2:172.21.15.117:3300/0 conn(0x7ff6bc0b0580 msgr2=0x7ff6bc05c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:28.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 --2- 172.21.15.61:0/1801611483 >> v2:172.21.15.117:3300/0 conn(0x7ff6bc0b0580 0x7ff6bc05c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 -- 172.21.15.61:0/1801611483 >> v1:172.21.15.117:6789/0 conn(0x7ff6bc0b23b0 legacy=0x7ff6bc146190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 -- 172.21.15.61:0/1801611483 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6bc0ae660 con 0x7ff6bc1468a0 2022-07-20T03:09:28.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c2840700 1 --2- 172.21.15.61:0/1801611483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc0b1bd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7ff6b40032f0 tx=0x7ff6b4009290).ready entity=mon.0 client_cookie=94521442601a4668 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c103d700 1 -- 172.21.15.61:0/1801611483 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b400b870 con 0x7ff6bc1468a0 2022-07-20T03:09:28.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c103d700 1 -- 172.21.15.61:0/1801611483 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b400b9d0 con 0x7ff6bc1468a0 2022-07-20T03:09:28.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c103d700 1 -- 172.21.15.61:0/1801611483 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b400bb40 con 0x7ff6bc1468a0 2022-07-20T03:09:28.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- 172.21.15.61:0/1801611483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 msgr2=0x7ff6bc0b1bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 --2- 172.21.15.61:0/1801611483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc0b1bd0 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7ff6b40032f0 tx=0x7ff6b4009290).stop 2022-07-20T03:09:28.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- 172.21.15.61:0/1801611483 shutdown_connections 2022-07-20T03:09:28.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 --2- 172.21.15.61:0/1801611483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc0b1bd0 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 --2- 172.21.15.61:0/1801611483 >> v2:172.21.15.117:3300/0 conn(0x7ff6bc0b0580 0x7ff6bc05c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- 172.21.15.61:0/1801611483 >> 172.21.15.61:0/1801611483 conn(0x7ff6bc0632d0 msgr2=0x7ff6bc0636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:28.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- 172.21.15.61:0/1801611483 shutdown_connections 2022-07-20T03:09:28.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.227+0000 7ff6c3041700 1 -- 172.21.15.61:0/1801611483 wait complete. 2022-07-20T03:09:28.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 Processor -- start 2022-07-20T03:09:28.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 -- start start 2022-07-20T03:09:28.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6bc0b0580 0x7ff6bc0659c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6bc0b23b0 0x7ff6bc140730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc140c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc15aa10 con 0x7ff6bc1468a0 2022-07-20T03:09:28.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc111330 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c3041700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6bc10f610 con 0x7ff6bc0b0580 2022-07-20T03:09:28.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c2840700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc140c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c183e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6bc0b23b0 0x7ff6bc140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c2840700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc140c20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44440/0 (socket says 172.21.15.61:44440) 2022-07-20T03:09:28.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c2840700 1 -- 172.21.15.61:0/3326140140 learned_addr learned my addr 172.21.15.61:0/3326140140 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:28.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c203f700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6bc0b0580 0x7ff6bc0659c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c183e700 1 -- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6bc0b0580 msgr2=0x7ff6bc0659c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c183e700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6bc0b0580 0x7ff6bc0659c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c183e700 1 -- 172.21.15.61:0/3326140140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 msgr2=0x7ff6bc140c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c183e700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc140c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c183e700 1 -- 172.21.15.61:0/3326140140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6b4009530 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.231+0000 7ff6c2840700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:28.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6c183e700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6bc0b23b0 0x7ff6bc140730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff6b000c000 tx=0x7ff6b0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b0011750 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6b00118b0 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6b0011c10 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6bc159ae0 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6bc0665a0 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6bc0b1390 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff6b0011d70 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6af7fe700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff69c053ad0 0x7ff69c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(41..41 src has 1..41) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff6b006c110 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.235+0000 7ff6c203f700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff69c053ad0 0x7ff69c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.239+0000 7ff6c203f700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff69c053ad0 0x7ff69c055f90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7ff6b800d000 tx=0x7ff6b8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.247+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff6b0043010 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.411+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff6bc063fd0 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6af7fe700 1 -- 172.21.15.61:0/3326140140 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v20) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff6b0046630 con 0x7ff6bc0b23b0 2022-07-20T03:09:28.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff69c053ad0 msgr2=0x7ff69c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff69c053ad0 0x7ff69c055f90 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7ff6b800d000 tx=0x7ff6b8000bc0).stop 2022-07-20T03:09:28.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6bc0b23b0 msgr2=0x7ff6bc140730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6bc0b23b0 0x7ff6bc140730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7ff6b000c000 tx=0x7ff6b0006df0).stop 2022-07-20T03:09:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 shutdown_connections 2022-07-20T03:09:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff69c053ad0 0x7ff69c055f90 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6bc0b23b0 0x7ff6bc140730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6bc1468a0 0x7ff6bc140c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 --2- 172.21.15.61:0/3326140140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6bc0b0580 0x7ff6bc0659c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 >> 172.21.15.61:0/3326140140 conn(0x7ff6bc0632d0 msgr2=0x7ff6bc0602c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 shutdown_connections 2022-07-20T03:09:28.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.507+0000 7ff6c3041700 1 -- 172.21.15.61:0/3326140140 wait complete. 2022-07-20T03:09:28.535 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:28.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 Processor -- start 2022-07-20T03:09:28.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 -- start start 2022-07-20T03:09:28.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6d700b5650 0x7f6d700b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d700b1200 0x7f6d7005c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d700aea30 con 0x7f6d7005bd90 2022-07-20T03:09:28.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d70142970 con 0x7f6d700b1200 2022-07-20T03:09:28.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d774e1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d700aec60 con 0x7f6d700b5650 2022-07-20T03:09:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d75cde700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d700b1200 0x7f6d7005c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d76ce0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6d7005bd90 0x7f6d70142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41046/0 (socket says 172.21.15.61:41046) 2022-07-20T03:09:28.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d76ce0700 1 -- 172.21.15.61:0/1396840047 learned_addr learned my addr 172.21.15.61:0/1396840047 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d764df700 1 --2- 172.21.15.61:0/1396840047 >> v2:172.21.15.117:3300/0 conn(0x7f6d700b5650 0x7f6d700b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d75cde700 1 -- 172.21.15.61:0/1396840047 >> v2:172.21.15.117:3300/0 conn(0x7f6d700b5650 msgr2=0x7f6d700b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:28.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d75cde700 1 --2- 172.21.15.61:0/1396840047 >> v2:172.21.15.117:3300/0 conn(0x7f6d700b5650 0x7f6d700b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d754dd700 1 -- 172.21.15.61:0/1396840047 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 924759841 0 0) 0x7f6d700aea30 con 0x7f6d7005bd90 2022-07-20T03:09:28.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d75cde700 1 -- 172.21.15.61:0/1396840047 >> v1:172.21.15.61:6789/0 conn(0x7f6d7005bd90 legacy=0x7f6d70142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d75cde700 1 -- 172.21.15.61:0/1396840047 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d700af180 con 0x7f6d700b1200 2022-07-20T03:09:28.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d75cde700 1 --2- 172.21.15.61:0/1396840047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d700b1200 0x7f6d7005c4f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f6d64002f70 tx=0x7f6d64006d90).ready entity=mon.0 client_cookie=f835cb6c46a9239a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d754dd700 1 -- 172.21.15.61:0/1396840047 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6400d4a0 con 0x7f6d700b1200 2022-07-20T03:09:28.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d754dd700 1 -- 172.21.15.61:0/1396840047 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d64000e90 con 0x7f6d700b1200 2022-07-20T03:09:28.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.647+0000 7f6d754dd700 1 -- 172.21.15.61:0/1396840047 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6400d7f0 con 0x7f6d700b1200 2022-07-20T03:09:28.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- 172.21.15.61:0/1396840047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d700b1200 msgr2=0x7f6d7005c4f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1396840047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d700b1200 0x7f6d7005c4f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f6d64002f70 tx=0x7f6d64006d90).stop 2022-07-20T03:09:28.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- 172.21.15.61:0/1396840047 shutdown_connections 2022-07-20T03:09:28.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1396840047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d700b1200 0x7f6d7005c4f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1396840047 >> v2:172.21.15.117:3300/0 conn(0x7f6d700b5650 0x7f6d700b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- 172.21.15.61:0/1396840047 >> 172.21.15.61:0/1396840047 conn(0x7f6d700634e0 msgr2=0x7f6d700b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:28.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- 172.21.15.61:0/1396840047 shutdown_connections 2022-07-20T03:09:28.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- 172.21.15.61:0/1396840047 wait complete. 2022-07-20T03:09:28.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 Processor -- start 2022-07-20T03:09:28.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- start start 2022-07-20T03:09:28.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 0x7f6d70155950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d700b1200 0x7f6d70155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d700b5650 0x7f6d70140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d70156500 con 0x7f6d7005bd90 2022-07-20T03:09:28.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d70152b10 con 0x7f6d700b5650 2022-07-20T03:09:28.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d774e1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d7013ced0 con 0x7f6d700b1200 2022-07-20T03:09:28.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d764df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 0x7f6d70155950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d764df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 0x7f6d70155950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44450/0 (socket says 172.21.15.61:44450) 2022-07-20T03:09:28.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d764df700 1 -- 172.21.15.61:0/1048258265 learned_addr learned my addr 172.21.15.61:0/1048258265 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:28.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d75cde700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d700b1200 0x7f6d70155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.651+0000 7f6d76ce0700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d700b5650 0x7f6d70140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d764df700 1 -- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d700b1200 msgr2=0x7f6d70155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d764df700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d700b1200 0x7f6d70155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d764df700 1 -- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d700b5650 msgr2=0x7f6d70140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d764df700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d700b5650 0x7f6d70140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d764df700 1 -- 172.21.15.61:0/1048258265 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6400b120 con 0x7f6d7005bd90 2022-07-20T03:09:28.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d764df700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 0x7f6d70155950 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f6d6c00c000 tx=0x7f6d6c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6c011750 con 0x7f6d7005bd90 2022-07-20T03:09:28.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d6c0118b0 con 0x7f6d7005bd90 2022-07-20T03:09:28.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d7013d170 con 0x7f6d7005bd90 2022-07-20T03:09:28.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d6c011c10 con 0x7f6d7005bd90 2022-07-20T03:09:28.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d7013d770 con 0x7f6d7005bd90 2022-07-20T03:09:28.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d700574e0 con 0x7f6d7005bd90 2022-07-20T03:09:28.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6d6c011d70 con 0x7f6d7005bd90 2022-07-20T03:09:28.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d637fe700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d50053a40 0x7f6d50055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6d6c06c280 con 0x7f6d7005bd90 2022-07-20T03:09:28.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.655+0000 7f6d75cde700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d50053a40 0x7f6d50055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.659+0000 7f6d75cde700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d50053a40 0x7f6d50055f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f6d6401a000 tx=0x7f6d64014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:28.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.663+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d6c043180 con 0x7f6d7005bd90 2022-07-20T03:09:28.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.823+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6d700b5d40 con 0x7f6d7005bd90 2022-07-20T03:09:28.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.823+0000 7f6d637fe700 1 -- 172.21.15.61:0/1048258265 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f6d6c0467a0 con 0x7f6d7005bd90 2022-07-20T03:09:28.831 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:28.832 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:28.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d50053a40 msgr2=0x7f6d50055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d50053a40 0x7f6d50055f00 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f6d6401a000 tx=0x7f6d64014000).stop 2022-07-20T03:09:28.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 msgr2=0x7f6d70155950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:28.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 0x7f6d70155950 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f6d6c00c000 tx=0x7f6d6c006df0).stop 2022-07-20T03:09:28.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 shutdown_connections 2022-07-20T03:09:28.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d50053a40 0x7f6d50055f00 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d700b5650 0x7f6d70140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d700b1200 0x7f6d70155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 --2- 172.21.15.61:0/1048258265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d7005bd90 0x7f6d70155950 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:28.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 >> 172.21.15.61:0/1048258265 conn(0x7f6d700634e0 msgr2=0x7f6d701432a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:28.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 shutdown_connections 2022-07-20T03:09:28.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.827+0000 7f6d774e1700 1 -- 172.21.15.61:0/1048258265 wait complete. 2022-07-20T03:09:28.842 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:28.850 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:09:28.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e40921700 1 Processor -- start 2022-07-20T03:09:28.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e40921700 1 -- start start 2022-07-20T03:09:28.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e40921700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5e3c0adff0 0x7f5e3c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:28.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e40921700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c147010 con 0x7f5e3c0b5650 2022-07-20T03:09:28.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e40921700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e3c1472b0 con 0x7f5e3c0af8e0 2022-07-20T03:09:28.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e40921700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e3c146d40 con 0x7f5e3c0adff0 2022-07-20T03:09:28.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5e3c0b5650 0x7f5e3c147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41056/0 (socket says 172.21.15.61:41056) 2022-07-20T03:09:28.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3b7fe700 1 -- 172.21.15.61:0/3854741958 learned_addr learned my addr 172.21.15.61:0/3854741958 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:28.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 --2- 172.21.15.61:0/3854741958 >> v2:172.21.15.117:3300/0 conn(0x7f5e3c0adff0 0x7f5e3c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:28.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3151438444 0 0) 0x7f5e3c147010 con 0x7f5e3c0b5650 2022-07-20T03:09:28.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e20003260 con 0x7f5e3c0b5650 2022-07-20T03:09:28.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3820178264 0 0) 0x7f5e20003260 con 0x7f5e3c0b5650 2022-07-20T03:09:29.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 >> v2:172.21.15.117:3300/0 conn(0x7f5e3c0adff0 msgr2=0x7f5e3c0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 --2- 172.21.15.61:0/3854741958 >> v2:172.21.15.117:3300/0 conn(0x7f5e3c0adff0 0x7f5e3c0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 >> v1:172.21.15.117:6789/0 conn(0x7f5e3c0af8e0 legacy=0x7f5e3c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c05c4a0 con 0x7f5e3c0b5650 2022-07-20T03:09:29.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f5e24001660 con 0x7f5e3c0b5650 2022-07-20T03:09:29.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 --2- 172.21.15.61:0/3854741958 >> v2:172.21.15.117:3300/0 conn(0x7f5e3c0adff0 0x7f5e3c0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:29.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 >> v1:172.21.15.61:6789/0 conn(0x7f5e3c0b5650 legacy=0x7f5e3c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e20003750 0x7f5e20003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e20004500 0x7f5e20004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004e70 0x7f5e2000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c146d40 con 0x7f5e20004500 2022-07-20T03:09:29.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2000bdd0 con 0x7f5e20003750 2022-07-20T03:09:29.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e2000bf60 con 0x7f5e20004e70 2022-07-20T03:09:29.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e20004500 0x7f5e20004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3b7fe700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e20003750 0x7f5e20003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3bfff700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004e70 0x7f5e2000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 -- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004e70 msgr2=0x7f5e2000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004e70 0x7f5e2000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 -- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e20003750 msgr2=0x7f5e20003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e20003750 0x7f5e20003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 -- 172.21.15.61:0/3854741958 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e20003260 con 0x7f5e20004500 2022-07-20T03:09:29.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3bfff700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004e70 0x7f5e2000b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:29.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3bfff700 1 -- 172.21.15.61:0/3854741958 reap_dead start 2022-07-20T03:09:29.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3affd700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e20004500 0x7f5e20004980 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f5e3400c000 tx=0x7f5e340091a0).ready entity=mon.0 client_cookie=666c5a200d721e8e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e3400e8f0 con 0x7f5e20004500 2022-07-20T03:09:29.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e3400ea50 con 0x7f5e20004500 2022-07-20T03:09:29.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.983+0000 7f5e3a7fc700 1 -- 172.21.15.61:0/3854741958 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e3400edb0 con 0x7f5e20004500 2022-07-20T03:09:29.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- 172.21.15.61:0/3854741958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e20004500 msgr2=0x7f5e20004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e20004500 0x7f5e20004980 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f5e3400c000 tx=0x7f5e340091a0).stop 2022-07-20T03:09:29.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- 172.21.15.61:0/3854741958 shutdown_connections 2022-07-20T03:09:29.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 --2- 172.21.15.61:0/3854741958 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e20004500 0x7f5e20004980 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- 172.21.15.61:0/3854741958 >> 172.21.15.61:0/3854741958 conn(0x7f5e3c0634e0 msgr2=0x7f5e3c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:29.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- 172.21.15.61:0/3854741958 shutdown_connections 2022-07-20T03:09:29.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- 172.21.15.61:0/3854741958 wait complete. 2022-07-20T03:09:29.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 Processor -- start 2022-07-20T03:09:29.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- start start 2022-07-20T03:09:29.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004500 0x7f5e3c1563b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e3c10ffe0 0x7f5e3c149510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 0x7f5e3c067330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c065940 con 0x7f5e3c066eb0 2022-07-20T03:09:29.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c065be0 con 0x7f5e3c10ffe0 2022-07-20T03:09:29.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e40921700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e3c159980 con 0x7f5e20004500 2022-07-20T03:09:29.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 0x7f5e3c067330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e3c10ffe0 0x7f5e3c149510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 0x7f5e3c067330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44474/0 (socket says 172.21.15.61:44474) 2022-07-20T03:09:29.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 -- 172.21.15.61:0/4003051222 learned_addr learned my addr 172.21.15.61:0/4003051222 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:29.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3b7fe700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004500 0x7f5e3c1563b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 -- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004500 msgr2=0x7f5e3c1563b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004500 0x7f5e3c1563b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 -- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e3c10ffe0 msgr2=0x7f5e3c149510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e3c10ffe0 0x7f5e3c149510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.987+0000 7f5e3bfff700 1 -- 172.21.15.61:0/4003051222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e3c067820 con 0x7f5e3c066eb0 2022-07-20T03:09:29.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e3bfff700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 0x7f5e3c067330 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5e2c000e90 tx=0x7f5e2c009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e2c007430 con 0x7f5e3c066eb0 2022-07-20T03:09:29.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e2c00e900 con 0x7f5e3c066eb0 2022-07-20T03:09:29.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e3c14a030 con 0x7f5e3c066eb0 2022-07-20T03:09:29.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e2c00ec60 con 0x7f5e3c066eb0 2022-07-20T03:09:29.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e3c10fd80 con 0x7f5e3c066eb0 2022-07-20T03:09:29.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5e2c01c060 con 0x7f5e3c066eb0 2022-07-20T03:09:29.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e3c159b40 con 0x7f5e3c066eb0 2022-07-20T03:09:29.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e38ff9700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e14053af0 0x7f5e14055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e2c06b500 con 0x7f5e3c066eb0 2022-07-20T03:09:29.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e3b7fe700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e14053af0 0x7f5e14055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.991+0000 7f5e3b7fe700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e14053af0 0x7f5e14055fb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5e24005000 tx=0x7f5e24001e10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:28.999+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e2c042460 con 0x7f5e3c066eb0 2022-07-20T03:09:29.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.123+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5e3c0574e0 con 0x7f5e3c066eb0 2022-07-20T03:09:29.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e38ff9700 1 -- 172.21.15.61:0/4003051222 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f5e2c045a80 con 0x7f5e3c066eb0 2022-07-20T03:09:29.131 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:29.131 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:09:28.502947+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:09:25.577917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T03:09:26.436916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:09:29.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e14053af0 msgr2=0x7f5e14055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e14053af0 0x7f5e14055fb0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f5e24005000 tx=0x7f5e24001e10).stop 2022-07-20T03:09:29.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 msgr2=0x7f5e3c067330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 0x7f5e3c067330 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f5e2c000e90 tx=0x7f5e2c009040).stop 2022-07-20T03:09:29.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 shutdown_connections 2022-07-20T03:09:29.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e14053af0 0x7f5e14055fb0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e3c10ffe0 0x7f5e3c149510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e3c066eb0 0x7f5e3c067330 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 --2- 172.21.15.61:0/4003051222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e20004500 0x7f5e3c1563b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 >> 172.21.15.61:0/4003051222 conn(0x7f5e3c0634e0 msgr2=0x7f5e3c0b05e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:29.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 shutdown_connections 2022-07-20T03:09:29.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.127+0000 7f5e40921700 1 -- 172.21.15.61:0/4003051222 wait complete. 2022-07-20T03:09:29.144 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:09:29.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07b4a41700 1 Processor -- start 2022-07-20T03:09:29.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07b4a41700 1 -- start start 2022-07-20T03:09:29.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07b4a41700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f07b00b34a0 0x7f07b013efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07b4a41700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b005f840 con 0x7f07b013f4a0 2022-07-20T03:09:29.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07b4a41700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07b0062540 con 0x7f07b00ab960 2022-07-20T03:09:29.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07b4a41700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07b005fa70 con 0x7f07b00b34a0 2022-07-20T03:09:29.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07aed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f07b013f4a0 0x7f07b0061e30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41078/0 (socket says 172.21.15.61:41078) 2022-07-20T03:09:29.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07aed9d700 1 -- 172.21.15.61:0/1593925461 learned_addr learned my addr 172.21.15.61:0/1593925461 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:29.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07ae59c700 1 --2- 172.21.15.61:0/1593925461 >> v2:172.21.15.117:3300/0 conn(0x7f07b00b34a0 0x7f07b013efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365721376 0 0) 0x7f07b005f840 con 0x7f07b013f4a0 2022-07-20T03:09:29.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0790003260 con 0x7f07b013f4a0 2022-07-20T03:09:29.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3067255256 0 0) 0x7f07b0062540 con 0x7f07b00ab960 2022-07-20T03:09:29.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f07b005f840 con 0x7f07b00ab960 2022-07-20T03:09:29.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1407125319 0 0) 0x7f0790003260 con 0x7f07b013f4a0 2022-07-20T03:09:29.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 >> v2:172.21.15.117:3300/0 conn(0x7f07b00b34a0 msgr2=0x7f07b013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 --2- 172.21.15.61:0/1593925461 >> v2:172.21.15.117:3300/0 conn(0x7f07b00b34a0 0x7f07b013efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 >> v1:172.21.15.117:6789/0 conn(0x7f07b00ab960 legacy=0x7f07b0061720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b005fd10 con 0x7f07b013f4a0 2022-07-20T03:09:29.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f07a80019e0 con 0x7f07b013f4a0 2022-07-20T03:09:29.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 >> v1:172.21.15.61:6789/0 conn(0x7f07b013f4a0 legacy=0x7f07b0061e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0790003930 0x7f0790003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f0790004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0790004e70 0x7f079000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b005fa70 con 0x7f0790004500 2022-07-20T03:09:29.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f079000bd20 con 0x7f0790003930 2022-07-20T03:09:29.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f079000bee0 con 0x7f0790004e70 2022-07-20T03:09:29.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f0790004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07aed9d700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0790004e70 0x7f079000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07ae59c700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0790003930 0x7f0790003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07a7fff700 1 -- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0790004e70 msgr2=0x7f079000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0790004e70 0x7f079000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07a7fff700 1 -- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0790003930 msgr2=0x7f0790003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0790003930 0x7f0790003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07a7fff700 1 -- 172.21.15.61:0/1593925461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0790003260 con 0x7f0790004500 2022-07-20T03:09:29.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.247+0000 7f07aed9d700 1 -- 172.21.15.61:0/1593925461 reap_dead start 2022-07-20T03:09:29.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f0790004980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f07a0001110 tx=0x7f07a0007a00).ready entity=mon.0 client_cookie=30cd00210253a1e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07a0007d10 con 0x7f0790004500 2022-07-20T03:09:29.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07a0007030 con 0x7f0790004500 2022-07-20T03:09:29.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07add9b700 1 -- 172.21.15.61:0/1593925461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07a0007390 con 0x7f0790004500 2022-07-20T03:09:29.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 -- 172.21.15.61:0/1593925461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 msgr2=0x7f0790004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f0790004980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f07a0001110 tx=0x7f07a0007a00).stop 2022-07-20T03:09:29.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 -- 172.21.15.61:0/1593925461 shutdown_connections 2022-07-20T03:09:29.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1593925461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f0790004980 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 -- 172.21.15.61:0/1593925461 >> 172.21.15.61:0/1593925461 conn(0x7f07b00b6bf0 msgr2=0x7f07b00b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:29.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 -- 172.21.15.61:0/1593925461 shutdown_connections 2022-07-20T03:09:29.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 -- 172.21.15.61:0/1593925461 wait complete. 2022-07-20T03:09:29.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 Processor -- start 2022-07-20T03:09:29.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.251+0000 7f07b4a41700 1 -- start start 2022-07-20T03:09:29.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f07b00b2330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07b01412a0 0x7f07b013c2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07b0142c70 0x7f07b013c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b01418a0 con 0x7f0790004500 2022-07-20T03:09:29.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0141b40 con 0x7f07b01412a0 2022-07-20T03:09:29.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07b0141fe0 con 0x7f07b0142c70 2022-07-20T03:09:29.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f07b00b2330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f07b00b2330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44490/0 (socket says 172.21.15.61:44490) 2022-07-20T03:09:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 -- 172.21.15.61:0/1483702216 learned_addr learned my addr 172.21.15.61:0/1483702216 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07b01412a0 0x7f07b013c2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07aed9d700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07b0142c70 0x7f07b013c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 -- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07b0142c70 msgr2=0x7f07b013c7d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07b0142c70 0x7f07b013c7d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 -- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07b01412a0 msgr2=0x7f07b013c2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07b01412a0 0x7f07b013c2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 -- 172.21.15.61:0/1483702216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07b0140d90 con 0x7f0790004500 2022-07-20T03:09:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07ae59c700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f07b00b2330 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f079800b000 tx=0x7f0798006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0798013680 con 0x7f0790004500 2022-07-20T03:09:29.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798000e90 con 0x7f0790004500 2022-07-20T03:09:29.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07980139d0 con 0x7f0790004500 2022-07-20T03:09:29.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07b013ccc0 con 0x7f0790004500 2022-07-20T03:09:29.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.255+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07b0151510 con 0x7f0790004500 2022-07-20T03:09:29.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.259+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0798013b30 con 0x7f0790004500 2022-07-20T03:09:29.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.259+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07b0141070 con 0x7f0790004500 2022-07-20T03:09:29.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.259+0000 7f07a77fe700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07940539b0 0x7f0794055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.259+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f079806c2f0 con 0x7f0790004500 2022-07-20T03:09:29.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.259+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07940539b0 0x7f0794055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.259+0000 7f07a7fff700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07940539b0 0x7f0794055e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f07a001a000 tx=0x7f07a0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.263+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0798043250 con 0x7f0790004500 2022-07-20T03:09:29.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.431+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f07b0141d00 con 0x7f0790004500 2022-07-20T03:09:29.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.435+0000 7f07a77fe700 1 -- 172.21.15.61:0/1483702216 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v16) v1 ==== 184+0+0 (secure 0 0 0) 0x7f0798018020 con 0x7f0790004500 2022-07-20T03:09:29.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07940539b0 msgr2=0x7f0794055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07940539b0 0x7f0794055e70 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f07a001a000 tx=0x7f07a0015000).stop 2022-07-20T03:09:29.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 msgr2=0x7f07b00b2330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f07b00b2330 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f079800b000 tx=0x7f0798006df0).stop 2022-07-20T03:09:29.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 shutdown_connections 2022-07-20T03:09:29.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07940539b0 0x7f0794055e70 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07b01412a0 0x7f07b013c2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07b0142c70 0x7f07b013c7d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 --2- 172.21.15.61:0/1483702216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0790004500 0x7f07b00b2330 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 >> 172.21.15.61:0/1483702216 conn(0x7f07b00b6bf0 msgr2=0x7f07b013e8a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:29.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 shutdown_connections 2022-07-20T03:09:29.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.439+0000 7f07b4a41700 1 -- 172.21.15.61:0/1483702216 wait complete. 2022-07-20T03:09:29.452 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:09:29.467 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:09:29.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b564d700 1 Processor -- start 2022-07-20T03:09:29.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b564d700 1 -- start start 2022-07-20T03:09:29.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b564d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb3b00acc90 0x7fb3b0143410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b564d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b0097fa0 con 0x7fb3b00ad070 2022-07-20T03:09:29.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b564d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3b00abff0 con 0x7fb3b0062260 2022-07-20T03:09:29.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b564d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3b0098200 con 0x7fb3b00acc90 2022-07-20T03:09:29.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b4e4c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb3b00ad070 0x7fb3b00b0d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41100/0 (socket says 172.21.15.61:41100) 2022-07-20T03:09:29.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3b4e4c700 1 -- 172.21.15.61:0/1600367471 learned_addr learned my addr 172.21.15.61:0/1600367471 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:29.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3affff700 1 --2- 172.21.15.61:0/1600367471 >> v2:172.21.15.117:3300/0 conn(0x7fb3b00acc90 0x7fb3b0143410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 389620529 0 0) 0x7fb3b0097fa0 con 0x7fb3b00ad070 2022-07-20T03:09:29.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb390003260 con 0x7fb3b00ad070 2022-07-20T03:09:29.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3438941834 0 0) 0x7fb3b00abff0 con 0x7fb3b0062260 2022-07-20T03:09:29.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.583+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3b0097fa0 con 0x7fb3b0062260 2022-07-20T03:09:29.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1863947183 0 0) 0x7fb390003260 con 0x7fb3b00ad070 2022-07-20T03:09:29.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 >> v2:172.21.15.117:3300/0 conn(0x7fb3b00acc90 msgr2=0x7fb3b0143410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 --2- 172.21.15.61:0/1600367471 >> v2:172.21.15.117:3300/0 conn(0x7fb3b00acc90 0x7fb3b0143410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 >> v1:172.21.15.117:6789/0 conn(0x7fb3b0062260 legacy=0x7fb3b0143900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b00ac3a0 con 0x7fb3b00ad070 2022-07-20T03:09:29.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fb3a8001a00 con 0x7fb3b00ad070 2022-07-20T03:09:29.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 --2- 172.21.15.61:0/1600367471 >> v2:172.21.15.117:3300/0 conn(0x7fb3b00acc90 0x7fb3b0143410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:29.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 >> v1:172.21.15.61:6789/0 conn(0x7fb3b00ad070 legacy=0x7fb3b00b0d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb390003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb390004500 0x7fb390004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb390004e70 0x7fb39000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0098200 con 0x7fb390004500 2022-07-20T03:09:29.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb39000bdd0 con 0x7fb3900038e0 2022-07-20T03:09:29.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb39000bf40 con 0x7fb390004e70 2022-07-20T03:09:29.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb390003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3a7fff700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb390004500 0x7fb390004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b4e4c700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb390004e70 0x7fb39000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 -- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb390004e70 msgr2=0x7fb39000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb390004e70 0x7fb39000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 -- 172.21.15.61:0/1600367471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb390004500 msgr2=0x7fb390004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb390004500 0x7fb390004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 -- 172.21.15.61:0/1600367471 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb390003260 con 0x7fb3900038e0 2022-07-20T03:09:29.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b4e4c700 1 -- 172.21.15.61:0/1600367471 reap_dead start 2022-07-20T03:09:29.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3a7fff700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb390004500 0x7fb390004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:29.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3affff700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb390003ce0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb3a000c000 tx=0x7fb3a0006df0).ready entity=mon.1 client_cookie=c84ccd8fb36737ed server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3a000e830 con 0x7fb3900038e0 2022-07-20T03:09:29.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3a000e990 con 0x7fb3900038e0 2022-07-20T03:09:29.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3af7fe700 1 -- 172.21.15.61:0/1600367471 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3a000ecf0 con 0x7fb3900038e0 2022-07-20T03:09:29.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 -- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 msgr2=0x7fb390003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb390003ce0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7fb3a000c000 tx=0x7fb3a0006df0).stop 2022-07-20T03:09:29.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 -- 172.21.15.61:0/1600367471 shutdown_connections 2022-07-20T03:09:29.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 --2- 172.21.15.61:0/1600367471 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb390003ce0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 -- 172.21.15.61:0/1600367471 >> 172.21.15.61:0/1600367471 conn(0x7fb3b00b6210 msgr2=0x7fb3b00b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:29.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 -- 172.21.15.61:0/1600367471 shutdown_connections 2022-07-20T03:09:29.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.587+0000 7fb3b564d700 1 -- 172.21.15.61:0/1600367471 wait complete. 2022-07-20T03:09:29.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 Processor -- start 2022-07-20T03:09:29.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 -- start start 2022-07-20T03:09:29.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb3b005c990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 0x7fb3b00674e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3b0066150 0x7fb3b00665d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0064e40 con 0x7fb3b005ffd0 2022-07-20T03:09:29.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b00607e0 con 0x7fb3900038e0 2022-07-20T03:09:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b564d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3b0060a80 con 0x7fb3b0066150 2022-07-20T03:09:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 0x7fb3b00674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 0x7fb3b00674e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44512/0 (socket says 172.21.15.61:44512) 2022-07-20T03:09:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 -- 172.21.15.61:0/523442596 learned_addr learned my addr 172.21.15.61:0/523442596 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3affff700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb3b005c990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3b4e4c700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3b0066150 0x7fb3b00665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 -- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3b0066150 msgr2=0x7fb3b00665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3b0066150 0x7fb3b00665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 -- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 msgr2=0x7fb3b005c990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb3b005c990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.591+0000 7fb3a7fff700 1 -- 172.21.15.61:0/523442596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3b0060d20 con 0x7fb3b005ffd0 2022-07-20T03:09:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3a7fff700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 0x7fb3b00674e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fb39800d000 tx=0x7fb398007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb39800fea0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3980116b0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3b00611c0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb398011a10 con 0x7fb3b005ffd0 2022-07-20T03:09:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3b0066ce0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb39801b030 con 0x7fb3b005ffd0 2022-07-20T03:09:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3b00679d0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3adffb700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb394053a60 0x7fb394055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb39806bb20 con 0x7fb3b005ffd0 2022-07-20T03:09:29.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3affff700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb394053a60 0x7fb394055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.595+0000 7fb3affff700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb394053a60 0x7fb394055f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb3a001d000 tx=0x7fb3a0019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.599+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb398042a80 con 0x7fb3b005ffd0 2022-07-20T03:09:29.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.771+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb3b0060eb0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.775+0000 7fb3adffb700 1 -- 172.21.15.61:0/523442596 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v17) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb3980460a0 con 0x7fb3b005ffd0 2022-07-20T03:09:29.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb394053a60 msgr2=0x7fb394055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb394053a60 0x7fb394055f20 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fb3a001d000 tx=0x7fb3a0019000).stop 2022-07-20T03:09:29.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 msgr2=0x7fb3b00674e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 0x7fb3b00674e0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fb39800d000 tx=0x7fb398007a00).stop 2022-07-20T03:09:29.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 shutdown_connections 2022-07-20T03:09:29.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb394053a60 0x7fb394055f20 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3b0066150 0x7fb3b00665d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3b005ffd0 0x7fb3b00674e0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 --2- 172.21.15.61:0/523442596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3900038e0 0x7fb3b005c990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 >> 172.21.15.61:0/523442596 conn(0x7fb3b00b6210 msgr2=0x7fb3b00b5200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:29.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 shutdown_connections 2022-07-20T03:09:29.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.779+0000 7fb3b564d700 1 -- 172.21.15.61:0/523442596 wait complete. 2022-07-20T03:09:29.794 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:09:29.803 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:09:29.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 Processor -- start 2022-07-20T03:09:29.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 -- start start 2022-07-20T03:09:29.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f81980b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f81980b2290 0x7f81980b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f81980b3e40 con 0x7f81980b9ef0 2022-07-20T03:09:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f81980ba2d0 con 0x7f81980b2290 2022-07-20T03:09:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f81a003a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f81980b4070 con 0x7f81980b3650 2022-07-20T03:09:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819ddd6700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f81980b9ef0 0x7f81980b48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41116/0 (socket says 172.21.15.61:41116) 2022-07-20T03:09:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819ddd6700 1 -- 172.21.15.61:0/4156689356 learned_addr learned my addr 172.21.15.61:0/4156689356 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819e5d7700 1 --2- 172.21.15.61:0/4156689356 >> v2:172.21.15.117:3300/0 conn(0x7f81980b2290 0x7f81980b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819d5d5700 1 --2- 172.21.15.61:0/4156689356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f81980b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819cdd4700 1 -- 172.21.15.61:0/4156689356 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 753769491 0 0) 0x7f81980b3e40 con 0x7f81980b9ef0 2022-07-20T03:09:29.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819cdd4700 1 -- 172.21.15.61:0/4156689356 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f817c003260 con 0x7f81980b9ef0 2022-07-20T03:09:29.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819d5d5700 1 -- 172.21.15.61:0/4156689356 >> v2:172.21.15.117:3300/0 conn(0x7f81980b2290 msgr2=0x7f81980b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:29.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819d5d5700 1 --2- 172.21.15.61:0/4156689356 >> v2:172.21.15.117:3300/0 conn(0x7f81980b2290 0x7f81980b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:29.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819d5d5700 1 -- 172.21.15.61:0/4156689356 >> v1:172.21.15.61:6789/0 conn(0x7f81980b9ef0 legacy=0x7f81980b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:29.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819d5d5700 1 -- 172.21.15.61:0/4156689356 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f81980b1df0 con 0x7f81980b3650 2022-07-20T03:09:29.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819d5d5700 1 --2- 172.21.15.61:0/4156689356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f81980b0750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f818c002f70 tx=0x7f818c006df0).ready entity=mon.2 client_cookie=e88ebddcc46e77eb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:29.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819cdd4700 1 -- 172.21.15.61:0/4156689356 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818c00d4a0 con 0x7f81980b3650 2022-07-20T03:09:29.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819cdd4700 1 -- 172.21.15.61:0/4156689356 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c000e90 con 0x7f81980b3650 2022-07-20T03:09:29.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.927+0000 7f819cdd4700 1 -- 172.21.15.61:0/4156689356 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818c00d7f0 con 0x7f81980b3650 2022-07-20T03:09:30.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- 172.21.15.61:0/4156689356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 msgr2=0x7f81980b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 --2- 172.21.15.61:0/4156689356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f81980b0750 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f818c002f70 tx=0x7f818c006df0).stop 2022-07-20T03:09:30.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- 172.21.15.61:0/4156689356 shutdown_connections 2022-07-20T03:09:30.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 --2- 172.21.15.61:0/4156689356 >> v2:172.21.15.117:3300/0 conn(0x7f81980b2290 0x7f81980b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 --2- 172.21.15.61:0/4156689356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f81980b0750 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- 172.21.15.61:0/4156689356 >> 172.21.15.61:0/4156689356 conn(0x7f8198067980 msgr2=0x7f81980b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:30.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- 172.21.15.61:0/4156689356 shutdown_connections 2022-07-20T03:09:30.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- 172.21.15.61:0/4156689356 wait complete. 2022-07-20T03:09:30.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 Processor -- start 2022-07-20T03:09:30.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- start start 2022-07-20T03:09:30.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 0x7f8198140300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f8198155830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81980b9ef0 0x7f81981449c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819813f2e0 con 0x7f81980b9ef0 2022-07-20T03:09:30.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f819813e250 con 0x7f81980b2290 2022-07-20T03:09:30.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f81a003a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f819813f4d0 con 0x7f81980b3650 2022-07-20T03:09:30.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f819ddd6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 0x7f8198140300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f819ddd6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 0x7f8198140300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54204/0 (socket says 172.21.15.61:54204) 2022-07-20T03:09:30.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f819ddd6700 1 -- 172.21.15.61:0/2324917094 learned_addr learned my addr 172.21.15.61:0/2324917094 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:30.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f819d5d5700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f8198155830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.931+0000 7f819e5d7700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81980b9ef0 0x7f81981449c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819d5d5700 1 -- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 msgr2=0x7f8198140300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819d5d5700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 0x7f8198140300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819d5d5700 1 -- 172.21.15.61:0/2324917094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81980b9ef0 msgr2=0x7f81981449c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819d5d5700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81980b9ef0 0x7f81981449c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819d5d5700 1 -- 172.21.15.61:0/2324917094 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f818c00b180 con 0x7f81980b3650 2022-07-20T03:09:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819ddd6700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 0x7f8198140300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819e5d7700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81980b9ef0 0x7f81981449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:30.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819d5d5700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f8198155830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f819813e590 tx=0x7f818c014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:30.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818c016070 con 0x7f81980b3650 2022-07-20T03:09:30.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f818c0137d0 con 0x7f81980b3650 2022-07-20T03:09:30.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f819813f770 con 0x7f81980b3650 2022-07-20T03:09:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f818c013b30 con 0x7f81980b3650 2022-07-20T03:09:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8198156320 con 0x7f81980b3650 2022-07-20T03:09:30.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f81980574e0 con 0x7f81980b3650 2022-07-20T03:09:30.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f818c025030 con 0x7f81980b3650 2022-07-20T03:09:30.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f818affd700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8180053a40 0x7f8180055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f818c06cd10 con 0x7f81980b3650 2022-07-20T03:09:30.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.935+0000 7f819ddd6700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8180053a40 0x7f8180055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.939+0000 7f819ddd6700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8180053a40 0x7f8180055f00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f819400d000 tx=0x7f8194009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:30.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:29.943+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f818c043c70 con 0x7f81980b3650 2022-07-20T03:09:30.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.115+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f81980b9990 con 0x7f81980b3650 2022-07-20T03:09:30.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.123+0000 7f818affd700 1 -- 172.21.15.61:0/2324917094 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v18) v1 ==== 184+0+0 (secure 0 0 0) 0x7f818c023020 con 0x7f81980b3650 2022-07-20T03:09:30.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8180053a40 msgr2=0x7f8180055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8180053a40 0x7f8180055f00 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7f819400d000 tx=0x7f8194009040).stop 2022-07-20T03:09:30.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 msgr2=0x7f8198155830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f8198155830 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f819813e590 tx=0x7f818c014000).stop 2022-07-20T03:09:30.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 shutdown_connections 2022-07-20T03:09:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8180053a40 0x7f8180055f00 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f81980b9ef0 0x7f81981449c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f81980b3650 0x7f8198155830 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 --2- 172.21.15.61:0/2324917094 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f81980b2290 0x7f8198140300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 >> 172.21.15.61:0/2324917094 conn(0x7f8198067980 msgr2=0x7f81980b5460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:30.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 shutdown_connections 2022-07-20T03:09:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.127+0000 7f81a003a700 1 -- 172.21.15.61:0/2324917094 wait complete. 2022-07-20T03:09:30.175 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:09:30.180 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:30.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.319+0000 7f686b608700 1 Processor -- start 2022-07-20T03:09:30.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.319+0000 7f686b608700 1 -- start start 2022-07-20T03:09:30.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68640b1e00 0x7f686414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68640acef0 0x7f6864146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f686405d480 con 0x7f686405c3f0 2022-07-20T03:09:30.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68640b21e0 con 0x7f68640b1e00 2022-07-20T03:09:30.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f686405cf40 con 0x7f68640acef0 2022-07-20T03:09:30.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68640b1e00 0x7f686414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68640b1e00 0x7f686414a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44536/0 (socket says 172.21.15.61:44536) 2022-07-20T03:09:30.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 -- 172.21.15.61:0/646960737 learned_addr learned my addr 172.21.15.61:0/646960737 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:30.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f6869ba5700 1 --2- 172.21.15.61:0/646960737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68640acef0 0x7f6864146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 -- 172.21.15.61:0/646960737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68640acef0 msgr2=0x7f6864146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:30.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 --2- 172.21.15.61:0/646960737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68640acef0 0x7f6864146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f6863fff700 1 -- 172.21.15.61:0/646960737 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3357536684 0 0) 0x7f686405d480 con 0x7f686405c3f0 2022-07-20T03:09:30.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 -- 172.21.15.61:0/646960737 >> v1:172.21.15.117:6789/0 conn(0x7f686405c3f0 legacy=0x7f68640ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 -- 172.21.15.61:0/646960737 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f686405d1e0 con 0x7f68640b1e00 2022-07-20T03:09:30.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f68693a4700 1 --2- 172.21.15.61:0/646960737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f686414a6d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f6858002f70 tx=0x7f6858006eb0).ready entity=mon.0 client_cookie=b5e95e7480cdb2f6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:30.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f6863fff700 1 -- 172.21.15.61:0/646960737 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f685800d570 con 0x7f68640b1e00 2022-07-20T03:09:30.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f6863fff700 1 -- 172.21.15.61:0/646960737 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6858000e90 con 0x7f68640b1e00 2022-07-20T03:09:30.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f6863fff700 1 -- 172.21.15.61:0/646960737 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f685800d8c0 con 0x7f68640b1e00 2022-07-20T03:09:30.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- 172.21.15.61:0/646960737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 msgr2=0x7f686414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 --2- 172.21.15.61:0/646960737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f686414a6d0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f6858002f70 tx=0x7f6858006eb0).stop 2022-07-20T03:09:30.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- 172.21.15.61:0/646960737 shutdown_connections 2022-07-20T03:09:30.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 --2- 172.21.15.61:0/646960737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68640acef0 0x7f6864146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 --2- 172.21.15.61:0/646960737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f686414a6d0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- 172.21.15.61:0/646960737 >> 172.21.15.61:0/646960737 conn(0x7f68640611c0 msgr2=0x7f6864062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:30.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- 172.21.15.61:0/646960737 shutdown_connections 2022-07-20T03:09:30.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- 172.21.15.61:0/646960737 wait complete. 2022-07-20T03:09:30.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 Processor -- start 2022-07-20T03:09:30.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.323+0000 7f686b608700 1 -- start start 2022-07-20T03:09:30.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68640acef0 0x7f68640ad900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f6864063e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6864064320 0x7f6864159cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864143820 con 0x7f68640b1e00 2022-07-20T03:09:30.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864065e10 con 0x7f68640acef0 2022-07-20T03:09:30.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6864149840 con 0x7f6864064320 2022-07-20T03:09:30.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f6864063e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f6864063e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44540/0 (socket says 172.21.15.61:44540) 2022-07-20T03:09:30.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 -- 172.21.15.61:0/752466555 learned_addr learned my addr 172.21.15.61:0/752466555 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:30.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f68693a4700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68640acef0 0x7f68640ad900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6869ba5700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6864064320 0x7f6864159cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 -- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6864064320 msgr2=0x7f6864159cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6864064320 0x7f6864159cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 -- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68640acef0 msgr2=0x7f68640ad900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68640acef0 0x7f68640ad900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 -- 172.21.15.61:0/752466555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f685800b1e0 con 0x7f68640b1e00 2022-07-20T03:09:30.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f6868ba3700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f6864063e30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f686414a360 tx=0x7f685c007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:30.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f685c002490 con 0x7f68640b1e00 2022-07-20T03:09:30.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f685c0025f0 con 0x7f68640b1e00 2022-07-20T03:09:30.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864149a70 con 0x7f68640b1e00 2022-07-20T03:09:30.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f685c002950 con 0x7f68640b1e00 2022-07-20T03:09:30.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.327+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6864149480 con 0x7f68640b1e00 2022-07-20T03:09:30.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.331+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68641491c0 con 0x7f68640b1e00 2022-07-20T03:09:30.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.331+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f685c010030 con 0x7f68640b1e00 2022-07-20T03:09:30.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.331+0000 7f68627fc700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6844053ad0 0x7f6844055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:30.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.331+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f685c06c0f0 con 0x7f68640b1e00 2022-07-20T03:09:30.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.331+0000 7f68693a4700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6844053ad0 0x7f6844055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:30.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.331+0000 7f68693a4700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6844053ad0 0x7f6844055f90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f685801c000 tx=0x7f6858015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:30.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.335+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68641491c0 con 0x7f68640b1e00 2022-07-20T03:09:30.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.499+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f686414a150 con 0x7f68640b1e00 2022-07-20T03:09:30.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.499+0000 7f68627fc700 1 -- 172.21.15.61:0/752466555 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f685c042f40 con 0x7f68640b1e00 2022-07-20T03:09:30.507 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:30.508 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:30.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6844053ad0 msgr2=0x7f6844055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6844053ad0 0x7f6844055f90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f685801c000 tx=0x7f6858015000).stop 2022-07-20T03:09:30.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 msgr2=0x7f6864063e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:30.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f6864063e30 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f686414a360 tx=0x7f685c007f70).stop 2022-07-20T03:09:30.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 shutdown_connections 2022-07-20T03:09:30.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6844053ad0 0x7f6844055f90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6864064320 0x7f6864159cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68640b1e00 0x7f6864063e30 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 --2- 172.21.15.61:0/752466555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68640acef0 0x7f68640ad900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:30.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 >> 172.21.15.61:0/752466555 conn(0x7f686405c3f0 msgr2=0x7f6864144fb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:30.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 shutdown_connections 2022-07-20T03:09:30.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:30.503+0000 7f686b608700 1 -- 172.21.15.61:0/752466555 wait complete. 2022-07-20T03:09:30.520 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:30.741 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:09:27.499383+0000', 'modified': '2022-07-20T03:09:28.502169+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5268': {'gid': 5268, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/844334511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 844334511}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 844334511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:09:30.742 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:09:30.743 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:09:30.875 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5268,"want_state":"up:active","state":"up:active","rank_uptime":3.3563735609999998,"mdsmap_epoch":20,"osdmap_epoch":42,"osdmap_epoch_barrier":41,"uptime":8.7964898839999996} 2022-07-20T03:09:30.885 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5268, "mdsmap_epoch": 20, "osdmap_epoch": 42, "osdmap_epoch_barrier": 41, "rank_uptime": 3.356373561, "state": "up:active", "uptime": 8.796489884, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:09:30.887 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:09:31.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.003+0000 7f9cbdf79700 1 Processor -- start 2022-07-20T03:09:31.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- start start 2022-07-20T03:09:31.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cb80ab390 0x7f9cb80ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb8061130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cb80accd0 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cb8060530 con 0x7f9cb80ab390 2022-07-20T03:09:31.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb80b22d0 con 0x7f9cb81427a0 2022-07-20T03:09:31.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cb80ab390 0x7f9cb80ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cb80ab390 0x7f9cb80ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44546/0 (socket says 172.21.15.61:44546) 2022-07-20T03:09:31.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 -- 172.21.15.61:0/4101925221 learned_addr learned my addr 172.21.15.61:0/4101925221 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:31.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbd778700 1 --2- 172.21.15.61:0/4101925221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb8061130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 -- 172.21.15.61:0/4101925221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 msgr2=0x7f9cb8061130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:31.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/4101925221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb8061130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 -- 172.21.15.61:0/4101925221 >> v1:172.21.15.117:6789/0 conn(0x7f9cb80b1ef0 legacy=0x7f9cb8142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 -- 172.21.15.61:0/4101925221 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb805f8e0 con 0x7f9cb80ab390 2022-07-20T03:09:31.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/4101925221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80ab390 0x7f9cb80ac7e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f9ca80032f0 tx=0x7f9ca8009290).ready entity=mon.0 client_cookie=c5f7abadf67505da server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb77fe700 1 -- 172.21.15.61:0/4101925221 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca800b8a0 con 0x7f9cb80ab390 2022-07-20T03:09:31.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb77fe700 1 -- 172.21.15.61:0/4101925221 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca800ba00 con 0x7f9cb80ab390 2022-07-20T03:09:31.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cb77fe700 1 -- 172.21.15.61:0/4101925221 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca800bb70 con 0x7f9cb80ab390 2022-07-20T03:09:31.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- 172.21.15.61:0/4101925221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80ab390 msgr2=0x7f9cb80ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/4101925221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80ab390 0x7f9cb80ac7e0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f9ca80032f0 tx=0x7f9ca8009290).stop 2022-07-20T03:09:31.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- 172.21.15.61:0/4101925221 shutdown_connections 2022-07-20T03:09:31.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/4101925221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb8061130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/4101925221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80ab390 0x7f9cb80ac7e0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- 172.21.15.61:0/4101925221 >> 172.21.15.61:0/4101925221 conn(0x7f9cb80b2ca0 msgr2=0x7f9cb80b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:31.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- 172.21.15.61:0/4101925221 shutdown_connections 2022-07-20T03:09:31.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.007+0000 7f9cbdf79700 1 -- 172.21.15.61:0/4101925221 wait complete. 2022-07-20T03:09:31.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 Processor -- start 2022-07-20T03:09:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- start start 2022-07-20T03:09:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 0x7f9cb809a390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 0x7f9cb8145760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb80992a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb809bd00 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb809bfa0 con 0x7f9cb80ab390 2022-07-20T03:09:31.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cb8145f40 con 0x7f9cb81427a0 2022-07-20T03:09:31.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 0x7f9cb8145760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbcf77700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 0x7f9cb809a390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 0x7f9cb8145760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44554/0 (socket says 172.21.15.61:44554) 2022-07-20T03:09:31.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbcf77700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 0x7f9cb809a390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54232/0 (socket says 172.21.15.61:54232) 2022-07-20T03:09:31.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2601067816 learned_addr learned my addr 172.21.15.61:0/2601067816 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:31.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbd778700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb80992a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 msgr2=0x7f9cb80992a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb80992a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 msgr2=0x7f9cb809a390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 0x7f9cb809a390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 -- 172.21.15.61:0/2601067816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca8009530 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbcf77700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 0x7f9cb809a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:31.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb7fff700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 0x7f9cb8145760 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f9ca80032f0 tx=0x7f9ca8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca800b8a0 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca8021480 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cb8099790 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca80217e0 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cb80ad7e0 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.011+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cb805fe20 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.015+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9ca800f070 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.015+0000 7f9cb5ffb700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c980539d0 0x7f9c98055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.015+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ca806d2b0 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.015+0000 7f9cbcf77700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c980539d0 0x7f9c98055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.015+0000 7f9cbcf77700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c980539d0 0x7f9c98055e90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9cac00d000 tx=0x7f9cac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.019+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ca8044140 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.147+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9cb80574e0 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cb5ffb700 1 -- 172.21.15.61:0/2601067816 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16086 (secure 0 0 0) 0x7f9ca8015070 con 0x7f9cb80b1ef0 2022-07-20T03:09:31.235 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:31.236 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:09:28.502947+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:09:25.577917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T03:09:26.436916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:09:31.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c980539d0 msgr2=0x7f9c98055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c980539d0 0x7f9c98055e90 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f9cac00d000 tx=0x7f9cac009040).stop 2022-07-20T03:09:31.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 msgr2=0x7f9cb8145760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 0x7f9cb8145760 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f9ca80032f0 tx=0x7f9ca8016000).stop 2022-07-20T03:09:31.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 shutdown_connections 2022-07-20T03:09:31.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c980539d0 0x7f9c98055e90 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cb81427a0 0x7f9cb80992a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cb80b1ef0 0x7f9cb8145760 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 --2- 172.21.15.61:0/2601067816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cb80ab390 0x7f9cb809a390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 >> 172.21.15.61:0/2601067816 conn(0x7f9cb80b2ca0 msgr2=0x7f9cb8060ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:31.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 shutdown_connections 2022-07-20T03:09:31.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.151+0000 7f9cbdf79700 1 -- 172.21.15.61:0/2601067816 wait complete. 2022-07-20T03:09:31.254 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:09:25.577917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:09:26.436916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:09:31.255 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:09:31.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:31.348 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:31.358 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:31.389 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:31.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.371+0000 7f12f6b94700 1 Processor -- start 2022-07-20T03:09:31.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.371+0000 7f12f6b94700 1 -- start start 2022-07-20T03:09:31.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f00b1590 0x7f12f0146b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f12f0063820 0x7f12f00600b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12f005bd90 0x7f12f00605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f0061800 con 0x7f12f005bd90 2022-07-20T03:09:31.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12f01461b0 con 0x7f12f0063820 2022-07-20T03:09:31.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00b1970 con 0x7f12f00b1590 2022-07-20T03:09:31.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f5131700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12f005bd90 0x7f12f00605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f00b1590 0x7f12f0146b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f5131700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12f005bd90 0x7f12f00605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44564/0 (socket says 172.21.15.61:44564) 2022-07-20T03:09:31.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f5131700 1 -- 172.21.15.61:0/3658201303 learned_addr learned my addr 172.21.15.61:0/3658201303 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:31.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12effff700 1 --2- 172.21.15.61:0/3658201303 >> v2:172.21.15.117:3300/0 conn(0x7f12f0063820 0x7f12f00600b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 -- 172.21.15.61:0/3658201303 >> v2:172.21.15.117:3300/0 conn(0x7f12f0063820 msgr2=0x7f12f00600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:31.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 --2- 172.21.15.61:0/3658201303 >> v2:172.21.15.117:3300/0 conn(0x7f12f0063820 0x7f12f00600b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 -- 172.21.15.61:0/3658201303 >> v2:172.21.15.61:3300/0 conn(0x7f12f005bd90 msgr2=0x7f12f00605a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:31.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 --2- 172.21.15.61:0/3658201303 >> v2:172.21.15.61:3300/0 conn(0x7f12f005bd90 0x7f12f00605a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 -- 172.21.15.61:0/3658201303 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f01463e0 con 0x7f12f00b1590 2022-07-20T03:09:31.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f4930700 1 --2- 172.21.15.61:0/3658201303 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f00b1590 0x7f12f0146b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f12f0063cf0 tx=0x7f12e0006df0).ready entity=mon.2 client_cookie=57576becfb80d9b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12ef7fe700 1 -- 172.21.15.61:0/3658201303 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e000d4a0 con 0x7f12f00b1590 2022-07-20T03:09:31.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12ef7fe700 1 -- 172.21.15.61:0/3658201303 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0000e90 con 0x7f12f00b1590 2022-07-20T03:09:31.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12ef7fe700 1 -- 172.21.15.61:0/3658201303 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e000d7f0 con 0x7f12f00b1590 2022-07-20T03:09:31.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- 172.21.15.61:0/3658201303 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f00b1590 msgr2=0x7f12f0146b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- 172.21.15.61:0/3658201303 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f00b1590 0x7f12f0146b20 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f12f0063cf0 tx=0x7f12e0006df0).stop 2022-07-20T03:09:31.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- 172.21.15.61:0/3658201303 shutdown_connections 2022-07-20T03:09:31.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- 172.21.15.61:0/3658201303 >> v2:172.21.15.61:3300/0 conn(0x7f12f005bd90 0x7f12f00605a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- 172.21.15.61:0/3658201303 >> v2:172.21.15.117:3300/0 conn(0x7f12f0063820 0x7f12f00600b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 --2- 172.21.15.61:0/3658201303 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f00b1590 0x7f12f0146b20 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- 172.21.15.61:0/3658201303 >> 172.21.15.61:0/3658201303 conn(0x7f12f00b2340 msgr2=0x7f12f00b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:31.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- 172.21.15.61:0/3658201303 shutdown_connections 2022-07-20T03:09:31.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.375+0000 7f12f6b94700 1 -- 172.21.15.61:0/3658201303 wait complete. 2022-07-20T03:09:31.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 Processor -- start 2022-07-20T03:09:31.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 -- start start 2022-07-20T03:09:31.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 0x7f12f01421c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f0063820 0x7f12f0155d50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f00b1590 0x7f12f0066480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0090000 con 0x7f12f005bd90 2022-07-20T03:09:31.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12f00652b0 con 0x7f12f00b1590 2022-07-20T03:09:31.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12f0065550 con 0x7f12f0063820 2022-07-20T03:09:31.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 0x7f12f01421c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 0x7f12f01421c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44566/0 (socket says 172.21.15.61:44566) 2022-07-20T03:09:31.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 -- 172.21.15.61:0/4080872567 learned_addr learned my addr 172.21.15.61:0/4080872567 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:31.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f5131700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f00b1590 0x7f12f0066480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 -- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f0063820 msgr2=0x7f12f0155d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12effff700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f0063820 0x7f12f0155d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f0063820 0x7f12f0155d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 -- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f00b1590 msgr2=0x7f12f0066480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f00b1590 0x7f12f0066480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 -- 172.21.15.61:0/4080872567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12e000b180 con 0x7f12f005bd90 2022-07-20T03:09:31.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f5131700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f00b1590 0x7f12f0066480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:31.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12effff700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f0063820 0x7f12f0155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:09:31.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f4930700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 0x7f12f01421c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f12f0063cf0 tx=0x7f12e0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e00139c0 con 0x7f12f005bd90 2022-07-20T03:09:31.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12e0013b20 con 0x7f12f005bd90 2022-07-20T03:09:31.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12e0013e80 con 0x7f12f005bd90 2022-07-20T03:09:31.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12f0066020 con 0x7f12f005bd90 2022-07-20T03:09:31.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.379+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12f0064810 con 0x7f12f005bd90 2022-07-20T03:09:31.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.383+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12f0146680 con 0x7f12f005bd90 2022-07-20T03:09:31.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.383+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f12e001c070 con 0x7f12f005bd90 2022-07-20T03:09:31.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.383+0000 7f12edffb700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12d0053940 0x7f12d0055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.383+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f12e006c8f0 con 0x7f12f005bd90 2022-07-20T03:09:31.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.383+0000 7f12effff700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12d0053940 0x7f12d0055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.383+0000 7f12effff700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12d0053940 0x7f12d0055e00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f12e400b000 tx=0x7f12e4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.391+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12e00436c0 con 0x7f12f005bd90 2022-07-20T03:09:31.567 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:09:31.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.515+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f12f00574e0 con 0x7f12f005bd90 2022-07-20T03:09:31.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12edffb700 1 -- 172.21.15.61:0/4080872567 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f12e0023070 con 0x7f12f005bd90 2022-07-20T03:09:31.568 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:09:31.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12d0053940 msgr2=0x7f12d0055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12d0053940 0x7f12d0055e00 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f12e400b000 tx=0x7f12e4008040).stop 2022-07-20T03:09:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 msgr2=0x7f12f01421c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 0x7f12f01421c0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f12f0063cf0 tx=0x7f12e0014000).stop 2022-07-20T03:09:31.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 shutdown_connections 2022-07-20T03:09:31.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12d0053940 0x7f12d0055e00 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12f00b1590 0x7f12f0066480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12f0063820 0x7f12f0155d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 --2- 172.21.15.61:0/4080872567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12f005bd90 0x7f12f01421c0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 >> 172.21.15.61:0/4080872567 conn(0x7f12f00b2340 msgr2=0x7f12f00aede0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:31.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 shutdown_connections 2022-07-20T03:09:31.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.519+0000 7f12f6b94700 1 -- 172.21.15.61:0/4080872567 wait complete. 2022-07-20T03:09:31.584 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:09:31.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.731+0000 7f3b89f66700 1 Processor -- start 2022-07-20T03:09:31.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.731+0000 7f3b89f66700 1 -- start start 2022-07-20T03:09:31.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3b840b58f0 0x7f3b840aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3b840b4b90 0x7f3b84143f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b01f0 0x7f3b840b0660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b840b4fd0 con 0x7f3b840b4b90 2022-07-20T03:09:31.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3b840b0b50 con 0x7f3b840b58f0 2022-07-20T03:09:31.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b840b4150 con 0x7f3b840b01f0 2022-07-20T03:09:31.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3b840b4b90 0x7f3b84143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3b840b4b90 0x7f3b84143f60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44576/0 (socket says 172.21.15.61:44576) 2022-07-20T03:09:31.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 -- 172.21.15.61:0/219248305 learned_addr learned my addr 172.21.15.61:0/219248305 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:31.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b88f64700 1 --2- 172.21.15.61:0/219248305 >> v2:172.21.15.117:3300/0 conn(0x7f3b840b58f0 0x7f3b840aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89765700 1 --2- 172.21.15.61:0/219248305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b01f0 0x7f3b840b0660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 -- 172.21.15.61:0/219248305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b01f0 msgr2=0x7f3b840b0660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:31.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/219248305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b01f0 0x7f3b840b0660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 -- 172.21.15.61:0/219248305 >> v2:172.21.15.117:3300/0 conn(0x7f3b840b58f0 msgr2=0x7f3b840aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:31.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/219248305 >> v2:172.21.15.117:3300/0 conn(0x7f3b840b58f0 0x7f3b840aa9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 -- 172.21.15.61:0/219248305 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b840b43f0 con 0x7f3b840b4b90 2022-07-20T03:09:31.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/219248305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b4b90 0x7f3b84143f60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f3b74002f70 tx=0x7f3b74006eb0).ready entity=mon.0 client_cookie=6f782f9039162a35 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7b7fe700 1 -- 172.21.15.61:0/219248305 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b7400d570 con 0x7f3b840b4b90 2022-07-20T03:09:31.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7b7fe700 1 -- 172.21.15.61:0/219248305 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b74000e90 con 0x7f3b840b4b90 2022-07-20T03:09:31.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b7b7fe700 1 -- 172.21.15.61:0/219248305 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b7400d900 con 0x7f3b840b4b90 2022-07-20T03:09:31.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 -- 172.21.15.61:0/219248305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b4b90 msgr2=0x7f3b84143f60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- 172.21.15.61:0/219248305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b4b90 0x7f3b84143f60 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f3b74002f70 tx=0x7f3b74006eb0).stop 2022-07-20T03:09:31.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 -- 172.21.15.61:0/219248305 shutdown_connections 2022-07-20T03:09:31.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- 172.21.15.61:0/219248305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b01f0 0x7f3b840b0660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- 172.21.15.61:0/219248305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b4b90 0x7f3b84143f60 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 --2- 172.21.15.61:0/219248305 >> v2:172.21.15.117:3300/0 conn(0x7f3b840b58f0 0x7f3b840aa9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.735+0000 7f3b89f66700 1 -- 172.21.15.61:0/219248305 >> 172.21.15.61:0/219248305 conn(0x7f3b840632b0 msgr2=0x7f3b84063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:31.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.739+0000 7f3b89f66700 1 -- 172.21.15.61:0/219248305 shutdown_connections 2022-07-20T03:09:31.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.739+0000 7f3b89f66700 1 -- 172.21.15.61:0/219248305 wait complete. 2022-07-20T03:09:31.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.739+0000 7f3b89f66700 1 Processor -- start 2022-07-20T03:09:31.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.739+0000 7f3b89f66700 1 -- start start 2022-07-20T03:09:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 0x7f3b8414fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b840b4b90 0x7f3b841455c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b58f0 0x7f3b841505c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8413fd60 con 0x7f3b840b01f0 2022-07-20T03:09:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3b841462e0 con 0x7f3b840b4b90 2022-07-20T03:09:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3b8414ffd0 con 0x7f3b840b58f0 2022-07-20T03:09:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b88f64700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 0x7f3b8414fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b88f64700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 0x7f3b8414fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44580/0 (socket says 172.21.15.61:44580) 2022-07-20T03:09:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b88f64700 1 -- 172.21.15.61:0/2925330804 learned_addr learned my addr 172.21.15.61:0/2925330804 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b840b4b90 0x7f3b841455c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89765700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b58f0 0x7f3b841505c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 -- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b58f0 msgr2=0x7f3b841505c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b58f0 0x7f3b841505c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 -- 172.21.15.61:0/2925330804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 msgr2=0x7f3b8414fae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 0x7f3b8414fae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 -- 172.21.15.61:0/2925330804 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3b7400b1e0 con 0x7f3b840b4b90 2022-07-20T03:09:31.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b88f64700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 0x7f3b8414fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89765700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b58f0 0x7f3b841505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b7bfff700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b840b4b90 0x7f3b841455c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f3b74002f70 tx=0x7f3b74015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b74011760 con 0x7f3b840b4b90 2022-07-20T03:09:31.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3b740118c0 con 0x7f3b840b4b90 2022-07-20T03:09:31.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3b840906b0 con 0x7f3b840b4b90 2022-07-20T03:09:31.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3b74011c20 con 0x7f3b840b4b90 2022-07-20T03:09:31.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.743+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3b84146730 con 0x7f3b840b4b90 2022-07-20T03:09:31.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.747+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3b8409b590 con 0x7f3b840b4b90 2022-07-20T03:09:31.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.747+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3b7401a040 con 0x7f3b840b4b90 2022-07-20T03:09:31.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.747+0000 7f3b79ffb700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b64053a40 0x7f3b64055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:31.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.747+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3b7406cad0 con 0x7f3b840b4b90 2022-07-20T03:09:31.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.747+0000 7f3b88f64700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b64053a40 0x7f3b64055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:31.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.747+0000 7f3b88f64700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b64053a40 0x7f3b64055f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3b8000b000 tx=0x7f3b80006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:31.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.759+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3b74043a30 con 0x7f3b840b4b90 2022-07-20T03:09:31.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.887+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3b84146420 con 0x7f3b840b4b90 2022-07-20T03:09:31.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.887+0000 7f3b79ffb700 1 -- 172.21.15.61:0/2925330804 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7f3b74047050 con 0x7f3b840b4b90 2022-07-20T03:09:31.903 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:31.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b64053a40 msgr2=0x7f3b64055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b64053a40 0x7f3b64055f00 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f3b8000b000 tx=0x7f3b80006df0).stop 2022-07-20T03:09:31.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b840b4b90 msgr2=0x7f3b841455c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:31.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b840b4b90 0x7f3b841455c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f3b74002f70 tx=0x7f3b74015000).stop 2022-07-20T03:09:31.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 shutdown_connections 2022-07-20T03:09:31.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3b64053a40 0x7f3b64055f00 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3b840b4b90 0x7f3b841455c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3b840b58f0 0x7f3b841505c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 --2- 172.21.15.61:0/2925330804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3b840b01f0 0x7f3b8414fae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:31.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 >> 172.21.15.61:0/2925330804 conn(0x7f3b840632b0 msgr2=0x7f3b840b1500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:31.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 shutdown_connections 2022-07-20T03:09:31.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:31.891+0000 7f3b89f66700 1 -- 172.21.15.61:0/2925330804 wait complete. 2022-07-20T03:09:31.923 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:09:32.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.043+0000 7f5f4020b700 1 Processor -- start 2022-07-20T03:09:32.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.043+0000 7f5f4020b700 1 -- start start 2022-07-20T03:09:32.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f380ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5f38099300 0x7f5f38099770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f380b3a50 con 0x7f5f381437e0 2022-07-20T03:09:32.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f3813d3b0 con 0x7f5f38099300 2022-07-20T03:09:32.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380ac670 con 0x7f5f380b3670 2022-07-20T03:09:32.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3dfa7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f380ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3e7a8700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5f38099300 0x7f5f38099770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3d7a6700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f5f381437e0 0x7f5f3813f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:49692/0 (socket says 172.21.15.61:49692) 2022-07-20T03:09:32.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3d7a6700 1 -- 172.21.15.61:0/1962472019 learned_addr learned my addr 172.21.15.61:0/1962472019 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:32.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3cfa5700 1 -- 172.21.15.61:0/1962472019 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1055924360 0 0) 0x7f5f380b3a50 con 0x7f5f381437e0 2022-07-20T03:09:32.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3cfa5700 1 -- 172.21.15.61:0/1962472019 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f20003260 con 0x7f5f381437e0 2022-07-20T03:09:32.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/1962472019 >> v2:172.21.15.117:3300/0 conn(0x7f5f38099300 msgr2=0x7f5f38099770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:32.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3dfa7700 1 --2- 172.21.15.61:0/1962472019 >> v2:172.21.15.117:3300/0 conn(0x7f5f38099300 0x7f5f38099770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/1962472019 >> v1:172.21.15.117:6789/0 conn(0x7f5f381437e0 legacy=0x7f5f3813f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/1962472019 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f381430a0 con 0x7f5f380b3670 2022-07-20T03:09:32.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3dfa7700 1 --2- 172.21.15.61:0/1962472019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f380ac180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5f28002f70 tx=0x7f5f28006df0).ready entity=mon.2 client_cookie=43b09f37f288fa38 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3cfa5700 1 -- 172.21.15.61:0/1962472019 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2800d4a0 con 0x7f5f380b3670 2022-07-20T03:09:32.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3cfa5700 1 -- 172.21.15.61:0/1962472019 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f28000e90 con 0x7f5f380b3670 2022-07-20T03:09:32.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f3cfa5700 1 -- 172.21.15.61:0/1962472019 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f2800d7f0 con 0x7f5f380b3670 2022-07-20T03:09:32.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- 172.21.15.61:0/1962472019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 msgr2=0x7f5f380ac180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 --2- 172.21.15.61:0/1962472019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f380ac180 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f5f28002f70 tx=0x7f5f28006df0).stop 2022-07-20T03:09:32.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- 172.21.15.61:0/1962472019 shutdown_connections 2022-07-20T03:09:32.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 --2- 172.21.15.61:0/1962472019 >> v2:172.21.15.117:3300/0 conn(0x7f5f38099300 0x7f5f38099770 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 --2- 172.21.15.61:0/1962472019 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f380ac180 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- 172.21.15.61:0/1962472019 >> 172.21.15.61:0/1962472019 conn(0x7f5f380b6c50 msgr2=0x7f5f380b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:32.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- 172.21.15.61:0/1962472019 shutdown_connections 2022-07-20T03:09:32.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.047+0000 7f5f4020b700 1 -- 172.21.15.61:0/1962472019 wait complete. 2022-07-20T03:09:32.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 Processor -- start 2022-07-20T03:09:32.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 -- start start 2022-07-20T03:09:32.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 0x7f5f380b4dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f3813fea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f381437e0 0x7f5f380ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3813d750 con 0x7f5f38099300 2022-07-20T03:09:32.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f3813fa50 con 0x7f5f381437e0 2022-07-20T03:09:32.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f380af4d0 con 0x7f5f380b3670 2022-07-20T03:09:32.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 0x7f5f380b4dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 0x7f5f380b4dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44594/0 (socket says 172.21.15.61:44594) 2022-07-20T03:09:32.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/3659358804 learned_addr learned my addr 172.21.15.61:0/3659358804 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:32.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3d7a6700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f3813fea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3e7a8700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f381437e0 0x7f5f380ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 msgr2=0x7f5f3813fea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f3813fea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f381437e0 msgr2=0x7f5f380ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f381437e0 0x7f5f380ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 -- 172.21.15.61:0/3659358804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f2800b180 con 0x7f5f38099300 2022-07-20T03:09:32.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f3dfa7700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 0x7f5f380b4dd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f5f28002f70 tx=0x7f5f28014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f28016070 con 0x7f5f38099300 2022-07-20T03:09:32.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f280137d0 con 0x7f5f38099300 2022-07-20T03:09:32.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f380af770 con 0x7f5f38099300 2022-07-20T03:09:32.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f28013b30 con 0x7f5f38099300 2022-07-20T03:09:32.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.051+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f380af1c0 con 0x7f5f38099300 2022-07-20T03:09:32.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.055+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f380574e0 con 0x7f5f38099300 2022-07-20T03:09:32.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.055+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5f2800f030 con 0x7f5f38099300 2022-07-20T03:09:32.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.055+0000 7f5f26ffd700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f18053a40 0x7f5f18055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.055+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f2806cc20 con 0x7f5f38099300 2022-07-20T03:09:32.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.055+0000 7f5f3d7a6700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f18053a40 0x7f5f18055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.059+0000 7f5f3d7a6700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f18053a40 0x7f5f18055f00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5f3400d000 tx=0x7f5f34009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.063+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f280439f0 con 0x7f5f38099300 2022-07-20T03:09:32.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.187+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5f3805db10 con 0x7f5f38099300 2022-07-20T03:09:32.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.187+0000 7f5f26ffd700 1 -- 172.21.15.61:0/3659358804 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5f28023020 con 0x7f5f38099300 2022-07-20T03:09:32.207 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:32.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f18053a40 msgr2=0x7f5f18055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f18053a40 0x7f5f18055f00 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f5f3400d000 tx=0x7f5f34009040).stop 2022-07-20T03:09:32.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 msgr2=0x7f5f380b4dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 0x7f5f380b4dd0 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f5f28002f70 tx=0x7f5f28014000).stop 2022-07-20T03:09:32.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 shutdown_connections 2022-07-20T03:09:32.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f18053a40 0x7f5f18055f00 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f381437e0 0x7f5f380ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f380b3670 0x7f5f3813fea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 --2- 172.21.15.61:0/3659358804 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f38099300 0x7f5f380b4dd0 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 >> 172.21.15.61:0/3659358804 conn(0x7f5f380b6c50 msgr2=0x7f5f38144850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:32.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 shutdown_connections 2022-07-20T03:09:32.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.191+0000 7f5f4020b700 1 -- 172.21.15.61:0/3659358804 wait complete. 2022-07-20T03:09:32.216 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:09:32.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.319+0000 7f003a7f4700 1 Processor -- start 2022-07-20T03:09:32.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.319+0000 7f003a7f4700 1 -- start start 2022-07-20T03:09:32.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f00340b32a0 0x7f00341468d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340b3f10 con 0x7f00340b2ec0 2022-07-20T03:09:32.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f00340b2620 con 0x7f003405d160 2022-07-20T03:09:32.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f00340b4ea0 con 0x7f00340b32a0 2022-07-20T03:09:32.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f00397f2700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f00340b2ec0 0x7f0034142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41200/0 (socket says 172.21.15.61:41200) 2022-07-20T03:09:32.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f00397f2700 1 -- 172.21.15.61:0/2895297615 learned_addr learned my addr 172.21.15.61:0/2895297615 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:32.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0039ff3700 1 --2- 172.21.15.61:0/2895297615 >> v2:172.21.15.117:3300/0 conn(0x7f00340b32a0 0x7f00341468d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0023fff700 1 -- 172.21.15.61:0/2895297615 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 556481080 0 0) 0x7f00340b3f10 con 0x7f00340b2ec0 2022-07-20T03:09:32.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0023fff700 1 -- 172.21.15.61:0/2895297615 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f001c003260 con 0x7f00340b2ec0 2022-07-20T03:09:32.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0039ff3700 1 -- 172.21.15.61:0/2895297615 >> v1:172.21.15.117:6789/0 conn(0x7f003405d160 legacy=0x7f00341461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0039ff3700 1 -- 172.21.15.61:0/2895297615 >> v1:172.21.15.61:6789/0 conn(0x7f00340b2ec0 legacy=0x7f0034142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0039ff3700 1 -- 172.21.15.61:0/2895297615 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00340b28c0 con 0x7f00340b32a0 2022-07-20T03:09:32.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0039ff3700 1 --2- 172.21.15.61:0/2895297615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00340b32a0 0x7f00341468d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f002c0032f0 tx=0x7f002c0091d0).ready entity=mon.1 client_cookie=bd6ad5daec53b1ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0023fff700 1 -- 172.21.15.61:0/2895297615 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002c00b6c0 con 0x7f00340b32a0 2022-07-20T03:09:32.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0023fff700 1 -- 172.21.15.61:0/2895297615 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c00b820 con 0x7f00340b32a0 2022-07-20T03:09:32.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f0023fff700 1 -- 172.21.15.61:0/2895297615 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002c00b990 con 0x7f00340b32a0 2022-07-20T03:09:32.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- 172.21.15.61:0/2895297615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00340b32a0 msgr2=0x7f00341468d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 --2- 172.21.15.61:0/2895297615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00340b32a0 0x7f00341468d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f002c0032f0 tx=0x7f002c0091d0).stop 2022-07-20T03:09:32.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- 172.21.15.61:0/2895297615 shutdown_connections 2022-07-20T03:09:32.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 --2- 172.21.15.61:0/2895297615 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f00340b32a0 0x7f00341468d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- 172.21.15.61:0/2895297615 >> 172.21.15.61:0/2895297615 conn(0x7f0034066e20 msgr2=0x7f0034067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:32.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- 172.21.15.61:0/2895297615 shutdown_connections 2022-07-20T03:09:32.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.323+0000 7f003a7f4700 1 -- 172.21.15.61:0/2895297615 wait complete. 2022-07-20T03:09:32.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 Processor -- start 2022-07-20T03:09:32.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 -- start start 2022-07-20T03:09:32.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003405d160 0x7f0034146ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00340b2ec0 0x7f0034152450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 0x7f0034152940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340af040 con 0x7f00340b32a0 2022-07-20T03:09:32.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00340acd80 con 0x7f003405d160 2022-07-20T03:09:32.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f00340acae0 con 0x7f00340b2ec0 2022-07-20T03:09:32.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 0x7f0034152940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 0x7f0034152940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44612/0 (socket says 172.21.15.61:44612) 2022-07-20T03:09:32.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 -- 172.21.15.61:0/3600327307 learned_addr learned my addr 172.21.15.61:0/3600327307 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:32.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f00397f2700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003405d160 0x7f0034146ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0038ff1700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00340b2ec0 0x7f0034152450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 -- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00340b2ec0 msgr2=0x7f0034152450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00340b2ec0 0x7f0034152450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 -- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003405d160 msgr2=0x7f0034146ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003405d160 0x7f0034146ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 -- 172.21.15.61:0/3600327307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c009470 con 0x7f00340b32a0 2022-07-20T03:09:32.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f0039ff3700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 0x7f0034152940 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f002c0032f0 tx=0x7f002c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002c00b6c0 con 0x7f00340b32a0 2022-07-20T03:09:32.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002c021630 con 0x7f00340b32a0 2022-07-20T03:09:32.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0034112560 con 0x7f00340b32a0 2022-07-20T03:09:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002c021990 con 0x7f00340b32a0 2022-07-20T03:09:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.327+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00340afaf0 con 0x7f00340b32a0 2022-07-20T03:09:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.331+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0034112a00 con 0x7f00340b32a0 2022-07-20T03:09:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.331+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f002c029070 con 0x7f00340b32a0 2022-07-20T03:09:32.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.331+0000 7f00227fc700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00140539b0 0x7f0014055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.331+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f002c06cfa0 con 0x7f00340b32a0 2022-07-20T03:09:32.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.331+0000 7f00397f2700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00140539b0 0x7f0014055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.331+0000 7f00397f2700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00140539b0 0x7f0014055e70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f003000d000 tx=0x7f0030009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.339+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0034112a00 con 0x7f00340b32a0 2022-07-20T03:09:32.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.463+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0034112a00 con 0x7f00340b32a0 2022-07-20T03:09:32.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f00227fc700 1 -- 172.21.15.61:0/3600327307 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0034112a00 con 0x7f00340b32a0 2022-07-20T03:09:32.478 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:32.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00140539b0 msgr2=0x7f0014055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00140539b0 0x7f0014055e70 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f003000d000 tx=0x7f0030009040).stop 2022-07-20T03:09:32.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 msgr2=0x7f0034152940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 0x7f0034152940 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f002c0032f0 tx=0x7f002c016000).stop 2022-07-20T03:09:32.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 shutdown_connections 2022-07-20T03:09:32.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00140539b0 0x7f0014055e70 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f00340b32a0 0x7f0034152940 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f00340b2ec0 0x7f0034152450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 --2- 172.21.15.61:0/3600327307 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f003405d160 0x7f0034146ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 >> 172.21.15.61:0/3600327307 conn(0x7f0034066e20 msgr2=0x7f00340b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:32.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 shutdown_connections 2022-07-20T03:09:32.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.467+0000 7f003a7f4700 1 -- 172.21.15.61:0/3600327307 wait complete. 2022-07-20T03:09:32.489 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:09:32.490 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:32.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.607+0000 7f6e8b04a700 1 Processor -- start 2022-07-20T03:09:32.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.607+0000 7f6e8b04a700 1 -- start start 2022-07-20T03:09:32.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6e840b2530 0x7f6e84095880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e84098ab0 0x7f6e84096fb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6e840b2910 con 0x7f6e840620c0 2022-07-20T03:09:32.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6e840980d0 con 0x7f6e840b2530 2022-07-20T03:09:32.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84098370 con 0x7f6e84098ab0 2022-07-20T03:09:32.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e895e7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6e840620c0 0x7f6e840974a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41218/0 (socket says 172.21.15.61:41218) 2022-07-20T03:09:32.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e895e7700 1 -- 172.21.15.61:0/1215693055 learned_addr learned my addr 172.21.15.61:0/1215693055 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:32.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e88de6700 1 --2- 172.21.15.61:0/1215693055 >> v2:172.21.15.117:3300/0 conn(0x7f6e840b2530 0x7f6e84095880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e83fff700 1 --2- 172.21.15.61:0/1215693055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e84098ab0 0x7f6e84096fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e837fe700 1 -- 172.21.15.61:0/1215693055 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4242981473 0 0) 0x7f6e840b2910 con 0x7f6e840620c0 2022-07-20T03:09:32.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e837fe700 1 -- 172.21.15.61:0/1215693055 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6e68003260 con 0x7f6e840620c0 2022-07-20T03:09:32.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e83fff700 1 -- 172.21.15.61:0/1215693055 >> v2:172.21.15.117:3300/0 conn(0x7f6e840b2530 msgr2=0x7f6e84095880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:32.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e83fff700 1 --2- 172.21.15.61:0/1215693055 >> v2:172.21.15.117:3300/0 conn(0x7f6e840b2530 0x7f6e84095880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e83fff700 1 -- 172.21.15.61:0/1215693055 >> v1:172.21.15.61:6789/0 conn(0x7f6e840620c0 legacy=0x7f6e840974a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e83fff700 1 -- 172.21.15.61:0/1215693055 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e84098610 con 0x7f6e84098ab0 2022-07-20T03:09:32.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e83fff700 1 --2- 172.21.15.61:0/1215693055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e84098ab0 0x7f6e84096fb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6e70002f70 tx=0x7f6e70006df0).ready entity=mon.2 client_cookie=be92f0e247c059a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e837fe700 1 -- 172.21.15.61:0/1215693055 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e7000d4a0 con 0x7f6e84098ab0 2022-07-20T03:09:32.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e837fe700 1 -- 172.21.15.61:0/1215693055 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70000e90 con 0x7f6e84098ab0 2022-07-20T03:09:32.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e837fe700 1 -- 172.21.15.61:0/1215693055 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e7000d7f0 con 0x7f6e84098ab0 2022-07-20T03:09:32.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- 172.21.15.61:0/1215693055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e84098ab0 msgr2=0x7f6e84096fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/1215693055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e84098ab0 0x7f6e84096fb0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f6e70002f70 tx=0x7f6e70006df0).stop 2022-07-20T03:09:32.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- 172.21.15.61:0/1215693055 shutdown_connections 2022-07-20T03:09:32.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/1215693055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e84098ab0 0x7f6e84096fb0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/1215693055 >> v2:172.21.15.117:3300/0 conn(0x7f6e840b2530 0x7f6e84095880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- 172.21.15.61:0/1215693055 >> 172.21.15.61:0/1215693055 conn(0x7f6e840b36a0 msgr2=0x7f6e840b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:32.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- 172.21.15.61:0/1215693055 shutdown_connections 2022-07-20T03:09:32.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.611+0000 7f6e8b04a700 1 -- 172.21.15.61:0/1215693055 wait complete. 2022-07-20T03:09:32.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 Processor -- start 2022-07-20T03:09:32.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 -- start start 2022-07-20T03:09:32.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e840620c0 0x7f6e840ae550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 0x7f6e840aea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e840b2530 0x7f6e8414fd80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840ad8e0 con 0x7f6e84098ab0 2022-07-20T03:09:32.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e840aef60 con 0x7f6e840b2530 2022-07-20T03:09:32.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6e84150a30 con 0x7f6e840620c0 2022-07-20T03:09:32.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 0x7f6e840aea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 0x7f6e840aea40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44624/0 (socket says 172.21.15.61:44624) 2022-07-20T03:09:32.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 -- 172.21.15.61:0/2980655381 learned_addr learned my addr 172.21.15.61:0/2980655381 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:32.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e895e7700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e840b2530 0x7f6e8414fd80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e88de6700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e840620c0 0x7f6e840ae550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 -- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e840620c0 msgr2=0x7f6e840ae550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e840620c0 0x7f6e840ae550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 -- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e840b2530 msgr2=0x7f6e8414fd80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e840b2530 0x7f6e8414fd80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 -- 172.21.15.61:0/2980655381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6e7000b180 con 0x7f6e84098ab0 2022-07-20T03:09:32.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e83fff700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 0x7f6e840aea40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f6e70002f70 tx=0x7f6e70014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e70016070 con 0x7f6e84098ab0 2022-07-20T03:09:32.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6e70022070 con 0x7f6e84098ab0 2022-07-20T03:09:32.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e84150d30 con 0x7f6e84098ab0 2022-07-20T03:09:32.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6e70013b70 con 0x7f6e84098ab0 2022-07-20T03:09:32.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.615+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e841429f0 con 0x7f6e84098ab0 2022-07-20T03:09:32.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.619+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e840574e0 con 0x7f6e84098ab0 2022-07-20T03:09:32.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.619+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6e7000f030 con 0x7f6e84098ab0 2022-07-20T03:09:32.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.619+0000 7f6e81ffb700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6e640539d0 0x7f6e64055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:32.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.619+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6e70071230 con 0x7f6e84098ab0 2022-07-20T03:09:32.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.619+0000 7f6e88de6700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6e640539d0 0x7f6e64055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:32.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.619+0000 7f6e88de6700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6e640539d0 0x7f6e64055e90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f6e7800b000 tx=0x7f6e78008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:32.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.627+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6e70048190 con 0x7f6e84098ab0 2022-07-20T03:09:32.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6e84150cc0 con 0x7f6e84098ab0 2022-07-20T03:09:32.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e81ffb700 1 -- 172.21.15.61:0/2980655381 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f6e7004b7b0 con 0x7f6e84098ab0 2022-07-20T03:09:32.804 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:32.804 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6e640539d0 msgr2=0x7f6e64055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6e640539d0 0x7f6e64055e90 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7f6e7800b000 tx=0x7f6e78008040).stop 2022-07-20T03:09:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 msgr2=0x7f6e840aea40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 0x7f6e840aea40 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f6e70002f70 tx=0x7f6e70014000).stop 2022-07-20T03:09:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 shutdown_connections 2022-07-20T03:09:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6e640539d0 0x7f6e64055e90 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6e840b2530 0x7f6e8414fd80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6e84098ab0 0x7f6e840aea40 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 --2- 172.21.15.61:0/2980655381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6e840620c0 0x7f6e840ae550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 >> 172.21.15.61:0/2980655381 conn(0x7f6e840b36a0 msgr2=0x7f6e8405c620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 shutdown_connections 2022-07-20T03:09:32.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:32.787+0000 7f6e8b04a700 1 -- 172.21.15.61:0/2980655381 wait complete. 2022-07-20T03:09:32.824 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:33.049 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:09:27.499383+0000', 'modified': '2022-07-20T03:09:28.502169+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5268': {'gid': 5268, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/844334511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 844334511}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 844334511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:09:33.050 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:09:33.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:09:33.169 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5268,"want_state":"up:active","state":"up:active","rank_uptime":5.6560599720000004,"mdsmap_epoch":20,"osdmap_epoch":42,"osdmap_epoch_barrier":41,"uptime":11.096176660999999} 2022-07-20T03:09:33.170 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5268, "mdsmap_epoch": 20, "osdmap_epoch": 42, "osdmap_epoch_barrier": 41, "rank_uptime": 5.656059972, "state": "up:active", "uptime": 11.096176661, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:09:33.171 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:09:33.186 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:09:33.187 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:09:33.188 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:33.190 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:33.201 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:09:33.223 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:09:33.224 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:09:33.293 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:09:33.321 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:09:33.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:33.330 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:09:33.423 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:09:33.424 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:33.490 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:09:33.515 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:09:33.516 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:09:33.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:09:33.560+0000 7fe997009080 -1 init, newargv = 0x55a08b610470 newargc=9ceph-fuse[ 2022-07-20T03:09:33.567 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:22475]: starting ceph client 2022-07-20T03:09:33.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22475]: starting fuse 2022-07-20T03:09:33.605 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:09:33.615 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:09:33.615 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:33.616 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:09:33.672 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:09:33.673 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:09:33.676 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:33.693 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:09:33.696 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:09:33.697 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:09:33.698 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:09:33.699 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:09:33.700 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:33.701 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:09:33.702 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:09:33.703 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:09:33.704 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:33.705 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:09:33.706 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:09:33.707 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:09:33.708 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:09:33.709 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:33.710 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:09:33.711 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:09:33.712 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:09:33.728 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:09:33.737 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:09:33.738 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:33.745 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:09:33.745 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:09:33.790 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22475.asok 2022-07-20T03:09:33.794 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22475.asok 2022-07-20T03:09:33.795 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:33.796 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22475.asok status 2022-07-20T03:09:33.926 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:09:33.927 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:09:33.927 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:09:33.928 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:09:33.928 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:09:33.929 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:09:33.929 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:09:33.930 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22475", 2022-07-20T03:09:33.934 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:09:33.935 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:09:33.935 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:09:33.935 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:09:33.936 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5440, 2022-07-20T03:09:33.936 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:09:33.937 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:09:33.937 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:09:33.937 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5440 2022-07-20T03:09:33.937 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:09:33.938 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:09:33.938 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:09:33.938 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:09:33.939 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3783591104 2022-07-20T03:09:33.939 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:09:33.939 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:09:33.939 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:09:33.940 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:09:33.940 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:09:33.940 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3783591104 2022-07-20T03:09:33.941 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:09:33.941 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5440 172.21.15.124:0/3783591104", 2022-07-20T03:09:33.941 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/3783591104", 2022-07-20T03:09:33.941 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:09:33.942 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 20, 2022-07-20T03:09:33.942 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 42, 2022-07-20T03:09:33.942 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:09:33.942 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:09:33.972 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:09:33.972 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:09:33.982 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:33.983 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:33.983 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:33.998 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:09:33.999 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:09:33.999 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:34.000 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:34.063 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:34.063 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:34.064 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:34.094 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:09:34.139 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:09:34.168 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:09:34.169 DEBUG:teuthology.orchestra.run.smithi124:> import sys 2022-07-20T03:09:34.170 DEBUG:teuthology.orchestra.run.smithi124:> import time 2022-07-20T03:09:34.171 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:09:34.172 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:34.173 DEBUG:teuthology.orchestra.run.smithi124:> n = 1000 2022-07-20T03:09:34.173 DEBUG:teuthology.orchestra.run.smithi124:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T03:09:34.177 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:34.180 DEBUG:teuthology.orchestra.run.smithi124:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T03:09:34.181 DEBUG:teuthology.orchestra.run.smithi124:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T03:09:34.182 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:34.183 DEBUG:teuthology.orchestra.run.smithi124:> for i in range(0, n): 2022-07-20T03:09:34.184 DEBUG:teuthology.orchestra.run.smithi124:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T03:09:34.186 DEBUG:teuthology.orchestra.run.smithi124:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T03:09:34.187 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content'"'"') 2022-07-20T03:09:34.189 DEBUG:teuthology.orchestra.run.smithi124:> if True: 2022-07-20T03:09:34.190 DEBUG:teuthology.orchestra.run.smithi124:> f.flush() 2022-07-20T03:09:34.191 DEBUG:teuthology.orchestra.run.smithi124:> os.fsync(f.fileno()) 2022-07-20T03:09:34.191 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:09:43.936 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:44.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 Processor -- start 2022-07-20T03:09:44.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 -- start start 2022-07-20T03:09:44.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbb540b1200 0x7fbb540b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb5405bd90 0x7fbb54142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbb540aea30 con 0x7fbb540b5650 2022-07-20T03:09:44.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbb54142970 con 0x7fbb540b1200 2022-07-20T03:09:44.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59c3a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb540aec60 con 0x7fbb5405bd90 2022-07-20T03:09:44.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb58c38700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbb540b5650 0x7fbb5405c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41230/0 (socket says 172.21.15.61:41230) 2022-07-20T03:09:44.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb58c38700 1 -- 172.21.15.61:0/2963028974 learned_addr learned my addr 172.21.15.61:0/2963028974 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:44.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.055+0000 7fbb59439700 1 --2- 172.21.15.61:0/2963028974 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb5405bd90 0x7fbb54142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4bfff700 1 --2- 172.21.15.61:0/2963028974 >> v2:172.21.15.117:3300/0 conn(0x7fbb540b1200 0x7fbb540b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4b7fe700 1 -- 172.21.15.61:0/2963028974 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167755326 0 0) 0x7fbb540aea30 con 0x7fbb540b5650 2022-07-20T03:09:44.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4b7fe700 1 -- 172.21.15.61:0/2963028974 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbb40003260 con 0x7fbb540b5650 2022-07-20T03:09:44.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59439700 1 -- 172.21.15.61:0/2963028974 >> v2:172.21.15.117:3300/0 conn(0x7fbb540b1200 msgr2=0x7fbb540b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:44.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59439700 1 --2- 172.21.15.61:0/2963028974 >> v2:172.21.15.117:3300/0 conn(0x7fbb540b1200 0x7fbb540b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59439700 1 -- 172.21.15.61:0/2963028974 >> v1:172.21.15.61:6789/0 conn(0x7fbb540b5650 legacy=0x7fbb5405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4b7fe700 1 -- 172.21.15.61:0/2963028974 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1155696296 0 0) 0x7fbb40003260 con 0x7fbb540b5650 2022-07-20T03:09:44.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59439700 1 -- 172.21.15.61:0/2963028974 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb540af180 con 0x7fbb5405bd90 2022-07-20T03:09:44.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59439700 1 --2- 172.21.15.61:0/2963028974 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb5405bd90 0x7fbb54142d60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fbb50001800 tx=0x7fbb4c0091d0).ready entity=mon.2 client_cookie=5499f2833b9c0a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4b7fe700 1 -- 172.21.15.61:0/2963028974 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb50001660 con 0x7fbb5405bd90 2022-07-20T03:09:44.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4b7fe700 1 -- 172.21.15.61:0/2963028974 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c00f510 con 0x7fbb5405bd90 2022-07-20T03:09:44.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb4b7fe700 1 -- 172.21.15.61:0/2963028974 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb4c00f680 con 0x7fbb5405bd90 2022-07-20T03:09:44.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 -- 172.21.15.61:0/2963028974 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb5405bd90 msgr2=0x7fbb54142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/2963028974 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb5405bd90 0x7fbb54142d60 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fbb50001800 tx=0x7fbb4c0091d0).stop 2022-07-20T03:09:44.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 -- 172.21.15.61:0/2963028974 shutdown_connections 2022-07-20T03:09:44.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/2963028974 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb5405bd90 0x7fbb54142d60 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/2963028974 >> v2:172.21.15.117:3300/0 conn(0x7fbb540b1200 0x7fbb540b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 -- 172.21.15.61:0/2963028974 >> 172.21.15.61:0/2963028974 conn(0x7fbb540634e0 msgr2=0x7fbb540b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:44.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 -- 172.21.15.61:0/2963028974 shutdown_connections 2022-07-20T03:09:44.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 -- 172.21.15.61:0/2963028974 wait complete. 2022-07-20T03:09:44.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 Processor -- start 2022-07-20T03:09:44.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.059+0000 7fbb59c3a700 1 -- start start 2022-07-20T03:09:44.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb5405bd90 0x7fbb541476f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb540b1200 0x7fbb54141e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 0x7fbb54142310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb54142800 con 0x7fbb540b5650 2022-07-20T03:09:44.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbb541564b0 con 0x7fbb5405bd90 2022-07-20T03:09:44.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbb5413d0d0 con 0x7fbb540b1200 2022-07-20T03:09:44.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 0x7fbb54142310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 0x7fbb54142310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44640/0 (socket says 172.21.15.61:44640) 2022-07-20T03:09:44.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 -- 172.21.15.61:0/35294996 learned_addr learned my addr 172.21.15.61:0/35294996 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:44.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb58c38700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb5405bd90 0x7fbb541476f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb4bfff700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb540b1200 0x7fbb54141e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 -- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb540b1200 msgr2=0x7fbb54141e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb540b1200 0x7fbb54141e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 -- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb5405bd90 msgr2=0x7fbb541476f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb5405bd90 0x7fbb541476f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 -- 172.21.15.61:0/35294996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb4c009470 con 0x7fbb540b5650 2022-07-20T03:09:44.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb58c38700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb5405bd90 0x7fbb541476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:44.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59439700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 0x7fbb54142310 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbb50001800 tx=0x7fbb4c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb4c0032d0 con 0x7fbb540b5650 2022-07-20T03:09:44.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbb5413d370 con 0x7fbb540b5650 2022-07-20T03:09:44.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbb4c011ce0 con 0x7fbb540b5650 2022-07-20T03:09:44.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbb54147f40 con 0x7fbb540b5650 2022-07-20T03:09:44.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbb4c021630 con 0x7fbb540b5650 2022-07-20T03:09:44.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.063+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbb54155740 con 0x7fbb540b5650 2022-07-20T03:09:44.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.067+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbb4c027020 con 0x7fbb540b5650 2022-07-20T03:09:44.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.067+0000 7fbb49ffb700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbb34053ad0 0x7fbb34055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.067+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbb4c06d170 con 0x7fbb540b5650 2022-07-20T03:09:44.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.067+0000 7fbb58c38700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbb34053ad0 0x7fbb34055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.067+0000 7fbb58c38700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbb34053ad0 0x7fbb34055f90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbb5000b000 tx=0x7fbb50001140).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.071+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbb4c044030 con 0x7fbb540b5650 2022-07-20T03:09:44.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.231+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbb540574e0 con 0x7fbb540b5650 2022-07-20T03:09:44.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb49ffb700 1 -- 172.21.15.61:0/35294996 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fbb4c015020 con 0x7fbb540b5650 2022-07-20T03:09:44.238 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:44.239 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:44.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbb34053ad0 msgr2=0x7fbb34055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbb34053ad0 0x7fbb34055f90 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fbb5000b000 tx=0x7fbb50001140).stop 2022-07-20T03:09:44.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 msgr2=0x7fbb54142310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 0x7fbb54142310 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fbb50001800 tx=0x7fbb4c016000).stop 2022-07-20T03:09:44.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 shutdown_connections 2022-07-20T03:09:44.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbb34053ad0 0x7fbb34055f90 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbb540b5650 0x7fbb54142310 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbb540b1200 0x7fbb54141e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 --2- 172.21.15.61:0/35294996 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbb5405bd90 0x7fbb541476f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 >> 172.21.15.61:0/35294996 conn(0x7fbb540634e0 msgr2=0x7fbb540b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:44.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 shutdown_connections 2022-07-20T03:09:44.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.235+0000 7fbb59c3a700 1 -- 172.21.15.61:0/35294996 wait complete. 2022-07-20T03:09:44.256 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:44.259 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_caps 20 2022-07-20T03:09:44.388 INFO:teuthology.orchestra.run.smithi117.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T03:09:44.389 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T03:09:44.390 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:44.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.495+0000 7f85bec4c700 1 Processor -- start 2022-07-20T03:09:44.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.495+0000 7f85bec4c700 1 -- start start 2022-07-20T03:09:44.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f85b813cb70 0x7f85b80b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b8142ba0 con 0x7f85b813c790 2022-07-20T03:09:44.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85b80ba120 con 0x7f85b80b3ca0 2022-07-20T03:09:44.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85b81429b0 con 0x7f85b813cb70 2022-07-20T03:09:44.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bdc4a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f85b813c790 0x7f85b80b9a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41242/0 (socket says 172.21.15.61:41242) 2022-07-20T03:09:44.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bdc4a700 1 -- 172.21.15.61:0/592705526 learned_addr learned my addr 172.21.15.61:0/592705526 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:44.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85be44b700 1 --2- 172.21.15.61:0/592705526 >> v2:172.21.15.117:3300/0 conn(0x7f85b813cb70 0x7f85b80b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bcc48700 1 -- 172.21.15.61:0/592705526 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3621936630 0 0) 0x7f85b8142ba0 con 0x7f85b813c790 2022-07-20T03:09:44.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bcc48700 1 -- 172.21.15.61:0/592705526 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85a0003260 con 0x7f85b813c790 2022-07-20T03:09:44.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bcc48700 1 -- 172.21.15.61:0/592705526 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855849725 0 0) 0x7f85a0003260 con 0x7f85b813c790 2022-07-20T03:09:44.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85be44b700 1 -- 172.21.15.61:0/592705526 >> v1:172.21.15.117:6789/0 conn(0x7f85b80b3ca0 legacy=0x7f85b8143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85be44b700 1 -- 172.21.15.61:0/592705526 >> v1:172.21.15.61:6789/0 conn(0x7f85b813c790 legacy=0x7f85b80b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85be44b700 1 -- 172.21.15.61:0/592705526 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b80b4c00 con 0x7f85b813cb70 2022-07-20T03:09:44.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85be44b700 1 --2- 172.21.15.61:0/592705526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b80b4710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f85b4001800 tx=0x7f85b00091d0).ready entity=mon.1 client_cookie=ba2be785482d093f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bcc48700 1 -- 172.21.15.61:0/592705526 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85b4001660 con 0x7f85b813cb70 2022-07-20T03:09:44.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bcc48700 1 -- 172.21.15.61:0/592705526 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b000b580 con 0x7f85b813cb70 2022-07-20T03:09:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bcc48700 1 -- 172.21.15.61:0/592705526 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85b000b6f0 con 0x7f85b813cb70 2022-07-20T03:09:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- 172.21.15.61:0/592705526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 msgr2=0x7f85b80b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 --2- 172.21.15.61:0/592705526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b80b4710 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f85b4001800 tx=0x7f85b00091d0).stop 2022-07-20T03:09:44.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- 172.21.15.61:0/592705526 shutdown_connections 2022-07-20T03:09:44.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 --2- 172.21.15.61:0/592705526 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b80b4710 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- 172.21.15.61:0/592705526 >> 172.21.15.61:0/592705526 conn(0x7f85b80678b0 msgr2=0x7f85b80b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:44.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- 172.21.15.61:0/592705526 shutdown_connections 2022-07-20T03:09:44.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.499+0000 7f85bec4c700 1 -- 172.21.15.61:0/592705526 wait complete. 2022-07-20T03:09:44.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 Processor -- start 2022-07-20T03:09:44.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 -- start start 2022-07-20T03:09:44.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85b80b3ca0 0x7f85b80b0fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 0x7f85b813ec00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b8152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8141410 con 0x7f85b813c790 2022-07-20T03:09:44.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85b8141600 con 0x7f85b813cb70 2022-07-20T03:09:44.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bec4c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85b813f290 con 0x7f85b80b3ca0 2022-07-20T03:09:44.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 0x7f85b813ec00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 0x7f85b813ec00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44652/0 (socket says 172.21.15.61:44652) 2022-07-20T03:09:44.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 -- 172.21.15.61:0/47481843 learned_addr learned my addr 172.21.15.61:0/47481843 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:44.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85be44b700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b8152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bdc4a700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85b80b3ca0 0x7f85b80b0fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 -- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85b80b3ca0 msgr2=0x7f85b80b0fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85b80b3ca0 0x7f85b80b0fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 -- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 msgr2=0x7f85b8152230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b8152230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 -- 172.21.15.61:0/47481843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b0009470 con 0x7f85b813c790 2022-07-20T03:09:44.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.503+0000 7f85bd449700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 0x7f85b813ec00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f85b8060950 tx=0x7f85a8007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a8012400 con 0x7f85b813c790 2022-07-20T03:09:44.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85a8012560 con 0x7f85b813c790 2022-07-20T03:09:44.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85b813e500 con 0x7f85b813c790 2022-07-20T03:09:44.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85a80128c0 con 0x7f85b813c790 2022-07-20T03:09:44.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85b8142420 con 0x7f85b813c790 2022-07-20T03:09:44.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85b813f0f0 con 0x7f85b813c790 2022-07-20T03:09:44.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f85a8019030 con 0x7f85b813c790 2022-07-20T03:09:44.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85a6ffd700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8598053920 0x7f8598055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f85a806b220 con 0x7f85b813c790 2022-07-20T03:09:44.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.507+0000 7f85bdc4a700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8598053920 0x7f8598055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.511+0000 7f85bdc4a700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8598053920 0x7f8598055de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f85b400b000 tx=0x7f85b4001920).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.515+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85b813f0f0 con 0x7f85b813c790 2022-07-20T03:09:44.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.679+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f85b813f0f0 con 0x7f85b813c790 2022-07-20T03:09:44.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.679+0000 7f85a6ffd700 1 -- 172.21.15.61:0/47481843 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f85b813f0f0 con 0x7f85b813c790 2022-07-20T03:09:44.690 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:44.690 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:44.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8598053920 msgr2=0x7f8598055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8598053920 0x7f8598055de0 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f85b400b000 tx=0x7f85b4001920).stop 2022-07-20T03:09:44.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 msgr2=0x7f85b813ec00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 0x7f85b813ec00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f85b8060950 tx=0x7f85a8007c20).stop 2022-07-20T03:09:44.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 shutdown_connections 2022-07-20T03:09:44.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8598053920 0x7f8598055de0 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85b813cb70 0x7f85b8152230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85b813c790 0x7f85b813ec00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 --2- 172.21.15.61:0/47481843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85b80b3ca0 0x7f85b80b0fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 >> 172.21.15.61:0/47481843 conn(0x7f85b80678b0 msgr2=0x7f85b80b7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:44.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 shutdown_connections 2022-07-20T03:09:44.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.683+0000 7f85bec4c700 1 -- 172.21.15.61:0/47481843 wait complete. 2022-07-20T03:09:44.702 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:44.716 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T03:09:44.827 INFO:teuthology.orchestra.run.smithi117.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T03:09:44.828 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T03:09:44.828 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:09:44.850 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:44.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.967+0000 7fe0bb59e700 1 Processor -- start 2022-07-20T03:09:44.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.967+0000 7fe0bb59e700 1 -- start start 2022-07-20T03:09:44.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0bc0b08c0 0x7fe0bc145b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe0bc060d80 0x7fe0bc0adf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0bc0b1d10 con 0x7fe0bc0b61d0 2022-07-20T03:09:44.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc0ad480 con 0x7fe0bc0b08c0 2022-07-20T03:09:44.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0bc0b65b0 con 0x7fe0bc060d80 2022-07-20T03:09:44.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0bc0b08c0 0x7fe0bc145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0ba59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe0bc0b61d0 0x7fe0bc145400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41256/0 (socket says 172.21.15.61:41256) 2022-07-20T03:09:44.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0ba59c700 1 -- 172.21.15.61:0/2192449852 learned_addr learned my addr 172.21.15.61:0/2192449852 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:44.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bad9d700 1 --2- 172.21.15.61:0/2192449852 >> v2:172.21.15.117:3300/0 conn(0x7fe0bc060d80 0x7fe0bc0adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:44.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b9d9b700 1 -- 172.21.15.61:0/2192449852 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 698745110 0 0) 0x7fe0bc0b1d10 con 0x7fe0bc0b61d0 2022-07-20T03:09:44.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b9d9b700 1 -- 172.21.15.61:0/2192449852 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe09c003260 con 0x7fe0bc0b61d0 2022-07-20T03:09:44.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/2192449852 >> v2:172.21.15.117:3300/0 conn(0x7fe0bc060d80 msgr2=0x7fe0bc0adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:44.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b1d9b700 1 --2- 172.21.15.61:0/2192449852 >> v2:172.21.15.117:3300/0 conn(0x7fe0bc060d80 0x7fe0bc0adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/2192449852 >> v1:172.21.15.61:6789/0 conn(0x7fe0bc0b61d0 legacy=0x7fe0bc145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/2192449852 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0bc0b1f70 con 0x7fe0bc0b08c0 2022-07-20T03:09:44.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bad9d700 1 --2- 172.21.15.61:0/2192449852 >> v2:172.21.15.117:3300/0 conn(0x7fe0bc060d80 0x7fe0bc0adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:44.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b1d9b700 1 --2- 172.21.15.61:0/2192449852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc145b10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fe0a4002f70 tx=0x7fe0a4006df0).ready entity=mon.0 client_cookie=242170060dfe1c18 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:44.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b9d9b700 1 -- 172.21.15.61:0/2192449852 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0a400d570 con 0x7fe0bc0b08c0 2022-07-20T03:09:44.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b9d9b700 1 -- 172.21.15.61:0/2192449852 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a4000e90 con 0x7fe0bc0b08c0 2022-07-20T03:09:44.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0b9d9b700 1 -- 172.21.15.61:0/2192449852 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0a400d8c0 con 0x7fe0bc0b08c0 2022-07-20T03:09:44.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- 172.21.15.61:0/2192449852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 msgr2=0x7fe0bc145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:44.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/2192449852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc145b10 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fe0a4002f70 tx=0x7fe0a4006df0).stop 2022-07-20T03:09:44.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- 172.21.15.61:0/2192449852 shutdown_connections 2022-07-20T03:09:44.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/2192449852 >> v2:172.21.15.117:3300/0 conn(0x7fe0bc060d80 0x7fe0bc0adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/2192449852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc145b10 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- 172.21.15.61:0/2192449852 >> 172.21.15.61:0/2192449852 conn(0x7fe0bc063cf0 msgr2=0x7fe0bc0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- 172.21.15.61:0/2192449852 shutdown_connections 2022-07-20T03:09:44.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.971+0000 7fe0bb59e700 1 -- 172.21.15.61:0/2192449852 wait complete. 2022-07-20T03:09:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 Processor -- start 2022-07-20T03:09:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 -- start start 2022-07-20T03:09:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0bc060d80 0x7fe0bc144110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc1431d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0bc0b61d0 0x7fe0bc155300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:44.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc066830 con 0x7fe0bc0b08c0 2022-07-20T03:09:44.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc1436c0 con 0x7fe0bc060d80 2022-07-20T03:09:44.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0bc064c20 con 0x7fe0bc0b61d0 2022-07-20T03:09:44.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc1431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc1431d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44666/0 (socket says 172.21.15.61:44666) 2022-07-20T03:09:45.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/512042578 learned_addr learned my addr 172.21.15.61:0/512042578 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0ba59c700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0bc060d80 0x7fe0bc144110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bad9d700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0bc0b61d0 0x7fe0bc155300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0bc0b61d0 msgr2=0x7fe0bc155300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0bc0b61d0 0x7fe0bc155300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0bc060d80 msgr2=0x7fe0bc144110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0bc060d80 0x7fe0bc144110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 -- 172.21.15.61:0/512042578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0a400b180 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b1d9b700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc1431d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fe0a4002f70 tx=0x7fe0a4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0a4006b20 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0a4013de0 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0bc064ec0 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0a400dea0 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.975+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0bc112d80 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.979+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe0bc065050 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.979+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe0a401c070 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.979+0000 7fe0b3fff700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0a0053ad0 0x7fe0a0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.979+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe0a406c710 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.979+0000 7fe0ba59c700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0a0053ad0 0x7fe0a0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.979+0000 7fe0ba59c700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0a0053ad0 0x7fe0a0055f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fe0ac00d000 tx=0x7fe0ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:44.987+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe0a40435d0 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.151+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe0bc0574e0 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.151+0000 7fe0b3fff700 1 -- 172.21.15.61:0/512042578 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fe0a4023020 con 0x7fe0bc0b08c0 2022-07-20T03:09:45.200 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:45.200 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:45.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0a0053ad0 msgr2=0x7fe0a0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0a0053ad0 0x7fe0a0055f90 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7fe0ac00d000 tx=0x7fe0ac009040).stop 2022-07-20T03:09:45.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 msgr2=0x7fe0bc1431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc1431d0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fe0a4002f70 tx=0x7fe0a4014000).stop 2022-07-20T03:09:45.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 shutdown_connections 2022-07-20T03:09:45.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0a0053ad0 0x7fe0a0055f90 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0bc0b61d0 0x7fe0bc155300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0bc0b08c0 0x7fe0bc1431d0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 --2- 172.21.15.61:0/512042578 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0bc060d80 0x7fe0bc144110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 >> 172.21.15.61:0/512042578 conn(0x7fe0bc063cf0 msgr2=0x7fe0bc0b4d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:45.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 shutdown_connections 2022-07-20T03:09:45.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.155+0000 7fe0bb59e700 1 -- 172.21.15.61:0/512042578 wait complete. 2022-07-20T03:09:45.223 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:45.224 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.b cache drop 2022-07-20T03:09:45.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 Processor -- start 2022-07-20T03:09:45.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 -- start start 2022-07-20T03:09:45.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f13e00af3e0 0x7f13e0146780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13e005d210 0x7f13e014a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13e0146c70 con 0x7f13e00b5990 2022-07-20T03:09:45.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e00ad050 con 0x7f13e005d210 2022-07-20T03:09:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13e4cf5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f13e00b17e0 con 0x7f13e00af3e0 2022-07-20T03:09:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13e005d210 0x7f13e014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13e005d210 0x7f13e014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44676/0 (socket says 172.21.15.61:44676) 2022-07-20T03:09:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13deffd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f13e00af3e0 0x7f13e0146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 -- 172.21.15.61:0/1484372732 learned_addr learned my addr 172.21.15.61:0/1484372732 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13de7fc700 1 -- 172.21.15.61:0/1484372732 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1593617301 0 0) 0x7f13e0146c70 con 0x7f13e00b5990 2022-07-20T03:09:45.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13de7fc700 1 -- 172.21.15.61:0/1484372732 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f13cc003260 con 0x7f13e00b5990 2022-07-20T03:09:45.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 -- 172.21.15.61:0/1484372732 >> v2:172.21.15.117:3300/0 conn(0x7f13e00af3e0 msgr2=0x7f13e0146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 --2- 172.21.15.61:0/1484372732 >> v2:172.21.15.117:3300/0 conn(0x7f13e00af3e0 0x7f13e0146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 -- 172.21.15.61:0/1484372732 >> v1:172.21.15.117:6789/0 conn(0x7f13e00b5990 legacy=0x7f13e0146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 -- 172.21.15.61:0/1484372732 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13e00b5d70 con 0x7f13e005d210 2022-07-20T03:09:45.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.335+0000 7f13dffff700 1 --2- 172.21.15.61:0/1484372732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a360 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f13d0002f70 tx=0x7f13d0006eb0).ready entity=mon.0 client_cookie=4f3066caa898addc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13de7fc700 1 -- 172.21.15.61:0/1484372732 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d000d610 con 0x7f13e005d210 2022-07-20T03:09:45.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13de7fc700 1 -- 172.21.15.61:0/1484372732 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d0000e90 con 0x7f13e005d210 2022-07-20T03:09:45.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13de7fc700 1 -- 172.21.15.61:0/1484372732 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d000d9a0 con 0x7f13e005d210 2022-07-20T03:09:45.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1484372732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 msgr2=0x7f13e014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1484372732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a360 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f13d0002f70 tx=0x7f13d0006eb0).stop 2022-07-20T03:09:45.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1484372732 shutdown_connections 2022-07-20T03:09:45.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1484372732 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a360 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1484372732 >> v2:172.21.15.117:3300/0 conn(0x7f13e00af3e0 0x7f13e0146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1484372732 >> 172.21.15.61:0/1484372732 conn(0x7f13e0063480 msgr2=0x7f13e00b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:45.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1484372732 shutdown_connections 2022-07-20T03:09:45.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1484372732 wait complete. 2022-07-20T03:09:45.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 Processor -- start 2022-07-20T03:09:45.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- start start 2022-07-20T03:09:45.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13e00af3e0 0x7f13e013f360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e00b5990 0x7f13e013f850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e015a5d0 con 0x7f13e005d210 2022-07-20T03:09:45.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13e0144b00 con 0x7f13e00b5990 2022-07-20T03:09:45.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13e4cf5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13e0144d60 con 0x7f13e00af3e0 2022-07-20T03:09:45.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a5b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44678/0 (socket says 172.21.15.61:44678) 2022-07-20T03:09:45.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13df7fe700 1 -- 172.21.15.61:0/1952429995 learned_addr learned my addr 172.21.15.61:0/1952429995 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13deffd700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13e00af3e0 0x7f13e013f360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.339+0000 7f13dffff700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e00b5990 0x7f13e013f850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13df7fe700 1 -- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13e00af3e0 msgr2=0x7f13e013f360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13df7fe700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13e00af3e0 0x7f13e013f360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13df7fe700 1 -- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e00b5990 msgr2=0x7f13e013f850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13df7fe700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e00b5990 0x7f13e013f850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13df7fe700 1 -- 172.21.15.61:0/1952429995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13d000b1e0 con 0x7f13e005d210 2022-07-20T03:09:45.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dffff700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e00b5990 0x7f13e013f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:45.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13df7fe700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a5b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f13d80014e0 tx=0x7f13d8007e50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dcff9700 1 -- 172.21.15.61:0/1952429995 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d8002030 con 0x7f13e005d210 2022-07-20T03:09:45.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dcff9700 1 -- 172.21.15.61:0/1952429995 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d8002190 con 0x7f13e005d210 2022-07-20T03:09:45.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13e0159750 con 0x7f13e005d210 2022-07-20T03:09:45.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dcff9700 1 -- 172.21.15.61:0/1952429995 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d80024f0 con 0x7f13e005d210 2022-07-20T03:09:45.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e0144700 con 0x7f13e005d210 2022-07-20T03:09:45.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dcff9700 1 -- 172.21.15.61:0/1952429995 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f13d8010070 con 0x7f13e005d210 2022-07-20T03:09:45.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dcff9700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f13c0053af0 0x7f13c0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13dcff9700 1 -- 172.21.15.61:0/1952429995 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13d806b4a0 con 0x7f13e005d210 2022-07-20T03:09:45.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13deffd700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f13c0053af0 0x7f13c0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13f01ae740 1 Processor -- start 2022-07-20T03:09:45.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.343+0000 7f13f01ae740 1 -- start start 2022-07-20T03:09:45.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a8ec10 0x1a88c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13deffd700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f13c0053af0 0x7f13c0055fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f13d400b000 tx=0x7f13d4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 0x1a89600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a52400 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x19c6010 con 0x1a89180 2022-07-20T03:09:45.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x19a6250 con 0x1a8ec10 2022-07-20T03:09:45.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a54400 con 0x1a51f80 2022-07-20T03:09:45.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13deffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 0x1a89600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13deffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 0x1a89600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44688/0 (socket says 172.21.15.61:44688) 2022-07-20T03:09:45.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13deffd700 1 -- 172.21.15.61:0/4179987822 learned_addr learned my addr 172.21.15.61:0/4179987822 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a8ec10 0x1a88c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13dffff700 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a52400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 -- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 msgr2=0x1a52400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a52400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 -- 172.21.15.61:0/4179987822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 msgr2=0x1a89600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 0x1a89600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 -- 172.21.15.61:0/4179987822 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x19fe030 con 0x1a8ec10 2022-07-20T03:09:45.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13deffd700 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 0x1a89600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:09:45.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13df7fe700 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a8ec10 0x1a88c90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f13d806b6b0 tx=0x7f13d8010000).ready entity=mon.1 client_cookie=ed152c65dddacccf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13e4cf5700 1 -- 172.21.15.61:0/4179987822 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d8044200 con 0x1a8ec10 2022-07-20T03:09:45.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13e4cf5700 1 -- 172.21.15.61:0/4179987822 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d8044460 con 0x1a8ec10 2022-07-20T03:09:45.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13e4cf5700 1 -- 172.21.15.61:0/4179987822 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d80457a0 con 0x1a8ec10 2022-07-20T03:09:45.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a8ec10 msgr2=0x1a88c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a8ec10 0x1a88c90 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f13d806b6b0 tx=0x7f13d8010000).stop 2022-07-20T03:09:45.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- 172.21.15.61:0/4179987822 shutdown_connections 2022-07-20T03:09:45.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a52400 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a89180 0x1a89600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 --2- 172.21.15.61:0/4179987822 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a8ec10 0x1a88c90 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- 172.21.15.61:0/4179987822 >> 172.21.15.61:0/4179987822 conn(0x1a532e0 msgr2=0x1a8e1a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:45.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- 172.21.15.61:0/4179987822 shutdown_connections 2022-07-20T03:09:45.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- 172.21.15.61:0/4179987822 wait complete. 2022-07-20T03:09:45.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 -- start start 2022-07-20T03:09:45.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.347+0000 7f13f01ae740 1 Processor -- start 2022-07-20T03:09:45.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 0x1a56070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1a9cc50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a8ec10 0x1a9d140 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a56560 con 0x1a51f80 2022-07-20T03:09:45.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a56780 con 0x1a89180 2022-07-20T03:09:45.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a56a20 con 0x1a8ec10 2022-07-20T03:09:45.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 10 client.0 ll_register_callbacks cb 0x1a0d610 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:09:45.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 0x1a56070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 0x1a56070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44692/0 (socket says 172.21.15.61:44692) 2022-07-20T03:09:45.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 -- 172.21.15.61:0/3606796222 learned_addr learned my addr 172.21.15.61:0/3606796222 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13deffd700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1a9cc50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13dffff700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a8ec10 0x1a9d140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 -- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a8ec10 msgr2=0x1a9d140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a8ec10 0x1a9d140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 -- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 msgr2=0x1a9cc50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1a9cc50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 -- 172.21.15.61:0/3606796222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13d40118b0 con 0x1a51f80 2022-07-20T03:09:45.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13df7fe700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 0x1a56070 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f13d806b6b0 tx=0x7f13d801d070).ready entity=mon.0 client_cookie=6f0784b7f1183f52 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-20T03:09:45.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 1 -- 172.21.15.61:0/3606796222 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d806ae60 con 0x1a51f80 2022-07-20T03:09:45.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 1 -- 172.21.15.61:0/3606796222 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13d802ec60 con 0x1a51f80 2022-07-20T03:09:45.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x19bbfc0 con 0x1a51f80 2022-07-20T03:09:45.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 1 -- 172.21.15.61:0/3606796222 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13d802ee20 con 0x1a51f80 2022-07-20T03:09:45.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 10 client.5457 fetch_fsmap learned FSMap version 20 2022-07-20T03:09:45.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x7f13e0144700 con 0x1a51f80 2022-07-20T03:09:45.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 1 -- 172.21.15.61:0/3606796222 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x19bbfc0 con 0x1a51f80 2022-07-20T03:09:45.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 1 -- 172.21.15.61:0/3606796222 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 20) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f13d8041eb0 con 0x1a51f80 2022-07-20T03:09:45.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 10 client.5457 fetch_fsmap finished waiting for FSMap version 20 2022-07-20T03:09:45.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 10 client.5457 resolve_mds: resolved ID 'b' to GID 5268 2022-07-20T03:09:45.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 20 client.5457 populate_metadata read hostname 'smithi061' 2022-07-20T03:09:45.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1a87210 0x1a875f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 4 client.5457 mds_command: new command op to 5268 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T03:09:45.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 --> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x19b4040 con 0x1a87210 2022-07-20T03:09:45.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13deffd700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1a87210 0x1a875f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13be7fc700 10 client.5457 ms_handle_connect on v2:172.21.15.117:6834/844334511 2022-07-20T03:09:45.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.351+0000 7f13deffd700 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1a87210 0x1a875f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=98e36e3113235588 server_cookie=949fd9d642b01f80 in_seq=0 out_seq=0 2022-07-20T03:09:45.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13be7fc700 1 -- 172.21.15.61:0/3606796222 <== mds.0 v2:172.21.15.117:6834/844334511 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x19b4040 con 0x1a87210 2022-07-20T03:09:45.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13be7fc700 10 client.5457 handle_command_reply: tid=0 2022-07-20T03:09:45.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 client.5457 shutdown 2022-07-20T03:09:45.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 msgr2=0x1a56070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 0x1a56070 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f13d806b6b0 tx=0x7f13d801d070).stop 2022-07-20T03:09:45.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 shutdown_connections 2022-07-20T03:09:45.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1a87210 0x1a875f0 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1a9cc50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a8ec10 0x1a9d140 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13be7fc700 0 client.5457 ms_handle_reset on v2:172.21.15.117:6834/844334511 2022-07-20T03:09:45.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3606796222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a51f80 0x1a56070 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 >> 172.21.15.61:0/3606796222 conn(0x1a532e0 msgr2=0x1a8e0b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:45.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 shutdown_connections 2022-07-20T03:09:45.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 1 -- 172.21.15.61:0/3606796222 wait complete. 2022-07-20T03:09:45.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.355+0000 7f13f01ae740 20 client.5457 trim_cache size 0 max 16384 2022-07-20T03:09:45.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.371+0000 7f13f01ae740 1 Processor -- start 2022-07-20T03:09:45.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.371+0000 7f13f01ae740 1 -- start start 2022-07-20T03:09:45.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a51f80 0x1a89a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a89180 0x1a89f10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1a55160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x19cd0a0 con 0x1a8ec10 2022-07-20T03:09:45.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x1a4c870 con 0x1a51f80 2022-07-20T03:09:45.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x1a9d1d0 con 0x1a89180 2022-07-20T03:09:45.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1a55160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1a55160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44704/0 (socket says 172.21.15.61:44704) 2022-07-20T03:09:45.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 -- 172.21.15.61:0/176219843 learned_addr learned my addr 172.21.15.61:0/176219843 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13df7fe700 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a51f80 0x1a89a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13deffd700 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a89180 0x1a89f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 -- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a89180 msgr2=0x1a89f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a89180 0x1a89f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 -- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a51f80 msgr2=0x1a89a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a51f80 0x1a89a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 -- 172.21.15.61:0/176219843 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x19989b0 con 0x1a8ec10 2022-07-20T03:09:45.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13dffff700 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1a55160 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f13d001c000 tx=0x7f13d0015000).ready entity=mon.0 client_cookie=5db73dbb5b90316d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13c4ff9700 1 -- 172.21.15.61:0/176219843 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d0025070 con 0x1a8ec10 2022-07-20T03:09:45.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13c4ff9700 1 -- 172.21.15.61:0/176219843 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d000dc90 con 0x1a8ec10 2022-07-20T03:09:45.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13c4ff9700 1 -- 172.21.15.61:0/176219843 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d00068b0 con 0x1a8ec10 2022-07-20T03:09:45.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- 172.21.15.61:0/176219843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 msgr2=0x1a55160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:45.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1a55160 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f13d001c000 tx=0x7f13d0015000).stop 2022-07-20T03:09:45.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- 172.21.15.61:0/176219843 shutdown_connections 2022-07-20T03:09:45.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1a55160 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a89180 0x1a89f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 --2- 172.21.15.61:0/176219843 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a51f80 0x1a89a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- 172.21.15.61:0/176219843 >> 172.21.15.61:0/176219843 conn(0x1a532e0 msgr2=0x1a9c960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:45.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- 172.21.15.61:0/176219843 shutdown_connections 2022-07-20T03:09:45.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- 172.21.15.61:0/176219843 wait complete. 2022-07-20T03:09:45.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 -- start start 2022-07-20T03:09:45.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.375+0000 7f13f01ae740 1 Processor -- start 2022-07-20T03:09:45.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a92860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1aebe40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1aec330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x1aec820 con 0x1a8ec10 2022-07-20T03:09:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x1aecce0 con 0x1a89180 2022-07-20T03:09:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x1aed8c0 con 0x1a51f80 2022-07-20T03:09:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 10 client.0 ll_register_callbacks cb 0x1a598a0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:09:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1aec330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13dffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1aec330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44710/0 (socket says 172.21.15.61:44710) 2022-07-20T03:09:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13dffff700 1 -- 172.21.15.61:0/3727951140 learned_addr learned my addr 172.21.15.61:0/3727951140 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:45.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1aebe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13df7fe700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a92860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 -- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 msgr2=0x1a92860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a92860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 -- 172.21.15.61:0/3727951140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 msgr2=0x1aec330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:45.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1aec330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:45.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 -- 172.21.15.61:0/3727951140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13e00b5d70 con 0x1a89180 2022-07-20T03:09:45.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13deffd700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1aebe40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x1aec960 tx=0x7f13d4016000).ready entity=mon.1 client_cookie=6b72417d068b9b97 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:45.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 10 client.0 ms_handle_connect on v2:172.21.15.117:3300/0 2022-07-20T03:09:45.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 1 -- 172.21.15.61:0/3727951140 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13d4014e30 con 0x1a89180 2022-07-20T03:09:45.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f13d8044460 con 0x1a89180 2022-07-20T03:09:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 1 -- 172.21.15.61:0/3727951140 <== mon.1 v2:172.21.15.117:3300/0 2 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f13d4019040 con 0x1a89180 2022-07-20T03:09:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 1 -- 172.21.15.61:0/3727951140 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_get_version_reply(handle=1 version=20) v2 ==== 24+0+0 (secure 0 0 0) 0x7f13d4011d10 con 0x1a89180 2022-07-20T03:09:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 10 client.5482 fetch_fsmap learned FSMap version 20 2022-07-20T03:09:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,fsmap=20,monmap=2+}) v3 -- 0x19b55b0 con 0x1a89180 2022-07-20T03:09:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 1 -- 172.21.15.61:0/3727951140 <== mon.1 v2:172.21.15.117:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13d8044460 con 0x1a89180 2022-07-20T03:09:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 1 -- 172.21.15.61:0/3727951140 <== mon.1 v2:172.21.15.117:3300/0 5 ==== fsmap(e 20) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f13d4013b00 con 0x1a89180 2022-07-20T03:09:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 10 client.5482 fetch_fsmap finished waiting for FSMap version 20 2022-07-20T03:09:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 10 client.5482 resolve_mds: resolved ID 'b' to GID 5268 2022-07-20T03:09:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 20 client.5482 populate_metadata read hostname 'smithi061' 2022-07-20T03:09:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1aeff00 0x1af02e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 4 client.5482 mds_command: new command op to 5268 tid=0[{"prefix": "cache drop"}] 2022-07-20T03:09:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 --> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] -- command(tid 0: {"prefix": "cache drop"}) v1 -- 0x18a9590 con 0x1aeff00 2022-07-20T03:09:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13df7fe700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1aeff00 0x1af02e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:45.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13bf7fe700 10 client.5482 ms_handle_connect on v2:172.21.15.117:6834/844334511 2022-07-20T03:09:45.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:45.379+0000 7f13df7fe700 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1aeff00 0x1af02e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=9fc978bd49084eb9 server_cookie=cd014a63eb740fae in_seq=0 out_seq=0 2022-07-20T03:09:56.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13bf7fe700 1 -- 172.21.15.61:0/3727951140 <== mds.0 v2:172.21.15.117:6834/844334511 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+380 (crc 0 0 0) 0x18a9590 con 0x1aeff00 2022-07-20T03:09:56.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13bf7fe700 10 client.5482 handle_command_reply: tid=0 2022-07-20T03:09:56.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 client.5482 shutdown 2022-07-20T03:09:56.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 msgr2=0x1aebe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:56.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1aebe40 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x1aec960 tx=0x7f13d4016000).stop 2022-07-20T03:09:56.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 shutdown_connections 2022-07-20T03:09:56.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:6834/844334511,v1:172.21.15.117:6835/844334511] conn(0x1aeff00 0x1af02e0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x1a89180 0x1aebe40 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13bf7fe700 0 client.5482 ms_handle_reset on v2:172.21.15.117:6834/844334511 2022-07-20T03:09:56.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x1a8ec10 0x1aec330 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 --2- 172.21.15.61:0/3727951140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x1a51f80 0x1a92860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 >> 172.21.15.61:0/3727951140 conn(0x1a532e0 msgr2=0x1a89a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:56.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 shutdown_connections 2022-07-20T03:09:56.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 1 -- 172.21.15.61:0/3727951140 wait complete. 2022-07-20T03:09:56.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13f01ae740 20 client.5482 trim_cache size 0 max 16384 2022-07-20T03:09:56.567 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:09:56.567 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-20T03:09:56.568 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-20T03:09:56.568 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(0) Success", 2022-07-20T03:09:56.569 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 220 2022-07-20T03:09:56.569 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:56.570 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-20T03:09:56.570 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-20T03:09:56.571 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-20T03:09:56.571 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:56.572 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-20T03:09:56.572 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 0 2022-07-20T03:09:56.573 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:56.573 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-20T03:09:56.573 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-20T03:09:56.574 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 20004, 2022-07-20T03:09:56.574 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 3100576 2022-07-20T03:09:56.582 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:09:56.583 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:09:56.583 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 11.169117004 2022-07-20T03:09:56.584 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:09:56.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f13c0053af0 msgr2=0x7f13c0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:56.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f13c0053af0 0x7f13c0055fb0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f13d400b000 tx=0x7f13d4006df0).stop 2022-07-20T03:09:56.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 msgr2=0x7f13e014a5b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:56.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a5b0 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f13d80014e0 tx=0x7f13d8007e50).stop 2022-07-20T03:09:56.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 shutdown_connections 2022-07-20T03:09:56.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f13c0053af0 0x7f13c0055fb0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13e00b5990 0x7f13e013f850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13e00af3e0 0x7f13e013f360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 --2- 172.21.15.61:0/1952429995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13e005d210 0x7f13e014a5b0 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:56.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 >> 172.21.15.61:0/1952429995 conn(0x7f13e0063480 msgr2=0x7f13e00b4210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:56.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 shutdown_connections 2022-07-20T03:09:56.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.551+0000 7f13e4cf5700 1 -- 172.21.15.61:0/1952429995 wait complete. 2022-07-20T03:09:56.631 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-20T03:09:56.632 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:56.633 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:56.633 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:56.671 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:09:56.672 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T03:09:56.673 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T03:09:56.703 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:09:56.704 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:09:56.705 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:56.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:56.782 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:09:56.783 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:56.862 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:09:56.863 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:09:56.864 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:09:56.865 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:56.882 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:09:56.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.987+0000 7fd5407ec700 1 Processor -- start 2022-07-20T03:09:56.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.987+0000 7fd5407ec700 1 -- start start 2022-07-20T03:09:57.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd5380b3de0 0x7fd53814d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5380b42a0 con 0x7fd5380afa10 2022-07-20T03:09:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd538060c90 con 0x7fd5380608b0 2022-07-20T03:09:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5380af410 con 0x7fd5380b3de0 2022-07-20T03:09:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53dd87700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd5380afa10 0x7fd5380b46d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41314/0 (socket says 172.21.15.61:41314) 2022-07-20T03:09:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53dd87700 1 -- 172.21.15.61:0/263979911 learned_addr learned my addr 172.21.15.61:0/263979911 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53ed89700 1 --2- 172.21.15.61:0/263979911 >> v2:172.21.15.117:3300/0 conn(0x7fd5380b3de0 0x7fd53814d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53d586700 1 -- 172.21.15.61:0/263979911 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3806843951 0 0) 0x7fd5380b42a0 con 0x7fd5380afa10 2022-07-20T03:09:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53d586700 1 -- 172.21.15.61:0/263979911 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd524003260 con 0x7fd5380afa10 2022-07-20T03:09:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53ed89700 1 -- 172.21.15.61:0/263979911 >> v1:172.21.15.117:6789/0 conn(0x7fd5380608b0 legacy=0x7fd53805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53ed89700 1 -- 172.21.15.61:0/263979911 >> v1:172.21.15.61:6789/0 conn(0x7fd5380afa10 legacy=0x7fd5380b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53ed89700 1 -- 172.21.15.61:0/263979911 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5380af6b0 con 0x7fd5380b3de0 2022-07-20T03:09:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53ed89700 1 --2- 172.21.15.61:0/263979911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b3de0 0x7fd53814d920 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fd528002f70 tx=0x7fd528006df0).ready entity=mon.1 client_cookie=d5a5af190a81d3dd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53d586700 1 -- 172.21.15.61:0/263979911 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd52800d610 con 0x7fd5380b3de0 2022-07-20T03:09:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53d586700 1 -- 172.21.15.61:0/263979911 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd528000e90 con 0x7fd5380b3de0 2022-07-20T03:09:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd53d586700 1 -- 172.21.15.61:0/263979911 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd52800d960 con 0x7fd5380b3de0 2022-07-20T03:09:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- 172.21.15.61:0/263979911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b3de0 msgr2=0x7fd53814d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 --2- 172.21.15.61:0/263979911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b3de0 0x7fd53814d920 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7fd528002f70 tx=0x7fd528006df0).stop 2022-07-20T03:09:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- 172.21.15.61:0/263979911 shutdown_connections 2022-07-20T03:09:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 --2- 172.21.15.61:0/263979911 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b3de0 0x7fd53814d920 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- 172.21.15.61:0/263979911 >> 172.21.15.61:0/263979911 conn(0x7fd5380b5ee0 msgr2=0x7fd5380b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- 172.21.15.61:0/263979911 shutdown_connections 2022-07-20T03:09:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.991+0000 7fd5407ec700 1 -- 172.21.15.61:0/263979911 wait complete. 2022-07-20T03:09:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 Processor -- start 2022-07-20T03:09:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 -- start start 2022-07-20T03:09:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380608b0 0x7fd538090c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380afa10 0x7fd538094930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 0x7fd538155330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd538090680 con 0x7fd5380b3de0 2022-07-20T03:09:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380b2f20 con 0x7fd5380afa10 2022-07-20T03:09:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5407ec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380b31c0 con 0x7fd5380608b0 2022-07-20T03:09:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 0x7fd538155330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 0x7fd538155330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44724/0 (socket says 172.21.15.61:44724) 2022-07-20T03:09:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 -- 172.21.15.61:0/2417415237 learned_addr learned my addr 172.21.15.61:0/2417415237 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53dd87700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380afa10 0x7fd538094930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53e588700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380608b0 0x7fd538090c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 -- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380608b0 msgr2=0x7fd538090c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380608b0 0x7fd538090c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 -- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380afa10 msgr2=0x7fd538094930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380afa10 0x7fd538094930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 -- 172.21.15.61:0/2417415237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52800b180 con 0x7fd5380b3de0 2022-07-20T03:09:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd53ed89700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 0x7fd538155330 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fd528002f70 tx=0x7fd528014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.995+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd528016070 con 0x7fd5380b3de0 2022-07-20T03:09:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd5380b3460 con 0x7fd5380b3de0 2022-07-20T03:09:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52800d290 con 0x7fd5380b3de0 2022-07-20T03:09:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd528013de0 con 0x7fd5380b3de0 2022-07-20T03:09:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd53809b720 con 0x7fd5380b3de0 2022-07-20T03:09:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd53809ba00 con 0x7fd5380b3de0 2022-07-20T03:09:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd52800f030 con 0x7fd5380b3de0 2022-07-20T03:09:57.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5377fe700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd518053ad0 0x7fd518055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd52806cbb0 con 0x7fd5380b3de0 2022-07-20T03:09:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd53e588700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd518053ad0 0x7fd518055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:56.999+0000 7fd53e588700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd518053ad0 0x7fd518055f90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd53000d000 tx=0x7fd530007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.007+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd528043980 con 0x7fd5380b3de0 2022-07-20T03:09:57.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.131+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd5380b4de0 con 0x7fd5380b3de0 2022-07-20T03:09:57.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.131+0000 7fd5377fe700 1 -- 172.21.15.61:0/2417415237 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v42) v1 ==== 74+0+16086 (secure 0 0 0) 0x7fd528023020 con 0x7fd5380b3de0 2022-07-20T03:09:57.141 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:57.141 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:09:28.502947+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":7,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:09:25.577917+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"41","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":8,"pool_name":"cephfs_data","create_time":"2022-07-20T03:09:26.436916+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"42","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:09:57.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd518053ad0 msgr2=0x7fd518055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd518053ad0 0x7fd518055f90 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7fd53000d000 tx=0x7fd530007f10).stop 2022-07-20T03:09:57.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 msgr2=0x7fd538155330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 0x7fd538155330 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fd528002f70 tx=0x7fd528014000).stop 2022-07-20T03:09:57.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 shutdown_connections 2022-07-20T03:09:57.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd518053ad0 0x7fd518055f90 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380afa10 0x7fd538094930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b3de0 0x7fd538155330 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 --2- 172.21.15.61:0/2417415237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380608b0 0x7fd538090c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 >> 172.21.15.61:0/2417415237 conn(0x7fd5380b5ee0 msgr2=0x7fd538145600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:57.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 shutdown_connections 2022-07-20T03:09:57.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.135+0000 7fd5407ec700 1 -- 172.21.15.61:0/2417415237 wait complete. 2022-07-20T03:09:57.168 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 7, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:09:25.577917+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '41', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 8, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:09:26.436916+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '42', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:09:57.170 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:09:57.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 Processor -- start 2022-07-20T03:09:57.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 -- start start 2022-07-20T03:09:57.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f07140b62a0 0x7f07140b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b01e0 0x7f071413fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07140b6680 con 0x7f07140b3ed0 2022-07-20T03:09:57.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07141465c0 con 0x7f07140b62a0 2022-07-20T03:09:57.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07198e4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0714142df0 con 0x7f07140b01e0 2022-07-20T03:09:57.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07137fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b01e0 0x7f071413fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07127fc700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f07140b3ed0 0x7f0714143530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:49832/0 (socket says 172.21.15.61:49832) 2022-07-20T03:09:57.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f0712ffd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f07140b62a0 0x7f07140b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.295+0000 7f07127fc700 1 -- 172.21.15.61:0/3229996597 learned_addr learned my addr 172.21.15.61:0/3229996597 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:57.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0712ffd700 1 -- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b01e0 msgr2=0x7f071413fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:57.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0712ffd700 1 --2- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b01e0 0x7f071413fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0712ffd700 1 -- 172.21.15.61:0/3229996597 >> v1:172.21.15.117:6789/0 conn(0x7f07140b3ed0 legacy=0x7f0714143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0712ffd700 1 -- 172.21.15.61:0/3229996597 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0714143090 con 0x7f07140b62a0 2022-07-20T03:09:57.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0712ffd700 1 --2- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b62a0 0x7f07140b43e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f07040032f0 tx=0x7f07040091d0).ready entity=mon.1 client_cookie=cf24e806436988b3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0711ffb700 1 -- 172.21.15.61:0/3229996597 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f070400b6c0 con 0x7f07140b62a0 2022-07-20T03:09:57.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0711ffb700 1 -- 172.21.15.61:0/3229996597 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f070400b820 con 0x7f07140b62a0 2022-07-20T03:09:57.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f0711ffb700 1 -- 172.21.15.61:0/3229996597 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f070400b990 con 0x7f07140b62a0 2022-07-20T03:09:57.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 -- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b62a0 msgr2=0x7f07140b43e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 --2- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b62a0 0x7f07140b43e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f07040032f0 tx=0x7f07040091d0).stop 2022-07-20T03:09:57.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 -- 172.21.15.61:0/3229996597 shutdown_connections 2022-07-20T03:09:57.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 --2- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b01e0 0x7f071413fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 --2- 172.21.15.61:0/3229996597 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b62a0 0x7f07140b43e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 -- 172.21.15.61:0/3229996597 >> 172.21.15.61:0/3229996597 conn(0x7f0714066f00 msgr2=0x7f07140672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 -- 172.21.15.61:0/3229996597 shutdown_connections 2022-07-20T03:09:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 -- 172.21.15.61:0/3229996597 wait complete. 2022-07-20T03:09:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 Processor -- start 2022-07-20T03:09:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.299+0000 7f07198e4700 1 -- start start 2022-07-20T03:09:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b01e0 0x7f0714140f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 0x7f07140af3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b62a0 0x7f0714151f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07140aea00 con 0x7f07140b3ed0 2022-07-20T03:09:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07140aeca0 con 0x7f07140b01e0 2022-07-20T03:09:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07140aef40 con 0x7f07140b62a0 2022-07-20T03:09:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 0x7f07140af3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 0x7f07140af3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44736/0 (socket says 172.21.15.61:44736) 2022-07-20T03:09:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 -- 172.21.15.61:0/2310281466 learned_addr learned my addr 172.21.15.61:0/2310281466 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f0712ffd700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b01e0 0x7f0714140f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07137fe700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b62a0 0x7f0714151f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 -- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b62a0 msgr2=0x7f0714151f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b62a0 0x7f0714151f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 -- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b01e0 msgr2=0x7f0714140f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b01e0 0x7f0714140f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 -- 172.21.15.61:0/2310281466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0704009470 con 0x7f07140b3ed0 2022-07-20T03:09:57.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07137fe700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b62a0 0x7f0714151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:57.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07127fc700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 0x7f07140af3e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f07140b3b70 tx=0x7f06fc007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06fc0070d0 con 0x7f07140b3ed0 2022-07-20T03:09:57.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f06fc007230 con 0x7f07140b3ed0 2022-07-20T03:09:57.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07140ae000 con 0x7f07140b3ed0 2022-07-20T03:09:57.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06fc005610 con 0x7f07140b3ed0 2022-07-20T03:09:57.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.303+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07141428c0 con 0x7f07140b3ed0 2022-07-20T03:09:57.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.307+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f06fc010070 con 0x7f07140b3ed0 2022-07-20T03:09:57.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.307+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07140af100 con 0x7f07140b3ed0 2022-07-20T03:09:57.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.307+0000 7f070bfff700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f06f8053a60 0x7f06f8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.307+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f06fc06b430 con 0x7f07140b3ed0 2022-07-20T03:09:57.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.307+0000 7f0712ffd700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f06f8053a60 0x7f06f8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.307+0000 7f0712ffd700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f06f8053a60 0x7f06f8055f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f070401b000 tx=0x7f0704016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.311+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f06fc0422f0 con 0x7f07140b3ed0 2022-07-20T03:09:57.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.435+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0714143c40 con 0x7f07140b3ed0 2022-07-20T03:09:57.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.435+0000 7f070bfff700 1 -- 172.21.15.61:0/2310281466 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v42) v1 ==== 110+0+10 (secure 0 0 0) 0x7f06fc018070 con 0x7f07140b3ed0 2022-07-20T03:09:57.446 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:09:57.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f06f8053a60 msgr2=0x7f06f8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f06f8053a60 0x7f06f8055f20 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f070401b000 tx=0x7f0704016000).stop 2022-07-20T03:09:57.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 msgr2=0x7f07140af3e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 0x7f07140af3e0 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f07140b3b70 tx=0x7f06fc007c40).stop 2022-07-20T03:09:57.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 shutdown_connections 2022-07-20T03:09:57.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f06f8053a60 0x7f06f8055f20 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07140b62a0 0x7f0714151f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07140b3ed0 0x7f07140af3e0 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 --2- 172.21.15.61:0/2310281466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07140b01e0 0x7f0714140f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 >> 172.21.15.61:0/2310281466 conn(0x7f0714066f00 msgr2=0x7f07140b1dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:57.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 shutdown_connections 2022-07-20T03:09:57.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.439+0000 7f07198e4700 1 -- 172.21.15.61:0/2310281466 wait complete. 2022-07-20T03:09:57.496 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:09:57.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.607+0000 7fdee2d99700 1 Processor -- start 2022-07-20T03:09:57.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.607+0000 7fdee2d99700 1 -- start start 2022-07-20T03:09:57.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc0af030 0x7fdedc05c570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdedc0b5690 0x7fdedc140730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdedc140c20 0x7fdedc1296f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdedc0af410 con 0x7fdedc140c20 2022-07-20T03:09:57.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdedc144050 con 0x7fdedc0b5690 2022-07-20T03:09:57.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc05be30 con 0x7fdedc0af030 2022-07-20T03:09:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdedc140c20 0x7fdedc1296f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdedc140c20 0x7fdedc1296f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44744/0 (socket says 172.21.15.61:44744) 2022-07-20T03:09:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 -- 172.21.15.61:0/3550257876 learned_addr learned my addr 172.21.15.61:0/3550257876 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:57.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee1d97700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc0af030 0x7fdedc05c570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee1596700 1 --2- 172.21.15.61:0/3550257876 >> v2:172.21.15.117:3300/0 conn(0x7fdedc0b5690 0x7fdedc140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 -- 172.21.15.61:0/3550257876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc0af030 msgr2=0x7fdedc05c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:57.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc0af030 0x7fdedc05c570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 -- 172.21.15.61:0/3550257876 >> v2:172.21.15.117:3300/0 conn(0x7fdedc0b5690 msgr2=0x7fdedc140730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:57.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 --2- 172.21.15.61:0/3550257876 >> v2:172.21.15.117:3300/0 conn(0x7fdedc0b5690 0x7fdedc140730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 -- 172.21.15.61:0/3550257876 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdedc05c0d0 con 0x7fdedc140c20 2022-07-20T03:09:57.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee1d97700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc0af030 0x7fdedc05c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:57.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee1596700 1 --2- 172.21.15.61:0/3550257876 >> v2:172.21.15.117:3300/0 conn(0x7fdedc0b5690 0x7fdedc140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2598700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc140c20 0x7fdedc1296f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fdecc002f70 tx=0x7fdecc006eb0).ready entity=mon.0 client_cookie=c62f2413e91a058f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee0d95700 1 -- 172.21.15.61:0/3550257876 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdecc00d570 con 0x7fdedc140c20 2022-07-20T03:09:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee0d95700 1 -- 172.21.15.61:0/3550257876 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdecc000e90 con 0x7fdedc140c20 2022-07-20T03:09:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee0d95700 1 -- 172.21.15.61:0/3550257876 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdecc00d900 con 0x7fdedc140c20 2022-07-20T03:09:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- 172.21.15.61:0/3550257876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc140c20 msgr2=0x7fdedc1296f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc140c20 0x7fdedc1296f0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7fdecc002f70 tx=0x7fdecc006eb0).stop 2022-07-20T03:09:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- 172.21.15.61:0/3550257876 shutdown_connections 2022-07-20T03:09:57.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc140c20 0x7fdedc1296f0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- 172.21.15.61:0/3550257876 >> v2:172.21.15.117:3300/0 conn(0x7fdedc0b5690 0x7fdedc140730 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 --2- 172.21.15.61:0/3550257876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc0af030 0x7fdedc05c570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- 172.21.15.61:0/3550257876 >> 172.21.15.61:0/3550257876 conn(0x7fdedc0634c0 msgr2=0x7fdedc0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:57.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- 172.21.15.61:0/3550257876 shutdown_connections 2022-07-20T03:09:57.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- 172.21.15.61:0/3550257876 wait complete. 2022-07-20T03:09:57.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 Processor -- start 2022-07-20T03:09:57.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.611+0000 7fdee2d99700 1 -- start start 2022-07-20T03:09:57.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdedc0af030 0x7fdedc066a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 0x7fdedc155af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc140c20 0x7fdedc12ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc155110 con 0x7fdedc0b5690 2022-07-20T03:09:57.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc1553b0 con 0x7fdedc0af030 2022-07-20T03:09:57.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdedc155650 con 0x7fdedc140c20 2022-07-20T03:09:57.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 0x7fdedc155af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 0x7fdedc155af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44746/0 (socket says 172.21.15.61:44746) 2022-07-20T03:09:57.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 -- 172.21.15.61:0/212630787 learned_addr learned my addr 172.21.15.61:0/212630787 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:57.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1d97700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdedc0af030 0x7fdedc066a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2598700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc140c20 0x7fdedc12ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 -- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc140c20 msgr2=0x7fdedc12ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc140c20 0x7fdedc12ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 -- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdedc0af030 msgr2=0x7fdedc066a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdedc0af030 0x7fdedc066a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 -- 172.21.15.61:0/212630787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdecc00b1e0 con 0x7fdedc0b5690 2022-07-20T03:09:57.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee1596700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 0x7fdedc155af0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fded800a000 tx=0x7fded8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fded8011780 con 0x7fdedc0b5690 2022-07-20T03:09:57.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fded80118e0 con 0x7fdedc0b5690 2022-07-20T03:09:57.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdedc0add10 con 0x7fdedc0b5690 2022-07-20T03:09:57.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fded8011c40 con 0x7fdedc0b5690 2022-07-20T03:09:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdedc12e950 con 0x7fdedc0b5690 2022-07-20T03:09:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fded801c460 con 0x7fdedc0b5690 2022-07-20T03:09:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdedc143d40 con 0x7fdedc0b5690 2022-07-20T03:09:57.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.615+0000 7fded2ffd700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdebc0539b0 0x7fdebc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:57.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.619+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fded806c340 con 0x7fdedc0b5690 2022-07-20T03:09:57.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.619+0000 7fdee1d97700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdebc0539b0 0x7fdebc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:57.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.619+0000 7fdee1d97700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdebc0539b0 0x7fdebc055e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fded400d000 tx=0x7fded4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:57.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.623+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fded80431a0 con 0x7fdedc0b5690 2022-07-20T03:09:57.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.743+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdedc0574e0 con 0x7fdedc0b5690 2022-07-20T03:09:57.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.743+0000 7fded2ffd700 1 -- 172.21.15.61:0/212630787 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v42) v1 ==== 92+0+10 (secure 0 0 0) 0x7fded80467c0 con 0x7fdedc0b5690 2022-07-20T03:09:57.928 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:57.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdebc0539b0 msgr2=0x7fdebc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdebc0539b0 0x7fdebc055e70 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fded400d000 tx=0x7fded4008040).stop 2022-07-20T03:09:57.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 msgr2=0x7fdedc155af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:57.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 0x7fdedc155af0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7fded800a000 tx=0x7fded8008040).stop 2022-07-20T03:09:57.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 shutdown_connections 2022-07-20T03:09:57.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdebc0539b0 0x7fdebc055e70 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdedc140c20 0x7fdedc12ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:57.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdedc0b5690 0x7fdedc155af0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 --2- 172.21.15.61:0/212630787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdedc0af030 0x7fdedc066a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 >> 172.21.15.61:0/212630787 conn(0x7fdedc0634c0 msgr2=0x7fdedc05c740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 shutdown_connections 2022-07-20T03:09:58.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:57.747+0000 7fdee2d99700 1 -- 172.21.15.61:0/212630787 wait complete. 2022-07-20T03:09:58.041 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:09:58.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 Processor -- start 2022-07-20T03:09:58.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 -- start start 2022-07-20T03:09:58.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcd00b9da0 0x7fdcd01468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd01419a0 0x7fdcd0141e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcd00ba180 con 0x7fdcd00b3cf0 2022-07-20T03:09:58.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcd00ad5b0 con 0x7fdcd00b9da0 2022-07-20T03:09:58.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd700d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd01461b0 con 0x7fdcd01419a0 2022-07-20T03:09:58.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd600b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcd00b9da0 0x7fdcd01468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd600b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcd00b9da0 0x7fdcd01468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44756/0 (socket says 172.21.15.61:44756) 2022-07-20T03:09:58.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd600b700 1 -- 172.21.15.61:0/2453976941 learned_addr learned my addr 172.21.15.61:0/2453976941 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:58.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.159+0000 7fdcd680c700 1 --2- 172.21.15.61:0/2453976941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd01419a0 0x7fdcd0141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd5009700 1 -- 172.21.15.61:0/2453976941 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 285702394 0 0) 0x7fdcd00ba180 con 0x7fdcd00b3cf0 2022-07-20T03:09:58.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd600b700 1 -- 172.21.15.61:0/2453976941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd01419a0 msgr2=0x7fdcd0141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:58.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd600b700 1 --2- 172.21.15.61:0/2453976941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd01419a0 0x7fdcd0141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd600b700 1 -- 172.21.15.61:0/2453976941 >> v1:172.21.15.117:6789/0 conn(0x7fdcd00b3cf0 legacy=0x7fdcd0142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd600b700 1 -- 172.21.15.61:0/2453976941 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcd0146450 con 0x7fdcd00b9da0 2022-07-20T03:09:58.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd600b700 1 --2- 172.21.15.61:0/2453976941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd01468f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fdcc0002f70 tx=0x7fdcc0006eb0).ready entity=mon.0 client_cookie=2485ba247ce924d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd5009700 1 -- 172.21.15.61:0/2453976941 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcc000d570 con 0x7fdcd00b9da0 2022-07-20T03:09:58.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd5009700 1 -- 172.21.15.61:0/2453976941 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcc0000e90 con 0x7fdcd00b9da0 2022-07-20T03:09:58.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd5009700 1 -- 172.21.15.61:0/2453976941 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcc000d8c0 con 0x7fdcd00b9da0 2022-07-20T03:09:58.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 -- 172.21.15.61:0/2453976941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 msgr2=0x7fdcd01468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2453976941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd01468f0 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fdcc0002f70 tx=0x7fdcc0006eb0).stop 2022-07-20T03:09:58.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 -- 172.21.15.61:0/2453976941 shutdown_connections 2022-07-20T03:09:58.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2453976941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd01419a0 0x7fdcd0141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2453976941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd01468f0 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 -- 172.21.15.61:0/2453976941 >> 172.21.15.61:0/2453976941 conn(0x7fdcd00678c0 msgr2=0x7fdcd00b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 -- 172.21.15.61:0/2453976941 shutdown_connections 2022-07-20T03:09:58.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.163+0000 7fdcd700d700 1 -- 172.21.15.61:0/2453976941 wait complete. 2022-07-20T03:09:58.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 Processor -- start 2022-07-20T03:09:58.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 -- start start 2022-07-20T03:09:58.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd00b3cf0 0x7fdcd013f7a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd013d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01419a0 0x7fdcd0151de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd0148ea0 con 0x7fdcd00b9da0 2022-07-20T03:09:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00b19a0 con 0x7fdcd01419a0 2022-07-20T03:09:58.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcd00b1c40 con 0x7fdcd00b3cf0 2022-07-20T03:09:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd013d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd013d7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44762/0 (socket says 172.21.15.61:44762) 2022-07-20T03:09:58.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd680c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01419a0 0x7fdcd0151de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 -- 172.21.15.61:0/2581411514 learned_addr learned my addr 172.21.15.61:0/2581411514 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:58.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd600b700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd00b3cf0 0x7fdcd013f7a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 -- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd00b3cf0 msgr2=0x7fdcd013f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd00b3cf0 0x7fdcd013f7a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 -- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01419a0 msgr2=0x7fdcd0151de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01419a0 0x7fdcd0151de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 -- 172.21.15.61:0/2581411514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcc000b1e0 con 0x7fdcd00b9da0 2022-07-20T03:09:58.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd600b700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd00b3cf0 0x7fdcd013f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:58.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd580a700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd013d7f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fdccc000ed0 tx=0x7fdccc007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdccc012400 con 0x7fdcd00b9da0 2022-07-20T03:09:58.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdccc012560 con 0x7fdcd00b9da0 2022-07-20T03:09:58.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcd013fc90 con 0x7fdcd00b9da0 2022-07-20T03:09:58.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdccc0128c0 con 0x7fdcd00b9da0 2022-07-20T03:09:58.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.167+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcd0141210 con 0x7fdcd00b9da0 2022-07-20T03:09:58.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.171+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcd0151b20 con 0x7fdcd00b9da0 2022-07-20T03:09:58.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.171+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdccc019030 con 0x7fdcd00b9da0 2022-07-20T03:09:58.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.171+0000 7fdcbf7fe700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdcb0053a40 0x7fdcb0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.171+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdccc06b260 con 0x7fdcd00b9da0 2022-07-20T03:09:58.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.171+0000 7fdcd600b700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdcb0053a40 0x7fdcb0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.171+0000 7fdcd600b700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdcb0053a40 0x7fdcb0055f00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fdcc001a000 tx=0x7fdcc0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.179+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdcd0151b20 con 0x7fdcd00b9da0 2022-07-20T03:09:58.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.307+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fdcd0151b20 con 0x7fdcd00b9da0 2022-07-20T03:09:58.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.307+0000 7fdcbf7fe700 1 -- 172.21.15.61:0/2581411514 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v42) v1 ==== 104+0+10 (secure 0 0 0) 0x7fdcd0151b20 con 0x7fdcd00b9da0 2022-07-20T03:09:58.313 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:58.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdcb0053a40 msgr2=0x7fdcb0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdcb0053a40 0x7fdcb0055f00 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7fdcc001a000 tx=0x7fdcc0015000).stop 2022-07-20T03:09:58.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 msgr2=0x7fdcd013d7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd013d7f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fdccc000ed0 tx=0x7fdccc007a90).stop 2022-07-20T03:09:58.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 shutdown_connections 2022-07-20T03:09:58.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdcb0053a40 0x7fdcb0055f00 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcd01419a0 0x7fdcd0151de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcd00b9da0 0x7fdcd013d7f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 --2- 172.21.15.61:0/2581411514 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcd00b3cf0 0x7fdcd013f7a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 >> 172.21.15.61:0/2581411514 conn(0x7fdcd00678c0 msgr2=0x7fdcd00b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 shutdown_connections 2022-07-20T03:09:58.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.311+0000 7fdcd700d700 1 -- 172.21.15.61:0/2581411514 wait complete. 2022-07-20T03:09:58.347 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:09:58.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 Processor -- start 2022-07-20T03:09:58.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 -- start start 2022-07-20T03:09:58.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f634c0b2ec0 0x7f634c0b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f634c05d160 0x7f634c142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f634c0b3f10 con 0x7f634c142700 2022-07-20T03:09:58.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634c0b4ea0 con 0x7f634c0b2ec0 2022-07-20T03:09:58.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63527e3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f634c142b30 con 0x7f634c05d160 2022-07-20T03:09:58.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.471+0000 7f63517e1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f634c0b2ec0 0x7f634c0b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f634c0b2ec0 0x7f634c0b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44768/0 (socket says 172.21.15.61:44768) 2022-07-20T03:09:58.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 -- 172.21.15.61:0/4288350900 learned_addr learned my addr 172.21.15.61:0/4288350900 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:58.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f6350fe0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f634c05d160 0x7f634c142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 -- 172.21.15.61:0/4288350900 >> v2:172.21.15.117:3300/0 conn(0x7f634c05d160 msgr2=0x7f634c142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:58.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 --2- 172.21.15.61:0/4288350900 >> v2:172.21.15.117:3300/0 conn(0x7f634c05d160 0x7f634c142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 -- 172.21.15.61:0/4288350900 >> v1:172.21.15.117:6789/0 conn(0x7f634c142700 legacy=0x7f634c1461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 -- 172.21.15.61:0/4288350900 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634c0b3790 con 0x7f634c0b2ec0 2022-07-20T03:09:58.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63517e1700 1 --2- 172.21.15.61:0/4288350900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c0b2ec0 0x7f634c0b32a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6340002f70 tx=0x7f6340006eb0).ready entity=mon.0 client_cookie=fc02e83155ac4772 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f6337fff700 1 -- 172.21.15.61:0/4288350900 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f634000d5c0 con 0x7f634c0b2ec0 2022-07-20T03:09:58.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f6337fff700 1 -- 172.21.15.61:0/4288350900 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6340000e90 con 0x7f634c0b2ec0 2022-07-20T03:09:58.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f6337fff700 1 -- 172.21.15.61:0/4288350900 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f634000d950 con 0x7f634c0b2ec0 2022-07-20T03:09:58.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 -- 172.21.15.61:0/4288350900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c0b2ec0 msgr2=0x7f634c0b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 --2- 172.21.15.61:0/4288350900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c0b2ec0 0x7f634c0b32a0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f6340002f70 tx=0x7f6340006eb0).stop 2022-07-20T03:09:58.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 -- 172.21.15.61:0/4288350900 shutdown_connections 2022-07-20T03:09:58.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 --2- 172.21.15.61:0/4288350900 >> v2:172.21.15.117:3300/0 conn(0x7f634c05d160 0x7f634c142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 --2- 172.21.15.61:0/4288350900 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c0b2ec0 0x7f634c0b32a0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 -- 172.21.15.61:0/4288350900 >> 172.21.15.61:0/4288350900 conn(0x7f634c066e20 msgr2=0x7f634c067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 -- 172.21.15.61:0/4288350900 shutdown_connections 2022-07-20T03:09:58.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 -- 172.21.15.61:0/4288350900 wait complete. 2022-07-20T03:09:58.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 Processor -- start 2022-07-20T03:09:58.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.475+0000 7f63527e3700 1 -- start start 2022-07-20T03:09:58.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 0x7f634c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f634c0b2ec0 0x7f634c144400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c142700 0x7f634c112840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0ada10 con 0x7f634c05d160 2022-07-20T03:09:58.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0aeab0 con 0x7f634c142700 2022-07-20T03:09:58.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f634c0acae0 con 0x7f634c0b2ec0 2022-07-20T03:09:58.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 0x7f634c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 0x7f634c155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44774/0 (socket says 172.21.15.61:44774) 2022-07-20T03:09:58.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 -- 172.21.15.61:0/457510115 learned_addr learned my addr 172.21.15.61:0/457510115 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:58.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f6350fe0700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f634c0b2ec0 0x7f634c144400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f6351fe2700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c142700 0x7f634c112840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 -- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f634c0b2ec0 msgr2=0x7f634c144400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f634c0b2ec0 0x7f634c144400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 -- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c142700 msgr2=0x7f634c112840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c142700 0x7f634c112840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 -- 172.21.15.61:0/457510115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634000b1e0 con 0x7f634c05d160 2022-07-20T03:09:58.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f6350fe0700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f634c0b2ec0 0x7f634c144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:58.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63517e1700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 0x7f634c155c30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6340002f70 tx=0x7f6340015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6340011860 con 0x7f634c05d160 2022-07-20T03:09:58.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63400119c0 con 0x7f634c05d160 2022-07-20T03:09:58.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f634c0acff0 con 0x7f634c05d160 2022-07-20T03:09:58.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6340011d20 con 0x7f634c05d160 2022-07-20T03:09:58.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.479+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f634c146d20 con 0x7f634c05d160 2022-07-20T03:09:58.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.483+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f634c0acd30 con 0x7f634c05d160 2022-07-20T03:09:58.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.483+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6340025030 con 0x7f634c05d160 2022-07-20T03:09:58.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.483+0000 7f633effd700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6328053a40 0x7f6328055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.483+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f634006d050 con 0x7f634c05d160 2022-07-20T03:09:58.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.483+0000 7f6350fe0700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6328053a40 0x7f6328055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.483+0000 7f6350fe0700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6328053a40 0x7f6328055f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f634800b000 tx=0x7f6348008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.491+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6340043fb0 con 0x7f634c05d160 2022-07-20T03:09:58.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.615+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f634c0b2620 con 0x7f634c05d160 2022-07-20T03:09:58.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f633effd700 1 -- 172.21.15.61:0/457510115 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v42) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6340023020 con 0x7f634c05d160 2022-07-20T03:09:58.657 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:09:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6328053a40 msgr2=0x7f6328055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6328053a40 0x7f6328055f00 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f634800b000 tx=0x7f6348008040).stop 2022-07-20T03:09:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 msgr2=0x7f634c155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 0x7f634c155c30 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f6340002f70 tx=0x7f6340015000).stop 2022-07-20T03:09:58.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 shutdown_connections 2022-07-20T03:09:58.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6328053a40 0x7f6328055f00 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f634c142700 0x7f634c112840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f634c0b2ec0 0x7f634c144400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 --2- 172.21.15.61:0/457510115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f634c05d160 0x7f634c155c30 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 >> 172.21.15.61:0/457510115 conn(0x7f634c066e20 msgr2=0x7f634c0b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 shutdown_connections 2022-07-20T03:09:58.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.619+0000 7f63527e3700 1 -- 172.21.15.61:0/457510115 wait complete. 2022-07-20T03:09:58.663 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:09:58.664 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:09:58.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 Processor -- start 2022-07-20T03:09:58.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 -- start start 2022-07-20T03:09:58.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdcb80b3ca0 0x7fdcb80b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 0x7fdcb8143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcb8142ba0 con 0x7fdcb813c790 2022-07-20T03:09:58.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcb81429b0 con 0x7fdcb80b3ca0 2022-07-20T03:09:58.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb8146330 con 0x7fdcb813cb70 2022-07-20T03:09:58.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 0x7fdcb8143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb67fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdcb80b3ca0 0x7fdcb80b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb6ffd700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fdcb813c790 0x7fdcb80b9a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:49884/0 (socket says 172.21.15.61:49884) 2022-07-20T03:09:58.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb67fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdcb80b3ca0 0x7fdcb80b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54464/0 (socket says 172.21.15.61:54464) 2022-07-20T03:09:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb6ffd700 1 -- 172.21.15.61:0/592277207 learned_addr learned my addr 172.21.15.61:0/592277207 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:09:58.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 0x7fdcb8143000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:59688/0 (socket says 172.21.15.61:59688) 2022-07-20T03:09:58.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 -- 172.21.15.61:0/592277207 >> v2:172.21.15.117:3300/0 conn(0x7fdcb80b3ca0 msgr2=0x7fdcb80b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:09:58.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 --2- 172.21.15.61:0/592277207 >> v2:172.21.15.117:3300/0 conn(0x7fdcb80b3ca0 0x7fdcb80b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 -- 172.21.15.61:0/592277207 >> v1:172.21.15.117:6789/0 conn(0x7fdcb813c790 legacy=0x7fdcb80b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 -- 172.21.15.61:0/592277207 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb80ba120 con 0x7fdcb813cb70 2022-07-20T03:09:58.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb77fe700 1 --2- 172.21.15.61:0/592277207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 0x7fdcb8143000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fdcb0002f70 tx=0x7fdcb0006df0).ready entity=mon.2 client_cookie=96e88bfb9edc8f89 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb5ffb700 1 -- 172.21.15.61:0/592277207 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb000d570 con 0x7fdcb813cb70 2022-07-20T03:09:58.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb5ffb700 1 -- 172.21.15.61:0/592277207 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdcb0000e90 con 0x7fdcb813cb70 2022-07-20T03:09:58.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.771+0000 7fdcb5ffb700 1 -- 172.21.15.61:0/592277207 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdcb000d8c0 con 0x7fdcb813cb70 2022-07-20T03:09:58.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/592277207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 msgr2=0x7fdcb8143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/592277207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 0x7fdcb8143000 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7fdcb0002f70 tx=0x7fdcb0006df0).stop 2022-07-20T03:09:58.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/592277207 shutdown_connections 2022-07-20T03:09:58.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/592277207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb813cb70 0x7fdcb8143000 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/592277207 >> v2:172.21.15.117:3300/0 conn(0x7fdcb80b3ca0 0x7fdcb80b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/592277207 >> 172.21.15.61:0/592277207 conn(0x7fdcb80678b0 msgr2=0x7fdcb80b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/592277207 shutdown_connections 2022-07-20T03:09:58.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/592277207 wait complete. 2022-07-20T03:09:58.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 Processor -- start 2022-07-20T03:09:58.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- start start 2022-07-20T03:09:58.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb80b3ca0 0x7fdcb8141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 0x7fdcb813cf80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcb813cb70 0x7fdcb80b02b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb80aeaf0 con 0x7fdcb813c790 2022-07-20T03:09:58.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb813eb90 con 0x7fdcb813cb70 2022-07-20T03:09:58.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcb813edf0 con 0x7fdcb80b3ca0 2022-07-20T03:09:58.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 0x7fdcb813cf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 0x7fdcb813cf80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44790/0 (socket says 172.21.15.61:44790) 2022-07-20T03:09:58.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 -- 172.21.15.61:0/575431494 learned_addr learned my addr 172.21.15.61:0/575431494 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:09:58.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb77fe700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcb813cb70 0x7fdcb80b02b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb6ffd700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb80b3ca0 0x7fdcb8141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 -- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb80b3ca0 msgr2=0x7fdcb8141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb80b3ca0 0x7fdcb8141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 -- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcb813cb70 msgr2=0x7fdcb80b02b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcb813cb70 0x7fdcb80b02b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 -- 172.21.15.61:0/575431494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcb000b180 con 0x7fdcb813c790 2022-07-20T03:09:58.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb77fe700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcb813cb70 0x7fdcb80b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:09:58.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb67fc700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 0x7fdcb813cf80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fdca000c000 tx=0x7fdca0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdca0010780 con 0x7fdcb813c790 2022-07-20T03:09:58.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdca00108e0 con 0x7fdcb813c790 2022-07-20T03:09:58.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcb80afe10 con 0x7fdcb813c790 2022-07-20T03:09:58.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdca0010c40 con 0x7fdcb813c790 2022-07-20T03:09:58.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.775+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcb813fed0 con 0x7fdcb813c790 2022-07-20T03:09:58.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.779+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcb81401e0 con 0x7fdcb813c790 2022-07-20T03:09:58.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.779+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdca001c460 con 0x7fdcb813c790 2022-07-20T03:09:58.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.779+0000 7fdcaffff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc9c0539b0 0x7fdc9c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:09:58.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.779+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdca006c2f0 con 0x7fdcb813c790 2022-07-20T03:09:58.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.779+0000 7fdcb6ffd700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc9c0539b0 0x7fdc9c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:09:58.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.779+0000 7fdcb6ffd700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc9c0539b0 0x7fdc9c055e70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fdca800b000 tx=0x7fdca8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:09:58.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.783+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdca00431f0 con 0x7fdcb813c790 2022-07-20T03:09:58.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.943+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdcb80affa0 con 0x7fdcb813c790 2022-07-20T03:09:58.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.943+0000 7fdcaffff700 1 -- 172.21.15.61:0/575431494 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7fdca0046810 con 0x7fdcb813c790 2022-07-20T03:09:58.988 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:09:58.988 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:09:58.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc9c0539b0 msgr2=0x7fdc9c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc9c0539b0 0x7fdc9c055e70 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7fdca800b000 tx=0x7fdca8009040).stop 2022-07-20T03:09:58.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 msgr2=0x7fdcb813cf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:09:58.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 0x7fdcb813cf80 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fdca000c000 tx=0x7fdca0006df0).stop 2022-07-20T03:09:58.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 shutdown_connections 2022-07-20T03:09:58.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc9c0539b0 0x7fdc9c055e70 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcb813cb70 0x7fdcb80b02b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcb813c790 0x7fdcb813cf80 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 --2- 172.21.15.61:0/575431494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcb80b3ca0 0x7fdcb8141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:09:58.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 >> 172.21.15.61:0/575431494 conn(0x7fdcb80678b0 msgr2=0x7fdcb8146660 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:09:58.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 shutdown_connections 2022-07-20T03:09:59.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:09:58.947+0000 7fdcb7fff700 1 -- 172.21.15.61:0/575431494 wait complete. 2022-07-20T03:09:59.000 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:09:59.206 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 20, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:09:27.499383+0000', 'modified': '2022-07-20T03:09:28.502169+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5268}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5268': {'gid': 5268, 'name': 'b', 'rank': 0, 'incarnation': 19, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/844334511', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 844334511}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 844334511}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [8], 'metadata_pool': 7, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:09:59.207 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:09:59.208 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:09:59.350 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5268,"want_state":"up:active","state":"up:active","rank_uptime":31.821128782999999,"mdsmap_epoch":20,"osdmap_epoch":42,"osdmap_epoch_barrier":41,"uptime":37.261247963000002} 2022-07-20T03:09:59.351 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5268, "mdsmap_epoch": 20, "osdmap_epoch": 42, "osdmap_epoch_barrier": 41, "rank_uptime": 31.821128783, "state": "up:active", "uptime": 37.261247963, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:09:59.351 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:09:59.352 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:09:59.352 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:09:59.353 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:59.353 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:59.394 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:09:59.427 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:09:59.428 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:09:59.498 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:09:59.499 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:09:59.499 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:59.500 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:09:59.523 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:09:59.525 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:09:59.573 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:09:59.615 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:09:59.616 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:09:59.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:09:59.644+0000 7fc905138080 -1 init, newargv = 0x557135450290 newargc=9 2022-07-20T03:09:59.647 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22549]: starting ceph client 2022-07-20T03:09:59.658 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22549]: starting fuse 2022-07-20T03:09:59.671 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:09:59.679 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:09:59.680 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:59.692 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:09:59.768 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:09:59.768 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:09:59.769 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:59.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:09:59.779 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:09:59.779 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:09:59.780 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:09:59.780 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:09:59.781 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:59.781 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:09:59.781 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:09:59.818 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:09:59.819 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:59.819 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:09:59.819 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:09:59.820 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:09:59.820 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:09:59.821 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:59.821 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:09:59.828 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:09:59.829 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:09:59.829 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:09:59.830 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:09:59.830 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:09:59.831 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:09:59.839 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:09:59.907 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22549.asok 2022-07-20T03:09:59.939 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22549.asok 2022-07-20T03:09:59.940 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:09:59.946 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22549.asok status 2022-07-20T03:10:00.056 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:10:00.057 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:10:00.058 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:10:00.058 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:10:00.059 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:10:00.059 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:10:00.060 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:10:00.060 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22549", 2022-07-20T03:10:00.060 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:10:00.061 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:10:00.064 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:10:00.065 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:10:00.065 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5502, 2022-07-20T03:10:00.065 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:10:00.066 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:10:00.066 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:10:00.067 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5502 2022-07-20T03:10:00.067 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:10:00.067 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:10:00.068 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:10:00.068 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:10:00.069 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1969269383 2022-07-20T03:10:00.069 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:10:00.070 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:10:00.070 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:10:00.070 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:10:00.071 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:10:00.071 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1969269383 2022-07-20T03:10:00.072 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:10:00.145 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5502 172.21.15.124:0/1969269383", 2022-07-20T03:10:00.145 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/1969269383", 2022-07-20T03:10:00.145 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:10:00.145 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 20, 2022-07-20T03:10:00.146 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 42, 2022-07-20T03:10:00.146 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:10:00.146 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:10:00.147 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:10:00.147 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:10:00.148 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:10:00.149 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:10:00.149 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:10:00.200 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:10:00.210 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:10:00.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:10:00.217 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:10:01.628 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:01.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:01.680 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:01.704 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:01.712 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:31.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:31.794 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:31.797 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:31.800 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:31.808 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:10:47.107 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:10:47.125 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:10:47.161 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:10:47.241 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:10:47.282 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:10:47.305 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:10:47.340 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:10:47.341 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:10:47.342 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:10:47.343 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:10:47.382 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:10:47.383 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:10:47.384 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:10:47.385 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:10:47.386 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:10:47.453 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22549]: fuse finished with error 0 and tester_r 0 2022-07-20T03:10:47.458 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:10:47.459 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:10:47.460 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:10:47.531 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:10:47.534 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:10:47.565 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:10:47.567 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:10:47.596 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:10:47.607 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:10:47.608 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:10:47.678 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:10:47.679 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:10:47.680 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:10:47.680 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:10:47.681 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:10:47.761 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:10:47.763 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:10:47.764 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:10:47.765 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:10:47.766 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:10:47.787 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:10:47.809 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:10:47.809 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:10:47.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:10:47.811 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:10:47.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:10:47.905 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:10:47.921 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:10:48.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.039+0000 7ff24759e700 1 Processor -- start 2022-07-20T03:10:48.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.039+0000 7ff24759e700 1 -- start start 2022-07-20T03:10:48.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2480b7b70 0x7ff248146860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:48.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff24812aad0 0x7ff24812af60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:48.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2480b7f50 con 0x7ff2480b1520 2022-07-20T03:10:48.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff248146120 con 0x7ff2480b7b70 2022-07-20T03:10:48.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2481463c0 con 0x7ff24812aad0 2022-07-20T03:10:48.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2480b7b70 0x7ff248146860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:48.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff246d9d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff24812aad0 0x7ff24812af60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:48.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2480b7b70 0x7ff248146860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44798/0 (socket says 172.21.15.61:44798) 2022-07-20T03:10:48.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 -- 172.21.15.61:0/1964295374 learned_addr learned my addr 172.21.15.61:0/1964295374 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:48.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 -- 172.21.15.61:0/1964295374 >> v2:172.21.15.117:3300/0 conn(0x7ff24812aad0 msgr2=0x7ff24812af60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:48.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 --2- 172.21.15.61:0/1964295374 >> v2:172.21.15.117:3300/0 conn(0x7ff24812aad0 0x7ff24812af60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:48.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 -- 172.21.15.61:0/1964295374 >> v1:172.21.15.117:6789/0 conn(0x7ff2480b1520 legacy=0x7ff248066870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:48.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 -- 172.21.15.61:0/1964295374 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff24812b450 con 0x7ff2480b7b70 2022-07-20T03:10:48.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24659c700 1 --2- 172.21.15.61:0/1964295374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff248146860 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7ff238002f70 tx=0x7ff238006eb0).ready entity=mon.0 client_cookie=2d831dbbc01e3cae server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:48.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24559a700 1 -- 172.21.15.61:0/1964295374 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff23800d5c0 con 0x7ff2480b7b70 2022-07-20T03:10:48.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24559a700 1 -- 172.21.15.61:0/1964295374 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff238000e90 con 0x7ff2480b7b70 2022-07-20T03:10:48.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24559a700 1 -- 172.21.15.61:0/1964295374 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff23800d950 con 0x7ff2480b7b70 2022-07-20T03:10:48.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- 172.21.15.61:0/1964295374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 msgr2=0x7ff248146860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:48.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 --2- 172.21.15.61:0/1964295374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff248146860 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7ff238002f70 tx=0x7ff238006eb0).stop 2022-07-20T03:10:48.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- 172.21.15.61:0/1964295374 shutdown_connections 2022-07-20T03:10:48.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 --2- 172.21.15.61:0/1964295374 >> v2:172.21.15.117:3300/0 conn(0x7ff24812aad0 0x7ff24812af60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:48.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 --2- 172.21.15.61:0/1964295374 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff248146860 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:48.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- 172.21.15.61:0/1964295374 >> 172.21.15.61:0/1964295374 conn(0x7ff2480656b0 msgr2=0x7ff2480b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:48.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- 172.21.15.61:0/1964295374 shutdown_connections 2022-07-20T03:10:48.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- 172.21.15.61:0/1964295374 wait complete. 2022-07-20T03:10:48.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 Processor -- start 2022-07-20T03:10:48.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.043+0000 7ff24759e700 1 -- start start 2022-07-20T03:10:48.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2480b1520 0x7ff24812c870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:48.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff2480b2a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:48.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff24812aad0 0x7ff248130d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:48.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24812f310 con 0x7ff2480b7b70 2022-07-20T03:10:48.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff24812ce30 con 0x7ff2480b1520 2022-07-20T03:10:48.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff248155580 con 0x7ff24812aad0 2022-07-20T03:10:48.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff2480b2a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:48.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff2480b2a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44804/0 (socket says 172.21.15.61:44804) 2022-07-20T03:10:48.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 -- 172.21.15.61:0/3057697515 learned_addr learned my addr 172.21.15.61:0/3057697515 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24659c700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2480b1520 0x7ff24812c870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff246d9d700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff24812aad0 0x7ff248130d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 -- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff24812aad0 msgr2=0x7ff248130d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:48.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff24812aad0 0x7ff248130d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:48.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 -- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2480b1520 msgr2=0x7ff24812c870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:48.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2480b1520 0x7ff24812c870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:48.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 -- 172.21.15.61:0/3057697515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23800b1e0 con 0x7ff2480b7b70 2022-07-20T03:10:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff245d9b700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff2480b2a40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff248146740 tx=0x7ff230007f30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff230006aa0 con 0x7ff2480b7b70 2022-07-20T03:10:48.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff230006c00 con 0x7ff2480b7b70 2022-07-20T03:10:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff248155820 con 0x7ff2480b7b70 2022-07-20T03:10:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff230007030 con 0x7ff2480b7b70 2022-07-20T03:10:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff248155ee0 con 0x7ff2480b7b70 2022-07-20T03:10:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.047+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2481559b0 con 0x7ff2480b7b70 2022-07-20T03:10:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.051+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff23001b060 con 0x7ff2480b7b70 2022-07-20T03:10:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.051+0000 7ff23f7fe700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff22c0539b0 0x7ff22c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.051+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff23006bb40 con 0x7ff2480b7b70 2022-07-20T03:10:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.051+0000 7ff24659c700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff22c0539b0 0x7ff22c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:48.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.051+0000 7ff24659c700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff22c0539b0 0x7ff22c055e70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ff23801c000 tx=0x7ff238015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:48.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.059+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff230042aa0 con 0x7ff2480b7b70 2022-07-20T03:10:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.179+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff248066f80 con 0x7ff2480b7b70 2022-07-20T03:10:48.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff23f7fe700 1 -- 172.21.15.61:0/3057697515 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v42) v1 ==== 81+0+33693 (secure 0 0 0) 0x7ff230019030 con 0x7ff2480b7b70 2022-07-20T03:10:48.262 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:10:48.263 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:10:48.263 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 42, 2022-07-20T03:10:48.264 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:10:48.264 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:10:48.265 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:09:28.502947+0000", 2022-07-20T03:10:48.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:10:48.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:10:48.291 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:10:48.292 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:10:48.292 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:10:48.293 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:10:48.293 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:10:48.293 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:10:48.303 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:10:48.303 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:10:48.304 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:10:48.304 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:10:48.305 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:10:48.306 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:10:48.306 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:10:48.307 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 8, 2022-07-20T03:10:48.307 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:10:48.308 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:10:48.308 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:10:48.308 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:10:48.309 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:10:48.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:48.309 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:10:48.310 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:10:48.310 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:10:48.315 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:10:48.315 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:10:48.316 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:10:48.316 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:10:48.322 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:10:48.323 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:10:48.323 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:10:48.323 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:10:48.347 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:10:48.348 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:10:48.348 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:10:48.348 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:10:48.349 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:10:48.356 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:10:48.356 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:10:48.356 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:10:48.367 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:10:48.369 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:10:48.370 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:10:48.370 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:10:48.378 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:48.378 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:10:48.426 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:10:48.426 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:10:48.427 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:10:48.427 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:10:48.428 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:10:48.428 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:10:48.428 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:10:48.429 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:10:48.429 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:10:48.447 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:10:48.448 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:10:48.448 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:10:48.449 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:10:48.449 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:10:48.450 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:10:48.450 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:10:48.452 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:10:48.453 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:10:48.453 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:10:48.454 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:10:48.454 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:10:48.455 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:10:48.455 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:10:48.456 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:10:48.456 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:10:48.457 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:10:48.457 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:48.457 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:10:48.485 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:10:48.485 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:10:48.485 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:10:48.486 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:10:48.502 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:10:48.503 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:10:48.530 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:10:48.531 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:10:48.531 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:10:48.531 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:10:48.532 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:10:48.577 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:10:48.577 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:48.577 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:10:48.577 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:10:48.578 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:48.613 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:48.613 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:48.634 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:10:48.634 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:10:48.672 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:10:48.672 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:10:48.680 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:10:48.680 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:10:48.681 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:10:48.681 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:10:48.682 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:10:48.682 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:10:48.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:10:48.685 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:10:48.686 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:10:48.687 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:10:48.687 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:10:48.687 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:10:48.688 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:10:48.688 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:10:48.689 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:10:48.689 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:10:48.690 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:10:48.690 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:10:48.691 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:10:48.691 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:48.691 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:10:48.693 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:10:48.693 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:10:48.694 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:10:48.695 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:10:48.754 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:10:48.755 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:10:48.755 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:10:48.755 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:10:48.756 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:10:48.756 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:10:48.757 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:10:48.757 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:10:48.898 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:10:48.898 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:10:48.899 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:10:48.899 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:10:48.899 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:10:48.900 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:10:48.900 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:10:48.900 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:10:48.920 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:10:48.942 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:10:48.943 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:10:48.943 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:10:48.944 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:10:48.944 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:10:48.945 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:48.945 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:10:48.955 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:10:48.955 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:10:48.956 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:10:48.956 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:10:48.957 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:10:48.957 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:10:48.957 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:10:49.077 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:10:49.077 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:10:49.078 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:10:49.078 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:10:49.079 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:10:49.079 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:10:49.080 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.080 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.084 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 7, 2022-07-20T03:10:49.084 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:10:49.085 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:09:25.577917+0000", 2022-07-20T03:10:49.085 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:10:49.086 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:10:49.110 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:10:49.111 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:10:49.111 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:10:49.112 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:10:49.136 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:10:49.149 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:10:49.165 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:10:49.166 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:10:49.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:10:49.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:10:49.199 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:10:49.199 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:10:49.200 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:10:49.200 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:10:49.201 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:10:49.201 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:10:49.202 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:10:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:10:49.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "41", 2022-07-20T03:10:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:10:49.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:10:49.249 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:10:49.249 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:10:49.306 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:10:49.311 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:10:49.312 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:10:49.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:10:49.314 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:10:49.314 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:10:49.315 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:10:49.315 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:10:49.315 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:10:49.316 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:10:49.316 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:10:49.317 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:10:49.420 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:10:49.421 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:10:49.421 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:10:49.422 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:10:49.423 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:10:49.423 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:10:49.424 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:10:49.425 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:10:49.425 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:10:49.426 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:10:49.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.427 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:10:49.427 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:10:49.428 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:10:49.428 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:10:49.437 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:10:49.438 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:10:49.438 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:10:49.439 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:10:49.439 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:10:49.440 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:10:49.440 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:10:49.440 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:10:49.441 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:10:49.451 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:10:49.452 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:10:49.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.453 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:10:49.453 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:10:49.454 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:10:49.454 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.454 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.455 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.472 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 8, 2022-07-20T03:10:49.475 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:10:49.475 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:09:26.436916+0000", 2022-07-20T03:10:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:10:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:10:49.482 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:10:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:10:49.483 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:10:49.509 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:10:49.509 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:10:49.509 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:10:49.510 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:10:49.510 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:10:49.510 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:10:49.513 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:10:49.513 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:10:49.514 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:10:49.514 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:10:49.515 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:10:49.515 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:10:49.515 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:10:49.516 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:10:49.516 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:10:49.516 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.516 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "42", 2022-07-20T03:10:49.517 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:10:49.517 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:10:49.517 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:10:49.517 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:10:49.518 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:10:49.518 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:10:49.518 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:10:49.519 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:10:49.519 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:10:49.526 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:10:49.526 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:10:49.539 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:10:49.539 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:10:49.540 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:10:49.540 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:10:49.540 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:10:49.540 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:10:49.541 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:10:49.541 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:10:49.541 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:10:49.541 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:10:49.542 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:10:49.542 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:10:49.542 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:10:49.601 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:10:49.602 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:10:49.602 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.603 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:10:49.603 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:10:49.604 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:10:49.604 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:10:49.605 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:10:49.605 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:10:49.605 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:10:49.606 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:10:49.606 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:10:49.607 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:10:49.609 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:10:49.609 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:10:49.611 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:10:49.612 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:10:49.612 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:10:49.613 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.613 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.646 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.647 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:10:49.668 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:10:49.668 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.668 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:10:49.670 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:10:49.670 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:49.671 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:49.671 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:49.671 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:10:49.672 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.673 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.674 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.674 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:10:49.674 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.675 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.675 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.675 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.675 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:10:49.675 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.676 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.676 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.676 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.676 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:49.697 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.697 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.697 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.698 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:10:49.698 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.698 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.699 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.699 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.699 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:10:49.699 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.700 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.700 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.700 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.700 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:49.701 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.701 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.701 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.701 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:10:49.701 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.702 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.703 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.703 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.703 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:10:49.703 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.704 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.704 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.704 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.704 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:49.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.705 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.705 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:10:49.729 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.730 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.731 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.732 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.733 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:10:49.733 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:10:49.734 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.747 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.747 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.747 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:10:49.748 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:10:49.748 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:10:49.748 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:10:49.748 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:49.765 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:49.765 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:49.765 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.772 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.772 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.773 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:10:49.773 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:10:49.773 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:49.773 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:49.774 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:49.774 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:49.774 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:49.774 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:49.775 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:49.775 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:10:49.775 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:49.776 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:49.776 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:49.776 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.776 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.776 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.777 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:10:49.777 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.777 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.777 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.778 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.778 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:10:49.778 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.779 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.779 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.779 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.780 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:49.780 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.780 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.780 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.780 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:10:49.781 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.781 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.781 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.782 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:10:49.783 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.783 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.784 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.784 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.784 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:49.784 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.785 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.785 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:10:49.794 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.794 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.795 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.795 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.795 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:10:49.795 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.796 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.796 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.796 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.796 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:49.797 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.797 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.797 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.797 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:10:49.799 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.799 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.799 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.800 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.800 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:10:49.800 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:10:49.800 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.801 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.801 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.801 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:10:49.801 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:10:49.802 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:10:49.807 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:10:49.808 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:49.808 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:49.808 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:49.809 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.809 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.809 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:10:49.810 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:10:49.810 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:49.810 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:49.811 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:49.811 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:49.811 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:49.812 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:49.813 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:49.813 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-20T03:10:49.813 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:49.822 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:49.822 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:49.822 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.823 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.823 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.823 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:10:49.823 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.824 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.824 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.824 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.824 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:10:49.825 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.825 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.825 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.825 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.840 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:49.841 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.841 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.842 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.842 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:10:49.843 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.843 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.844 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.844 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.845 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:10:49.845 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.846 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.846 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.847 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.854 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:49.855 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.855 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.855 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.856 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:10:49.856 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.856 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.856 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.857 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.857 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:10:49.857 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.857 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.865 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.865 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.866 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:49.866 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.866 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.867 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.867 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:10:49.867 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.867 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.868 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.868 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.868 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:10:49.868 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:10:49.869 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.869 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.869 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.869 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:10:49.870 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:10:49.870 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:10:49.870 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:10:49.875 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:49.876 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:49.876 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:49.876 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.877 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.877 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:10:49.877 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:10:49.878 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:49.878 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:49.878 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:49.879 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:49.879 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:49.879 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:49.879 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:49.880 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:10:49.880 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:49.880 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:49.881 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:49.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.881 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.882 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.883 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:10:49.883 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.884 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.884 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.884 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.884 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:10:49.885 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.886 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.887 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.888 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.888 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:49.888 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.889 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.890 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.890 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:10:49.891 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.891 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.892 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.894 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:10:49.894 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.895 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.895 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.898 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:49.898 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.900 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.901 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:10:49.901 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.923 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.924 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:49.924 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:10:49.932 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.933 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:49.933 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:49.947 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:49.948 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:49.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:49.948 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:49.949 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:49.949 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:10:49.981 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:49.981 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.027 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.028 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:10:50.029 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:10:50.029 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.030 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.031 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:10:50.035 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:10:50.035 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:10:50.036 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:10:50.036 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:50.037 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:50.037 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:50.037 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.038 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.038 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.039 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:10:50.039 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:10:50.040 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:50.061 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:50.061 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:50.062 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:50.063 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:50.063 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:50.064 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:50.064 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:10:50.064 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:50.065 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:50.065 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:50.066 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.067 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:10:50.068 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.070 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:10:50.070 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.071 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.071 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.072 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:50.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.073 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.073 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.081 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:10:50.082 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.082 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.083 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.084 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:10:50.084 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.095 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.096 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.096 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:50.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.098 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.098 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:10:50.099 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.113 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.113 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:10:50.114 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.115 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.115 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.116 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.116 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:50.117 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.127 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.128 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:10:50.134 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:10:50.142 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:10:50.143 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.144 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:10:50.145 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:10:50.145 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:10:50.173 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:10:50.173 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:50.174 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:50.174 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:50.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.175 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:10:50.175 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:10:50.175 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:50.176 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:50.176 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:50.176 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:50.177 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:50.177 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:50.177 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:50.177 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:10:50.182 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:50.183 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:50.183 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:50.184 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.189 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.190 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:10:50.190 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.191 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.214 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:10:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.229 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.229 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.230 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:50.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.231 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:10:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:10:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:10:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:10:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:10:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:10:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:10:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:10:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:10:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:10:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:10:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:10:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:10:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:50.339 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:50.340 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:50.340 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:10:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:50.342 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:10:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:10:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:50.529 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.530 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.531 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:10:50.531 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.531 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.532 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.533 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:10:50.533 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.660 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.661 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.690 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.691 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:50.699 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.699 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.700 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.700 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:10:50.701 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.777 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.778 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.778 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:10:50.779 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.779 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.780 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.780 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.781 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:50.781 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.783 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.783 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:10:50.784 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.784 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.784 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.785 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:10:50.786 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:10:50.786 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.787 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.809 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:10:50.813 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:10:50.814 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:10:50.814 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:10:50.815 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:50.815 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:50.816 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:50.816 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.816 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.817 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.817 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:10:50.818 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:10:50.818 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:10:50.819 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:10:50.819 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:10:50.820 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:10:50.843 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:10:50.844 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:10:50.844 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:10:50.845 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 38, 2022-07-20T03:10:50.845 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:10:50.846 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:10:50.847 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:10:50.847 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.848 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:10:50.869 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:50.870 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.870 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.910 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.911 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:10:50.911 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:50.912 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.912 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.912 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.927 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:10:50.927 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:50.928 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.928 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:50.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:10:50.948 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:50.949 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.949 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:50.952 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:50.953 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:10:50.953 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:50.954 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:50.954 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:50.955 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:50.955 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:10:51.000 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:51.000 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.000 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:51.001 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:10:51.001 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:51.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.001 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.002 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:51.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:10:51.003 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:51.003 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:51.003 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:51.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.004 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:10:51.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:10:51.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.005 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:10:51.005 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:10:51.005 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:51.005 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:10:51.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:10:51.075 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:10:51.075 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:51.076 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:51.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.076 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:10:51.076 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:10:51.077 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:10:51.077 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:10:51.077 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:10:51.077 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:10:51.077 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:10:51.107 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:10:51.108 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:51.110 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:10:51.110 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:10:51.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.111 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:10:51.111 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.111 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.112 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.147 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.148 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.148 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:10:51.149 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.149 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.150 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:10:51.150 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.151 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.151 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.152 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.152 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.152 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:10:51.153 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.168 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.169 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:10:51.169 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.170 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.170 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.171 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.171 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.172 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:10:51.199 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.200 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:10:51.200 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.201 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.201 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.234 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.234 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:10:51.235 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.235 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.236 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:10:51.237 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.240 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.240 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.241 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.241 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.258 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:10:51.258 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.259 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:10:51.260 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.260 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.260 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.261 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.261 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.268 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:10:51.269 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.270 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.270 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:10:51.270 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.270 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.271 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.271 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.271 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.271 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:10:51.272 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.272 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:10:51.272 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:10:51.273 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:10:51.273 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:10:51.273 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:10:51.273 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:10:51.274 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:10:51.274 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:10:51.274 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:10:51.274 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:51.275 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:10:51.275 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:10:51.283 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:10:51.285 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:10:51.285 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:10:51.286 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:10:51.286 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:10:51.287 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:10:51.287 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:10:51.288 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:10:51.288 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:10:51.289 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:10:51.290 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:10:51.290 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:10:51.291 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:10:51.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:10:51.292 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:10:51.299 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:10:51.300 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:10:51.300 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:10:51.300 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:10:51.300 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:10:51.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff22c0539b0 msgr2=0x7ff22c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:51.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff22c0539b0 0x7ff22c055e70 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ff23801c000 tx=0x7ff238015000).stop 2022-07-20T03:10:51.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 msgr2=0x7ff2480b2a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff2480b2a40 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7ff248146740 tx=0x7ff230007f30).stop 2022-07-20T03:10:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 shutdown_connections 2022-07-20T03:10:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff22c0539b0 0x7ff22c055e70 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff24812aad0 0x7ff248130d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2480b7b70 0x7ff2480b2a40 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 --2- 172.21.15.61:0/3057697515 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2480b1520 0x7ff24812c870 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 >> 172.21.15.61:0/3057697515 conn(0x7ff2480656b0 msgr2=0x7ff248128730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 shutdown_connections 2022-07-20T03:10:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:48.183+0000 7ff24759e700 1 -- 172.21.15.61:0/3057697515 wait complete. 2022-07-20T03:10:51.307 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:10:51.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.415+0000 7f2306cd9700 1 Processor -- start 2022-07-20T03:10:51.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.415+0000 7f2306cd9700 1 -- start start 2022-07-20T03:10:51.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2306cd9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 0x7f23000b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:51.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2306cd9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f23001433d0 con 0x7f23000b0570 2022-07-20T03:10:51.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2306cd9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2300143670 con 0x7f23000aea10 2022-07-20T03:10:51.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2306cd9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300143b10 con 0x7f23001424e0 2022-07-20T03:10:51.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2305cd7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f23000b0570 0x7f2300141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41408/0 (socket says 172.21.15.61:41408) 2022-07-20T03:10:51.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2305cd7700 1 -- 172.21.15.61:0/926604985 learned_addr learned my addr 172.21.15.61:0/926604985 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:10:51.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 0x7f23000b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3533376161 0 0) 0x7f23001433d0 con 0x7f23000b0570 2022-07-20T03:10:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f22e8003260 con 0x7f23000b0570 2022-07-20T03:10:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1440092050 0 0) 0x7f2300143670 con 0x7f23000aea10 2022-07-20T03:10:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f23001433d0 con 0x7f23000aea10 2022-07-20T03:10:51.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2470629254 0 0) 0x7f22e8003260 con 0x7f23000b0570 2022-07-20T03:10:51.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 msgr2=0x7f23000b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:51.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 0x7f23000b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 >> v1:172.21.15.117:6789/0 conn(0x7f23000aea10 legacy=0x7f23000ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:51.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23000b1010 con 0x7f23000b0570 2022-07-20T03:10:51.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f22fc001660 con 0x7f23000b0570 2022-07-20T03:10:51.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 >> v1:172.21.15.61:6789/0 conn(0x7f23000b0570 legacy=0x7f2300141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:51.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e8003910 0x7f22e8003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:51.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22e8004500 0x7f22e8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:51.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 0x7f22e8004e90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:51.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2300143b10 con 0x7f22e8004500 2022-07-20T03:10:51.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22e800b800 con 0x7f22e8003910 2022-07-20T03:10:51.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22e800b990 con 0x7f23001424e0 2022-07-20T03:10:51.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22e8004500 0x7f22e8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:51.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2305cd7700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e8003910 0x7f22e8003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:51.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23064d8700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 0x7f22e8004e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:51.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 -- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 msgr2=0x7f22e8004e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:51.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23001424e0 0x7f22e8004e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 -- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e8003910 msgr2=0x7f22e8003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:51.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22e8003910 0x7f22e8003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:51.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 -- 172.21.15.61:0/926604985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22e8003260 con 0x7f22e8004500 2022-07-20T03:10:51.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f23054d6700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22e8004500 0x7f22e8004980 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f22e80054d0 tx=0x7f22f0006df0).ready entity=mon.0 client_cookie=2c7ad5feb703c7fd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:51.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.419+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22f000e830 con 0x7f22e8004500 2022-07-20T03:10:51.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22f000be90 con 0x7f22e8004500 2022-07-20T03:10:51.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2304cd5700 1 -- 172.21.15.61:0/926604985 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22f000eb80 con 0x7f22e8004500 2022-07-20T03:10:51.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- 172.21.15.61:0/926604985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22e8004500 msgr2=0x7f22e8004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:51.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 --2- 172.21.15.61:0/926604985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22e8004500 0x7f22e8004980 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f22e80054d0 tx=0x7f22f0006df0).stop 2022-07-20T03:10:51.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f23064d8700 1 -- 172.21.15.61:0/926604985 reap_dead start 2022-07-20T03:10:51.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- 172.21.15.61:0/926604985 shutdown_connections 2022-07-20T03:10:51.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- 172.21.15.61:0/926604985 >> 172.21.15.61:0/926604985 conn(0x7f23000b9360 msgr2=0x7f23000b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:51.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- 172.21.15.61:0/926604985 shutdown_connections 2022-07-20T03:10:51.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- 172.21.15.61:0/926604985 wait complete. 2022-07-20T03:10:51.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 Processor -- start 2022-07-20T03:10:51.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- start start 2022-07-20T03:10:51.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 0x7f2300112930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:51.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23000b4ee0 0x7f23001527f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23000b3300 0x7f23000b2240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f230013e7a0 con 0x7f2300112550 2022-07-20T03:10:52.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23000b3bd0 con 0x7f23000b4ee0 2022-07-20T03:10:52.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2306cd9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2300152ce0 con 0x7f23000b3300 2022-07-20T03:10:52.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2305cd7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 0x7f2300112930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2305cd7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 0x7f2300112930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44822/0 (socket says 172.21.15.61:44822) 2022-07-20T03:10:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f2305cd7700 1 -- 172.21.15.61:0/3661066627 learned_addr learned my addr 172.21.15.61:0/3661066627 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f23054d6700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23000b4ee0 0x7f23001527f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.423+0000 7f23064d8700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23000b3300 0x7f23000b2240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2305cd7700 1 -- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23000b3300 msgr2=0x7f23000b2240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2305cd7700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23000b3300 0x7f23000b2240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2305cd7700 1 -- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23000b4ee0 msgr2=0x7f23001527f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2305cd7700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23000b4ee0 0x7f23001527f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2305cd7700 1 -- 172.21.15.61:0/3661066627 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2300151e10 con 0x7f2300112550 2022-07-20T03:10:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2305cd7700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 0x7f2300112930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f2300152e70 tx=0x7f22fc001d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22fc0116b0 con 0x7f2300112550 2022-07-20T03:10:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22fc011810 con 0x7f2300112550 2022-07-20T03:10:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f23001520b0 con 0x7f2300112550 2022-07-20T03:10:52.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22fc011b90 con 0x7f2300112550 2022-07-20T03:10:52.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f23000b2730 con 0x7f2300112550 2022-07-20T03:10:52.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23000574e0 con 0x7f2300112550 2022-07-20T03:10:52.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f22fc01b030 con 0x7f2300112550 2022-07-20T03:10:52.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f22eeffd700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22e0053a40 0x7f22e0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f22fc06b7c0 con 0x7f2300112550 2022-07-20T03:10:52.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.427+0000 7f23054d6700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22e0053a40 0x7f22e0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.431+0000 7f23054d6700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22e0053a40 0x7f22e0055f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f22f0007000 tx=0x7f22f0019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:52.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.435+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f22fc042770 con 0x7f2300112550 2022-07-20T03:10:52.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.599+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f230005d470 con 0x7f2300112550 2022-07-20T03:10:52.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.599+0000 7f22eeffd700 1 -- 172.21.15.61:0/3661066627 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 20 v20) v1 ==== 94+0+2138 (secure 0 0 0) 0x7f22fc019020 con 0x7f2300112550 2022-07-20T03:10:52.098 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:10:52.098 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":20,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":20,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:09:28.502169+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5268},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5268":{"gid":5268,"name":"b","rank":0,"incarnation":19,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/844334511","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":844334511},{"type":"v1","addr":"172.21.15.117:6835","nonce":844334511}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:10:52.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22e0053a40 msgr2=0x7f22e0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22e0053a40 0x7f22e0055f00 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f22f0007000 tx=0x7f22f0019000).stop 2022-07-20T03:10:52.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 msgr2=0x7f2300112930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 0x7f2300112930 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f2300152e70 tx=0x7f22fc001d00).stop 2022-07-20T03:10:52.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 shutdown_connections 2022-07-20T03:10:52.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22e0053a40 0x7f22e0055f00 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23000b4ee0 0x7f23001527f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23000b3300 0x7f23000b2240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 --2- 172.21.15.61:0/3661066627 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2300112550 0x7f2300112930 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 >> 172.21.15.61:0/3661066627 conn(0x7f23000b9360 msgr2=0x7f23001411a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:52.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 shutdown_connections 2022-07-20T03:10:52.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:51.603+0000 7f2306cd9700 1 -- 172.21.15.61:0/3661066627 wait complete. 2022-07-20T03:10:52.113 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 20 2022-07-20T03:10:52.113 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:10:52.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.243+0000 7fe600903700 1 Processor -- start 2022-07-20T03:10:52.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- start start 2022-07-20T03:10:52.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5fc0b3560 0x7fe5fc05fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc144920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe5fc144e10 0x7fe5fc0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc141f80 con 0x7fe5fc0b3560 2022-07-20T03:10:52.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5fc144250 con 0x7fe5fc144e10 2022-07-20T03:10:52.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0b3940 con 0x7fe5fc098b80 2022-07-20T03:10:52.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5fc0b3560 0x7fe5fc05fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5fc0b3560 0x7fe5fc05fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44830/0 (socket says 172.21.15.61:44830) 2022-07-20T03:10:52.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 -- 172.21.15.61:0/3633737975 learned_addr learned my addr 172.21.15.61:0/3633737975 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:52.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fb59e700 1 --2- 172.21.15.61:0/3633737975 >> v2:172.21.15.117:3300/0 conn(0x7fe5fc144e10 0x7fe5fc0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fa59c700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc144920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 -- 172.21.15.61:0/3633737975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 msgr2=0x7fe5fc144920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:52.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc144920 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 -- 172.21.15.61:0/3633737975 >> v2:172.21.15.117:3300/0 conn(0x7fe5fc144e10 msgr2=0x7fe5fc0b1a20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:52.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 --2- 172.21.15.61:0/3633737975 >> v2:172.21.15.117:3300/0 conn(0x7fe5fc144e10 0x7fe5fc0b1a20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 -- 172.21.15.61:0/3633737975 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc09a170 con 0x7fe5fc0b3560 2022-07-20T03:10:52.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fb59e700 1 --2- 172.21.15.61:0/3633737975 >> v2:172.21.15.117:3300/0 conn(0x7fe5fc144e10 0x7fe5fc0b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:10:52.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fa59c700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:10:52.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5fad9d700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc05fa50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe5f0002f70 tx=0x7fe5f0006eb0).ready entity=mon.0 client_cookie=e857d34c1a86b86e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:52.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5f9d9b700 1 -- 172.21.15.61:0/3633737975 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f000d570 con 0x7fe5fc0b3560 2022-07-20T03:10:52.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5f9d9b700 1 -- 172.21.15.61:0/3633737975 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5f0000e90 con 0x7fe5fc0b3560 2022-07-20T03:10:52.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe5f9d9b700 1 -- 172.21.15.61:0/3633737975 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5f000d900 con 0x7fe5fc0b3560 2022-07-20T03:10:52.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- 172.21.15.61:0/3633737975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 msgr2=0x7fe5fc05fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc05fa50 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fe5f0002f70 tx=0x7fe5f0006eb0).stop 2022-07-20T03:10:52.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- 172.21.15.61:0/3633737975 shutdown_connections 2022-07-20T03:10:52.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- 172.21.15.61:0/3633737975 >> v2:172.21.15.117:3300/0 conn(0x7fe5fc144e10 0x7fe5fc0b1a20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc144920 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 --2- 172.21.15.61:0/3633737975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc05fa50 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- 172.21.15.61:0/3633737975 >> 172.21.15.61:0/3633737975 conn(0x7fe5fc0b6b20 msgr2=0x7fe5fc0b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:52.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- 172.21.15.61:0/3633737975 shutdown_connections 2022-07-20T03:10:52.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.247+0000 7fe600903700 1 -- 172.21.15.61:0/3633737975 wait complete. 2022-07-20T03:10:52.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 Processor -- start 2022-07-20T03:10:52.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 -- start start 2022-07-20T03:10:52.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc153330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5fc144e10 0x7fe5fc0aff10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc066a30 con 0x7fe5fc0b3560 2022-07-20T03:10:52.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0ae500 con 0x7fe5fc144e10 2022-07-20T03:10:52.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5fc0afab0 con 0x7fe5fc098b80 2022-07-20T03:10:52.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc153330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc153330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44836/0 (socket says 172.21.15.61:44836) 2022-07-20T03:10:52.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 -- 172.21.15.61:0/1484060156 learned_addr learned my addr 172.21.15.61:0/1484060156 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:52.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fad9d700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fb59e700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5fc144e10 0x7fe5fc0aff10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 -- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 msgr2=0x7fe5fc143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 -- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5fc144e10 msgr2=0x7fe5fc0aff10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:52.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5fc144e10 0x7fe5fc0aff10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:52.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 -- 172.21.15.61:0/1484060156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5f000b1e0 con 0x7fe5fc0b3560 2022-07-20T03:10:52.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fb59e700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5fc144e10 0x7fe5fc0aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:10:52.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5fa59c700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc153330 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe5ec007000 tx=0x7fe5ec008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:52.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5ec010580 con 0x7fe5fc0b3560 2022-07-20T03:10:52.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5ec0106e0 con 0x7fe5fc0b3560 2022-07-20T03:10:52.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5fc066790 con 0x7fe5fc0b3560 2022-07-20T03:10:52.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5ec010a40 con 0x7fe5fc0b3560 2022-07-20T03:10:52.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.251+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5fc153a40 con 0x7fe5fc0b3560 2022-07-20T03:10:52.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.255+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5fc144480 con 0x7fe5fc0b3560 2022-07-20T03:10:52.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.255+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe5ec010ba0 con 0x7fe5fc0b3560 2022-07-20T03:10:52.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.255+0000 7fe5ebfff700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5d8053a40 0x7fe5d8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:52.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.255+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(42..42 src has 1..42) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe5ec06bec0 con 0x7fe5fc0b3560 2022-07-20T03:10:52.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.255+0000 7fe5fad9d700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5d8053a40 0x7fe5d8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:52.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.255+0000 7fe5fad9d700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5d8053a40 0x7fe5d8055f00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe5f001a000 tx=0x7fe5f0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:52.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.263+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe5ec042cb0 con 0x7fe5fc0b3560 2022-07-20T03:10:52.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:52.431+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fe5fc0574e0 con 0x7fe5fc0b3560 2022-07-20T03:10:53.159 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:10:53.155+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:10:53.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe5ebfff700 1 -- 172.21.15.61:0/1484060156 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v21) v1 ==== 161+0+0 (secure 0 0 0) 0x7fe5ec0462d0 con 0x7fe5fc0b3560 2022-07-20T03:10:53.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5d8053a40 msgr2=0x7fe5d8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5d8053a40 0x7fe5d8055f00 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fe5f001a000 tx=0x7fe5f0015000).stop 2022-07-20T03:10:53.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 msgr2=0x7fe5fc153330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc153330 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fe5ec007000 tx=0x7fe5ec008040).stop 2022-07-20T03:10:53.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 shutdown_connections 2022-07-20T03:10:53.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5d8053a40 0x7fe5d8055f00 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5fc144e10 0x7fe5fc0aff10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5fc0b3560 0x7fe5fc153330 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.171+0000 7fe600903700 1 --2- 172.21.15.61:0/1484060156 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5fc098b80 0x7fe5fc143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.175+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 >> 172.21.15.61:0/1484060156 conn(0x7fe5fc0b6b20 msgr2=0x7fe5fc0b2a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:53.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.175+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 shutdown_connections 2022-07-20T03:10:53.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.175+0000 7fe600903700 1 -- 172.21.15.61:0/1484060156 wait complete. 2022-07-20T03:10:53.255 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:10:53.255 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:10:53.259 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:10:53.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.367+0000 7f06822a2700 1 Processor -- start 2022-07-20T03:10:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- start start 2022-07-20T03:10:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f067c0b1e00 0x7f067c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f067c05c3f0 0x7f067c0ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c05bd90 0x7f067c0accd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c05d480 con 0x7f067c05c3f0 2022-07-20T03:10:53.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f067c0ad1c0 con 0x7f067c0b1e00 2022-07-20T03:10:53.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c0b21e0 con 0x7f067c05bd90 2022-07-20T03:10:53.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f067c05c3f0 0x7f067c0ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f067c05c3f0 0x7f067c0ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44842/0 (socket says 172.21.15.61:44842) 2022-07-20T03:10:53.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 -- 172.21.15.61:0/1355410436 learned_addr learned my addr 172.21.15.61:0/1355410436 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f068083f700 1 --2- 172.21.15.61:0/1355410436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c05bd90 0x7f067c0accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067bfff700 1 --2- 172.21.15.61:0/1355410436 >> v2:172.21.15.117:3300/0 conn(0x7f067c0b1e00 0x7f067c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 -- 172.21.15.61:0/1355410436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c05bd90 msgr2=0x7f067c0accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 --2- 172.21.15.61:0/1355410436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c05bd90 0x7f067c0accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 -- 172.21.15.61:0/1355410436 >> v2:172.21.15.117:3300/0 conn(0x7f067c0b1e00 msgr2=0x7f067c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:53.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 --2- 172.21.15.61:0/1355410436 >> v2:172.21.15.117:3300/0 conn(0x7f067c0b1e00 0x7f067c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 -- 172.21.15.61:0/1355410436 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f067c05cf40 con 0x7f067c05c3f0 2022-07-20T03:10:53.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067bfff700 1 --2- 172.21.15.61:0/1355410436 >> v2:172.21.15.117:3300/0 conn(0x7f067c0b1e00 0x7f067c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:10:53.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067b7fe700 1 --2- 172.21.15.61:0/1355410436 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c0ac7e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f066c002f70 tx=0x7f066c006eb0).ready entity=mon.0 client_cookie=b56603162313ed9e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067affd700 1 -- 172.21.15.61:0/1355410436 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f066c00d570 con 0x7f067c05c3f0 2022-07-20T03:10:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067affd700 1 -- 172.21.15.61:0/1355410436 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f066c000e90 con 0x7f067c05c3f0 2022-07-20T03:10:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f067affd700 1 -- 172.21.15.61:0/1355410436 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f066c00d900 con 0x7f067c05c3f0 2022-07-20T03:10:53.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- 172.21.15.61:0/1355410436 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 msgr2=0x7f067c0ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- 172.21.15.61:0/1355410436 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c0ac7e0 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f066c002f70 tx=0x7f066c006eb0).stop 2022-07-20T03:10:53.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- 172.21.15.61:0/1355410436 shutdown_connections 2022-07-20T03:10:53.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- 172.21.15.61:0/1355410436 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c05bd90 0x7f067c0accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- 172.21.15.61:0/1355410436 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c0ac7e0 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 --2- 172.21.15.61:0/1355410436 >> v2:172.21.15.117:3300/0 conn(0x7f067c0b1e00 0x7f067c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- 172.21.15.61:0/1355410436 >> 172.21.15.61:0/1355410436 conn(0x7f067c0611c0 msgr2=0x7f067c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:53.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- 172.21.15.61:0/1355410436 shutdown_connections 2022-07-20T03:10:53.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.371+0000 7f06822a2700 1 -- 172.21.15.61:0/1355410436 wait complete. 2022-07-20T03:10:53.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 Processor -- start 2022-07-20T03:10:53.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 -- start start 2022-07-20T03:10:53.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c147950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f067c0b1e00 0x7f067c156eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c0b5100 0x7f067c09e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c156550 con 0x7f067c05c3f0 2022-07-20T03:10:53.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f067c156770 con 0x7f067c0b1e00 2022-07-20T03:10:53.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f067c156a10 con 0x7f067c0b5100 2022-07-20T03:10:53.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c147950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c147950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44848/0 (socket says 172.21.15.61:44848) 2022-07-20T03:10:53.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f067c0b1e00 0x7f067c156eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 -- 172.21.15.61:0/1383488951 learned_addr learned my addr 172.21.15.61:0/1383488951 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:53.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f068083f700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c0b5100 0x7f067c09e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 -- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c0b5100 msgr2=0x7f067c09e080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c0b5100 0x7f067c09e080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 -- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f067c0b1e00 msgr2=0x7f067c156eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f067c0b1e00 0x7f067c156eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 -- 172.21.15.61:0/1383488951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f066c00b1e0 con 0x7f067c05c3f0 2022-07-20T03:10:53.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067b7fe700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f067c0b1e00 0x7f067c156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:10:53.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f067bfff700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c147950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0670007000 tx=0x7f0670008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:53.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f067000ce80 con 0x7f067c05c3f0 2022-07-20T03:10:53.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0670010580 con 0x7f067c05c3f0 2022-07-20T03:10:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06700108e0 con 0x7f067c05c3f0 2022-07-20T03:10:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.375+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f067c0b4d00 con 0x7f067c05c3f0 2022-07-20T03:10:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f067c1573a0 con 0x7f067c05c3f0 2022-07-20T03:10:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f067c0b4e90 con 0x7f067c05c3f0 2022-07-20T03:10:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0670010a40 con 0x7f067c05c3f0 2022-07-20T03:10:53.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f06797fa700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f065c053a40 0x7f065c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f067006bbc0 con 0x7f067c05c3f0 2022-07-20T03:10:53.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f067b7fe700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f065c053a40 0x7f065c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:53.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.379+0000 7f067b7fe700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f065c053a40 0x7f065c055f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f066c01c000 tx=0x7f066c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:53.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.383+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0670042ad0 con 0x7f067c05c3f0 2022-07-20T03:10:53.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f067c05d1e0 con 0x7f067c05c3f0 2022-07-20T03:10:53.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06797fa700 1 -- 172.21.15.61:0/1383488951 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 21 v21) v1 ==== 94+0+1777 (secure 0 0 0) 0x7f0670020250 con 0x7f067c05c3f0 2022-07-20T03:10:53.662 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:10:53.664 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":21,"default_fscid":3,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18}],"filesystems":[{"mdsmap":{"epoch":21,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:09:27.499383+0000","modified":"2022-07-20T03:10:53.160942+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":43,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[8],"metadata_pool":7,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":3}]} 2022-07-20T03:10:53.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f065c053a40 msgr2=0x7f065c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f065c053a40 0x7f065c055f00 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f066c01c000 tx=0x7f066c015000).stop 2022-07-20T03:10:53.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 msgr2=0x7f067c147950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:53.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c147950 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f0670007000 tx=0x7f0670008040).stop 2022-07-20T03:10:53.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 shutdown_connections 2022-07-20T03:10:53.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f065c053a40 0x7f065c055f00 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f067c0b1e00 0x7f067c156eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f067c0b5100 0x7f067c09e080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 --2- 172.21.15.61:0/1383488951 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f067c05c3f0 0x7f067c147950 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:53.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 >> 172.21.15.61:0/1383488951 conn(0x7f067c05bd90 msgr2=0x7f067c0ac700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:53.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.551+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 shutdown_connections 2022-07-20T03:10:53.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.555+0000 7f06822a2700 1 -- 172.21.15.61:0/1383488951 wait complete. 2022-07-20T03:10:53.775 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 21 2022-07-20T03:10:53.776 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:10:53.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.887+0000 7f9f2722a700 1 Processor -- start 2022-07-20T03:10:53.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.887+0000 7f9f2722a700 1 -- start start 2022-07-20T03:10:53.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f200b3cb0 0x7f9f200aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:53.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200ae280 con 0x7f9f200b6800 2022-07-20T03:10:53.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f200b6be0 con 0x7f9f200ad0e0 2022-07-20T03:10:53.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f200ae4b0 con 0x7f9f200b3cb0 2022-07-20T03:10:53.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f200b3cb0 0x7f9f200aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f200b3cb0 0x7f9f200aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44858/0 (socket says 172.21.15.61:44858) 2022-07-20T03:10:54.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 -- 172.21.15.61:0/1482418867 learned_addr learned my addr 172.21.15.61:0/1482418867 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:54.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f26228700 1 --1- 172.21.15.61:0/1482418867 >> v1:172.21.15.61:6789/0 conn(0x7f9f200b6800 0x7f9f20145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41454/0 (socket says 172.21.15.61:41454) 2022-07-20T03:10:54.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 -- 172.21.15.61:0/1482418867 >> v1:172.21.15.117:6789/0 conn(0x7f9f200ad0e0 legacy=0x7f9f201418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:54.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 -- 172.21.15.61:0/1482418867 >> v1:172.21.15.61:6789/0 conn(0x7f9f200b6800 legacy=0x7f9f20145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 -- 172.21.15.61:0/1482418867 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f200ae750 con 0x7f9f200b3cb0 2022-07-20T03:10:54.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25a27700 1 --2- 172.21.15.61:0/1482418867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b3cb0 0x7f9f200aebf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f9f200b3a30 tx=0x7f9f14006eb0).ready entity=mon.0 client_cookie=42a60174b6bc5423 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:54.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25226700 1 -- 172.21.15.61:0/1482418867 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f1400d610 con 0x7f9f200b3cb0 2022-07-20T03:10:54.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25226700 1 -- 172.21.15.61:0/1482418867 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f14000e90 con 0x7f9f200b3cb0 2022-07-20T03:10:54.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f25226700 1 -- 172.21.15.61:0/1482418867 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f1400d960 con 0x7f9f200b3cb0 2022-07-20T03:10:54.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- 172.21.15.61:0/1482418867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b3cb0 msgr2=0x7f9f200aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 --2- 172.21.15.61:0/1482418867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b3cb0 0x7f9f200aebf0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f9f200b3a30 tx=0x7f9f14006eb0).stop 2022-07-20T03:10:54.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- 172.21.15.61:0/1482418867 shutdown_connections 2022-07-20T03:10:54.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 --2- 172.21.15.61:0/1482418867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b3cb0 0x7f9f200aebf0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- 172.21.15.61:0/1482418867 >> 172.21.15.61:0/1482418867 conn(0x7f9f200b9dd0 msgr2=0x7f9f200b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:54.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- 172.21.15.61:0/1482418867 shutdown_connections 2022-07-20T03:10:54.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- 172.21.15.61:0/1482418867 wait complete. 2022-07-20T03:10:54.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 Processor -- start 2022-07-20T03:10:54.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.891+0000 7f9f2722a700 1 -- start start 2022-07-20T03:10:54.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f200ad0e0 0x7f9f200b2a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:54.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f200b3cb0 0x7f9f20112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:54.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 0x7f9f20113180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:54.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20115fa0 con 0x7f9f200b6800 2022-07-20T03:10:54.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f201139e0 con 0x7f9f200ad0e0 2022-07-20T03:10:54.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f20113c80 con 0x7f9f200b3cb0 2022-07-20T03:10:54.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 0x7f9f20113180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26228700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f200ad0e0 0x7f9f200b2a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f25a27700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f200b3cb0 0x7f9f20112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26228700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f200ad0e0 0x7f9f200b2a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54542/0 (socket says 172.21.15.61:54542) 2022-07-20T03:10:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26228700 1 -- 172.21.15.61:0/2397279947 learned_addr learned my addr 172.21.15.61:0/2397279947 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:54.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 0x7f9f20113180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44866/0 (socket says 172.21.15.61:44866) 2022-07-20T03:10:54.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 -- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f200b3cb0 msgr2=0x7f9f20112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f200b3cb0 0x7f9f20112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 -- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f200ad0e0 msgr2=0x7f9f200b2a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f200ad0e0 0x7f9f200b2a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 -- 172.21.15.61:0/2397279947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f1400b1e0 con 0x7f9f200b6800 2022-07-20T03:10:54.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f26a29700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 0x7f9f20113180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f9f18002320 tx=0x7f9f18007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:54.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f1800ce60 con 0x7f9f200b6800 2022-07-20T03:10:54.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f18006aa0 con 0x7f9f200b6800 2022-07-20T03:10:54.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f20113670 con 0x7f9f200b6800 2022-07-20T03:10:54.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f18006e00 con 0x7f9f200b6800 2022-07-20T03:10:54.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f20115bb0 con 0x7f9f200b6800 2022-07-20T03:10:54.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.895+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f20113800 con 0x7f9f200b6800 2022-07-20T03:10:54.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.899+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f18010030 con 0x7f9f200b6800 2022-07-20T03:10:54.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.899+0000 7f9f137fe700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f00053a40 0x7f9f00055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:54.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.899+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9f1806bd90 con 0x7f9f200b6800 2022-07-20T03:10:54.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.899+0000 7f9f26228700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f00053a40 0x7f9f00055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.899+0000 7f9f26228700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f00053a40 0x7f9f00055f00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9f1c00d000 tx=0x7f9f1c007c40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:54.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:53.907+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f18042c00 con 0x7f9f200b6800 2022-07-20T03:10:54.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.071+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9f200b5530 con 0x7f9f200b6800 2022-07-20T03:10:54.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.443+0000 7f9f137fe700 1 -- 172.21.15.61:0/2397279947 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v22) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9f18046220 con 0x7f9f200b6800 2022-07-20T03:10:54.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f00053a40 msgr2=0x7f9f00055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f00053a40 0x7f9f00055f00 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9f1c00d000 tx=0x7f9f1c007c40).stop 2022-07-20T03:10:54.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 msgr2=0x7f9f20113180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 0x7f9f20113180 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f9f18002320 tx=0x7f9f18007ef0).stop 2022-07-20T03:10:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 shutdown_connections 2022-07-20T03:10:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f00053a40 0x7f9f00055f00 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f200b6800 0x7f9f20113180 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f200b3cb0 0x7f9f20112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 --2- 172.21.15.61:0/2397279947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f200ad0e0 0x7f9f200b2a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 >> 172.21.15.61:0/2397279947 conn(0x7f9f200b9dd0 msgr2=0x7f9f20145e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 shutdown_connections 2022-07-20T03:10:54.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.447+0000 7f9f2722a700 1 -- 172.21.15.61:0/2397279947 wait complete. 2022-07-20T03:10:54.551 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:10:54.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 Processor -- start 2022-07-20T03:10:54.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 -- start start 2022-07-20T03:10:54.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b6840 0x7f77e40b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:54.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f77e40b4bc0 0x7f77e4141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77e4066b10 con 0x7f77e40b3cf0 2022-07-20T03:10:54.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77e40b6c20 con 0x7f77e40b4bc0 2022-07-20T03:10:54.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40aead0 con 0x7f77e40b6840 2022-07-20T03:10:54.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e27fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f77e40b3cf0 0x7f77e4146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41470/0 (socket says 172.21.15.61:41470) 2022-07-20T03:10:54.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e27fc700 1 -- 172.21.15.61:0/54101619 learned_addr learned my addr 172.21.15.61:0/54101619 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:10:54.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e2ffd700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b6840 0x7f77e40b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.703+0000 7f77e37fe700 1 --2- 172.21.15.61:0/54101619 >> v2:172.21.15.117:3300/0 conn(0x7f77e40b4bc0 0x7f77e4141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:54.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e1ffb700 1 -- 172.21.15.61:0/54101619 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2529694787 0 0) 0x7f77e4066b10 con 0x7f77e40b3cf0 2022-07-20T03:10:54.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e1ffb700 1 -- 172.21.15.61:0/54101619 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f77c8003260 con 0x7f77e40b3cf0 2022-07-20T03:10:54.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e37fe700 1 -- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b6840 msgr2=0x7f77e40b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:54.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e37fe700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b6840 0x7f77e40b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:54.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e37fe700 1 -- 172.21.15.61:0/54101619 >> v1:172.21.15.61:6789/0 conn(0x7f77e40b3cf0 legacy=0x7f77e4146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e37fe700 1 -- 172.21.15.61:0/54101619 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77e40aed70 con 0x7f77e40b4bc0 2022-07-20T03:10:54.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e2ffd700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b6840 0x7f77e40b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:10:54.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e37fe700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b4bc0 0x7f77e4141a90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f77dc0032f0 tx=0x7f77dc0091d0).ready entity=mon.1 client_cookie=6d2f58f84a3857ed server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:54.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e1ffb700 1 -- 172.21.15.61:0/54101619 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77dc00b600 con 0x7f77e40b4bc0 2022-07-20T03:10:54.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e1ffb700 1 -- 172.21.15.61:0/54101619 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77dc00b760 con 0x7f77e40b4bc0 2022-07-20T03:10:54.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e1ffb700 1 -- 172.21.15.61:0/54101619 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77dc00b8d0 con 0x7f77e40b4bc0 2022-07-20T03:10:54.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 -- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b4bc0 msgr2=0x7f77e4141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:54.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b4bc0 0x7f77e4141a90 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f77dc0032f0 tx=0x7f77dc0091d0).stop 2022-07-20T03:10:54.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 -- 172.21.15.61:0/54101619 shutdown_connections 2022-07-20T03:10:54.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b4bc0 0x7f77e4141a90 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 --2- 172.21.15.61:0/54101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b6840 0x7f77e40b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 -- 172.21.15.61:0/54101619 >> 172.21.15.61:0/54101619 conn(0x7f77e40b9e10 msgr2=0x7f77e40b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:55.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 -- 172.21.15.61:0/54101619 shutdown_connections 2022-07-20T03:10:55.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 -- 172.21.15.61:0/54101619 wait complete. 2022-07-20T03:10:55.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 Processor -- start 2022-07-20T03:10:55.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.707+0000 7f77e3fff700 1 -- start start 2022-07-20T03:10:55.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 0x7f77e4112ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b4bc0 0x7f77e413e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b6840 0x7f77e413e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40b2d90 con 0x7f77e40b3cf0 2022-07-20T03:10:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40b1c60 con 0x7f77e40b6840 2022-07-20T03:10:55.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77e40b19c0 con 0x7f77e40b4bc0 2022-07-20T03:10:55.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 0x7f77e4112ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 0x7f77e4112ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44876/0 (socket says 172.21.15.61:44876) 2022-07-20T03:10:55.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 -- 172.21.15.61:0/3601960200 learned_addr learned my addr 172.21.15.61:0/3601960200 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e27fc700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b4bc0 0x7f77e413e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e37fe700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b6840 0x7f77e413e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 -- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b4bc0 msgr2=0x7f77e413e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b4bc0 0x7f77e413e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 -- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b6840 msgr2=0x7f77e413e5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b6840 0x7f77e413e5c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 -- 172.21.15.61:0/3601960200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d800b180 con 0x7f77e40b3cf0 2022-07-20T03:10:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e2ffd700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 0x7f77e4112ae0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f77d800c000 tx=0x7f77d8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:55.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77d8010780 con 0x7f77e40b3cf0 2022-07-20T03:10:55.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77d80108e0 con 0x7f77e40b3cf0 2022-07-20T03:10:55.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77d8010c40 con 0x7f77e40b3cf0 2022-07-20T03:10:55.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77dc009470 con 0x7f77e40b3cf0 2022-07-20T03:10:55.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77e413eab0 con 0x7f77e40b3cf0 2022-07-20T03:10:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.711+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77e40b50e0 con 0x7f77e40b3cf0 2022-07-20T03:10:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.715+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f77d801c460 con 0x7f77e40b3cf0 2022-07-20T03:10:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.715+0000 7f77d3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77c0053ad0 0x7f77c0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.715+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(43..43 src has 1..43) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f77d806c4e0 con 0x7f77e40b3cf0 2022-07-20T03:10:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.715+0000 7f77e27fc700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77c0053ad0 0x7f77c0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.715+0000 7f77e27fc700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77c0053ad0 0x7f77c0055f90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f77d400d000 tx=0x7f77d4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:55.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.719+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77d8043200 con 0x7f77e40b3cf0 2022-07-20T03:10:55.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:54.843+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f77e40574e0 con 0x7f77e40b3cf0 2022-07-20T03:10:55.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.447+0000 7f77d3fff700 1 -- 172.21.15.61:0/3601960200 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v44) v1 ==== 185+0+0 (secure 0 0 0) 0x7f77d8046820 con 0x7f77e40b3cf0 2022-07-20T03:10:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77c0053ad0 msgr2=0x7f77c0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77c0053ad0 0x7f77c0055f90 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f77d400d000 tx=0x7f77d4009040).stop 2022-07-20T03:10:55.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 msgr2=0x7f77e4112ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 0x7f77e4112ae0 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f77d800c000 tx=0x7f77d8006df0).stop 2022-07-20T03:10:55.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 shutdown_connections 2022-07-20T03:10:55.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77c0053ad0 0x7f77c0055f90 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77e40b6840 0x7f77e413e5c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77e40b4bc0 0x7f77e413e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 --2- 172.21.15.61:0/3601960200 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77e40b3cf0 0x7f77e4112ae0 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 >> 172.21.15.61:0/3601960200 conn(0x7f77e40b9e10 msgr2=0x7f77e405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:55.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 shutdown_connections 2022-07-20T03:10:55.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.455+0000 7f77e3fff700 1 -- 172.21.15.61:0/3601960200 wait complete. 2022-07-20T03:10:55.585 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:10:55.597 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:10:55.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.719+0000 7fa830e6e700 1 Processor -- start 2022-07-20T03:10:55.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.719+0000 7fa830e6e700 1 -- start start 2022-07-20T03:10:55.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa830e6e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa82c146780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa830e6e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c146c70 con 0x7fa82c0b5990 2022-07-20T03:10:55.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa830e6e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa82c0b5d70 con 0x7fa82c0af3e0 2022-07-20T03:10:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa830e6e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c0ad050 con 0x7fa82c05d210 2022-07-20T03:10:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa82c0b5990 0x7fa82c146070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41482/0 (socket says 172.21.15.61:41482) 2022-07-20T03:10:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82b7fe700 1 -- 172.21.15.61:0/206345567 learned_addr learned my addr 172.21.15.61:0/206345567 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:10:55.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa82c146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3733755716 0 0) 0x7fa82c146c70 con 0x7fa82c0b5990 2022-07-20T03:10:55.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa80c003260 con 0x7fa82c0b5990 2022-07-20T03:10:55.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2012611040 0 0) 0x7fa82c0b5d70 con 0x7fa82c0af3e0 2022-07-20T03:10:55.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa82c146c70 con 0x7fa82c0af3e0 2022-07-20T03:10:55.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 150270284 0 0) 0x7fa80c003260 con 0x7fa82c0b5990 2022-07-20T03:10:55.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 msgr2=0x7fa82c146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:55.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa82c146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 >> v1:172.21.15.117:6789/0 conn(0x7fa82c0af3e0 legacy=0x7fa82c14a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0b1010 con 0x7fa82c0b5990 2022-07-20T03:10:55.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fa81c001b60 con 0x7fa82c0b5990 2022-07-20T03:10:55.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 >> v1:172.21.15.61:6789/0 conn(0x7fa82c0b5990 legacy=0x7fa82c146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa80c003910 0x7fa80c003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa80c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa80c0049f0 0x7fa80c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c00b750 con 0x7fa80c0049f0 2022-07-20T03:10:55.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c00b8c0 con 0x7fa80c003910 2022-07-20T03:10:55.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa80c00ba80 con 0x7fa82c05d210 2022-07-20T03:10:55.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82affd700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa80c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82b7fe700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa80c003910 0x7fa80c003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa80c0049f0 0x7fa80c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:55.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 -- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 msgr2=0x7fa80c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:55.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa80c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 -- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa80c003910 msgr2=0x7fa80c003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:55.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa80c003910 0x7fa80c003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:55.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 -- 172.21.15.61:0/206345567 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa80c003260 con 0x7fa80c0049f0 2022-07-20T03:10:55.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82affd700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c05d210 0x7fa80c004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:10:55.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.723+0000 7fa82bfff700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa80c0049f0 0x7fa80c004e70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa82400c000 tx=0x7fa824006df0).ready entity=mon.0 client_cookie=3b0a694c64e55579 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:55.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa82400e830 con 0x7fa80c0049f0 2022-07-20T03:10:55.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa82400e990 con 0x7fa80c0049f0 2022-07-20T03:10:55.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa82a7fc700 1 -- 172.21.15.61:0/206345567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa82400ecf0 con 0x7fa80c0049f0 2022-07-20T03:10:55.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 -- 172.21.15.61:0/206345567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa80c0049f0 msgr2=0x7fa80c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:55.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 --2- 172.21.15.61:0/206345567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa80c0049f0 0x7fa80c004e70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7fa82400c000 tx=0x7fa824006df0).stop 2022-07-20T03:10:55.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa82bfff700 1 -- 172.21.15.61:0/206345567 reap_dead start 2022-07-20T03:10:55.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 -- 172.21.15.61:0/206345567 shutdown_connections 2022-07-20T03:10:55.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 -- 172.21.15.61:0/206345567 >> 172.21.15.61:0/206345567 conn(0x7fa82c063480 msgr2=0x7fa82c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:55.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 -- 172.21.15.61:0/206345567 shutdown_connections 2022-07-20T03:10:55.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 -- 172.21.15.61:0/206345567 wait complete. 2022-07-20T03:10:55.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 Processor -- start 2022-07-20T03:10:55.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.727+0000 7fa830e6e700 1 -- start start 2022-07-20T03:10:55.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 0x7fa82c149cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c1562f0 0x7fa82c15a050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 0x7fa82c148260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:55.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c14a2f0 con 0x7fa82c15a540 2022-07-20T03:10:55.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c159910 con 0x7fa82c1562f0 2022-07-20T03:10:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa82c159bb0 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 0x7fa82c148260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 0x7fa82c149cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c1562f0 0x7fa82c15a050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 0x7fa82c149cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:59800/0 (socket says 172.21.15.61:59800) 2022-07-20T03:10:56.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 -- 172.21.15.61:0/2320353068 learned_addr learned my addr 172.21.15.61:0/2320353068 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:56.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82bfff700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 0x7fa82c148260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44900/0 (socket says 172.21.15.61:44900) 2022-07-20T03:10:56.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 -- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c1562f0 msgr2=0x7fa82c15a050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:56.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c1562f0 0x7fa82c15a050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:56.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 -- 172.21.15.61:0/2320353068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 msgr2=0x7fa82c148260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:56.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 0x7fa82c148260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:56.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 -- 172.21.15.61:0/2320353068 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa82c0b4280 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82bfff700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 0x7fa82c148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:10:56.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa82b7fe700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 0x7fa82c149cc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa81c00d000 tx=0x7fa81c009040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:56.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa81c005c60 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa81c005dc0 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa82c156750 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa81c0072c0 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa82c148970 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.731+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa82c0b4410 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.735+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa81c01a030 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.735+0000 7fa828ff9700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa810053920 0x7fa810055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.735+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(44..44 src has 1..44) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fa81c06c390 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.735+0000 7fa82affd700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa810053920 0x7fa810055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.735+0000 7fa82affd700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa810053920 0x7fa810055de0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fa814003000 tx=0x7fa814007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:56.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.743+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa81c043450 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:55.867+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa82c0574e0 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.451+0000 7fa828ff9700 1 -- 172.21.15.61:0/2320353068 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v45) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa81c018030 con 0x7fa82c0b3ea0 2022-07-20T03:10:56.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa810053920 msgr2=0x7fa810055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:56.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa810053920 0x7fa810055de0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7fa814003000 tx=0x7fa814007a00).stop 2022-07-20T03:10:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 msgr2=0x7fa82c149cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 0x7fa82c149cc0 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fa81c00d000 tx=0x7fa81c009040).stop 2022-07-20T03:10:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 shutdown_connections 2022-07-20T03:10:56.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa810053920 0x7fa810055de0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:56.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa82c1562f0 0x7fa82c15a050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:56.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa82c15a540 0x7fa82c148260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:56.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 --2- 172.21.15.61:0/2320353068 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa82c0b3ea0 0x7fa82c149cc0 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:56.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 >> 172.21.15.61:0/2320353068 conn(0x7fa82c063480 msgr2=0x7fa82c145e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:56.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 shutdown_connections 2022-07-20T03:10:56.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.455+0000 7fa830e6e700 1 -- 172.21.15.61:0/2320353068 wait complete. 2022-07-20T03:10:56.549 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:10:56.561 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:10:56.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.731+0000 7feffd6e2700 1 Processor -- start 2022-07-20T03:10:56.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.731+0000 7feffd6e2700 1 -- start start 2022-07-20T03:10:56.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7feff80b9ea0 0x7feff80b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:56.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff8142dc0 con 0x7feff80b00f0 2022-07-20T03:10:56.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feff80ba280 con 0x7feff80b3df0 2022-07-20T03:10:56.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feff8146280 con 0x7feff80b9ea0 2022-07-20T03:10:56.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff67fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feff80b00f0 0x7feff8143490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41502/0 (socket says 172.21.15.61:41502) 2022-07-20T03:10:56.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff67fc700 1 -- 172.21.15.61:0/1346298332 learned_addr learned my addr 172.21.15.61:0/1346298332 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:10:56.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff6ffd700 1 --2- 172.21.15.61:0/1346298332 >> v2:172.21.15.117:3300/0 conn(0x7feff80b9ea0 0x7feff80b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:56.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1282166460 0 0) 0x7feff8142dc0 con 0x7feff80b00f0 2022-07-20T03:10:56.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fefd8003260 con 0x7feff80b00f0 2022-07-20T03:10:56.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3704104183 0 0) 0x7feff80ba280 con 0x7feff80b3df0 2022-07-20T03:10:56.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feff8142dc0 con 0x7feff80b3df0 2022-07-20T03:10:57.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff6ffd700 1 -- 172.21.15.61:0/1346298332 >> v1:172.21.15.117:6789/0 conn(0x7feff80b3df0 legacy=0x7feff8141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff6ffd700 1 -- 172.21.15.61:0/1346298332 >> v1:172.21.15.61:6789/0 conn(0x7feff80b00f0 legacy=0x7feff8143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff6ffd700 1 -- 172.21.15.61:0/1346298332 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff8142ff0 con 0x7feff80b9ea0 2022-07-20T03:10:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff6ffd700 1 --2- 172.21.15.61:0/1346298332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff80b4300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fefe0002f70 tx=0x7fefe0006df0).ready entity=mon.1 client_cookie=7007e8a4ee5769db server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe000d540 con 0x7feff80b9ea0 2022-07-20T03:10:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe0000e90 con 0x7feff80b9ea0 2022-07-20T03:10:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feff5ffb700 1 -- 172.21.15.61:0/1346298332 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe000d890 con 0x7feff80b9ea0 2022-07-20T03:10:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 -- 172.21.15.61:0/1346298332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 msgr2=0x7feff80b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 --2- 172.21.15.61:0/1346298332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff80b4300 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7fefe0002f70 tx=0x7fefe0006df0).stop 2022-07-20T03:10:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 -- 172.21.15.61:0/1346298332 shutdown_connections 2022-07-20T03:10:57.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 --2- 172.21.15.61:0/1346298332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff80b4300 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.735+0000 7feffd6e2700 1 -- 172.21.15.61:0/1346298332 >> 172.21.15.61:0/1346298332 conn(0x7feff8067960 msgr2=0x7feff80b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:57.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 -- 172.21.15.61:0/1346298332 shutdown_connections 2022-07-20T03:10:57.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 -- 172.21.15.61:0/1346298332 wait complete. 2022-07-20T03:10:57.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 Processor -- start 2022-07-20T03:10:57.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 -- start start 2022-07-20T03:10:57.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feff80b00f0 0x7feff80b2e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 0x7feff80af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff813f000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff81142a0 con 0x7feff80b3df0 2022-07-20T03:10:57.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feff8155510 con 0x7feff80b9ea0 2022-07-20T03:10:57.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feffd6e2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feff80adfd0 con 0x7feff80b00f0 2022-07-20T03:10:57.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feff67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 0x7feff80af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feff67fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 0x7feff80af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44912/0 (socket says 172.21.15.61:44912) 2022-07-20T03:10:57.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feff67fc700 1 -- 172.21.15.61:0/2455581418 learned_addr learned my addr 172.21.15.61:0/2455581418 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:57.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feff77fe700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff813f000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.739+0000 7feff6ffd700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feff80b00f0 0x7feff80b2e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff67fc700 1 -- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feff80b00f0 msgr2=0x7feff80b2e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff67fc700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feff80b00f0 0x7feff80b2e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff67fc700 1 -- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 msgr2=0x7feff813f000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff67fc700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff813f000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff67fc700 1 -- 172.21.15.61:0/2455581418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fefe000b180 con 0x7feff80b3df0 2022-07-20T03:10:57.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff6ffd700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feff80b00f0 0x7feff80b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:10:57.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff77fe700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff813f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:10:57.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feff67fc700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 0x7feff80af520 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fefe80011d0 tx=0x7fefe8007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:57.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe800ce70 con 0x7feff80b3df0 2022-07-20T03:10:57.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fefe80072e0 con 0x7feff80b3df0 2022-07-20T03:10:57.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feff80aeb80 con 0x7feff80b3df0 2022-07-20T03:10:57.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fefe8005800 con 0x7feff80b3df0 2022-07-20T03:10:57.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feff80b3330 con 0x7feff80b3df0 2022-07-20T03:10:57.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feff80574e0 con 0x7feff80b3df0 2022-07-20T03:10:57.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fefe8010070 con 0x7feff80b3df0 2022-07-20T03:10:57.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.743+0000 7fefeffff700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fefdc053ad0 0x7fefdc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.747+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fefe806b420 con 0x7feff80b3df0 2022-07-20T03:10:57.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.747+0000 7feff6ffd700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fefdc053ad0 0x7fefdc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.747+0000 7feff6ffd700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fefdc053ad0 0x7fefdc055f90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fefe0018000 tx=0x7fefe0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:57.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.755+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fefe80425c0 con 0x7feff80b3df0 2022-07-20T03:10:57.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.879+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7feff8067ea0 con 0x7feff80b3df0 2022-07-20T03:10:57.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7fefeffff700 1 -- 172.21.15.61:0/2455581418 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v5) v1 ==== 104+0+0 (secure 0 0 0) 0x7fefe8045be0 con 0x7feff80b3df0 2022-07-20T03:10:57.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fefdc053ad0 msgr2=0x7fefdc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fefdc053ad0 0x7fefdc055f90 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7fefe0018000 tx=0x7fefe0014000).stop 2022-07-20T03:10:57.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 msgr2=0x7feff80af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 0x7feff80af520 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fefe80011d0 tx=0x7fefe8007b50).stop 2022-07-20T03:10:57.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 shutdown_connections 2022-07-20T03:10:57.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fefdc053ad0 0x7fefdc055f90 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feff80b9ea0 0x7feff813f000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feff80b3df0 0x7feff80af520 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 --2- 172.21.15.61:0/2455581418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feff80b00f0 0x7feff80b2e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 >> 172.21.15.61:0/2455581418 conn(0x7feff8067960 msgr2=0x7feff8140ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:57.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 shutdown_connections 2022-07-20T03:10:57.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:56.887+0000 7feffd6e2700 1 -- 172.21.15.61:0/2455581418 wait complete. 2022-07-20T03:10:57.259 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead' 2022-07-20T03:10:57.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.371+0000 7face76dd700 1 Processor -- start 2022-07-20T03:10:57.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.371+0000 7face76dd700 1 -- start start 2022-07-20T03:10:57.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7face00aedc0 0x7face013f2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00acdb0 con 0x7face00b0080 2022-07-20T03:10:57.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7face00b1450 con 0x7face00ac9d0 2022-07-20T03:10:57.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7face00af260 con 0x7face00aedc0 2022-07-20T03:10:57.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face5c7a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7face00b0080 0x7face013f790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41520/0 (socket says 172.21.15.61:41520) 2022-07-20T03:10:57.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face5c7a700 1 -- 172.21.15.61:0/259483437 learned_addr learned my addr 172.21.15.61:0/259483437 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:10:57.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face4c78700 1 --2- 172.21.15.61:0/259483437 >> v2:172.21.15.117:3300/0 conn(0x7face00aedc0 0x7face013f2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7facd3fff700 1 -- 172.21.15.61:0/259483437 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2449347922 0 0) 0x7face00acdb0 con 0x7face00b0080 2022-07-20T03:10:57.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7facd3fff700 1 -- 172.21.15.61:0/259483437 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7facc8003260 con 0x7face00b0080 2022-07-20T03:10:57.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face4c78700 1 -- 172.21.15.61:0/259483437 >> v1:172.21.15.117:6789/0 conn(0x7face00ac9d0 legacy=0x7face013e780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7facd3fff700 1 -- 172.21.15.61:0/259483437 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388323185 0 0) 0x7face00b1450 con 0x7face00ac9d0 2022-07-20T03:10:57.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face4c78700 1 -- 172.21.15.61:0/259483437 >> v1:172.21.15.61:6789/0 conn(0x7face00b0080 legacy=0x7face013f790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face4c78700 1 -- 172.21.15.61:0/259483437 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7face00af450 con 0x7face00aedc0 2022-07-20T03:10:57.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face4c78700 1 --2- 172.21.15.61:0/259483437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00aedc0 0x7face013f2a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7facdc002f70 tx=0x7facdc006e80).ready entity=mon.1 client_cookie=f0108fcede644738 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:57.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7facd3fff700 1 -- 172.21.15.61:0/259483437 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facdc00f730 con 0x7face00aedc0 2022-07-20T03:10:57.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7facd3fff700 1 -- 172.21.15.61:0/259483437 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc000e90 con 0x7face00aedc0 2022-07-20T03:10:57.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7facd3fff700 1 -- 172.21.15.61:0/259483437 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facdc00fa80 con 0x7face00aedc0 2022-07-20T03:10:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- 172.21.15.61:0/259483437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00aedc0 msgr2=0x7face013f2a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 --2- 172.21.15.61:0/259483437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00aedc0 0x7face013f2a0 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7facdc002f70 tx=0x7facdc006e80).stop 2022-07-20T03:10:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- 172.21.15.61:0/259483437 shutdown_connections 2022-07-20T03:10:57.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 --2- 172.21.15.61:0/259483437 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00aedc0 0x7face013f2a0 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- 172.21.15.61:0/259483437 >> 172.21.15.61:0/259483437 conn(0x7face00611c0 msgr2=0x7face0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- 172.21.15.61:0/259483437 shutdown_connections 2022-07-20T03:10:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.375+0000 7face76dd700 1 -- 172.21.15.61:0/259483437 wait complete. 2022-07-20T03:10:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 Processor -- start 2022-07-20T03:10:57.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- start start 2022-07-20T03:10:57.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7face00aedc0 0x7face00a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 0x7face00a3e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00a1510 0x7face013bed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face00a3130 con 0x7face00b0080 2022-07-20T03:10:57.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7face00a4370 con 0x7face00a1510 2022-07-20T03:10:57.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7face00a4590 con 0x7face00aedc0 2022-07-20T03:10:57.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 0x7face00a3e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 0x7face00a3e80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44926/0 (socket says 172.21.15.61:44926) 2022-07-20T03:10:57.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 -- 172.21.15.61:0/993439652 learned_addr learned my addr 172.21.15.61:0/993439652 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:57.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face5c7a700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00a1510 0x7face013bed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face5479700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7face00aedc0 0x7face00a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 -- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7face00aedc0 msgr2=0x7face00a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7face00aedc0 0x7face00a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 -- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00a1510 msgr2=0x7face013bed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:57.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00a1510 0x7face013bed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:57.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 -- 172.21.15.61:0/993439652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7facdc00c310 con 0x7face00b0080 2022-07-20T03:10:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face4c78700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 0x7face00a3e80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7face00a44b0 tx=0x7facdc017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:57.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facdc017b00 con 0x7face00b0080 2022-07-20T03:10:57.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7facdc017c60 con 0x7face00b0080 2022-07-20T03:10:57.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7facdc023450 con 0x7face00b0080 2022-07-20T03:10:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7face00a4830 con 0x7face00b0080 2022-07-20T03:10:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7face013c3c0 con 0x7face00b0080 2022-07-20T03:10:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.379+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7face00574e0 con 0x7face00b0080 2022-07-20T03:10:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.383+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7facdc020070 con 0x7face00b0080 2022-07-20T03:10:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.383+0000 7facd27fc700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7facc0053a40 0x7facc0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.383+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7facdc06e810 con 0x7face00b0080 2022-07-20T03:10:57.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.383+0000 7face5479700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7facc0053a40 0x7facc0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:57.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.383+0000 7face5479700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7facc0053a40 0x7facc0055f00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7facd400d000 tx=0x7facd4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:57.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.391+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7facdc045940 con 0x7face00b0080 2022-07-20T03:10:57.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:57.559+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]} v 0) v1 -- 0x7face00a2840 con 0x7face00b0080 2022-07-20T03:10:58.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.455+0000 7facd27fc700 1 -- 172.21.15.61:0/993439652 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead"]}]=0 v180) v1 ==== 144+0+0 (secure 0 0 0) 0x7facdc028070 con 0x7face00b0080 2022-07-20T03:10:58.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7facc0053a40 msgr2=0x7facc0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7facc0053a40 0x7facc0055f00 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7facd400d000 tx=0x7facd4009040).stop 2022-07-20T03:10:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 msgr2=0x7face00a3e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 0x7face00a3e80 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7face00a44b0 tx=0x7facdc017000).stop 2022-07-20T03:10:58.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 shutdown_connections 2022-07-20T03:10:58.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7facc0053a40 0x7facc0055f00 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:58.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7face00a1510 0x7face013bed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:58.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7face00b0080 0x7face00a3e80 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:58.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 --2- 172.21.15.61:0/993439652 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7face00aedc0 0x7face00a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:58.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 >> 172.21.15.61:0/993439652 conn(0x7face00ac9d0 msgr2=0x7face00b3ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:58.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 shutdown_connections 2022-07-20T03:10:58.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.459+0000 7face76dd700 1 -- 172.21.15.61:0/993439652 wait complete. 2022-07-20T03:10:58.496 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T03:10:58.497 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T03:10:58.499 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-20T03:10:59.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 Processor -- start 2022-07-20T03:10:59.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 -- start start 2022-07-20T03:10:59.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d00b23b0 0x7ff1d00abec0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 0x7ff1d013efb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff1d00ac680 0x7ff1d013f4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d00ab4e0 con 0x7ff1d01408a0 2022-07-20T03:10:59.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1d00ab780 con 0x7ff1d00ac680 2022-07-20T03:10:59.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1d4cf8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d00aba20 con 0x7ff1d00b23b0 2022-07-20T03:10:59.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1cdd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 0x7ff1d013efb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1cdd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 0x7ff1d013efb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44934/0 (socket says 172.21.15.61:44934) 2022-07-20T03:10:59.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1cdd9b700 1 -- 172.21.15.61:0/30890909 learned_addr learned my addr 172.21.15.61:0/30890909 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:59.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/30890909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d00b23b0 0x7ff1d00abec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.615+0000 7ff1ced9d700 1 --2- 172.21.15.61:0/30890909 >> v2:172.21.15.117:3300/0 conn(0x7ff1d00ac680 0x7ff1d013f4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1ce59c700 1 -- 172.21.15.61:0/30890909 >> v2:172.21.15.117:3300/0 conn(0x7ff1d00ac680 msgr2=0x7ff1d013f4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/30890909 >> v2:172.21.15.117:3300/0 conn(0x7ff1d00ac680 0x7ff1d013f4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1ce59c700 1 -- 172.21.15.61:0/30890909 >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 msgr2=0x7ff1d013efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/30890909 >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 0x7ff1d013efb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1ce59c700 1 -- 172.21.15.61:0/30890909 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1d00b2790 con 0x7ff1d00b23b0 2022-07-20T03:10:59.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1cdd9b700 1 --2- 172.21.15.61:0/30890909 >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 0x7ff1d013efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:10:59.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/30890909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d00b23b0 0x7ff1d00abec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff1c0002f70 tx=0x7ff1c0006df0).ready entity=mon.2 client_cookie=50b35c039befbce3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:59.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1cd59a700 1 -- 172.21.15.61:0/30890909 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c000d4a0 con 0x7ff1d00b23b0 2022-07-20T03:10:59.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1cd59a700 1 -- 172.21.15.61:0/30890909 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c0000e90 con 0x7ff1d00b23b0 2022-07-20T03:10:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1cd59a700 1 -- 172.21.15.61:0/30890909 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c000d7f0 con 0x7ff1d00b23b0 2022-07-20T03:10:59.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/30890909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d00b23b0 msgr2=0x7ff1d00abec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/30890909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d00b23b0 0x7ff1d00abec0 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7ff1c0002f70 tx=0x7ff1c0006df0).stop 2022-07-20T03:10:59.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/30890909 shutdown_connections 2022-07-20T03:10:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/30890909 >> v2:172.21.15.117:3300/0 conn(0x7ff1d00ac680 0x7ff1d013f4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/30890909 >> v2:172.21.15.61:3300/0 conn(0x7ff1d01408a0 0x7ff1d013efb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/30890909 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d00b23b0 0x7ff1d00abec0 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/30890909 >> 172.21.15.61:0/30890909 conn(0x7ff1d00b3680 msgr2=0x7ff1d00b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:59.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/30890909 shutdown_connections 2022-07-20T03:10:59.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/30890909 wait complete. 2022-07-20T03:10:59.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 Processor -- start 2022-07-20T03:10:59.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.619+0000 7ff1d4cf8700 1 -- start start 2022-07-20T03:10:59.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d00ac680 0x7ff1d013f1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 0x7ff1d0152800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d01408a0 0x7ff1d0152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d013f7a0 con 0x7ff1d00b23b0 2022-07-20T03:10:59.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d0094c10 con 0x7ff1d00ac680 2022-07-20T03:10:59.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1d01520c0 con 0x7ff1d01408a0 2022-07-20T03:10:59.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 0x7ff1d0152800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 0x7ff1d0152800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44940/0 (socket says 172.21.15.61:44940) 2022-07-20T03:10:59.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 -- 172.21.15.61:0/1581904775 learned_addr learned my addr 172.21.15.61:0/1581904775 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:59.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1ced9d700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d01408a0 0x7ff1d0152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d00ac680 0x7ff1d013f1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 -- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d01408a0 msgr2=0x7ff1d0152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:59.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d01408a0 0x7ff1d0152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 -- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d00ac680 msgr2=0x7ff1d013f1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:59.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d00ac680 0x7ff1d013f1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 -- 172.21.15.61:0/1581904775 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1c8009530 con 0x7ff1d00b23b0 2022-07-20T03:10:59.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1cdd9b700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 0x7ff1d0152800 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7ff1c8006b60 tx=0x7ff1c8009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:59.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c800c920 con 0x7ff1d00b23b0 2022-07-20T03:10:59.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff1c800ca80 con 0x7ff1d00b23b0 2022-07-20T03:10:59.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1c000b180 con 0x7ff1d00b23b0 2022-07-20T03:10:59.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff1c800cde0 con 0x7ff1d00b23b0 2022-07-20T03:10:59.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.623+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1d0093d60 con 0x7ff1d00b23b0 2022-07-20T03:10:59.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.627+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff1c8022070 con 0x7ff1d00b23b0 2022-07-20T03:10:59.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.627+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1d0151f10 con 0x7ff1d00b23b0 2022-07-20T03:10:59.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.627+0000 7ff1c77fe700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff1a8053920 0x7ff1a8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.627+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff1c806c460 con 0x7ff1d00b23b0 2022-07-20T03:10:59.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.627+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff1a8053920 0x7ff1a8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.627+0000 7ff1ce59c700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff1a8053920 0x7ff1a8055de0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff1c001a000 tx=0x7ff1c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:59.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.635+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff1c8043590 con 0x7ff1d00b23b0 2022-07-20T03:10:59.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:58.799+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7ff1d00ab0b0 con 0x7ff1d00b23b0 2022-07-20T03:10:59.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.463+0000 7ff1c77fe700 1 -- 172.21.15.61:0/1581904775 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v181) v1 ==== 155+0+0 (secure 0 0 0) 0x7ff1c8046bb0 con 0x7ff1d00b23b0 2022-07-20T03:10:59.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff1a8053920 msgr2=0x7ff1a8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:59.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff1a8053920 0x7ff1a8055de0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7ff1c001a000 tx=0x7ff1c0014000).stop 2022-07-20T03:10:59.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 msgr2=0x7ff1d0152800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:59.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 0x7ff1d0152800 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7ff1c8006b60 tx=0x7ff1c8009290).stop 2022-07-20T03:10:59.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 shutdown_connections 2022-07-20T03:10:59.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff1a8053920 0x7ff1a8055de0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1d01408a0 0x7ff1d0152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff1d00b23b0 0x7ff1d0152800 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 --2- 172.21.15.61:0/1581904775 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1d00ac680 0x7ff1d013f1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 >> 172.21.15.61:0/1581904775 conn(0x7ff1d00b3680 msgr2=0x7ff1d00b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:10:59.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 shutdown_connections 2022-07-20T03:10:59.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.467+0000 7ff1d4cf8700 1 -- 172.21.15.61:0/1581904775 wait complete. 2022-07-20T03:10:59.669 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:10:59.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 Processor -- start 2022-07-20T03:10:59.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 -- start start 2022-07-20T03:10:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c80b2ec0 0x7fc5c80b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc5c805d160 0x7fc5c8142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:10:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5c80b3f10 con 0x7fc5c8142700 2022-07-20T03:10:59.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5c8142b90 con 0x7fc5c805d160 2022-07-20T03:10:59.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.787+0000 7fc5cd898700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c80b4ea0 con 0x7fc5c80b2ec0 2022-07-20T03:10:59.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc5c805d160 0x7fc5c8142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cc896700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c80b2ec0 0x7fc5c80b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:10:59.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc5c805d160 0x7fc5c8142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54626/0 (socket says 172.21.15.61:54626) 2022-07-20T03:10:59.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 -- 172.21.15.61:0/3711952399 learned_addr learned my addr 172.21.15.61:0/3711952399 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:10:59.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 -- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c80b2ec0 msgr2=0x7fc5c80b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:10:59.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 --2- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c80b2ec0 0x7fc5c80b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:10:59.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 -- 172.21.15.61:0/3711952399 >> v1:172.21.15.117:6789/0 conn(0x7fc5c8142700 legacy=0x7fc5c81461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:10:59.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 -- 172.21.15.61:0/3711952399 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5c80b3790 con 0x7fc5c805d160 2022-07-20T03:10:59.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c77fe700 1 -- 172.21.15.61:0/3711952399 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3021542375 0 0) 0x7fc5c80b3f10 con 0x7fc5c8142700 2022-07-20T03:10:59.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cc896700 1 --2- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c80b2ec0 0x7fc5c80b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:10:59.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c7fff700 1 --2- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c805d160 0x7fc5c8142210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc5c80b4fe0 tx=0x7fc5bc006eb0).ready entity=mon.1 client_cookie=a107ca0508a3a6d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:10:59.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c77fe700 1 -- 172.21.15.61:0/3711952399 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5bc00d5c0 con 0x7fc5c805d160 2022-07-20T03:11:00.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c77fe700 1 -- 172.21.15.61:0/3711952399 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc000e90 con 0x7fc5c805d160 2022-07-20T03:11:00.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5c77fe700 1 -- 172.21.15.61:0/3711952399 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5bc00d910 con 0x7fc5c805d160 2022-07-20T03:11:00.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 -- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c805d160 msgr2=0x7fc5c8142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:00.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 --2- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c805d160 0x7fc5c8142210 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7fc5c80b4fe0 tx=0x7fc5bc006eb0).stop 2022-07-20T03:11:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 -- 172.21.15.61:0/3711952399 shutdown_connections 2022-07-20T03:11:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 --2- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c805d160 0x7fc5c8142210 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 --2- 172.21.15.61:0/3711952399 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c80b2ec0 0x7fc5c80b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 -- 172.21.15.61:0/3711952399 >> 172.21.15.61:0/3711952399 conn(0x7fc5c8066e20 msgr2=0x7fc5c8067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 -- 172.21.15.61:0/3711952399 shutdown_connections 2022-07-20T03:11:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.791+0000 7fc5cd898700 1 -- 172.21.15.61:0/3711952399 wait complete. 2022-07-20T03:11:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 Processor -- start 2022-07-20T03:11:00.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 -- start start 2022-07-20T03:11:00.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 0x7fc5c80ad890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:00.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c80b2ec0 0x7fc5c8152520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:00.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c8142700 0x7fc5c8152a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:00.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8146200 con 0x7fc5c805d160 2022-07-20T03:11:00.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c81464a0 con 0x7fc5c80b2ec0 2022-07-20T03:11:00.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5c8146740 con 0x7fc5c8142700 2022-07-20T03:11:00.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 0x7fc5c80ad890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:00.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 0x7fc5c80ad890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44952/0 (socket says 172.21.15.61:44952) 2022-07-20T03:11:00.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 -- 172.21.15.61:0/803543191 learned_addr learned my addr 172.21.15.61:0/803543191 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:00.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5c7fff700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c80b2ec0 0x7fc5c8152520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:00.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd097700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c8142700 0x7fc5c8152a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:00.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 -- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c8142700 msgr2=0x7fc5c8152a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:00.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c8142700 0x7fc5c8152a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 -- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c80b2ec0 msgr2=0x7fc5c8152520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:00.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c80b2ec0 0x7fc5c8152520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 -- 172.21.15.61:0/803543191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5b800b180 con 0x7fc5c805d160 2022-07-20T03:11:00.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cc896700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 0x7fc5c80ad890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fc5b800c000 tx=0x7fc5b8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:00.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b800c7e0 con 0x7fc5c805d160 2022-07-20T03:11:00.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5b800c940 con 0x7fc5c805d160 2022-07-20T03:11:00.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.795+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5bc00b1e0 con 0x7fc5c805d160 2022-07-20T03:11:00.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5b800cca0 con 0x7fc5c805d160 2022-07-20T03:11:00.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5c8146be0 con 0x7fc5c805d160 2022-07-20T03:11:00.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5c8146900 con 0x7fc5c805d160 2022-07-20T03:11:00.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc5b801c460 con 0x7fc5c805d160 2022-07-20T03:11:00.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5c5ffb700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5a8053af0 0x7fc5a8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:00.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc5b806c0e0 con 0x7fc5c805d160 2022-07-20T03:11:00.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.799+0000 7fc5c7fff700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5a8053af0 0x7fc5a8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:00.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.803+0000 7fc5c7fff700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5a8053af0 0x7fc5a8055fb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc5bc01b000 tx=0x7fc5bc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:00.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.807+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5b80432c0 con 0x7fc5c805d160 2022-07-20T03:11:00.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.939+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fc5c80b2620 con 0x7fc5c805d160 2022-07-20T03:11:00.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.947+0000 7fc5c5ffb700 1 -- 172.21.15.61:0/803543191 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v6)=0 v6) v1 ==== 122+0+0 (secure 0 0 0) 0x7fc5b800ce00 con 0x7fc5c805d160 2022-07-20T03:11:00.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5a8053af0 msgr2=0x7fc5a8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:00.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5a8053af0 0x7fc5a8055fb0 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fc5bc01b000 tx=0x7fc5bc015000).stop 2022-07-20T03:11:00.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 msgr2=0x7fc5c80ad890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:00.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 0x7fc5c80ad890 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7fc5b800c000 tx=0x7fc5b8006df0).stop 2022-07-20T03:11:00.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 shutdown_connections 2022-07-20T03:11:00.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5a8053af0 0x7fc5a8055fb0 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5c80b2ec0 0x7fc5c8152520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5c8142700 0x7fc5c8152a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 --2- 172.21.15.61:0/803543191 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5c805d160 0x7fc5c80ad890 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:00.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 >> 172.21.15.61:0/803543191 conn(0x7fc5c8066e20 msgr2=0x7fc5c805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:00.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 shutdown_connections 2022-07-20T03:11:00.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:10:59.951+0000 7fc5cd898700 1 -- 172.21.15.61:0/803543191 wait complete. 2022-07-20T03:11:00.462 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:11:00.463 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:11:00.549 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:11:00.623 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:11:00.653 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:11:00.687 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:11:00.756 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:11:00.759 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:00.760 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:00.760 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:00.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:11:00.956 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:11:00.957 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:11:00.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:00.973 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:00.973 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:11:01.002 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:11:01.022 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:11:01.035 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:11:01.036 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:01.036 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:01.037 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:11:01.121 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:11:01.143 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:11:01.144 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:11:01.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:11:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.307+0000 7f5660f25700 1 Processor -- start 2022-07-20T03:11:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.307+0000 7f5660f25700 1 -- start start 2022-07-20T03:11:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f565c0b21f0 0x7f565c0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:01.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f565c1438b0 0x7f565c05fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:01.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f565c060ec0 con 0x7f565c0ac8a0 2022-07-20T03:11:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c143d20 con 0x7f565c0b21f0 2022-07-20T03:11:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f565c05ff40 con 0x7f565c1438b0 2022-07-20T03:11:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565a59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f565c0b21f0 0x7f565c0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5659d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f565c0ac8a0 0x7f565c140260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41558/0 (socket says 172.21.15.61:41558) 2022-07-20T03:11:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5659d9b700 1 -- 172.21.15.61:0/1988019520 learned_addr learned my addr 172.21.15.61:0/1988019520 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565ad9d700 1 --2- 172.21.15.61:0/1988019520 >> v2:172.21.15.117:3300/0 conn(0x7f565c1438b0 0x7f565c05fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565959a700 1 -- 172.21.15.61:0/1988019520 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654607478 0 0) 0x7f565c060ec0 con 0x7f565c0ac8a0 2022-07-20T03:11:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565959a700 1 -- 172.21.15.61:0/1988019520 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5640003260 con 0x7f565c0ac8a0 2022-07-20T03:11:01.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565a59c700 1 -- 172.21.15.61:0/1988019520 >> v2:172.21.15.117:3300/0 conn(0x7f565c1438b0 msgr2=0x7f565c05fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:01.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565a59c700 1 --2- 172.21.15.61:0/1988019520 >> v2:172.21.15.117:3300/0 conn(0x7f565c1438b0 0x7f565c05fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:01.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565a59c700 1 -- 172.21.15.61:0/1988019520 >> v1:172.21.15.61:6789/0 conn(0x7f565c0ac8a0 legacy=0x7f565c140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:01.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565a59c700 1 -- 172.21.15.61:0/1988019520 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565c13fb20 con 0x7f565c0b21f0 2022-07-20T03:11:01.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565a59c700 1 --2- 172.21.15.61:0/1988019520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c0b25d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f565c143e60 tx=0x7f5650006df0).ready entity=mon.0 client_cookie=2562cf345e982aad server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565959a700 1 -- 172.21.15.61:0/1988019520 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f565000d4f0 con 0x7f565c0b21f0 2022-07-20T03:11:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565959a700 1 -- 172.21.15.61:0/1988019520 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5650000e90 con 0x7f565c0b21f0 2022-07-20T03:11:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f565959a700 1 -- 172.21.15.61:0/1988019520 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f565000d840 con 0x7f565c0b21f0 2022-07-20T03:11:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- 172.21.15.61:0/1988019520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 msgr2=0x7f565c0b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:01.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 --2- 172.21.15.61:0/1988019520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c0b25d0 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f565c143e60 tx=0x7f5650006df0).stop 2022-07-20T03:11:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- 172.21.15.61:0/1988019520 shutdown_connections 2022-07-20T03:11:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 --2- 172.21.15.61:0/1988019520 >> v2:172.21.15.117:3300/0 conn(0x7f565c1438b0 0x7f565c05fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 --2- 172.21.15.61:0/1988019520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c0b25d0 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:01.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- 172.21.15.61:0/1988019520 >> 172.21.15.61:0/1988019520 conn(0x7f565c0b36b0 msgr2=0x7f565c0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- 172.21.15.61:0/1988019520 shutdown_connections 2022-07-20T03:11:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.311+0000 7f5660f25700 1 -- 172.21.15.61:0/1988019520 wait complete. 2022-07-20T03:11:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 Processor -- start 2022-07-20T03:11:01.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 -- start start 2022-07-20T03:11:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f565c0ac8a0 0x7f565c09a0f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c155960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f565c1438b0 0x7f565c155e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:01.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c097620 con 0x7f565c0b21f0 2022-07-20T03:11:01.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f565c09a5e0 con 0x7f565c0ac8a0 2022-07-20T03:11:01.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f565c0960c0 con 0x7f565c1438b0 2022-07-20T03:11:01.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c155960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:01.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c155960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44968/0 (socket says 172.21.15.61:44968) 2022-07-20T03:11:01.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 -- 172.21.15.61:0/2996305070 learned_addr learned my addr 172.21.15.61:0/2996305070 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f565a59c700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f565c0ac8a0 0x7f565c09a0f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f565ad9d700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f565c1438b0 0x7f565c155e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 -- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f565c1438b0 msgr2=0x7f565c155e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:01.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f565c1438b0 0x7f565c155e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:01.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 -- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f565c0ac8a0 msgr2=0x7f565c09a0f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:01.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f565c0ac8a0 0x7f565c09a0f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:01.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 -- 172.21.15.61:0/2996305070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f565000b180 con 0x7f565c0b21f0 2022-07-20T03:11:01.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f565a59c700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f565c0ac8a0 0x7f565c09a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:01.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5659d9b700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c155960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f564c001490 tx=0x7f564c007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:01.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f564c005ba0 con 0x7f565c0b21f0 2022-07-20T03:11:01.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f564c005d00 con 0x7f565c0b21f0 2022-07-20T03:11:01.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f565c096360 con 0x7f565c0b21f0 2022-07-20T03:11:01.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f564c0072c0 con 0x7f565c0b21f0 2022-07-20T03:11:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.315+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f565c096730 con 0x7f565c0b21f0 2022-07-20T03:11:01.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.319+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f565c097790 con 0x7f565c0b21f0 2022-07-20T03:11:01.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.319+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f564c010070 con 0x7f565c0b21f0 2022-07-20T03:11:01.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.319+0000 7f564b7fe700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5638053920 0x7f5638055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:01.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.319+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f564c06b650 con 0x7f565c0b21f0 2022-07-20T03:11:02.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.319+0000 7f565a59c700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5638053920 0x7f5638055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:02.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.319+0000 7f565a59c700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5638053920 0x7f5638055de0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f565001a000 tx=0x7f5650014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:02.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.327+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f565c097790 con 0x7f565c0b21f0 2022-07-20T03:11:02.070 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:02.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.455+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f565c097760 con 0x7f565c0b21f0 2022-07-20T03:11:02.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f564b7fe700 1 -- 172.21.15.61:0/2996305070 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v45) v1 ==== 81+0+28875 (secure 0 0 0) 0x7f564c042700 con 0x7f565c0b21f0 2022-07-20T03:11:02.078 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:02.079 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:11:02.080 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 45, 2022-07-20T03:11:02.080 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:11:02.081 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:11:02.081 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:10:56.447172+0000", 2022-07-20T03:11:02.081 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:11:02.115 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:11:02.118 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:11:02.119 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:11:02.119 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:11:02.120 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:11:02.120 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:11:02.120 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:11:02.121 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:11:02.121 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:11:02.147 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:11:02.147 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:11:02.148 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:11:02.148 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:11:02.180 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:11:02.180 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 8, 2022-07-20T03:11:02.190 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:11:02.191 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:11:02.191 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:11:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:11:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:11:02.192 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.211 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:11:02.214 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:11:02.217 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:11:02.218 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:11:02.218 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:11:02.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:11:02.219 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:11:02.219 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:11:02.220 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:11:02.250 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:11:02.251 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:11:02.251 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:11:02.252 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:11:02.252 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:11:02.252 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:11:02.253 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:11:02.253 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:11:02.254 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:11:02.254 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:11:02.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:11:02.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:11:02.256 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:11:02.256 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:11:02.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.257 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:11:02.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:11:02.310 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:11:02.311 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:11:02.311 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:11:02.312 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:11:02.312 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:11:02.313 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:11:02.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:11:02.360 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:11:02.360 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:11:02.373 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:11:02.373 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:11:02.374 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:11:02.374 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:11:02.375 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:11:02.375 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:11:02.442 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:11:02.442 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:11:02.443 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:11:02.443 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:11:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:11:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:11:02.444 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:11:02.445 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:11:02.445 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:11:02.534 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:11:02.534 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.535 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:11:02.535 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:11:02.536 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:11:02.536 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:11:02.537 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:11:02.537 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:11:02.547 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:11:02.548 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:11:02.548 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:11:02.549 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:11:02.549 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:11:02.553 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:11:02.553 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:11:02.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.554 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:11:02.554 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:11:02.554 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.554 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.555 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.555 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:11:02.555 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:11:02.555 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:11:02.556 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:11:02.556 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:11:02.556 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:11:02.577 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:11:02.578 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:11:02.578 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:11:02.579 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:11:02.579 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:11:02.579 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:11:02.580 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:11:02.580 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:11:02.581 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:11:02.581 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:11:02.582 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:11:02.582 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:11:02.623 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:11:02.623 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:11:02.624 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:11:02.625 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:11:02.626 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:11:02.626 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.627 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:11:02.628 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:11:02.629 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:11:02.629 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:11:02.630 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:11:02.630 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:11:02.631 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:11:02.631 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:11:02.631 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:11:02.632 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:11:02.645 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:11:02.646 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:11:02.659 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:11:02.671 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:11:02.671 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:11:02.672 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:11:02.672 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:11:02.672 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:11:02.672 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:11:02.673 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:11:02.673 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:11:02.673 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:11:02.673 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:11:02.673 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:11:02.685 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:11:02.686 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:11:02.686 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:11:02.687 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.687 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:11:02.688 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:11:02.688 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:11:02.689 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:11:02.701 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:11:02.701 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:11:02.702 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:11:02.702 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:11:02.703 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:11:02.703 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:11:02.704 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:11:02.704 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:11:02.705 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:11:02.705 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:11:02.705 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.707 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.707 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:11:02.707 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:11:02.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.708 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:11:02.709 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:11:02.709 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:02.715 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:02.716 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:02.716 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:02.742 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:02.743 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:02.743 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:02.743 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:11:02.744 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:02.744 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:02.744 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:02.744 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.744 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.750 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.750 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:11:02.751 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.751 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.751 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.751 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.752 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:11:02.752 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.752 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.752 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.753 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:02.753 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.753 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.753 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.754 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:11:02.754 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.754 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.754 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.755 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.755 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:11:02.755 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.755 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.755 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.756 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.756 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:02.756 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.756 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.757 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.757 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:11:02.757 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.757 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.757 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.758 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.758 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:11:02.758 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.758 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.759 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.759 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.759 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:02.759 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.759 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.760 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.760 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:11:02.760 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.761 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.761 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.761 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:11:02.761 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:11:02.761 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.762 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.762 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.762 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:11:02.782 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:11:02.782 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:11:02.783 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:11:02.783 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:02.783 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:02.783 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:02.784 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.784 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.784 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.784 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:11:02.785 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:11:02.785 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:02.785 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:02.785 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:02.786 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:02.786 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:02.786 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:02.786 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:02.787 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:11:02.787 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:02.787 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:02.787 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:02.787 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.788 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.788 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.788 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:11:02.788 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.789 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.789 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.789 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.789 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:11:02.790 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.790 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.791 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.791 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.791 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:02.792 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.792 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.793 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.793 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:11:02.793 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.794 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.794 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.795 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.795 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:11:02.796 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.796 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.796 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.797 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.797 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:02.816 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.817 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.817 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.817 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:11:02.818 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.818 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.818 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.818 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.819 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:11:02.819 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.819 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.819 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.820 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.820 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:02.820 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.820 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.821 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.821 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:11:02.821 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.821 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.822 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.829 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.830 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:11:02.830 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:11:02.830 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.830 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.846 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.846 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:11:02.847 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:11:02.847 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:11:02.847 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:11:02.847 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:02.848 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:02.848 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:02.848 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.848 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.849 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:11:02.849 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:11:02.849 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:02.849 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:02.850 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:02.850 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:02.850 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:02.850 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:02.850 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:02.851 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 30, 2022-07-20T03:11:02.851 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:02.851 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:02.851 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:02.852 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.852 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.852 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.852 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:11:02.852 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:02.853 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.853 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.853 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.853 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:11:02.893 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:02.894 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.933 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.933 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.934 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:02.934 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.934 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.935 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.935 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:11:02.935 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:02.935 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.935 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.936 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.936 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:11:02.936 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:02.936 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.937 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.937 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.937 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:02.937 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:02.938 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.964 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:02.964 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:11:02.964 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:02.965 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.965 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:02.972 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:02.978 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:11:02.978 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:02.979 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:02.979 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:02.985 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:02.986 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:03.010 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.010 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.011 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.011 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:11:03.077 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:03.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.078 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.078 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:11:03.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:11:03.080 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.080 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.081 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:11:03.082 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:11:03.083 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:11:03.083 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:11:03.084 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:03.084 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:03.085 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:03.086 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.196 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.196 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:11:03.197 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:11:03.206 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:03.207 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:03.228 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:03.228 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:03.228 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:03.229 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:03.229 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:03.229 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:11:03.239 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:03.239 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:03.239 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:03.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.241 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:11:03.242 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.242 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:11:03.244 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.244 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.244 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.245 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:03.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.246 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:11:03.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.258 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:11:03.259 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.260 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.260 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.261 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:03.261 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.402 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.403 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:11:03.457 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.457 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.458 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.458 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.459 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:11:03.459 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.460 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.460 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.461 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:03.511 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.511 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.512 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.512 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:11:03.513 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.513 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.514 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.514 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.600 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:11:03.601 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:11:03.601 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.633 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.636 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.637 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:11:03.638 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:11:03.638 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:11:03.639 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:11:03.639 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:03.639 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:03.640 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:03.759 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.759 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.781 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:11:03.782 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:11:03.782 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:03.783 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:03.802 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:03.803 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:03.803 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:03.831 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:03.831 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:03.832 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:11:03.832 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:03.833 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:03.833 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:03.833 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.834 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.834 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.846 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:11:03.847 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:03.847 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.850 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.854 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:11:03.863 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:03.863 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.864 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.864 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.865 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:03.865 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:03.866 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.921 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:03.922 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:11:03.923 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:03.923 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.924 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:03.924 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:03.924 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:11:03.925 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:03.925 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:03.926 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:03.926 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:03.927 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:03.927 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.035 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.035 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:11:04.036 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:04.036 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.036 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.036 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.037 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:11:04.037 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:04.037 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.060 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.061 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:04.086 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.094 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:11:04.197 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:04.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.198 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:11:04.199 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:11:04.262 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.275 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.275 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.275 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:11:04.276 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:11:04.276 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:11:04.276 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:11:04.276 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.278 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:11:04.278 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:11:04.278 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:04.279 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:11:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:04.280 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:04.281 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:04.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.281 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.281 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:11:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.283 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.283 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:11:04.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.285 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.286 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.286 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:04.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.287 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:11:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:11:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.289 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.290 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.290 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:04.300 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:11:04.301 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.302 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:11:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:11:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:11:04.306 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:11:04.306 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.306 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:11:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:11:04.307 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:11:04.308 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:11:04.308 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:04.308 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:04.308 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:04.309 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.319 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.319 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:11:04.319 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:11:04.320 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:04.320 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:04.321 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:04.389 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:04.389 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:04.390 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:04.390 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 40, 2022-07-20T03:11:04.390 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:04.390 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:04.391 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:04.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.391 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.392 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.392 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:11:04.392 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.392 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.393 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:11:04.393 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.394 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.421 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.422 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.422 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:04.423 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.423 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:11:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.446 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.447 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:11:04.465 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.465 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.466 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.466 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.526 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.527 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:11:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.528 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:11:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.529 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.530 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.531 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:11:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.533 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.533 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:11:04.533 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:11:04.533 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.534 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.534 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.534 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:11:04.535 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:11:04.535 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:11:04.535 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:11:04.535 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:04.536 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:04.536 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:04.545 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.545 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.545 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:11:04.546 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:11:04.546 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:11:04.546 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:11:04.546 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:11:04.547 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:11:04.547 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:11:04.547 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:11:04.548 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:11:04.557 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 38, 2022-07-20T03:11:04.557 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:11:04.558 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:11:04.558 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:11:04.559 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.559 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.559 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.559 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:11:04.560 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.560 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.560 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.561 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.561 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:11:04.561 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.561 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.562 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.563 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.563 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:11:04.563 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.563 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.564 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.564 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:11:04.564 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.564 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.565 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.565 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.565 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:11:04.565 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.566 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.566 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.566 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.566 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:11:04.567 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.567 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.567 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.568 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:11:04.568 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.568 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.569 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.569 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:11:04.569 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.569 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.570 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.594 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.595 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:11:04.595 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:11:04.595 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.601 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:11:04.602 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:11:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.603 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:11:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:11:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:11:04.604 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:11:04.605 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:11:04.606 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:11:04.632 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:11:04.632 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:11:04.632 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:11:04.632 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:11:04.633 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:04.633 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:04.633 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:11:04.633 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:11:04.636 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.640 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:11:04.641 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:04.641 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:04.641 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:04.641 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:04.687 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:04.687 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:11:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.688 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:11:04.689 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:04.689 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:04.689 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:04.689 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:04.703 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:04.703 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:11:04.703 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:04.730 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.730 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.731 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:11:04.733 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:04.733 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:04.734 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:04.734 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:04.734 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:04.734 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:11:04.765 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:04.808 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:11:04.809 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:04.810 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:04.811 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:11:04.811 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:04.908 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.913 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.914 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:11:04.914 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:04.915 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:04.915 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:04.916 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:04.942 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:04.942 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:11:04.943 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:04.943 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.944 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:04.944 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:11:04.945 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:04.974 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:04.979 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:04.980 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:04.985 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:04.987 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:11:04.987 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:04.999 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:04.999 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:05.000 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:11:05.024 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:05.024 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:05.025 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:05.025 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:05.025 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:05.049 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:11:05.050 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:05.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:05.051 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:05.051 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:11:05.051 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:11:05.052 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:11:05.052 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:11:05.053 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:11:05.075 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:11:05.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:11:05.076 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:11:05.079 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:05.079 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:11:05.079 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:11:05.080 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:11:05.080 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:11:05.080 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:11:05.081 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:11:05.081 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6835/844334511": "2022-07-21T03:10:52.435396+0000", 2022-07-20T03:11:05.081 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6834/844334511": "2022-07-21T03:10:52.435396+0000" 2022-07-20T03:11:05.082 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:05.082 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:11:05.083 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:11:05.083 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:11:05.084 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:11:05.085 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:11:05.086 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:11:05.086 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:11:05.088 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:11:05.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:05.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:05.089 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:11:05.090 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:11:05.091 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:11:05.092 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:11:05.092 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:11:05.092 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:11:05.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5638053920 msgr2=0x7f5638055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5638053920 0x7f5638055de0 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f565001a000 tx=0x7f5650014000).stop 2022-07-20T03:11:05.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 msgr2=0x7f565c155960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c155960 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f564c001490 tx=0x7f564c007e20).stop 2022-07-20T03:11:05.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 shutdown_connections 2022-07-20T03:11:05.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5638053920 0x7f5638055de0 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f565c1438b0 0x7f565c155e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f565c0b21f0 0x7f565c155960 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 --2- 172.21.15.61:0/2996305070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f565c0ac8a0 0x7f565c09a0f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 >> 172.21.15.61:0/2996305070 conn(0x7f565c0b36b0 msgr2=0x7f565c05cd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:05.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 shutdown_connections 2022-07-20T03:11:05.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:01.459+0000 7f5660f25700 1 -- 172.21.15.61:0/2996305070 wait complete. 2022-07-20T03:11:05.274 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:05.278 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:05.304 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:05.308 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:05.359 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:05.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf97f1700 1 Processor -- start 2022-07-20T03:11:05.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf97f1700 1 -- start start 2022-07-20T03:11:05.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf97f1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febf414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf97f1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf405d480 con 0x7febf40acef0 2022-07-20T03:11:05.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf97f1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7febf40b21e0 con 0x7febf405c3f0 2022-07-20T03:11:05.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf97f1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf405cf40 con 0x7febf40b1e00 2022-07-20T03:11:05.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7febf40acef0 0x7febf4146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41576/0 (socket says 172.21.15.61:41576) 2022-07-20T03:11:05.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf37fe700 1 -- 172.21.15.61:0/1308590243 learned_addr learned my addr 172.21.15.61:0/1308590243 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:05.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febf414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3685068355 0 0) 0x7febf405d480 con 0x7febf40acef0 2022-07-20T03:11:05.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7febd4003260 con 0x7febf40acef0 2022-07-20T03:11:05.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3036772806 0 0) 0x7febd4003260 con 0x7febf40acef0 2022-07-20T03:11:05.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 msgr2=0x7febf414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:05.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febf414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 >> v1:172.21.15.117:6789/0 conn(0x7febf405c3f0 legacy=0x7febf40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf405d1e0 con 0x7febf40acef0 2022-07-20T03:11:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7febe4001660 con 0x7febf40acef0 2022-07-20T03:11:05.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 >> v1:172.21.15.61:6789/0 conn(0x7febf40acef0 legacy=0x7febf4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febd4003780 0x7febd4003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febd4004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febd40049f0 0x7febd4004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf405cf40 con 0x7febd4003780 2022-07-20T03:11:05.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febd400b800 con 0x7febd40049f0 2022-07-20T03:11:05.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febd400b9c0 con 0x7febf40b1e00 2022-07-20T03:11:05.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febd4003780 0x7febd4003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf37fe700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febd40049f0 0x7febd4004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf27fc700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febd4004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 -- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 msgr2=0x7febd4004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:05.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febd4004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 -- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febd40049f0 msgr2=0x7febd4004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:05.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febd40049f0 0x7febd4004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 -- 172.21.15.61:0/1308590243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febd4003260 con 0x7febd4003780 2022-07-20T03:11:05.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf27fc700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf40b1e00 0x7febd4004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:11:05.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf37fe700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febd40049f0 0x7febd4004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:05.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf2ffd700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febd4003780 0x7febd4003b60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7febdc00c000 tx=0x7febdc006df0).ready entity=mon.0 client_cookie=14beaf1392b5ed11 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:05.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.383+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febdc00e800 con 0x7febd4003780 2022-07-20T03:11:05.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febdc00e960 con 0x7febd4003780 2022-07-20T03:11:05.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf1ffb700 1 -- 172.21.15.61:0/1308590243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febdc00ecc0 con 0x7febd4003780 2022-07-20T03:11:05.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- 172.21.15.61:0/1308590243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febd4003780 msgr2=0x7febd4003b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 --2- 172.21.15.61:0/1308590243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febd4003780 0x7febd4003b60 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7febdc00c000 tx=0x7febdc006df0).stop 2022-07-20T03:11:05.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf37fe700 1 -- 172.21.15.61:0/1308590243 reap_dead start 2022-07-20T03:11:05.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- 172.21.15.61:0/1308590243 shutdown_connections 2022-07-20T03:11:05.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- 172.21.15.61:0/1308590243 >> 172.21.15.61:0/1308590243 conn(0x7febf40611c0 msgr2=0x7febf4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:05.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- 172.21.15.61:0/1308590243 shutdown_connections 2022-07-20T03:11:05.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- 172.21.15.61:0/1308590243 wait complete. 2022-07-20T03:11:05.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 Processor -- start 2022-07-20T03:11:05.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- start start 2022-07-20T03:11:05.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf414bec0 0x7febf40a1980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf414a490 0x7febf40a02a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 0x7febf4143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf414c2a0 con 0x7febf40a0790 2022-07-20T03:11:05.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7febf409ea90 con 0x7febf414a490 2022-07-20T03:11:05.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.387+0000 7febf97f1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7febf409ed30 con 0x7febf414bec0 2022-07-20T03:11:05.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 0x7febf4143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 0x7febf4143400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44990/0 (socket says 172.21.15.61:44990) 2022-07-20T03:11:05.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 -- 172.21.15.61:0/3590766997 learned_addr learned my addr 172.21.15.61:0/3590766997 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:05.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf27fc700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf414a490 0x7febf40a02a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf2ffd700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf414bec0 0x7febf40a1980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 -- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf414bec0 msgr2=0x7febf40a1980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf414bec0 0x7febf40a1980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 -- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf414a490 msgr2=0x7febf40a02a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf414a490 0x7febf40a02a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 -- 172.21.15.61:0/3590766997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7febf40a1240 con 0x7febf40a0790 2022-07-20T03:11:05.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf27fc700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf414a490 0x7febf40a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:05.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf37fe700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 0x7febf4143400 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7febe4000e90 tx=0x7febf40ac1a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:05.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febe4010640 con 0x7febf40a0790 2022-07-20T03:11:05.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7febe40107a0 con 0x7febf40a0790 2022-07-20T03:11:05.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7febf40a14e0 con 0x7febf40a0790 2022-07-20T03:11:05.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7febe4010b00 con 0x7febf40a0790 2022-07-20T03:11:05.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7febf4149e90 con 0x7febf40a0790 2022-07-20T03:11:05.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7febf409eef0 con 0x7febf40a0790 2022-07-20T03:11:05.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7febe401a020 con 0x7febf40a0790 2022-07-20T03:11:05.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febebfff700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd8053ad0 0x7febd8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.391+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7febe406b540 con 0x7febf40a0790 2022-07-20T03:11:05.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.395+0000 7febf2ffd700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd8053ad0 0x7febd8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.395+0000 7febf2ffd700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd8053ad0 0x7febd8055f90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7febdc01a000 tx=0x7febdc019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:05.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.403+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7febe4042780 con 0x7febf40a0790 2022-07-20T03:11:05.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.563+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7febf40a1670 con 0x7febf40a0790 2022-07-20T03:11:05.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febebfff700 1 -- 172.21.15.61:0/3590766997 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+1212 (secure 0 0 0) 0x7febf40a1670 con 0x7febf40a0790 2022-07-20T03:11:05.700 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:05.700 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18},{"gid":5557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/3870422415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3870422415},{"type":"v1","addr":"172.21.15.117:6835","nonce":3870422415}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[]} 2022-07-20T03:11:05.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd8053ad0 msgr2=0x7febd8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd8053ad0 0x7febd8055f90 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7febdc01a000 tx=0x7febdc019000).stop 2022-07-20T03:11:05.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 msgr2=0x7febf4143400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 0x7febf4143400 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7febe4000e90 tx=0x7febf40ac1a0).stop 2022-07-20T03:11:05.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 shutdown_connections 2022-07-20T03:11:05.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7febd8053ad0 0x7febd8055f90 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7febf414a490 0x7febf40a02a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7febf40a0790 0x7febf4143400 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 --2- 172.21.15.61:0/3590766997 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7febf414bec0 0x7febf40a1980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 >> 172.21.15.61:0/3590766997 conn(0x7febf4062af0 msgr2=0x7febf4144560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:05.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 shutdown_connections 2022-07-20T03:11:05.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.567+0000 7febf97f1700 1 -- 172.21.15.61:0/3590766997 wait complete. 2022-07-20T03:11:05.704 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 23 2022-07-20T03:11:05.723 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:05.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 Processor -- start 2022-07-20T03:11:05.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- start start 2022-07-20T03:11:05.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9cc80adff0 0x7f9cc80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cc80af8e0 0x7f9cc805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cc8147010 con 0x7f9cc80b5650 2022-07-20T03:11:05.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc81472b0 con 0x7f9cc80af8e0 2022-07-20T03:11:05.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cc8146d40 con 0x7f9cc80adff0 2022-07-20T03:11:05.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cc80af8e0 0x7f9cc805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cc80af8e0 0x7f9cc805bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:44998/0 (socket says 172.21.15.61:44998) 2022-07-20T03:11:05.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 -- 172.21.15.61:0/1226230152 learned_addr learned my addr 172.21.15.61:0/1226230152 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:05.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cccc5e700 1 --2- 172.21.15.61:0/1226230152 >> v2:172.21.15.117:3300/0 conn(0x7f9cc80adff0 0x7f9cc80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 -- 172.21.15.61:0/1226230152 >> v2:172.21.15.117:3300/0 conn(0x7f9cc80adff0 msgr2=0x7f9cc80b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:05.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 --2- 172.21.15.61:0/1226230152 >> v2:172.21.15.117:3300/0 conn(0x7f9cc80adff0 0x7f9cc80b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 -- 172.21.15.61:0/1226230152 >> v1:172.21.15.117:6789/0 conn(0x7f9cc80b5650 legacy=0x7f9cc8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 -- 172.21.15.61:0/1226230152 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cc80acae0 con 0x7f9cc80af8e0 2022-07-20T03:11:05.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cccc5e700 1 --2- 172.21.15.61:0/1226230152 >> v2:172.21.15.117:3300/0 conn(0x7f9cc80adff0 0x7f9cc80b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:05.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9ccdc60700 1 --2- 172.21.15.61:0/1226230152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc805bd90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f9cb8002f70 tx=0x7f9cb8006eb0).ready entity=mon.0 client_cookie=a223497a6ccf5ec4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:05.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cc7fff700 1 -- 172.21.15.61:0/1226230152 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cb800d610 con 0x7f9cc80af8e0 2022-07-20T03:11:05.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cc7fff700 1 -- 172.21.15.61:0/1226230152 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cb8000e90 con 0x7f9cc80af8e0 2022-07-20T03:11:05.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cc7fff700 1 -- 172.21.15.61:0/1226230152 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cb800d9a0 con 0x7f9cc80af8e0 2022-07-20T03:11:05.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- 172.21.15.61:0/1226230152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 msgr2=0x7f9cc805bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 --2- 172.21.15.61:0/1226230152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc805bd90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f9cb8002f70 tx=0x7f9cb8006eb0).stop 2022-07-20T03:11:05.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- 172.21.15.61:0/1226230152 shutdown_connections 2022-07-20T03:11:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 --2- 172.21.15.61:0/1226230152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc805bd90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 --2- 172.21.15.61:0/1226230152 >> v2:172.21.15.117:3300/0 conn(0x7f9cc80adff0 0x7f9cc80b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- 172.21.15.61:0/1226230152 >> 172.21.15.61:0/1226230152 conn(0x7f9cc80634e0 msgr2=0x7f9cc80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:05.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.875+0000 7f9cce461700 1 -- 172.21.15.61:0/1226230152 shutdown_connections 2022-07-20T03:11:05.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- 172.21.15.61:0/1226230152 wait complete. 2022-07-20T03:11:05.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 Processor -- start 2022-07-20T03:11:05.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- start start 2022-07-20T03:11:05.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cc80adff0 0x7f9cc81598b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc8146000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cc80b5650 0x7f9cc81464f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:05.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc8067230 con 0x7f9cc80af8e0 2022-07-20T03:11:05.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc815a320 con 0x7f9cc80b5650 2022-07-20T03:11:05.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cc815a5c0 con 0x7f9cc80adff0 2022-07-20T03:11:05.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cccc5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc8146000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cc80adff0 0x7f9cc81598b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cccc5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc8146000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45002/0 (socket says 172.21.15.61:45002) 2022-07-20T03:11:05.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cccc5e700 1 -- 172.21.15.61:0/3909297713 learned_addr learned my addr 172.21.15.61:0/3909297713 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:05.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccdc60700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cc80b5650 0x7f9cc81464f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:05.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 -- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cc80b5650 msgr2=0x7f9cc81464f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cc80b5650 0x7f9cc81464f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 -- 172.21.15.61:0/3909297713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 msgr2=0x7f9cc8146000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:05.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc8146000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:05.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 -- 172.21.15.61:0/3909297713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cb800b1e0 con 0x7f9cc80adff0 2022-07-20T03:11:05.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cccc5e700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc8146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:05.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9ccd45f700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cc80adff0 0x7f9cc81598b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f9cc8060b70 tx=0x7f9cbc007b90).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:05.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cbc00ccd0 con 0x7f9cc80adff0 2022-07-20T03:11:05.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9cbc00ce30 con 0x7f9cc80adff0 2022-07-20T03:11:06.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cc8157480 con 0x7f9cc80adff0 2022-07-20T03:11:06.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9cbc0125f0 con 0x7f9cc80adff0 2022-07-20T03:11:06.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.879+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cc8067640 con 0x7f9cc80adff0 2022-07-20T03:11:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.883+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cc80673a0 con 0x7f9cc80adff0 2022-07-20T03:11:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.883+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9cbc019030 con 0x7f9cc80adff0 2022-07-20T03:11:06.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.883+0000 7f9cc67fc700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ca8053a40 0x7f9ca8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.883+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(45..45 src has 1..45) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9cbc06b3c0 con 0x7f9cc80adff0 2022-07-20T03:11:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.883+0000 7f9cccc5e700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ca8053a40 0x7f9ca8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.887+0000 7f9cccc5e700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ca8053a40 0x7f9ca8055f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f9cc000d000 tx=0x7f9cc0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:06.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:05.895+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9cbc0425a0 con 0x7f9cc80adff0 2022-07-20T03:11:06.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.059+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9cc80673a0 con 0x7f9cc80adff0 2022-07-20T03:11:06.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cc67fc700 1 -- 172.21.15.61:0/3909297713 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 23 v23) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f9cc80673a0 con 0x7f9cc80adff0 2022-07-20T03:11:06.088 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:06.089 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":23,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5259,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/2598488564","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2598488564},{"type":"v1","addr":"172.21.15.61:6833","nonce":2598488564}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":18},{"gid":5557,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/3870422415","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3870422415},{"type":"v1","addr":"172.21.15.117:6835","nonce":3870422415}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":23}],"filesystems":[]} 2022-07-20T03:11:06.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ca8053a40 msgr2=0x7f9ca8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:06.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ca8053a40 0x7f9ca8055f00 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f9cc000d000 tx=0x7f9cc0008040).stop 2022-07-20T03:11:06.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cc80adff0 msgr2=0x7f9cc81598b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:06.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cc80adff0 0x7f9cc81598b0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f9cc8060b70 tx=0x7f9cbc007b90).stop 2022-07-20T03:11:06.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 shutdown_connections 2022-07-20T03:11:06.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ca8053a40 0x7f9ca8055f00 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cc80b5650 0x7f9cc81464f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cc80af8e0 0x7f9cc8146000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 --2- 172.21.15.61:0/3909297713 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cc80adff0 0x7f9cc81598b0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.063+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 >> 172.21.15.61:0/3909297713 conn(0x7f9cc80634e0 msgr2=0x7f9cc80afea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:06.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.067+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 shutdown_connections 2022-07-20T03:11:06.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.067+0000 7f9cce461700 1 -- 172.21.15.61:0/3909297713 wait complete. 2022-07-20T03:11:06.117 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 23 2022-07-20T03:11:06.143 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:11:06.144 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:11:06.145 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:11:06.145 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:11:06.145 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:11:06.146 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:11:06.146 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:11:06.147 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:11:06.170 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:11:06.171 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:11:06.172 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:11:06.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:11:06.186 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:11:06.207 DEBUG:teuthology.parallel:result is None 2022-07-20T03:11:06.224 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:11:06.225 DEBUG:teuthology.parallel:result is None 2022-07-20T03:11:06.226 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:11:06.287 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:11:06.303 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:11:06.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 Processor -- start 2022-07-20T03:11:06.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 -- start start 2022-07-20T03:11:06.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff1000acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff100146770 0x7ff10014c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff1000b46e0 con 0x7ff1000b5860 2022-07-20T03:11:06.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff1000ad990 con 0x7ff100146770 2022-07-20T03:11:06.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff1000b4b10 con 0x7ff1000b4f80 2022-07-20T03:11:06.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0feffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff1000b5860 0x7ff100146060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41618/0 (socket says 172.21.15.61:41618) 2022-07-20T03:11:06.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0ff7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff100146770 0x7ff10014c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0feffd700 1 -- 172.21.15.61:0/4244741353 learned_addr learned my addr 172.21.15.61:0/4244741353 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:06.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fe7fc700 1 --2- 172.21.15.61:0/4244741353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff1000acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fdffb700 1 -- 172.21.15.61:0/4244741353 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423196592 0 0) 0x7ff1000b46e0 con 0x7ff1000b5860 2022-07-20T03:11:06.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fdffb700 1 -- 172.21.15.61:0/4244741353 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0e0003260 con 0x7ff1000b5860 2022-07-20T03:11:06.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/4244741353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 msgr2=0x7ff1000acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:06.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0ff7fe700 1 --2- 172.21.15.61:0/4244741353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff1000acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/4244741353 >> v1:172.21.15.61:6789/0 conn(0x7ff1000b5860 legacy=0x7ff100146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:06.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/4244741353 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff1000ad3d0 con 0x7ff100146770 2022-07-20T03:11:06.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0ff7fe700 1 --2- 172.21.15.61:0/4244741353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff10014c670 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7ff0f0002f70 tx=0x7ff0f0006df0).ready entity=mon.0 client_cookie=e67cfa9b9461c202 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:06.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fdffb700 1 -- 172.21.15.61:0/4244741353 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f000d4a0 con 0x7ff100146770 2022-07-20T03:11:06.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fdffb700 1 -- 172.21.15.61:0/4244741353 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f0000e90 con 0x7ff100146770 2022-07-20T03:11:06.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.339+0000 7ff0fdffb700 1 -- 172.21.15.61:0/4244741353 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f000d830 con 0x7ff100146770 2022-07-20T03:11:06.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- 172.21.15.61:0/4244741353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 msgr2=0x7ff10014c670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:06.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 --2- 172.21.15.61:0/4244741353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff10014c670 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7ff0f0002f70 tx=0x7ff0f0006df0).stop 2022-07-20T03:11:06.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- 172.21.15.61:0/4244741353 shutdown_connections 2022-07-20T03:11:06.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 --2- 172.21.15.61:0/4244741353 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff10014c670 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 --2- 172.21.15.61:0/4244741353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff1000acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- 172.21.15.61:0/4244741353 >> 172.21.15.61:0/4244741353 conn(0x7ff1000b5e10 msgr2=0x7ff1000b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:06.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- 172.21.15.61:0/4244741353 shutdown_connections 2022-07-20T03:11:06.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- 172.21.15.61:0/4244741353 wait complete. 2022-07-20T03:11:06.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 Processor -- start 2022-07-20T03:11:06.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- start start 2022-07-20T03:11:06.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff100114890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1000b5860 0x7ff100112700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff100151a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff100113f30 con 0x7ff100146770 2022-07-20T03:11:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff100114d80 con 0x7ff1000b5860 2022-07-20T03:11:06.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff100114150 con 0x7ff1000b4f80 2022-07-20T03:11:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff100151a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff100151a80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45030/0 (socket says 172.21.15.61:45030) 2022-07-20T03:11:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/1290385139 learned_addr learned my addr 172.21.15.61:0/1290385139 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:06.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0fe7fc700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1000b5860 0x7ff100112700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0feffd700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff100114890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 msgr2=0x7ff100114890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:06.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff100114890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1000b5860 msgr2=0x7ff100112700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:06.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1000b5860 0x7ff100112700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:06.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 -- 172.21.15.61:0/1290385139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0f000b180 con 0x7ff100146770 2022-07-20T03:11:06.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.343+0000 7ff0ff7fe700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff100151a80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7ff0f0002f70 tx=0x7ff0f0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:06.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f0012930 con 0x7ff100146770 2022-07-20T03:11:06.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0f0012a90 con 0x7ff100146770 2022-07-20T03:11:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff1001143f0 con 0x7ff100146770 2022-07-20T03:11:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0f0012df0 con 0x7ff100146770 2022-07-20T03:11:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1000b1bb0 con 0x7ff100146770 2022-07-20T03:11:06.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff1000b1740 con 0x7ff100146770 2022-07-20T03:11:06.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff0f001c070 con 0x7ff100146770 2022-07-20T03:11:06.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0f7fff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0e4053a60 0x7ff0e4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:06.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(46..46 src has 1..46) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff0f006cae0 con 0x7ff100146770 2022-07-20T03:11:06.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0feffd700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0e4053a60 0x7ff0e4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:06.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.347+0000 7ff0feffd700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0e4053a60 0x7ff0e4055f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ff0e800d000 tx=0x7ff0e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:06.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.355+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0f0043cc0 con 0x7ff100146770 2022-07-20T03:11:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:06.491+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7ff100114580 con 0x7ff100146770 2022-07-20T03:11:07.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.279+0000 7ff0f7fff700 1 -- 172.21.15.61:0/1290385139 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v47) v1 ==== 97+0+0 (secure 0 0 0) 0x7ff0f0019030 con 0x7ff100146770 2022-07-20T03:11:07.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0e4053a60 msgr2=0x7ff0e4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:07.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0e4053a60 0x7ff0e4055f20 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7ff0e800d000 tx=0x7ff0e8009040).stop 2022-07-20T03:11:07.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 msgr2=0x7ff100151a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:07.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff100151a80 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7ff0f0002f70 tx=0x7ff0f0014000).stop 2022-07-20T03:11:07.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 shutdown_connections 2022-07-20T03:11:07.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0e4053a60 0x7ff0e4055f20 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff1000b5860 0x7ff100112700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff100146770 0x7ff100151a80 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 --2- 172.21.15.61:0/1290385139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff1000b4f80 0x7ff100114890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 >> 172.21.15.61:0/1290385139 conn(0x7ff1000b5e10 msgr2=0x7ff1000b82a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:07.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 shutdown_connections 2022-07-20T03:11:07.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.283+0000 7ff0fffff700 1 -- 172.21.15.61:0/1290385139 wait complete. 2022-07-20T03:11:07.295 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:11:07.331 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:11:07.332 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:11:07.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 Processor -- start 2022-07-20T03:11:07.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 -- start start 2022-07-20T03:11:07.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4060ed0 0x7f77d4143810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:07.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77d4062cf0 0x7f77d413c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:07.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77d40b1a00 con 0x7f77d40b1620 2022-07-20T03:11:07.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77d4140940 con 0x7f77d4062cf0 2022-07-20T03:11:07.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.491+0000 7f77db82c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d405c0a0 con 0x7f77d4060ed0 2022-07-20T03:11:07.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d9dc9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77d4062cf0 0x7f77d413c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:07.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d95c8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f77d40b1620 0x7f77d4143100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41632/0 (socket says 172.21.15.61:41632) 2022-07-20T03:11:07.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d95c8700 1 -- 172.21.15.61:0/639918795 learned_addr learned my addr 172.21.15.61:0/639918795 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:07.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d8dc7700 1 --2- 172.21.15.61:0/639918795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4060ed0 0x7f77d4143810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:07.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d9dc9700 1 -- 172.21.15.61:0/639918795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4060ed0 msgr2=0x7f77d4143810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:07.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d9dc9700 1 --2- 172.21.15.61:0/639918795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4060ed0 0x7f77d4143810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d9dc9700 1 -- 172.21.15.61:0/639918795 >> v1:172.21.15.61:6789/0 conn(0x7f77d40b1620 legacy=0x7f77d4143100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:07.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77c3fff700 1 -- 172.21.15.61:0/639918795 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1159338471 0 0) 0x7f77d40b1a00 con 0x7f77d40b1620 2022-07-20T03:11:07.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d9dc9700 1 -- 172.21.15.61:0/639918795 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77d40ae0f0 con 0x7f77d4062cf0 2022-07-20T03:11:07.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77d9dc9700 1 --2- 172.21.15.61:0/639918795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d4062cf0 0x7f77d413c710 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f77cc0032f0 tx=0x7f77cc0091d0).ready entity=mon.0 client_cookie=532aff158af7189f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77c3fff700 1 -- 172.21.15.61:0/639918795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77cc00b6c0 con 0x7f77d4062cf0 2022-07-20T03:11:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77c3fff700 1 -- 172.21.15.61:0/639918795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77cc00b820 con 0x7f77d4062cf0 2022-07-20T03:11:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77c3fff700 1 -- 172.21.15.61:0/639918795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77cc00b990 con 0x7f77d4062cf0 2022-07-20T03:11:07.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 -- 172.21.15.61:0/639918795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d4062cf0 msgr2=0x7f77d413c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 --2- 172.21.15.61:0/639918795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d4062cf0 0x7f77d413c710 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7f77cc0032f0 tx=0x7f77cc0091d0).stop 2022-07-20T03:11:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 -- 172.21.15.61:0/639918795 shutdown_connections 2022-07-20T03:11:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 --2- 172.21.15.61:0/639918795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d4062cf0 0x7f77d413c710 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 --2- 172.21.15.61:0/639918795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4060ed0 0x7f77d4143810 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 -- 172.21.15.61:0/639918795 >> 172.21.15.61:0/639918795 conn(0x7f77d40b23d0 msgr2=0x7f77d40b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:07.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 -- 172.21.15.61:0/639918795 shutdown_connections 2022-07-20T03:11:07.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 -- 172.21.15.61:0/639918795 wait complete. 2022-07-20T03:11:07.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 Processor -- start 2022-07-20T03:11:07.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.495+0000 7f77db82c700 1 -- start start 2022-07-20T03:11:07.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77d4060ed0 0x7f77d4155a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:07.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4062cf0 0x7f77d40ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:07.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 0x7f77d413a430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:07.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d413e300 con 0x7f77d40b1620 2022-07-20T03:11:07.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40b3700 con 0x7f77d4060ed0 2022-07-20T03:11:07.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77d40b4590 con 0x7f77d4062cf0 2022-07-20T03:11:07.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 0x7f77d413a430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:07.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 0x7f77d413a430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45044/0 (socket says 172.21.15.61:45044) 2022-07-20T03:11:07.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 -- 172.21.15.61:0/1591286424 learned_addr learned my addr 172.21.15.61:0/1591286424 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:07.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d8dc7700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4062cf0 0x7f77d40ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:07.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d95c8700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77d4060ed0 0x7f77d4155a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:07.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 -- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4062cf0 msgr2=0x7f77d40ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:07.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4062cf0 0x7f77d40ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 -- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77d4060ed0 msgr2=0x7f77d4155a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:07.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77d4060ed0 0x7f77d4155a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:07.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 -- 172.21.15.61:0/1591286424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77cc009470 con 0x7f77d40b1620 2022-07-20T03:11:07.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d95c8700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77d4060ed0 0x7f77d4155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:07.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77d9dc9700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 0x7f77d413a430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f77cc0032f0 tx=0x7f77cc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:07.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77cc00b990 con 0x7f77d40b1620 2022-07-20T03:11:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77d40b5460 con 0x7f77d40b1620 2022-07-20T03:11:07.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77cc0107a0 con 0x7f77d40b1620 2022-07-20T03:11:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77d413aac0 con 0x7f77d40b1620 2022-07-20T03:11:07.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77cc00b6c0 con 0x7f77d40b1620 2022-07-20T03:11:07.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.499+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77d40b55f0 con 0x7f77d40b1620 2022-07-20T03:11:07.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.503+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f77cc027020 con 0x7f77d40b1620 2022-07-20T03:11:07.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.503+0000 7f77c27fc700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77b4053ad0 0x7f77b4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:07.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.503+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f77cc06d060 con 0x7f77d40b1620 2022-07-20T03:11:07.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.503+0000 7f77d95c8700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77b4053ad0 0x7f77b4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.503+0000 7f77d95c8700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77b4053ad0 0x7f77b4055f90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f77d000b000 tx=0x7f77d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:07.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.507+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77cc044160 con 0x7f77d40b1620 2022-07-20T03:11:07.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.675+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f77d40b3900 con 0x7f77d40b1620 2022-07-20T03:11:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.675+0000 7f77c27fc700 1 -- 172.21.15.61:0/1591286424 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v18) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f77cc015020 con 0x7f77d40b1620 2022-07-20T03:11:07.776 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:07.776 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:11:07.777 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:11:07.777 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.778 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:11:07.778 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:11:07.786 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.786 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:11:07.787 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.787 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.788 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.788 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.789 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.789 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.789 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:11:07.809 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:11:07.810 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.810 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:11:07.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.812 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.812 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.813 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.813 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.813 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:11:07.814 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:11:07.814 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.815 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.816 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.856 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.856 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.857 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.857 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:11:07.896 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:11:07.902 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.903 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.903 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.903 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.904 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.904 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.905 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.905 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:11:07.917 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:11:07.920 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.920 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.921 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.921 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.922 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.922 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.931 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.931 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:11:07.932 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:11:07.932 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.934 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.934 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.935 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.942 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.942 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.943 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.946 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:11:07.946 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:11:07.946 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.947 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.947 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.947 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.947 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.947 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.948 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.948 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:11:07.948 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:11:07.966 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.967 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.977 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.977 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.993 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.993 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:07.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:07.994 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:11:07.994 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:11:07.995 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:07.995 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:07.996 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:07.996 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:07.997 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:07.997 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.019 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.035 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:11:08.036 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:11:08.037 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.037 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:08.037 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:08.039 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:08.040 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.041 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.041 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:11:08.042 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:11:08.081 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.081 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:11:08.106 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:11:08.107 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:11:08.108 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.109 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.109 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:11:08.109 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:11:08.110 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.140 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:11:08.140 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:11:08.141 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:11:08.142 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.147 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.147 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:11:08.148 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:11:08.148 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.148 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:11:08.149 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:11:08.149 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:11:08.149 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.198 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.199 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:11:08.199 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:11:08.199 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.200 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:11:08.202 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:11:08.202 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:11:08.202 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:08.202 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.203 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.203 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:11:08.203 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:11:08.204 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.204 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:11:08.204 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:11:08.204 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:08.204 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:08.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:08.205 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:11:08.205 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:11:08.206 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:11:08.255 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:11:08.256 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:11:08.257 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:11:08.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:08.284 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:11:08.285 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:11:08.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77b4053ad0 msgr2=0x7f77b4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77b4053ad0 0x7f77b4055f90 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f77d000b000 tx=0x7f77d0009040).stop 2022-07-20T03:11:08.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 msgr2=0x7f77d413a430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 0x7f77d413a430 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f77cc0032f0 tx=0x7f77cc016000).stop 2022-07-20T03:11:08.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 shutdown_connections 2022-07-20T03:11:08.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77b4053ad0 0x7f77b4055f90 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77d40b1620 0x7f77d413a430 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77d4062cf0 0x7f77d40ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 --2- 172.21.15.61:0/1591286424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77d4060ed0 0x7f77d4155a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 >> 172.21.15.61:0/1591286424 conn(0x7f77d40b23d0 msgr2=0x7f77d4143100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:08.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 shutdown_connections 2022-07-20T03:11:08.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:07.679+0000 7f77db82c700 1 -- 172.21.15.61:0/1591286424 wait complete. 2022-07-20T03:11:08.405 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:11:08.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.575+0000 7f62fc926700 1 Processor -- start 2022-07-20T03:11:08.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.575+0000 7f62fc926700 1 -- start start 2022-07-20T03:11:08.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80af8e0 0x7f62f80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:08.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f8147010 con 0x7f62f80b5650 2022-07-20T03:11:08.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f62f81472b0 con 0x7f62f80adff0 2022-07-20T03:11:08.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8146d40 con 0x7f62f80af8e0 2022-07-20T03:11:08.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f6d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f62f80b5650 0x7f62f8147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41646/0 (socket says 172.21.15.61:41646) 2022-07-20T03:11:08.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f6d9d700 1 -- 172.21.15.61:0/4193223680 learned_addr learned my addr 172.21.15.61:0/4193223680 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f759e700 1 --2- 172.21.15.61:0/4193223680 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80af8e0 0x7f62f80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:08.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f5d9b700 1 -- 172.21.15.61:0/4193223680 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2004115684 0 0) 0x7f62f8147010 con 0x7f62f80b5650 2022-07-20T03:11:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f5d9b700 1 -- 172.21.15.61:0/4193223680 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f62dc003260 con 0x7f62f80b5650 2022-07-20T03:11:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f759e700 1 -- 172.21.15.61:0/4193223680 >> v1:172.21.15.117:6789/0 conn(0x7f62f80adff0 legacy=0x7f62f805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f759e700 1 -- 172.21.15.61:0/4193223680 >> v1:172.21.15.61:6789/0 conn(0x7f62f80b5650 legacy=0x7f62f8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f759e700 1 -- 172.21.15.61:0/4193223680 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62f805c4a0 con 0x7f62f80af8e0 2022-07-20T03:11:08.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f759e700 1 --2- 172.21.15.61:0/4193223680 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80af8e0 0x7f62f80b5a30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f62e8002f70 tx=0x7f62e8006df0).ready entity=mon.2 client_cookie=4fe5eaea3facfe31 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:08.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f5d9b700 1 -- 172.21.15.61:0/4193223680 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62e800d570 con 0x7f62f80af8e0 2022-07-20T03:11:08.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f5d9b700 1 -- 172.21.15.61:0/4193223680 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62e8000e90 con 0x7f62f80af8e0 2022-07-20T03:11:08.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62f5d9b700 1 -- 172.21.15.61:0/4193223680 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62e800d8c0 con 0x7f62f80af8e0 2022-07-20T03:11:08.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- 172.21.15.61:0/4193223680 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80af8e0 msgr2=0x7f62f80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 --2- 172.21.15.61:0/4193223680 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80af8e0 0x7f62f80b5a30 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f62e8002f70 tx=0x7f62e8006df0).stop 2022-07-20T03:11:08.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- 172.21.15.61:0/4193223680 shutdown_connections 2022-07-20T03:11:08.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 --2- 172.21.15.61:0/4193223680 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80af8e0 0x7f62f80b5a30 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- 172.21.15.61:0/4193223680 >> 172.21.15.61:0/4193223680 conn(0x7f62f80634e0 msgr2=0x7f62f80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:08.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- 172.21.15.61:0/4193223680 shutdown_connections 2022-07-20T03:11:08.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.579+0000 7f62fc926700 1 -- 172.21.15.61:0/4193223680 wait complete. 2022-07-20T03:11:08.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 Processor -- start 2022-07-20T03:11:08.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 -- start start 2022-07-20T03:11:08.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 0x7f62f80675c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:08.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f80af8e0 0x7f62f8110fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:08.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f80b5650 0x7f62f810f670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:08.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f810eb60 con 0x7f62f80af8e0 2022-07-20T03:11:08.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62f8067000 con 0x7f62f80b5650 2022-07-20T03:11:08.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62fc926700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f62f810e420 con 0x7f62f80adff0 2022-07-20T03:11:08.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.583+0000 7f62f659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f80af8e0 0x7f62f8110fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:08.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 0x7f62f80675c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:08.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 0x7f62f80675c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:59958/0 (socket says 172.21.15.61:59958) 2022-07-20T03:11:08.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 -- 172.21.15.61:0/3115106213 learned_addr learned my addr 172.21.15.61:0/3115106213 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:08.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f759e700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f80b5650 0x7f62f810f670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:08.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 -- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f80b5650 msgr2=0x7f62f810f670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f80b5650 0x7f62f810f670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 -- 172.21.15.61:0/3115106213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f80af8e0 msgr2=0x7f62f8110fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f80af8e0 0x7f62f8110fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 -- 172.21.15.61:0/3115106213 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62e800b180 con 0x7f62f80adff0 2022-07-20T03:11:08.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f659c700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f80af8e0 0x7f62f8110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:08.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f759e700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f80b5650 0x7f62f810f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:08.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62f6d9d700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 0x7f62f80675c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f62f810eca0 tx=0x7f62f0007e50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:08.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f0005ba0 con 0x7f62f80adff0 2022-07-20T03:11:08.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f62f0005d00 con 0x7f62f80adff0 2022-07-20T03:11:08.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62f810e6c0 con 0x7f62f80adff0 2022-07-20T03:11:08.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f62f00072c0 con 0x7f62f80adff0 2022-07-20T03:11:08.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62f8111490 con 0x7f62f80adff0 2022-07-20T03:11:08.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62f8066e00 con 0x7f62f80adff0 2022-07-20T03:11:08.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.587+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f62f0010070 con 0x7f62f80adff0 2022-07-20T03:11:08.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.591+0000 7f62e7fff700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62d4053a40 0x7f62d4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:08.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.591+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f62f006b3f0 con 0x7f62f80adff0 2022-07-20T03:11:08.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.591+0000 7f62f659c700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62d4053a40 0x7f62d4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:08.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.591+0000 7f62f659c700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62d4053a40 0x7f62d4055f00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f62ec00d000 tx=0x7f62ec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:08.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.599+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f62f0042680 con 0x7f62f80adff0 2022-07-20T03:11:08.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.727+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f62f80574e0 con 0x7f62f80adff0 2022-07-20T03:11:08.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.727+0000 7f62e7fff700 1 -- 172.21.15.61:0/3115106213 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v47) v1 ==== 74+0+13229 (secure 0 0 0) 0x7f62f0045ca0 con 0x7f62f80adff0 2022-07-20T03:11:08.916 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:08.916 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":47,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:11:07.276320+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":8,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":30,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":40,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":38,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:11:08.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62d4053a40 msgr2=0x7f62d4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62d4053a40 0x7f62d4055f00 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f62ec00d000 tx=0x7f62ec009040).stop 2022-07-20T03:11:08.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 msgr2=0x7f62f80675c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:08.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 0x7f62f80675c0 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7f62f810eca0 tx=0x7f62f0007e50).stop 2022-07-20T03:11:08.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 shutdown_connections 2022-07-20T03:11:08.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f62d4053a40 0x7f62d4055f00 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f62f80b5650 0x7f62f810f670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f62f80af8e0 0x7f62f8110fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 --2- 172.21.15.61:0/3115106213 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f62f80adff0 0x7f62f80675c0 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:08.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 >> 172.21.15.61:0/3115106213 conn(0x7f62f80634e0 msgr2=0x7f62f81462c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:08.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 shutdown_connections 2022-07-20T03:11:08.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:08.731+0000 7f62fc926700 1 -- 172.21.15.61:0/3115106213 wait complete. 2022-07-20T03:11:08.930 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:11:08.931 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:11:09.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.071+0000 7f0f32378700 1 Processor -- start 2022-07-20T03:11:09.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.071+0000 7f0f32378700 1 -- start start 2022-07-20T03:11:09.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0f2c0b9da0 0x7f0f2c1468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0f2c0b3cf0 0x7f0f2c142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0f2c0ba180 con 0x7f0f2c142f40 2022-07-20T03:11:09.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c0ad5b0 con 0x7f0f2c0b3cf0 2022-07-20T03:11:09.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0f2c1461b0 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0f2c0b3cf0 0x7f0f2c142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0f2c0b3cf0 0x7f0f2c142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45062/0 (socket says 172.21.15.61:45062) 2022-07-20T03:11:09.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 -- 172.21.15.61:0/3862317157 learned_addr learned my addr 172.21.15.61:0/3862317157 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:09.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f31376700 1 --2- 172.21.15.61:0/3862317157 >> v2:172.21.15.117:3300/0 conn(0x7f0f2c0b9da0 0x7f0f2c1468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 -- 172.21.15.61:0/3862317157 >> v2:172.21.15.117:3300/0 conn(0x7f0f2c0b9da0 msgr2=0x7f0f2c1468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:09.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 --2- 172.21.15.61:0/3862317157 >> v2:172.21.15.117:3300/0 conn(0x7f0f2c0b9da0 0x7f0f2c1468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 -- 172.21.15.61:0/3862317157 >> v1:172.21.15.117:6789/0 conn(0x7f0f2c142f40 legacy=0x7f0f2c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:09.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 -- 172.21.15.61:0/3862317157 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f2c146450 con 0x7f0f2c0b3cf0 2022-07-20T03:11:09.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f30b75700 1 --2- 172.21.15.61:0/3862317157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b3cf0 0x7f0f2c142a50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f0f2c066cd0 tx=0x7f0f1c006eb0).ready entity=mon.0 client_cookie=7657d7c928d23e62 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f23fff700 1 -- 172.21.15.61:0/3862317157 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f1c00d5c0 con 0x7f0f2c0b3cf0 2022-07-20T03:11:09.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f23fff700 1 -- 172.21.15.61:0/3862317157 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c000e90 con 0x7f0f2c0b3cf0 2022-07-20T03:11:09.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f23fff700 1 -- 172.21.15.61:0/3862317157 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f1c00d910 con 0x7f0f2c0b3cf0 2022-07-20T03:11:09.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- 172.21.15.61:0/3862317157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b3cf0 msgr2=0x7f0f2c142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 --2- 172.21.15.61:0/3862317157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b3cf0 0x7f0f2c142a50 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f0f2c066cd0 tx=0x7f0f1c006eb0).stop 2022-07-20T03:11:09.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- 172.21.15.61:0/3862317157 shutdown_connections 2022-07-20T03:11:09.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 --2- 172.21.15.61:0/3862317157 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b3cf0 0x7f0f2c142a50 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 --2- 172.21.15.61:0/3862317157 >> v2:172.21.15.117:3300/0 conn(0x7f0f2c0b9da0 0x7f0f2c1468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- 172.21.15.61:0/3862317157 >> 172.21.15.61:0/3862317157 conn(0x7f0f2c0678c0 msgr2=0x7f0f2c0b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:09.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- 172.21.15.61:0/3862317157 shutdown_connections 2022-07-20T03:11:09.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.075+0000 7f0f32378700 1 -- 172.21.15.61:0/3862317157 wait complete. 2022-07-20T03:11:09.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 Processor -- start 2022-07-20T03:11:09.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 -- start start 2022-07-20T03:11:09.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0f2c0b3cf0 0x7f0f2c113290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 0x7f0f2c144270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f2c142f40 0x7f0f2c151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c1128b0 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c112b50 con 0x7f0f2c142f40 2022-07-20T03:11:09.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f32378700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0f2c112df0 con 0x7f0f2c0b3cf0 2022-07-20T03:11:09.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 0x7f0f2c144270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 0x7f0f2c144270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45068/0 (socket says 172.21.15.61:45068) 2022-07-20T03:11:09.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 -- 172.21.15.61:0/1170706033 learned_addr learned my addr 172.21.15.61:0/1170706033 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:09.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f31376700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0f2c0b3cf0 0x7f0f2c113290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f31b77700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f2c142f40 0x7f0f2c151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 -- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0f2c0b3cf0 msgr2=0x7f0f2c113290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0f2c0b3cf0 0x7f0f2c113290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 -- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f2c142f40 msgr2=0x7f0f2c151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f2c142f40 0x7f0f2c151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 -- 172.21.15.61:0/1170706033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c00b1e0 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f30b75700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 0x7f0f2c144270 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f0f2c066cd0 tx=0x7f0f1c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f1c00bec0 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.079+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0f1c00de80 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0f1c011d70 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f2c0b1fa0 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f2c145c70 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f2c0b1190 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0f1c01a040 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f227fc700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0f14053a40 0x7f0f14055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f0f1c06c940 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f31376700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0f14053a40 0x7f0f14055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.083+0000 7f0f31376700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0f14053a40 0x7f0f14055f00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0f2800c000 tx=0x7f0f28008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.091+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0f2c0b1190 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.219+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f0f2c0b1190 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.219+0000 7f0f227fc700 1 -- 172.21.15.61:0/1170706033 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v47) v1 ==== 110+0+10 (secure 0 0 0) 0x7f0f2c0b1190 con 0x7f0f2c0b9da0 2022-07-20T03:11:09.336 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:11:09.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0f14053a40 msgr2=0x7f0f14055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0f14053a40 0x7f0f14055f00 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0f2800c000 tx=0x7f0f28008040).stop 2022-07-20T03:11:09.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 msgr2=0x7f0f2c144270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 0x7f0f2c144270 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7f0f2c066cd0 tx=0x7f0f1c015000).stop 2022-07-20T03:11:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 shutdown_connections 2022-07-20T03:11:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0f14053a40 0x7f0f14055f00 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0f2c142f40 0x7f0f2c151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0f2c0b9da0 0x7f0f2c144270 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 --2- 172.21.15.61:0/1170706033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0f2c0b3cf0 0x7f0f2c113290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 >> 172.21.15.61:0/1170706033 conn(0x7f0f2c0678c0 msgr2=0x7f0f2c0b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 shutdown_connections 2022-07-20T03:11:09.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.223+0000 7f0f32378700 1 -- 172.21.15.61:0/1170706033 wait complete. 2022-07-20T03:11:09.389 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:11:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6f15700 1 Processor -- start 2022-07-20T03:11:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6f15700 1 -- start start 2022-07-20T03:11:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6f15700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f95a00aea20 0x7f95a00b1480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6f15700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a00af3f0 con 0x7f95a00afd70 2022-07-20T03:11:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6f15700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f95a013fff0 con 0x7f95a00acae0 2022-07-20T03:11:09.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6f15700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f95a00af150 con 0x7f95a00aea20 2022-07-20T03:11:09.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a5f13700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f95a00aea20 0x7f95a00b1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6714700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f95a00afd70 0x7f95a013de40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41674/0 (socket says 172.21.15.61:41674) 2022-07-20T03:11:09.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a6714700 1 -- 172.21.15.61:0/968126135 learned_addr learned my addr 172.21.15.61:0/968126135 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:09.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 751831543 0 0) 0x7f95a00af3f0 con 0x7f95a00afd70 2022-07-20T03:11:09.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9588003260 con 0x7f95a00afd70 2022-07-20T03:11:09.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.543+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2226433291 0 0) 0x7f95a013fff0 con 0x7f95a00acae0 2022-07-20T03:11:09.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f95a00af3f0 con 0x7f95a00acae0 2022-07-20T03:11:09.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a5f13700 1 -- 172.21.15.61:0/968126135 >> v1:172.21.15.117:6789/0 conn(0x7f95a00acae0 legacy=0x7f95a013f8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a5f13700 1 -- 172.21.15.61:0/968126135 >> v1:172.21.15.61:6789/0 conn(0x7f95a00afd70 legacy=0x7f95a013de40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a5f13700 1 -- 172.21.15.61:0/968126135 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95a00b0720 con 0x7f95a00aea20 2022-07-20T03:11:09.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a5f13700 1 --2- 172.21.15.61:0/968126135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00aea20 0x7f95a00b1480 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f9594002f70 tx=0x7f9594006df0).ready entity=mon.0 client_cookie=75fa74e4c438ba75 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f959400d540 con 0x7f95a00aea20 2022-07-20T03:11:09.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9594000e90 con 0x7f95a00aea20 2022-07-20T03:11:09.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a4f11700 1 -- 172.21.15.61:0/968126135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f959400d890 con 0x7f95a00aea20 2022-07-20T03:11:09.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 -- 172.21.15.61:0/968126135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00aea20 msgr2=0x7f95a00b1480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 --2- 172.21.15.61:0/968126135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00aea20 0x7f95a00b1480 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f9594002f70 tx=0x7f9594006df0).stop 2022-07-20T03:11:09.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 -- 172.21.15.61:0/968126135 shutdown_connections 2022-07-20T03:11:09.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 --2- 172.21.15.61:0/968126135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00aea20 0x7f95a00b1480 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 -- 172.21.15.61:0/968126135 >> 172.21.15.61:0/968126135 conn(0x7f95a00634c0 msgr2=0x7f95a00b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:09.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 -- 172.21.15.61:0/968126135 shutdown_connections 2022-07-20T03:11:09.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 -- 172.21.15.61:0/968126135 wait complete. 2022-07-20T03:11:09.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 Processor -- start 2022-07-20T03:11:09.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.547+0000 7f95a6f15700 1 -- start start 2022-07-20T03:11:09.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f95a00acae0 0x7f95a0142880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a00aea20 0x7f95a013d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 0x7f95a013c460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0067900 con 0x7f95a00afd70 2022-07-20T03:11:09.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0065df0 con 0x7f95a00aea20 2022-07-20T03:11:09.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f95a0066090 con 0x7f95a00acae0 2022-07-20T03:11:09.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6714700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 0x7f95a013c460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6714700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 0x7f95a013c460 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45084/0 (socket says 172.21.15.61:45084) 2022-07-20T03:11:09.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6714700 1 -- 172.21.15.61:0/2972290520 learned_addr learned my addr 172.21.15.61:0/2972290520 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:09.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f95a00acae0 0x7f95a0142880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5712700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a00aea20 0x7f95a013d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 -- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a00aea20 msgr2=0x7f95a013d240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a00aea20 0x7f95a013d240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 -- 172.21.15.61:0/2972290520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 msgr2=0x7f95a013c460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 0x7f95a013c460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 -- 172.21.15.61:0/2972290520 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f959400b180 con 0x7f95a00acae0 2022-07-20T03:11:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6714700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 0x7f95a013c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5712700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a00aea20 0x7f95a013d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:09.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a5f13700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f95a00acae0 0x7f95a0142880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f95a0067a40 tx=0x7f9594014000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9594013da0 con 0x7f95a00acae0 2022-07-20T03:11:09.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f959400d1c0 con 0x7f95a00acae0 2022-07-20T03:11:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95a0142e40 con 0x7f95a00acae0 2022-07-20T03:11:09.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f959401e5f0 con 0x7f95a00acae0 2022-07-20T03:11:09.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.551+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95a0066750 con 0x7f95a00acae0 2022-07-20T03:11:09.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.555+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95a0066250 con 0x7f95a00acae0 2022-07-20T03:11:09.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.555+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9594025030 con 0x7f95a00acae0 2022-07-20T03:11:09.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.555+0000 7f9592ffd700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9580053af0 0x7f9580055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.555+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f959406c540 con 0x7f95a00acae0 2022-07-20T03:11:09.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.555+0000 7f95a5712700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9580053af0 0x7f9580055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.555+0000 7f95a5712700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9580053af0 0x7f9580055fb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f959c00d000 tx=0x7f959c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.563+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f95940437d0 con 0x7f95a00acae0 2022-07-20T03:11:09.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.695+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f95a00b01e0 con 0x7f95a00acae0 2022-07-20T03:11:09.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.695+0000 7f9592ffd700 1 -- 172.21.15.61:0/2972290520 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v47) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9594023020 con 0x7f95a00acae0 2022-07-20T03:11:09.711 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:09.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9580053af0 msgr2=0x7f9580055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9580053af0 0x7f9580055fb0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f959c00d000 tx=0x7f959c007a20).stop 2022-07-20T03:11:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f95a00acae0 msgr2=0x7f95a0142880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f95a00acae0 0x7f95a0142880 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f95a0067a40 tx=0x7f9594014000).stop 2022-07-20T03:11:09.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 shutdown_connections 2022-07-20T03:11:09.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9580053af0 0x7f9580055fb0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f95a00aea20 0x7f95a013d240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f95a00afd70 0x7f95a013c460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 --2- 172.21.15.61:0/2972290520 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f95a00acae0 0x7f95a0142880 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 >> 172.21.15.61:0/2972290520 conn(0x7f95a00634c0 msgr2=0x7f95a00b3850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 shutdown_connections 2022-07-20T03:11:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.699+0000 7f95a6f15700 1 -- 172.21.15.61:0/2972290520 wait complete. 2022-07-20T03:11:09.737 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:11:09.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.891+0000 7f703b7b4700 1 Processor -- start 2022-07-20T03:11:09.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.891+0000 7f703b7b4700 1 -- start start 2022-07-20T03:11:09.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f703405c3f0 0x7f703414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f70340acef0 0x7f7034146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:09.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f703405d480 con 0x7f70340b1e00 2022-07-20T03:11:09.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70340b21e0 con 0x7f70340acef0 2022-07-20T03:11:09.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703405cf40 con 0x7f703405c3f0 2022-07-20T03:11:09.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039d51700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f70340acef0 0x7f7034146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039550700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f70340b1e00 0x7f70340ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41688/0 (socket says 172.21.15.61:41688) 2022-07-20T03:11:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039550700 1 -- 172.21.15.61:0/1791026746 learned_addr learned my addr 172.21.15.61:0/1791026746 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7038d4f700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f703405c3f0 0x7f703414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039d51700 1 -- 172.21.15.61:0/1791026746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f703405c3f0 msgr2=0x7f703414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039d51700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f703405c3f0 0x7f703414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039d51700 1 -- 172.21.15.61:0/1791026746 >> v1:172.21.15.61:6789/0 conn(0x7f70340b1e00 legacy=0x7f70340ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f702bfff700 1 -- 172.21.15.61:0/1791026746 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1887872029 0 0) 0x7f703405d480 con 0x7f70340b1e00 2022-07-20T03:11:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039d51700 1 -- 172.21.15.61:0/1791026746 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703405d1e0 con 0x7f70340acef0 2022-07-20T03:11:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7038d4f700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f703405c3f0 0x7f703414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f7039d51700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340acef0 0x7f7034146410 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f702c0032f0 tx=0x7f702c0091d0).ready entity=mon.0 client_cookie=3620ebe2fe3a1370 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f702bfff700 1 -- 172.21.15.61:0/1791026746 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f702c00b6c0 con 0x7f70340acef0 2022-07-20T03:11:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f702bfff700 1 -- 172.21.15.61:0/1791026746 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c00b820 con 0x7f70340acef0 2022-07-20T03:11:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f702bfff700 1 -- 172.21.15.61:0/1791026746 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f702c00b990 con 0x7f70340acef0 2022-07-20T03:11:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- 172.21.15.61:0/1791026746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340acef0 msgr2=0x7f7034146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340acef0 0x7f7034146410 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f702c0032f0 tx=0x7f702c0091d0).stop 2022-07-20T03:11:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- 172.21.15.61:0/1791026746 shutdown_connections 2022-07-20T03:11:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340acef0 0x7f7034146410 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 --2- 172.21.15.61:0/1791026746 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f703405c3f0 0x7f703414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- 172.21.15.61:0/1791026746 >> 172.21.15.61:0/1791026746 conn(0x7f70340611c0 msgr2=0x7f7034062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:10.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- 172.21.15.61:0/1791026746 shutdown_connections 2022-07-20T03:11:10.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.895+0000 7f703b7b4700 1 -- 172.21.15.61:0/1791026746 wait complete. 2022-07-20T03:11:10.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 Processor -- start 2022-07-20T03:11:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 -- start start 2022-07-20T03:11:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70340acef0 0x7f7034149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70340b1e00 0x7f703414bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 0x7f70340a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7034149cd0 con 0x7f70340a06a0 2022-07-20T03:11:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f703409e950 con 0x7f70340acef0 2022-07-20T03:11:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f703409e5f0 con 0x7f70340b1e00 2022-07-20T03:11:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 0x7f70340a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039550700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70340acef0 0x7f7034149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 0x7f70340a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45098/0 (socket says 172.21.15.61:45098) 2022-07-20T03:11:10.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 -- 172.21.15.61:0/2146249249 learned_addr learned my addr 172.21.15.61:0/2146249249 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:10.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7038d4f700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70340b1e00 0x7f703414bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 -- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70340b1e00 msgr2=0x7f703414bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70340b1e00 0x7f703414bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 -- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70340acef0 msgr2=0x7f7034149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70340acef0 0x7f7034149ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 -- 172.21.15.61:0/2146249249 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f703000b180 con 0x7f70340a06a0 2022-07-20T03:11:10.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7038d4f700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70340b1e00 0x7f703414bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:10.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f7039d51700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 0x7f70340a0b20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f702c0032f0 tx=0x7f702c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f702c00b6c0 con 0x7f70340a06a0 2022-07-20T03:11:10.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f702c021630 con 0x7f70340a06a0 2022-07-20T03:11:10.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f702c009470 con 0x7f70340a06a0 2022-07-20T03:11:10.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f702c021990 con 0x7f70340a06a0 2022-07-20T03:11:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.899+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f703409ee40 con 0x7f70340a06a0 2022-07-20T03:11:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.903+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70340b04b0 con 0x7f70340a06a0 2022-07-20T03:11:10.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.903+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f702c027020 con 0x7f70340a06a0 2022-07-20T03:11:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.903+0000 7f702a7fc700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f701c053ad0 0x7f701c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.903+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(47..47 src has 1..47) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f702c06cff0 con 0x7f70340a06a0 2022-07-20T03:11:10.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.903+0000 7f7039550700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f701c053ad0 0x7f701c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.903+0000 7f7039550700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f701c053ad0 0x7f701c055f90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f702400d000 tx=0x7f7024007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:10.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:09.911+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f702c0441d0 con 0x7f70340a06a0 2022-07-20T03:11:10.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.039+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f70340574e0 con 0x7f70340a06a0 2022-07-20T03:11:10.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.039+0000 7f702a7fc700 1 -- 172.21.15.61:0/2146249249 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v47) v1 ==== 84+0+143 (secure 0 0 0) 0x7f702c015090 con 0x7f70340a06a0 2022-07-20T03:11:10.341 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:10.342 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:11:10.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:10.342 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:11:10.342 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:11:10.364 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:10.364 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:11:10.365 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:11:10.365 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:11:10.365 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:10.377 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:11:10.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f701c053ad0 msgr2=0x7f701c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f701c053ad0 0x7f701c055f90 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f702400d000 tx=0x7f7024007c20).stop 2022-07-20T03:11:10.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 msgr2=0x7f70340a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 0x7f70340a0b20 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7f702c0032f0 tx=0x7f702c016000).stop 2022-07-20T03:11:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 shutdown_connections 2022-07-20T03:11:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f701c053ad0 0x7f701c055f90 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70340a06a0 0x7f70340a0b20 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70340b1e00 0x7f703414bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 --2- 172.21.15.61:0/2146249249 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70340acef0 0x7f7034149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 >> 172.21.15.61:0/2146249249 conn(0x7f703405c3f0 msgr2=0x7f7034069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:10.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 shutdown_connections 2022-07-20T03:11:10.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.043+0000 7f703b7b4700 1 -- 172.21.15.61:0/2146249249 wait complete. 2022-07-20T03:11:10.469 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:11:10.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:11:10.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 Processor -- start 2022-07-20T03:11:10.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 -- start start 2022-07-20T03:11:10.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 0x7fd7001473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd7000b1400 0x7fd700147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd70005cc70 con 0x7fd7000af940 2022-07-20T03:11:10.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7000b1070 con 0x7fd7000b1400 2022-07-20T03:11:10.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7000b5dd0 con 0x7fd7000b59f0 2022-07-20T03:11:10.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6feffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 0x7fd7001473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6ff7fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd7000b1400 0x7fd700147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6feffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 0x7fd7001473a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:60006/0 (socket says 172.21.15.61:60006) 2022-07-20T03:11:10.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6feffd700 1 -- 172.21.15.61:0/2193822821 learned_addr learned my addr 172.21.15.61:0/2193822821 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:10.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6ff7fe700 1 -- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 msgr2=0x7fd7001473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:10.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6ff7fe700 1 --2- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 0x7fd7001473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6ff7fe700 1 -- 172.21.15.61:0/2193822821 >> v1:172.21.15.117:6789/0 conn(0x7fd7000af940 legacy=0x7fd70014a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6ff7fe700 1 -- 172.21.15.61:0/2193822821 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd70005c760 con 0x7fd7000b1400 2022-07-20T03:11:10.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6feffd700 1 --2- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 0x7fd7001473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:10.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6ff7fe700 1 --2- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000b1400 0x7fd700147890 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd6f80032f0 tx=0x7fd6f80091d0).ready entity=mon.1 client_cookie=eae6d8588773962b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:10.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fe7fc700 1 -- 172.21.15.61:0/2193822821 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f800b6c0 con 0x7fd7000b1400 2022-07-20T03:11:10.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fe7fc700 1 -- 172.21.15.61:0/2193822821 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6f800b820 con 0x7fd7000b1400 2022-07-20T03:11:10.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.623+0000 7fd6fe7fc700 1 -- 172.21.15.61:0/2193822821 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6f800b990 con 0x7fd7000b1400 2022-07-20T03:11:10.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000b1400 msgr2=0x7fd700147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000b1400 0x7fd700147890 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fd6f80032f0 tx=0x7fd6f80091d0).stop 2022-07-20T03:11:10.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- 172.21.15.61:0/2193822821 shutdown_connections 2022-07-20T03:11:10.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000b1400 0x7fd700147890 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2193822821 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b59f0 0x7fd7001473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- 172.21.15.61:0/2193822821 >> 172.21.15.61:0/2193822821 conn(0x7fd700063480 msgr2=0x7fd7000b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:10.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- 172.21.15.61:0/2193822821 shutdown_connections 2022-07-20T03:11:10.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- 172.21.15.61:0/2193822821 wait complete. 2022-07-20T03:11:10.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 Processor -- start 2022-07-20T03:11:10.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- start start 2022-07-20T03:11:10.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000af940 0x7fd700063e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 0x7fd70013d5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7000b59f0 0x7fd70013dae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd700066bc0 con 0x7fd7000b59f0 2022-07-20T03:11:10.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd700067390 con 0x7fd7000af940 2022-07-20T03:11:10.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6fffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd70010e2e0 con 0x7fd7000b1400 2022-07-20T03:11:10.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 0x7fd70013d5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7000b59f0 0x7fd70013dae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 0x7fd70013d5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:60014/0 (socket says 172.21.15.61:60014) 2022-07-20T03:11:10.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 -- 172.21.15.61:0/2504914859 learned_addr learned my addr 172.21.15.61:0/2504914859 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:10.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6feffd700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000af940 0x7fd700063e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 -- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000af940 msgr2=0x7fd700063e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000af940 0x7fd700063e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 -- 172.21.15.61:0/2504914859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7000b59f0 msgr2=0x7fd70013dae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7000b59f0 0x7fd70013dae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 -- 172.21.15.61:0/2504914859 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6f000b1e0 con 0x7fd7000b1400 2022-07-20T03:11:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.627+0000 7fd6f67fc700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 0x7fd70013d5f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd700147280 tx=0x7fd6e8007bd0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e8007030 con 0x7fd7000b1400 2022-07-20T03:11:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd6e8007190 con 0x7fd7000b1400 2022-07-20T03:11:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8009470 con 0x7fd7000b1400 2022-07-20T03:11:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6e8006510 con 0x7fd7000b1400 2022-07-20T03:11:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd70013dfd0 con 0x7fd7000b1400 2022-07-20T03:11:10.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd700066dc0 con 0x7fd7000b1400 2022-07-20T03:11:10.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd6e801b070 con 0x7fd7000b1400 2022-07-20T03:11:10.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fcff9700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd6e4053ad0 0x7fd6e4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:10.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(48..48 src has 1..48) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fd6e806b1c0 con 0x7fd7000b1400 2022-07-20T03:11:10.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.631+0000 7fd6feffd700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd6e4053ad0 0x7fd6e4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:10.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.635+0000 7fd6feffd700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd6e4053ad0 0x7fd6e4055f90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fd6f000b000 tx=0x7fd6f0006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:10.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.639+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd6e8042340 con 0x7fd7000b1400 2022-07-20T03:11:10.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:10.767+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fd70005c9d0 con 0x7fd7000b1400 2022-07-20T03:11:11.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.327+0000 7fd6fcff9700 1 -- 172.21.15.61:0/2504914859 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v49) v1 ==== 133+0+0 (secure 0 0 0) 0x7fd6e8045960 con 0x7fd7000b1400 2022-07-20T03:11:11.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd6e4053ad0 msgr2=0x7fd6e4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:11.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd6e4053ad0 0x7fd6e4055f90 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7fd6f000b000 tx=0x7fd6f0006eb0).stop 2022-07-20T03:11:11.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 msgr2=0x7fd70013d5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:11.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 0x7fd70013d5f0 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7fd700147280 tx=0x7fd6e8007bd0).stop 2022-07-20T03:11:11.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 shutdown_connections 2022-07-20T03:11:11.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd6e4053ad0 0x7fd6e4055f90 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7000b59f0 0x7fd70013dae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7000b1400 0x7fd70013d5f0 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 --2- 172.21.15.61:0/2504914859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7000af940 0x7fd700063e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 >> 172.21.15.61:0/2504914859 conn(0x7fd700063480 msgr2=0x7fd7000adc90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:11.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 shutdown_connections 2022-07-20T03:11:11.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.331+0000 7fd6fffff700 1 -- 172.21.15.61:0/2504914859 wait complete. 2022-07-20T03:11:11.354 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:11:11.381 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:11:11.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.503+0000 7f148f59e700 1 Processor -- start 2022-07-20T03:11:11.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.503+0000 7f148f59e700 1 -- start start 2022-07-20T03:11:11.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f14900ac610 0x7f14900aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:11.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 0x7f149005fad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:11.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14900aced0 con 0x7f14900ac9f0 2022-07-20T03:11:11.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1490143c80 con 0x7f14900ac610 2022-07-20T03:11:11.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1490143f20 con 0x7f14900b0b90 2022-07-20T03:11:11.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 0x7f149005fad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 0x7f149005fad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:60022/0 (socket says 172.21.15.61:60022) 2022-07-20T03:11:11.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 -- 172.21.15.61:0/1186067625 learned_addr learned my addr 172.21.15.61:0/1186067625 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148e59c700 1 --2- 172.21.15.61:0/1186067625 >> v2:172.21.15.117:3300/0 conn(0x7f14900ac610 0x7f14900aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:11.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148ed9d700 1 --1- 172.21.15.61:0/1186067625 >> v1:172.21.15.117:6789/0 conn(0x7f14900ac9f0 0x7f1490140520 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:50224/0 (socket says 172.21.15.61:50224) 2022-07-20T03:11:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 -- 172.21.15.61:0/1186067625 >> v2:172.21.15.117:3300/0 conn(0x7f14900ac610 msgr2=0x7f14900aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 --2- 172.21.15.61:0/1186067625 >> v2:172.21.15.117:3300/0 conn(0x7f14900ac610 0x7f14900aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 -- 172.21.15.61:0/1186067625 >> v1:172.21.15.117:6789/0 conn(0x7f14900ac9f0 legacy=0x7f1490140520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 -- 172.21.15.61:0/1186067625 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f149005ffc0 con 0x7f14900b0b90 2022-07-20T03:11:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148d59a700 1 -- 172.21.15.61:0/1186067625 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1741744704 0 0) 0x7f14900aced0 con 0x7f14900ac9f0 2022-07-20T03:11:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148dd9b700 1 --2- 172.21.15.61:0/1186067625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 0x7f149005fad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1480002f70 tx=0x7f1480006f40).ready entity=mon.2 client_cookie=aa73422fc0322c42 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:11.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148d59a700 1 -- 172.21.15.61:0/1186067625 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f148000da80 con 0x7f14900b0b90 2022-07-20T03:11:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148d59a700 1 -- 172.21.15.61:0/1186067625 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1480000e90 con 0x7f14900b0b90 2022-07-20T03:11:11.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148d59a700 1 -- 172.21.15.61:0/1186067625 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f148000ddd0 con 0x7f14900b0b90 2022-07-20T03:11:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- 172.21.15.61:0/1186067625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 msgr2=0x7f149005fad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 --2- 172.21.15.61:0/1186067625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 0x7f149005fad0 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f1480002f70 tx=0x7f1480006f40).stop 2022-07-20T03:11:11.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- 172.21.15.61:0/1186067625 shutdown_connections 2022-07-20T03:11:11.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 --2- 172.21.15.61:0/1186067625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900b0b90 0x7f149005fad0 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 --2- 172.21.15.61:0/1186067625 >> v2:172.21.15.117:3300/0 conn(0x7f14900ac610 0x7f14900aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- 172.21.15.61:0/1186067625 >> 172.21.15.61:0/1186067625 conn(0x7f14900b61f0 msgr2=0x7f14900b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:11.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- 172.21.15.61:0/1186067625 shutdown_connections 2022-07-20T03:11:11.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.507+0000 7f148f59e700 1 -- 172.21.15.61:0/1186067625 wait complete. 2022-07-20T03:11:11.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 Processor -- start 2022-07-20T03:11:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 -- start start 2022-07-20T03:11:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900ac610 0x7f1490145480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:11.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14900ac9f0 0x7f14900adc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14900b0b90 0x7f14900ae160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:11.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1490143700 con 0x7f14900b0b90 2022-07-20T03:11:11.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1490064c90 con 0x7f14900ac9f0 2022-07-20T03:11:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14901428a0 con 0x7f14900ac610 2022-07-20T03:11:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14900b0b90 0x7f14900ae160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:11.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148dd9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14900ac9f0 0x7f14900adc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148dd9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14900ac9f0 0x7f14900adc70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54806/0 (socket says 172.21.15.61:54806) 2022-07-20T03:11:11.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148dd9b700 1 -- 172.21.15.61:0/381430738 learned_addr learned my addr 172.21.15.61:0/381430738 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148e59c700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900ac610 0x7f1490145480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 -- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900ac610 msgr2=0x7f1490145480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:11.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900ac610 0x7f1490145480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 -- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14900ac9f0 msgr2=0x7f14900adc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:11.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14900ac9f0 0x7f14900adc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 -- 172.21.15.61:0/381430738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f148000b460 con 0x7f14900b0b90 2022-07-20T03:11:11.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148e59c700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900ac610 0x7f1490145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148ed9d700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14900b0b90 0x7f14900ae160 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f1488001250 tx=0x7f1488007d00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1488005ae0 con 0x7f14900b0b90 2022-07-20T03:11:11.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1488005c40 con 0x7f14900b0b90 2022-07-20T03:11:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1490064f30 con 0x7f14900b0b90 2022-07-20T03:11:11.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.511+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14880070d0 con 0x7f14900b0b90 2022-07-20T03:11:11.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.515+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1490142d40 con 0x7f14900b0b90 2022-07-20T03:11:11.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.515+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1490142a60 con 0x7f14900b0b90 2022-07-20T03:11:11.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.515+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1488010070 con 0x7f14900b0b90 2022-07-20T03:11:11.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.515+0000 7f147f7fe700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f146c053a40 0x7f146c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:11.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.515+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(49..49 src has 1..49) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f148806b5b0 con 0x7f14900b0b90 2022-07-20T03:11:11.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.515+0000 7f148e59c700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f146c053a40 0x7f146c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:11.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.519+0000 7f148e59c700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f146c053a40 0x7f146c055f00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f148400b000 tx=0x7f1484006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:11.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.523+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14880426e0 con 0x7f14900b0b90 2022-07-20T03:11:11.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:11.651+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f1490140c30 con 0x7f14900b0b90 2022-07-20T03:11:12.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.327+0000 7f147f7fe700 1 -- 172.21.15.61:0/381430738 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v50) v1 ==== 125+0+0 (secure 0 0 0) 0x7f1488045d00 con 0x7f14900b0b90 2022-07-20T03:11:12.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.331+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f146c053a40 msgr2=0x7f146c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.331+0000 7f148f59e700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f146c053a40 0x7f146c055f00 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f148400b000 tx=0x7f1484006df0).stop 2022-07-20T03:11:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.331+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14900b0b90 msgr2=0x7f14900ae160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.331+0000 7f148f59e700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14900b0b90 0x7f14900ae160 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f1488001250 tx=0x7f1488007d00).stop 2022-07-20T03:11:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 shutdown_connections 2022-07-20T03:11:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f146c053a40 0x7f146c055f00 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14900ac9f0 0x7f14900adc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14900b0b90 0x7f14900ae160 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 --2- 172.21.15.61:0/381430738 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14900ac610 0x7f1490145480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 >> 172.21.15.61:0/381430738 conn(0x7f14900b61f0 msgr2=0x7f149009b330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:12.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 shutdown_connections 2022-07-20T03:11:12.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.335+0000 7f148f59e700 1 -- 172.21.15.61:0/381430738 wait complete. 2022-07-20T03:11:12.363 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:11:12.364 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:11:12.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.483+0000 7fc22870a700 1 Processor -- start 2022-07-20T03:11:12.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.483+0000 7fc22870a700 1 -- start start 2022-07-20T03:11:12.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22005bd90 0x7fc2200ac600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc22013eed0 0x7fc22013f340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc2200ad070 con 0x7fc2200b1700 2022-07-20T03:11:12.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc2200acaf0 con 0x7fc22013eed0 2022-07-20T03:11:12.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200603e0 con 0x7fc22005bd90 2022-07-20T03:11:12.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc22013eed0 0x7fc22013f340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc22013eed0 0x7fc22013f340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45134/0 (socket says 172.21.15.61:45134) 2022-07-20T03:11:12.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 -- 172.21.15.61:0/3919222895 learned_addr learned my addr 172.21.15.61:0/3919222895 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:12.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc225ca5700 1 --2- 172.21.15.61:0/3919222895 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22005bd90 0x7fc2200ac600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 -- 172.21.15.61:0/3919222895 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22005bd90 msgr2=0x7fc2200ac600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:12.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 --2- 172.21.15.61:0/3919222895 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22005bd90 0x7fc2200ac600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 -- 172.21.15.61:0/3919222895 >> v1:172.21.15.117:6789/0 conn(0x7fc2200b1700 legacy=0x7fc2200abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 -- 172.21.15.61:0/3919222895 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc2200ace10 con 0x7fc22013eed0 2022-07-20T03:11:12.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc226ca7700 1 --2- 172.21.15.61:0/3919222895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22013eed0 0x7fc22013f340 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fc2180032f0 tx=0x7fc218009290).ready entity=mon.0 client_cookie=693f4287cb5645de server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:12.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc2254a4700 1 -- 172.21.15.61:0/3919222895 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21800b8a0 con 0x7fc22013eed0 2022-07-20T03:11:12.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc2254a4700 1 -- 172.21.15.61:0/3919222895 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc21800ba00 con 0x7fc22013eed0 2022-07-20T03:11:12.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc2254a4700 1 -- 172.21.15.61:0/3919222895 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc21800bb70 con 0x7fc22013eed0 2022-07-20T03:11:12.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- 172.21.15.61:0/3919222895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22013eed0 msgr2=0x7fc22013f340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 --2- 172.21.15.61:0/3919222895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22013eed0 0x7fc22013f340 secure :-1 s=READY pgs=479 cs=0 l=1 rev1=1 rx=0x7fc2180032f0 tx=0x7fc218009290).stop 2022-07-20T03:11:12.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- 172.21.15.61:0/3919222895 shutdown_connections 2022-07-20T03:11:12.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 --2- 172.21.15.61:0/3919222895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22013eed0 0x7fc22013f340 unknown :-1 s=CLOSED pgs=479 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 --2- 172.21.15.61:0/3919222895 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22005bd90 0x7fc2200ac600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- 172.21.15.61:0/3919222895 >> 172.21.15.61:0/3919222895 conn(0x7fc2200b5910 msgr2=0x7fc2200b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:12.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- 172.21.15.61:0/3919222895 shutdown_connections 2022-07-20T03:11:12.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.487+0000 7fc22870a700 1 -- 172.21.15.61:0/3919222895 wait complete. 2022-07-20T03:11:12.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.491+0000 7fc22870a700 1 Processor -- start 2022-07-20T03:11:12.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.491+0000 7fc22870a700 1 -- start start 2022-07-20T03:11:12.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 0x7fc2200b5130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2200b1700 0x7fc2200668a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22013eed0 0x7fc220155b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc2200644d0 con 0x7fc22005bd90 2022-07-20T03:11:12.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc220064770 con 0x7fc2200b1700 2022-07-20T03:11:12.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc220064a10 con 0x7fc22013eed0 2022-07-20T03:11:12.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 0x7fc2200b5130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 0x7fc2200b5130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45138/0 (socket says 172.21.15.61:45138) 2022-07-20T03:11:12.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 -- 172.21.15.61:0/3674848218 learned_addr learned my addr 172.21.15.61:0/3674848218 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:12.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc225ca5700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2200b1700 0x7fc2200668a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc226ca7700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22013eed0 0x7fc220155b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 -- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22013eed0 msgr2=0x7fc220155b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22013eed0 0x7fc220155b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 -- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2200b1700 msgr2=0x7fc2200668a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2200b1700 0x7fc2200668a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 -- 172.21.15.61:0/3674848218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc218009530 con 0x7fc22005bd90 2022-07-20T03:11:12.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc2264a6700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 0x7fc2200b5130 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fc220061b40 tx=0x7fc210007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:12.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc210012400 con 0x7fc22005bd90 2022-07-20T03:11:12.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc210012560 con 0x7fc22005bd90 2022-07-20T03:11:12.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc220135d00 con 0x7fc22005bd90 2022-07-20T03:11:12.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2100128c0 con 0x7fc22005bd90 2022-07-20T03:11:12.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.495+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc220137520 con 0x7fc22005bd90 2022-07-20T03:11:12.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.499+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc210019030 con 0x7fc22005bd90 2022-07-20T03:11:12.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.499+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc220064330 con 0x7fc22005bd90 2022-07-20T03:11:12.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.499+0000 7fc21f7fe700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc200053ad0 0x7fc200055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.499+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(50..50 src has 1..50) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fc21006b140 con 0x7fc22005bd90 2022-07-20T03:11:12.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.499+0000 7fc225ca5700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc200053ad0 0x7fc200055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.499+0000 7fc225ca5700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc200053ad0 0x7fc200055f90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fc21400b000 tx=0x7fc214006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:12.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.503+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc220064330 con 0x7fc22005bd90 2022-07-20T03:11:12.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.671+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fc220064330 con 0x7fc22005bd90 2022-07-20T03:11:12.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.683+0000 7fc21f7fe700 1 -- 172.21.15.61:0/3674848218 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 9 and data pool 10 v26) v1 ==== 188+0+0 (secure 0 0 0) 0x7fc220064330 con 0x7fc22005bd90 2022-07-20T03:11:12.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc200053ad0 msgr2=0x7fc200055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc200053ad0 0x7fc200055f90 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fc21400b000 tx=0x7fc214006df0).stop 2022-07-20T03:11:12.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 msgr2=0x7fc2200b5130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 0x7fc2200b5130 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7fc220061b40 tx=0x7fc210007c50).stop 2022-07-20T03:11:12.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 shutdown_connections 2022-07-20T03:11:12.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc200053ad0 0x7fc200055f90 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc2200b1700 0x7fc2200668a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc22013eed0 0x7fc220155b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 --2- 172.21.15.61:0/3674848218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc22005bd90 0x7fc2200b5130 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.687+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 >> 172.21.15.61:0/3674848218 conn(0x7fc2200b5910 msgr2=0x7fc22013f2e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:12.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.691+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 shutdown_connections 2022-07-20T03:11:12.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.691+0000 7fc22870a700 1 -- 172.21.15.61:0/3674848218 wait complete. 2022-07-20T03:11:12.699 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 9 and data pool 10 2022-07-20T03:11:12.717 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:11:12.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.823+0000 7f6dbd7f9700 1 Processor -- start 2022-07-20T03:11:12.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.823+0000 7f6dbd7f9700 1 -- start start 2022-07-20T03:11:12.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6db805f9e0 0x7f6db80606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db8143e00 con 0x7f6db80b1bb0 2022-07-20T03:11:12.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db80b2050 con 0x7f6db8062140 2022-07-20T03:11:12.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db8061660 con 0x7f6db805f9e0 2022-07-20T03:11:12.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6db80b1bb0 0x7f6db8142fe0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41756/0 (socket says 172.21.15.61:41756) 2022-07-20T03:11:12.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6db805f9e0 0x7f6db80606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db77fe700 1 -- 172.21.15.61:0/789769446 learned_addr learned my addr 172.21.15.61:0/789769446 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:12.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db7fff700 1 -- 172.21.15.61:0/789769446 >> v1:172.21.15.117:6789/0 conn(0x7f6db8062140 legacy=0x7f6db81436f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db7fff700 1 -- 172.21.15.61:0/789769446 >> v1:172.21.15.61:6789/0 conn(0x7f6db80b1bb0 legacy=0x7f6db8142fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db7fff700 1 -- 172.21.15.61:0/789769446 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8060200 con 0x7f6db805f9e0 2022-07-20T03:11:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db7fff700 1 --2- 172.21.15.61:0/789769446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db805f9e0 0x7f6db80606a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f6db8060f40 tx=0x7f6da8006d90).ready entity=mon.0 client_cookie=2a6c90a95168faaa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db6ffd700 1 -- 172.21.15.61:0/789769446 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da800f680 con 0x7f6db805f9e0 2022-07-20T03:11:12.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db6ffd700 1 -- 172.21.15.61:0/789769446 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da8000e90 con 0x7f6db805f9e0 2022-07-20T03:11:12.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6db6ffd700 1 -- 172.21.15.61:0/789769446 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da800f9d0 con 0x7f6db805f9e0 2022-07-20T03:11:12.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/789769446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db805f9e0 msgr2=0x7f6db80606a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/789769446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db805f9e0 0x7f6db80606a0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f6db8060f40 tx=0x7f6da8006d90).stop 2022-07-20T03:11:12.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/789769446 shutdown_connections 2022-07-20T03:11:12.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/789769446 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db805f9e0 0x7f6db80606a0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/789769446 >> 172.21.15.61:0/789769446 conn(0x7f6db80b2ca0 msgr2=0x7f6db80b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:12.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/789769446 shutdown_connections 2022-07-20T03:11:12.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.827+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/789769446 wait complete. 2022-07-20T03:11:12.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 Processor -- start 2022-07-20T03:11:12.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 -- start start 2022-07-20T03:11:12.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db805f9e0 0x7f6db8143090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 0x7f6db814ff80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db80b1bb0 0x7f6db8153c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:12.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8150f40 con 0x7f6db8062140 2022-07-20T03:11:12.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db8153550 con 0x7f6db805f9e0 2022-07-20T03:11:12.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbd7f9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db80afd10 con 0x7f6db80b1bb0 2022-07-20T03:11:12.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6db77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 0x7f6db814ff80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6db7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db805f9e0 0x7f6db8143090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6db77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 0x7f6db814ff80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45164/0 (socket says 172.21.15.61:45164) 2022-07-20T03:11:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6db77fe700 1 -- 172.21.15.61:0/25738179 learned_addr learned my addr 172.21.15.61:0/25738179 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:12.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.831+0000 7f6dbcff8700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db80b1bb0 0x7f6db8153c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:12.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db77fe700 1 -- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db80b1bb0 msgr2=0x7f6db8153c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db77fe700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db80b1bb0 0x7f6db8153c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db77fe700 1 -- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db805f9e0 msgr2=0x7f6db8143090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:12.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db77fe700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db805f9e0 0x7f6db8143090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:12.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db77fe700 1 -- 172.21.15.61:0/25738179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da800b120 con 0x7f6db8062140 2022-07-20T03:11:12.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db77fe700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 0x7f6db814ff80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f6db8064340 tx=0x7f6dac007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:12.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dac0070d0 con 0x7f6db8062140 2022-07-20T03:11:12.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac007230 con 0x7f6db8062140 2022-07-20T03:11:12.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db8064070 con 0x7f6db8062140 2022-07-20T03:11:12.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dac005610 con 0x7f6db8062140 2022-07-20T03:11:12.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db8150470 con 0x7f6db8062140 2022-07-20T03:11:12.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db8098bf0 con 0x7f6db8062140 2022-07-20T03:11:13.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.835+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6dac010070 con 0x7f6db8062140 2022-07-20T03:11:13.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.839+0000 7f6db57fa700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d98053ad0 0x7f6d98055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.839+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6dac06b320 con 0x7f6db8062140 2022-07-20T03:11:13.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.839+0000 7f6db7fff700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d98053ad0 0x7f6d98055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.839+0000 7f6db7fff700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d98053ad0 0x7f6d98055f90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f6da801b000 tx=0x7f6da8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.847+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6dac068e90 con 0x7f6db8062140 2022-07-20T03:11:13.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.971+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6db80574e0 con 0x7f6db8062140 2022-07-20T03:11:13.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.971+0000 7f6db57fa700 1 -- 172.21.15.61:0/25738179 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16101 (secure 0 0 0) 0x7f6dac047020 con 0x7f6db8062140 2022-07-20T03:11:13.041 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:13.041 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:11:12.676420+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:11:10.825678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T03:11:11.707209+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:11:13.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d98053ad0 msgr2=0x7f6d98055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d98053ad0 0x7f6d98055f90 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f6da801b000 tx=0x7f6da8015000).stop 2022-07-20T03:11:13.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 msgr2=0x7f6db814ff80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 0x7f6db814ff80 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7f6db8064340 tx=0x7f6dac007c40).stop 2022-07-20T03:11:13.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 shutdown_connections 2022-07-20T03:11:13.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d98053ad0 0x7f6d98055f90 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db80b1bb0 0x7f6db8153c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db8062140 0x7f6db814ff80 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 --2- 172.21.15.61:0/25738179 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db805f9e0 0x7f6db8143090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 >> 172.21.15.61:0/25738179 conn(0x7f6db80b2ca0 msgr2=0x7f6db80ab180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:13.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 shutdown_connections 2022-07-20T03:11:13.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:12.975+0000 7f6dbd7f9700 1 -- 172.21.15.61:0/25738179 wait complete. 2022-07-20T03:11:13.179 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:11:13.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.299+0000 7fe3c8992700 1 Processor -- start 2022-07-20T03:11:13.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.299+0000 7fe3c8992700 1 -- start start 2022-07-20T03:11:13.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe3c40b0570 0x7fe3c40b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe3c40aea10 0x7fe3c4141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3c41433d0 con 0x7fe3c41422c0 2022-07-20T03:11:13.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c4143b10 con 0x7fe3c40aea10 2022-07-20T03:11:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3c4143670 con 0x7fe3c40b0570 2022-07-20T03:11:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe3c40aea10 0x7fe3c4141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c359e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe3c41422c0 0x7fe3c40ad350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41772/0 (socket says 172.21.15.61:41772) 2022-07-20T03:11:13.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c359e700 1 -- 172.21.15.61:0/279826438 learned_addr learned my addr 172.21.15.61:0/279826438 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c2d9d700 1 --2- 172.21.15.61:0/279826438 >> v2:172.21.15.117:3300/0 conn(0x7fe3c40b0570 0x7fe3c40b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c1d9b700 1 -- 172.21.15.61:0/279826438 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3269058682 0 0) 0x7fe3c41433d0 con 0x7fe3c41422c0 2022-07-20T03:11:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c1d9b700 1 -- 172.21.15.61:0/279826438 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe3a8003260 con 0x7fe3c41422c0 2022-07-20T03:11:13.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c259c700 1 -- 172.21.15.61:0/279826438 >> v2:172.21.15.117:3300/0 conn(0x7fe3c40b0570 msgr2=0x7fe3c40b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c259c700 1 --2- 172.21.15.61:0/279826438 >> v2:172.21.15.117:3300/0 conn(0x7fe3c40b0570 0x7fe3c40b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c259c700 1 -- 172.21.15.61:0/279826438 >> v1:172.21.15.61:6789/0 conn(0x7fe3c41422c0 legacy=0x7fe3c40ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c259c700 1 -- 172.21.15.61:0/279826438 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3c4142710 con 0x7fe3c40aea10 2022-07-20T03:11:13.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c259c700 1 --2- 172.21.15.61:0/279826438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40aea10 0x7fe3c4141dd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fe3b4002f70 tx=0x7fe3b4006df0).ready entity=mon.0 client_cookie=f3a89ad135f9c121 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c1d9b700 1 -- 172.21.15.61:0/279826438 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b400d4f0 con 0x7fe3c40aea10 2022-07-20T03:11:13.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c1d9b700 1 -- 172.21.15.61:0/279826438 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b4000e90 con 0x7fe3c40aea10 2022-07-20T03:11:13.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c1d9b700 1 -- 172.21.15.61:0/279826438 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b400d840 con 0x7fe3c40aea10 2022-07-20T03:11:13.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- 172.21.15.61:0/279826438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40aea10 msgr2=0x7fe3c4141dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 --2- 172.21.15.61:0/279826438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40aea10 0x7fe3c4141dd0 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7fe3b4002f70 tx=0x7fe3b4006df0).stop 2022-07-20T03:11:13.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- 172.21.15.61:0/279826438 shutdown_connections 2022-07-20T03:11:13.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 --2- 172.21.15.61:0/279826438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40aea10 0x7fe3c4141dd0 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 --2- 172.21.15.61:0/279826438 >> v2:172.21.15.117:3300/0 conn(0x7fe3c40b0570 0x7fe3c40b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- 172.21.15.61:0/279826438 >> 172.21.15.61:0/279826438 conn(0x7fe3c40b9360 msgr2=0x7fe3c40b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:13.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- 172.21.15.61:0/279826438 shutdown_connections 2022-07-20T03:11:13.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.303+0000 7fe3c8992700 1 -- 172.21.15.61:0/279826438 wait complete. 2022-07-20T03:11:13.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 Processor -- start 2022-07-20T03:11:13.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 -- start start 2022-07-20T03:11:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c40aea10 0x7fe3c40b2100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 0x7fe3c413eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c41422c0 0x7fe3c413f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40b4d80 con 0x7fe3c40b0570 2022-07-20T03:11:13.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c40b5220 con 0x7fe3c40aea10 2022-07-20T03:11:13.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3c4146890 con 0x7fe3c41422c0 2022-07-20T03:11:13.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 0x7fe3c413eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 0x7fe3c413eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45176/0 (socket says 172.21.15.61:45176) 2022-07-20T03:11:13.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 -- 172.21.15.61:0/3386286484 learned_addr learned my addr 172.21.15.61:0/3386286484 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:13.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c359e700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c41422c0 0x7fe3c413f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c2d9d700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c40aea10 0x7fe3c40b2100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 -- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c41422c0 msgr2=0x7fe3c413f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c41422c0 0x7fe3c413f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 -- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c40aea10 msgr2=0x7fe3c40b2100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c40aea10 0x7fe3c40b2100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 -- 172.21.15.61:0/3386286484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3b400b180 con 0x7fe3c40b0570 2022-07-20T03:11:13.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c2d9d700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c40aea10 0x7fe3c40b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:13.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c259c700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 0x7fe3c413eb70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fe3b4002f70 tx=0x7fe3b4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b400de60 con 0x7fe3c40b0570 2022-07-20T03:11:13.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3b401e400 con 0x7fe3c40b0570 2022-07-20T03:11:13.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3c40b4f80 con 0x7fe3c40b0570 2022-07-20T03:11:13.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3b401e760 con 0x7fe3c40b0570 2022-07-20T03:11:13.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.307+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3c4146580 con 0x7fe3c40b0570 2022-07-20T03:11:13.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.311+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3c40b2be0 con 0x7fe3c40b0570 2022-07-20T03:11:13.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.311+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe3b4025030 con 0x7fe3c40b0570 2022-07-20T03:11:13.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.311+0000 7fe3b3fff700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3a00539d0 0x7fe3a0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.311+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe3b406ca80 con 0x7fe3c40b0570 2022-07-20T03:11:13.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.311+0000 7fe3c2d9d700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3a00539d0 0x7fe3a0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.311+0000 7fe3c2d9d700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3a00539d0 0x7fe3a0055e90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe3b800b000 tx=0x7fe3b8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.319+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3c40b2be0 con 0x7fe3c40b0570 2022-07-20T03:11:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.447+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3c40b2be0 con 0x7fe3c40b0570 2022-07-20T03:11:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3b3fff700 1 -- 172.21.15.61:0/3386286484 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v51) v1 ==== 74+0+16101 (secure 0 0 0) 0x7fe3c40b2be0 con 0x7fe3c40b0570 2022-07-20T03:11:13.493 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:13.493 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:11:12.676420+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:11:10.825678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T03:11:11.707209+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:11:13.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3a00539d0 msgr2=0x7fe3a0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3a00539d0 0x7fe3a0055e90 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7fe3b800b000 tx=0x7fe3b8008040).stop 2022-07-20T03:11:13.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 msgr2=0x7fe3c413eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 0x7fe3c413eb70 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fe3b4002f70 tx=0x7fe3b4014000).stop 2022-07-20T03:11:13.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 shutdown_connections 2022-07-20T03:11:13.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3a00539d0 0x7fe3a0055e90 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3c41422c0 0x7fe3c413f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3c40b0570 0x7fe3c413eb70 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 --2- 172.21.15.61:0/3386286484 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3c40aea10 0x7fe3c40b2100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 >> 172.21.15.61:0/3386286484 conn(0x7fe3c40b9360 msgr2=0x7fe3c40acee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:13.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 shutdown_connections 2022-07-20T03:11:13.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.451+0000 7fe3c8992700 1 -- 172.21.15.61:0/3386286484 wait complete. 2022-07-20T03:11:13.529 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:11:13.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f72ba700 1 Processor -- start 2022-07-20T03:11:13.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f72ba700 1 -- start start 2022-07-20T03:11:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f72ba700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff3f014b590 0x7ff3f014d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f72ba700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f00ba180 con 0x7ff3f00b3cf0 2022-07-20T03:11:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f72ba700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3f01461b0 con 0x7ff3f00b9da0 2022-07-20T03:11:13.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f72ba700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3f00ad5b0 con 0x7ff3f014b590 2022-07-20T03:11:13.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f5ab7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff3f00b3cf0 0x7ff3f0142a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41782/0 (socket says 172.21.15.61:41782) 2022-07-20T03:11:13.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f5ab7700 1 -- 172.21.15.61:0/3140305701 learned_addr learned my addr 172.21.15.61:0/3140305701 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:13.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.651+0000 7ff3f6ab9700 1 --2- 172.21.15.61:0/3140305701 >> v2:172.21.15.117:3300/0 conn(0x7ff3f014b590 0x7ff3f014d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f52b6700 1 -- 172.21.15.61:0/3140305701 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1120369706 0 0) 0x7ff3f00ba180 con 0x7ff3f00b3cf0 2022-07-20T03:11:13.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f52b6700 1 -- 172.21.15.61:0/3140305701 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3d8003260 con 0x7ff3f00b3cf0 2022-07-20T03:11:13.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/3140305701 >> v1:172.21.15.117:6789/0 conn(0x7ff3f00b9da0 legacy=0x7ff3f01468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/3140305701 >> v1:172.21.15.61:6789/0 conn(0x7ff3f00b3cf0 legacy=0x7ff3f0142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/3140305701 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f0146450 con 0x7ff3f014b590 2022-07-20T03:11:13.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 --2- 172.21.15.61:0/3140305701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f014b590 0x7ff3f014d980 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff3f00ad3d0 tx=0x7ff3e80091d0).ready entity=mon.1 client_cookie=1a8dad44ac9ec1d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f52b6700 1 -- 172.21.15.61:0/3140305701 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3e800b680 con 0x7ff3f014b590 2022-07-20T03:11:13.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f52b6700 1 -- 172.21.15.61:0/3140305701 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e800b7e0 con 0x7ff3f014b590 2022-07-20T03:11:13.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f52b6700 1 -- 172.21.15.61:0/3140305701 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3e800b950 con 0x7ff3f014b590 2022-07-20T03:11:13.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- 172.21.15.61:0/3140305701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f014b590 msgr2=0x7ff3f014d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/3140305701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f014b590 0x7ff3f014d980 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7ff3f00ad3d0 tx=0x7ff3e80091d0).stop 2022-07-20T03:11:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- 172.21.15.61:0/3140305701 shutdown_connections 2022-07-20T03:11:13.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/3140305701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f014b590 0x7ff3f014d980 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- 172.21.15.61:0/3140305701 >> 172.21.15.61:0/3140305701 conn(0x7ff3f00678c0 msgr2=0x7ff3f00b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:13.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- 172.21.15.61:0/3140305701 shutdown_connections 2022-07-20T03:11:13.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- 172.21.15.61:0/3140305701 wait complete. 2022-07-20T03:11:13.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 Processor -- start 2022-07-20T03:11:13.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- start start 2022-07-20T03:11:13.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f00b3cf0 0x7ff3f013db00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3f00b9da0 0x7ff3f0155170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 0x7ff3f0144dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f013e6a0 con 0x7ff3f014b590 2022-07-20T03:11:13.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f0141250 con 0x7ff3f00b3cf0 2022-07-20T03:11:13.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f72ba700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f01414f0 con 0x7ff3f00b9da0 2022-07-20T03:11:13.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 0x7ff3f0144dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 0x7ff3f0144dc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45190/0 (socket says 172.21.15.61:45190) 2022-07-20T03:11:13.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/1992308227 learned_addr learned my addr 172.21.15.61:0/1992308227 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:13.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f5ab7700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3f00b9da0 0x7ff3f0155170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.655+0000 7ff3f62b8700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f00b3cf0 0x7ff3f013db00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3f00b9da0 msgr2=0x7ff3f0155170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f6ab9700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3f00b9da0 0x7ff3f0155170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f00b3cf0 msgr2=0x7ff3f013db00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:13.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f6ab9700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f00b3cf0 0x7ff3f013db00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:13.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f6ab9700 1 -- 172.21.15.61:0/1992308227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3e8009470 con 0x7ff3f014b590 2022-07-20T03:11:13.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f6ab9700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 0x7ff3f0144dc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7ff3f00ad3d0 tx=0x7ff3e8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3e800b680 con 0x7ff3f014b590 2022-07-20T03:11:13.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f0143fc0 con 0x7ff3f014b590 2022-07-20T03:11:13.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3e8021630 con 0x7ff3f014b590 2022-07-20T03:11:13.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3e8021990 con 0x7ff3f014b590 2022-07-20T03:11:13.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f0140cd0 con 0x7ff3f014b590 2022-07-20T03:11:13.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff3e8028070 con 0x7ff3f014b590 2022-07-20T03:11:13.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f0141090 con 0x7ff3f014b590 2022-07-20T03:11:13.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3e37fe700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3d00539d0 0x7ff3d0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.659+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(51..51 src has 1..51) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff3e806cd80 con 0x7ff3f014b590 2022-07-20T03:11:13.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.663+0000 7ff3f62b8700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3d00539d0 0x7ff3d0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:13.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.663+0000 7ff3f62b8700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3d00539d0 0x7ff3d0055e90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ff3ec00d000 tx=0x7ff3ec007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:13.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.667+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3e8043c40 con 0x7ff3f014b590 2022-07-20T03:11:13.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:13.839+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7ff3f0143160 con 0x7ff3f014b590 2022-07-20T03:11:14.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.703+0000 7ff3e37fe700 1 -- 172.21.15.61:0/1992308227 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v29) v1 ==== 118+0+0 (secure 0 0 0) 0x7ff3e806e020 con 0x7ff3f014b590 2022-07-20T03:11:14.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3d00539d0 msgr2=0x7ff3d0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3d00539d0 0x7ff3d0055e90 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7ff3ec00d000 tx=0x7ff3ec007b90).stop 2022-07-20T03:11:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 msgr2=0x7ff3f0144dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:14.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 0x7ff3f0144dc0 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7ff3f00ad3d0 tx=0x7ff3e8016000).stop 2022-07-20T03:11:14.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 shutdown_connections 2022-07-20T03:11:14.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3d00539d0 0x7ff3d0055e90 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:14.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3f014b590 0x7ff3f0144dc0 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:14.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3f00b9da0 0x7ff3f0155170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:14.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 --2- 172.21.15.61:0/1992308227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3f00b3cf0 0x7ff3f013db00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:14.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 >> 172.21.15.61:0/1992308227 conn(0x7ff3f00678c0 msgr2=0x7ff3f0069e00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:14.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 shutdown_connections 2022-07-20T03:11:14.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.707+0000 7ff3f72ba700 1 -- 172.21.15.61:0/1992308227 wait complete. 2022-07-20T03:11:14.863 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:15.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 Processor -- start 2022-07-20T03:11:15.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- start start 2022-07-20T03:11:15.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c098160 0x7f5e6c098540 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e6c0affe0 0x7f5e6c14a640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e6c05c110 con 0x7f5e6c14ab30 2022-07-20T03:11:15.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e6c05c3b0 con 0x7f5e6c0affe0 2022-07-20T03:11:15.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c05c650 con 0x7f5e6c098160 2022-07-20T03:11:15.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e6c0affe0 0x7f5e6c14a640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e6c0affe0 0x7f5e6c14a640 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45200/0 (socket says 172.21.15.61:45200) 2022-07-20T03:11:15.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 -- 172.21.15.61:0/1408179918 learned_addr learned my addr 172.21.15.61:0/1408179918 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:15.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e6a59c700 1 --2- 172.21.15.61:0/1408179918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c098160 0x7f5e6c098540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 -- 172.21.15.61:0/1408179918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c098160 msgr2=0x7f5e6c098540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:15.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 --2- 172.21.15.61:0/1408179918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c098160 0x7f5e6c098540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 -- 172.21.15.61:0/1408179918 >> v1:172.21.15.117:6789/0 conn(0x7f5e6c14ab30 legacy=0x7f5e6c1462e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:15.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 -- 172.21.15.61:0/1408179918 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6c098a30 con 0x7f5e6c0affe0 2022-07-20T03:11:15.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e69d9b700 1 --2- 172.21.15.61:0/1408179918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c14a640 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f5e6c05c9d0 tx=0x7f5e60006eb0).ready entity=mon.0 client_cookie=9813a213819f3bac server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:15.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e6959a700 1 -- 172.21.15.61:0/1408179918 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e6000d570 con 0x7f5e6c0affe0 2022-07-20T03:11:15.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e6959a700 1 -- 172.21.15.61:0/1408179918 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e60000e90 con 0x7f5e6c0affe0 2022-07-20T03:11:15.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e6959a700 1 -- 172.21.15.61:0/1408179918 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e6000d8c0 con 0x7f5e6c0affe0 2022-07-20T03:11:15.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- 172.21.15.61:0/1408179918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c0affe0 msgr2=0x7f5e6c14a640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/1408179918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c14a640 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f5e6c05c9d0 tx=0x7f5e60006eb0).stop 2022-07-20T03:11:15.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- 172.21.15.61:0/1408179918 shutdown_connections 2022-07-20T03:11:15.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/1408179918 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c14a640 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/1408179918 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c098160 0x7f5e6c098540 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- 172.21.15.61:0/1408179918 >> 172.21.15.61:0/1408179918 conn(0x7f5e6c0b5ee0 msgr2=0x7f5e6c0b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:15.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- 172.21.15.61:0/1408179918 shutdown_connections 2022-07-20T03:11:15.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 -- 172.21.15.61:0/1408179918 wait complete. 2022-07-20T03:11:15.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.979+0000 7f5e70d9f700 1 Processor -- start 2022-07-20T03:11:15.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 -- start start 2022-07-20T03:11:15.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 0x7f5e6c14b9f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c0b8a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c14ab30 0x7f5e6c14bf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c099d50 con 0x7f5e6c098160 2022-07-20T03:11:15.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c159d70 con 0x7f5e6c0affe0 2022-07-20T03:11:15.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e6c15a010 con 0x7f5e6c14ab30 2022-07-20T03:11:15.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 0x7f5e6c14b9f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c14ab30 0x7f5e6c14bf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e69d9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c0b8a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 0x7f5e6c14b9f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45208/0 (socket says 172.21.15.61:45208) 2022-07-20T03:11:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6a59c700 1 -- 172.21.15.61:0/4017810616 learned_addr learned my addr 172.21.15.61:0/4017810616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:15.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 -- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e6c0affe0 msgr2=0x7f5e6c0b8a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c0b8a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 -- 172.21.15.61:0/4017810616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 msgr2=0x7f5e6c14b9f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 0x7f5e6c14b9f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 -- 172.21.15.61:0/4017810616 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e6000b1e0 con 0x7f5e6c14ab30 2022-07-20T03:11:15.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6a59c700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 0x7f5e6c14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:15.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e6ad9d700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c14ab30 0x7f5e6c14bf60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5e640032b0 tx=0x7f5e64009060).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:15.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e6400be50 con 0x7f5e6c14ab30 2022-07-20T03:11:15.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e6400e470 con 0x7f5e6c14ab30 2022-07-20T03:11:15.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e6c0999f0 con 0x7f5e6c14ab30 2022-07-20T03:11:15.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e6400e7d0 con 0x7f5e6c14ab30 2022-07-20T03:11:15.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.983+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e6c14aff0 con 0x7f5e6c14ab30 2022-07-20T03:11:15.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.987+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e6c15a1d0 con 0x7f5e6c14ab30 2022-07-20T03:11:15.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.987+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5e64018030 con 0x7f5e6c14ab30 2022-07-20T03:11:15.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.987+0000 7f5e5b7fe700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e48053ad0 0x7f5e48055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.987+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e6406b6d0 con 0x7f5e6c14ab30 2022-07-20T03:11:15.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.987+0000 7f5e6a59c700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e48053ad0 0x7f5e48055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.987+0000 7f5e6a59c700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e48053ad0 0x7f5e48055f90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f5e5c008000 tx=0x7f5e5c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:15.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:14.995+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e64042550 con 0x7f5e6c14ab30 2022-07-20T03:11:15.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.155+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e6c05bd90 con 0x7f5e6c14ab30 2022-07-20T03:11:15.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.155+0000 7f5e5b7fe700 1 -- 172.21.15.61:0/4017810616 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f5e64045b70 con 0x7f5e6c14ab30 2022-07-20T03:11:15.161 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:15.161 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:15.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e48053ad0 msgr2=0x7f5e48055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e48053ad0 0x7f5e48055f90 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f5e5c008000 tx=0x7f5e5c008040).stop 2022-07-20T03:11:15.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c14ab30 msgr2=0x7f5e6c14bf60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c14ab30 0x7f5e6c14bf60 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f5e640032b0 tx=0x7f5e64009060).stop 2022-07-20T03:11:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 shutdown_connections 2022-07-20T03:11:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e48053ad0 0x7f5e48055f90 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e6c0affe0 0x7f5e6c0b8a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e6c14ab30 0x7f5e6c14bf60 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 --2- 172.21.15.61:0/4017810616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e6c098160 0x7f5e6c14b9f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 >> 172.21.15.61:0/4017810616 conn(0x7f5e6c0b5ee0 msgr2=0x7f5e6c09c570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 shutdown_connections 2022-07-20T03:11:15.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.159+0000 7f5e70d9f700 1 -- 172.21.15.61:0/4017810616 wait complete. 2022-07-20T03:11:15.190 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:15.193 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:11:15.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.391+0000 7f0dd3fff700 1 Processor -- start 2022-07-20T03:11:15.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.391+0000 7f0dd3fff700 1 -- start start 2022-07-20T03:11:15.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd3fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0dd40b5650 0x7f0dd40b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd3fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd4147010 con 0x7f0dd40adff0 2022-07-20T03:11:15.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd3fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0dd41472b0 con 0x7f0dd40af8e0 2022-07-20T03:11:15.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd3fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0dd4146d40 con 0x7f0dd40b5650 2022-07-20T03:11:15.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0dd40adff0 0x7f0dd4147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41810/0 (socket says 172.21.15.61:41810) 2022-07-20T03:11:15.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 -- 172.21.15.61:0/1673751131 learned_addr learned my addr 172.21.15.61:0/1673751131 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:15.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd2ffd700 1 --2- 172.21.15.61:0/1673751131 >> v2:172.21.15.117:3300/0 conn(0x7f0dd40b5650 0x7f0dd40b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 912324247 0 0) 0x7f0dd4147010 con 0x7f0dd40adff0 2022-07-20T03:11:15.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0db4003260 con 0x7f0dd40adff0 2022-07-20T03:11:15.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652932782 0 0) 0x7f0dd41472b0 con 0x7f0dd40af8e0 2022-07-20T03:11:15.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0dd4147010 con 0x7f0dd40af8e0 2022-07-20T03:11:15.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2568118409 0 0) 0x7f0db4003260 con 0x7f0dd40adff0 2022-07-20T03:11:15.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 >> v2:172.21.15.117:3300/0 conn(0x7f0dd40b5650 msgr2=0x7f0dd40b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:15.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 --2- 172.21.15.61:0/1673751131 >> v2:172.21.15.117:3300/0 conn(0x7f0dd40b5650 0x7f0dd40b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 >> v1:172.21.15.117:6789/0 conn(0x7f0dd40af8e0 legacy=0x7f0dd405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd405c4a0 con 0x7f0dd40adff0 2022-07-20T03:11:15.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f0dbc001b60 con 0x7f0dd40adff0 2022-07-20T03:11:15.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 >> v1:172.21.15.61:6789/0 conn(0x7f0dd40adff0 legacy=0x7f0dd4147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db4003930 0x7f0db4003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0db4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db4004e70 0x7f0db400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4146d40 con 0x7f0db4004500 2022-07-20T03:11:15.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0db400bdd0 con 0x7f0db4004e70 2022-07-20T03:11:15.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0db400bf90 con 0x7f0db4003930 2022-07-20T03:11:15.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0db4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd2ffd700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db4003930 0x7f0db4003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd37fe700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db4004e70 0x7f0db400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 -- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db4003930 msgr2=0x7f0db4003d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:15.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0db4003930 0x7f0db4003d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 -- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db4004e70 msgr2=0x7f0db400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:15.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0db4004e70 0x7f0db400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 -- 172.21.15.61:0/1673751131 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0db4003260 con 0x7f0db4004500 2022-07-20T03:11:15.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd37fe700 1 -- 172.21.15.61:0/1673751131 reap_dead start 2022-07-20T03:11:15.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0db4004980 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f0dbc001930 tx=0x7f0dbc00c440).ready entity=mon.0 client_cookie=eaa4bd6c2090c80a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:15.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dbc00cad0 con 0x7f0db4004500 2022-07-20T03:11:15.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dbc00cc30 con 0x7f0db4004500 2022-07-20T03:11:15.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd1ffb700 1 -- 172.21.15.61:0/1673751131 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dbc005860 con 0x7f0db4004500 2022-07-20T03:11:15.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd3fff700 1 -- 172.21.15.61:0/1673751131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 msgr2=0x7f0db4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.395+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0db4004980 secure :-1 s=READY pgs=491 cs=0 l=1 rev1=1 rx=0x7f0dbc001930 tx=0x7f0dbc00c440).stop 2022-07-20T03:11:15.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 -- 172.21.15.61:0/1673751131 shutdown_connections 2022-07-20T03:11:15.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/1673751131 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0db4004980 unknown :-1 s=CLOSED pgs=491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 -- 172.21.15.61:0/1673751131 >> 172.21.15.61:0/1673751131 conn(0x7f0dd40634e0 msgr2=0x7f0dd40b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:15.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 -- 172.21.15.61:0/1673751131 shutdown_connections 2022-07-20T03:11:15.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 -- 172.21.15.61:0/1673751131 wait complete. 2022-07-20T03:11:15.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 Processor -- start 2022-07-20T03:11:15.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.399+0000 7f0dd3fff700 1 -- start start 2022-07-20T03:11:15.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0dd4067230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dd411db20 0x7f0dd41557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dd4155cc0 0x7f0dd4156140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4157150 con 0x7f0db4004500 2022-07-20T03:11:15.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4066e20 con 0x7f0dd4155cc0 2022-07-20T03:11:15.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0dd4156770 con 0x7f0dd411db20 2022-07-20T03:11:15.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0dd4067230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0dd4067230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45224/0 (socket says 172.21.15.61:45224) 2022-07-20T03:11:15.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 -- 172.21.15.61:0/219785555 learned_addr learned my addr 172.21.15.61:0/219785555 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:15.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd37fe700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dd4155cc0 0x7f0dd4156140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dd411db20 0x7f0dd41557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 -- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dd411db20 msgr2=0x7f0dd41557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dd411db20 0x7f0dd41557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 -- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dd4155cc0 msgr2=0x7f0dd4156140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dd4155cc0 0x7f0dd4156140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 -- 172.21.15.61:0/219785555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0dd4156a10 con 0x7f0db4004500 2022-07-20T03:11:15.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd2ffd700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0dd4067230 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f0dd4157030 tx=0x7f0dc4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:15.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dc4013680 con 0x7f0db4004500 2022-07-20T03:11:15.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0dc4000e90 con 0x7f0db4004500 2022-07-20T03:11:15.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0dd4156cb0 con 0x7f0db4004500 2022-07-20T03:11:15.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0dc40139d0 con 0x7f0db4004500 2022-07-20T03:11:15.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.403+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0dd411b5c0 con 0x7f0db4004500 2022-07-20T03:11:15.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.407+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0dd4156e40 con 0x7f0db4004500 2022-07-20T03:11:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.407+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0dc4013b30 con 0x7f0db4004500 2022-07-20T03:11:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.407+0000 7f0dcbfff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0db8053920 0x7f0db8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.407+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0dc406c280 con 0x7f0db4004500 2022-07-20T03:11:15.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.407+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0db8053920 0x7f0db8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.407+0000 7f0dd27fc700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0db8053920 0x7f0db8055de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f0dbc017000 tx=0x7f0dbc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:15.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.415+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0dc40431e0 con 0x7f0db4004500 2022-07-20T03:11:15.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.539+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f0dd40574e0 con 0x7f0db4004500 2022-07-20T03:11:15.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dcbfff700 1 -- 172.21.15.61:0/219785555 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16088 (secure 0 0 0) 0x7f0dc4018020 con 0x7f0db4004500 2022-07-20T03:11:15.783 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:15.784 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":52,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:11:13.680406+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:11:10.825678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T03:11:11.707209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:11:15.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0db8053920 msgr2=0x7f0db8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0db8053920 0x7f0db8055de0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f0dbc017000 tx=0x7f0dbc015000).stop 2022-07-20T03:11:15.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 msgr2=0x7f0dd4067230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:15.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0dd4067230 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f0dd4157030 tx=0x7f0dc4006df0).stop 2022-07-20T03:11:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 shutdown_connections 2022-07-20T03:11:15.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0db8053920 0x7f0db8055de0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0dd4155cc0 0x7f0dd4156140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0dd411db20 0x7f0dd41557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 --2- 172.21.15.61:0/219785555 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0db4004500 0x7f0dd4067230 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:15.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 >> 172.21.15.61:0/219785555 conn(0x7f0dd40634e0 msgr2=0x7f0dd40ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:15.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 shutdown_connections 2022-07-20T03:11:15.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.543+0000 7f0dd3fff700 1 -- 172.21.15.61:0/219785555 wait complete. 2022-07-20T03:11:15.809 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:11:15.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 Processor -- start 2022-07-20T03:11:15.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 -- start start 2022-07-20T03:11:15.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f54c80b1600 0x7f54c80aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805bd90 0x7f54c8141cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:15.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c80b19e0 con 0x7f54c805fd90 2022-07-20T03:11:15.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c8060fe0 con 0x7f54c80b1600 2022-07-20T03:11:16.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54ceb42700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c80ac740 con 0x7f54c805bd90 2022-07-20T03:11:16.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54c7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f54c805fd90 0x7f54c80613d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41834/0 (socket says 172.21.15.61:41834) 2022-07-20T03:11:16.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54c7fff700 1 -- 172.21.15.61:0/275888368 learned_addr learned my addr 172.21.15.61:0/275888368 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:16.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54cc8de700 1 --2- 172.21.15.61:0/275888368 >> v2:172.21.15.117:3300/0 conn(0x7f54c80b1600 0x7f54c80aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.939+0000 7f54cd0df700 1 --2- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805bd90 0x7f54c8141cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54c77fe700 1 -- 172.21.15.61:0/275888368 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 654288536 0 0) 0x7f54c80b19e0 con 0x7f54c805fd90 2022-07-20T03:11:16.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54c77fe700 1 -- 172.21.15.61:0/275888368 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f54b4003260 con 0x7f54c805fd90 2022-07-20T03:11:16.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54cc8de700 1 -- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805bd90 msgr2=0x7f54c8141cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:16.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54cc8de700 1 --2- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805bd90 0x7f54c8141cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54cc8de700 1 -- 172.21.15.61:0/275888368 >> v1:172.21.15.61:6789/0 conn(0x7f54c805fd90 legacy=0x7f54c80613d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54cc8de700 1 -- 172.21.15.61:0/275888368 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c809b900 con 0x7f54c80b1600 2022-07-20T03:11:16.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54cc8de700 1 --2- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c80b1600 0x7f54c80aa9d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f54b8002f70 tx=0x7f54b8006d90).ready entity=mon.1 client_cookie=3fe16a63d09ab1bb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:16.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54c77fe700 1 -- 172.21.15.61:0/275888368 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54b800e590 con 0x7f54c80b1600 2022-07-20T03:11:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54c77fe700 1 -- 172.21.15.61:0/275888368 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b8000e90 con 0x7f54c80b1600 2022-07-20T03:11:16.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54c77fe700 1 -- 172.21.15.61:0/275888368 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54b800e8e0 con 0x7f54c80b1600 2022-07-20T03:11:16.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 -- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c80b1600 msgr2=0x7f54c80aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 --2- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c80b1600 0x7f54c80aa9d0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f54b8002f70 tx=0x7f54b8006d90).stop 2022-07-20T03:11:16.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 -- 172.21.15.61:0/275888368 shutdown_connections 2022-07-20T03:11:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 --2- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805bd90 0x7f54c8141cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 --2- 172.21.15.61:0/275888368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c80b1600 0x7f54c80aa9d0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 -- 172.21.15.61:0/275888368 >> 172.21.15.61:0/275888368 conn(0x7f54c80b23b0 msgr2=0x7f54c80b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:16.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 -- 172.21.15.61:0/275888368 shutdown_connections 2022-07-20T03:11:16.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.943+0000 7f54ceb42700 1 -- 172.21.15.61:0/275888368 wait complete. 2022-07-20T03:11:16.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 Processor -- start 2022-07-20T03:11:16.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 -- start start 2022-07-20T03:11:16.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 0x7f54c8092110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805fd90 0x7f54c80b46a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 0x7f54c809aae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c808fc00 con 0x7f54c80b1600 2022-07-20T03:11:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c808fea0 con 0x7f54c805bd90 2022-07-20T03:11:16.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c8090340 con 0x7f54c805fd90 2022-07-20T03:11:16.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 0x7f54c809aae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cc8de700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 0x7f54c8092110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cc8de700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 0x7f54c8092110 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:54918/0 (socket says 172.21.15.61:54918) 2022-07-20T03:11:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cc8de700 1 -- 172.21.15.61:0/3562396125 learned_addr learned my addr 172.21.15.61:0/3562396125 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:16.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 0x7f54c809aae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45242/0 (socket says 172.21.15.61:45242) 2022-07-20T03:11:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54c7fff700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805fd90 0x7f54c80b46a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 -- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805fd90 msgr2=0x7f54c80b46a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805fd90 0x7f54c80b46a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 -- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 msgr2=0x7f54c8092110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 0x7f54c8092110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 -- 172.21.15.61:0/3562396125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54b800b120 con 0x7f54c80b1600 2022-07-20T03:11:16.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cc8de700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 0x7f54c8092110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:16.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54cd0df700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 0x7f54c809aae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f54c000c000 tx=0x7f54c00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54c0012600 con 0x7f54c80b1600 2022-07-20T03:11:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54c0012760 con 0x7f54c80b1600 2022-07-20T03:11:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c808f7e0 con 0x7f54c80b1600 2022-07-20T03:11:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54c0012ac0 con 0x7f54c80b1600 2022-07-20T03:11:16.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.947+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c80af150 con 0x7f54c80b1600 2022-07-20T03:11:16.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.951+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c80b1dc0 con 0x7f54c80b1600 2022-07-20T03:11:16.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.951+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f54c0022070 con 0x7f54c80b1600 2022-07-20T03:11:16.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.951+0000 7f54c5ffb700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a8053a40 0x7f54a8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.951+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f54c006c570 con 0x7f54c80b1600 2022-07-20T03:11:16.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.951+0000 7f54cc8de700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a8053a40 0x7f54a8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.951+0000 7f54cc8de700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a8053a40 0x7f54a8055f00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f54b801a000 tx=0x7f54b8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:16.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:15.959+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54c0043360 con 0x7f54c80b1600 2022-07-20T03:11:16.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.127+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f54c80574e0 con 0x7f54c80b1600 2022-07-20T03:11:16.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.135+0000 7f54c5ffb700 1 -- 172.21.15.61:0/3562396125 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v19) v1 ==== 184+0+0 (secure 0 0 0) 0x7f54c0016020 con 0x7f54c80b1600 2022-07-20T03:11:16.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a8053a40 msgr2=0x7f54a8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a8053a40 0x7f54a8055f00 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f54b801a000 tx=0x7f54b8014000).stop 2022-07-20T03:11:16.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 msgr2=0x7f54c809aae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 0x7f54c809aae0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7f54c000c000 tx=0x7f54c00091d0).stop 2022-07-20T03:11:16.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 shutdown_connections 2022-07-20T03:11:16.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a8053a40 0x7f54a8055f00 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c80b1600 0x7f54c809aae0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c805fd90 0x7f54c80b46a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 --2- 172.21.15.61:0/3562396125 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c805bd90 0x7f54c8092110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 >> 172.21.15.61:0/3562396125 conn(0x7f54c80b23b0 msgr2=0x7f54c8148370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:16.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 shutdown_connections 2022-07-20T03:11:16.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.139+0000 7f54ceb42700 1 -- 172.21.15.61:0/3562396125 wait complete. 2022-07-20T03:11:16.304 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:11:16.305 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:11:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cce01b700 1 Processor -- start 2022-07-20T03:11:16.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cce01b700 1 -- start start 2022-07-20T03:11:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cce01b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1dc0 0x7f6cc80ae4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cce01b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc80b19e0 con 0x7f6cc80b1600 2022-07-20T03:11:16.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cce01b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cc8140a60 con 0x7f6cc805bd90 2022-07-20T03:11:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cce01b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8140cd0 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6cc80b1600 0x7f6cc8141410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41844/0 (socket says 172.21.15.61:41844) 2022-07-20T03:11:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc77fe700 1 -- 172.21.15.61:0/3857529261 learned_addr learned my addr 172.21.15.61:0/3857529261 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:16.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc6ffd700 1 --2- 172.21.15.61:0/3857529261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1dc0 0x7f6cc80ae4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2378593998 0 0) 0x7f6cc80b19e0 con 0x7f6cc80b1600 2022-07-20T03:11:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cb4003260 con 0x7f6cc80b1600 2022-07-20T03:11:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2424783902 0 0) 0x7f6cc8140a60 con 0x7f6cc805bd90 2022-07-20T03:11:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.479+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cc80b19e0 con 0x7f6cc805bd90 2022-07-20T03:11:16.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc6ffd700 1 -- 172.21.15.61:0/3857529261 >> v1:172.21.15.117:6789/0 conn(0x7f6cc805bd90 legacy=0x7f6cc80ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc6ffd700 1 -- 172.21.15.61:0/3857529261 >> v1:172.21.15.61:6789/0 conn(0x7f6cc80b1600 legacy=0x7f6cc8141410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc6ffd700 1 -- 172.21.15.61:0/3857529261 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc8140f70 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc6ffd700 1 --2- 172.21.15.61:0/3857529261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1dc0 0x7f6cc80ae4a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6cbc002f70 tx=0x7f6cbc006df0).ready entity=mon.2 client_cookie=666f0ae3eead8339 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:16.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cbc00d5c0 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cbc000e90 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc67fc700 1 -- 172.21.15.61:0/3857529261 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cbc00d910 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- 172.21.15.61:0/3857529261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1dc0 msgr2=0x7f6cc80ae4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 --2- 172.21.15.61:0/3857529261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1dc0 0x7f6cc80ae4a0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f6cbc002f70 tx=0x7f6cbc006df0).stop 2022-07-20T03:11:16.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- 172.21.15.61:0/3857529261 shutdown_connections 2022-07-20T03:11:16.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 --2- 172.21.15.61:0/3857529261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1dc0 0x7f6cc80ae4a0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- 172.21.15.61:0/3857529261 >> 172.21.15.61:0/3857529261 conn(0x7f6cc80b23b0 msgr2=0x7f6cc80b27c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:16.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- 172.21.15.61:0/3857529261 shutdown_connections 2022-07-20T03:11:16.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- 172.21.15.61:0/3857529261 wait complete. 2022-07-20T03:11:16.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 Processor -- start 2022-07-20T03:11:16.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- start start 2022-07-20T03:11:16.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cc805bd90 0x7f6cc8143710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1600 0x7f6cc8090950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 0x7f6cc8090e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8066cc0 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8066f60 con 0x7f6cc805bd90 2022-07-20T03:11:16.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cce01b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cc8067200 con 0x7f6cc80b1600 2022-07-20T03:11:16.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 0x7f6cc8090e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc6ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1600 0x7f6cc8090950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 0x7f6cc8090e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45256/0 (socket says 172.21.15.61:45256) 2022-07-20T03:11:16.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc7fff700 1 -- 172.21.15.61:0/2244095222 learned_addr learned my addr 172.21.15.61:0/2244095222 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:16.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.483+0000 7f6cc77fe700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cc805bd90 0x7f6cc8143710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc7fff700 1 -- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1600 msgr2=0x7f6cc8090950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc7fff700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1600 0x7f6cc8090950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc7fff700 1 -- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cc805bd90 msgr2=0x7f6cc8143710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc7fff700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cc805bd90 0x7f6cc8143710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc7fff700 1 -- 172.21.15.61:0/2244095222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cbc00b180 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc7fff700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 0x7f6cc8090e40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f6cc0001110 tx=0x7f6cc0007a50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:16.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc0012400 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc0012560 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cc8064a40 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc00128c0 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cc8065400 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cc8066b20 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6cc0019030 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.487+0000 7f6cc4ff9700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ca8053af0 0x7f6ca8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.491+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6cc006b1e0 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.491+0000 7f6cc77fe700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ca8053af0 0x7f6ca8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:16.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.491+0000 7f6cc77fe700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ca8053af0 0x7f6ca8055fb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6cb800d000 tx=0x7f6cb8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:16.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.495+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6cc8066b20 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.667+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6cc8066b20 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cc4ff9700 1 -- 172.21.15.61:0/2244095222 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v20) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6cc8066b20 con 0x7f6cc80b1dc0 2022-07-20T03:11:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ca8053af0 msgr2=0x7f6ca8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ca8053af0 0x7f6ca8055fb0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f6cb800d000 tx=0x7f6cb8009040).stop 2022-07-20T03:11:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 msgr2=0x7f6cc8090e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:16.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 0x7f6cc8090e40 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7f6cc0001110 tx=0x7f6cc0007a50).stop 2022-07-20T03:11:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 shutdown_connections 2022-07-20T03:11:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ca8053af0 0x7f6ca8055fb0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cc80b1dc0 0x7f6cc8090e40 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cc80b1600 0x7f6cc8090950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 --2- 172.21.15.61:0/2244095222 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cc805bd90 0x7f6cc8143710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:16.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 >> 172.21.15.61:0/2244095222 conn(0x7f6cc80b23b0 msgr2=0x7f6cc80b4d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 shutdown_connections 2022-07-20T03:11:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.675+0000 7f6cce01b700 1 -- 172.21.15.61:0/2244095222 wait complete. 2022-07-20T03:11:16.703 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:11:16.704 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:11:16.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 Processor -- start 2022-07-20T03:11:16.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 -- start start 2022-07-20T03:11:17.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0af0d0 0x7f860c0abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f860c0a9690 0x7f860c0ac2f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f860c0b4290 con 0x7f860c0b4c20 2022-07-20T03:11:17.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f860c0af4b0 con 0x7f860c0a9690 2022-07-20T03:11:17.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c0a9340 con 0x7f860c0af0d0 2022-07-20T03:11:17.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f8609d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f860c0a9690 0x7f860c0ac2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f8609d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f860c0a9690 0x7f860c0ac2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45262/0 (socket says 172.21.15.61:45262) 2022-07-20T03:11:17.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f8609d9b700 1 -- 172.21.15.61:0/3766253434 learned_addr learned my addr 172.21.15.61:0/3766253434 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:17.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860ad9d700 1 --1- 172.21.15.61:0/3766253434 >> v1:172.21.15.61:6789/0 conn(0x7f860c0b4c20 0x7f860c1409b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41862/0 (socket says 172.21.15.61:41862) 2022-07-20T03:11:17.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.855+0000 7f860a59c700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0af0d0 0x7f860c0abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f8609d9b700 1 -- 172.21.15.61:0/3766253434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0af0d0 msgr2=0x7f860c0abe00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:17.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f8609d9b700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0af0d0 0x7f860c0abe00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f8609d9b700 1 -- 172.21.15.61:0/3766253434 >> v1:172.21.15.61:6789/0 conn(0x7f860c0b4c20 legacy=0x7f860c1409b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f8609d9b700 1 -- 172.21.15.61:0/3766253434 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f860c0b44e0 con 0x7f860c0a9690 2022-07-20T03:11:17.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860a59c700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0af0d0 0x7f860c0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:11:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f8609d9b700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0a9690 0x7f860c0ac2f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f85f8002f70 tx=0x7f85f8006eb0).ready entity=mon.0 client_cookie=151347d0f5530caf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:17.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860959a700 1 -- 172.21.15.61:0/3766253434 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f800d5c0 con 0x7f860c0a9690 2022-07-20T03:11:17.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860959a700 1 -- 172.21.15.61:0/3766253434 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8000e90 con 0x7f860c0a9690 2022-07-20T03:11:17.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860959a700 1 -- 172.21.15.61:0/3766253434 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f800d950 con 0x7f860c0a9690 2022-07-20T03:11:17.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 -- 172.21.15.61:0/3766253434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0a9690 msgr2=0x7f860c0ac2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0a9690 0x7f860c0ac2f0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f85f8002f70 tx=0x7f85f8006eb0).stop 2022-07-20T03:11:17.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 -- 172.21.15.61:0/3766253434 shutdown_connections 2022-07-20T03:11:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0a9690 0x7f860c0ac2f0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 --2- 172.21.15.61:0/3766253434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0af0d0 0x7f860c0abe00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 -- 172.21.15.61:0/3766253434 >> 172.21.15.61:0/3766253434 conn(0x7f860c063150 msgr2=0x7f860c063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:17.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 -- 172.21.15.61:0/3766253434 shutdown_connections 2022-07-20T03:11:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 -- 172.21.15.61:0/3766253434 wait complete. 2022-07-20T03:11:17.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 Processor -- start 2022-07-20T03:11:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.859+0000 7f860b59e700 1 -- start start 2022-07-20T03:11:17.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f860c0a9690 0x7f860c145c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 0x7f860c064f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0b4c20 0x7f860c09e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c066120 con 0x7f860c0af0d0 2022-07-20T03:11:17.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f860c09e200 con 0x7f860c0a9690 2022-07-20T03:11:17.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f860c065440 con 0x7f860c0b4c20 2022-07-20T03:11:17.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 0x7f860c064f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 0x7f860c064f50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45268/0 (socket says 172.21.15.61:45268) 2022-07-20T03:11:17.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 -- 172.21.15.61:0/2574627478 learned_addr learned my addr 172.21.15.61:0/2574627478 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:17.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860a59c700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f860c0a9690 0x7f860c145c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860ad9d700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0b4c20 0x7f860c09e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 -- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0b4c20 msgr2=0x7f860c09e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0b4c20 0x7f860c09e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 -- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f860c0a9690 msgr2=0x7f860c145c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f860c0a9690 0x7f860c145c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 -- 172.21.15.61:0/2574627478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f800b1e0 con 0x7f860c0af0d0 2022-07-20T03:11:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860ad9d700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0b4c20 0x7f860c09e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f8609d9b700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 0x7f860c064f50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f85f8002f70 tx=0x7f85f8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:17.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f80119d0 con 0x7f860c0af0d0 2022-07-20T03:11:17.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85f8011b30 con 0x7f860c0af0d0 2022-07-20T03:11:17.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f860c141e70 con 0x7f860c0af0d0 2022-07-20T03:11:17.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85f8011e90 con 0x7f860c0af0d0 2022-07-20T03:11:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f860c142430 con 0x7f860c0af0d0 2022-07-20T03:11:17.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.863+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f860c0b4780 con 0x7f860c0af0d0 2022-07-20T03:11:17.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.867+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f85f8025030 con 0x7f860c0af0d0 2022-07-20T03:11:17.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.867+0000 7f86037fe700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85e8053ad0 0x7f85e8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.867+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f85f806d1e0 con 0x7f860c0af0d0 2022-07-20T03:11:17.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.867+0000 7f860a59c700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85e8053ad0 0x7f85e8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.867+0000 7f860a59c700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85e8053ad0 0x7f85e8055f90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f85fc00d000 tx=0x7f85fc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:17.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:16.871+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85f8044140 con 0x7f860c0af0d0 2022-07-20T03:11:17.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.035+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f860c0574e0 con 0x7f860c0af0d0 2022-07-20T03:11:17.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.039+0000 7f86037fe700 1 -- 172.21.15.61:0/2574627478 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v21) v1 ==== 184+0+0 (secure 0 0 0) 0x7f85f8023020 con 0x7f860c0af0d0 2022-07-20T03:11:17.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85e8053ad0 msgr2=0x7f85e8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85e8053ad0 0x7f85e8055f90 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f85fc00d000 tx=0x7f85fc008040).stop 2022-07-20T03:11:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 msgr2=0x7f860c064f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 0x7f860c064f50 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f85f8002f70 tx=0x7f85f8015000).stop 2022-07-20T03:11:17.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 shutdown_connections 2022-07-20T03:11:17.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85e8053ad0 0x7f85e8055f90 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f860c0b4c20 0x7f860c09e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f860c0af0d0 0x7f860c064f50 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 --2- 172.21.15.61:0/2574627478 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f860c0a9690 0x7f860c145c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 >> 172.21.15.61:0/2574627478 conn(0x7f860c063150 msgr2=0x7f860c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:17.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 shutdown_connections 2022-07-20T03:11:17.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.043+0000 7f860b59e700 1 -- 172.21.15.61:0/2574627478 wait complete. 2022-07-20T03:11:17.202 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:11:17.203 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:17.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 Processor -- start 2022-07-20T03:11:17.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 -- start start 2022-07-20T03:11:17.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb8980b1e00 0x7fb89814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8980acef0 0x7fb898146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb89805d480 con 0x7fb89805c3f0 2022-07-20T03:11:17.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8980b21e0 con 0x7fb8980b1e00 2022-07-20T03:11:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89f249700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89805cf40 con 0x7fb8980acef0 2022-07-20T03:11:17.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb89805c3f0 0x7fb8980ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41874/0 (socket says 172.21.15.61:41874) 2022-07-20T03:11:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 -- 172.21.15.61:0/3736026861 learned_addr learned my addr 172.21.15.61:0/3736026861 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:17.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89cfe5700 1 --2- 172.21.15.61:0/3736026861 >> v2:172.21.15.117:3300/0 conn(0x7fb8980b1e00 0x7fb89814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3115364702 0 0) 0x7fb89805d480 con 0x7fb89805c3f0 2022-07-20T03:11:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89d7e6700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8980acef0 0x7fb898146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb884003260 con 0x7fb89805c3f0 2022-07-20T03:11:17.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 547971353 0 0) 0x7fb884003260 con 0x7fb89805c3f0 2022-07-20T03:11:17.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8980acef0 msgr2=0x7fb898146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:17.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8980acef0 0x7fb898146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 >> v2:172.21.15.117:3300/0 conn(0x7fb8980b1e00 msgr2=0x7fb89814a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:17.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 --2- 172.21.15.61:0/3736026861 >> v2:172.21.15.117:3300/0 conn(0x7fb8980b1e00 0x7fb89814a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89805d1e0 con 0x7fb89805c3f0 2022-07-20T03:11:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fb88c001680 con 0x7fb89805c3f0 2022-07-20T03:11:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 >> v1:172.21.15.61:6789/0 conn(0x7fb89805c3f0 legacy=0x7fb8980ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb884003750 0x7fb884003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb884004500 0x7fb884004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb884004e70 0x7fb88400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8980b21e0 con 0x7fb884004500 2022-07-20T03:11:17.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89805cf40 con 0x7fb884004e70 2022-07-20T03:11:17.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb88400bd20 con 0x7fb884003750 2022-07-20T03:11:17.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb884004500 0x7fb884004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89cfe5700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb884003750 0x7fb884003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb89d7e6700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb884004e70 0x7fb88400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 -- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb884003750 msgr2=0x7fb884003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:17.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb884003750 0x7fb884003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 -- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb884004e70 msgr2=0x7fb88400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:17.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb884004e70 0x7fb88400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 -- 172.21.15.61:0/3736026861 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb884003260 con 0x7fb884004500 2022-07-20T03:11:17.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb897fff700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb884004500 0x7fb884004980 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fb88c001390 tx=0x7fb8980ac1a0).ready entity=mon.0 client_cookie=6b0c7058cf177921 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:17.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c007170 con 0x7fb884004500 2022-07-20T03:11:17.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c0072d0 con 0x7fb884004500 2022-07-20T03:11:17.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.347+0000 7fb8977fe700 1 -- 172.21.15.61:0/3736026861 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c00c990 con 0x7fb884004500 2022-07-20T03:11:17.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 -- 172.21.15.61:0/3736026861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb884004500 msgr2=0x7fb884004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 --2- 172.21.15.61:0/3736026861 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb884004500 0x7fb884004980 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fb88c001390 tx=0x7fb8980ac1a0).stop 2022-07-20T03:11:17.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89d7e6700 1 -- 172.21.15.61:0/3736026861 reap_dead start 2022-07-20T03:11:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 -- 172.21.15.61:0/3736026861 shutdown_connections 2022-07-20T03:11:17.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 -- 172.21.15.61:0/3736026861 >> 172.21.15.61:0/3736026861 conn(0x7fb8980611c0 msgr2=0x7fb898062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:17.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 -- 172.21.15.61:0/3736026861 shutdown_connections 2022-07-20T03:11:17.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 -- 172.21.15.61:0/3736026861 wait complete. 2022-07-20T03:11:17.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 Processor -- start 2022-07-20T03:11:17.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.351+0000 7fb89f249700 1 -- start start 2022-07-20T03:11:17.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 0x7fb8981558b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 0x7fb898155da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8981485d0 0x7fb898148a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb898064100 con 0x7fb8980a1b30 2022-07-20T03:11:17.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8981571f0 con 0x7fb8981485d0 2022-07-20T03:11:17.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89809e640 con 0x7fb898063d20 2022-07-20T03:11:17.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 0x7fb898155da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89cfe5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 0x7fb8981558b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89d7e6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8981485d0 0x7fb898148a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89cfe5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 0x7fb8981558b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:60190/0 (socket says 172.21.15.61:60190) 2022-07-20T03:11:17.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 0x7fb898155da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45288/0 (socket says 172.21.15.61:45288) 2022-07-20T03:11:17.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89cfe5700 1 -- 172.21.15.61:0/1553801228 learned_addr learned my addr 172.21.15.61:0/1553801228 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:17.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 -- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 msgr2=0x7fb8981558b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 0x7fb8981558b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 -- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8981485d0 msgr2=0x7fb898148a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8981485d0 0x7fb898148a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 -- 172.21.15.61:0/1553801228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb898156f10 con 0x7fb8980a1b30 2022-07-20T03:11:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89cfe5700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 0x7fb8981558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:17.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89d7e6700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8981485d0 0x7fb898148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:17.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb897fff700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 0x7fb898155da0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fb88c001390 tx=0x7fb88c013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:17.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c007170 con 0x7fb8980a1b30 2022-07-20T03:11:17.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb88c001af0 con 0x7fb8980a1b30 2022-07-20T03:11:17.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb898156290 con 0x7fb8980a1b30 2022-07-20T03:11:17.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb88c004730 con 0x7fb8980a1b30 2022-07-20T03:11:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.355+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb89809e0c0 con 0x7fb8980a1b30 2022-07-20T03:11:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.359+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb89809e4a0 con 0x7fb8980a1b30 2022-07-20T03:11:17.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.359+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb88c027070 con 0x7fb8980a1b30 2022-07-20T03:11:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.359+0000 7fb895ffb700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb878053af0 0x7fb878055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.359+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb88c06c300 con 0x7fb8980a1b30 2022-07-20T03:11:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.359+0000 7fb89cfe5700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb878053af0 0x7fb878055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:17.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.359+0000 7fb89cfe5700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb878053af0 0x7fb878055fb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb88800b000 tx=0x7fb8880091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:17.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.363+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb89809e4a0 con 0x7fb8980a1b30 2022-07-20T03:11:17.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.527+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb89809e4a0 con 0x7fb8980a1b30 2022-07-20T03:11:17.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.527+0000 7fb895ffb700 1 -- 172.21.15.61:0/1553801228 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fb89809e4a0 con 0x7fb8980a1b30 2022-07-20T03:11:17.628 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:17.628 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:17.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb878053af0 msgr2=0x7fb878055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb878053af0 0x7fb878055fb0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7fb88800b000 tx=0x7fb8880091d0).stop 2022-07-20T03:11:17.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 msgr2=0x7fb898155da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:17.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 0x7fb898155da0 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7fb88c001390 tx=0x7fb88c013000).stop 2022-07-20T03:11:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 shutdown_connections 2022-07-20T03:11:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb878053af0 0x7fb878055fb0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8981485d0 0x7fb898148a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8980a1b30 0x7fb898155da0 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 --2- 172.21.15.61:0/1553801228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb898063d20 0x7fb8981558b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:17.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 >> 172.21.15.61:0/1553801228 conn(0x7fb884004500 msgr2=0x7fb8981461e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:17.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 shutdown_connections 2022-07-20T03:11:17.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:17.531+0000 7fb89f249700 1 -- 172.21.15.61:0/1553801228 wait complete. 2022-07-20T03:11:17.632 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:17.841 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:11:12.676790+0000', 'modified': '2022-07-20T03:11:14.703208+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5610}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5610': {'gid': 5610, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2246781672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2246781672}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2246781672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:11:17.842 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:11:17.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:11:17.995 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5610,"want_state":"up:active","state":"up:active","rank_uptime":5.2747130670000004,"mdsmap_epoch":29,"osdmap_epoch":52,"osdmap_epoch_barrier":52,"uptime":11.654879960000001} 2022-07-20T03:11:17.996 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5610, "mdsmap_epoch": 29, "osdmap_epoch": 52, "osdmap_epoch_barrier": 52, "rank_uptime": 5.274713067, "state": "up:active", "uptime": 11.65487996, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:11:17.997 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:11:18.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 Processor -- start 2022-07-20T03:11:18.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 -- start start 2022-07-20T03:11:18.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71180b5990 0x7f711813fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aecf0 0x7f7118140020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71180b5d70 con 0x7f71180aff80 2022-07-20T03:11:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71180651c0 con 0x7f71180b5990 2022-07-20T03:11:18.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71180b0690 con 0x7f71180aecf0 2022-07-20T03:11:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71180b5990 0x7f711813fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71180b5990 0x7f711813fb30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45294/0 (socket says 172.21.15.61:45294) 2022-07-20T03:11:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 -- 172.21.15.61:0/980572645 learned_addr learned my addr 172.21.15.61:0/980572645 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:18.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f71177fe700 1 --2- 172.21.15.61:0/980572645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aecf0 0x7f7118140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 -- 172.21.15.61:0/980572645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aecf0 msgr2=0x7f7118140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 --2- 172.21.15.61:0/980572645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aecf0 0x7f7118140020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 -- 172.21.15.61:0/980572645 >> v1:172.21.15.61:6789/0 conn(0x7f71180aff80 legacy=0x7f711805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 -- 172.21.15.61:0/980572645 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71180b0930 con 0x7f71180b5990 2022-07-20T03:11:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7117fff700 1 --2- 172.21.15.61:0/980572645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180b5990 0x7f711813fb30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f71100032f0 tx=0x7f7110009290).ready entity=mon.0 client_cookie=fdaf79b7a0062d8b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7116ffd700 1 -- 172.21.15.61:0/980572645 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f711000b8a0 con 0x7f71180b5990 2022-07-20T03:11:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7116ffd700 1 -- 172.21.15.61:0/980572645 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f711000ba00 con 0x7f71180b5990 2022-07-20T03:11:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f7116ffd700 1 -- 172.21.15.61:0/980572645 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f711000bb70 con 0x7f71180b5990 2022-07-20T03:11:18.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 -- 172.21.15.61:0/980572645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180b5990 msgr2=0x7f711813fb30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.123+0000 7f711d261700 1 --2- 172.21.15.61:0/980572645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180b5990 0x7f711813fb30 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f71100032f0 tx=0x7f7110009290).stop 2022-07-20T03:11:18.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- 172.21.15.61:0/980572645 shutdown_connections 2022-07-20T03:11:18.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 --2- 172.21.15.61:0/980572645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aecf0 0x7f7118140020 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 --2- 172.21.15.61:0/980572645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180b5990 0x7f711813fb30 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- 172.21.15.61:0/980572645 >> 172.21.15.61:0/980572645 conn(0x7f71180634e0 msgr2=0x7f71180b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- 172.21.15.61:0/980572645 shutdown_connections 2022-07-20T03:11:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- 172.21.15.61:0/980572645 wait complete. 2022-07-20T03:11:18.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 Processor -- start 2022-07-20T03:11:18.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- start start 2022-07-20T03:11:18.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 0x7f71181435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aff80 0x7f71181443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71180b5990 0x7f71181448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7118113940 con 0x7f71180aecf0 2022-07-20T03:11:18.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7118115250 con 0x7f71180b5990 2022-07-20T03:11:18.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711d261700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7118114fb0 con 0x7f71180aff80 2022-07-20T03:11:18.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 0x7f71181435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 0x7f71181435c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45300/0 (socket says 172.21.15.61:45300) 2022-07-20T03:11:18.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 -- 172.21.15.61:0/3399110840 learned_addr learned my addr 172.21.15.61:0/3399110840 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:18.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f71177fe700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aff80 0x7f71181443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711ca60700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71180b5990 0x7f71181448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 -- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aff80 msgr2=0x7f71181443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aff80 0x7f71181443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 -- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71180b5990 msgr2=0x7f71181448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71180b5990 0x7f71181448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f7117fff700 1 -- 172.21.15.61:0/3399110840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7110009530 con 0x7f71180aecf0 2022-07-20T03:11:18.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f711ca60700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71180b5990 0x7f71181448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:18.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.127+0000 7f71177fe700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aff80 0x7f71181443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:18.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f7117fff700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 0x7f71181435c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f71181432e0 tx=0x7f7110016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7110021480 con 0x7f71180aecf0 2022-07-20T03:11:18.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71100215e0 con 0x7f71180aecf0 2022-07-20T03:11:18.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7118143ca0 con 0x7f71180aecf0 2022-07-20T03:11:18.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7110021940 con 0x7f71180aecf0 2022-07-20T03:11:18.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7118112ce0 con 0x7f71180aecf0 2022-07-20T03:11:18.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71180574e0 con 0x7f71180aecf0 2022-07-20T03:11:18.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f711000f070 con 0x7f71180aecf0 2022-07-20T03:11:18.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71157fa700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70f8053a40 0x7f70f8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f711006d0d0 con 0x7f71180aecf0 2022-07-20T03:11:18.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.131+0000 7f71177fe700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70f8053a40 0x7f70f8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.135+0000 7f71177fe700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70f8053a40 0x7f70f8055f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f710800c000 tx=0x7f7108008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.135+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71100440e0 con 0x7f71180aecf0 2022-07-20T03:11:18.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7118067ea0 con 0x7f71180aecf0 2022-07-20T03:11:18.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f71157fa700 1 -- 172.21.15.61:0/3399110840 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16088 (secure 0 0 0) 0x7f7110015020 con 0x7f71180aecf0 2022-07-20T03:11:18.298 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:18.298 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":52,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:11:13.680406+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:11:10.825678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T03:11:11.707209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:11:18.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70f8053a40 msgr2=0x7f70f8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70f8053a40 0x7f70f8055f00 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f710800c000 tx=0x7f7108008040).stop 2022-07-20T03:11:18.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 msgr2=0x7f71181435c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 0x7f71181435c0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7f71181432e0 tx=0x7f7110016000).stop 2022-07-20T03:11:18.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 shutdown_connections 2022-07-20T03:11:18.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70f8053a40 0x7f70f8055f00 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71180b5990 0x7f71181448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71180aff80 0x7f71181443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 --2- 172.21.15.61:0/3399110840 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71180aecf0 0x7f71181435c0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.263+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 >> 172.21.15.61:0/3399110840 conn(0x7f71180634e0 msgr2=0x7f711805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:18.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.267+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 shutdown_connections 2022-07-20T03:11:18.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.267+0000 7f711d261700 1 -- 172.21.15.61:0/3399110840 wait complete. 2022-07-20T03:11:18.325 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:11:10.825678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:11:11.707209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:11:18.335 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:11:18.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.511+0000 7fb4bf8b6700 1 Processor -- start 2022-07-20T03:11:18.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.511+0000 7fb4bf8b6700 1 -- start start 2022-07-20T03:11:18.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b809df40 0x7fb4b8145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb4b8060ee0 0x7fb4b809cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4b80607e0 con 0x7fb4b80b4d60 2022-07-20T03:11:18.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4b8060d20 con 0x7fb4b8060ee0 2022-07-20T03:11:18.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b8060a80 con 0x7fb4b809df40 2022-07-20T03:11:18.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bce51700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb4b80b4d60 0x7fb4b8145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41908/0 (socket says 172.21.15.61:41908) 2022-07-20T03:11:18.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bce51700 1 -- 172.21.15.61:0/63302409 learned_addr learned my addr 172.21.15.61:0/63302409 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:18.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bd652700 1 --2- 172.21.15.61:0/63302409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b809df40 0x7fb4b8145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bde53700 1 --2- 172.21.15.61:0/63302409 >> v2:172.21.15.117:3300/0 conn(0x7fb4b8060ee0 0x7fb4b809cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4affff700 1 -- 172.21.15.61:0/63302409 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3160382939 0 0) 0x7fb4b80607e0 con 0x7fb4b80b4d60 2022-07-20T03:11:18.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4affff700 1 -- 172.21.15.61:0/63302409 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb49c003260 con 0x7fb4b80b4d60 2022-07-20T03:11:18.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bd652700 1 -- 172.21.15.61:0/63302409 >> v2:172.21.15.117:3300/0 conn(0x7fb4b8060ee0 msgr2=0x7fb4b809cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:18.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bd652700 1 --2- 172.21.15.61:0/63302409 >> v2:172.21.15.117:3300/0 conn(0x7fb4b8060ee0 0x7fb4b809cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bd652700 1 -- 172.21.15.61:0/63302409 >> v1:172.21.15.61:6789/0 conn(0x7fb4b80b4d60 legacy=0x7fb4b8145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bd652700 1 -- 172.21.15.61:0/63302409 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b809e320 con 0x7fb4b809df40 2022-07-20T03:11:18.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bd652700 1 --2- 172.21.15.61:0/63302409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b809df40 0x7fb4b8145680 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fb4a8002f70 tx=0x7fb4a8006d90).ready entity=mon.2 client_cookie=14cb66df18703071 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4affff700 1 -- 172.21.15.61:0/63302409 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4a800e590 con 0x7fb4b809df40 2022-07-20T03:11:18.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4affff700 1 -- 172.21.15.61:0/63302409 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a8000e90 con 0x7fb4b809df40 2022-07-20T03:11:18.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4affff700 1 -- 172.21.15.61:0/63302409 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4a800e8e0 con 0x7fb4b809df40 2022-07-20T03:11:18.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/63302409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b809df40 msgr2=0x7fb4b8145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/63302409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b809df40 0x7fb4b8145680 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fb4a8002f70 tx=0x7fb4a8006d90).stop 2022-07-20T03:11:18.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/63302409 shutdown_connections 2022-07-20T03:11:18.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/63302409 >> v2:172.21.15.117:3300/0 conn(0x7fb4b8060ee0 0x7fb4b809cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/63302409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b809df40 0x7fb4b8145680 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/63302409 >> 172.21.15.61:0/63302409 conn(0x7fb4b80b5f30 msgr2=0x7fb4b80b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:18.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/63302409 shutdown_connections 2022-07-20T03:11:18.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.515+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/63302409 wait complete. 2022-07-20T03:11:18.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 Processor -- start 2022-07-20T03:11:18.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 -- start start 2022-07-20T03:11:18.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4b8060ee0 0x7fb4b80b1500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 0x7fb4b80b8020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b80b4d60 0x7fb4b8090090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b80b2530 con 0x7fb4b809df40 2022-07-20T03:11:18.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b80b08a0 con 0x7fb4b8060ee0 2022-07-20T03:11:18.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bf8b6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4b80b0560 con 0x7fb4b80b4d60 2022-07-20T03:11:18.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 0x7fb4b80b8020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 0x7fb4b80b8020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45314/0 (socket says 172.21.15.61:45314) 2022-07-20T03:11:18.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 -- 172.21.15.61:0/2894204243 learned_addr learned my addr 172.21.15.61:0/2894204243 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:18.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bde53700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b80b4d60 0x7fb4b8090090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bd652700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4b8060ee0 0x7fb4b80b1500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 -- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b80b4d60 msgr2=0x7fb4b8090090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b80b4d60 0x7fb4b8090090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 -- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4b8060ee0 msgr2=0x7fb4b80b1500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4b8060ee0 0x7fb4b80b1500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 -- 172.21.15.61:0/2894204243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a800b120 con 0x7fb4b809df40 2022-07-20T03:11:18.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.519+0000 7fb4bce51700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 0x7fb4b80b8020 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fb4b4001110 tx=0x7fb4b4007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4b400ce00 con 0x7fb4b809df40 2022-07-20T03:11:18.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b40070d0 con 0x7fb4b809df40 2022-07-20T03:11:18.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4b80b2790 con 0x7fb4b809df40 2022-07-20T03:11:18.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4b4007430 con 0x7fb4b809df40 2022-07-20T03:11:18.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4b8155560 con 0x7fb4b809df40 2022-07-20T03:11:18.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4b80574e0 con 0x7fb4b809df40 2022-07-20T03:11:18.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb4b4010070 con 0x7fb4b809df40 2022-07-20T03:11:18.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4ae7fc700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4a0053920 0x7fb4a0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4b406b4c0 con 0x7fb4b809df40 2022-07-20T03:11:18.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.523+0000 7fb4bd652700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4a0053920 0x7fb4a0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.527+0000 7fb4bd652700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4a0053920 0x7fb4a0055de0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb4a801a000 tx=0x7fb4a8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.535+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4b4042380 con 0x7fb4b809df40 2022-07-20T03:11:18.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.659+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb4b8061810 con 0x7fb4b809df40 2022-07-20T03:11:18.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4ae7fc700 1 -- 172.21.15.61:0/2894204243 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v52) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb4b4018030 con 0x7fb4b809df40 2022-07-20T03:11:18.773 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:11:18.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4a0053920 msgr2=0x7fb4a0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4a0053920 0x7fb4a0055de0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7fb4a801a000 tx=0x7fb4a8014000).stop 2022-07-20T03:11:18.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 msgr2=0x7fb4b80b8020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 0x7fb4b80b8020 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7fb4b4001110 tx=0x7fb4b4007aa0).stop 2022-07-20T03:11:18.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 shutdown_connections 2022-07-20T03:11:18.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4a0053920 0x7fb4a0055de0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4b80b4d60 0x7fb4b8090090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4b809df40 0x7fb4b80b8020 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 --2- 172.21.15.61:0/2894204243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4b8060ee0 0x7fb4b80b1500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 >> 172.21.15.61:0/2894204243 conn(0x7fb4b80b5f30 msgr2=0x7fb4b8144a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:18.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 shutdown_connections 2022-07-20T03:11:18.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.663+0000 7fb4bf8b6700 1 -- 172.21.15.61:0/2894204243 wait complete. 2022-07-20T03:11:18.798 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:11:18.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.931+0000 7f73cffff700 1 Processor -- start 2022-07-20T03:11:18.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.931+0000 7f73cffff700 1 -- start start 2022-07-20T03:11:18.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.931+0000 7f73cffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d00b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:18.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.931+0000 7f73cffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d00b3f10 con 0x7f73d01461c0 2022-07-20T03:11:18.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.931+0000 7f73cffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73d00b3790 con 0x7f73d00b2ec0 2022-07-20T03:11:18.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.931+0000 7f73cffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00b4ea0 con 0x7f73d005d160 2022-07-20T03:11:18.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f73d01461c0 0x7f73d0146650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41924/0 (socket says 172.21.15.61:41924) 2022-07-20T03:11:18.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cf7fe700 1 -- 172.21.15.61:0/1654649167 learned_addr learned my addr 172.21.15.61:0/1654649167 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:18.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73ce7fc700 1 --2- 172.21.15.61:0/1654649167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d00b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:18.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cdffb700 1 -- 172.21.15.61:0/1654649167 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3395164849 0 0) 0x7f73d00b3f10 con 0x7f73d01461c0 2022-07-20T03:11:18.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cdffb700 1 -- 172.21.15.61:0/1654649167 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f73b0003260 con 0x7f73d01461c0 2022-07-20T03:11:18.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cdffb700 1 -- 172.21.15.61:0/1654649167 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282427453 0 0) 0x7f73d00b3790 con 0x7f73d00b2ec0 2022-07-20T03:11:18.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73ce7fc700 1 -- 172.21.15.61:0/1654649167 >> v1:172.21.15.117:6789/0 conn(0x7f73d00b2ec0 legacy=0x7f73d0142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73ce7fc700 1 -- 172.21.15.61:0/1654649167 >> v1:172.21.15.61:6789/0 conn(0x7f73d01461c0 legacy=0x7f73d0146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73ce7fc700 1 -- 172.21.15.61:0/1654649167 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73d00b2620 con 0x7f73d005d160 2022-07-20T03:11:18.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73ce7fc700 1 --2- 172.21.15.61:0/1654649167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d00b32a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f73b8002f70 tx=0x7f73b8006df0).ready entity=mon.2 client_cookie=8ac6ae8248bd5178 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:18.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cdffb700 1 -- 172.21.15.61:0/1654649167 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73b800d570 con 0x7f73d005d160 2022-07-20T03:11:18.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cdffb700 1 -- 172.21.15.61:0/1654649167 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73b8000e90 con 0x7f73d005d160 2022-07-20T03:11:18.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cdffb700 1 -- 172.21.15.61:0/1654649167 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73b800d8c0 con 0x7f73d005d160 2022-07-20T03:11:18.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 -- 172.21.15.61:0/1654649167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 msgr2=0x7f73d00b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:18.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 --2- 172.21.15.61:0/1654649167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d00b32a0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f73b8002f70 tx=0x7f73b8006df0).stop 2022-07-20T03:11:18.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 -- 172.21.15.61:0/1654649167 shutdown_connections 2022-07-20T03:11:18.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 --2- 172.21.15.61:0/1654649167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d00b32a0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:18.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 -- 172.21.15.61:0/1654649167 >> 172.21.15.61:0/1654649167 conn(0x7f73d0066e20 msgr2=0x7f73d0067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:18.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 -- 172.21.15.61:0/1654649167 shutdown_connections 2022-07-20T03:11:19.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.935+0000 7f73cffff700 1 -- 172.21.15.61:0/1654649167 wait complete. 2022-07-20T03:11:19.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 Processor -- start 2022-07-20T03:11:19.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 -- start start 2022-07-20T03:11:19.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d01450b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 0x7f73d00af480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 0x7f73d0152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d013b910 con 0x7f73d01461c0 2022-07-20T03:11:19.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73d0141570 con 0x7f73d00b2ec0 2022-07-20T03:11:19.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73d00aedb0 con 0x7f73d005d160 2022-07-20T03:11:19.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 0x7f73d0152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73ce7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 0x7f73d00af480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73ce7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 0x7f73d00af480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55010/0 (socket says 172.21.15.61:55010) 2022-07-20T03:11:19.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73ce7fc700 1 -- 172.21.15.61:0/3091448095 learned_addr learned my addr 172.21.15.61:0/3091448095 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:19.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 0x7f73d0152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45332/0 (socket says 172.21.15.61:45332) 2022-07-20T03:11:19.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73ceffd700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d01450b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 -- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 msgr2=0x7f73d01450b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d01450b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 -- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 msgr2=0x7f73d00af480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 0x7f73d00af480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 -- 172.21.15.61:0/3091448095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73b800b180 con 0x7f73d01461c0 2022-07-20T03:11:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73ce7fc700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 0x7f73d00af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:19.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cf7fe700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 0x7f73d0152230 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f73d00aebb0 tx=0x7f73c8007d40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c8005ba0 con 0x7f73d01461c0 2022-07-20T03:11:19.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f73c8005d00 con 0x7f73d01461c0 2022-07-20T03:11:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73c80072c0 con 0x7f73d01461c0 2022-07-20T03:11:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73d00acae0 con 0x7f73d01461c0 2022-07-20T03:11:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.939+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73d0141cc0 con 0x7f73d01461c0 2022-07-20T03:11:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.943+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73d00af1e0 con 0x7f73d01461c0 2022-07-20T03:11:19.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.943+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f73c8010070 con 0x7f73d01461c0 2022-07-20T03:11:19.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.943+0000 7f73c7fff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b4053a40 0x7f73b4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.943+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f73c806b6f0 con 0x7f73d01461c0 2022-07-20T03:11:19.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.943+0000 7f73ceffd700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b4053a40 0x7f73b4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.943+0000 7f73ceffd700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b4053a40 0x7f73b4055f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f73c000d000 tx=0x7f73c0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:18.951+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73d00af1e0 con 0x7f73d01461c0 2022-07-20T03:11:19.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.079+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f73d00af1e0 con 0x7f73d01461c0 2022-07-20T03:11:19.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.079+0000 7f73c7fff700 1 -- 172.21.15.61:0/3091448095 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v52) v1 ==== 92+0+10 (secure 0 0 0) 0x7f73d00af1e0 con 0x7f73d01461c0 2022-07-20T03:11:19.139 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:19.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b4053a40 msgr2=0x7f73b4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b4053a40 0x7f73b4055f00 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f73c000d000 tx=0x7f73c0007b90).stop 2022-07-20T03:11:19.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 msgr2=0x7f73d0152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 0x7f73d0152230 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f73d00aebb0 tx=0x7f73c8007d40).stop 2022-07-20T03:11:19.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 shutdown_connections 2022-07-20T03:11:19.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f73b4053a40 0x7f73b4055f00 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73d00b2ec0 0x7f73d00af480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73d01461c0 0x7f73d0152230 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 --2- 172.21.15.61:0/3091448095 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73d005d160 0x7f73d01450b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 >> 172.21.15.61:0/3091448095 conn(0x7f73d0066e20 msgr2=0x7f73d0146ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:19.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 shutdown_connections 2022-07-20T03:11:19.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.083+0000 7f73cffff700 1 -- 172.21.15.61:0/3091448095 wait complete. 2022-07-20T03:11:19.190 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:11:19.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 Processor -- start 2022-07-20T03:11:19.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 -- start start 2022-07-20T03:11:19.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fac380b3cf0 0x7fac38142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac381419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac380ba180 con 0x7fac380b9da0 2022-07-20T03:11:19.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac380ad5b0 con 0x7fac380b3cf0 2022-07-20T03:11:19.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac3cfaa700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac381461b0 con 0x7fac38142f40 2022-07-20T03:11:19.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac36ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fac380b3cf0 0x7fac38142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.307+0000 7fac377fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fac380b9da0 0x7fac381468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41934/0 (socket says 172.21.15.61:41934) 2022-07-20T03:11:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac377fe700 1 -- 172.21.15.61:0/1626916105 learned_addr learned my addr 172.21.15.61:0/1626916105 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:19.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac37fff700 1 --2- 172.21.15.61:0/1626916105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac381419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac36ffd700 1 -- 172.21.15.61:0/1626916105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 msgr2=0x7fac381419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac36ffd700 1 --2- 172.21.15.61:0/1626916105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac381419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac36ffd700 1 -- 172.21.15.61:0/1626916105 >> v1:172.21.15.61:6789/0 conn(0x7fac380b9da0 legacy=0x7fac381468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac367fc700 1 -- 172.21.15.61:0/1626916105 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 268077820 0 0) 0x7fac380ba180 con 0x7fac380b9da0 2022-07-20T03:11:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac36ffd700 1 -- 172.21.15.61:0/1626916105 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac38146450 con 0x7fac380b3cf0 2022-07-20T03:11:19.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac36ffd700 1 --2- 172.21.15.61:0/1626916105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b3cf0 0x7fac38142a50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fac20002f70 tx=0x7fac20006df0).ready entity=mon.0 client_cookie=102bae5b06d9498b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac367fc700 1 -- 172.21.15.61:0/1626916105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac2000d570 con 0x7fac380b3cf0 2022-07-20T03:11:19.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac367fc700 1 -- 172.21.15.61:0/1626916105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac20000e90 con 0x7fac380b3cf0 2022-07-20T03:11:19.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac367fc700 1 -- 172.21.15.61:0/1626916105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac2000d8c0 con 0x7fac380b3cf0 2022-07-20T03:11:19.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 -- 172.21.15.61:0/1626916105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b3cf0 msgr2=0x7fac38142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/1626916105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b3cf0 0x7fac38142a50 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fac20002f70 tx=0x7fac20006df0).stop 2022-07-20T03:11:19.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 -- 172.21.15.61:0/1626916105 shutdown_connections 2022-07-20T03:11:19.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/1626916105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac381419a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/1626916105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b3cf0 0x7fac38142a50 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 -- 172.21.15.61:0/1626916105 >> 172.21.15.61:0/1626916105 conn(0x7fac380678c0 msgr2=0x7fac380b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:19.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 -- 172.21.15.61:0/1626916105 shutdown_connections 2022-07-20T03:11:19.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.311+0000 7fac3cfaa700 1 -- 172.21.15.61:0/1626916105 wait complete. 2022-07-20T03:11:19.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 Processor -- start 2022-07-20T03:11:19.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 -- start start 2022-07-20T03:11:19.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac380b3cf0 0x7fac380affd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 0x7fac3813ea30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac3813ef20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38155060 con 0x7fac380b9da0 2022-07-20T03:11:19.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac38155f70 con 0x7fac380b3cf0 2022-07-20T03:11:19.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac380af890 con 0x7fac38142f40 2022-07-20T03:11:19.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 0x7fac3813ea30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 0x7fac3813ea30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45342/0 (socket says 172.21.15.61:45342) 2022-07-20T03:11:19.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 -- 172.21.15.61:0/245386497 learned_addr learned my addr 172.21.15.61:0/245386497 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:19.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac37fff700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac3813ef20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac377fe700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac380b3cf0 0x7fac380affd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 -- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 msgr2=0x7fac3813ef20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac3813ef20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 -- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac380b3cf0 msgr2=0x7fac380affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac380b3cf0 0x7fac380affd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 -- 172.21.15.61:0/245386497 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac2000b180 con 0x7fac380b9da0 2022-07-20T03:11:19.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac37fff700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac3813ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:19.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac36ffd700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 0x7fac3813ea30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fac38145220 tx=0x7fac20014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac20016070 con 0x7fac380b9da0 2022-07-20T03:11:19.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac20013d10 con 0x7fac380b9da0 2022-07-20T03:11:19.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac38145300 con 0x7fac380b9da0 2022-07-20T03:11:19.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac2001e640 con 0x7fac380b9da0 2022-07-20T03:11:19.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.315+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac38152bc0 con 0x7fac380b9da0 2022-07-20T03:11:19.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.319+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac380af6b0 con 0x7fac380b9da0 2022-07-20T03:11:19.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.319+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fac2000f030 con 0x7fac380b9da0 2022-07-20T03:11:19.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.319+0000 7fac34ff9700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fac1c0539b0 0x7fac1c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.319+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fac2006cbe0 con 0x7fac380b9da0 2022-07-20T03:11:19.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.319+0000 7fac377fe700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fac1c0539b0 0x7fac1c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.319+0000 7fac377fe700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fac1c0539b0 0x7fac1c055e70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fac2800b000 tx=0x7fac28009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.331+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fac20043aa0 con 0x7fac380b9da0 2022-07-20T03:11:19.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.455+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fac380574e0 con 0x7fac380b9da0 2022-07-20T03:11:19.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac34ff9700 1 -- 172.21.15.61:0/245386497 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+10 (secure 0 0 0) 0x7fac20023020 con 0x7fac380b9da0 2022-07-20T03:11:19.533 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:19.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fac1c0539b0 msgr2=0x7fac1c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fac1c0539b0 0x7fac1c055e70 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7fac2800b000 tx=0x7fac28009040).stop 2022-07-20T03:11:19.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 msgr2=0x7fac3813ea30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 0x7fac3813ea30 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fac38145220 tx=0x7fac20014000).stop 2022-07-20T03:11:19.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 shutdown_connections 2022-07-20T03:11:19.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fac1c0539b0 0x7fac1c055e70 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac38142f40 0x7fac3813ef20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac380b9da0 0x7fac3813ea30 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 --2- 172.21.15.61:0/245386497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac380b3cf0 0x7fac380affd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 >> 172.21.15.61:0/245386497 conn(0x7fac380678c0 msgr2=0x7fac380b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:19.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 shutdown_connections 2022-07-20T03:11:19.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.459+0000 7fac3cfaa700 1 -- 172.21.15.61:0/245386497 wait complete. 2022-07-20T03:11:19.583 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:11:19.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 Processor -- start 2022-07-20T03:11:19.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 -- start start 2022-07-20T03:11:19.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5f600b2ec0 0x7f5f600b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5f6005d160 0x7f5f60142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5f600b3f10 con 0x7f5f60142700 2022-07-20T03:11:19.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f600b4ea0 con 0x7f5f600b2ec0 2022-07-20T03:11:19.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f66d5e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5f60142b30 con 0x7f5f6005d160 2022-07-20T03:11:19.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.707+0000 7f5f65d5c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5f600b2ec0 0x7f5f600b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f6655d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5f60142700 0x7f5f601461c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41950/0 (socket says 172.21.15.61:41950) 2022-07-20T03:11:19.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f6655d700 1 -- 172.21.15.61:0/281258959 learned_addr learned my addr 172.21.15.61:0/281258959 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:19.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f6555b700 1 --2- 172.21.15.61:0/281258959 >> v2:172.21.15.117:3300/0 conn(0x7f5f6005d160 0x7f5f60142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f64d5a700 1 -- 172.21.15.61:0/281258959 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 379117897 0 0) 0x7f5f600b3f10 con 0x7f5f60142700 2022-07-20T03:11:19.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f64d5a700 1 -- 172.21.15.61:0/281258959 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5f48003260 con 0x7f5f60142700 2022-07-20T03:11:19.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f65d5c700 1 -- 172.21.15.61:0/281258959 >> v2:172.21.15.117:3300/0 conn(0x7f5f6005d160 msgr2=0x7f5f60142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:19.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f65d5c700 1 --2- 172.21.15.61:0/281258959 >> v2:172.21.15.117:3300/0 conn(0x7f5f6005d160 0x7f5f60142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f65d5c700 1 -- 172.21.15.61:0/281258959 >> v1:172.21.15.61:6789/0 conn(0x7f5f60142700 legacy=0x7f5f601461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f65d5c700 1 -- 172.21.15.61:0/281258959 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f600b3790 con 0x7f5f600b2ec0 2022-07-20T03:11:19.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f65d5c700 1 --2- 172.21.15.61:0/281258959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f600b32a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f5f54002f70 tx=0x7f5f54006df0).ready entity=mon.0 client_cookie=db8839c8aaf600c7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f64d5a700 1 -- 172.21.15.61:0/281258959 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f5400d4f0 con 0x7f5f600b2ec0 2022-07-20T03:11:19.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f64d5a700 1 -- 172.21.15.61:0/281258959 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f54000e90 con 0x7f5f600b2ec0 2022-07-20T03:11:19.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f64d5a700 1 -- 172.21.15.61:0/281258959 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f5400d840 con 0x7f5f600b2ec0 2022-07-20T03:11:19.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 -- 172.21.15.61:0/281258959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 msgr2=0x7f5f600b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/281258959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f600b32a0 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f5f54002f70 tx=0x7f5f54006df0).stop 2022-07-20T03:11:19.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 -- 172.21.15.61:0/281258959 shutdown_connections 2022-07-20T03:11:19.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/281258959 >> v2:172.21.15.117:3300/0 conn(0x7f5f6005d160 0x7f5f60142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/281258959 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f600b32a0 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 -- 172.21.15.61:0/281258959 >> 172.21.15.61:0/281258959 conn(0x7f5f60066e20 msgr2=0x7f5f60067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:19.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 -- 172.21.15.61:0/281258959 shutdown_connections 2022-07-20T03:11:19.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.711+0000 7f5f66d5e700 1 -- 172.21.15.61:0/281258959 wait complete. 2022-07-20T03:11:19.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 Processor -- start 2022-07-20T03:11:19.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 -- start start 2022-07-20T03:11:19.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f6005d160 0x7f5f600ad7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f6013e5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f60142700 0x7f5f60145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600aef60 con 0x7f5f600b2ec0 2022-07-20T03:11:19.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f600ae890 con 0x7f5f6005d160 2022-07-20T03:11:19.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60146a20 con 0x7f5f60142700 2022-07-20T03:11:19.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f6013e5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f6013e5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45360/0 (socket says 172.21.15.61:45360) 2022-07-20T03:11:19.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 -- 172.21.15.61:0/3773043158 learned_addr learned my addr 172.21.15.61:0/3773043158 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:19.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f65d5c700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f6005d160 0x7f5f600ad7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6655d700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f60142700 0x7f5f60145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 -- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f60142700 msgr2=0x7f5f60145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f60142700 0x7f5f60145050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 -- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f6005d160 msgr2=0x7f5f600ad7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f6005d160 0x7f5f600ad7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 -- 172.21.15.61:0/3773043158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f5400b180 con 0x7f5f600b2ec0 2022-07-20T03:11:19.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f6555b700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f6013e5e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f5f5c00c000 tx=0x7f5f5c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f5c011750 con 0x7f5f600b2ec0 2022-07-20T03:11:19.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5f5c0118b0 con 0x7f5f600b2ec0 2022-07-20T03:11:19.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5f5c011c10 con 0x7f5f600b2ec0 2022-07-20T03:11:19.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f60146cc0 con 0x7f5f600b2ec0 2022-07-20T03:11:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.715+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f600acae0 con 0x7f5f600b2ec0 2022-07-20T03:11:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.719+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f600b2620 con 0x7f5f600b2ec0 2022-07-20T03:11:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.719+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5f5c011d70 con 0x7f5f600b2ec0 2022-07-20T03:11:19.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.719+0000 7f5f52ffd700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f40053a40 0x7f5f40055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.719+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5f5c06c230 con 0x7f5f600b2ec0 2022-07-20T03:11:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.719+0000 7f5f65d5c700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f40053a40 0x7f5f40055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.719+0000 7f5f65d5c700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f40053a40 0x7f5f40055f00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f5f5401a000 tx=0x7f5f54014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:19.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.727+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5f5c043020 con 0x7f5f600b2ec0 2022-07-20T03:11:19.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.851+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f5f600574e0 con 0x7f5f600b2ec0 2022-07-20T03:11:19.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.851+0000 7f5f52ffd700 1 -- 172.21.15.61:0/3773043158 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+10 (secure 0 0 0) 0x7f5f5c046640 con 0x7f5f600b2ec0 2022-07-20T03:11:19.963 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:19.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f40053a40 msgr2=0x7f5f40055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f40053a40 0x7f5f40055f00 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f5f5401a000 tx=0x7f5f54014000).stop 2022-07-20T03:11:19.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 msgr2=0x7f5f6013e5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:19.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f6013e5e0 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f5f5c00c000 tx=0x7f5f5c006df0).stop 2022-07-20T03:11:19.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 shutdown_connections 2022-07-20T03:11:19.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5f40053a40 0x7f5f40055f00 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5f60142700 0x7f5f60145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5f600b2ec0 0x7f5f6013e5e0 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 --2- 172.21.15.61:0/3773043158 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5f6005d160 0x7f5f600ad7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:19.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 >> 172.21.15.61:0/3773043158 conn(0x7f5f60066e20 msgr2=0x7f5f60141100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:19.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 shutdown_connections 2022-07-20T03:11:19.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:19.855+0000 7f5f66d5e700 1 -- 172.21.15.61:0/3773043158 wait complete. 2022-07-20T03:11:20.089 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:11:20.092 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:20.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 Processor -- start 2022-07-20T03:11:20.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 -- start start 2022-07-20T03:11:20.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f49300b5650 0x7f49300b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:20.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f493005bd90 0x7f4930142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:20.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f49300aea30 con 0x7f49300b1200 2022-07-20T03:11:20.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4930142970 con 0x7f49300b5650 2022-07-20T03:11:20.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4936138700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f49300aec60 con 0x7f493005bd90 2022-07-20T03:11:20.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935136700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f49300b5650 0x7f49300b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:20.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4934935700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f49300b1200 0x7f493005c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:41964/0 (socket says 172.21.15.61:41964) 2022-07-20T03:11:20.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4934935700 1 -- 172.21.15.61:0/3171458895 learned_addr learned my addr 172.21.15.61:0/3171458895 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:20.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935937700 1 --2- 172.21.15.61:0/3171458895 >> v2:172.21.15.117:3300/0 conn(0x7f493005bd90 0x7f4930142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:20.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935136700 1 -- 172.21.15.61:0/3171458895 >> v2:172.21.15.117:3300/0 conn(0x7f493005bd90 msgr2=0x7f4930142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:20.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935136700 1 --2- 172.21.15.61:0/3171458895 >> v2:172.21.15.117:3300/0 conn(0x7f493005bd90 0x7f4930142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935136700 1 -- 172.21.15.61:0/3171458895 >> v1:172.21.15.61:6789/0 conn(0x7f49300b1200 legacy=0x7f493005c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:20.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f492ffff700 1 -- 172.21.15.61:0/3171458895 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1850272570 0 0) 0x7f49300aea30 con 0x7f49300b1200 2022-07-20T03:11:20.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935136700 1 -- 172.21.15.61:0/3171458895 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f49300af180 con 0x7f49300b5650 2022-07-20T03:11:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935937700 1 --2- 172.21.15.61:0/3171458895 >> v2:172.21.15.117:3300/0 conn(0x7f493005bd90 0x7f4930142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:11:20.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f4935136700 1 --2- 172.21.15.61:0/3171458895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b5650 0x7f49300b5a30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f49280032f0 tx=0x7f49280091d0).ready entity=mon.0 client_cookie=db47f3e2349cd864 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:20.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f492ffff700 1 -- 172.21.15.61:0/3171458895 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4928009ea0 con 0x7f49300b5650 2022-07-20T03:11:20.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f492ffff700 1 -- 172.21.15.61:0/3171458895 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f492800b870 con 0x7f49300b5650 2022-07-20T03:11:20.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.247+0000 7f492ffff700 1 -- 172.21.15.61:0/3171458895 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f492800b9e0 con 0x7f49300b5650 2022-07-20T03:11:20.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 -- 172.21.15.61:0/3171458895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b5650 msgr2=0x7f49300b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:20.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 --2- 172.21.15.61:0/3171458895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b5650 0x7f49300b5a30 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f49280032f0 tx=0x7f49280091d0).stop 2022-07-20T03:11:20.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 -- 172.21.15.61:0/3171458895 shutdown_connections 2022-07-20T03:11:20.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 --2- 172.21.15.61:0/3171458895 >> v2:172.21.15.117:3300/0 conn(0x7f493005bd90 0x7f4930142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 --2- 172.21.15.61:0/3171458895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b5650 0x7f49300b5a30 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 -- 172.21.15.61:0/3171458895 >> 172.21.15.61:0/3171458895 conn(0x7f49300634e0 msgr2=0x7f49300b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:20.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 -- 172.21.15.61:0/3171458895 shutdown_connections 2022-07-20T03:11:20.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 -- 172.21.15.61:0/3171458895 wait complete. 2022-07-20T03:11:20.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 Processor -- start 2022-07-20T03:11:20.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.251+0000 7f4936138700 1 -- start start 2022-07-20T03:11:20.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f493005bd90 0x7f49300b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:20.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 0x7f4930155650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:20.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49300b5650 0x7f4930155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:20.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4930144d30 con 0x7f49300b1200 2022-07-20T03:11:20.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4930145b20 con 0x7f49300b5650 2022-07-20T03:11:20.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4930151f70 con 0x7f493005bd90 2022-07-20T03:11:20.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 0x7f4930155650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:20.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 0x7f4930155650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45374/0 (socket says 172.21.15.61:45374) 2022-07-20T03:11:20.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 -- 172.21.15.61:0/3974346322 learned_addr learned my addr 172.21.15.61:0/3974346322 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:20.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4935136700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f493005bd90 0x7f49300b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:20.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4935937700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49300b5650 0x7f4930155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:20.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 -- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f493005bd90 msgr2=0x7f49300b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:20.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f493005bd90 0x7f49300b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 -- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49300b5650 msgr2=0x7f4930155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:20.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49300b5650 0x7f4930155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 -- 172.21.15.61:0/3974346322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4928009470 con 0x7f49300b1200 2022-07-20T03:11:20.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4934935700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 0x7f4930155650 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f4924001050 tx=0x7f4924001820).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:20.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f492400ce70 con 0x7f49300b1200 2022-07-20T03:11:20.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f49240070d0 con 0x7f49300b1200 2022-07-20T03:11:20.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4930142640 con 0x7f49300b1200 2022-07-20T03:11:20.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f49240054e0 con 0x7f49300b1200 2022-07-20T03:11:20.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4930140ef0 con 0x7f49300b1200 2022-07-20T03:11:20.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.255+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4930145960 con 0x7f49300b1200 2022-07-20T03:11:20.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.259+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4924010070 con 0x7f49300b1200 2022-07-20T03:11:20.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.259+0000 7f492e7fc700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4910053a40 0x7f4910055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:20.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.259+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f492406b3d0 con 0x7f49300b1200 2022-07-20T03:11:20.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.259+0000 7f4935136700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4910053a40 0x7f4910055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:20.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.259+0000 7f4935136700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4910053a40 0x7f4910055f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f492801b000 tx=0x7f4928016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:20.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.263+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4924042330 con 0x7f49300b1200 2022-07-20T03:11:20.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.423+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f49300574e0 con 0x7f49300b1200 2022-07-20T03:11:20.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f492e7fc700 1 -- 172.21.15.61:0/3974346322 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f4924045950 con 0x7f49300b1200 2022-07-20T03:11:20.851 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:20.851 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:20.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4910053a40 msgr2=0x7f4910055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:20.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4910053a40 0x7f4910055f00 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f492801b000 tx=0x7f4928016000).stop 2022-07-20T03:11:20.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 msgr2=0x7f4930155650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:20.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 0x7f4930155650 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7f4924001050 tx=0x7f4924001820).stop 2022-07-20T03:11:20.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 shutdown_connections 2022-07-20T03:11:20.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4910053a40 0x7f4910055f00 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f49300b5650 0x7f4930155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f49300b1200 0x7f4930155650 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 --2- 172.21.15.61:0/3974346322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f493005bd90 0x7f49300b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:20.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 >> 172.21.15.61:0/3974346322 conn(0x7f49300634e0 msgr2=0x7f49300b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 shutdown_connections 2022-07-20T03:11:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:20.427+0000 7f4936138700 1 -- 172.21.15.61:0/3974346322 wait complete. 2022-07-20T03:11:20.930 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:21.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:11:12.676790+0000', 'modified': '2022-07-20T03:11:14.703208+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5610}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5610': {'gid': 5610, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2246781672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2246781672}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2246781672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:11:21.166 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:11:21.167 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:11:21.288 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5610,"want_state":"up:active","state":"up:active","rank_uptime":8.5977771290000007,"mdsmap_epoch":29,"osdmap_epoch":52,"osdmap_epoch_barrier":52,"uptime":14.977942232} 2022-07-20T03:11:21.289 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5610, "mdsmap_epoch": 29, "osdmap_epoch": 52, "osdmap_epoch_barrier": 52, "rank_uptime": 8.597777129, "state": "up:active", "uptime": 14.977942232, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:11:21.298 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:11:21.299 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:11:21.300 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:11:21.300 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:21.301 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:21.372 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:11:21.442 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:11:21.444 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:11:21.462 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:11:21.463 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:11:21.465 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:21.466 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:11:21.490 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:11:21.491 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:21.567 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:11:21.579 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:11:21.580 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:11:21.607 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:11:21.608 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:11:21.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:21.609 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:11:21.631 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:11:21.628+0000 7fcacf19a080 -1 init, newargv = 0x55a4fb96f210 newargc=9 2022-07-20T03:11:21.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22637]: starting ceph client 2022-07-20T03:11:21.644 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22637]: starting fuse 2022-07-20T03:11:21.648 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:11:21.662 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:11:21.663 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:21.664 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:11:21.683 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:11:21.690 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:11:21.691 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:11:21.708 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:11:21.709 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:21.767 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:11:21.768 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:11:21.769 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:11:21.769 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:21.799 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:11:21.800 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:11:21.816 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:11:21.836 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:11:21.837 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:21.858 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:11:21.859 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:11:21.859 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:11:21.860 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:11:21.861 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:11:21.916 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:21.917 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:11:21.930 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:11:21.978 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22637.asok 2022-07-20T03:11:22.014 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22637.asok 2022-07-20T03:11:22.015 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:22.016 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22637.asok status 2022-07-20T03:11:22.113 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:11:22.113 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:11:22.115 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:11:22.115 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:11:22.115 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:11:22.116 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:11:22.116 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:11:22.116 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22637", 2022-07-20T03:11:22.116 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:11:22.117 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:22.117 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:11:22.117 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:11:22.118 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5790, 2022-07-20T03:11:22.118 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:11:22.119 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:11:22.119 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:11:22.119 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5790 2022-07-20T03:11:22.119 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:22.120 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:11:22.120 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:11:22.120 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:11:22.120 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3834564268 2022-07-20T03:11:22.121 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:11:22.121 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:22.121 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:11:22.122 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:11:22.123 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:11:22.123 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3834564268 2022-07-20T03:11:22.123 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:22.123 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5790 172.21.15.124:0/3834564268", 2022-07-20T03:11:22.124 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/3834564268", 2022-07-20T03:11:22.124 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:11:22.124 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 29, 2022-07-20T03:11:22.126 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 52, 2022-07-20T03:11:22.126 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:11:22.126 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:11:22.126 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:11:22.127 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:11:22.129 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:22.130 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:22.130 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:22.229 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:11:22.229 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:11:22.230 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:22.231 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:22.274 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:22.275 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:22.286 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:22.367 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:11:22.368 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:11:22.369 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:11:22.390 DEBUG:teuthology.orchestra.run.smithi124:> import sys 2022-07-20T03:11:22.390 DEBUG:teuthology.orchestra.run.smithi124:> import time 2022-07-20T03:11:22.396 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:11:22.399 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:22.400 DEBUG:teuthology.orchestra.run.smithi124:> n = 1000 2022-07-20T03:11:22.400 DEBUG:teuthology.orchestra.run.smithi124:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T03:11:22.401 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:22.401 DEBUG:teuthology.orchestra.run.smithi124:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T03:11:22.401 DEBUG:teuthology.orchestra.run.smithi124:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T03:11:22.402 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:22.402 DEBUG:teuthology.orchestra.run.smithi124:> for i in range(0, n): 2022-07-20T03:11:22.410 DEBUG:teuthology.orchestra.run.smithi124:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T03:11:22.410 DEBUG:teuthology.orchestra.run.smithi124:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T03:11:22.411 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content'"'"') 2022-07-20T03:11:22.411 DEBUG:teuthology.orchestra.run.smithi124:> if True: 2022-07-20T03:11:22.411 DEBUG:teuthology.orchestra.run.smithi124:> f.flush() 2022-07-20T03:11:22.412 DEBUG:teuthology.orchestra.run.smithi124:> os.fsync(f.fileno()) 2022-07-20T03:11:22.412 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:11:31.703 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:31.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 Processor -- start 2022-07-20T03:11:31.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- start start 2022-07-20T03:11:31.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84dc0608b0 0x7f84dc05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:31.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0afa10 0x7f84dc0b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:31.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84dc0b42a0 con 0x7f84dc0b3de0 2022-07-20T03:11:31.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84dc060c90 con 0x7f84dc0608b0 2022-07-20T03:11:31.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0af410 con 0x7f84dc0afa10 2022-07-20T03:11:31.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84dc0608b0 0x7f84dc05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:31.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84dc0608b0 0x7f84dc05bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45378/0 (socket says 172.21.15.61:45378) 2022-07-20T03:11:31.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 -- 172.21.15.61:0/994516969 learned_addr learned my addr 172.21.15.61:0/994516969 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:31.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84db7fe700 1 --2- 172.21.15.61:0/994516969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0afa10 0x7f84dc0b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:31.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 -- 172.21.15.61:0/994516969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0afa10 msgr2=0x7f84dc0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:31.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 --2- 172.21.15.61:0/994516969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0afa10 0x7f84dc0b46d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:31.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 -- 172.21.15.61:0/994516969 >> v1:172.21.15.117:6789/0 conn(0x7f84dc0b3de0 legacy=0x7f84dc146350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:31.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 -- 172.21.15.61:0/994516969 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84dc0af6b0 con 0x7f84dc0608b0 2022-07-20T03:11:31.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84dbfff700 1 --2- 172.21.15.61:0/994516969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc05bd90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f84cc002f70 tx=0x7f84cc006eb0).ready entity=mon.0 client_cookie=64dd842202a74789 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:31.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84daffd700 1 -- 172.21.15.61:0/994516969 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84cc00d5c0 con 0x7f84dc0608b0 2022-07-20T03:11:31.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84daffd700 1 -- 172.21.15.61:0/994516969 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc000e90 con 0x7f84dc0608b0 2022-07-20T03:11:31.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84daffd700 1 -- 172.21.15.61:0/994516969 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84cc00d950 con 0x7f84dc0608b0 2022-07-20T03:11:31.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- 172.21.15.61:0/994516969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 msgr2=0x7f84dc05bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:31.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 --2- 172.21.15.61:0/994516969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc05bd90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f84cc002f70 tx=0x7f84cc006eb0).stop 2022-07-20T03:11:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- 172.21.15.61:0/994516969 shutdown_connections 2022-07-20T03:11:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 --2- 172.21.15.61:0/994516969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0afa10 0x7f84dc0b46d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 --2- 172.21.15.61:0/994516969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc05bd90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:31.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- 172.21.15.61:0/994516969 >> 172.21.15.61:0/994516969 conn(0x7f84dc0b5ee0 msgr2=0x7f84dc0b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- 172.21.15.61:0/994516969 shutdown_connections 2022-07-20T03:11:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.883+0000 7f84e244a700 1 -- 172.21.15.61:0/994516969 wait complete. 2022-07-20T03:11:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 Processor -- start 2022-07-20T03:11:31.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 -- start start 2022-07-20T03:11:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc09de50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84dc0afa10 0x7f84dc0901f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:31.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0b3de0 0x7f84dc1526d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc0937f0 con 0x7f84dc0608b0 2022-07-20T03:11:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc092c80 con 0x7f84dc0afa10 2022-07-20T03:11:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84dc151f90 con 0x7f84dc0b3de0 2022-07-20T03:11:31.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc09de50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:31.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84db7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84dc0afa10 0x7f84dc0901f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:31.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e09e7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0b3de0 0x7f84dc1526d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:31.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84db7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84dc0afa10 0x7f84dc0901f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55066/0 (socket says 172.21.15.61:55066) 2022-07-20T03:11:31.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84db7fe700 1 -- 172.21.15.61:0/2697327380 learned_addr learned my addr 172.21.15.61:0/2697327380 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:31.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 -- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0b3de0 msgr2=0x7f84dc1526d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:31.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0b3de0 0x7f84dc1526d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:31.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 -- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84dc0afa10 msgr2=0x7f84dc0901f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:31.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84dc0afa10 0x7f84dc0901f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:31.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 -- 172.21.15.61:0/2697327380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84cc00b1e0 con 0x7f84dc0608b0 2022-07-20T03:11:31.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84dbfff700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc09de50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f84cc002f70 tx=0x7f84cc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:31.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84cc00d940 con 0x7f84dc0608b0 2022-07-20T03:11:31.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84cc011b80 con 0x7f84dc0608b0 2022-07-20T03:11:31.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.887+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84dc152230 con 0x7f84dc0608b0 2022-07-20T03:11:31.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84cc01e460 con 0x7f84dc0608b0 2022-07-20T03:11:31.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84dc0906e0 con 0x7f84dc0608b0 2022-07-20T03:11:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f84cc025030 con 0x7f84dc0608b0 2022-07-20T03:11:31.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84dc05c280 con 0x7f84dc0608b0 2022-07-20T03:11:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84d97fa700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84c0053a40 0x7f84c0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f84cc06cfd0 con 0x7f84dc0608b0 2022-07-20T03:11:31.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84db7fe700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84c0053a40 0x7f84c0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:31.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.891+0000 7f84db7fe700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84c0053a40 0x7f84c0055f00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f84c400c000 tx=0x7f84c4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:31.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:31.895+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84cc043f30 con 0x7f84dc0608b0 2022-07-20T03:11:32.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f84dc091360 con 0x7f84dc0608b0 2022-07-20T03:11:32.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84d97fa700 1 -- 172.21.15.61:0/2697327380 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f84cc023090 con 0x7f84dc0608b0 2022-07-20T03:11:32.090 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:32.091 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:32.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84c0053a40 msgr2=0x7f84c0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84c0053a40 0x7f84c0055f00 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f84c400c000 tx=0x7f84c4008040).stop 2022-07-20T03:11:32.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 msgr2=0x7f84dc09de50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc09de50 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7f84cc002f70 tx=0x7f84cc015000).stop 2022-07-20T03:11:32.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 shutdown_connections 2022-07-20T03:11:32.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84c0053a40 0x7f84c0055f00 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84dc0afa10 0x7f84dc0901f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84dc0b3de0 0x7f84dc1526d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 --2- 172.21.15.61:0/2697327380 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84dc0608b0 0x7f84dc09de50 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 >> 172.21.15.61:0/2697327380 conn(0x7f84dc0b5ee0 msgr2=0x7f84dc0b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:32.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 shutdown_connections 2022-07-20T03:11:32.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.063+0000 7f84e244a700 1 -- 172.21.15.61:0/2697327380 wait complete. 2022-07-20T03:11:32.304 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:32.305 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-20T03:11:32.421 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T03:11:32.422 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T03:11:32.423 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:32.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 Processor -- start 2022-07-20T03:11:32.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 -- start start 2022-07-20T03:11:32.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f23580b7b70 0x7f23580b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23580b31e0 0x7f235813d810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2358066160 0x7f235813dd00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23580b7f50 con 0x7f23580b31e0 2022-07-20T03:11:32.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f23580b2800 con 0x7f23580b7b70 2022-07-20T03:11:32.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235df53700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23580b2aa0 con 0x7f2358066160 2022-07-20T03:11:32.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23580b31e0 0x7f235813d810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f23580b31e0 0x7f235813d810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45394/0 (socket says 172.21.15.61:45394) 2022-07-20T03:11:32.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 -- 172.21.15.61:0/731963043 learned_addr learned my addr 172.21.15.61:0/731963043 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:32.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235d752700 1 --2- 172.21.15.61:0/731963043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2358066160 0x7f235813dd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f235cf51700 1 --2- 172.21.15.61:0/731963043 >> v2:172.21.15.117:3300/0 conn(0x7f23580b7b70 0x7f23580b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 -- 172.21.15.61:0/731963043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2358066160 msgr2=0x7f235813dd00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:32.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 --2- 172.21.15.61:0/731963043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2358066160 0x7f235813dd00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 -- 172.21.15.61:0/731963043 >> v2:172.21.15.117:3300/0 conn(0x7f23580b7b70 msgr2=0x7f23580b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:32.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 --2- 172.21.15.61:0/731963043 >> v2:172.21.15.117:3300/0 conn(0x7f23580b7b70 0x7f23580b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 -- 172.21.15.61:0/731963043 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f23580b2d40 con 0x7f23580b31e0 2022-07-20T03:11:32.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f2357fff700 1 --2- 172.21.15.61:0/731963043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23580b31e0 0x7f235813d810 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f234c002f70 tx=0x7f234c006eb0).ready entity=mon.0 client_cookie=cb6810158c5307d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:32.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f23577fe700 1 -- 172.21.15.61:0/731963043 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f234c00d570 con 0x7f23580b31e0 2022-07-20T03:11:32.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f23577fe700 1 -- 172.21.15.61:0/731963043 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f234c000e90 con 0x7f23580b31e0 2022-07-20T03:11:32.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.531+0000 7f23577fe700 1 -- 172.21.15.61:0/731963043 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f234c00d900 con 0x7f23580b31e0 2022-07-20T03:11:32.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- 172.21.15.61:0/731963043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23580b31e0 msgr2=0x7f235813d810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- 172.21.15.61:0/731963043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23580b31e0 0x7f235813d810 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f234c002f70 tx=0x7f234c006eb0).stop 2022-07-20T03:11:32.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- 172.21.15.61:0/731963043 shutdown_connections 2022-07-20T03:11:32.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- 172.21.15.61:0/731963043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2358066160 0x7f235813dd00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- 172.21.15.61:0/731963043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f23580b31e0 0x7f235813d810 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- 172.21.15.61:0/731963043 >> v2:172.21.15.117:3300/0 conn(0x7f23580b7b70 0x7f23580b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- 172.21.15.61:0/731963043 >> 172.21.15.61:0/731963043 conn(0x7f23580656b0 msgr2=0x7f23580b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:32.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- 172.21.15.61:0/731963043 shutdown_connections 2022-07-20T03:11:32.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- 172.21.15.61:0/731963043 wait complete. 2022-07-20T03:11:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 Processor -- start 2022-07-20T03:11:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- start start 2022-07-20T03:11:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 0x7f2358143900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23580b31e0 0x7f2358141fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23580b7b70 0x7f23580ad5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2358143e90 con 0x7f2358066160 2022-07-20T03:11:32.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f23580ace90 con 0x7f23580b31e0 2022-07-20T03:11:32.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235df53700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f23580ad130 con 0x7f23580b7b70 2022-07-20T03:11:32.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.535+0000 7f235cf51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 0x7f2358143900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 0x7f2358143900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45398/0 (socket says 172.21.15.61:45398) 2022-07-20T03:11:32.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 -- 172.21.15.61:0/2545340729 learned_addr learned my addr 172.21.15.61:0/2545340729 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:32.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f2357fff700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23580b31e0 0x7f2358141fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235d752700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23580b7b70 0x7f23580ad5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 -- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23580b7b70 msgr2=0x7f23580ad5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23580b7b70 0x7f23580ad5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 -- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23580b31e0 msgr2=0x7f2358141fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23580b31e0 0x7f2358141fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 -- 172.21.15.61:0/2545340729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f234c00b1e0 con 0x7f2358066160 2022-07-20T03:11:32.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235cf51700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 0x7f2358143900 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f234800c000 tx=0x7f2348006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2348011750 con 0x7f2358066160 2022-07-20T03:11:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f23480118b0 con 0x7f2358066160 2022-07-20T03:11:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2358140940 con 0x7f2358066160 2022-07-20T03:11:32.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2348011c10 con 0x7f2358066160 2022-07-20T03:11:32.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2358142a70 con 0x7f2358066160 2022-07-20T03:11:32.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f23580574e0 con 0x7f2358066160 2022-07-20T03:11:32.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.539+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2348011d70 con 0x7f2358066160 2022-07-20T03:11:32.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.543+0000 7f2355ffb700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2338053a40 0x7f2338055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:32.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.543+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f234806c2e0 con 0x7f2358066160 2022-07-20T03:11:32.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.543+0000 7f2357fff700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2338053a40 0x7f2338055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:32.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.543+0000 7f2357fff700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2338053a40 0x7f2338055f00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f234c01c000 tx=0x7f234c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:32.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.547+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f23480431e0 con 0x7f2358066160 2022-07-20T03:11:32.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2358068630 con 0x7f2358066160 2022-07-20T03:11:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f2355ffb700 1 -- 172.21.15.61:0/2545340729 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f2348046800 con 0x7f2358066160 2022-07-20T03:11:32.810 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:32.811 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:32.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2338053a40 msgr2=0x7f2338055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2338053a40 0x7f2338055f00 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f234c01c000 tx=0x7f234c015000).stop 2022-07-20T03:11:32.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 msgr2=0x7f2358143900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:32.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 0x7f2358143900 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f234800c000 tx=0x7f2348006df0).stop 2022-07-20T03:11:32.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 shutdown_connections 2022-07-20T03:11:32.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2338053a40 0x7f2338055f00 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f23580b31e0 0x7f2358141fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f23580b7b70 0x7f23580ad5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 --2- 172.21.15.61:0/2545340729 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2358066160 0x7f2358143900 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:32.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 >> 172.21.15.61:0/2545340729 conn(0x7f23580656b0 msgr2=0x7f23580604f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:32.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 shutdown_connections 2022-07-20T03:11:32.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:32.715+0000 7f235df53700 1 -- 172.21.15.61:0/2545340729 wait complete. 2022-07-20T03:11:32.826 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:32.834 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T03:11:32.984 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T03:11:32.985 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T03:11:32.986 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:11:33.050 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:33.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.159+0000 7f3974570700 1 Processor -- start 2022-07-20T03:11:33.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.159+0000 7f3974570700 1 -- start start 2022-07-20T03:11:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3974570700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c14ba40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3974570700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c0aee00 con 0x7f396c0aea20 2022-07-20T03:11:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3974570700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f396c144470 con 0x7f396c0675f0 2022-07-20T03:11:33.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3974570700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c144710 con 0x7f396c149650 2022-07-20T03:11:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f397230c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f396c0aea20 0x7f396c140a90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42004/0 (socket says 172.21.15.61:42004) 2022-07-20T03:11:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f397230c700 1 -- 172.21.15.61:0/1886218614 learned_addr learned my addr 172.21.15.61:0/1886218614 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3972b0d700 1 --2- 172.21.15.61:0/1886218614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c14ba40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3972b0d700 1 -- 172.21.15.61:0/1886218614 >> v1:172.21.15.117:6789/0 conn(0x7f396c0675f0 legacy=0x7f396c1411a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3972b0d700 1 -- 172.21.15.61:0/1886218614 >> v1:172.21.15.61:6789/0 conn(0x7f396c0aea20 legacy=0x7f396c140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3972b0d700 1 -- 172.21.15.61:0/1886218614 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f396c1449b0 con 0x7f396c149650 2022-07-20T03:11:33.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.163+0000 7f3972b0d700 1 --2- 172.21.15.61:0/1886218614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c14ba40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f39640032f0 tx=0x7f39640091d0).ready entity=mon.2 client_cookie=b724489a01a69390 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:33.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f397130a700 1 -- 172.21.15.61:0/1886218614 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f396400b8a0 con 0x7f396c149650 2022-07-20T03:11:33.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f397130a700 1 -- 172.21.15.61:0/1886218614 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f396400ba00 con 0x7f396c149650 2022-07-20T03:11:33.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f397130a700 1 -- 172.21.15.61:0/1886218614 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f396400bb70 con 0x7f396c149650 2022-07-20T03:11:33.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 -- 172.21.15.61:0/1886218614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 msgr2=0x7f396c14ba40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 --2- 172.21.15.61:0/1886218614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c14ba40 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f39640032f0 tx=0x7f39640091d0).stop 2022-07-20T03:11:33.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 -- 172.21.15.61:0/1886218614 shutdown_connections 2022-07-20T03:11:33.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 --2- 172.21.15.61:0/1886218614 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c14ba40 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 -- 172.21.15.61:0/1886218614 >> 172.21.15.61:0/1886218614 conn(0x7f396c0611c0 msgr2=0x7f396c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:33.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 -- 172.21.15.61:0/1886218614 shutdown_connections 2022-07-20T03:11:33.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 -- 172.21.15.61:0/1886218614 wait complete. 2022-07-20T03:11:33.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 Processor -- start 2022-07-20T03:11:33.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.167+0000 7f3974570700 1 -- start start 2022-07-20T03:11:33.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3974570700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 0x7f396c139780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:33.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3974570700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c154700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:33.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3974570700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f396c132190 0x7f396c064cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:33.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3974570700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c153d20 con 0x7f396c0aea20 2022-07-20T03:11:33.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3974570700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f396c153fc0 con 0x7f396c132190 2022-07-20T03:11:33.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3974570700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f396c154260 con 0x7f396c149650 2022-07-20T03:11:33.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 0x7f396c139780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:33.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 0x7f396c139780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45414/0 (socket says 172.21.15.61:45414) 2022-07-20T03:11:33.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3971b0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c154700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:33.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 -- 172.21.15.61:0/1293142608 learned_addr learned my addr 172.21.15.61:0/1293142608 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:33.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f3972b0d700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f396c132190 0x7f396c064cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:33.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 -- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 msgr2=0x7f396c154700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c154700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 -- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f396c132190 msgr2=0x7f396c064cb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f396c132190 0x7f396c064cb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 -- 172.21.15.61:0/1293142608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3964009470 con 0x7f396c0aea20 2022-07-20T03:11:33.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.171+0000 7f397230c700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 0x7f396c139780 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f396c141c50 tx=0x7f3968007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:33.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39680070d0 con 0x7f396c0aea20 2022-07-20T03:11:33.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3968007230 con 0x7f396c0aea20 2022-07-20T03:11:33.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f396c132b70 con 0x7f396c0aea20 2022-07-20T03:11:33.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3968005610 con 0x7f396c0aea20 2022-07-20T03:11:33.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f396c132860 con 0x7f396c0aea20 2022-07-20T03:11:33.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f396c154420 con 0x7f396c0aea20 2022-07-20T03:11:33.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3968010070 con 0x7f396c0aea20 2022-07-20T03:11:33.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f395b7fe700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f394c0539b0 0x7f394c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f396806b230 con 0x7f396c0aea20 2022-07-20T03:11:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.175+0000 7f3971b0b700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f394c0539b0 0x7f394c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:33.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.179+0000 7f3971b0b700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f394c0539b0 0x7f394c055e70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f395c00d000 tx=0x7f395c000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:33.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.183+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3968042080 con 0x7f396c0aea20 2022-07-20T03:11:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.343+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f396c132860 con 0x7f396c0aea20 2022-07-20T03:11:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f395b7fe700 1 -- 172.21.15.61:0/1293142608 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f396c132860 con 0x7f396c0aea20 2022-07-20T03:11:33.538 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:33.558 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:33.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f394c0539b0 msgr2=0x7f394c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f394c0539b0 0x7f394c055e70 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f395c00d000 tx=0x7f395c000bc0).stop 2022-07-20T03:11:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 msgr2=0x7f396c139780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:33.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 0x7f396c139780 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f396c141c50 tx=0x7f3968007c00).stop 2022-07-20T03:11:33.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 shutdown_connections 2022-07-20T03:11:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f394c0539b0 0x7f394c055e70 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f396c132190 0x7f396c064cb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f396c149650 0x7f396c154700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 --2- 172.21.15.61:0/1293142608 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f396c0aea20 0x7f396c139780 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:33.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 >> 172.21.15.61:0/1293142608 conn(0x7f396c0675f0 msgr2=0x7f396c1337c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:33.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 shutdown_connections 2022-07-20T03:11:33.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.347+0000 7f3974570700 1 -- 172.21.15.61:0/1293142608 wait complete. 2022-07-20T03:11:33.654 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:33.655 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 5 2022-07-20T03:11:34.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.775+0000 7f0d70a2d700 1 Processor -- start 2022-07-20T03:11:34.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.775+0000 7f0d70a2d700 1 -- start start 2022-07-20T03:11:34.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.775+0000 7f0d70a2d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0d6c0b59f0 0x7f0d6c1407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.775+0000 7f0d70a2d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0b5dd0 con 0x7f0d6c140cd0 2022-07-20T03:11:34.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.775+0000 7f0d70a2d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d6c0652a0 con 0x7f0d6c0aef30 2022-07-20T03:11:34.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.775+0000 7f0d70a2d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d6c0b1750 con 0x7f0d6c0b59f0 2022-07-20T03:11:34.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0d6c140cd0 0x7f0d6c1445a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42020/0 (socket says 172.21.15.61:42020) 2022-07-20T03:11:34.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6affd700 1 -- 172.21.15.61:0/1408556662 learned_addr learned my addr 172.21.15.61:0/1408556662 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:34.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/1408556662 >> v2:172.21.15.117:3300/0 conn(0x7f0d6c0b59f0 0x7f0d6c1407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2531571586 0 0) 0x7f0d6c0b5dd0 con 0x7f0d6c140cd0 2022-07-20T03:11:34.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d4c003260 con 0x7f0d6c140cd0 2022-07-20T03:11:34.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4155866349 0 0) 0x7f0d6c0652a0 con 0x7f0d6c0aef30 2022-07-20T03:11:34.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d6c0b5dd0 con 0x7f0d6c0aef30 2022-07-20T03:11:34.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1408556662 >> v1:172.21.15.117:6789/0 conn(0x7f0d6c0aef30 legacy=0x7f0d6c129340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 329661477 0 0) 0x7f0d4c003260 con 0x7f0d6c140cd0 2022-07-20T03:11:34.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1408556662 >> v1:172.21.15.61:6789/0 conn(0x7f0d6c140cd0 legacy=0x7f0d6c1445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1408556662 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d6c143e60 con 0x7f0d6c0b59f0 2022-07-20T03:11:34.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/1408556662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c1407e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0d5c002f70 tx=0x7f0d5c006df0).ready entity=mon.1 client_cookie=eab04ac10d3a21a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c00d540 con 0x7f0d6c0b59f0 2022-07-20T03:11:34.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c000e90 con 0x7f0d6c0b59f0 2022-07-20T03:11:34.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d6a7fc700 1 -- 172.21.15.61:0/1408556662 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c00d890 con 0x7f0d6c0b59f0 2022-07-20T03:11:34.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1408556662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 msgr2=0x7f0d6c1407e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1408556662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c1407e0 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f0d5c002f70 tx=0x7f0d5c006df0).stop 2022-07-20T03:11:34.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1408556662 shutdown_connections 2022-07-20T03:11:34.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1408556662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c1407e0 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1408556662 >> 172.21.15.61:0/1408556662 conn(0x7f0d6c063480 msgr2=0x7f0d6c0b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:34.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1408556662 shutdown_connections 2022-07-20T03:11:34.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.779+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1408556662 wait complete. 2022-07-20T03:11:34.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 Processor -- start 2022-07-20T03:11:34.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 -- start start 2022-07-20T03:11:34.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 0x7f0d6c12e840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c12ed30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6c140cd0 0x7f0d6c14fff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c130f10 con 0x7f0d6c0aef30 2022-07-20T03:11:34.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c1311b0 con 0x7f0d6c0b59f0 2022-07-20T03:11:34.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.783+0000 7f0d70a2d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d6c12cdd0 con 0x7f0d6c140cd0 2022-07-20T03:11:34.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 0x7f0d6c12e840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 0x7f0d6c12e840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45426/0 (socket says 172.21.15.61:45426) 2022-07-20T03:11:34.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1904471360 learned_addr learned my addr 172.21.15.61:0/1904471360 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:34.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6affd700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c12ed30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6bfff700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6c140cd0 0x7f0d6c14fff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6c140cd0 msgr2=0x7f0d6c14fff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6c140cd0 0x7f0d6c14fff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 msgr2=0x7f0d6c12ed30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c12ed30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/1904471360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d5c00b180 con 0x7f0d6c0aef30 2022-07-20T03:11:34.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6affd700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c12ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:34.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 0x7f0d6c12e840 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f0d5c002f70 tx=0x7f0d5c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d68ff9700 1 -- 172.21.15.61:0/1904471360 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c016070 con 0x7f0d6c0aef30 2022-07-20T03:11:34.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d68ff9700 1 -- 172.21.15.61:0/1904471360 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c013850 con 0x7f0d6c0aef30 2022-07-20T03:11:34.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6c12ca40 con 0x7f0d6c0aef30 2022-07-20T03:11:34.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d68ff9700 1 -- 172.21.15.61:0/1904471360 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c013bb0 con 0x7f0d6c0aef30 2022-07-20T03:11:34.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.787+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c12dd50 con 0x7f0d6c0aef30 2022-07-20T03:11:34.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d68ff9700 1 -- 172.21.15.61:0/1904471360 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0d5c025030 con 0x7f0d6c0aef30 2022-07-20T03:11:34.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d68ff9700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d500539b0 0x7f0d50055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d68ff9700 1 -- 172.21.15.61:0/1904471360 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d5c06cc80 con 0x7f0d6c0aef30 2022-07-20T03:11:34.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d6affd700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d500539b0 0x7f0d50055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d6affd700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d500539b0 0x7f0d50055e70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0d5400d000 tx=0x7f0d540079c0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d7bef3740 1 Processor -- start 2022-07-20T03:11:34.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.791+0000 7f0d7bef3740 1 -- start start 2022-07-20T03:11:34.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 0x2626420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f0350 0x25f1e80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f2370 0x25eba90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x255d210 con 0x262b110 2022-07-20T03:11:34.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x253d450 con 0x25f0350 2022-07-20T03:11:34.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x25f0dd0 con 0x25f2370 2022-07-20T03:11:34.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 0x2626420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 0x2626420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45434/0 (socket says 172.21.15.61:45434) 2022-07-20T03:11:34.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3307050598 learned_addr learned my addr 172.21.15.61:0/3307050598 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:34.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f0350 0x25f1e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6bfff700 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f2370 0x25eba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f2370 msgr2=0x25eba90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f2370 0x25eba90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f0350 msgr2=0x25f1e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f0350 0x25f1e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3307050598 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x2594ad0 con 0x262b110 2022-07-20T03:11:34.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 0x2626420 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f0d5c06ce90 tx=0x7f0d5c010000).ready entity=mon.0 client_cookie=9bcbfd8eebb9e3b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d70a2d700 1 -- 172.21.15.61:0/3307050598 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c043fb0 con 0x262b110 2022-07-20T03:11:34.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d70a2d700 1 -- 172.21.15.61:0/3307050598 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c044110 con 0x262b110 2022-07-20T03:11:34.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d70a2d700 1 -- 172.21.15.61:0/3307050598 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c046f80 con 0x262b110 2022-07-20T03:11:34.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3307050598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 msgr2=0x2626420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 0x2626420 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7f0d5c06ce90 tx=0x7f0d5c010000).stop 2022-07-20T03:11:34.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3307050598 shutdown_connections 2022-07-20T03:11:34.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f0350 0x25f1e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f2370 0x25eba90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3307050598 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x262b110 0x2626420 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3307050598 >> 172.21.15.61:0/3307050598 conn(0x25748f0 msgr2=0x262a850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:34.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3307050598 shutdown_connections 2022-07-20T03:11:34.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3307050598 wait complete. 2022-07-20T03:11:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 -- start start 2022-07-20T03:11:34.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.795+0000 7f0d7bef3740 1 Processor -- start 2022-07-20T03:11:34.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x2635550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x25f41a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x2638b20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x26267a0 con 0x25f0350 2022-07-20T03:11:34.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x2635bf0 con 0x25f2370 2022-07-20T03:11:34.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x2635e90 con 0x262b110 2022-07-20T03:11:34.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 10 client.0 ll_register_callbacks cb 0x25a40b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:11:34.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x2635550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x2635550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45440/0 (socket says 172.21.15.61:45440) 2022-07-20T03:11:34.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/2803343981 learned_addr learned my addr 172.21.15.61:0/2803343981 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:34.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6affd700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x25f41a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6bfff700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x2638b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 msgr2=0x2638b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x2638b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 msgr2=0x25f41a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x25f41a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/2803343981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d6c12dd50 con 0x25f0350 2022-07-20T03:11:34.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6affd700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x25f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:34.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x2635550 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f0d5c06ce90 tx=0x7f0d5c01f000).ready entity=mon.0 client_cookie=35201355c52b52a7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-20T03:11:34.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 1 -- 172.21.15.61:0/2803343981 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c006040 con 0x25f0350 2022-07-20T03:11:34.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 1 -- 172.21.15.61:0/2803343981 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d5c03ac50 con 0x25f0350 2022-07-20T03:11:34.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x25531c0 con 0x25f0350 2022-07-20T03:11:34.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 1 -- 172.21.15.61:0/2803343981 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d5c03ae10 con 0x25f0350 2022-07-20T03:11:34.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 10 client.5817 fetch_fsmap learned FSMap version 29 2022-07-20T03:11:34.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x254c7b0 con 0x25f0350 2022-07-20T03:11:34.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 1 -- 172.21.15.61:0/2803343981 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x25531c0 con 0x25f0350 2022-07-20T03:11:34.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 1 -- 172.21.15.61:0/2803343981 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 29) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f0d5c06c3b0 con 0x25f0350 2022-07-20T03:11:34.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 10 client.5817 fetch_fsmap finished waiting for FSMap version 29 2022-07-20T03:11:34.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 10 client.5817 resolve_mds: resolved ID 'a' to GID 5610 2022-07-20T03:11:34.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 20 client.5817 populate_metadata read hostname 'smithi061' 2022-07-20T03:11:34.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x25f8810 0x2639010 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 4 client.5817 mds_command: new command op to 5610 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T03:11:34.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 --> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x254b240 con 0x25f8810 2022-07-20T03:11:34.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6affd700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x25f8810 0x2639010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d4a7fc700 10 client.5817 ms_handle_connect on v2:172.21.15.61:6832/2246781672 2022-07-20T03:11:34.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.799+0000 7f0d6affd700 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x25f8810 0x2639010 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=1be4b2669766a50a server_cookie=841b88ccc6f883aa in_seq=0 out_seq=0 2022-07-20T03:11:34.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d4a7fc700 1 -- 172.21.15.61:0/2803343981 <== mds.0 v2:172.21.15.61:6832/2246781672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0x254b240 con 0x25f8810 2022-07-20T03:11:34.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d4a7fc700 10 client.5817 handle_command_reply: tid=0 2022-07-20T03:11:34.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 client.5817 shutdown 2022-07-20T03:11:34.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 msgr2=0x2635550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x2635550 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7f0d5c06ce90 tx=0x7f0d5c01f000).stop 2022-07-20T03:11:34.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 shutdown_connections 2022-07-20T03:11:34.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x25f8810 0x2639010 crc :-1 s=READY pgs=11 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x25f41a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x2638b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/2803343981 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x2635550 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 >> 172.21.15.61:0/2803343981 conn(0x25748f0 msgr2=0x25ee150 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:34.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d4a7fc700 0 client.5817 ms_handle_reset on v2:172.21.15.61:6832/2246781672 2022-07-20T03:11:34.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 shutdown_connections 2022-07-20T03:11:34.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 1 -- 172.21.15.61:0/2803343981 wait complete. 2022-07-20T03:11:34.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.803+0000 7f0d7bef3740 20 client.5817 trim_cache size 0 max 16384 2022-07-20T03:11:34.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 Processor -- start 2022-07-20T03:11:34.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 -- start start 2022-07-20T03:11:34.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f0350 0x2622960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 0x25f3760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x262b110 0x25f3c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x2635c50 con 0x25f2370 2022-07-20T03:11:34.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x256ef10 con 0x262b110 2022-07-20T03:11:34.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x255e4f0 con 0x25f0350 2022-07-20T03:11:34.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 0x25f3760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 0x25f3760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45450/0 (socket says 172.21.15.61:45450) 2022-07-20T03:11:34.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 -- 172.21.15.61:0/3597794386 learned_addr learned my addr 172.21.15.61:0/3597794386 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:34.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6bfff700 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x262b110 0x25f3c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f0350 0x2622960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 -- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f0350 msgr2=0x2622960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f0350 0x2622960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 -- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x262b110 msgr2=0x25f3c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x262b110 0x25f3c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 -- 172.21.15.61:0/3597794386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x25553a0 con 0x25f2370 2022-07-20T03:11:34.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.819+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 0x25f3760 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x25ed220 tx=0x7f0d54019040).ready entity=mon.0 client_cookie=5120094558029867 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d60ff9700 1 -- 172.21.15.61:0/3597794386 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d54017690 con 0x25f2370 2022-07-20T03:11:34.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d60ff9700 1 -- 172.21.15.61:0/3597794386 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d54029070 con 0x25f2370 2022-07-20T03:11:34.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d60ff9700 1 -- 172.21.15.61:0/3597794386 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d540179e0 con 0x25f2370 2022-07-20T03:11:34.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3597794386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 msgr2=0x25f3760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:34.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 0x25f3760 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x25ed220 tx=0x7f0d54019040).stop 2022-07-20T03:11:34.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3597794386 shutdown_connections 2022-07-20T03:11:34.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x262b110 0x25f3c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f2370 0x25f3760 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3597794386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x25f0350 0x2622960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3597794386 >> 172.21.15.61:0/3597794386 conn(0x25748f0 msgr2=0x262a760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:34.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3597794386 shutdown_connections 2022-07-20T03:11:34.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3597794386 wait complete. 2022-07-20T03:11:34.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- start start 2022-07-20T03:11:34.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 Processor -- start 2022-07-20T03:11:34.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x262da40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x262bd60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x262c250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x262c740 con 0x25f0350 2022-07-20T03:11:34.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x2623f00 con 0x25f2370 2022-07-20T03:11:34.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x26241a0 con 0x262b110 2022-07-20T03:11:34.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d7bef3740 10 client.0 ll_register_callbacks cb 0x7f0d5c020030 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:11:34.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x262da40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x262bd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x262da40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45456/0 (socket says 172.21.15.61:45456) 2022-07-20T03:11:34.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3584061137 learned_addr learned my addr 172.21.15.61:0/3584061137 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:34.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x262bd60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55134/0 (socket says 172.21.15.61:55134) 2022-07-20T03:11:34.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6bfff700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x262c250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 msgr2=0x262c250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x262c250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 msgr2=0x262bd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:34.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x262bd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:34.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 -- 172.21.15.61:0/3584061137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d54004db0 con 0x25f0350 2022-07-20T03:11:34.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x262bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:11:34.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d6b7fe700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x262da40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f0d5c029000 tx=0x7f0d5c037000).ready entity=mon.0 client_cookie=55330e7ec12e8bb4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:34.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d4b7fe700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-20T03:11:34.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.823+0000 7f0d4b7fe700 1 -- 172.21.15.61:0/3584061137 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5c006980 con 0x25f0350 2022-07-20T03:11:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d4b7fe700 1 -- 172.21.15.61:0/3584061137 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d5c06c8e0 con 0x25f0350 2022-07-20T03:11:34.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7f0d5c044110 con 0x25f0350 2022-07-20T03:11:34.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d4b7fe700 1 -- 172.21.15.61:0/3584061137 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=29) v2 ==== 24+0+0 (secure 0 0 0) 0x7f0d5c0473e0 con 0x25f0350 2022-07-20T03:11:34.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 10 client.5823 fetch_fsmap learned FSMap version 29 2022-07-20T03:11:34.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=29,monmap=2+}) v3 -- 0x2551d90 con 0x25f0350 2022-07-20T03:11:34.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d4b7fe700 1 -- 172.21.15.61:0/3584061137 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5c044110 con 0x25f0350 2022-07-20T03:11:34.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d4b7fe700 1 -- 172.21.15.61:0/3584061137 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 29) v1 ==== 1247+0+0 (secure 0 0 0) 0x7f0d5c06c3b0 con 0x25f0350 2022-07-20T03:11:34.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 10 client.5823 fetch_fsmap finished waiting for FSMap version 29 2022-07-20T03:11:34.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 10 client.5823 resolve_mds: resolved ID 'a' to GID 5610 2022-07-20T03:11:34.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 20 client.5823 populate_metadata read hostname 'smithi061' 2022-07-20T03:11:34.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x2623560 0x2638a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:34.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 4 client.5823 mds_command: new command op to 5610 tid=0[{"prefix": "cache drop", "timeout": 5}] 2022-07-20T03:11:34.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 --> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] -- command(tid 0: {"prefix": "cache drop", "timeout": 5}) v1 -- 0x24405e0 con 0x2623560 2022-07-20T03:11:34.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x2623560 0x2638a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:34.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d6affd700 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x2623560 0x2638a30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=45e7d0d1d4e12e4 server_cookie=2bc4f0adee174c7e in_seq=0 out_seq=0 2022-07-20T03:11:34.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:33.827+0000 7f0d4b7fe700 10 client.5823 ms_handle_connect on v2:172.21.15.61:6832/2246781672 2022-07-20T03:11:35.518 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:35.536 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:35.541 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:35.551 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:35.558 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:11:39.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d4b7fe700 1 -- 172.21.15.61:0/3584061137 <== mds.0 v2:172.21.15.61:6832/2246781672 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+404 (crc 0 0 0) 0x24405e0 con 0x2623560 2022-07-20T03:11:39.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d4b7fe700 10 client.5823 handle_command_reply: tid=0 2022-07-20T03:11:39.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 client.5823 shutdown 2022-07-20T03:11:39.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 msgr2=0x262da40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x262da40 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f0d5c029000 tx=0x7f0d5c037000).stop 2022-07-20T03:11:39.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 shutdown_connections 2022-07-20T03:11:39.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:6832/2246781672,v1:172.21.15.61:6833/2246781672] conn(0x2623560 0x2638a30 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x25f2370 0x262bd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x262b110 0x262c250 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 --2- 172.21.15.61:0/3584061137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x25f0350 0x262da40 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d4b7fe700 0 client.5823 ms_handle_reset on v2:172.21.15.61:6832/2246781672 2022-07-20T03:11:39.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.019+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 >> 172.21.15.61:0/3584061137 conn(0x25748f0 msgr2=0x2622960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:39.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 shutdown_connections 2022-07-20T03:11:39.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d7bef3740 1 -- 172.21.15.61:0/3584061137 wait complete. 2022-07-20T03:11:39.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d7bef3740 20 client.5823 trim_cache size 0 max 16384 2022-07-20T03:11:39.133 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:11:39.133 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-20T03:11:39.133 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": -110, 2022-07-20T03:11:39.133 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(110) Connection timed out", 2022-07-20T03:11:39.154 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 120 2022-07-20T03:11:39.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:39.155 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-20T03:11:39.155 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-20T03:11:39.156 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-20T03:11:39.156 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:39.171 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-20T03:11:39.171 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 0 2022-07-20T03:11:39.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:39.171 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-20T03:11:39.172 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-20T03:11:39.172 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 20001, 2022-07-20T03:11:39.172 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 3100384 2022-07-20T03:11:39.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:11:39.173 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:11:39.173 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 5.1935359209999996 2022-07-20T03:11:39.174 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:11:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d500539b0 msgr2=0x7f0d50055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d500539b0 0x7f0d50055e70 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f0d5400d000 tx=0x7f0d540079c0).stop 2022-07-20T03:11:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 msgr2=0x7f0d6c12e840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 0x7f0d6c12e840 secure :-1 s=READY pgs=528 cs=0 l=1 rev1=1 rx=0x7f0d5c002f70 tx=0x7f0d5c014000).stop 2022-07-20T03:11:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 shutdown_connections 2022-07-20T03:11:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d500539b0 0x7f0d50055e70 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d6c0b59f0 0x7f0d6c12ed30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6c140cd0 0x7f0d6c14fff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 --2- 172.21.15.61:0/1904471360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6c0aef30 0x7f0d6c12e840 unknown :-1 s=CLOSED pgs=528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 >> 172.21.15.61:0/1904471360 conn(0x7f0d6c063480 msgr2=0x7f0d6c0643b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 shutdown_connections 2022-07-20T03:11:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.023+0000 7f0d70a2d700 1 -- 172.21.15.61:0/1904471360 wait complete. 2022-07-20T03:11:39.231 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-20T03:11:39.237 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:39.238 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:39.238 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:39.249 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:11:39.251 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T03:11:39.252 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T03:11:39.253 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:11:39.253 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:11:39.271 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:39.293 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:39.294 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:39.303 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:39.352 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:11:39.353 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:11:39.354 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:11:39.355 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:39.373 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:11:39.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.483+0000 7f688b53b700 1 Processor -- start 2022-07-20T03:11:39.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.483+0000 7f688b53b700 1 -- start start 2022-07-20T03:11:39.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68840b61d0 0x7f6884143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f68841434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f68841439e0 0x7f688413ec50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68840add80 con 0x7f68840b61d0 2022-07-20T03:11:39.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68840afac0 con 0x7f68841439e0 2022-07-20T03:11:39.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68840b65b0 con 0x7f68840b18d0 2022-07-20T03:11:39.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68840b61d0 0x7f6884143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68840b61d0 0x7f6884143000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45470/0 (socket says 172.21.15.61:45470) 2022-07-20T03:11:39.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688ad3a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f68841439e0 0x7f688413ec50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f6889d38700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f68841434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 -- 172.21.15.61:0/2960882465 learned_addr learned my addr 172.21.15.61:0/2960882465 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:39.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 -- 172.21.15.61:0/2960882465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 msgr2=0x7f68841434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:39.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 --2- 172.21.15.61:0/2960882465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f68841434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 -- 172.21.15.61:0/2960882465 >> v2:172.21.15.117:3300/0 conn(0x7f68841439e0 msgr2=0x7f688413ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:39.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 --2- 172.21.15.61:0/2960882465 >> v2:172.21.15.117:3300/0 conn(0x7f68841439e0 0x7f688413ec50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 -- 172.21.15.61:0/2960882465 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688405c3e0 con 0x7f68840b61d0 2022-07-20T03:11:39.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688a539700 1 --2- 172.21.15.61:0/2960882465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884143000 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f6880002f70 tx=0x7f6880006f40).ready entity=mon.0 client_cookie=da99d9e9eaff2c5c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f6889537700 1 -- 172.21.15.61:0/2960882465 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6880013070 con 0x7f68840b61d0 2022-07-20T03:11:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f6889537700 1 -- 172.21.15.61:0/2960882465 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6880000e90 con 0x7f68840b61d0 2022-07-20T03:11:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f6889537700 1 -- 172.21.15.61:0/2960882465 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f688000eb90 con 0x7f68840b61d0 2022-07-20T03:11:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- 172.21.15.61:0/2960882465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 msgr2=0x7f6884143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- 172.21.15.61:0/2960882465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884143000 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7f6880002f70 tx=0x7f6880006f40).stop 2022-07-20T03:11:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- 172.21.15.61:0/2960882465 shutdown_connections 2022-07-20T03:11:39.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- 172.21.15.61:0/2960882465 >> v2:172.21.15.117:3300/0 conn(0x7f68841439e0 0x7f688413ec50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- 172.21.15.61:0/2960882465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f68841434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 --2- 172.21.15.61:0/2960882465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884143000 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- 172.21.15.61:0/2960882465 >> 172.21.15.61:0/2960882465 conn(0x7f6884063cf0 msgr2=0x7f68840b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:39.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- 172.21.15.61:0/2960882465 shutdown_connections 2022-07-20T03:11:39.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.487+0000 7f688b53b700 1 -- 172.21.15.61:0/2960882465 wait complete. 2022-07-20T03:11:39.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 Processor -- start 2022-07-20T03:11:39.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 -- start start 2022-07-20T03:11:39.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f6884142300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884064210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68841439e0 0x7f688413b8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f688413bde0 con 0x7f68840b61d0 2022-07-20T03:11:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6884150dc0 con 0x7f68841439e0 2022-07-20T03:11:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688b53b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68840648b0 con 0x7f68840b18d0 2022-07-20T03:11:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884064210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884064210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45476/0 (socket says 172.21.15.61:45476) 2022-07-20T03:11:39.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 -- 172.21.15.61:0/3513907416 learned_addr learned my addr 172.21.15.61:0/3513907416 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:39.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688ad3a700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68841439e0 0x7f688413b8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f688a539700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f6884142300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 -- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 msgr2=0x7f6884142300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f6884142300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 -- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68841439e0 msgr2=0x7f688413b8f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68841439e0 0x7f688413b8f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 -- 172.21.15.61:0/3513907416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f688000c2e0 con 0x7f68840b61d0 2022-07-20T03:11:39.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f6889d38700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884064210 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f687400c000 tx=0x7f6874006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:39.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6874011750 con 0x7f68840b61d0 2022-07-20T03:11:39.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.491+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68740118b0 con 0x7f68840b61d0 2022-07-20T03:11:39.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6884064b50 con 0x7f68840b61d0 2022-07-20T03:11:39.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6874011c10 con 0x7f68840b61d0 2022-07-20T03:11:39.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6884151410 con 0x7f68840b61d0 2022-07-20T03:11:39.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6874011d70 con 0x7f68840b61d0 2022-07-20T03:11:39.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6884150f30 con 0x7f68840b61d0 2022-07-20T03:11:39.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f68737fe700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68640539d0 0x7f6864055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f687406c170 con 0x7f68840b61d0 2022-07-20T03:11:39.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f688a539700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68640539d0 0x7f6864055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.495+0000 7f688a539700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68640539d0 0x7f6864055e90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f688001b000 tx=0x7f6880015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:39.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.503+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6884150f30 con 0x7f68840b61d0 2022-07-20T03:11:39.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.623+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6884150f30 con 0x7f68840b61d0 2022-07-20T03:11:39.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f68737fe700 1 -- 172.21.15.61:0/3513907416 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v52) v1 ==== 74+0+16088 (secure 0 0 0) 0x7f6884150f30 con 0x7f68840b61d0 2022-07-20T03:11:39.690 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:39.690 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":52,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:11:13.680406+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":9,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:11:10.825678+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"51","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":10,"pool_name":"cephfs_data","create_time":"2022-07-20T03:11:11.707209+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"52","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:11:39.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68640539d0 msgr2=0x7f6864055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68640539d0 0x7f6864055e90 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f688001b000 tx=0x7f6880015000).stop 2022-07-20T03:11:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 msgr2=0x7f6884064210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884064210 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7f687400c000 tx=0x7f6874006df0).stop 2022-07-20T03:11:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 shutdown_connections 2022-07-20T03:11:39.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68640539d0 0x7f6864055e90 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68841439e0 0x7f688413b8f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68840b61d0 0x7f6884064210 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 --2- 172.21.15.61:0/3513907416 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68840b18d0 0x7f6884142300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 >> 172.21.15.61:0/3513907416 conn(0x7f6884063cf0 msgr2=0x7f6884140f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:39.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 shutdown_connections 2022-07-20T03:11:39.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.627+0000 7f688b53b700 1 -- 172.21.15.61:0/3513907416 wait complete. 2022-07-20T03:11:39.708 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 9, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:11:10.825678+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '51', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 10, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:11:11.707209+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '52', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:11:39.709 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:11:39.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 Processor -- start 2022-07-20T03:11:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 -- start start 2022-07-20T03:11:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6ce80b59f0 0x7f6ce81473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80af940 0x7f6ce8147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ce80b1400 0x7f6ce814a3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce805cc70 con 0x7f6ce80b1400 2022-07-20T03:11:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ce80b5dd0 con 0x7f6ce80b59f0 2022-07-20T03:11:39.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.827+0000 7f6cec875700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce805c760 con 0x7f6ce80af940 2022-07-20T03:11:39.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ce80b1400 0x7f6ce814a3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6ce80b1400 0x7f6ce814a3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45486/0 (socket says 172.21.15.61:45486) 2022-07-20T03:11:39.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 -- 172.21.15.61:0/911737739 learned_addr learned my addr 172.21.15.61:0/911737739 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce77fe700 1 --2- 172.21.15.61:0/911737739 >> v2:172.21.15.117:3300/0 conn(0x7f6ce80b59f0 0x7f6ce81473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce6ffd700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80af940 0x7f6ce8147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 -- 172.21.15.61:0/911737739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80af940 msgr2=0x7f6ce8147890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80af940 0x7f6ce8147890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 -- 172.21.15.61:0/911737739 >> v2:172.21.15.117:3300/0 conn(0x7f6ce80b59f0 msgr2=0x7f6ce81473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:39.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 --2- 172.21.15.61:0/911737739 >> v2:172.21.15.117:3300/0 conn(0x7f6ce80b59f0 0x7f6ce81473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:39.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 -- 172.21.15.61:0/911737739 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce8147d80 con 0x7f6ce80b1400 2022-07-20T03:11:39.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce6ffd700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80af940 0x7f6ce8147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:39.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce7fff700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce814a3c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f6ce00032f0 tx=0x7f6ce0009290).ready entity=mon.0 client_cookie=b2f7482d456cea6b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:39.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce67fc700 1 -- 172.21.15.61:0/911737739 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ce000b870 con 0x7f6ce80b1400 2022-07-20T03:11:39.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce67fc700 1 -- 172.21.15.61:0/911737739 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ce000b9d0 con 0x7f6ce80b1400 2022-07-20T03:11:39.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6ce67fc700 1 -- 172.21.15.61:0/911737739 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ce000bb40 con 0x7f6ce80b1400 2022-07-20T03:11:39.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 -- 172.21.15.61:0/911737739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 msgr2=0x7f6ce814a3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce814a3c0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f6ce00032f0 tx=0x7f6ce0009290).stop 2022-07-20T03:11:40.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 -- 172.21.15.61:0/911737739 shutdown_connections 2022-07-20T03:11:40.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce814a3c0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 --2- 172.21.15.61:0/911737739 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80af940 0x7f6ce8147890 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 --2- 172.21.15.61:0/911737739 >> v2:172.21.15.117:3300/0 conn(0x7f6ce80b59f0 0x7f6ce81473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 -- 172.21.15.61:0/911737739 >> 172.21.15.61:0/911737739 conn(0x7f6ce8063480 msgr2=0x7f6ce80b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:40.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 -- 172.21.15.61:0/911737739 shutdown_connections 2022-07-20T03:11:40.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 -- 172.21.15.61:0/911737739 wait complete. 2022-07-20T03:11:40.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 Processor -- start 2022-07-20T03:11:40.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.831+0000 7f6cec875700 1 -- start start 2022-07-20T03:11:40.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce80af940 0x7f6ce8065510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce8159880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80b59f0 0x7f6ce814a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce813f1b0 con 0x7f6ce80b1400 2022-07-20T03:11:40.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce81115a0 con 0x7f6ce80af940 2022-07-20T03:11:40.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ce8063a00 con 0x7f6ce80b59f0 2022-07-20T03:11:40.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce8159880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce8159880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45490/0 (socket says 172.21.15.61:45490) 2022-07-20T03:11:40.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 -- 172.21.15.61:0/3831744857 learned_addr learned my addr 172.21.15.61:0/3831744857 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:40.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce77fe700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce80af940 0x7f6ce8065510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce7fff700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80b59f0 0x7f6ce814a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 -- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80b59f0 msgr2=0x7f6ce814a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80b59f0 0x7f6ce814a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 -- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce80af940 msgr2=0x7f6ce8065510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce80af940 0x7f6ce8065510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 -- 172.21.15.61:0/3831744857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ce0009530 con 0x7f6ce80b1400 2022-07-20T03:11:40.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce6ffd700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce8159880 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f6ce8066980 tx=0x7f6cd8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd8011780 con 0x7f6ce80b1400 2022-07-20T03:11:40.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd80118e0 con 0x7f6ce80b1400 2022-07-20T03:11:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ce8159d70 con 0x7f6ce80b1400 2022-07-20T03:11:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd8011c40 con 0x7f6ce80b1400 2022-07-20T03:11:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.835+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ce8064a20 con 0x7f6ce80b1400 2022-07-20T03:11:40.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.839+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ce8064be0 con 0x7f6ce80b1400 2022-07-20T03:11:40.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.839+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6cd8012420 con 0x7f6ce80b1400 2022-07-20T03:11:40.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.839+0000 7f6ce4ff9700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ccc053af0 0x7f6ccc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.839+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6cd806bae0 con 0x7f6ce80b1400 2022-07-20T03:11:40.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.839+0000 7f6ce77fe700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ccc053af0 0x7f6ccc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.839+0000 7f6ce77fe700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ccc053af0 0x7f6ccc055fb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6cd000b000 tx=0x7f6cd0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.843+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6cd8042940 con 0x7f6ce80b1400 2022-07-20T03:11:40.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.967+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6ce8159d70 con 0x7f6ce80b1400 2022-07-20T03:11:40.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.967+0000 7f6ce4ff9700 1 -- 172.21.15.61:0/3831744857 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v52) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6ce8159d70 con 0x7f6ce80b1400 2022-07-20T03:11:40.125 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:11:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ccc053af0 msgr2=0x7f6ccc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ccc053af0 0x7f6ccc055fb0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f6cd000b000 tx=0x7f6cd0006df0).stop 2022-07-20T03:11:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 msgr2=0x7f6ce8159880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce8159880 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7f6ce8066980 tx=0x7f6cd8008040).stop 2022-07-20T03:11:40.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 shutdown_connections 2022-07-20T03:11:40.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ccc053af0 0x7f6ccc055fb0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ce80b59f0 0x7f6ce814a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ce80b1400 0x7f6ce8159880 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 --2- 172.21.15.61:0/3831744857 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ce80af940 0x7f6ce8065510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 >> 172.21.15.61:0/3831744857 conn(0x7f6ce8063480 msgr2=0x7f6ce813fb70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:40.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 shutdown_connections 2022-07-20T03:11:40.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:39.971+0000 7f6cec875700 1 -- 172.21.15.61:0/3831744857 wait complete. 2022-07-20T03:11:40.150 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:11:40.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.267+0000 7f76a7bd7700 1 Processor -- start 2022-07-20T03:11:40.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- start start 2022-07-20T03:11:40.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f76a005fa50 0x7f76a0062160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f76a005ce90 0x7f76a0143220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76a005d2c0 con 0x7f76a00b2550 2022-07-20T03:11:40.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a00b2930 con 0x7f76a005fa50 2022-07-20T03:11:40.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76a0061780 con 0x7f76a005ce90 2022-07-20T03:11:40.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f76a005fa50 0x7f76a0062160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f76a005fa50 0x7f76a0062160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45498/0 (socket says 172.21.15.61:45498) 2022-07-20T03:11:40.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 -- 172.21.15.61:0/3166842423 learned_addr learned my addr 172.21.15.61:0/3166842423 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:40.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5973700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f76a00b2550 0x7f76a0146260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42094/0 (socket says 172.21.15.61:42094) 2022-07-20T03:11:40.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a6174700 1 --2- 172.21.15.61:0/3166842423 >> v2:172.21.15.117:3300/0 conn(0x7f76a005ce90 0x7f76a0143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 -- 172.21.15.61:0/3166842423 >> v2:172.21.15.117:3300/0 conn(0x7f76a005ce90 msgr2=0x7f76a0143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:40.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 --2- 172.21.15.61:0/3166842423 >> v2:172.21.15.117:3300/0 conn(0x7f76a005ce90 0x7f76a0143220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 -- 172.21.15.61:0/3166842423 >> v1:172.21.15.61:6789/0 conn(0x7f76a00b2550 legacy=0x7f76a0146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 -- 172.21.15.61:0/3166842423 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76a0061a20 con 0x7f76a005fa50 2022-07-20T03:11:40.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a4971700 1 -- 172.21.15.61:0/3166842423 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3852947587 0 0) 0x7f76a005d2c0 con 0x7f76a00b2550 2022-07-20T03:11:40.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a6174700 1 --2- 172.21.15.61:0/3166842423 >> v2:172.21.15.117:3300/0 conn(0x7f76a005ce90 0x7f76a0143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:40.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a5172700 1 --2- 172.21.15.61:0/3166842423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a0062160 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f7690002f70 tx=0x7f7690006eb0).ready entity=mon.0 client_cookie=7c3fe7901df9a8f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a4971700 1 -- 172.21.15.61:0/3166842423 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f769000d610 con 0x7f76a005fa50 2022-07-20T03:11:40.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a4971700 1 -- 172.21.15.61:0/3166842423 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7690000e90 con 0x7f76a005fa50 2022-07-20T03:11:40.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a4971700 1 -- 172.21.15.61:0/3166842423 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f769000d960 con 0x7f76a005fa50 2022-07-20T03:11:40.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3166842423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 msgr2=0x7f76a0062160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3166842423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a0062160 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f7690002f70 tx=0x7f7690006eb0).stop 2022-07-20T03:11:40.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3166842423 shutdown_connections 2022-07-20T03:11:40.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3166842423 >> v2:172.21.15.117:3300/0 conn(0x7f76a005ce90 0x7f76a0143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3166842423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a0062160 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3166842423 >> 172.21.15.61:0/3166842423 conn(0x7f76a00b36c0 msgr2=0x7f76a00b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:40.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3166842423 shutdown_connections 2022-07-20T03:11:40.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.271+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3166842423 wait complete. 2022-07-20T03:11:40.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 Processor -- start 2022-07-20T03:11:40.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 -- start start 2022-07-20T03:11:40.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76a005ce90 0x7f76a009b3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a00af650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76a00b2550 0x7f76a0142d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a009ac70 con 0x7f76a005fa50 2022-07-20T03:11:40.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76a0099640 con 0x7f76a005ce90 2022-07-20T03:11:40.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76a009ae70 con 0x7f76a00b2550 2022-07-20T03:11:40.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a00af650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a00af650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45504/0 (socket says 172.21.15.61:45504) 2022-07-20T03:11:40.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 -- 172.21.15.61:0/3995616795 learned_addr learned my addr 172.21.15.61:0/3995616795 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:40.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a6174700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76a00b2550 0x7f76a0142d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5973700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76a005ce90 0x7f76a009b3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 -- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76a00b2550 msgr2=0x7f76a0142d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76a00b2550 0x7f76a0142d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 -- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76a005ce90 msgr2=0x7f76a009b3b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76a005ce90 0x7f76a009b3b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 -- 172.21.15.61:0/3995616795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f769000b1e0 con 0x7f76a005fa50 2022-07-20T03:11:40.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5973700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76a005ce90 0x7f76a009b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:11:40.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a5172700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a00af650 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f7690002f70 tx=0x7f7690015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f769000d290 con 0x7f76a005fa50 2022-07-20T03:11:40.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f769000de90 con 0x7f76a005fa50 2022-07-20T03:11:40.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76a009b110 con 0x7f76a005fa50 2022-07-20T03:11:40.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7690011e20 con 0x7f76a005fa50 2022-07-20T03:11:40.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.275+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76a0142390 con 0x7f76a005fa50 2022-07-20T03:11:40.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.279+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76a0061cc0 con 0x7f76a005fa50 2022-07-20T03:11:40.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.279+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7690025030 con 0x7f76a005fa50 2022-07-20T03:11:40.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.279+0000 7f768effd700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7680053ad0 0x7f7680055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.279+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f769006d060 con 0x7f76a005fa50 2022-07-20T03:11:40.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.279+0000 7f76a5973700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7680053ad0 0x7f7680055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.279+0000 7f76a5973700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7680053ad0 0x7f7680055f90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f769c00b000 tx=0x7f769c0070e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.287+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7690043fc0 con 0x7f76a005fa50 2022-07-20T03:11:40.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f76a0142070 con 0x7f76a005fa50 2022-07-20T03:11:40.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f768effd700 1 -- 172.21.15.61:0/3995616795 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v52) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7690023070 con 0x7f76a005fa50 2022-07-20T03:11:40.563 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7680053ad0 msgr2=0x7f7680055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7680053ad0 0x7f7680055f90 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f769c00b000 tx=0x7f769c0070e0).stop 2022-07-20T03:11:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 msgr2=0x7f76a00af650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a00af650 secure :-1 s=READY pgs=539 cs=0 l=1 rev1=1 rx=0x7f7690002f70 tx=0x7f7690015000).stop 2022-07-20T03:11:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 shutdown_connections 2022-07-20T03:11:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7680053ad0 0x7f7680055f90 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76a00b2550 0x7f76a0142d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f76a005fa50 0x7f76a00af650 unknown :-1 s=CLOSED pgs=539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 --2- 172.21.15.61:0/3995616795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f76a005ce90 0x7f76a009b3b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 >> 172.21.15.61:0/3995616795 conn(0x7f76a00b36c0 msgr2=0x7f76a00ac970 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:40.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 shutdown_connections 2022-07-20T03:11:40.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.415+0000 7f76a7bd7700 1 -- 172.21.15.61:0/3995616795 wait complete. 2022-07-20T03:11:40.574 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:11:40.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.699+0000 7f67fd9b8700 1 Processor -- start 2022-07-20T03:11:40.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.699+0000 7f67fd9b8700 1 -- start start 2022-07-20T03:11:40.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f67f80b3ed0 0x7f67f8143530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f80b6680 con 0x7f67f8143a20 2022-07-20T03:11:40.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67f81465c0 con 0x7f67f80b62a0 2022-07-20T03:11:40.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67f80b4af0 con 0x7f67f80b3ed0 2022-07-20T03:11:40.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f77fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f67f8143a20 0x7f67f813fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42112/0 (socket says 172.21.15.61:42112) 2022-07-20T03:11:40.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f77fe700 1 -- 172.21.15.61:0/3068552900 learned_addr learned my addr 172.21.15.61:0/3068552900 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:40.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f67fc700 1 --2- 172.21.15.61:0/3068552900 >> v2:172.21.15.117:3300/0 conn(0x7f67f80b3ed0 0x7f67f8143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f5ffb700 1 -- 172.21.15.61:0/3068552900 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3922051584 0 0) 0x7f67f80b6680 con 0x7f67f8143a20 2022-07-20T03:11:40.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f5ffb700 1 -- 172.21.15.61:0/3068552900 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f67d8003260 con 0x7f67f8143a20 2022-07-20T03:11:40.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f5ffb700 1 -- 172.21.15.61:0/3068552900 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1418039660 0 0) 0x7f67d8003260 con 0x7f67f8143a20 2022-07-20T03:11:40.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f67fc700 1 -- 172.21.15.61:0/3068552900 >> v1:172.21.15.117:6789/0 conn(0x7f67f80b62a0 legacy=0x7f67f80b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f67fc700 1 -- 172.21.15.61:0/3068552900 >> v1:172.21.15.61:6789/0 conn(0x7f67f8143a20 legacy=0x7f67f813fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f67fc700 1 -- 172.21.15.61:0/3068552900 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67f8142df0 con 0x7f67f80b3ed0 2022-07-20T03:11:40.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f67fc700 1 --2- 172.21.15.61:0/3068552900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b3ed0 0x7f67f8143530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f67f0001b80 tx=0x7f67e8006df0).ready entity=mon.1 client_cookie=b7a758ff38ed41ee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f5ffb700 1 -- 172.21.15.61:0/3068552900 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67f00019e0 con 0x7f67f80b3ed0 2022-07-20T03:11:40.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f5ffb700 1 -- 172.21.15.61:0/3068552900 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e8000e90 con 0x7f67f80b3ed0 2022-07-20T03:11:40.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67f5ffb700 1 -- 172.21.15.61:0/3068552900 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67e800d4f0 con 0x7f67f80b3ed0 2022-07-20T03:11:40.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- 172.21.15.61:0/3068552900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b3ed0 msgr2=0x7f67f8143530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/3068552900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b3ed0 0x7f67f8143530 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f67f0001b80 tx=0x7f67e8006df0).stop 2022-07-20T03:11:40.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- 172.21.15.61:0/3068552900 shutdown_connections 2022-07-20T03:11:40.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/3068552900 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b3ed0 0x7f67f8143530 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- 172.21.15.61:0/3068552900 >> 172.21.15.61:0/3068552900 conn(0x7f67f8066f00 msgr2=0x7f67f80672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:40.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- 172.21.15.61:0/3068552900 shutdown_connections 2022-07-20T03:11:40.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.703+0000 7f67fd9b8700 1 -- 172.21.15.61:0/3068552900 wait complete. 2022-07-20T03:11:40.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 Processor -- start 2022-07-20T03:11:40.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 -- start start 2022-07-20T03:11:40.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f80b3ed0 0x7f67f80ada30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b62a0 0x7f67f8152b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f8143a20 0x7f67f8155780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f80ae730 con 0x7f67f80b3ed0 2022-07-20T03:11:40.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67f8153070 con 0x7f67f80b62a0 2022-07-20T03:11:40.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67fd9b8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67f80ba320 con 0x7f67f8143a20 2022-07-20T03:11:40.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f80b3ed0 0x7f67f80ada30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f80b3ed0 0x7f67f80ada30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45516/0 (socket says 172.21.15.61:45516) 2022-07-20T03:11:40.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f6ffd700 1 -- 172.21.15.61:0/2980205480 learned_addr learned my addr 172.21.15.61:0/2980205480 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:40.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f77fe700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f8143a20 0x7f67f8155780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b62a0 0x7f67f8152b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 -- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f8143a20 msgr2=0x7f67f8155780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f8143a20 0x7f67f8155780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 -- 172.21.15.61:0/2980205480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f80b3ed0 msgr2=0x7f67f80ada30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f80b3ed0 0x7f67f80ada30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 -- 172.21.15.61:0/2980205480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67e800b180 con 0x7f67f80b62a0 2022-07-20T03:11:40.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.707+0000 7f67f67fc700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b62a0 0x7f67f8152b80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f67f0001b80 tx=0x7f67e8014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67e8016070 con 0x7f67f80b62a0 2022-07-20T03:11:40.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67e8013bd0 con 0x7f67f80b62a0 2022-07-20T03:11:40.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67f80ae960 con 0x7f67f80b62a0 2022-07-20T03:11:40.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67e800d4f0 con 0x7f67f80b62a0 2022-07-20T03:11:40.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67f80ad390 con 0x7f67f80b62a0 2022-07-20T03:11:40.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67f8143090 con 0x7f67f80b62a0 2022-07-20T03:11:40.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f67e800f030 con 0x7f67f80b62a0 2022-07-20T03:11:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67effff700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67dc053ad0 0x7f67dc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f67e806c9c0 con 0x7f67f80b62a0 2022-07-20T03:11:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.711+0000 7f67f6ffd700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67dc053ad0 0x7f67dc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.715+0000 7f67f6ffd700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67dc053ad0 0x7f67dc055f90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f67e000d000 tx=0x7f67e0007c50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:40.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.723+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67e80439d0 con 0x7f67f80b62a0 2022-07-20T03:11:40.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.847+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f67f8155e90 con 0x7f67f80b62a0 2022-07-20T03:11:40.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67effff700 1 -- 172.21.15.61:0/2980205480 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v52) v1 ==== 104+0+10 (secure 0 0 0) 0x7f67e8046ff0 con 0x7f67f80b62a0 2022-07-20T03:11:40.910 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:40.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67dc053ad0 msgr2=0x7f67dc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67dc053ad0 0x7f67dc055f90 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f67e000d000 tx=0x7f67e0007c50).stop 2022-07-20T03:11:40.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b62a0 msgr2=0x7f67f8152b80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:40.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b62a0 0x7f67f8152b80 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f67f0001b80 tx=0x7f67e8014000).stop 2022-07-20T03:11:40.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 shutdown_connections 2022-07-20T03:11:40.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f67dc053ad0 0x7f67dc055f90 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67f80b62a0 0x7f67f8152b80 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67f8143a20 0x7f67f8155780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 --2- 172.21.15.61:0/2980205480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67f80b3ed0 0x7f67f80ada30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:40.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 >> 172.21.15.61:0/2980205480 conn(0x7f67f8066f00 msgr2=0x7f67f80b24a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:40.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 shutdown_connections 2022-07-20T03:11:40.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:40.851+0000 7f67fd9b8700 1 -- 172.21.15.61:0/2980205480 wait complete. 2022-07-20T03:11:40.956 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:11:41.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.079+0000 7f09affff700 1 Processor -- start 2022-07-20T03:11:41.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.079+0000 7f09affff700 1 -- start start 2022-07-20T03:11:41.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.079+0000 7f09affff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b5920 0x7f09b013fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.079+0000 7f09affff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f09b00b4ee0 0x7f09b0142530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.079+0000 7f09affff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09b00b5d00 con 0x7f09b00ab990 2022-07-20T03:11:41.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09b013f090 con 0x7f09b00b4ee0 2022-07-20T03:11:41.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b013f330 con 0x7f09b00b5920 2022-07-20T03:11:41.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09af7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f09b00ab990 0x7f09b013e2a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42126/0 (socket says 172.21.15.61:42126) 2022-07-20T03:11:41.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09af7fe700 1 -- 172.21.15.61:0/153381875 learned_addr learned my addr 172.21.15.61:0/153381875 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:41.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09ae7fc700 1 --2- 172.21.15.61:0/153381875 >> v2:172.21.15.117:3300/0 conn(0x7f09b00b4ee0 0x7f09b0142530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09aeffd700 1 --2- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b5920 0x7f09b013fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09adffb700 1 -- 172.21.15.61:0/153381875 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 562579843 0 0) 0x7f09b00b5d00 con 0x7f09b00ab990 2022-07-20T03:11:41.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09adffb700 1 -- 172.21.15.61:0/153381875 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0994003260 con 0x7f09b00ab990 2022-07-20T03:11:41.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09ae7fc700 1 -- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b5920 msgr2=0x7f09b013fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:11:41.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09ae7fc700 1 --2- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b5920 0x7f09b013fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09ae7fc700 1 -- 172.21.15.61:0/153381875 >> v1:172.21.15.61:6789/0 conn(0x7f09b00ab990 legacy=0x7f09b013e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09ae7fc700 1 -- 172.21.15.61:0/153381875 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09b013f5d0 con 0x7f09b00b4ee0 2022-07-20T03:11:41.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09ae7fc700 1 --2- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b4ee0 0x7f09b0142530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f09a4002f70 tx=0x7f09a4006df0).ready entity=mon.1 client_cookie=5755bd391e9e3da8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:41.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09adffb700 1 -- 172.21.15.61:0/153381875 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a400d4a0 con 0x7f09b00b4ee0 2022-07-20T03:11:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09adffb700 1 -- 172.21.15.61:0/153381875 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a4000e90 con 0x7f09b00b4ee0 2022-07-20T03:11:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09adffb700 1 -- 172.21.15.61:0/153381875 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a400d7f0 con 0x7f09b00b4ee0 2022-07-20T03:11:41.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b4ee0 msgr2=0x7f09b0142530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 --2- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b4ee0 0x7f09b0142530 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f09a4002f70 tx=0x7f09a4006df0).stop 2022-07-20T03:11:41.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- 172.21.15.61:0/153381875 shutdown_connections 2022-07-20T03:11:41.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 --2- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b4ee0 0x7f09b0142530 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 --2- 172.21.15.61:0/153381875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b5920 0x7f09b013fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- 172.21.15.61:0/153381875 >> 172.21.15.61:0/153381875 conn(0x7f09b00632b0 msgr2=0x7f09b0063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:41.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- 172.21.15.61:0/153381875 shutdown_connections 2022-07-20T03:11:41.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.083+0000 7f09affff700 1 -- 172.21.15.61:0/153381875 wait complete. 2022-07-20T03:11:41.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 Processor -- start 2022-07-20T03:11:41.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 -- start start 2022-07-20T03:11:41.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 0x7f09b0140430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b4ee0 0x7f09b0064520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b5920 0x7f09b00b14d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b00b25c0 con 0x7f09b00ab990 2022-07-20T03:11:41.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0134be0 con 0x7f09b00b5920 2022-07-20T03:11:41.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09affff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f09b0064d00 con 0x7f09b00b4ee0 2022-07-20T03:11:41.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 0x7f09b0140430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 0x7f09b0140430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45530/0 (socket says 172.21.15.61:45530) 2022-07-20T03:11:41.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 -- 172.21.15.61:0/692881284 learned_addr learned my addr 172.21.15.61:0/692881284 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:41.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09ae7fc700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b4ee0 0x7f09b0064520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09af7fe700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b5920 0x7f09b00b14d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 -- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b4ee0 msgr2=0x7f09b0064520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b4ee0 0x7f09b0064520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 -- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b5920 msgr2=0x7f09b00b14d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b5920 0x7f09b00b14d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 -- 172.21.15.61:0/692881284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f09a400b180 con 0x7f09b00ab990 2022-07-20T03:11:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.087+0000 7f09aeffd700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 0x7f09b0140430 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f09a000c000 tx=0x7f09a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a0011980 con 0x7f09b00ab990 2022-07-20T03:11:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f09a0011ae0 con 0x7f09b00ab990 2022-07-20T03:11:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09b0064fa0 con 0x7f09b00ab990 2022-07-20T03:11:41.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f09a0011e40 con 0x7f09b00ab990 2022-07-20T03:11:41.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09b0142f90 con 0x7f09b00ab990 2022-07-20T03:11:41.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f09b00b11f0 con 0x7f09b00ab990 2022-07-20T03:11:41.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f09a0021070 con 0x7f09b00ab990 2022-07-20T03:11:41.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f099ffff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f098c053920 0x7f098c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f09a006dcd0 con 0x7f09b00ab990 2022-07-20T03:11:41.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.091+0000 7f09ae7fc700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f098c053920 0x7f098c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.095+0000 7f09ae7fc700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f098c053920 0x7f098c055de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f09a4018000 tx=0x7f09a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:41.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.103+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f09a0044b90 con 0x7f09b00ab990 2022-07-20T03:11:41.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.227+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f09b0064b00 con 0x7f09b00ab990 2022-07-20T03:11:41.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f099ffff700 1 -- 172.21.15.61:0/692881284 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v52) v1 ==== 100+0+10 (secure 0 0 0) 0x7f09a00481b0 con 0x7f09b00ab990 2022-07-20T03:11:41.238 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:11:41.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f098c053920 msgr2=0x7f098c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f098c053920 0x7f098c055de0 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f09a4018000 tx=0x7f09a4014000).stop 2022-07-20T03:11:41.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 msgr2=0x7f09b0140430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 0x7f09b0140430 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7f09a000c000 tx=0x7f09a0006df0).stop 2022-07-20T03:11:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 shutdown_connections 2022-07-20T03:11:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f098c053920 0x7f098c055de0 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f09b00b5920 0x7f09b00b14d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f09b00b4ee0 0x7f09b0064520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 --2- 172.21.15.61:0/692881284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f09b00ab990 0x7f09b0140430 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 >> 172.21.15.61:0/692881284 conn(0x7f09b00632b0 msgr2=0x7f09b00abe00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:41.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 shutdown_connections 2022-07-20T03:11:41.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.231+0000 7f09affff700 1 -- 172.21.15.61:0/692881284 wait complete. 2022-07-20T03:11:41.281 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:11:41.282 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:11:41.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0dcc0700 1 Processor -- start 2022-07-20T03:11:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0dcc0700 1 -- start start 2022-07-20T03:11:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0dcc0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2d080b1440 0x7f2d0813e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0dcc0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d08064140 con 0x7f2d08063d60 2022-07-20T03:11:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0dcc0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d08143800 con 0x7f2d080af150 2022-07-20T03:11:41.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0dcc0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d08143aa0 con 0x7f2d080b1440 2022-07-20T03:11:41.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0ccbe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2d08063d60 0x7f2d080b57e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42138/0 (socket says 172.21.15.61:42138) 2022-07-20T03:11:41.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d0ccbe700 1 -- 172.21.15.61:0/2425862456 learned_addr learned my addr 172.21.15.61:0/2425862456 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:11:41.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.439+0000 7f2d07fff700 1 --2- 172.21.15.61:0/2425862456 >> v2:172.21.15.117:3300/0 conn(0x7f2d080b1440 0x7f2d0813e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d077fe700 1 -- 172.21.15.61:0/2425862456 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1316244354 0 0) 0x7f2d08064140 con 0x7f2d08063d60 2022-07-20T03:11:41.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d077fe700 1 -- 172.21.15.61:0/2425862456 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2cec003260 con 0x7f2d08063d60 2022-07-20T03:11:41.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d07fff700 1 -- 172.21.15.61:0/2425862456 >> v1:172.21.15.117:6789/0 conn(0x7f2d080af150 legacy=0x7f2d0813eb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d07fff700 1 -- 172.21.15.61:0/2425862456 >> v1:172.21.15.61:6789/0 conn(0x7f2d08063d60 legacy=0x7f2d080b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d07fff700 1 -- 172.21.15.61:0/2425862456 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d080b1050 con 0x7f2d080b1440 2022-07-20T03:11:41.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d07fff700 1 --2- 172.21.15.61:0/2425862456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d0813e690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f2cf4002f70 tx=0x7f2cf4006d90).ready entity=mon.1 client_cookie=d694a46d171ebb1a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:41.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d077fe700 1 -- 172.21.15.61:0/2425862456 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf400d4a0 con 0x7f2d080b1440 2022-07-20T03:11:41.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d077fe700 1 -- 172.21.15.61:0/2425862456 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf4000e90 con 0x7f2d080b1440 2022-07-20T03:11:41.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d077fe700 1 -- 172.21.15.61:0/2425862456 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf400d7f0 con 0x7f2d080b1440 2022-07-20T03:11:41.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2425862456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 msgr2=0x7f2d0813e690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2425862456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d0813e690 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f2cf4002f70 tx=0x7f2cf4006d90).stop 2022-07-20T03:11:41.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2425862456 shutdown_connections 2022-07-20T03:11:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2425862456 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d0813e690 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2425862456 >> 172.21.15.61:0/2425862456 conn(0x7f2d080634e0 msgr2=0x7f2d080b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2425862456 shutdown_connections 2022-07-20T03:11:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2425862456 wait complete. 2022-07-20T03:11:41.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 Processor -- start 2022-07-20T03:11:41.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.443+0000 7f2d0dcc0700 1 -- start start 2022-07-20T03:11:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d08063d60 0x7f2d08065280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 0x7f2d0813a6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d081387e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d0813b100 con 0x7f2d080af150 2022-07-20T03:11:41.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d081369d0 con 0x7f2d080b1440 2022-07-20T03:11:41.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d08136c70 con 0x7f2d08063d60 2022-07-20T03:11:41.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 0x7f2d0813a6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 0x7f2d0813a6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45546/0 (socket says 172.21.15.61:45546) 2022-07-20T03:11:41.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 -- 172.21.15.61:0/2336977831 learned_addr learned my addr 172.21.15.61:0/2336977831 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:11:41.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0ccbe700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d08063d60 0x7f2d08065280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0d4bf700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d081387e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 -- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d08063d60 msgr2=0x7f2d08065280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d08063d60 0x7f2d08065280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 -- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 msgr2=0x7f2d081387e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d081387e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 -- 172.21.15.61:0/2336977831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2cf400b120 con 0x7f2d080af150 2022-07-20T03:11:41.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d07fff700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 0x7f2d0813a6b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f2cf4002f70 tx=0x7f2cf4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:41.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf4011880 con 0x7f2d080af150 2022-07-20T03:11:41.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2cf40119e0 con 0x7f2d080af150 2022-07-20T03:11:41.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2cf4011d40 con 0x7f2d080af150 2022-07-20T03:11:41.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d081361c0 con 0x7f2d080af150 2022-07-20T03:11:41.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d0813aba0 con 0x7f2d080af150 2022-07-20T03:11:41.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.447+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d0813af60 con 0x7f2d080af150 2022-07-20T03:11:41.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.451+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2cf400f070 con 0x7f2d080af150 2022-07-20T03:11:41.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.451+0000 7f2d05ffb700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2ce80539b0 0x7f2ce8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:11:41.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.451+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2cf406d250 con 0x7f2d080af150 2022-07-20T03:11:41.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.451+0000 7f2d0ccbe700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2ce80539b0 0x7f2ce8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:11:41.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.451+0000 7f2d0ccbe700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2ce80539b0 0x7f2ce8055e70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f2cfc00d000 tx=0x7f2cfc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:11:41.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.455+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d0813af60 con 0x7f2d080af150 2022-07-20T03:11:41.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f2d0813af60 con 0x7f2d080af150 2022-07-20T03:11:41.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d05ffb700 1 -- 172.21.15.61:0/2336977831 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7f2d0813af60 con 0x7f2d080af150 2022-07-20T03:11:41.624 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:11:41.624 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:11:41.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2ce80539b0 msgr2=0x7f2ce8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2ce80539b0 0x7f2ce8055e70 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f2cfc00d000 tx=0x7f2cfc009040).stop 2022-07-20T03:11:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 msgr2=0x7f2d0813a6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:11:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 0x7f2d0813a6b0 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7f2cf4002f70 tx=0x7f2cf4014000).stop 2022-07-20T03:11:41.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 shutdown_connections 2022-07-20T03:11:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2ce80539b0 0x7f2ce8055e70 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d080b1440 0x7f2d081387e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d080af150 0x7f2d0813a6b0 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 --2- 172.21.15.61:0/2336977831 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d08063d60 0x7f2d08065280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:11:41.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 >> 172.21.15.61:0/2336977831 conn(0x7f2d080634e0 msgr2=0x7f2d080af8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:11:41.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 shutdown_connections 2022-07-20T03:11:41.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:11:41.619+0000 7f2d0dcc0700 1 -- 172.21.15.61:0/2336977831 wait complete. 2022-07-20T03:11:41.669 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:11:41.883 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 29, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:11:12.676790+0000', 'modified': '2022-07-20T03:11:14.703208+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5610}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5610': {'gid': 5610, 'name': 'a', 'rank': 0, 'incarnation': 26, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2246781672', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2246781672}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2246781672}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [10], 'metadata_pool': 9, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:11:41.884 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:11:41.885 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:11:42.011 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5610,"want_state":"up:active","state":"up:active","rank_uptime":29.314877637999999,"mdsmap_epoch":29,"osdmap_epoch":52,"osdmap_epoch_barrier":52,"uptime":35.695046544} 2022-07-20T03:11:42.012 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5610, "mdsmap_epoch": 29, "osdmap_epoch": 52, "osdmap_epoch_barrier": 52, "rank_uptime": 29.314877638, "state": "up:active", "uptime": 35.695046544, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:11:42.013 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:11:42.020 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:11:42.029 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:11:42.030 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.031 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:42.064 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:11:42.146 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:11:42.147 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:11:42.162 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:11:42.164 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:11:42.166 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.167 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:11:42.214 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:11:42.215 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:42.218 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:11:42.268 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:11:42.269 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:11:42.311 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:11:42.296+0000 7fc0c4f30080 -1 init, newargv = 0x55cfb9141430 newargc=9ceph-fuse[ 2022-07-20T03:11:42.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:22723]: starting ceph client 2022-07-20T03:11:42.312 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22723]: starting fuse 2022-07-20T03:11:42.349 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:11:42.354 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:11:42.355 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.356 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:11:42.403 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:11:42.404 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:11:42.404 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.405 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:11:42.405 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:11:42.405 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:11:42.406 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:11:42.406 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:11:42.407 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:42.407 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:11:42.408 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:11:42.408 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:11:42.408 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:42.409 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:11:42.409 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:11:42.410 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:11:42.411 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:11:42.411 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:42.411 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:11:42.412 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:11:42.412 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:11:42.413 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:11:42.413 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:11:42.413 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:11:42.414 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:11:42.414 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:11:42.466 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22723.asok 2022-07-20T03:11:42.486 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22723.asok 2022-07-20T03:11:42.487 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.503 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22723.asok status 2022-07-20T03:11:42.635 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:11:42.635 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:11:42.636 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:11:42.636 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:11:42.637 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:11:42.637 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:11:42.637 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:11:42.649 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22723", 2022-07-20T03:11:42.649 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:11:42.650 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:42.651 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:11:42.651 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:11:42.652 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 5863, 2022-07-20T03:11:42.652 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:11:42.653 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:11:42.653 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:11:42.654 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 5863 2022-07-20T03:11:42.654 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:42.656 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:11:42.656 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:11:42.657 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:11:42.657 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2315858686 2022-07-20T03:11:42.658 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:11:42.658 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:42.659 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:11:42.659 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:11:42.659 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:11:42.660 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2315858686 2022-07-20T03:11:42.661 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:11:42.662 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.5863 172.21.15.124:0/2315858686", 2022-07-20T03:11:42.662 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/2315858686", 2022-07-20T03:11:42.663 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:11:42.665 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 29, 2022-07-20T03:11:42.679 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 52, 2022-07-20T03:11:42.690 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:11:42.690 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:11:42.691 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:11:42.691 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:11:42.692 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.693 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:11:42.711 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:11:42.739 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:11:42.754 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:11:42.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:11:42.755 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:05.725 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:05.761 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:05.785 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:05.788 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:05.797 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:35.861 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:35.907 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:35.927 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:35.939 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:35.958 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:12:36.305 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:12:36.306 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:36.345 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:36.370 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:36.451 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:36.485 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:36.508 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:12:36.509 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:36.517 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:36.518 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:36.533 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:12:36.534 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:12:36.535 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:12:36.536 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:12:36.537 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:36.579 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22723]: fuse finished with error 0 and tester_r 0 2022-07-20T03:12:36.581 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:36.582 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:36.582 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:36.633 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:12:36.637 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:12:36.761 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:12:36.762 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:36.782 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:36.783 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:36.784 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:12:36.816 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:36.820 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:12:36.820 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:12:36.822 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:12:36.822 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:12:36.888 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:36.891 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:12:36.892 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:36.893 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:36.893 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:12:36.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:36.930 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:12:36.931 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:12:36.934 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:12:36.935 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:12:36.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:36.958 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:12:36.959 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:12:37.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 Processor -- start 2022-07-20T03:12:37.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- start start 2022-07-20T03:12:37.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f07a00ac3a0 0x7f07a0143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:37.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07a005fa50 0x7f07a013e9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:37.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07a0060980 con 0x7f07a00b1ef0 2022-07-20T03:12:37.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a00b22d0 con 0x7f07a005fa50 2022-07-20T03:12:37.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07a0098810 con 0x7f07a00ac3a0 2022-07-20T03:12:37.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a4b78700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07a005fa50 0x7f07a013e9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:37.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f07a00b1ef0 0x7f07a0142f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42150/0 (socket says 172.21.15.61:42150) 2022-07-20T03:12:37.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079ffff700 1 -- 172.21.15.61:0/2078843501 learned_addr learned my addr 172.21.15.61:0/2078843501 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:37.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079f7fe700 1 --2- 172.21.15.61:0/2078843501 >> v2:172.21.15.117:3300/0 conn(0x7f07a00ac3a0 0x7f07a0143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:37.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079effd700 1 -- 172.21.15.61:0/2078843501 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 476083035 0 0) 0x7f07a0060980 con 0x7f07a00b1ef0 2022-07-20T03:12:37.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079effd700 1 -- 172.21.15.61:0/2078843501 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0780003260 con 0x7f07a00b1ef0 2022-07-20T03:12:37.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a4b78700 1 -- 172.21.15.61:0/2078843501 >> v2:172.21.15.117:3300/0 conn(0x7f07a00ac3a0 msgr2=0x7f07a0143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:37.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a4b78700 1 --2- 172.21.15.61:0/2078843501 >> v2:172.21.15.117:3300/0 conn(0x7f07a00ac3a0 0x7f07a0143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a4b78700 1 -- 172.21.15.61:0/2078843501 >> v1:172.21.15.61:6789/0 conn(0x7f07a00b1ef0 legacy=0x7f07a0142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a4b78700 1 -- 172.21.15.61:0/2078843501 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07a0060bb0 con 0x7f07a005fa50 2022-07-20T03:12:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079f7fe700 1 --2- 172.21.15.61:0/2078843501 >> v2:172.21.15.117:3300/0 conn(0x7f07a00ac3a0 0x7f07a0143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a4b78700 1 --2- 172.21.15.61:0/2078843501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a005fa50 0x7f07a013e9a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f0798002f70 tx=0x7f0798006df0).ready entity=mon.0 client_cookie=34c3ef678429bca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079effd700 1 -- 172.21.15.61:0/2078843501 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f079800d570 con 0x7f07a005fa50 2022-07-20T03:12:37.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079effd700 1 -- 172.21.15.61:0/2078843501 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0798000e90 con 0x7f07a005fa50 2022-07-20T03:12:37.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f079effd700 1 -- 172.21.15.61:0/2078843501 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f079800d8c0 con 0x7f07a005fa50 2022-07-20T03:12:37.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- 172.21.15.61:0/2078843501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a005fa50 msgr2=0x7f07a013e9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 --2- 172.21.15.61:0/2078843501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a005fa50 0x7f07a013e9a0 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f0798002f70 tx=0x7f0798006df0).stop 2022-07-20T03:12:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- 172.21.15.61:0/2078843501 shutdown_connections 2022-07-20T03:12:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 --2- 172.21.15.61:0/2078843501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a005fa50 0x7f07a013e9a0 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 --2- 172.21.15.61:0/2078843501 >> v2:172.21.15.117:3300/0 conn(0x7f07a00ac3a0 0x7f07a0143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:37.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- 172.21.15.61:0/2078843501 >> 172.21.15.61:0/2078843501 conn(0x7f07a00b2ca0 msgr2=0x7f07a00b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:37.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- 172.21.15.61:0/2078843501 shutdown_connections 2022-07-20T03:12:37.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.103+0000 7f07a5379700 1 -- 172.21.15.61:0/2078843501 wait complete. 2022-07-20T03:12:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 Processor -- start 2022-07-20T03:12:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 -- start start 2022-07-20T03:12:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07a005fa50 0x7f07a01423b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 0x7f07a00af2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07a00b1ef0 0x7f07a009b2c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a009be30 con 0x7f07a00ac3a0 2022-07-20T03:12:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07a009ae20 con 0x7f07a00b1ef0 2022-07-20T03:12:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a5379700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07a00af7d0 con 0x7f07a005fa50 2022-07-20T03:12:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 0x7f07a00af2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 0x7f07a00af2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45560/0 (socket says 172.21.15.61:45560) 2022-07-20T03:12:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 -- 172.21.15.61:0/1145445995 learned_addr learned my addr 172.21.15.61:0/1145445995 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079ffff700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07a005fa50 0x7f07a01423b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f07a4b78700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07a00b1ef0 0x7f07a009b2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 -- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07a005fa50 msgr2=0x7f07a01423b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07a005fa50 0x7f07a01423b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 -- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07a00b1ef0 msgr2=0x7f07a009b2c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07a00b1ef0 0x7f07a009b2c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 -- 172.21.15.61:0/1145445995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f079800b180 con 0x7f07a00ac3a0 2022-07-20T03:12:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.107+0000 7f079f7fe700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 0x7f07a00af2e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f079000a000 tx=0x7f0790008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0790013620 con 0x7f07a00ac3a0 2022-07-20T03:12:37.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0790013780 con 0x7f07a00ac3a0 2022-07-20T03:12:37.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07a014b160 con 0x7f07a00ac3a0 2022-07-20T03:12:37.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0790013ae0 con 0x7f07a00ac3a0 2022-07-20T03:12:37.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07a00aeb80 con 0x7f07a00ac3a0 2022-07-20T03:12:37.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07a009bfa0 con 0x7f07a00ac3a0 2022-07-20T03:12:37.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0790013c40 con 0x7f07a00ac3a0 2022-07-20T03:12:37.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079d7fa700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07840539b0 0x7f0784055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:37.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f079006bcb0 con 0x7f07a00ac3a0 2022-07-20T03:12:37.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.111+0000 7f079ffff700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07840539b0 0x7f0784055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:37.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.115+0000 7f079ffff700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07840539b0 0x7f0784055e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f078800d000 tx=0x7f0788009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:37.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.119+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07a009bfa0 con 0x7f07a00ac3a0 2022-07-20T03:12:37.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.243+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f07a009afe0 con 0x7f07a00ac3a0 2022-07-20T03:12:37.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.243+0000 7f079d7fa700 1 -- 172.21.15.61:0/1145445995 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v52) v1 ==== 81+0+33695 (secure 0 0 0) 0x7f0790042bb0 con 0x7f07a00ac3a0 2022-07-20T03:12:37.249 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:37.249 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:12:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 52, 2022-07-20T03:12:37.250 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:12:37.251 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:12:37.251 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:11:13.680406+0000", 2022-07-20T03:12:37.252 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:12:37.252 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:12:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:12:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:12:37.253 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:12:37.255 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:12:37.255 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:12:37.256 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:12:37.256 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:12:37.257 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:37.257 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:12:37.258 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:12:37.288 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:12:37.289 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:12:37.289 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:12:37.345 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 10, 2022-07-20T03:12:37.350 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:12:37.351 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:12:37.351 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:12:37.351 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:12:37.352 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:12:37.352 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:37.353 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:12:37.353 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:12:37.354 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:12:37.406 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:12:37.407 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:12:37.408 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:12:37.408 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:12:37.409 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:12:37.409 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:12:37.443 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:12:37.443 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:12:37.458 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:12:37.458 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:12:37.459 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:12:37.459 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:12:37.459 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:12:37.459 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:12:37.459 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:12:37.460 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:12:37.460 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:12:37.460 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:12:37.460 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:12:37.461 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:12:37.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.461 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:12:37.461 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:12:37.461 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:12:37.478 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:12:37.478 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:12:37.479 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:12:37.479 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:12:37.479 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:12:37.479 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:12:37.479 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:12:37.480 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:12:37.480 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:12:37.480 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:12:37.480 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:12:37.481 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:12:37.481 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:12:37.481 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:12:37.481 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:12:37.481 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:12:37.482 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:12:37.482 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:12:37.516 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:12:37.516 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:12:37.517 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:12:37.517 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:12:37.517 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:12:37.517 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:12:37.538 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.539 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:12:37.540 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:12:37.540 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:12:37.541 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:12:37.541 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:12:37.542 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:12:37.542 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:12:37.543 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:12:37.543 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:12:37.543 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:12:37.544 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:12:37.544 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:12:37.545 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:12:37.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.545 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:12:37.546 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:12:37.546 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:37.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.569 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:37.570 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:12:37.570 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:12:37.571 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:12:37.571 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:12:37.571 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:12:37.572 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:12:37.572 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:12:37.573 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:12:37.573 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:12:37.574 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:12:37.574 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:12:37.575 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:12:37.575 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:12:37.576 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:12:37.576 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:12:37.576 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:12:37.577 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:12:37.577 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:12:37.659 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:12:37.660 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:12:37.662 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:12:37.663 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:12:37.663 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:12:37.664 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.665 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:12:37.665 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:12:37.666 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:12:37.666 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:12:37.667 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:12:37.667 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:12:37.667 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:12:37.668 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:12:37.668 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:12:37.669 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:12:37.670 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:12:37.713 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:12:37.714 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:12:37.714 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:12:37.727 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:12:37.728 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:12:37.728 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:12:37.729 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:12:37.729 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:12:37.730 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:12:37.736 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:12:37.737 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:12:37.743 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:12:37.744 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:12:37.753 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:12:37.757 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:12:37.757 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:12:37.769 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.770 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:12:37.770 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:12:37.772 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:12:37.783 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:12:37.784 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:12:37.785 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:12:37.786 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:12:37.787 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:12:37.787 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:12:37.788 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:12:37.790 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:12:37.790 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:12:37.791 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:12:37.791 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:12:37.792 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:37.792 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.793 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:37.794 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 9, 2022-07-20T03:12:37.795 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:12:37.796 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:11:10.825678+0000", 2022-07-20T03:12:37.797 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:12:37.798 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:12:37.798 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:12:37.799 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:12:37.799 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:12:37.801 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:12:37.801 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:12:37.801 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:12:37.802 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:12:37.802 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:12:37.803 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:12:37.803 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:12:37.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:12:37.812 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:12:37.812 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:12:37.813 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:12:37.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:12:37.815 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:12:37.815 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:12:37.815 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:12:37.815 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "51", 2022-07-20T03:12:37.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:12:37.817 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:12:37.817 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:12:37.817 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:12:37.818 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:12:37.818 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:12:37.818 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:12:37.819 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:12:37.819 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:12:37.819 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:12:37.826 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:12:37.826 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:12:37.827 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:12:37.827 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:12:37.828 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:12:37.828 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:12:37.828 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:12:37.829 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:12:37.829 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:12:37.830 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:12:37.830 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:12:37.830 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:12:37.831 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:12:37.831 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:12:37.832 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:12:37.832 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:12:37.833 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.833 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:12:37.834 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:12:37.834 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:12:37.836 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:12:37.837 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:12:37.837 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:12:37.838 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:12:37.854 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:12:37.863 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:12:37.872 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:12:37.872 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:12:37.873 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:12:37.891 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:12:37.891 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:12:37.893 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:12:37.893 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.893 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:12:37.893 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:12:37.894 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:12:37.894 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:37.894 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:37.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:37.899 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 10, 2022-07-20T03:12:37.900 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:12:37.900 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:11:11.707209+0000", 2022-07-20T03:12:37.909 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:12:37.909 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:12:37.937 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:12:37.937 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:12:37.938 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:12:37.938 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:12:37.938 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:12:37.938 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:12:37.938 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:12:37.939 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:12:37.939 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:12:37.939 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:12:37.939 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:12:37.940 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:12:37.940 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:12:37.950 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:12:37.951 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:12:37.951 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:12:37.951 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:12:37.955 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:12:37.955 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.955 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "52", 2022-07-20T03:12:37.955 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:12:37.956 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:12:37.956 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:12:37.956 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:12:37.956 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:12:37.957 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:12:37.957 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:12:37.957 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:12:37.957 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:12:37.962 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:12:37.980 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:12:37.981 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:12:37.981 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:12:37.982 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:12:37.982 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:12:37.983 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:12:37.984 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:12:37.984 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:12:37.985 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:12:37.986 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:12:37.989 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:12:37.989 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:12:37.990 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:12:37.991 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:12:37.992 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:12:37.994 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:12:37.994 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:37.995 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:12:37.995 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:12:37.996 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:12:37.997 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:12:37.997 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:12:38.001 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:12:38.019 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:12:38.019 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:12:38.021 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:12:38.022 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:12:38.023 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:12:38.027 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:12:38.027 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:12:38.031 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:12:38.031 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:12:38.031 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.032 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.032 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.032 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:38.032 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:12:38.033 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.033 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:12:38.033 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:12:38.033 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.034 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.034 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.034 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.034 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.035 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:38.055 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.056 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.056 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.060 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.060 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:12:38.061 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.064 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:12:38.066 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.067 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.067 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.069 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.070 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.070 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:12:38.072 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.088 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.088 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:12:38.089 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.090 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.090 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.092 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.092 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.094 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.104 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:12:38.104 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.106 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.106 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.108 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:12:38.109 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.111 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.112 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.112 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.115 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.116 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:12:38.119 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.120 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.122 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.122 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:12:38.123 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:38.123 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.124 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.124 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.124 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:12:38.125 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:12:38.126 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:12:38.126 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:12:38.127 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.127 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.128 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.128 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.131 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:12:38.131 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:12:38.132 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.132 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.132 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.133 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.134 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.134 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.135 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:38.135 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.136 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.136 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:12:38.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.139 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.140 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.141 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:12:38.142 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.142 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.144 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.145 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.145 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.146 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:12:38.147 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.147 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.148 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:12:38.150 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.152 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.152 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.154 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.155 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.156 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:12:38.157 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.158 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.159 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.159 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.160 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:12:38.160 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.161 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.161 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.164 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.165 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.165 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.165 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.166 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:12:38.167 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.168 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.169 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:12:38.170 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:38.171 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.171 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.173 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:12:38.174 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:12:38.174 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:12:38.175 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:12:38.176 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.176 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.177 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.177 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.179 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.179 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:12:38.192 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:12:38.192 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.193 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.193 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.205 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.205 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.206 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.206 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.206 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:38.214 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.214 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.214 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:12:38.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.217 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.218 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.218 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.218 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:12:38.219 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.219 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.220 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.220 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.221 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:12:38.222 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.222 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.222 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:12:38.223 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.223 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.223 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.223 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.224 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.224 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.225 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:12:38.225 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.239 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.240 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:12:38.241 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.241 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.241 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.241 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.242 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.242 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:12:38.243 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.247 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:12:38.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:38.251 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.251 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.254 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:12:38.255 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:12:38.256 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:12:38.257 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:12:38.258 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.258 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.259 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.260 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.261 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.262 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:12:38.263 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:12:38.271 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.282 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.283 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.283 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.284 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.285 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.288 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.289 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-20T03:12:38.289 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.290 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.290 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.291 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.291 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.292 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.292 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:12:38.293 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.294 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.295 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:12:38.315 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.315 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.316 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.317 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.317 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.318 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.318 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.319 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:12:38.319 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.320 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.320 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.320 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:12:38.321 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.322 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.326 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.334 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.334 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.337 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:12:38.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.339 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.340 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.341 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:12:38.341 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.341 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.342 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.343 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.343 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.344 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.345 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:12:38.346 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.346 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.347 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.349 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.349 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:12:38.350 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:38.350 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.351 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.352 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.352 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:12:38.353 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:12:38.353 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:12:38.354 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:12:38.354 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.354 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.355 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.359 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.361 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.362 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:12:38.362 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:12:38.362 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.363 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.381 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.381 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.381 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.411 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.439 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.449 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:38.451 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.452 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.453 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.454 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.456 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:12:38.485 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.487 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.487 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.488 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:12:38.488 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.489 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.489 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.490 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.490 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.491 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.491 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.491 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.492 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:12:38.492 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.492 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.493 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.493 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.502 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:12:38.502 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.503 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.503 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.504 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.504 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.505 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.505 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.505 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.506 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:12:38.507 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.507 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.508 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.509 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.509 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:12:38.510 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.510 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.510 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.511 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.512 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.512 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.512 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.513 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.513 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:12:38.539 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.539 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.539 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.540 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.540 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:12:38.540 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:38.540 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.541 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.541 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.541 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:12:38.541 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:12:38.556 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:12:38.556 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:12:38.557 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.557 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.557 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.557 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.558 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:12:38.558 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:12:38.559 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.559 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.559 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.559 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.559 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.560 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.560 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.560 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:38.560 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.561 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.561 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.561 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.561 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.590 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.591 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:12:38.591 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.591 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.591 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.592 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.592 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:12:38.592 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.592 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.593 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.593 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.593 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.593 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.593 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.594 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.594 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:12:38.594 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.594 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.595 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.595 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.595 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:12:38.595 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.596 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.596 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.596 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.596 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.596 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.597 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.597 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.597 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:12:38.597 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.598 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.598 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.598 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.598 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:12:38.599 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.599 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.599 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.599 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.599 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.600 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.600 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.600 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.656 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:12:38.658 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.658 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.659 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.670 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.687 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:12:38.687 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:38.688 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.697 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.697 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.698 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:12:38.698 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:12:38.722 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:12:38.723 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:12:38.723 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.724 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.726 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.727 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.727 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.728 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:12:38.728 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:12:38.729 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.729 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.730 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.730 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.731 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.731 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.731 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.740 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:38.740 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.741 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.751 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.751 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.752 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.752 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.753 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:12:38.753 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.754 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.762 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.763 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.763 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:12:38.764 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.764 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.764 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.765 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.766 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.767 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.767 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.767 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.768 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:12:38.768 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.769 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.769 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.775 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.777 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:12:38.779 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.779 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.780 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.781 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.781 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.782 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.782 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.782 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.782 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:12:38.783 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.783 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.783 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.783 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.784 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:12:38.784 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.785 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.785 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.785 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.785 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.786 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.786 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.786 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.787 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:12:38.787 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.787 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.788 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.789 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:12:38.789 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:38.789 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.789 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.790 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.790 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:12:38.790 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:12:38.790 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:12:38.791 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:12:38.791 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.791 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.791 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.792 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.792 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.797 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.798 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:12:38.798 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:12:38.799 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:38.799 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:38.799 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:38.799 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:38.800 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:38.800 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:38.800 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:38.800 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-20T03:12:38.801 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:38.801 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:38.801 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:38.801 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.802 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.802 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.827 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:12:38.827 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.828 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.828 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.828 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:12:38.829 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.829 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.829 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.829 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.850 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:38.851 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.862 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.862 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.864 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:12:38.864 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.864 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.864 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.865 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.865 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:12:38.883 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.883 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.884 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.884 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.884 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:38.885 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.885 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.885 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:12:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.886 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.891 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.893 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:12:38.893 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.893 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.894 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.895 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.897 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:38.897 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:38.900 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.900 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:38.901 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:12:38.902 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.902 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.904 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:38.904 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:12:38.905 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:38.907 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.907 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.908 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.908 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:12:38.910 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:12:38.910 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:12:38.911 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:12:38.911 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:38.913 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:38.913 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:38.914 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:38.914 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:38.922 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:38.922 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:12:38.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.923 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:12:38.924 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:38.924 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:38.925 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:38.929 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:38.929 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:38.929 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:12:38.930 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:38.930 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.931 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.931 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:12:38.931 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:38.931 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:38.955 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:38.995 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:38.995 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:38.996 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:12:38.996 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:38.997 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:38.997 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:38.997 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:12:38.999 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:38.999 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:39.000 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:39.000 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:39.001 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:39.001 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:12:39.002 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:39.002 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:39.003 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:39.003 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:12:39.003 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:39.004 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:39.004 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:39.005 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:39.009 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:39.009 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:12:39.010 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:39.011 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:39.012 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:39.012 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:12:39.013 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:39.013 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:39.014 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:39.015 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:39.015 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:39.016 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:12:39.017 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:39.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:39.019 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:39.019 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:12:39.020 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:39.025 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:39.025 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:39.025 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:39.026 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:39.026 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:12:39.026 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:39.026 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:39.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:39.027 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:12:39.027 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:39.027 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:39.028 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:39.028 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:39.028 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:39.028 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:12:39.029 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:39.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:39.029 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:39.029 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:12:39.029 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:39.030 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:39.030 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:39.030 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:39.030 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:39.031 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:12:39.031 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:39.032 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:39.033 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:39.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:12:39.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:12:39.033 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:12:39.034 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:12:39.050 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:12:39.050 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:12:39.051 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:12:39.051 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:12:39.051 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:12:39.051 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:12:39.052 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:12:39.052 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:12:39.057 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:12:39.058 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:39.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:39.058 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:12:39.059 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:12:39.059 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:12:39.059 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:12:39.059 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:12:39.060 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:12:39.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07840539b0 msgr2=0x7f0784055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07840539b0 0x7f0784055e70 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f078800d000 tx=0x7f0788009040).stop 2022-07-20T03:12:39.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 msgr2=0x7f07a00af2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 0x7f07a00af2e0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f079000a000 tx=0x7f0790008040).stop 2022-07-20T03:12:39.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 shutdown_connections 2022-07-20T03:12:39.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07840539b0 0x7f0784055e70 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07a00b1ef0 0x7f07a009b2c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07a00ac3a0 0x7f07a00af2e0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 --2- 172.21.15.61:0/1145445995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07a005fa50 0x7f07a01423b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 >> 172.21.15.61:0/1145445995 conn(0x7f07a00b2ca0 msgr2=0x7f07a013faa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:39.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 shutdown_connections 2022-07-20T03:12:39.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:37.247+0000 7f07a5379700 1 -- 172.21.15.61:0/1145445995 wait complete. 2022-07-20T03:12:39.082 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:39.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 Processor -- start 2022-07-20T03:12:39.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- start start 2022-07-20T03:12:39.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd39c0b1e00 0x7fd39c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd39c0acef0 0x7fd39c146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd39c05d480 con 0x7fd39c05c3f0 2022-07-20T03:12:39.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39c0b21e0 con 0x7fd39c0acef0 2022-07-20T03:12:39.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd39c05cf40 con 0x7fd39c0b1e00 2022-07-20T03:12:39.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd393fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd39c05c3f0 0x7fd39c0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42166/0 (socket says 172.21.15.61:42166) 2022-07-20T03:12:39.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd393fff700 1 -- 172.21.15.61:0/824348025 learned_addr learned my addr 172.21.15.61:0/824348025 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:39.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a12e9700 1 --2- 172.21.15.61:0/824348025 >> v2:172.21.15.61:3300/0 conn(0x7fd39c0acef0 0x7fd39c146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a0ae8700 1 --2- 172.21.15.61:0/824348025 >> v2:172.21.15.117:3300/0 conn(0x7fd39c0b1e00 0x7fd39c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3937fe700 1 -- 172.21.15.61:0/824348025 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3578059025 0 0) 0x7fd39c05d480 con 0x7fd39c05c3f0 2022-07-20T03:12:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3937fe700 1 -- 172.21.15.61:0/824348025 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd388003260 con 0x7fd39c05c3f0 2022-07-20T03:12:39.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a12e9700 1 -- 172.21.15.61:0/824348025 >> v2:172.21.15.117:3300/0 conn(0x7fd39c0b1e00 msgr2=0x7fd39c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a12e9700 1 --2- 172.21.15.61:0/824348025 >> v2:172.21.15.117:3300/0 conn(0x7fd39c0b1e00 0x7fd39c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a12e9700 1 -- 172.21.15.61:0/824348025 >> v1:172.21.15.61:6789/0 conn(0x7fd39c05c3f0 legacy=0x7fd39c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a12e9700 1 -- 172.21.15.61:0/824348025 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd39c05d1e0 con 0x7fd39c0acef0 2022-07-20T03:12:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a12e9700 1 --2- 172.21.15.61:0/824348025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c0acef0 0x7fd39c146410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7fd3940032f0 tx=0x7fd3940091d0).ready entity=mon.0 client_cookie=18e040b9f1f23eef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3937fe700 1 -- 172.21.15.61:0/824348025 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39400b600 con 0x7fd39c0acef0 2022-07-20T03:12:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3937fe700 1 -- 172.21.15.61:0/824348025 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd39400b760 con 0x7fd39c0acef0 2022-07-20T03:12:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3937fe700 1 -- 172.21.15.61:0/824348025 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39400b8d0 con 0x7fd39c0acef0 2022-07-20T03:12:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/824348025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c0acef0 msgr2=0x7fd39c146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/824348025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c0acef0 0x7fd39c146410 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7fd3940032f0 tx=0x7fd3940091d0).stop 2022-07-20T03:12:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/824348025 shutdown_connections 2022-07-20T03:12:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/824348025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c0acef0 0x7fd39c146410 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/824348025 >> v2:172.21.15.117:3300/0 conn(0x7fd39c0b1e00 0x7fd39c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/824348025 >> 172.21.15.61:0/824348025 conn(0x7fd39c0611c0 msgr2=0x7fd39c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/824348025 shutdown_connections 2022-07-20T03:12:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.199+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/824348025 wait complete. 2022-07-20T03:12:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 Processor -- start 2022-07-20T03:12:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 -- start start 2022-07-20T03:12:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd39c0acef0 0x7fd39c0a18a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd39c0b1e00 0x7fd39c09ec30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 0x7fd39c0a06f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c09e560 con 0x7fd39c149770 2022-07-20T03:12:39.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c149cf0 con 0x7fd39c0acef0 2022-07-20T03:12:39.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd39c09e790 con 0x7fd39c0b1e00 2022-07-20T03:12:39.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 0x7fd39c0a06f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 0x7fd39c0a06f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45574/0 (socket says 172.21.15.61:45574) 2022-07-20T03:12:39.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 -- 172.21.15.61:0/446880867 learned_addr learned my addr 172.21.15.61:0/446880867 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:39.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a0ae8700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd39c0acef0 0x7fd39c0a18a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd393fff700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd39c0b1e00 0x7fd39c09ec30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 -- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd39c0b1e00 msgr2=0x7fd39c09ec30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd39c0b1e00 0x7fd39c09ec30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 -- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd39c0acef0 msgr2=0x7fd39c0a18a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd39c0acef0 0x7fd39c0a18a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 -- 172.21.15.61:0/446880867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394009470 con 0x7fd39c149770 2022-07-20T03:12:39.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a12e9700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 0x7fd39c0a06f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fd3940032f0 tx=0x7fd394016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:39.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39400b600 con 0x7fd39c149770 2022-07-20T03:12:39.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd394011b90 con 0x7fd39c149770 2022-07-20T03:12:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd39c09e080 con 0x7fd39c149770 2022-07-20T03:12:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd39401f3f0 con 0x7fd39c149770 2022-07-20T03:12:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.203+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd39c09f120 con 0x7fd39c149770 2022-07-20T03:12:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.207+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd39c0574e0 con 0x7fd39c149770 2022-07-20T03:12:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.207+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd394026030 con 0x7fd39c149770 2022-07-20T03:12:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.207+0000 7fd391ffb700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd37c053ad0 0x7fd37c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.207+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd39406c830 con 0x7fd39c149770 2022-07-20T03:12:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.207+0000 7fd3a0ae8700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd37c053ad0 0x7fd37c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.211+0000 7fd3a0ae8700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd37c053ad0 0x7fd37c055f90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd39800b000 tx=0x7fd398006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:39.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.215+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3940471c0 con 0x7fd39c149770 2022-07-20T03:12:39.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.375+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd39c09e950 con 0x7fd39c149770 2022-07-20T03:12:39.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd391ffb700 1 -- 172.21.15.61:0/446880867 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 29 v29) v1 ==== 94+0+2139 (secure 0 0 0) 0x7fd394024870 con 0x7fd39c149770 2022-07-20T03:12:39.405 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:39.406 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":29,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":29,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:11:14.703208+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5610},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5610":{"gid":5610,"name":"a","rank":0,"incarnation":26,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2246781672","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2246781672},{"type":"v1","addr":"172.21.15.61:6833","nonce":2246781672}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:12:39.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd37c053ad0 msgr2=0x7fd37c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd37c053ad0 0x7fd37c055f90 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fd39800b000 tx=0x7fd398006df0).stop 2022-07-20T03:12:39.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 msgr2=0x7fd39c0a06f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 0x7fd39c0a06f0 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7fd3940032f0 tx=0x7fd394016000).stop 2022-07-20T03:12:39.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 shutdown_connections 2022-07-20T03:12:39.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd37c053ad0 0x7fd37c055f90 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd39c149770 0x7fd39c0a06f0 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd39c0b1e00 0x7fd39c09ec30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 --2- 172.21.15.61:0/446880867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd39c0acef0 0x7fd39c0a18a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 >> 172.21.15.61:0/446880867 conn(0x7fd39c05c3f0 msgr2=0x7fd39c069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:39.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 shutdown_connections 2022-07-20T03:12:39.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.379+0000 7fd3a2d4c700 1 -- 172.21.15.61:0/446880867 wait complete. 2022-07-20T03:12:39.462 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 29 2022-07-20T03:12:39.462 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:12:39.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 Processor -- start 2022-07-20T03:12:39.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 -- start start 2022-07-20T03:12:39.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f96380b4f80 0x7f96380acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9638146770 0x7f963814c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f96380b46e0 con 0x7f96380b5860 2022-07-20T03:12:39.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380ad990 con 0x7f96380b4f80 2022-07-20T03:12:39.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f963d1dd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f96380b4b10 con 0x7f9638146770 2022-07-20T03:12:39.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.571+0000 7f96377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f96380b4f80 0x7f96380acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f96380b4f80 0x7f96380acae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45580/0 (socket says 172.21.15.61:45580) 2022-07-20T03:12:39.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 -- 172.21.15.61:0/1637273799 learned_addr learned my addr 172.21.15.61:0/1637273799 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:39.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963c9dc700 1 --2- 172.21.15.61:0/1637273799 >> v2:172.21.15.117:3300/0 conn(0x7f9638146770 0x7f963814c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 -- 172.21.15.61:0/1637273799 >> v2:172.21.15.117:3300/0 conn(0x7f9638146770 msgr2=0x7f963814c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:39.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 --2- 172.21.15.61:0/1637273799 >> v2:172.21.15.117:3300/0 conn(0x7f9638146770 0x7f963814c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 -- 172.21.15.61:0/1637273799 >> v1:172.21.15.117:6789/0 conn(0x7f96380b5860 legacy=0x7f9638146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 -- 172.21.15.61:0/1637273799 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f96380ad3d0 con 0x7f96380b4f80 2022-07-20T03:12:39.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f96377fe700 1 --2- 172.21.15.61:0/1637273799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96380b4f80 0x7f96380acae0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f96200032f0 tx=0x7f9620009290).ready entity=mon.0 client_cookie=8dc4a22019cf9a36 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:39.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f9636ffd700 1 -- 172.21.15.61:0/1637273799 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962000b870 con 0x7f96380b4f80 2022-07-20T03:12:39.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f9636ffd700 1 -- 172.21.15.61:0/1637273799 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f962000b9d0 con 0x7f96380b4f80 2022-07-20T03:12:39.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f9636ffd700 1 -- 172.21.15.61:0/1637273799 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f962000bb40 con 0x7f96380b4f80 2022-07-20T03:12:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 -- 172.21.15.61:0/1637273799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96380b4f80 msgr2=0x7f96380acae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1637273799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96380b4f80 0x7f96380acae0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f96200032f0 tx=0x7f9620009290).stop 2022-07-20T03:12:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 -- 172.21.15.61:0/1637273799 shutdown_connections 2022-07-20T03:12:39.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1637273799 >> v2:172.21.15.117:3300/0 conn(0x7f9638146770 0x7f963814c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1637273799 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96380b4f80 0x7f96380acae0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 -- 172.21.15.61:0/1637273799 >> 172.21.15.61:0/1637273799 conn(0x7f96380b5e10 msgr2=0x7f96380b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:39.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 -- 172.21.15.61:0/1637273799 shutdown_connections 2022-07-20T03:12:39.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.575+0000 7f963d1dd700 1 -- 172.21.15.61:0/1637273799 wait complete. 2022-07-20T03:12:39.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 Processor -- start 2022-07-20T03:12:39.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 -- start start 2022-07-20T03:12:39.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96380b4f80 0x7f96380b2930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96380b5860 0x7f96381552e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 0x7f9638114aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9638113b40 con 0x7f9638146770 2022-07-20T03:12:39.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96380afb50 con 0x7f96380b4f80 2022-07-20T03:12:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f96380afdf0 con 0x7f96380b5860 2022-07-20T03:12:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 0x7f9638114aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 0x7f9638114aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45590/0 (socket says 172.21.15.61:45590) 2022-07-20T03:12:39.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 -- 172.21.15.61:0/1940617477 learned_addr learned my addr 172.21.15.61:0/1940617477 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:39.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f9637fff700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96380b4f80 0x7f96380b2930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f96377fe700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96380b5860 0x7f96381552e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 -- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96380b5860 msgr2=0x7f96381552e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96380b5860 0x7f96381552e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 -- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96380b4f80 msgr2=0x7f96380b2930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:39.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96380b4f80 0x7f96380b2930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:39.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 -- 172.21.15.61:0/1940617477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9620009530 con 0x7f9638146770 2022-07-20T03:12:39.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f96377fe700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96380b5860 0x7f96381552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:39.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963c9dc700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 0x7f9638114aa0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f963000a000 tx=0x7f9630008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:39.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9630011780 con 0x7f9638146770 2022-07-20T03:12:39.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96300118e0 con 0x7f9638146770 2022-07-20T03:12:39.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9638114600 con 0x7f9638146770 2022-07-20T03:12:39.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9630011c40 con 0x7f9638146770 2022-07-20T03:12:39.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.579+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f96380af6b0 con 0x7f9638146770 2022-07-20T03:12:39.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.583+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9638114790 con 0x7f9638146770 2022-07-20T03:12:39.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.583+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9630012420 con 0x7f9638146770 2022-07-20T03:12:39.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.583+0000 7f96357fa700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f961c053a40 0x7f961c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:39.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.583+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(52..52 src has 1..52) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f963006bdb0 con 0x7f9638146770 2022-07-20T03:12:39.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.583+0000 7f9637fff700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f961c053a40 0x7f961c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:39.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.583+0000 7f9637fff700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f961c053a40 0x7f961c055f00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f962800d000 tx=0x7f9628000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:39.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.587+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9630070070 con 0x7f9638146770 2022-07-20T03:12:39.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:39.751+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f96380af0f0 con 0x7f9638146770 2022-07-20T03:12:40.211 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:12:40.175+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:12:40.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.183+0000 7f96357fa700 1 -- 172.21.15.61:0/1940617477 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v30) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9630042ba0 con 0x7f9638146770 2022-07-20T03:12:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f961c053a40 msgr2=0x7f961c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f961c053a40 0x7f961c055f00 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f962800d000 tx=0x7f9628000bc0).stop 2022-07-20T03:12:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 msgr2=0x7f9638114aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 0x7f9638114aa0 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7f963000a000 tx=0x7f9630008040).stop 2022-07-20T03:12:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 shutdown_connections 2022-07-20T03:12:40.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f961c053a40 0x7f961c055f00 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9638146770 0x7f9638114aa0 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96380b5860 0x7f96381552e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 --2- 172.21.15.61:0/1940617477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96380b4f80 0x7f96380b2930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 >> 172.21.15.61:0/1940617477 conn(0x7f96380b5e10 msgr2=0x7f963813fa50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:40.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 shutdown_connections 2022-07-20T03:12:40.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.191+0000 7f963d1dd700 1 -- 172.21.15.61:0/1940617477 wait complete. 2022-07-20T03:12:40.236 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:12:40.236 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:40.245 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:12:40.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 Processor -- start 2022-07-20T03:12:40.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 -- start start 2022-07-20T03:12:40.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17ac0b1ef0 0x7f17ac0605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac13f980 0x7f17ac0abff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f17ac0ac4e0 0x7f17ac0ab180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ac140940 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17ac0b22d0 con 0x7f17ac0ac4e0 2022-07-20T03:12:40.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b2e0f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac05fbc0 con 0x7f17ac13f980 2022-07-20T03:12:40.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17ac0b1ef0 0x7f17ac0605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17ac0b1ef0 0x7f17ac0605a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45606/0 (socket says 172.21.15.61:45606) 2022-07-20T03:12:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 -- 172.21.15.61:0/3451262636 learned_addr learned my addr 172.21.15.61:0/3451262636 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b260e700 1 --2- 172.21.15.61:0/3451262636 >> v2:172.21.15.117:3300/0 conn(0x7f17ac0ac4e0 0x7f17ac0ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b160c700 1 --2- 172.21.15.61:0/3451262636 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac13f980 0x7f17ac0abff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 -- 172.21.15.61:0/3451262636 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac13f980 msgr2=0x7f17ac0abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:40.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 --2- 172.21.15.61:0/3451262636 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac13f980 0x7f17ac0abff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 -- 172.21.15.61:0/3451262636 >> v2:172.21.15.117:3300/0 conn(0x7f17ac0ac4e0 msgr2=0x7f17ac0ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:40.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 --2- 172.21.15.61:0/3451262636 >> v2:172.21.15.117:3300/0 conn(0x7f17ac0ac4e0 0x7f17ac0ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 -- 172.21.15.61:0/3451262636 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17ac05fe60 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b1e0d700 1 --2- 172.21.15.61:0/3451262636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac0605a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f17a8002f70 tx=0x7f17a8006f40).ready entity=mon.0 client_cookie=86b44b15b28c0a6e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:40.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b0e0b700 1 -- 172.21.15.61:0/3451262636 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17a800d9a0 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b0e0b700 1 -- 172.21.15.61:0/3451262636 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17a8000e90 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.355+0000 7f17b0e0b700 1 -- 172.21.15.61:0/3451262636 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17a800dd30 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3451262636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 msgr2=0x7f17ac0605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3451262636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac0605a0 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7f17a8002f70 tx=0x7f17a8006f40).stop 2022-07-20T03:12:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3451262636 shutdown_connections 2022-07-20T03:12:40.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3451262636 >> v2:172.21.15.117:3300/0 conn(0x7f17ac0ac4e0 0x7f17ac0ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3451262636 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac13f980 0x7f17ac0abff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3451262636 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac0605a0 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3451262636 >> 172.21.15.61:0/3451262636 conn(0x7f17ac0b2ca0 msgr2=0x7f17ac0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3451262636 shutdown_connections 2022-07-20T03:12:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3451262636 wait complete. 2022-07-20T03:12:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 Processor -- start 2022-07-20T03:12:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- start start 2022-07-20T03:12:40.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac0ac4e0 0x7f17ac1435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac139620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17ac13f980 0x7f17ac139b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac13a000 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac1425a0 con 0x7f17ac13f980 2022-07-20T03:12:40.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b2e0f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17ac142840 con 0x7f17ac0ac4e0 2022-07-20T03:12:40.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b160c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac139620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b160c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac139620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45614/0 (socket says 172.21.15.61:45614) 2022-07-20T03:12:40.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.359+0000 7f17b160c700 1 -- 172.21.15.61:0/3314102081 learned_addr learned my addr 172.21.15.61:0/3314102081 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:40.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b260e700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17ac13f980 0x7f17ac139b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b1e0d700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac0ac4e0 0x7f17ac1435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b160c700 1 -- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac0ac4e0 msgr2=0x7f17ac1435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b160c700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac0ac4e0 0x7f17ac1435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b160c700 1 -- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17ac13f980 msgr2=0x7f17ac139b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b160c700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17ac13f980 0x7f17ac139b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b160c700 1 -- 172.21.15.61:0/3314102081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17a800b460 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b160c700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac139620 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f179c00c000 tx=0x7f179c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:40.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f179c011750 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f179c0118b0 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17ac142ae0 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f179c011c10 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17ac0ad640 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17ac142c70 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f179c011d70 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f179affd700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f178c053920 0x7f178c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f179c06c310 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.363+0000 7f17b1e0d700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f178c053920 0x7f178c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.367+0000 7f17b1e0d700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f178c053920 0x7f178c055de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f17a801b000 tx=0x7f17a8006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:40.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.371+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f179c043220 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.539+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f17ac060100 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.539+0000 7f179affd700 1 -- 172.21.15.61:0/3314102081 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 30 v30) v1 ==== 94+0+1778 (secure 0 0 0) 0x7f179c046840 con 0x7f17ac0b1ef0 2022-07-20T03:12:40.680 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:40.681 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":30,"default_fscid":4,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24}],"filesystems":[{"mdsmap":{"epoch":30,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:11:12.676790+0000","modified":"2022-07-20T03:12:40.181907+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":53,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[10],"metadata_pool":9,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":4}]} 2022-07-20T03:12:40.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f178c053920 msgr2=0x7f178c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f178c053920 0x7f178c055de0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f17a801b000 tx=0x7f17a8006f40).stop 2022-07-20T03:12:40.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 msgr2=0x7f17ac139620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac139620 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f179c00c000 tx=0x7f179c006df0).stop 2022-07-20T03:12:40.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 shutdown_connections 2022-07-20T03:12:40.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f178c053920 0x7f178c055de0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17ac13f980 0x7f17ac139b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17ac0b1ef0 0x7f17ac139620 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 --2- 172.21.15.61:0/3314102081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17ac0ac4e0 0x7f17ac1435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 >> 172.21.15.61:0/3314102081 conn(0x7f17ac0b2ca0 msgr2=0x7f17ac13ece0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:40.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 shutdown_connections 2022-07-20T03:12:40.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.543+0000 7f17b2e0f700 1 -- 172.21.15.61:0/3314102081 wait complete. 2022-07-20T03:12:40.751 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 30 2022-07-20T03:12:40.752 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:12:40.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 Processor -- start 2022-07-20T03:12:40.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 -- start start 2022-07-20T03:12:40.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d8c0b12e0 0x7f6d8c0b16c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6d8c0acd70 0x7f6d8c0af4e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c05bd90 0x7f6d8c0af9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8c0abd50 con 0x7f6d8c0b12e0 2022-07-20T03:12:40.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d8c0abff0 con 0x7f6d8c0acd70 2022-07-20T03:12:40.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d92169700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c0ac290 con 0x7f6d8c05bd90 2022-07-20T03:12:40.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d8b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d8c0b12e0 0x7f6d8c0b16c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d8b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d8c0b12e0 0x7f6d8c0b16c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45620/0 (socket says 172.21.15.61:45620) 2022-07-20T03:12:40.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/2716721692 learned_addr learned my addr 172.21.15.61:0/2716721692 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:40.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d8bfff700 1 --2- 172.21.15.61:0/2716721692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c05bd90 0x7f6d8c0af9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.871+0000 7f6d83fff700 1 --2- 172.21.15.61:0/2716721692 >> v2:172.21.15.117:3300/0 conn(0x7f6d8c0acd70 0x7f6d8c0af4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/2716721692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c05bd90 msgr2=0x7f6d8c0af9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:40.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8b7fe700 1 --2- 172.21.15.61:0/2716721692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c05bd90 0x7f6d8c0af9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/2716721692 >> v2:172.21.15.117:3300/0 conn(0x7f6d8c0acd70 msgr2=0x7f6d8c0af4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:40.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8b7fe700 1 --2- 172.21.15.61:0/2716721692 >> v2:172.21.15.117:3300/0 conn(0x7f6d8c0acd70 0x7f6d8c0af4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/2716721692 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d8c0ac530 con 0x7f6d8c0b12e0 2022-07-20T03:12:40.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8b7fe700 1 --2- 172.21.15.61:0/2716721692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0b12e0 0x7f6d8c0b16c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f6d7c002f70 tx=0x7f6d7c006eb0).ready entity=mon.0 client_cookie=aa320ecc500d66bc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:40.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8affd700 1 -- 172.21.15.61:0/2716721692 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7c00d570 con 0x7f6d8c0b12e0 2022-07-20T03:12:40.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8affd700 1 -- 172.21.15.61:0/2716721692 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c000e90 con 0x7f6d8c0b12e0 2022-07-20T03:12:40.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d8affd700 1 -- 172.21.15.61:0/2716721692 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7c00d900 con 0x7f6d8c0b12e0 2022-07-20T03:12:40.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 -- 172.21.15.61:0/2716721692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0b12e0 msgr2=0x7f6d8c0b16c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 --2- 172.21.15.61:0/2716721692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0b12e0 0x7f6d8c0b16c0 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7f6d7c002f70 tx=0x7f6d7c006eb0).stop 2022-07-20T03:12:40.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 -- 172.21.15.61:0/2716721692 shutdown_connections 2022-07-20T03:12:40.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 --2- 172.21.15.61:0/2716721692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c05bd90 0x7f6d8c0af9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 --2- 172.21.15.61:0/2716721692 >> v2:172.21.15.117:3300/0 conn(0x7f6d8c0acd70 0x7f6d8c0af4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 --2- 172.21.15.61:0/2716721692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0b12e0 0x7f6d8c0b16c0 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 -- 172.21.15.61:0/2716721692 >> 172.21.15.61:0/2716721692 conn(0x7f6d8c0b23d0 msgr2=0x7f6d8c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:40.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 -- 172.21.15.61:0/2716721692 shutdown_connections 2022-07-20T03:12:40.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.875+0000 7f6d92169700 1 -- 172.21.15.61:0/2716721692 wait complete. 2022-07-20T03:12:40.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 Processor -- start 2022-07-20T03:12:40.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 -- start start 2022-07-20T03:12:40.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 0x7f6d8c0b4a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 0x7f6d8c1526f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c0b12e0 0x7f6d8c152be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:40.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c08f610 con 0x7f6d8c0acd70 2022-07-20T03:12:40.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c093880 con 0x7f6d8c05bd90 2022-07-20T03:12:40.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d8c08f840 con 0x7f6d8c0b12e0 2022-07-20T03:12:40.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 0x7f6d8c1526f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 0x7f6d8c0b4a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 0x7f6d8c1526f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45628/0 (socket says 172.21.15.61:45628) 2022-07-20T03:12:40.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 0x7f6d8c0b4a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55306/0 (socket says 172.21.15.61:55306) 2022-07-20T03:12:40.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d83fff700 1 -- 172.21.15.61:0/3592837875 learned_addr learned my addr 172.21.15.61:0/3592837875 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:40.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c0b12e0 0x7f6d8c152be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:40.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c0b12e0 msgr2=0x7f6d8c152be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c0b12e0 0x7f6d8c152be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/3592837875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 msgr2=0x7f6d8c1526f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:40.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 0x7f6d8c1526f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:40.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 -- 172.21.15.61:0/3592837875 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d7c00b1e0 con 0x7f6d8c05bd90 2022-07-20T03:12:40.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d83fff700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 0x7f6d8c1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:40.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d8b7fe700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 0x7f6d8c0b4a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6d7c002f70 tx=0x7f6d7c015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:40.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7c011760 con 0x7f6d8c05bd90 2022-07-20T03:12:40.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d7c0118c0 con 0x7f6d8c05bd90 2022-07-20T03:12:40.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d8c093b10 con 0x7f6d8c05bd90 2022-07-20T03:12:40.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d7c011c20 con 0x7f6d8c05bd90 2022-07-20T03:12:40.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.879+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d8c091980 con 0x7f6d8c05bd90 2022-07-20T03:12:40.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.883+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d8c08f980 con 0x7f6d8c05bd90 2022-07-20T03:12:41.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.883+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6d7c01a040 con 0x7f6d8c05bd90 2022-07-20T03:12:41.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.883+0000 7f6d897fa700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d70053ad0 0x7f6d70055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.883+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f6d7c06cc30 con 0x7f6d8c05bd90 2022-07-20T03:12:41.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.883+0000 7f6d83fff700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d70053ad0 0x7f6d70055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.883+0000 7f6d83fff700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d70053ad0 0x7f6d70055f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f6d7400b000 tx=0x7f6d74006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:41.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:40.891+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d8c08f980 con 0x7f6d8c05bd90 2022-07-20T03:12:41.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.059+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f6d8c08f980 con 0x7f6d8c05bd90 2022-07-20T03:12:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.203+0000 7f6d897fa700 1 -- 172.21.15.61:0/3592837875 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v31) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6d8c08f980 con 0x7f6d8c05bd90 2022-07-20T03:12:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d70053ad0 msgr2=0x7f6d70055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d70053ad0 0x7f6d70055f90 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f6d7400b000 tx=0x7f6d74006df0).stop 2022-07-20T03:12:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 msgr2=0x7f6d8c0b4a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 0x7f6d8c0b4a90 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f6d7c002f70 tx=0x7f6d7c015000).stop 2022-07-20T03:12:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 shutdown_connections 2022-07-20T03:12:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d70053ad0 0x7f6d70055f90 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d8c0b12e0 0x7f6d8c152be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d8c0acd70 0x7f6d8c1526f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 --2- 172.21.15.61:0/3592837875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d8c05bd90 0x7f6d8c0b4a90 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 >> 172.21.15.61:0/3592837875 conn(0x7f6d8c0b23d0 msgr2=0x7f6d8c13fc20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:41.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 shutdown_connections 2022-07-20T03:12:41.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.207+0000 7f6d92169700 1 -- 172.21.15.61:0/3592837875 wait complete. 2022-07-20T03:12:41.230 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:12:41.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 Processor -- start 2022-07-20T03:12:41.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 -- start start 2022-07-20T03:12:41.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3080633b0 0x7fc30805c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc30805c990 0x7fc308145fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3080b6db0 con 0x7fc3080b69d0 2022-07-20T03:12:41.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc308063880 con 0x7fc30805c990 2022-07-20T03:12:41.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30fff8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc308063b20 con 0x7fc3080633b0 2022-07-20T03:12:41.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30e595700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc30805c990 0x7fc308145fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30dd94700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc3080b69d0 0x7fc30805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42232/0 (socket says 172.21.15.61:42232) 2022-07-20T03:12:41.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30dd94700 1 -- 172.21.15.61:0/3445621946 learned_addr learned my addr 172.21.15.61:0/3445621946 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:41.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30d593700 1 --2- 172.21.15.61:0/3445621946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3080633b0 0x7fc30805c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30e595700 1 -- 172.21.15.61:0/3445621946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3080633b0 msgr2=0x7fc30805c4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30e595700 1 --2- 172.21.15.61:0/3445621946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3080633b0 0x7fc30805c4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30e595700 1 -- 172.21.15.61:0/3445621946 >> v1:172.21.15.61:6789/0 conn(0x7fc3080b69d0 legacy=0x7fc30805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30cd92700 1 -- 172.21.15.61:0/3445621946 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3526041246 0 0) 0x7fc3080b6db0 con 0x7fc3080b69d0 2022-07-20T03:12:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30e595700 1 -- 172.21.15.61:0/3445621946 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc308063dc0 con 0x7fc30805c990 2022-07-20T03:12:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.379+0000 7fc30e595700 1 --2- 172.21.15.61:0/3445621946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc30805c990 0x7fc308145fa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fc3000032f0 tx=0x7fc3000091d0).ready entity=mon.0 client_cookie=b150c9ee98b22d83 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30cd92700 1 -- 172.21.15.61:0/3445621946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc30000b6c0 con 0x7fc30805c990 2022-07-20T03:12:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30cd92700 1 -- 172.21.15.61:0/3445621946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc30000b820 con 0x7fc30805c990 2022-07-20T03:12:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30cd92700 1 -- 172.21.15.61:0/3445621946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc30000b990 con 0x7fc30805c990 2022-07-20T03:12:41.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 -- 172.21.15.61:0/3445621946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc30805c990 msgr2=0x7fc308145fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:41.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3445621946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc30805c990 0x7fc308145fa0 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fc3000032f0 tx=0x7fc3000091d0).stop 2022-07-20T03:12:41.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 -- 172.21.15.61:0/3445621946 shutdown_connections 2022-07-20T03:12:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3445621946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc30805c990 0x7fc308145fa0 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3445621946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3080633b0 0x7fc30805c4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 -- 172.21.15.61:0/3445621946 >> 172.21.15.61:0/3445621946 conn(0x7fc3080b9f70 msgr2=0x7fc3080b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:41.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 -- 172.21.15.61:0/3445621946 shutdown_connections 2022-07-20T03:12:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 -- 172.21.15.61:0/3445621946 wait complete. 2022-07-20T03:12:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.383+0000 7fc30fff8700 1 Processor -- start 2022-07-20T03:12:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 -- start start 2022-07-20T03:12:41.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc30805c990 0x7fc3081491c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 0x7fc30814a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3080b69d0 0x7fc308144de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc308143bc0 con 0x7fc3080633b0 2022-07-20T03:12:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc30814b070 con 0x7fc3080b69d0 2022-07-20T03:12:41.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc30814b310 con 0x7fc30805c990 2022-07-20T03:12:41.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 0x7fc30814a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 0x7fc30814a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45642/0 (socket says 172.21.15.61:45642) 2022-07-20T03:12:41.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30dd94700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc30805c990 0x7fc3081491c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 -- 172.21.15.61:0/3636600167 learned_addr learned my addr 172.21.15.61:0/3636600167 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30e595700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3080b69d0 0x7fc308144de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 -- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc30805c990 msgr2=0x7fc3081491c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc30805c990 0x7fc3081491c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 -- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3080b69d0 msgr2=0x7fc308144de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3080b69d0 0x7fc308144de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 -- 172.21.15.61:0/3636600167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc300009470 con 0x7fc3080633b0 2022-07-20T03:12:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30d593700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 0x7fc30814a9b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fc2fc00c000 tx=0x7fc2fc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:41.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2fc010780 con 0x7fc3080633b0 2022-07-20T03:12:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc2fc0108e0 con 0x7fc3080633b0 2022-07-20T03:12:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc30814a1c0 con 0x7fc3080633b0 2022-07-20T03:12:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc2fc010c40 con 0x7fc3080633b0 2022-07-20T03:12:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.387+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc308143930 con 0x7fc3080633b0 2022-07-20T03:12:41.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.391+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3080574e0 con 0x7fc3080633b0 2022-07-20T03:12:41.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.391+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc2fc01c460 con 0x7fc3080633b0 2022-07-20T03:12:41.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.391+0000 7fc2faffd700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc2e8053ad0 0x7fc2e8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:41.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.391+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(53..53 src has 1..53) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fc2fc06c370 con 0x7fc3080633b0 2022-07-20T03:12:41.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.391+0000 7fc30dd94700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc2e8053ad0 0x7fc2e8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:41.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.391+0000 7fc30dd94700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc2e8053ad0 0x7fc2e8055f90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fc30400d000 tx=0x7fc304007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:41.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.399+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc2fc043180 con 0x7fc3080633b0 2022-07-20T03:12:41.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:41.527+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fc3080ba380 con 0x7fc3080633b0 2022-07-20T03:12:42.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.215+0000 7fc2faffd700 1 -- 172.21.15.61:0/3636600167 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v54) v1 ==== 185+0+0 (secure 0 0 0) 0x7fc2fc0467a0 con 0x7fc3080633b0 2022-07-20T03:12:42.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc2e8053ad0 msgr2=0x7fc2e8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:42.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc2e8053ad0 0x7fc2e8055f90 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7fc30400d000 tx=0x7fc304007c20).stop 2022-07-20T03:12:42.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 msgr2=0x7fc30814a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 0x7fc30814a9b0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7fc2fc00c000 tx=0x7fc2fc006df0).stop 2022-07-20T03:12:42.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 shutdown_connections 2022-07-20T03:12:42.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc2e8053ad0 0x7fc2e8055f90 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3080b69d0 0x7fc308144de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3080633b0 0x7fc30814a9b0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 --2- 172.21.15.61:0/3636600167 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc30805c990 0x7fc3081491c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 >> 172.21.15.61:0/3636600167 conn(0x7fc3080b9f70 msgr2=0x7fc308066e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:42.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 shutdown_connections 2022-07-20T03:12:42.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.223+0000 7fc30fff8700 1 -- 172.21.15.61:0/3636600167 wait complete. 2022-07-20T03:12:42.277 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:12:42.294 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:12:42.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.419+0000 7ff3c9d94700 1 Processor -- start 2022-07-20T03:12:42.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.419+0000 7ff3c9d94700 1 -- start start 2022-07-20T03:12:42.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff3c40b3cf0 0x7ff3c4146040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:42.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c4146530 0x7ff3c4141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:42.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3c4066b10 con 0x7ff3c40b6840 2022-07-20T03:12:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3c40b5c90 con 0x7ff3c40b3cf0 2022-07-20T03:12:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c40b6c20 con 0x7ff3c4146530 2022-07-20T03:12:42.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c8d92700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff3c40b6840 0x7ff3c40b5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42246/0 (socket says 172.21.15.61:42246) 2022-07-20T03:12:42.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c8d92700 1 -- 172.21.15.61:0/12944857 learned_addr learned my addr 172.21.15.61:0/12944857 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:42.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9593700 1 --2- 172.21.15.61:0/12944857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c4146530 0x7ff3c4141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:42.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c3fff700 1 --2- 172.21.15.61:0/12944857 >> v2:172.21.15.117:3300/0 conn(0x7ff3c40b3cf0 0x7ff3c4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:42.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c37fe700 1 -- 172.21.15.61:0/12944857 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 240176754 0 0) 0x7ff3c4066b10 con 0x7ff3c40b6840 2022-07-20T03:12:42.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c37fe700 1 -- 172.21.15.61:0/12944857 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff3a4003260 con 0x7ff3c40b6840 2022-07-20T03:12:42.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9593700 1 -- 172.21.15.61:0/12944857 >> v2:172.21.15.117:3300/0 conn(0x7ff3c40b3cf0 msgr2=0x7ff3c4146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:42.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9593700 1 --2- 172.21.15.61:0/12944857 >> v2:172.21.15.117:3300/0 conn(0x7ff3c40b3cf0 0x7ff3c4146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9593700 1 -- 172.21.15.61:0/12944857 >> v1:172.21.15.61:6789/0 conn(0x7ff3c40b6840 legacy=0x7ff3c40b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:42.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9593700 1 -- 172.21.15.61:0/12944857 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3c40aead0 con 0x7ff3c4146530 2022-07-20T03:12:42.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c3fff700 1 --2- 172.21.15.61:0/12944857 >> v2:172.21.15.117:3300/0 conn(0x7ff3c40b3cf0 0x7ff3c4146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:42.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9593700 1 --2- 172.21.15.61:0/12944857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c4146530 0x7ff3c4141a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff3bc0032f0 tx=0x7ff3bc0091d0).ready entity=mon.2 client_cookie=5f98833a2d3e4711 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:42.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c37fe700 1 -- 172.21.15.61:0/12944857 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3bc00b600 con 0x7ff3c4146530 2022-07-20T03:12:42.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c37fe700 1 -- 172.21.15.61:0/12944857 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3bc00b760 con 0x7ff3c4146530 2022-07-20T03:12:42.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c37fe700 1 -- 172.21.15.61:0/12944857 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3bc00b8d0 con 0x7ff3c4146530 2022-07-20T03:12:42.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- 172.21.15.61:0/12944857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c4146530 msgr2=0x7ff3c4141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:42.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/12944857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c4146530 0x7ff3c4141a90 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7ff3bc0032f0 tx=0x7ff3bc0091d0).stop 2022-07-20T03:12:42.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- 172.21.15.61:0/12944857 shutdown_connections 2022-07-20T03:12:42.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/12944857 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c4146530 0x7ff3c4141a90 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/12944857 >> v2:172.21.15.117:3300/0 conn(0x7ff3c40b3cf0 0x7ff3c4146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- 172.21.15.61:0/12944857 >> 172.21.15.61:0/12944857 conn(0x7ff3c40b9e10 msgr2=0x7ff3c40b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:42.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- 172.21.15.61:0/12944857 shutdown_connections 2022-07-20T03:12:42.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.423+0000 7ff3c9d94700 1 -- 172.21.15.61:0/12944857 wait complete. 2022-07-20T03:12:42.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 Processor -- start 2022-07-20T03:12:42.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 -- start start 2022-07-20T03:12:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c40b3cf0 0x7ff3c41435c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:42.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 0x7ff3c40aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3c4146530 0x7ff3c4139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c4137cd0 con 0x7ff3c40b6840 2022-07-20T03:12:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c40b03f0 con 0x7ff3c4146530 2022-07-20T03:12:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9d94700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3c40b1fb0 con 0x7ff3c40b3cf0 2022-07-20T03:12:42.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 0x7ff3c40aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:42.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 0x7ff3c40aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45656/0 (socket says 172.21.15.61:45656) 2022-07-20T03:12:42.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 -- 172.21.15.61:0/2193406390 learned_addr learned my addr 172.21.15.61:0/2193406390 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:42.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9593700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3c4146530 0x7ff3c4139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:42.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c8d92700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c40b3cf0 0x7ff3c41435c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:42.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 -- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c40b3cf0 msgr2=0x7ff3c41435c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:42.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c40b3cf0 0x7ff3c41435c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 -- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3c4146530 msgr2=0x7ff3c4139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3c4146530 0x7ff3c4139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:42.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 -- 172.21.15.61:0/2193406390 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3ac00b180 con 0x7ff3c40b6840 2022-07-20T03:12:42.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c9593700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3c4146530 0x7ff3c4139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:42.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.427+0000 7ff3c3fff700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 0x7ff3c40aff00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7ff3ac00c000 tx=0x7ff3ac006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:42.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3ac010780 con 0x7ff3c40b6840 2022-07-20T03:12:42.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff3ac0108e0 con 0x7ff3c40b6840 2022-07-20T03:12:42.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3bc009470 con 0x7ff3c40b6840 2022-07-20T03:12:42.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3ac010c40 con 0x7ff3c40b6840 2022-07-20T03:12:42.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3c40b29c0 con 0x7ff3c40b6840 2022-07-20T03:12:42.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3c40aed70 con 0x7ff3c40b6840 2022-07-20T03:12:42.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff3ac01c460 con 0x7ff3c40b6840 2022-07-20T03:12:42.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c1ffb700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3a8053940 0x7ff3a8055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:42.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(54..54 src has 1..54) v4 ==== 5721+0+0 (secure 0 0 0) 0x7ff3ac06c210 con 0x7ff3c40b6840 2022-07-20T03:12:42.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.431+0000 7ff3c8d92700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3a8053940 0x7ff3a8055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:42.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.435+0000 7ff3c8d92700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3a8053940 0x7ff3a8055e00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff3b400d000 tx=0x7ff3b4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:42.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.439+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff3ac043270 con 0x7ff3c40b6840 2022-07-20T03:12:42.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:42.563+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff3c40574e0 con 0x7ff3c40b6840 2022-07-20T03:12:43.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.227+0000 7ff3c1ffb700 1 -- 172.21.15.61:0/2193406390 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v55) v1 ==== 173+0+0 (secure 0 0 0) 0x7ff3ac046890 con 0x7ff3c40b6840 2022-07-20T03:12:43.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3a8053940 msgr2=0x7ff3a8055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3a8053940 0x7ff3a8055e00 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff3b400d000 tx=0x7ff3b4009040).stop 2022-07-20T03:12:43.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 msgr2=0x7ff3c40aff00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 0x7ff3c40aff00 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7ff3ac00c000 tx=0x7ff3ac006df0).stop 2022-07-20T03:12:43.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 shutdown_connections 2022-07-20T03:12:43.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff3a8053940 0x7ff3a8055e00 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3c4146530 0x7ff3c4139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3c40b6840 0x7ff3c40aff00 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 --2- 172.21.15.61:0/2193406390 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3c40b3cf0 0x7ff3c41435c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 >> 172.21.15.61:0/2193406390 conn(0x7ff3c40b9e10 msgr2=0x7ff3c40b5dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:43.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 shutdown_connections 2022-07-20T03:12:43.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.231+0000 7ff3c9d94700 1 -- 172.21.15.61:0/2193406390 wait complete. 2022-07-20T03:12:43.257 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:12:43.260 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:12:43.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 Processor -- start 2022-07-20T03:12:43.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 -- start start 2022-07-20T03:12:43.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1080b4bc0 0x7fa10814d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa108066b10 con 0x7fa1080b3cf0 2022-07-20T03:12:43.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1080b6c20 con 0x7fa1080b6840 2022-07-20T03:12:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1080b5c90 con 0x7fa1080b4bc0 2022-07-20T03:12:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1080b4bc0 0x7fa10814d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1080b4bc0 0x7fa10814d610 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45666/0 (socket says 172.21.15.61:45666) 2022-07-20T03:12:43.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 -- 172.21.15.61:0/2625586043 learned_addr learned my addr 172.21.15.61:0/2625586043 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:43.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1067fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa1080b3cf0 0x7fa108146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42262/0 (socket says 172.21.15.61:42262) 2022-07-20T03:12:43.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 -- 172.21.15.61:0/2625586043 >> v1:172.21.15.117:6789/0 conn(0x7fa1080b6840 legacy=0x7fa1080b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 -- 172.21.15.61:0/2625586043 >> v1:172.21.15.61:6789/0 conn(0x7fa1080b3cf0 legacy=0x7fa108146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 -- 172.21.15.61:0/2625586043 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1080aead0 con 0x7fa1080b4bc0 2022-07-20T03:12:43.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa1077fe700 1 --2- 172.21.15.61:0/2625586043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b4bc0 0x7fa10814d610 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fa1000032f0 tx=0x7fa100009290).ready entity=mon.0 client_cookie=50633e7185a57a5d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:43.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa105ffb700 1 -- 172.21.15.61:0/2625586043 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa10000b8a0 con 0x7fa1080b4bc0 2022-07-20T03:12:43.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa105ffb700 1 -- 172.21.15.61:0/2625586043 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa10000ba00 con 0x7fa1080b4bc0 2022-07-20T03:12:43.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa105ffb700 1 -- 172.21.15.61:0/2625586043 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa10000bb70 con 0x7fa1080b4bc0 2022-07-20T03:12:43.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 -- 172.21.15.61:0/2625586043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b4bc0 msgr2=0x7fa10814d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.403+0000 7fa107fff700 1 --2- 172.21.15.61:0/2625586043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b4bc0 0x7fa10814d610 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fa1000032f0 tx=0x7fa100009290).stop 2022-07-20T03:12:43.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- 172.21.15.61:0/2625586043 shutdown_connections 2022-07-20T03:12:43.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 --2- 172.21.15.61:0/2625586043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b4bc0 0x7fa10814d610 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- 172.21.15.61:0/2625586043 >> 172.21.15.61:0/2625586043 conn(0x7fa1080b9e10 msgr2=0x7fa1080b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:43.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- 172.21.15.61:0/2625586043 shutdown_connections 2022-07-20T03:12:43.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- 172.21.15.61:0/2625586043 wait complete. 2022-07-20T03:12:43.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 Processor -- start 2022-07-20T03:12:43.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- start start 2022-07-20T03:12:43.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 0x7fa10813ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1080b4bc0 0x7fa108151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1080b6840 0x7fa108147f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa108148a40 con 0x7fa1080b3cf0 2022-07-20T03:12:43.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1081487a0 con 0x7fa1080b4bc0 2022-07-20T03:12:43.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa108155580 con 0x7fa1080b6840 2022-07-20T03:12:43.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 0x7fa10813ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 0x7fa10813ee20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45668/0 (socket says 172.21.15.61:45668) 2022-07-20T03:12:43.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 -- 172.21.15.61:0/892803758 learned_addr learned my addr 172.21.15.61:0/892803758 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:43.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa1067fc700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1080b4bc0 0x7fa108151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa1077fe700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1080b6840 0x7fa108147f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 -- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1080b6840 msgr2=0x7fa108147f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1080b6840 0x7fa108147f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 -- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1080b4bc0 msgr2=0x7fa108151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1080b4bc0 0x7fa108151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 -- 172.21.15.61:0/892803758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa100009530 con 0x7fa1080b3cf0 2022-07-20T03:12:43.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa1077fe700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1080b6840 0x7fa108147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:43.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa106ffd700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 0x7fa10813ee20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fa108067e80 tx=0x7fa0f0007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:43.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0f0007030 con 0x7fa1080b3cf0 2022-07-20T03:12:43.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa0f0007190 con 0x7fa1080b3cf0 2022-07-20T03:12:43.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa0f0006510 con 0x7fa1080b3cf0 2022-07-20T03:12:43.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa108152330 con 0x7fa1080b3cf0 2022-07-20T03:12:43.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.407+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1080b0d30 con 0x7fa1080b3cf0 2022-07-20T03:12:43.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.411+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa108155740 con 0x7fa1080b3cf0 2022-07-20T03:12:43.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.411+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa0f0010030 con 0x7fa1080b3cf0 2022-07-20T03:12:43.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.411+0000 7fa0fffff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa0ec053a40 0x7fa0ec055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.411+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa0f006b7a0 con 0x7fa1080b3cf0 2022-07-20T03:12:43.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.411+0000 7fa1067fc700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa0ec053a40 0x7fa0ec055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.411+0000 7fa1067fc700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa0ec053a40 0x7fa0ec055f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fa0f800d000 tx=0x7fa0f8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:43.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.415+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa0f00429e0 con 0x7fa1080b3cf0 2022-07-20T03:12:43.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.539+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa1080aed70 con 0x7fa1080b3cf0 2022-07-20T03:12:43.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.547+0000 7fa0fffff700 1 -- 172.21.15.61:0/892803758 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v7) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa0f0046000 con 0x7fa1080b3cf0 2022-07-20T03:12:43.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa0ec053a40 msgr2=0x7fa0ec055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa0ec053a40 0x7fa0ec055f00 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fa0f800d000 tx=0x7fa0f8009040).stop 2022-07-20T03:12:43.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 msgr2=0x7fa10813ee20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 0x7fa10813ee20 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7fa108067e80 tx=0x7fa0f0007c10).stop 2022-07-20T03:12:43.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 shutdown_connections 2022-07-20T03:12:43.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa0ec053a40 0x7fa0ec055f00 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1080b4bc0 0x7fa108151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1080b6840 0x7fa108147f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 --2- 172.21.15.61:0/892803758 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1080b3cf0 0x7fa10813ee20 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 >> 172.21.15.61:0/892803758 conn(0x7fa1080b9e10 msgr2=0x7fa1080b81e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:43.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 shutdown_connections 2022-07-20T03:12:43.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.551+0000 7fa107fff700 1 -- 172.21.15.61:0/892803758 wait complete. 2022-07-20T03:12:43.645 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout' 2022-07-20T03:12:43.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 Processor -- start 2022-07-20T03:12:43.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 -- start start 2022-07-20T03:12:43.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77000adff0 0x7f77000b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000af8e0 0x7f770005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7700147010 con 0x7f77000b5650 2022-07-20T03:12:43.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77001472b0 con 0x7f77000adff0 2022-07-20T03:12:43.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7700146d40 con 0x7f77000af8e0 2022-07-20T03:12:43.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77000adff0 0x7f77000b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77000adff0 0x7f77000b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45678/0 (socket says 172.21.15.61:45678) 2022-07-20T03:12:43.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3937554192 learned_addr learned my addr 172.21.15.61:0/3937554192 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:43.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76fed9d700 1 --2- 172.21.15.61:0/3937554192 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000af8e0 0x7f770005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3937554192 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000af8e0 msgr2=0x7f770005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:43.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 --2- 172.21.15.61:0/3937554192 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000af8e0 0x7f770005bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3937554192 >> v1:172.21.15.117:6789/0 conn(0x7f77000b5650 legacy=0x7f7700147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3937554192 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77000acae0 con 0x7f77000adff0 2022-07-20T03:12:43.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76f5d9b700 1 --2- 172.21.15.61:0/3937554192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000adff0 0x7f77000b5a30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8006eb0).ready entity=mon.0 client_cookie=b10e68e68ddb08ef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:43.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76fdd9b700 1 -- 172.21.15.61:0/3937554192 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e800d5c0 con 0x7f77000adff0 2022-07-20T03:12:43.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76fdd9b700 1 -- 172.21.15.61:0/3937554192 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e8000e90 con 0x7f77000adff0 2022-07-20T03:12:43.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76fdd9b700 1 -- 172.21.15.61:0/3937554192 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e800d950 con 0x7f77000adff0 2022-07-20T03:12:43.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 -- 172.21.15.61:0/3937554192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000adff0 msgr2=0x7f77000b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.763+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3937554192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000adff0 0x7f77000b5a30 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8006eb0).stop 2022-07-20T03:12:43.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- 172.21.15.61:0/3937554192 shutdown_connections 2022-07-20T03:12:43.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3937554192 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000af8e0 0x7f770005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3937554192 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000adff0 0x7f77000b5a30 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- 172.21.15.61:0/3937554192 >> 172.21.15.61:0/3937554192 conn(0x7f77000634e0 msgr2=0x7f77000b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:43.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- 172.21.15.61:0/3937554192 shutdown_connections 2022-07-20T03:12:43.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- 172.21.15.61:0/3937554192 wait complete. 2022-07-20T03:12:43.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 Processor -- start 2022-07-20T03:12:43.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- start start 2022-07-20T03:12:43.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77000adff0 0x7f77000674e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 0x7f770010e3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000b5650 0x7f7700111090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700111580 con 0x7f77000af8e0 2022-07-20T03:12:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700066310 con 0x7f77000adff0 2022-07-20T03:12:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76ff59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7700063d20 con 0x7f77000b5650 2022-07-20T03:12:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76f5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 0x7f770010e3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76f5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 0x7f770010e3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45684/0 (socket says 172.21.15.61:45684) 2022-07-20T03:12:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3473182149 learned_addr learned my addr 172.21.15.61:0/3473182149 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76fe59c700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77000adff0 0x7f77000674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.767+0000 7f76fed9d700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000b5650 0x7f7700111090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000b5650 msgr2=0x7f7700111090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f5d9b700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000b5650 0x7f7700111090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77000adff0 msgr2=0x7f77000674e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f5d9b700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77000adff0 0x7f77000674e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f5d9b700 1 -- 172.21.15.61:0/3473182149 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e800b1e0 con 0x7f77000af8e0 2022-07-20T03:12:43.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f5d9b700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 0x7f770010e3d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:43.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8011710 con 0x7f77000af8e0 2022-07-20T03:12:43.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76e8011870 con 0x7f77000af8e0 2022-07-20T03:12:43.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7700111cc0 con 0x7f77000af8e0 2022-07-20T03:12:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f76e8011bd0 con 0x7f77000af8e0 2022-07-20T03:12:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7700064360 con 0x7f77000af8e0 2022-07-20T03:12:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f76e8025030 con 0x7f77000af8e0 2022-07-20T03:12:43.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7700063ee0 con 0x7f77000af8e0 2022-07-20T03:12:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f7fff700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76e4053a40 0x7f76e4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.771+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f76e806cc10 con 0x7f77000af8e0 2022-07-20T03:12:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.775+0000 7f76fe59c700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76e4053a40 0x7f76e4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:43.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.775+0000 7f76fe59c700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76e4053a40 0x7f76e4055f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f76f000d000 tx=0x7f76f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:43.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.779+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f76e8043e50 con 0x7f77000af8e0 2022-07-20T03:12:43.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:43.947+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]} v 0) v1 -- 0x7f77000574e0 con 0x7f77000af8e0 2022-07-20T03:12:44.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.227+0000 7f76f7fff700 1 -- 172.21.15.61:0/3473182149 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_dead_timeout"]}]=0 v246) v1 ==== 152+0+0 (secure 0 0 0) 0x7f76e8023020 con 0x7f77000af8e0 2022-07-20T03:12:44.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.231+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76e4053a40 msgr2=0x7f76e4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:44.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.231+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76e4053a40 0x7f76e4055f00 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f76f000d000 tx=0x7f76f0009040).stop 2022-07-20T03:12:44.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.231+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 msgr2=0x7f770010e3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:44.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.231+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 0x7f770010e3d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7f76e8002f70 tx=0x7f76e8015000).stop 2022-07-20T03:12:44.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 shutdown_connections 2022-07-20T03:12:44.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76e4053a40 0x7f76e4055f00 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77000b5650 0x7f7700111090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77000af8e0 0x7f770010e3d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 --2- 172.21.15.61:0/3473182149 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77000adff0 0x7f77000674e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 >> 172.21.15.61:0/3473182149 conn(0x7f77000634e0 msgr2=0x7f7700146350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:44.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 shutdown_connections 2022-07-20T03:12:44.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.235+0000 7f76ff59e700 1 -- 172.21.15.61:0/3473182149 wait complete. 2022-07-20T03:12:44.264 INFO:tasks.cephfs_test_runner:test_drop_cache_command_dead_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T03:12:44.265 INFO:tasks.cephfs_test_runner:Starting test: test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) 2022-07-20T03:12:44.266 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-20T03:12:44.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.367+0000 7f4db6922700 1 Processor -- start 2022-07-20T03:12:44.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.367+0000 7f4db6922700 1 -- start start 2022-07-20T03:12:44.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4db00b2530 0x7f4db0141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:44.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00acdd0 0x7f4db01424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:44.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4db00b2910 con 0x7f4db00614b0 2022-07-20T03:12:44.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4db00abb60 con 0x7f4db00b2530 2022-07-20T03:12:44.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db00abe00 con 0x7f4db00acdd0 2022-07-20T03:12:44.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4db00b2530 0x7f4db0141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:44.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4db00b2530 0x7f4db0141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45690/0 (socket says 172.21.15.61:45690) 2022-07-20T03:12:44.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/2711734002 learned_addr learned my addr 172.21.15.61:0/2711734002 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:44.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4da7fff700 1 --2- 172.21.15.61:0/2711734002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00acdd0 0x7f4db01424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/2711734002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00acdd0 msgr2=0x7f4db01424b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- 172.21.15.61:0/2711734002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00acdd0 0x7f4db01424b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/2711734002 >> v1:172.21.15.117:6789/0 conn(0x7f4db00614b0 legacy=0x7f4db0146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:44.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/2711734002 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4db01429a0 con 0x7f4db00b2530 2022-07-20T03:12:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- 172.21.15.61:0/2711734002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00b2530 0x7f4db0141fc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f4da0002f70 tx=0x7f4da0006eb0).ready entity=mon.0 client_cookie=897add129ca6b769 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daf7fe700 1 -- 172.21.15.61:0/2711734002 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4da000d570 con 0x7f4db00b2530 2022-07-20T03:12:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daf7fe700 1 -- 172.21.15.61:0/2711734002 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da0000e90 con 0x7f4db00b2530 2022-07-20T03:12:44.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daf7fe700 1 -- 172.21.15.61:0/2711734002 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4da000d900 con 0x7f4db00b2530 2022-07-20T03:12:44.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- 172.21.15.61:0/2711734002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00b2530 msgr2=0x7f4db0141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:44.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- 172.21.15.61:0/2711734002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00b2530 0x7f4db0141fc0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f4da0002f70 tx=0x7f4da0006eb0).stop 2022-07-20T03:12:44.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- 172.21.15.61:0/2711734002 shutdown_connections 2022-07-20T03:12:44.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- 172.21.15.61:0/2711734002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00acdd0 0x7f4db01424b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- 172.21.15.61:0/2711734002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00b2530 0x7f4db0141fc0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- 172.21.15.61:0/2711734002 >> 172.21.15.61:0/2711734002 conn(0x7f4db00b36b0 msgr2=0x7f4db00b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- 172.21.15.61:0/2711734002 shutdown_connections 2022-07-20T03:12:44.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- 172.21.15.61:0/2711734002 wait complete. 2022-07-20T03:12:44.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 Processor -- start 2022-07-20T03:12:44.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- start start 2022-07-20T03:12:44.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 0x7f4db0098720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:44.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4db00acdd0 0x7f4db00af820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:44.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00b2530 0x7f4db01527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:44.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db0097290 con 0x7f4db00614b0 2022-07-20T03:12:44.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4db009ac10 con 0x7f4db00acdd0 2022-07-20T03:12:44.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db6922700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4db009aeb0 con 0x7f4db00b2530 2022-07-20T03:12:44.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 0x7f4db0098720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:44.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 0x7f4db0098720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45696/0 (socket says 172.21.15.61:45696) 2022-07-20T03:12:44.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/523114786 learned_addr learned my addr 172.21.15.61:0/523114786 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:44.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4da7fff700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4db00acdd0 0x7f4db00af820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:44.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4db4ebf700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00b2530 0x7f4db01527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:44.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00b2530 msgr2=0x7f4db01527c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:44.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00b2530 0x7f4db01527c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4db00acdd0 msgr2=0x7f4db00af820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4db00acdd0 0x7f4db00af820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:44.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.371+0000 7f4daffff700 1 -- 172.21.15.61:0/523114786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4da000b1e0 con 0x7f4db00614b0 2022-07-20T03:12:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4daffff700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 0x7f4db0098720 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f4da0002f70 tx=0x7f4da0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4da0011760 con 0x7f4db00614b0 2022-07-20T03:12:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4da00118c0 con 0x7f4db00614b0 2022-07-20T03:12:44.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4db009b150 con 0x7f4db00614b0 2022-07-20T03:12:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4da0011c20 con 0x7f4db00614b0 2022-07-20T03:12:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4db0152210 con 0x7f4db00614b0 2022-07-20T03:12:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4db009c000 con 0x7f4db00614b0 2022-07-20T03:12:44.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4da0025030 con 0x7f4db00614b0 2022-07-20T03:12:44.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4dadffb700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4d94053a60 0x7f4d94055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:44.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4da006cdd0 con 0x7f4db00614b0 2022-07-20T03:12:44.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4da7fff700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4d94053a60 0x7f4d94055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:44.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.375+0000 7f4da7fff700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4d94053a60 0x7f4d94055f20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f4d9800b000 tx=0x7f4d98006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:44.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.379+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4db009c000 con 0x7f4db00614b0 2022-07-20T03:12:44.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:44.547+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f4db009c000 con 0x7f4db00614b0 2022-07-20T03:12:45.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.239+0000 7f4dadffb700 1 -- 172.21.15.61:0/523114786 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v247) v1 ==== 150+0+0 (secure 0 0 0) 0x7f4db009c000 con 0x7f4db00614b0 2022-07-20T03:12:45.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.239+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4d94053a60 msgr2=0x7f4d94055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.239+0000 7f4db6922700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4d94053a60 0x7f4d94055f20 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f4d9800b000 tx=0x7f4d98006df0).stop 2022-07-20T03:12:45.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.239+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 msgr2=0x7f4db0098720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.239+0000 7f4db6922700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 0x7f4db0098720 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f4da0002f70 tx=0x7f4da0015000).stop 2022-07-20T03:12:45.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 shutdown_connections 2022-07-20T03:12:45.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4d94053a60 0x7f4d94055f20 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4db00acdd0 0x7f4db00af820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4db00b2530 0x7f4db01527c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 --2- 172.21.15.61:0/523114786 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4db00614b0 0x7f4db0098720 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 >> 172.21.15.61:0/523114786 conn(0x7f4db00b36b0 msgr2=0x7f4db00b0a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:45.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 shutdown_connections 2022-07-20T03:12:45.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.243+0000 7f4db6922700 1 -- 172.21.15.61:0/523114786 wait complete. 2022-07-20T03:12:45.275 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:12:45.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0f14b700 1 Processor -- start 2022-07-20T03:12:45.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0f14b700 1 -- start start 2022-07-20T03:12:45.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0f14b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7c0805bf30 0x7f7c0805d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:45.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0f14b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080b4230 con 0x7f7c08143ea0 2022-07-20T03:12:45.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0f14b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c080b3850 con 0x7f7c080adfa0 2022-07-20T03:12:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0f14b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c080677b0 con 0x7f7c0805bf30 2022-07-20T03:12:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e94a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7c08143ea0 0x7f7c08144330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42308/0 (socket says 172.21.15.61:42308) 2022-07-20T03:12:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e94a700 1 -- 172.21.15.61:0/2818630912 learned_addr learned my addr 172.21.15.61:0/2818630912 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e149700 1 --2- 172.21.15.61:0/2818630912 >> v2:172.21.15.117:3300/0 conn(0x7f7c0805bf30 0x7f7c0805d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0d147700 1 -- 172.21.15.61:0/2818630912 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3801802696 0 0) 0x7f7c080b4230 con 0x7f7c08143ea0 2022-07-20T03:12:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0d147700 1 -- 172.21.15.61:0/2818630912 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7bf0003260 con 0x7f7c08143ea0 2022-07-20T03:12:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e149700 1 -- 172.21.15.61:0/2818630912 >> v1:172.21.15.117:6789/0 conn(0x7f7c080adfa0 legacy=0x7f7c0813a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0d147700 1 -- 172.21.15.61:0/2818630912 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 901816426 0 0) 0x7f7bf0003260 con 0x7f7c08143ea0 2022-07-20T03:12:45.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e149700 1 -- 172.21.15.61:0/2818630912 >> v1:172.21.15.61:6789/0 conn(0x7f7c08143ea0 legacy=0x7f7c08144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e149700 1 -- 172.21.15.61:0/2818630912 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c080b3af0 con 0x7f7c0805bf30 2022-07-20T03:12:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0e149700 1 --2- 172.21.15.61:0/2818630912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c0805bf30 0x7f7c0805d0d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7c00001b80 tx=0x7f7c04006df0).ready entity=mon.1 client_cookie=290e82de4b2a1b95 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0d147700 1 -- 172.21.15.61:0/2818630912 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c000019e0 con 0x7f7c0805bf30 2022-07-20T03:12:45.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0d147700 1 -- 172.21.15.61:0/2818630912 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c04000e90 con 0x7f7c0805bf30 2022-07-20T03:12:45.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.387+0000 7f7c0d147700 1 -- 172.21.15.61:0/2818630912 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c0400d4f0 con 0x7f7c0805bf30 2022-07-20T03:12:45.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 -- 172.21.15.61:0/2818630912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c0805bf30 msgr2=0x7f7c0805d0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/2818630912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c0805bf30 0x7f7c0805d0d0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f7c00001b80 tx=0x7f7c04006df0).stop 2022-07-20T03:12:45.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 -- 172.21.15.61:0/2818630912 shutdown_connections 2022-07-20T03:12:45.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/2818630912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c0805bf30 0x7f7c0805d0d0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 -- 172.21.15.61:0/2818630912 >> 172.21.15.61:0/2818630912 conn(0x7f7c08064b90 msgr2=0x7f7c08064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:45.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 -- 172.21.15.61:0/2818630912 shutdown_connections 2022-07-20T03:12:45.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 -- 172.21.15.61:0/2818630912 wait complete. 2022-07-20T03:12:45.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 Processor -- start 2022-07-20T03:12:45.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.391+0000 7f7c0f14b700 1 -- start start 2022-07-20T03:12:45.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 0x7f7c0813f6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c080adfa0 0x7f7c080ae840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c08143ea0 0x7f7c08153000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c0813e620 con 0x7f7c0805bf30 2022-07-20T03:12:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c080af100 con 0x7f7c080adfa0 2022-07-20T03:12:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c081404b0 con 0x7f7c08143ea0 2022-07-20T03:12:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 0x7f7c0813f6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 0x7f7c0813f6f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45712/0 (socket says 172.21.15.61:45712) 2022-07-20T03:12:45.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 -- 172.21.15.61:0/1162674252 learned_addr learned my addr 172.21.15.61:0/1162674252 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:45.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e94a700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c08143ea0 0x7f7c08153000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:45.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0d948700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c080adfa0 0x7f7c080ae840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:45.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 -- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c08143ea0 msgr2=0x7f7c08153000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c08143ea0 0x7f7c08153000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 -- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c080adfa0 msgr2=0x7f7c080ae840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c080adfa0 0x7f7c080ae840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 -- 172.21.15.61:0/1162674252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c0400b180 con 0x7f7c0805bf30 2022-07-20T03:12:45.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0e149700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 0x7f7c0813f6f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f7c080af240 tx=0x7f7c04014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:45.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c04016030 con 0x7f7c0805bf30 2022-07-20T03:12:45.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c04013af0 con 0x7f7c0805bf30 2022-07-20T03:12:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c0813efb0 con 0x7f7c0805bf30 2022-07-20T03:12:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c0400d4f0 con 0x7f7c0805bf30 2022-07-20T03:12:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c080aeed0 con 0x7f7c0805bf30 2022-07-20T03:12:45.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.395+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c080b3d90 con 0x7f7c0805bf30 2022-07-20T03:12:45.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.399+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7c04024020 con 0x7f7c0805bf30 2022-07-20T03:12:45.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.399+0000 7f7bf77fe700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7be8053ad0 0x7f7be8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:45.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.399+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7c04017070 con 0x7f7c0805bf30 2022-07-20T03:12:45.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.399+0000 7f7c0d948700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7be8053ad0 0x7f7be8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.399+0000 7f7c0d948700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7be8053ad0 0x7f7be8055f90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7bf800d000 tx=0x7f7bf8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.403+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7c04043d20 con 0x7f7c0805bf30 2022-07-20T03:12:45.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.527+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7c080574e0 con 0x7f7c0805bf30 2022-07-20T03:12:45.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.535+0000 7f7bf77fe700 1 -- 172.21.15.61:0/1162674252 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v8)=0 v8) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7c04047340 con 0x7f7c0805bf30 2022-07-20T03:12:45.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7be8053ad0 msgr2=0x7f7be8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7be8053ad0 0x7f7be8055f90 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f7bf800d000 tx=0x7f7bf8009040).stop 2022-07-20T03:12:45.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 msgr2=0x7f7c0813f6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:45.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 0x7f7c0813f6f0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f7c080af240 tx=0x7f7c04014000).stop 2022-07-20T03:12:45.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 shutdown_connections 2022-07-20T03:12:45.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7be8053ad0 0x7f7be8055f90 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c080adfa0 0x7f7c080ae840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c08143ea0 0x7f7c08153000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 --2- 172.21.15.61:0/1162674252 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c0805bf30 0x7f7c0813f6f0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:45.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 >> 172.21.15.61:0/1162674252 conn(0x7f7c08064b90 msgr2=0x7f7c0813d830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:45.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 shutdown_connections 2022-07-20T03:12:45.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.539+0000 7f7c0f14b700 1 -- 172.21.15.61:0/1162674252 wait complete. 2022-07-20T03:12:45.559 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:12:45.560 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:45.581 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:45.606 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:45.648 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:45.691 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:12:45.735 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:12:45.736 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:45.737 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:45.741 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:45.754 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:12:45.756 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:45.757 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:12:45.758 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:45.759 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:45.760 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:12:45.801 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:45.804 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:12:45.805 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:12:45.806 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:45.809 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:45.811 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:12:45.854 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:12:45.868 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:12:45.869 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:12:45.870 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:12:46.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.987+0000 7eff08c0f700 1 Processor -- start 2022-07-20T03:12:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.987+0000 7eff08c0f700 1 -- start start 2022-07-20T03:12:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.987+0000 7eff08c0f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7eff040b9ef0 0x7eff040b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:46.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.987+0000 7eff08c0f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7eff040b2960 0x7eff04142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.987+0000 7eff08c0f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff040ba2d0 con 0x7eff0405d160 2022-07-20T03:12:46.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff040b4560 con 0x7eff040b9ef0 2022-07-20T03:12:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff040b4800 con 0x7eff040b2960 2022-07-20T03:12:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7eff040b9ef0 0x7eff040b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:46.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7eff040b9ef0 0x7eff040b4e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45720/0 (socket says 172.21.15.61:45720) 2022-07-20T03:12:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 -- 172.21.15.61:0/590354883 learned_addr learned my addr 172.21.15.61:0/590354883 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff02d9d700 1 --2- 172.21.15.61:0/590354883 >> v2:172.21.15.117:3300/0 conn(0x7eff040b2960 0x7eff04142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:46.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 -- 172.21.15.61:0/590354883 >> v2:172.21.15.117:3300/0 conn(0x7eff040b2960 msgr2=0x7eff04142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:46.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 --2- 172.21.15.61:0/590354883 >> v2:172.21.15.117:3300/0 conn(0x7eff040b2960 0x7eff04142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 -- 172.21.15.61:0/590354883 >> v1:172.21.15.117:6789/0 conn(0x7eff0405d160 legacy=0x7eff040b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:46.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 -- 172.21.15.61:0/590354883 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff040b1db0 con 0x7eff040b9ef0 2022-07-20T03:12:46.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0259c700 1 --2- 172.21.15.61:0/590354883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff040b4e50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7efef4002f70 tx=0x7efef4006eb0).ready entity=mon.0 client_cookie=1cf39cf007ba8cb9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:46.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0159a700 1 -- 172.21.15.61:0/590354883 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef400d5c0 con 0x7eff040b9ef0 2022-07-20T03:12:46.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0159a700 1 -- 172.21.15.61:0/590354883 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4000e90 con 0x7eff040b9ef0 2022-07-20T03:12:46.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff0159a700 1 -- 172.21.15.61:0/590354883 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef400d950 con 0x7eff040b9ef0 2022-07-20T03:12:46.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- 172.21.15.61:0/590354883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 msgr2=0x7eff040b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:46.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 --2- 172.21.15.61:0/590354883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff040b4e50 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7efef4002f70 tx=0x7efef4006eb0).stop 2022-07-20T03:12:46.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- 172.21.15.61:0/590354883 shutdown_connections 2022-07-20T03:12:46.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 --2- 172.21.15.61:0/590354883 >> v2:172.21.15.117:3300/0 conn(0x7eff040b2960 0x7eff04142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 --2- 172.21.15.61:0/590354883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff040b4e50 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- 172.21.15.61:0/590354883 >> 172.21.15.61:0/590354883 conn(0x7eff040679d0 msgr2=0x7eff040b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:46.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- 172.21.15.61:0/590354883 shutdown_connections 2022-07-20T03:12:46.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.991+0000 7eff08c0f700 1 -- 172.21.15.61:0/590354883 wait complete. 2022-07-20T03:12:46.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 Processor -- start 2022-07-20T03:12:46.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 -- start start 2022-07-20T03:12:46.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff0405d160 0x7eff040b8bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:46.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 0x7eff040adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:46.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff04152680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:46.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff04138ab0 con 0x7eff040b9ef0 2022-07-20T03:12:46.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff040ae3f0 con 0x7eff040b2960 2022-07-20T03:12:46.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff0413e410 con 0x7eff0405d160 2022-07-20T03:12:46.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff02d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff04152680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:46.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff01d9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 0x7eff040adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:46.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff0405d160 0x7eff040b8bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:46.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff02d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff04152680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45730/0 (socket says 172.21.15.61:45730) 2022-07-20T03:12:46.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff02d9d700 1 -- 172.21.15.61:0/3866758059 learned_addr learned my addr 172.21.15.61:0/3866758059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:46.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff01d9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 0x7eff040adf00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55408/0 (socket says 172.21.15.61:55408) 2022-07-20T03:12:46.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 -- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 msgr2=0x7eff040adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:46.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 0x7eff040adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 -- 172.21.15.61:0/3866758059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 msgr2=0x7eff04152680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:46.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff04152680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 -- 172.21.15.61:0/3866758059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efef400b1e0 con 0x7eff0405d160 2022-07-20T03:12:46.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff02d9d700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff04152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:46.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff01d9b700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 0x7eff040adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:46.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff0259c700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff0405d160 0x7eff040b8bd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7efef4002f70 tx=0x7efef4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:46.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef40116e0 con 0x7eff0405d160 2022-07-20T03:12:46.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efef4011840 con 0x7eff0405d160 2022-07-20T03:12:46.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.995+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff04141330 con 0x7eff0405d160 2022-07-20T03:12:46.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efef4011ba0 con 0x7eff0405d160 2022-07-20T03:12:46.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff04151ef0 con 0x7eff0405d160 2022-07-20T03:12:46.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff04155500 con 0x7eff0405d160 2022-07-20T03:12:46.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efef401a040 con 0x7eff0405d160 2022-07-20T03:12:46.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7efefb7fe700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efee8053ad0 0x7efee8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:46.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7efef406c940 con 0x7eff0405d160 2022-07-20T03:12:46.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:45.999+0000 7eff01d9b700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efee8053ad0 0x7efee8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:46.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.003+0000 7eff01d9b700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efee8053ad0 0x7efee8055f90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7efeec00d000 tx=0x7efeec009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:46.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.007+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efef4043a70 con 0x7eff0405d160 2022-07-20T03:12:46.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.139+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7eff040b3780 con 0x7eff0405d160 2022-07-20T03:12:46.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.139+0000 7efefb7fe700 1 -- 172.21.15.61:0/3866758059 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v55) v1 ==== 81+0+28876 (secure 0 0 0) 0x7efef4047090 con 0x7eff0405d160 2022-07-20T03:12:46.153 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:46.153 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:12:46.175 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 55, 2022-07-20T03:12:46.176 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:12:46.176 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:12:46.177 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:12:43.216843+0000", 2022-07-20T03:12:46.177 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:12:46.177 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:12:46.178 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:12:46.178 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:12:46.179 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:12:46.179 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:12:46.180 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:12:46.180 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:12:46.180 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:12:46.181 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:46.181 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:12:46.182 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:12:46.182 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:12:46.183 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:12:46.183 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:12:46.184 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 10, 2022-07-20T03:12:46.184 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:12:46.185 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:12:46.185 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:12:46.185 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:12:46.186 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:12:46.186 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.187 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:12:46.188 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:12:46.188 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:12:46.189 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:12:46.189 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:12:46.190 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:12:46.190 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:12:46.190 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:12:46.191 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:12:46.191 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:12:46.192 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:12:46.192 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:12:46.193 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:12:46.193 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:12:46.193 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:12:46.195 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:12:46.195 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:12:46.196 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:12:46.196 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:12:46.196 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:12:46.197 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:12:46.197 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:12:46.198 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:12:46.198 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.198 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:12:46.199 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:12:46.199 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:12:46.200 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:12:46.200 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:12:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:12:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:12:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:12:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:12:46.201 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:12:46.202 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:12:46.202 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:12:46.202 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:12:46.202 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:12:46.203 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:12:46.203 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:12:46.203 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:12:46.203 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:12:46.204 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:12:46.204 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:12:46.204 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:12:46.204 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:12:46.204 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:12:46.205 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:12:46.205 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:12:46.205 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:12:46.205 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:12:46.206 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.206 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:12:46.206 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:12:46.206 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:12:46.206 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:12:46.207 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:12:46.207 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:12:46.207 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:12:46.207 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:12:46.208 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:12:46.208 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:12:46.208 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:12:46.208 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:12:46.208 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:12:46.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.209 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:12:46.209 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:12:46.209 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.210 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:12:46.210 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:12:46.211 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:12:46.218 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:12:46.219 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:12:46.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:12:46.219 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:12:46.219 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:12:46.220 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:12:46.220 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:12:46.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:12:46.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:12:46.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:12:46.221 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:12:46.221 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:12:46.221 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:12:46.221 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:12:46.222 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:12:46.222 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:12:46.222 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:12:46.222 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:12:46.223 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:12:46.223 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:12:46.223 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.223 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:12:46.224 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:12:46.224 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:12:46.224 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:12:46.225 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:12:46.225 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:12:46.225 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:12:46.226 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:12:46.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:12:46.226 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:12:46.227 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:12:46.227 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:12:46.227 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:12:46.227 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:12:46.228 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:12:46.228 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:12:46.228 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:12:46.228 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:12:46.229 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:12:46.229 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:12:46.229 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:12:46.229 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:12:46.230 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:12:46.230 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:12:46.241 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:12:46.241 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:12:46.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:12:46.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.242 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:12:46.242 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:12:46.242 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:12:46.243 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:12:46.243 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:12:46.243 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:12:46.243 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:12:46.244 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:12:46.244 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:12:46.244 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:12:46.244 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:12:46.245 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:12:46.245 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:12:46.245 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:12:46.245 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.245 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.246 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:46.246 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:12:46.246 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.247 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:12:46.247 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:12:46.247 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.247 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.248 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.248 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.248 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.249 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:46.249 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.249 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.249 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.250 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:12:46.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.251 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:12:46.252 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.252 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.252 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.253 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:12:46.254 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.255 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.255 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:12:46.280 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.281 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.281 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.287 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.288 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.289 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.289 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:12:46.290 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.290 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.301 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.301 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.302 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:12:46.302 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.303 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.303 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.304 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.313 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.314 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.315 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.315 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:12:46.316 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.317 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.317 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.348 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:12:46.352 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:12:46.352 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.353 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.354 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.354 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:12:46.355 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:12:46.355 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:12:46.356 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:12:46.356 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.357 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.360 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.360 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.361 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.362 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:12:46.362 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:12:46.363 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.363 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.364 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.364 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.365 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.366 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.367 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.367 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:46.368 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.369 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.369 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.370 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.370 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.371 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.372 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:12:46.373 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.373 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:12:46.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.376 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.377 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.378 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.379 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.380 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.381 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:12:46.382 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.382 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.383 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.384 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:12:46.438 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.439 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.444 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.444 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.445 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.454 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:12:46.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.455 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.456 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:12:46.457 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.457 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.458 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.463 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.463 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.464 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.464 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.464 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.464 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:12:46.465 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.465 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.470 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.470 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.503 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:12:46.503 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:12:46.504 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.504 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.504 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.504 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:12:46.504 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:12:46.505 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:12:46.505 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:12:46.505 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.505 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.506 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.506 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.506 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.543 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:12:46.543 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:12:46.544 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.544 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.551 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.551 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.552 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.552 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.553 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.553 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:46.555 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.555 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.556 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.557 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.557 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:12:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.559 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.559 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:12:46.559 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.567 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.567 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.568 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.569 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.570 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.570 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.571 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:12:46.571 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.572 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:12:46.573 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.574 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.574 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.575 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.576 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.576 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.585 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.585 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:12:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.586 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.587 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.588 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.588 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:12:46.589 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.589 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.590 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.590 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.591 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.593 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.594 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.595 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.595 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:12:46.596 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.596 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.597 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.597 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.597 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:12:46.598 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:12:46.598 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.599 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.599 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.599 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:12:46.600 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:12:46.600 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:12:46.601 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:12:46.601 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.602 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.603 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.604 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.605 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.606 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.606 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:12:46.607 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:12:46.607 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.608 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.609 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.609 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.610 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.611 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.611 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.612 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-20T03:12:46.612 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.613 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.613 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.614 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.614 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.615 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.615 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:12:46.616 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.617 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.617 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.618 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.618 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:12:46.619 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.619 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.620 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.621 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.621 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.621 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.622 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.622 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.623 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:12:46.623 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.624 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.624 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.625 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.625 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:12:46.625 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.653 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.654 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.655 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.655 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.656 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.656 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.657 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.657 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:12:46.658 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.658 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.667 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.667 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.668 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:12:46.669 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.674 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.675 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.675 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.676 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.676 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.676 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.677 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.682 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:12:46.684 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.684 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.684 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.685 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.685 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:12:46.686 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:12:46.686 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.687 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.687 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.687 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:12:46.688 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:12:46.688 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:12:46.689 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:12:46.689 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.689 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.690 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.690 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.691 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.691 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.692 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:12:46.692 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:12:46.693 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.694 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.695 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.695 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.696 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.696 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.697 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.697 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:46.698 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.698 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.699 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.699 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.700 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.700 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.700 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:12:46.701 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.701 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.701 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.702 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:12:46.702 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.702 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.702 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.703 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.703 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.703 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.704 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.704 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.704 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:12:46.704 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.704 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.705 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.705 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:12:46.706 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.706 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.706 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.706 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.707 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.707 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.707 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.707 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.707 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:12:46.708 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.708 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.708 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.709 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:12:46.709 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.709 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.709 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.711 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.711 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.711 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.711 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:12:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.712 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.713 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.713 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.713 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:12:46.713 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:12:46.714 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.714 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.715 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.715 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:12:46.715 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:12:46.715 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:12:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:12:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.716 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.717 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.717 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.717 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.717 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:12:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:12:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.719 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.719 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.719 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.719 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.720 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.721 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.721 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.721 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:12:46.722 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.722 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.722 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.723 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.723 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:12:46.723 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.723 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.724 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.724 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.724 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.724 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:12:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.725 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.726 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.726 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.726 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:12:46.726 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:12:46.729 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.729 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.729 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.729 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.730 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:12:46.730 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.730 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.730 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.731 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.732 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.732 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:12:46.732 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.732 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.733 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.733 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.733 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:12:46.733 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:12:46.734 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.734 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.734 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.734 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:12:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:12:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:12:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:12:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.736 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.736 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.736 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.736 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.737 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.737 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:12:46.737 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:12:46.737 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.738 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.738 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.738 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.738 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.739 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.739 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.739 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 50, 2022-07-20T03:12:46.739 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.740 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.740 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.740 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.740 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.740 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.741 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:12:46.741 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.741 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.741 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.742 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.742 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:12:46.742 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.742 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.743 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.743 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.743 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.743 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:12:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:12:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.746 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:12:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:12:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:12:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.753 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:12:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:12:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.755 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.755 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:12:46.755 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:12:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:12:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:12:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.756 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.757 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.758 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:12:46.758 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:12:46.758 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:12:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:12:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:12:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:12:46.759 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:12:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:12:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:12:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-20T03:12:46.760 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:12:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:12:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:12:46.761 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:12:46.773 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.773 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.773 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.774 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.775 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:12:46.775 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.775 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.776 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.776 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.777 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:12:46.777 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.778 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.779 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.779 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:12:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.782 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:12:46.782 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.783 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.784 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:12:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.786 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:12:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:12:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:12:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:12:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:12:46.792 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:12:46.793 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:12:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:12:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:12:46.798 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.800 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:12:46.800 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:12:46.801 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:12:46.801 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:12:46.802 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:12:46.802 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:12:46.803 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:12:46.803 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:46.804 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.804 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:12:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:12:46.806 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:12:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.842 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:12:46.846 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.846 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:12:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:12:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:12:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:12:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:12:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:12:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:12:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:12:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:12:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:12:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:12:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:12:46.871 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:12:46.872 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:12:46.872 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:12:46.872 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:12:46.873 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:12:46.873 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:12:46.873 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:12:46.894 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.894 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:12:46.894 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:12:46.895 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:12:46.895 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:12:46.895 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:12:46.895 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:12:46.896 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2246781672": "2022-07-21T03:12:39.757099+0000", 2022-07-20T03:12:46.896 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2246781672": "2022-07-21T03:12:39.757099+0000" 2022-07-20T03:12:46.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.896 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:12:46.917 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:12:46.917 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:12:46.917 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:12:46.917 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:12:46.918 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:12:46.918 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:12:46.918 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:12:46.918 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:46.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:46.919 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:12:46.919 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:12:46.919 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:12:46.920 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:12:46.920 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:12:46.920 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:12:46.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efee8053ad0 msgr2=0x7efee8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:46.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efee8053ad0 0x7efee8055f90 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7efeec00d000 tx=0x7efeec009040).stop 2022-07-20T03:12:46.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff0405d160 msgr2=0x7eff040b8bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:46.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff0405d160 0x7eff040b8bd0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7efef4002f70 tx=0x7efef4015000).stop 2022-07-20T03:12:46.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 shutdown_connections 2022-07-20T03:12:46.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efee8053ad0 0x7efee8055f90 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff040b2960 0x7eff040adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff040b9ef0 0x7eff04152680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 --2- 172.21.15.61:0/3866758059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff0405d160 0x7eff040b8bd0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:46.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 >> 172.21.15.61:0/3866758059 conn(0x7eff040679d0 msgr2=0x7eff040b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:46.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 shutdown_connections 2022-07-20T03:12:46.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:46.143+0000 7eff08c0f700 1 -- 172.21.15.61:0/3866758059 wait complete. 2022-07-20T03:12:46.927 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:47.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.035+0000 7f20e5750700 1 Processor -- start 2022-07-20T03:12:47.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- start start 2022-07-20T03:12:47.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f20e00b1e00 0x7f20e014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e00acef0 0x7f20e0146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20e005d480 con 0x7f20e005c3f0 2022-07-20T03:12:47.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20e00b21e0 con 0x7f20e00b1e00 2022-07-20T03:12:47.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e005cf40 con 0x7f20e00acef0 2022-07-20T03:12:47.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20de7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f20e005c3f0 0x7f20e00ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42332/0 (socket says 172.21.15.61:42332) 2022-07-20T03:12:47.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20de7fc700 1 -- 172.21.15.61:0/2694152357 learned_addr learned my addr 172.21.15.61:0/2694152357 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:47.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20deffd700 1 --2- 172.21.15.61:0/2694152357 >> v2:172.21.15.61:3300/0 conn(0x7f20e00b1e00 0x7f20e014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20df7fe700 1 --2- 172.21.15.61:0/2694152357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e00acef0 0x7f20e0146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20ddffb700 1 -- 172.21.15.61:0/2694152357 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3482144487 0 0) 0x7f20e005d480 con 0x7f20e005c3f0 2022-07-20T03:12:47.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20ddffb700 1 -- 172.21.15.61:0/2694152357 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20c4003260 con 0x7f20e005c3f0 2022-07-20T03:12:47.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20deffd700 1 -- 172.21.15.61:0/2694152357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e00acef0 msgr2=0x7f20e0146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:47.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20deffd700 1 --2- 172.21.15.61:0/2694152357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e00acef0 0x7f20e0146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20deffd700 1 -- 172.21.15.61:0/2694152357 >> v1:172.21.15.61:6789/0 conn(0x7f20e005c3f0 legacy=0x7f20e00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20deffd700 1 -- 172.21.15.61:0/2694152357 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20e005d1e0 con 0x7f20e00b1e00 2022-07-20T03:12:47.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20deffd700 1 --2- 172.21.15.61:0/2694152357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e014a6d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f20d4002f70 tx=0x7f20d4006df0).ready entity=mon.0 client_cookie=8db1e8eb7fa03376 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20ddffb700 1 -- 172.21.15.61:0/2694152357 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d400d4a0 con 0x7f20e00b1e00 2022-07-20T03:12:47.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20ddffb700 1 -- 172.21.15.61:0/2694152357 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d4000e90 con 0x7f20e00b1e00 2022-07-20T03:12:47.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20ddffb700 1 -- 172.21.15.61:0/2694152357 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d400d7f0 con 0x7f20e00b1e00 2022-07-20T03:12:47.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- 172.21.15.61:0/2694152357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 msgr2=0x7f20e014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 --2- 172.21.15.61:0/2694152357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e014a6d0 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f20d4002f70 tx=0x7f20d4006df0).stop 2022-07-20T03:12:47.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- 172.21.15.61:0/2694152357 shutdown_connections 2022-07-20T03:12:47.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 --2- 172.21.15.61:0/2694152357 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e00acef0 0x7f20e0146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 --2- 172.21.15.61:0/2694152357 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e014a6d0 unknown :-1 s=CLOSED pgs=575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- 172.21.15.61:0/2694152357 >> 172.21.15.61:0/2694152357 conn(0x7f20e00611c0 msgr2=0x7f20e0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:47.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- 172.21.15.61:0/2694152357 shutdown_connections 2022-07-20T03:12:47.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.039+0000 7f20e5750700 1 -- 172.21.15.61:0/2694152357 wait complete. 2022-07-20T03:12:47.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 Processor -- start 2022-07-20T03:12:47.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 -- start start 2022-07-20T03:12:47.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e00acef0 0x7f20e00645b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e00aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e0065ca0 0x7f20e0149300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0064f40 con 0x7f20e00b1e00 2022-07-20T03:12:47.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0063a80 con 0x7f20e00acef0 2022-07-20T03:12:47.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20e0063d20 con 0x7f20e0065ca0 2022-07-20T03:12:47.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e00aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20deffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e00acef0 0x7f20e00645b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e00aeb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45742/0 (socket says 172.21.15.61:45742) 2022-07-20T03:12:47.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 -- 172.21.15.61:0/3144465669 learned_addr learned my addr 172.21.15.61:0/3144465669 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:47.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20df7fe700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e0065ca0 0x7f20e0149300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 -- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e0065ca0 msgr2=0x7f20e0149300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e0065ca0 0x7f20e0149300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 -- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e00acef0 msgr2=0x7f20e00645b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e00acef0 0x7f20e00645b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 -- 172.21.15.61:0/3144465669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d400b180 con 0x7f20e00b1e00 2022-07-20T03:12:47.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20de7fc700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e00aeb90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f20d0001110 tx=0x7f20d0007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d00070d0 con 0x7f20e00b1e00 2022-07-20T03:12:47.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d0007230 con 0x7f20e00b1e00 2022-07-20T03:12:47.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20e0143690 con 0x7f20e00b1e00 2022-07-20T03:12:47.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d0005610 con 0x7f20e00b1e00 2022-07-20T03:12:47.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.043+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20e014a530 con 0x7f20e00b1e00 2022-07-20T03:12:47.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.047+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20e0063ee0 con 0x7f20e00b1e00 2022-07-20T03:12:47.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.047+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f20d001a060 con 0x7f20e00b1e00 2022-07-20T03:12:47.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.047+0000 7f20cffff700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20bc053b20 0x7f20bc055fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.047+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f20d006b300 con 0x7f20e00b1e00 2022-07-20T03:12:47.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.047+0000 7f20deffd700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20bc053b20 0x7f20bc055fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.047+0000 7f20deffd700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20bc053b20 0x7f20bc055fe0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f20d401a000 tx=0x7f20d4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.055+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20d0042540 con 0x7f20e00b1e00 2022-07-20T03:12:47.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.219+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f20e00b04b0 con 0x7f20e00b1e00 2022-07-20T03:12:47.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20cffff700 1 -- 172.21.15.61:0/3144465669 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1206 (secure 0 0 0) 0x7f20d0018070 con 0x7f20e00b1e00 2022-07-20T03:12:47.223 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:47.223 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/602338255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":602338255},{"type":"v1","addr":"172.21.15.61:6833","nonce":602338255}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":32}],"filesystems":[]} 2022-07-20T03:12:47.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20bc053b20 msgr2=0x7f20bc055fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20bc053b20 0x7f20bc055fe0 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f20d401a000 tx=0x7f20d4014000).stop 2022-07-20T03:12:47.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 msgr2=0x7f20e00aeb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e00aeb90 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f20d0001110 tx=0x7f20d0007a80).stop 2022-07-20T03:12:47.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 shutdown_connections 2022-07-20T03:12:47.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20bc053b20 0x7f20bc055fe0 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20e0065ca0 0x7f20e0149300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20e00b1e00 0x7f20e00aeb90 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 --2- 172.21.15.61:0/3144465669 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20e00acef0 0x7f20e00645b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 >> 172.21.15.61:0/3144465669 conn(0x7f20e005c3f0 msgr2=0x7f20e0145720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:47.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 shutdown_connections 2022-07-20T03:12:47.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.223+0000 7f20e5750700 1 -- 172.21.15.61:0/3144465669 wait complete. 2022-07-20T03:12:47.238 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 32 2022-07-20T03:12:47.240 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:47.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 Processor -- start 2022-07-20T03:12:47.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 -- start start 2022-07-20T03:12:47.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fec381424e0 0x7fec380b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec380aea10 0x7fec380ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec381433d0 con 0x7fec380b0570 2022-07-20T03:12:47.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec38143b10 con 0x7fec380aea10 2022-07-20T03:12:47.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec3cb3f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec38143670 con 0x7fec381424e0 2022-07-20T03:12:47.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec37fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec380aea10 0x7fec380ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec37fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec380aea10 0x7fec380ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45752/0 (socket says 172.21.15.61:45752) 2022-07-20T03:12:47.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec37fff700 1 -- 172.21.15.61:0/4224169213 learned_addr learned my addr 172.21.15.61:0/4224169213 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:47.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.335+0000 7fec2ffff700 1 --2- 172.21.15.61:0/4224169213 >> v2:172.21.15.117:3300/0 conn(0x7fec381424e0 0x7fec380b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec37fff700 1 -- 172.21.15.61:0/4224169213 >> v2:172.21.15.117:3300/0 conn(0x7fec381424e0 msgr2=0x7fec380b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:47.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec37fff700 1 --2- 172.21.15.61:0/4224169213 >> v2:172.21.15.117:3300/0 conn(0x7fec381424e0 0x7fec380b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec37fff700 1 -- 172.21.15.61:0/4224169213 >> v1:172.21.15.117:6789/0 conn(0x7fec380b0570 legacy=0x7fec38141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec37fff700 1 -- 172.21.15.61:0/4224169213 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec380b1010 con 0x7fec380aea10 2022-07-20T03:12:47.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec37fff700 1 --2- 172.21.15.61:0/4224169213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec380ad350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fec300032f0 tx=0x7fec30009290).ready entity=mon.0 client_cookie=d8fcfa22ad23a62f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec36ffd700 1 -- 172.21.15.61:0/4224169213 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec3000b870 con 0x7fec380aea10 2022-07-20T03:12:47.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec36ffd700 1 -- 172.21.15.61:0/4224169213 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec3000b9d0 con 0x7fec380aea10 2022-07-20T03:12:47.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec36ffd700 1 -- 172.21.15.61:0/4224169213 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec3000bb40 con 0x7fec380aea10 2022-07-20T03:12:47.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 -- 172.21.15.61:0/4224169213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 msgr2=0x7fec380ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/4224169213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec380ad350 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fec300032f0 tx=0x7fec30009290).stop 2022-07-20T03:12:47.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 -- 172.21.15.61:0/4224169213 shutdown_connections 2022-07-20T03:12:47.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/4224169213 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec380ad350 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/4224169213 >> v2:172.21.15.117:3300/0 conn(0x7fec381424e0 0x7fec380b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 -- 172.21.15.61:0/4224169213 >> 172.21.15.61:0/4224169213 conn(0x7fec380b9360 msgr2=0x7fec380b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:47.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 -- 172.21.15.61:0/4224169213 shutdown_connections 2022-07-20T03:12:47.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 -- 172.21.15.61:0/4224169213 wait complete. 2022-07-20T03:12:47.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 Processor -- start 2022-07-20T03:12:47.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.339+0000 7fec3cb3f700 1 -- start start 2022-07-20T03:12:47.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec38112b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec380b0570 0x7fec380b38c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec381424e0 0x7fec380b3db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec380b2d20 con 0x7fec380aea10 2022-07-20T03:12:47.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec38115110 con 0x7fec381424e0 2022-07-20T03:12:47.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec381126d0 con 0x7fec380b0570 2022-07-20T03:12:47.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec38112b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec38112b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45754/0 (socket says 172.21.15.61:45754) 2022-07-20T03:12:47.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 -- 172.21.15.61:0/3811662346 learned_addr learned my addr 172.21.15.61:0/3811662346 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:47.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec37fff700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec381424e0 0x7fec380b3db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec2ffff700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec380b0570 0x7fec380b38c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 -- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec380b0570 msgr2=0x7fec380b38c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec380b0570 0x7fec380b38c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 -- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec381424e0 msgr2=0x7fec380b3db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec381424e0 0x7fec380b3db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 -- 172.21.15.61:0/3811662346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec30009530 con 0x7fec380aea10 2022-07-20T03:12:47.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec377fe700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec38112b70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fec38066710 tx=0x7fec28007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec28007030 con 0x7fec380aea10 2022-07-20T03:12:47.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec28007190 con 0x7fec380aea10 2022-07-20T03:12:47.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec38113060 con 0x7fec380aea10 2022-07-20T03:12:47.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec28006510 con 0x7fec380aea10 2022-07-20T03:12:47.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec380b4c40 con 0x7fec380aea10 2022-07-20T03:12:47.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fec38112890 con 0x7fec380aea10 2022-07-20T03:12:47.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.343+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fec28010030 con 0x7fec380aea10 2022-07-20T03:12:47.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.347+0000 7fec357fa700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec1c053a40 0x7fec1c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.347+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(55..55 src has 1..55) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fec2806b6f0 con 0x7fec380aea10 2022-07-20T03:12:47.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.347+0000 7fec2ffff700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec1c053a40 0x7fec1c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.347+0000 7fec2ffff700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec1c053a40 0x7fec1c055f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fec2000b000 tx=0x7fec20006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.355+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fec28042930 con 0x7fec380aea10 2022-07-20T03:12:47.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.515+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fec380ace30 con 0x7fec380aea10 2022-07-20T03:12:47.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.515+0000 7fec357fa700 1 -- 172.21.15.61:0/3811662346 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 32 v32) v1 ==== 94+0+1206 (secure 0 0 0) 0x7fec28045f50 con 0x7fec380aea10 2022-07-20T03:12:47.518 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:47.518 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":32,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5147,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/840207306","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":840207306},{"type":"v1","addr":"172.21.15.117:6835","nonce":840207306}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":24},{"gid":5898,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/602338255","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":602338255},{"type":"v1","addr":"172.21.15.61:6833","nonce":602338255}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":32}],"filesystems":[]} 2022-07-20T03:12:47.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec1c053a40 msgr2=0x7fec1c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec1c053a40 0x7fec1c055f00 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7fec2000b000 tx=0x7fec20006df0).stop 2022-07-20T03:12:47.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 msgr2=0x7fec38112b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec38112b70 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7fec38066710 tx=0x7fec28007c10).stop 2022-07-20T03:12:47.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 shutdown_connections 2022-07-20T03:12:47.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec1c053a40 0x7fec1c055f00 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec381424e0 0x7fec380b3db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec380b0570 0x7fec380b38c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 --2- 172.21.15.61:0/3811662346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec380aea10 0x7fec38112b70 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 >> 172.21.15.61:0/3811662346 conn(0x7fec380b9360 msgr2=0x7fec38141200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:47.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 shutdown_connections 2022-07-20T03:12:47.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.519+0000 7fec3cb3f700 1 -- 172.21.15.61:0/3811662346 wait complete. 2022-07-20T03:12:47.523 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 32 2022-07-20T03:12:47.531 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:12:47.531 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:12:47.532 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:12:47.532 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:12:47.532 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:12:47.533 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:12:47.533 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:12:47.533 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:12:47.553 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:12:47.553 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:12:47.554 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:12:47.554 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:12:47.557 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:12:47.557 DEBUG:teuthology.parallel:result is None 2022-07-20T03:12:47.558 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:12:47.559 DEBUG:teuthology.parallel:result is None 2022-07-20T03:12:47.559 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:12:47.628 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:12:47.629 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:12:47.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 Processor -- start 2022-07-20T03:12:47.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 -- start start 2022-07-20T03:12:47.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c40af8e0 0x7f77c40b0f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f77c4144050 0x7f77c413fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77c405c9a0 con 0x7f77c40af030 2022-07-20T03:12:47.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77c4065570 con 0x7f77c4144050 2022-07-20T03:12:47.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9834700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c40af410 con 0x7f77c40af8e0 2022-07-20T03:12:47.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c8832700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f77c40af030 0x7f77c4064e60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42372/0 (socket says 172.21.15.61:42372) 2022-07-20T03:12:47.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c8832700 1 -- 172.21.15.61:0/3796862992 learned_addr learned my addr 172.21.15.61:0/3796862992 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:47.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77bbfff700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c40af8e0 0x7f77c40b0f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9033700 1 --2- 172.21.15.61:0/3796862992 >> v2:172.21.15.117:3300/0 conn(0x7f77c4144050 0x7f77c413fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9033700 1 -- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c40af8e0 msgr2=0x7f77c40b0f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:47.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9033700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c40af8e0 0x7f77c40b0f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9033700 1 -- 172.21.15.61:0/3796862992 >> v1:172.21.15.61:6789/0 conn(0x7f77c40af030 legacy=0x7f77c4064e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9033700 1 -- 172.21.15.61:0/3796862992 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77c405cc00 con 0x7f77c4144050 2022-07-20T03:12:47.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77bbfff700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c40af8e0 0x7f77c40b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:47.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.663+0000 7f77c9033700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c4144050 0x7f77c413fa00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f77bc0032f0 tx=0x7f77bc0091d0).ready entity=mon.1 client_cookie=1ef2c1e72b65e4e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.667+0000 7f77bb7fe700 1 -- 172.21.15.61:0/3796862992 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77bc00b6c0 con 0x7f77c4144050 2022-07-20T03:12:47.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.667+0000 7f77bb7fe700 1 -- 172.21.15.61:0/3796862992 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77bc00b820 con 0x7f77c4144050 2022-07-20T03:12:47.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.667+0000 7f77bb7fe700 1 -- 172.21.15.61:0/3796862992 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77bc00b990 con 0x7f77c4144050 2022-07-20T03:12:47.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 -- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c4144050 msgr2=0x7f77c413fa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c4144050 0x7f77c413fa00 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f77bc0032f0 tx=0x7f77bc0091d0).stop 2022-07-20T03:12:47.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 -- 172.21.15.61:0/3796862992 shutdown_connections 2022-07-20T03:12:47.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c4144050 0x7f77c413fa00 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 --2- 172.21.15.61:0/3796862992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c40af8e0 0x7f77c40b0f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 -- 172.21.15.61:0/3796862992 >> 172.21.15.61:0/3796862992 conn(0x7f77c40629f0 msgr2=0x7f77c4062dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:47.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 -- 172.21.15.61:0/3796862992 shutdown_connections 2022-07-20T03:12:47.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 -- 172.21.15.61:0/3796862992 wait complete. 2022-07-20T03:12:47.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 Processor -- start 2022-07-20T03:12:47.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.671+0000 7f77c9834700 1 -- start start 2022-07-20T03:12:47.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c40af030 0x7f77c4146750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 0x7f77c414fec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c4144050 0x7f77c410fd60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c410ea50 con 0x7f77c40af8e0 2022-07-20T03:12:47.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77c413fef0 con 0x7f77c40af030 2022-07-20T03:12:47.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77c4140190 con 0x7f77c4144050 2022-07-20T03:12:47.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 0x7f77c414fec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 0x7f77c414fec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45782/0 (socket says 172.21.15.61:45782) 2022-07-20T03:12:47.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c8832700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c40af030 0x7f77c4146750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 -- 172.21.15.61:0/2038550867 learned_addr learned my addr 172.21.15.61:0/2038550867 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:47.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9033700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c4144050 0x7f77c410fd60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 -- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c4144050 msgr2=0x7f77c410fd60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c4144050 0x7f77c410fd60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 -- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c40af030 msgr2=0x7f77c4146750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:47.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c40af030 0x7f77c4146750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:47.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 -- 172.21.15.61:0/2038550867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b400b180 con 0x7f77c40af8e0 2022-07-20T03:12:47.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77bbfff700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 0x7f77c414fec0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f77b400c000 tx=0x7f77b4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77b4010780 con 0x7f77c40af8e0 2022-07-20T03:12:47.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77b40108e0 con 0x7f77c40af8e0 2022-07-20T03:12:47.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77bc009470 con 0x7f77c40af8e0 2022-07-20T03:12:47.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77b4010c40 con 0x7f77c40af8e0 2022-07-20T03:12:47.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77c41503b0 con 0x7f77c40af8e0 2022-07-20T03:12:47.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77c413fd20 con 0x7f77c40af8e0 2022-07-20T03:12:47.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.675+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f77b401c460 con 0x7f77c40af8e0 2022-07-20T03:12:47.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.679+0000 7f77b9ffb700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77a4053ad0 0x7f77a4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:47.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.679+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(56..56 src has 1..56) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f77b406c0e0 con 0x7f77c40af8e0 2022-07-20T03:12:47.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.679+0000 7f77c8832700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77a4053ad0 0x7f77a4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:47.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.679+0000 7f77c8832700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77a4053ad0 0x7f77a4055f90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f77c000d000 tx=0x7f77c0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:47.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.683+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77b40432c0 con 0x7f77c40af8e0 2022-07-20T03:12:47.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:47.823+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f77c4064720 con 0x7f77c40af8e0 2022-07-20T03:12:48.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.671+0000 7f77b9ffb700 1 -- 172.21.15.61:0/2038550867 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v57) v1 ==== 97+0+0 (secure 0 0 0) 0x7f77b40468e0 con 0x7f77c40af8e0 2022-07-20T03:12:48.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77a4053ad0 msgr2=0x7f77a4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:48.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77a4053ad0 0x7f77a4055f90 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f77c000d000 tx=0x7f77c0009040).stop 2022-07-20T03:12:48.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 msgr2=0x7f77c414fec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:48.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 0x7f77c414fec0 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7f77b400c000 tx=0x7f77b4006df0).stop 2022-07-20T03:12:48.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 shutdown_connections 2022-07-20T03:12:48.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77a4053ad0 0x7f77a4055f90 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:48.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77c4144050 0x7f77c410fd60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:48.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77c40af8e0 0x7f77c414fec0 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:48.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 --2- 172.21.15.61:0/2038550867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77c40af030 0x7f77c4146750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:48.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.675+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 >> 172.21.15.61:0/2038550867 conn(0x7f77c40629f0 msgr2=0x7f77c40b40e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:48.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.679+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 shutdown_connections 2022-07-20T03:12:48.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.679+0000 7f77c9834700 1 -- 172.21.15.61:0/2038550867 wait complete. 2022-07-20T03:12:48.700 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:12:48.704 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:12:48.763 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:12:49.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.887+0000 7fc9cccb2700 1 Processor -- start 2022-07-20T03:12:49.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.887+0000 7fc9cccb2700 1 -- start start 2022-07-20T03:12:49.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c805c3f0 0x7fc9c814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc9c80acef0 0x7fc9c8146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9c805d480 con 0x7fc9c80b1e00 2022-07-20T03:12:49.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9c80b21e0 con 0x7fc9c80acef0 2022-07-20T03:12:49.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c805cf40 con 0x7fc9c805c3f0 2022-07-20T03:12:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc9c80acef0 0x7fc9c8146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc9c80acef0 0x7fc9c8146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45792/0 (socket says 172.21.15.61:45792) 2022-07-20T03:12:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/3083859865 learned_addr learned my addr 172.21.15.61:0/3083859865 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:49.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c5d9b700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c805c3f0 0x7fc9c814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/3083859865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c805c3f0 msgr2=0x7fc9c814a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c805c3f0 0x7fc9c814a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/3083859865 >> v1:172.21.15.117:6789/0 conn(0x7fc9c80b1e00 legacy=0x7fc9c80ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/3083859865 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c805d1e0 con 0x7fc9c80acef0 2022-07-20T03:12:49.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c5d9b700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c805c3f0 0x7fc9c814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:49.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c6d9d700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c80acef0 0x7fc9c8146410 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fc9c0002f70 tx=0x7fc9c0006eb0).ready entity=mon.0 client_cookie=4dd0e9a865f8ecd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c559a700 1 -- 172.21.15.61:0/3083859865 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9c000d5c0 con 0x7fc9c80acef0 2022-07-20T03:12:49.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c559a700 1 -- 172.21.15.61:0/3083859865 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c0000e90 con 0x7fc9c80acef0 2022-07-20T03:12:49.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9c559a700 1 -- 172.21.15.61:0/3083859865 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9c000d950 con 0x7fc9c80acef0 2022-07-20T03:12:49.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- 172.21.15.61:0/3083859865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c80acef0 msgr2=0x7fc9c8146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c80acef0 0x7fc9c8146410 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7fc9c0002f70 tx=0x7fc9c0006eb0).stop 2022-07-20T03:12:49.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- 172.21.15.61:0/3083859865 shutdown_connections 2022-07-20T03:12:49.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c80acef0 0x7fc9c8146410 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/3083859865 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c805c3f0 0x7fc9c814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- 172.21.15.61:0/3083859865 >> 172.21.15.61:0/3083859865 conn(0x7fc9c80611c0 msgr2=0x7fc9c8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:49.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- 172.21.15.61:0/3083859865 shutdown_connections 2022-07-20T03:12:49.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.891+0000 7fc9cccb2700 1 -- 172.21.15.61:0/3083859865 wait complete. 2022-07-20T03:12:49.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 Processor -- start 2022-07-20T03:12:49.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 -- start start 2022-07-20T03:12:49.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c80acef0 0x7fc9c80b34f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9c80b1e00 0x7fc9c81432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 0x7fc9c8142ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c8149730 con 0x7fc9c8142620 2022-07-20T03:12:49.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c814a2d0 con 0x7fc9c80b1e00 2022-07-20T03:12:49.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9c8143790 con 0x7fc9c80acef0 2022-07-20T03:12:49.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 0x7fc9c8142ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 0x7fc9c8142ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45798/0 (socket says 172.21.15.61:45798) 2022-07-20T03:12:49.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/2881071479 learned_addr learned my addr 172.21.15.61:0/2881071479 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:49.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c5d9b700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9c80b1e00 0x7fc9c81432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c659c700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c80acef0 0x7fc9c80b34f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c80acef0 msgr2=0x7fc9c80b34f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c80acef0 0x7fc9c80b34f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9c80b1e00 msgr2=0x7fc9c81432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9c80b1e00 0x7fc9c81432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 -- 172.21.15.61:0/2881071479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9c000b1e0 con 0x7fc9c8142620 2022-07-20T03:12:49.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9c6d9d700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 0x7fc9c8142ab0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fc9c0002f70 tx=0x7fc9c0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9c000d940 con 0x7fc9c8142620 2022-07-20T03:12:49.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9c0011b80 con 0x7fc9c8142620 2022-07-20T03:12:49.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9c814a570 con 0x7fc9c8142620 2022-07-20T03:12:49.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9c001e460 con 0x7fc9c8142620 2022-07-20T03:12:49.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.895+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9c8149b60 con 0x7fc9c8142620 2022-07-20T03:12:49.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.899+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9c81498a0 con 0x7fc9c8142620 2022-07-20T03:12:49.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.899+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc9c0025030 con 0x7fc9c8142620 2022-07-20T03:12:49.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.899+0000 7fc9b77fe700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9a4053a40 0x7fc9a4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.899+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc9c006cdd0 con 0x7fc9c8142620 2022-07-20T03:12:49.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.899+0000 7fc9c659c700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9a4053a40 0x7fc9a4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.899+0000 7fc9c659c700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9a4053a40 0x7fc9a4055f00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc9bc00d000 tx=0x7fc9bc007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:48.907+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc9c81498a0 con 0x7fc9c8142620 2022-07-20T03:12:49.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.071+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fc9c8149870 con 0x7fc9c8142620 2022-07-20T03:12:49.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.071+0000 7fc9b77fe700 1 -- 172.21.15.61:0/2881071479 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v21) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fc9c0023020 con 0x7fc9c8142620 2022-07-20T03:12:49.086 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:49.086 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:12:49.086 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:12:49.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.087 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:12:49.088 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:12:49.088 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.088 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:12:49.089 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.089 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.089 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.089 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.093 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:12:49.106 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:12:49.106 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.106 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:12:49.107 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.107 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.107 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.108 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.108 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.117 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:12:49.117 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:12:49.118 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.118 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.118 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.118 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.123 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.124 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:12:49.124 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:12:49.125 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.125 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.126 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.126 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.126 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.131 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:12:49.131 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:12:49.132 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.132 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.136 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.136 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.137 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:12:49.138 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:12:49.138 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.138 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.143 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.143 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.144 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.146 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:12:49.146 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:12:49.165 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.166 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.166 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.166 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.167 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.167 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.167 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.167 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:12:49.168 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:12:49.168 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.170 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.171 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.171 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.171 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.172 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.173 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:12:49.174 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:12:49.174 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.174 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.174 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.175 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.175 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.176 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.176 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:12:49.176 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:12:49.177 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.177 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.177 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.177 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.178 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.178 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:12:49.179 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:12:49.181 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.182 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:12:49.182 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:12:49.182 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:12:49.182 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.183 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.183 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:12:49.183 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:12:49.184 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.184 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:12:49.184 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:12:49.184 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:12:49.185 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.185 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.186 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.186 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:12:49.186 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:12:49.187 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.188 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:12:49.188 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:12:49.199 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:12:49.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.267 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:12:49.268 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:12:49.269 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.269 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:12:49.294 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:12:49.295 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:12:49.295 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.298 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.300 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:12:49.345 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:12:49.369 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.370 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:12:49.372 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:12:49.373 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.373 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:49.379 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:49.380 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:12:49.380 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:12:49.381 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:12:49.381 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:12:49.406 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:12:49.416 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:12:49.416 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.416 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:49.417 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:12:49.418 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:12:49.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.071+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9a4053a40 msgr2=0x7fc9a4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.071+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9a4053a40 0x7fc9a4055f00 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fc9bc00d000 tx=0x7fc9bc007b70).stop 2022-07-20T03:12:49.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.071+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 msgr2=0x7fc9c8142ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.071+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 0x7fc9c8142ab0 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fc9c0002f70 tx=0x7fc9c0015000).stop 2022-07-20T03:12:49.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 shutdown_connections 2022-07-20T03:12:49.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9a4053a40 0x7fc9a4055f00 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9c80b1e00 0x7fc9c81432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9c8142620 0x7fc9c8142ab0 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 --2- 172.21.15.61:0/2881071479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9c80acef0 0x7fc9c80b34f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 >> 172.21.15.61:0/2881071479 conn(0x7fc9c805c3f0 msgr2=0x7fc9c814c110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:49.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 shutdown_connections 2022-07-20T03:12:49.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.075+0000 7fc9cccb2700 1 -- 172.21.15.61:0/2881071479 wait complete. 2022-07-20T03:12:49.461 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:12:49.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 Processor -- start 2022-07-20T03:12:49.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 -- start start 2022-07-20T03:12:49.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f016c0683d0 0x7f016c0b0380 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f016c0b18a0 0x7f016c13eba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016c0acff0 con 0x7f016c0b18a0 2022-07-20T03:12:49.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f016c0ad4c0 con 0x7f016c0683d0 2022-07-20T03:12:49.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f01739a0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016c13f090 con 0x7f016c0b2540 2022-07-20T03:12:49.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f016c0b18a0 0x7f016c13eba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f016c0b18a0 0x7f016c13eba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45802/0 (socket says 172.21.15.61:45802) 2022-07-20T03:12:49.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 -- 172.21.15.61:0/135456536 learned_addr learned my addr 172.21.15.61:0/135456536 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:49.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f017173c700 1 --2- 172.21.15.61:0/135456536 >> v2:172.21.15.117:3300/0 conn(0x7f016c0683d0 0x7f016c0b0380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0171f3d700 1 --2- 172.21.15.61:0/135456536 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 -- 172.21.15.61:0/135456536 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 msgr2=0x7f016c142720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:49.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 --2- 172.21.15.61:0/135456536 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c142720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 -- 172.21.15.61:0/135456536 >> v2:172.21.15.117:3300/0 conn(0x7f016c0683d0 msgr2=0x7f016c0b0380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:49.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 --2- 172.21.15.61:0/135456536 >> v2:172.21.15.117:3300/0 conn(0x7f016c0683d0 0x7f016c0b0380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 -- 172.21.15.61:0/135456536 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016c0ad220 con 0x7f016c0b18a0 2022-07-20T03:12:49.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f017173c700 1 --2- 172.21.15.61:0/135456536 >> v2:172.21.15.117:3300/0 conn(0x7f016c0683d0 0x7f016c0b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:49.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f0170f3b700 1 --2- 172.21.15.61:0/135456536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0b18a0 0x7f016c13eba0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f01640032f0 tx=0x7f0164009290).ready entity=mon.0 client_cookie=f981d642102f8725 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f016bfff700 1 -- 172.21.15.61:0/135456536 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016400b680 con 0x7f016c0b18a0 2022-07-20T03:12:49.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f016bfff700 1 -- 172.21.15.61:0/135456536 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016400b7e0 con 0x7f016c0b18a0 2022-07-20T03:12:49.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.575+0000 7f016bfff700 1 -- 172.21.15.61:0/135456536 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016400b990 con 0x7f016c0b18a0 2022-07-20T03:12:49.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 -- 172.21.15.61:0/135456536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0b18a0 msgr2=0x7f016c13eba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 --2- 172.21.15.61:0/135456536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0b18a0 0x7f016c13eba0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f01640032f0 tx=0x7f0164009290).stop 2022-07-20T03:12:49.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 -- 172.21.15.61:0/135456536 shutdown_connections 2022-07-20T03:12:49.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 --2- 172.21.15.61:0/135456536 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c142720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 --2- 172.21.15.61:0/135456536 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0b18a0 0x7f016c13eba0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 --2- 172.21.15.61:0/135456536 >> v2:172.21.15.117:3300/0 conn(0x7f016c0683d0 0x7f016c0b0380 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 -- 172.21.15.61:0/135456536 >> 172.21.15.61:0/135456536 conn(0x7f016c062f30 msgr2=0x7f016c063340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:49.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 -- 172.21.15.61:0/135456536 shutdown_connections 2022-07-20T03:12:49.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 -- 172.21.15.61:0/135456536 wait complete. 2022-07-20T03:12:49.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 Processor -- start 2022-07-20T03:12:49.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.579+0000 7f01739a0700 1 -- start start 2022-07-20T03:12:49.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 0x7f016c1551f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f016c0b18a0 0x7f016c145780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c067570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c1463f0 con 0x7f016c0683d0 2022-07-20T03:12:49.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f016c147700 con 0x7f016c0b18a0 2022-07-20T03:12:49.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f016c145c70 con 0x7f016c0b2540 2022-07-20T03:12:49.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 0x7f016c1551f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 0x7f016c1551f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45808/0 (socket says 172.21.15.61:45808) 2022-07-20T03:12:49.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 -- 172.21.15.61:0/2086602267 learned_addr learned my addr 172.21.15.61:0/2086602267 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:49.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f0170f3b700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f016c0b18a0 0x7f016c145780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f0171f3d700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c067570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 -- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 msgr2=0x7f016c067570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c067570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 -- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f016c0b18a0 msgr2=0x7f016c145780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f016c0b18a0 0x7f016c145780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 -- 172.21.15.61:0/2086602267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f016000b180 con 0x7f016c0683d0 2022-07-20T03:12:49.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f017173c700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 0x7f016c1551f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f016000c000 tx=0x7f0160006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016000c890 con 0x7f016c0683d0 2022-07-20T03:12:49.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f016000c9f0 con 0x7f016c0683d0 2022-07-20T03:12:49.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0164009530 con 0x7f016c0683d0 2022-07-20T03:12:49.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f016000cd50 con 0x7f016c0683d0 2022-07-20T03:12:49.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f016c067ae0 con 0x7f016c0683d0 2022-07-20T03:12:49.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.583+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f016c142280 con 0x7f016c0683d0 2022-07-20T03:12:49.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.587+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f016001c460 con 0x7f016c0683d0 2022-07-20T03:12:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.587+0000 7f016a7fc700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f014c0539b0 0x7f014c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.587+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f016006c180 con 0x7f016c0683d0 2022-07-20T03:12:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.587+0000 7f0170f3b700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f014c0539b0 0x7f014c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.587+0000 7f0170f3b700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f014c0539b0 0x7f014c055e70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f016401a000 tx=0x7f0164017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.595+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f01600433b0 con 0x7f016c0683d0 2022-07-20T03:12:49.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.723+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f016c0574e0 con 0x7f016c0683d0 2022-07-20T03:12:49.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.723+0000 7f016a7fc700 1 -- 172.21.15.61:0/2086602267 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v57) v1 ==== 74+0+13230 (secure 0 0 0) 0x7f01600469d0 con 0x7f016c0683d0 2022-07-20T03:12:49.733 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:49.736 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":57,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:12:48.669473+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":10,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":50,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:12:49.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f014c0539b0 msgr2=0x7f014c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f014c0539b0 0x7f014c055e70 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f016401a000 tx=0x7f0164017000).stop 2022-07-20T03:12:49.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 msgr2=0x7f016c1551f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 0x7f016c1551f0 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f016000c000 tx=0x7f0160006df0).stop 2022-07-20T03:12:49.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 shutdown_connections 2022-07-20T03:12:49.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f014c0539b0 0x7f014c055e70 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f016c0b18a0 0x7f016c145780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f016c0b2540 0x7f016c067570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 --2- 172.21.15.61:0/2086602267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f016c0683d0 0x7f016c1551f0 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 >> 172.21.15.61:0/2086602267 conn(0x7f016c062f30 msgr2=0x7f016c0ae3d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:49.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 shutdown_connections 2022-07-20T03:12:49.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.727+0000 7f01739a0700 1 -- 172.21.15.61:0/2086602267 wait complete. 2022-07-20T03:12:49.749 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:12:49.750 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:12:49.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.867+0000 7f933b2d5700 1 Processor -- start 2022-07-20T03:12:49.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.867+0000 7f933b2d5700 1 -- start start 2022-07-20T03:12:49.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f93340b9da0 0x7f93341468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93340b3cf0 0x7f9334142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9334142f40 0x7f933405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340ad5b0 con 0x7f93340b3cf0 2022-07-20T03:12:49.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93340ba180 con 0x7f93340b9da0 2022-07-20T03:12:49.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93341461b0 con 0x7f9334142f40 2022-07-20T03:12:49.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93340b3cf0 0x7f9334142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93340b3cf0 0x7f9334142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45818/0 (socket says 172.21.15.61:45818) 2022-07-20T03:12:49.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 -- 172.21.15.61:0/1370407430 learned_addr learned my addr 172.21.15.61:0/1370407430 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:49.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933aad4700 1 --2- 172.21.15.61:0/1370407430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9334142f40 0x7f933405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933a2d3700 1 --2- 172.21.15.61:0/1370407430 >> v2:172.21.15.117:3300/0 conn(0x7f93340b9da0 0x7f93341468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 -- 172.21.15.61:0/1370407430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9334142f40 msgr2=0x7f933405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:49.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 --2- 172.21.15.61:0/1370407430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9334142f40 0x7f933405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 -- 172.21.15.61:0/1370407430 >> v2:172.21.15.117:3300/0 conn(0x7f93340b9da0 msgr2=0x7f93341468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:49.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 --2- 172.21.15.61:0/1370407430 >> v2:172.21.15.117:3300/0 conn(0x7f93340b9da0 0x7f93341468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 -- 172.21.15.61:0/1370407430 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9334146450 con 0x7f93340b3cf0 2022-07-20T03:12:49.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933a2d3700 1 --2- 172.21.15.61:0/1370407430 >> v2:172.21.15.117:3300/0 conn(0x7f93340b9da0 0x7f93341468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:12:49.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f9339ad2700 1 --2- 172.21.15.61:0/1370407430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f9334142a50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f9330002f70 tx=0x7f9330006eb0).ready entity=mon.0 client_cookie=dd4dbaf4bf26dd50 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f93392d1700 1 -- 172.21.15.61:0/1370407430 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f933000d570 con 0x7f93340b3cf0 2022-07-20T03:12:49.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f93392d1700 1 -- 172.21.15.61:0/1370407430 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9330000e90 con 0x7f93340b3cf0 2022-07-20T03:12:49.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f93392d1700 1 -- 172.21.15.61:0/1370407430 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f933000d900 con 0x7f93340b3cf0 2022-07-20T03:12:49.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- 172.21.15.61:0/1370407430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 msgr2=0x7f9334142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- 172.21.15.61:0/1370407430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f9334142a50 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7f9330002f70 tx=0x7f9330006eb0).stop 2022-07-20T03:12:49.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- 172.21.15.61:0/1370407430 shutdown_connections 2022-07-20T03:12:49.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- 172.21.15.61:0/1370407430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9334142f40 0x7f933405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- 172.21.15.61:0/1370407430 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f9334142a50 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 --2- 172.21.15.61:0/1370407430 >> v2:172.21.15.117:3300/0 conn(0x7f93340b9da0 0x7f93341468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- 172.21.15.61:0/1370407430 >> 172.21.15.61:0/1370407430 conn(0x7f93340678c0 msgr2=0x7f93340b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:49.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- 172.21.15.61:0/1370407430 shutdown_connections 2022-07-20T03:12:49.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.871+0000 7f933b2d5700 1 -- 172.21.15.61:0/1370407430 wait complete. 2022-07-20T03:12:49.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 Processor -- start 2022-07-20T03:12:49.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 -- start start 2022-07-20T03:12:49.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f93341442c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93340b9da0 0x7f93341453e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9334142f40 0x7f933413d7f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93341554b0 con 0x7f93340b3cf0 2022-07-20T03:12:49.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9334155750 con 0x7f9334142f40 2022-07-20T03:12:49.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933b2d5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93341559f0 con 0x7f93340b9da0 2022-07-20T03:12:49.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f93341442c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f93341442c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45822/0 (socket says 172.21.15.61:45822) 2022-07-20T03:12:49.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 -- 172.21.15.61:0/606746385 learned_addr learned my addr 172.21.15.61:0/606746385 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:49.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f9339ad2700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93340b9da0 0x7f93341453e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933aad4700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9334142f40 0x7f933413d7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 -- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93340b9da0 msgr2=0x7f93341453e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93340b9da0 0x7f93341453e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 -- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9334142f40 msgr2=0x7f933413d7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:49.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9334142f40 0x7f933413d7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:49.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 -- 172.21.15.61:0/606746385 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f933000b1e0 con 0x7f93340b3cf0 2022-07-20T03:12:49.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f933a2d3700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f93341442c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f9324007000 tx=0x7f9324008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.875+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9324010580 con 0x7f93340b3cf0 2022-07-20T03:12:49.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93240106e0 con 0x7f93340b3cf0 2022-07-20T03:12:49.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93341551a0 con 0x7f93340b3cf0 2022-07-20T03:12:49.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9324010a40 con 0x7f93340b3cf0 2022-07-20T03:12:49.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93341458d0 con 0x7f93340b3cf0 2022-07-20T03:12:49.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9334155bb0 con 0x7f93340b3cf0 2022-07-20T03:12:49.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9324010ba0 con 0x7f93340b3cf0 2022-07-20T03:12:49.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f93237fe700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9314053920 0x7f9314055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:49.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f932406bc90 con 0x7f93340b3cf0 2022-07-20T03:12:49.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.879+0000 7f9339ad2700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9314053920 0x7f9314055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:49.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.883+0000 7f9339ad2700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9314053920 0x7f9314055de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f933001c000 tx=0x7f9330015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:49.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:49.887+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9324042dc0 con 0x7f93340b3cf0 2022-07-20T03:12:50.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f93340574e0 con 0x7f93340b3cf0 2022-07-20T03:12:50.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f93237fe700 1 -- 172.21.15.61:0/606746385 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v57) v1 ==== 110+0+10 (secure 0 0 0) 0x7f93240463e0 con 0x7f93340b3cf0 2022-07-20T03:12:50.021 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:12:50.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9314053920 msgr2=0x7f9314055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9314053920 0x7f9314055de0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f933001c000 tx=0x7f9330015000).stop 2022-07-20T03:12:50.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 msgr2=0x7f93341442c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f93341442c0 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f9324007000 tx=0x7f9324008040).stop 2022-07-20T03:12:50.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 shutdown_connections 2022-07-20T03:12:50.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9314053920 0x7f9314055de0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9334142f40 0x7f933413d7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93340b9da0 0x7f93341453e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 --2- 172.21.15.61:0/606746385 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93340b3cf0 0x7f93341442c0 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.015+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 >> 172.21.15.61:0/606746385 conn(0x7f93340678c0 msgr2=0x7f93340b9330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:50.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.019+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 shutdown_connections 2022-07-20T03:12:50.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.019+0000 7f933b2d5700 1 -- 172.21.15.61:0/606746385 wait complete. 2022-07-20T03:12:50.044 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:12:50.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.163+0000 7fdd97385700 1 Processor -- start 2022-07-20T03:12:50.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 -- start start 2022-07-20T03:12:50.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd9005fa50 0x7fdd90062160 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdd9005ce90 0x7fdd90143220 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd9005d2c0 con 0x7fdd900b2550 2022-07-20T03:12:50.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd900b2930 con 0x7fdd9005fa50 2022-07-20T03:12:50.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd90061780 con 0x7fdd9005ce90 2022-07-20T03:12:50.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd9005fa50 0x7fdd90062160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd9005fa50 0x7fdd90062160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45830/0 (socket says 172.21.15.61:45830) 2022-07-20T03:12:50.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 -- 172.21.15.61:0/3195532453 learned_addr learned my addr 172.21.15.61:0/3195532453 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:50.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd95922700 1 --2- 172.21.15.61:0/3195532453 >> v2:172.21.15.117:3300/0 conn(0x7fdd9005ce90 0x7fdd90143220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 -- 172.21.15.61:0/3195532453 >> v2:172.21.15.117:3300/0 conn(0x7fdd9005ce90 msgr2=0x7fdd90143220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:50.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 --2- 172.21.15.61:0/3195532453 >> v2:172.21.15.117:3300/0 conn(0x7fdd9005ce90 0x7fdd90143220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 -- 172.21.15.61:0/3195532453 >> v1:172.21.15.117:6789/0 conn(0x7fdd900b2550 legacy=0x7fdd90146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 -- 172.21.15.61:0/3195532453 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd90061a20 con 0x7fdd9005fa50 2022-07-20T03:12:50.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd95922700 1 --2- 172.21.15.61:0/3195532453 >> v2:172.21.15.117:3300/0 conn(0x7fdd9005ce90 0x7fdd90143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:50.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd94920700 1 --2- 172.21.15.61:0/3195532453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005fa50 0x7fdd90062160 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fdd8c002f70 tx=0x7fdd8c006eb0).ready entity=mon.0 client_cookie=eb4fb6eacfa8a2ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd83fff700 1 -- 172.21.15.61:0/3195532453 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd8c00d5c0 con 0x7fdd9005fa50 2022-07-20T03:12:50.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd83fff700 1 -- 172.21.15.61:0/3195532453 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd8c000e90 con 0x7fdd9005fa50 2022-07-20T03:12:50.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd83fff700 1 -- 172.21.15.61:0/3195532453 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd8c00d950 con 0x7fdd9005fa50 2022-07-20T03:12:50.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 -- 172.21.15.61:0/3195532453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005fa50 msgr2=0x7fdd90062160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.167+0000 7fdd97385700 1 --2- 172.21.15.61:0/3195532453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005fa50 0x7fdd90062160 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7fdd8c002f70 tx=0x7fdd8c006eb0).stop 2022-07-20T03:12:50.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- 172.21.15.61:0/3195532453 shutdown_connections 2022-07-20T03:12:50.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 --2- 172.21.15.61:0/3195532453 >> v2:172.21.15.117:3300/0 conn(0x7fdd9005ce90 0x7fdd90143220 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 --2- 172.21.15.61:0/3195532453 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005fa50 0x7fdd90062160 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- 172.21.15.61:0/3195532453 >> 172.21.15.61:0/3195532453 conn(0x7fdd900b36c0 msgr2=0x7fdd900b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:50.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- 172.21.15.61:0/3195532453 shutdown_connections 2022-07-20T03:12:50.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- 172.21.15.61:0/3195532453 wait complete. 2022-07-20T03:12:50.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 Processor -- start 2022-07-20T03:12:50.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- start start 2022-07-20T03:12:50.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 0x7fdd90155530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd9005fa50 0x7fdd90144830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd900b2550 0x7fdd90096570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90099200 con 0x7fdd9005ce90 2022-07-20T03:12:50.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd900994a0 con 0x7fdd900b2550 2022-07-20T03:12:50.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd97385700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd90145470 con 0x7fdd9005fa50 2022-07-20T03:12:50.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 0x7fdd90155530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 0x7fdd90155530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45834/0 (socket says 172.21.15.61:45834) 2022-07-20T03:12:50.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 -- 172.21.15.61:0/3675050152 learned_addr learned my addr 172.21.15.61:0/3675050152 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:50.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd94920700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd9005fa50 0x7fdd90144830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95922700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd900b2550 0x7fdd90096570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 -- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd9005fa50 msgr2=0x7fdd90144830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd9005fa50 0x7fdd90144830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 -- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd900b2550 msgr2=0x7fdd90096570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd900b2550 0x7fdd90096570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 -- 172.21.15.61:0/3675050152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd88009470 con 0x7fdd9005ce90 2022-07-20T03:12:50.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.171+0000 7fdd95121700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 0x7fdd90155530 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fdd900632b0 tx=0x7fdd84007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd84012400 con 0x7fdd9005ce90 2022-07-20T03:12:50.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd84012560 con 0x7fdd9005ce90 2022-07-20T03:12:50.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd8c00b1e0 con 0x7fdd9005ce90 2022-07-20T03:12:50.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd840128c0 con 0x7fdd9005ce90 2022-07-20T03:12:50.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd9009b540 con 0x7fdd9005ce90 2022-07-20T03:12:50.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd9009a590 con 0x7fdd9005ce90 2022-07-20T03:12:50.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdd84019030 con 0x7fdd9005ce90 2022-07-20T03:12:50.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd827fc700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd78053a60 0x7fdd78055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdd8406ae20 con 0x7fdd9005ce90 2022-07-20T03:12:50.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd94920700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd78053a60 0x7fdd78055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.175+0000 7fdd94920700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd78053a60 0x7fdd78055f20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fdd8c01c000 tx=0x7fdd8c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.183+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd9009a590 con 0x7fdd9005ce90 2022-07-20T03:12:50.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.307+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fdd9009a590 con 0x7fdd9005ce90 2022-07-20T03:12:50.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd827fc700 1 -- 172.21.15.61:0/3675050152 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v57) v1 ==== 92+0+10 (secure 0 0 0) 0x7fdd9009a590 con 0x7fdd9005ce90 2022-07-20T03:12:50.319 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:12:50.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd78053a60 msgr2=0x7fdd78055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd78053a60 0x7fdd78055f20 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fdd8c01c000 tx=0x7fdd8c015000).stop 2022-07-20T03:12:50.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 msgr2=0x7fdd90155530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 0x7fdd90155530 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7fdd900632b0 tx=0x7fdd84007c50).stop 2022-07-20T03:12:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 shutdown_connections 2022-07-20T03:12:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd78053a60 0x7fdd78055f20 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd900b2550 0x7fdd90096570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd9005fa50 0x7fdd90144830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 --2- 172.21.15.61:0/3675050152 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd9005ce90 0x7fdd90155530 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 >> 172.21.15.61:0/3675050152 conn(0x7fdd900b36c0 msgr2=0x7fdd900afd80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 shutdown_connections 2022-07-20T03:12:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.311+0000 7fdd97385700 1 -- 172.21.15.61:0/3675050152 wait complete. 2022-07-20T03:12:50.334 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:12:50.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 Processor -- start 2022-07-20T03:12:50.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 -- start start 2022-07-20T03:12:50.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a40b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4a405bd90 0x7fe4a4142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4a40aea30 con 0x7fe4a40b1200 2022-07-20T03:12:50.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4a4142970 con 0x7fe4a405bd90 2022-07-20T03:12:50.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4aa01d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a40aec60 con 0x7fe4a40b5650 2022-07-20T03:12:50.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a981c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4a405bd90 0x7fe4a4142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a981c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe4a405bd90 0x7fe4a4142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45846/0 (socket says 172.21.15.61:45846) 2022-07-20T03:12:50.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a981c700 1 -- 172.21.15.61:0/282477811 learned_addr learned my addr 172.21.15.61:0/282477811 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:50.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a901b700 1 --2- 172.21.15.61:0/282477811 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a40b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a981c700 1 -- 172.21.15.61:0/282477811 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 msgr2=0x7fe4a40b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:50.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a981c700 1 --2- 172.21.15.61:0/282477811 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a40b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.447+0000 7fe4a981c700 1 -- 172.21.15.61:0/282477811 >> v1:172.21.15.117:6789/0 conn(0x7fe4a40b1200 legacy=0x7fe4a405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4a981c700 1 -- 172.21.15.61:0/282477811 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4a40af180 con 0x7fe4a405bd90 2022-07-20T03:12:50.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4a981c700 1 --2- 172.21.15.61:0/282477811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a405bd90 0x7fe4a4142d60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fe49c0032f0 tx=0x7fe49c009290).ready entity=mon.0 client_cookie=7c8b798edf871855 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe493fff700 1 -- 172.21.15.61:0/282477811 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe49c00b8a0 con 0x7fe4a405bd90 2022-07-20T03:12:50.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe493fff700 1 -- 172.21.15.61:0/282477811 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe49c00ba00 con 0x7fe4a405bd90 2022-07-20T03:12:50.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe493fff700 1 -- 172.21.15.61:0/282477811 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe49c00bb70 con 0x7fe4a405bd90 2022-07-20T03:12:50.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 -- 172.21.15.61:0/282477811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a405bd90 msgr2=0x7fe4a4142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/282477811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a405bd90 0x7fe4a4142d60 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fe49c0032f0 tx=0x7fe49c009290).stop 2022-07-20T03:12:50.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 -- 172.21.15.61:0/282477811 shutdown_connections 2022-07-20T03:12:50.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/282477811 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a405bd90 0x7fe4a4142d60 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/282477811 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a40b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 -- 172.21.15.61:0/282477811 >> 172.21.15.61:0/282477811 conn(0x7fe4a40634e0 msgr2=0x7fe4a40b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:50.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 -- 172.21.15.61:0/282477811 shutdown_connections 2022-07-20T03:12:50.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.451+0000 7fe4aa01d700 1 -- 172.21.15.61:0/282477811 wait complete. 2022-07-20T03:12:50.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 Processor -- start 2022-07-20T03:12:50.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 -- start start 2022-07-20T03:12:50.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4a405bd90 0x7fe4a4146310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 0x7fe4a4145270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a413f480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4138eb0 con 0x7fe4a40b1200 2022-07-20T03:12:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a413d890 con 0x7fe4a405bd90 2022-07-20T03:12:50.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4a4147960 con 0x7fe4a40b5650 2022-07-20T03:12:50.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 0x7fe4a4145270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 0x7fe4a4145270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45850/0 (socket says 172.21.15.61:45850) 2022-07-20T03:12:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a901b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4a405bd90 0x7fe4a4146310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 -- 172.21.15.61:0/3946107465 learned_addr learned my addr 172.21.15.61:0/3946107465 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:50.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a981c700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a413f480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 -- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 msgr2=0x7fe4a413f480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a413f480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 -- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4a405bd90 msgr2=0x7fe4a4146310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4a405bd90 0x7fe4a4146310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 -- 172.21.15.61:0/3946107465 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe49c009530 con 0x7fe4a40b1200 2022-07-20T03:12:50.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4a881a700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 0x7fe4a4145270 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fe4a40b0c00 tx=0x7fe494007ca0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe494012400 con 0x7fe4a40b1200 2022-07-20T03:12:50.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe494012560 con 0x7fe4a40b1200 2022-07-20T03:12:50.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4a413d3f0 con 0x7fe4a40b1200 2022-07-20T03:12:50.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4940128c0 con 0x7fe4a40b1200 2022-07-20T03:12:50.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.455+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4a4145e70 con 0x7fe4a40b1200 2022-07-20T03:12:50.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.459+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4a4147b60 con 0x7fe4a40b1200 2022-07-20T03:12:50.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.459+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe494019030 con 0x7fe4a40b1200 2022-07-20T03:12:50.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.459+0000 7fe4927fc700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe484053a60 0x7fe484055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.459+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fe49406aeb0 con 0x7fe4a40b1200 2022-07-20T03:12:50.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.459+0000 7fe4a901b700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe484053a60 0x7fe484055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.459+0000 7fe4a901b700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe484053a60 0x7fe484055f20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fe4a000c000 tx=0x7fe4a0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.463+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe494042030 con 0x7fe4a40b1200 2022-07-20T03:12:50.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.591+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fe4a413d580 con 0x7fe4a40b1200 2022-07-20T03:12:50.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.591+0000 7fe4927fc700 1 -- 172.21.15.61:0/3946107465 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v57) v1 ==== 84+0+143 (secure 0 0 0) 0x7fe494045650 con 0x7fe4a40b1200 2022-07-20T03:12:50.658 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:50.659 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:12:50.659 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:50.659 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:12:50.660 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:12:50.660 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:12:50.660 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:12:50.660 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:12:50.660 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:12:50.661 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:12:50.661 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:12:50.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe484053a60 msgr2=0x7fe484055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe484053a60 0x7fe484055f20 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fe4a000c000 tx=0x7fe4a0008040).stop 2022-07-20T03:12:50.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 msgr2=0x7fe4a4145270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 0x7fe4a4145270 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7fe4a40b0c00 tx=0x7fe494007ca0).stop 2022-07-20T03:12:50.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 shutdown_connections 2022-07-20T03:12:50.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe484053a60 0x7fe484055f20 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4a40b5650 0x7fe4a413f480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4a40b1200 0x7fe4a4145270 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 --2- 172.21.15.61:0/3946107465 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4a405bd90 0x7fe4a4146310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 >> 172.21.15.61:0/3946107465 conn(0x7fe4a40634e0 msgr2=0x7fe4a413bcc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:50.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 shutdown_connections 2022-07-20T03:12:50.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.595+0000 7fe4aa01d700 1 -- 172.21.15.61:0/3946107465 wait complete. 2022-07-20T03:12:50.673 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:12:50.674 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:12:50.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.811+0000 7fb531426700 1 Processor -- start 2022-07-20T03:12:50.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.811+0000 7fb531426700 1 -- start start 2022-07-20T03:12:50.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb52c09df40 0x7fb52c145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c0607e0 con 0x7fb52c060ee0 2022-07-20T03:12:50.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb52c060a80 con 0x7fb52c0b4d60 2022-07-20T03:12:50.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb52c060d20 con 0x7fb52c09df40 2022-07-20T03:12:50.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb52c060ee0 0x7fb52c09cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42458/0 (socket says 172.21.15.61:42458) 2022-07-20T03:12:50.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52b7fe700 1 -- 172.21.15.61:0/678044524 learned_addr learned my addr 172.21.15.61:0/678044524 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:50.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52affd700 1 --2- 172.21.15.61:0/678044524 >> v2:172.21.15.117:3300/0 conn(0x7fb52c09df40 0x7fb52c145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb529ffb700 1 -- 172.21.15.61:0/678044524 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4218714607 0 0) 0x7fb52c0607e0 con 0x7fb52c060ee0 2022-07-20T03:12:50.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb529ffb700 1 -- 172.21.15.61:0/678044524 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb50c003260 con 0x7fb52c060ee0 2022-07-20T03:12:50.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52affd700 1 -- 172.21.15.61:0/678044524 >> v1:172.21.15.117:6789/0 conn(0x7fb52c0b4d60 legacy=0x7fb52c145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52affd700 1 -- 172.21.15.61:0/678044524 >> v1:172.21.15.61:6789/0 conn(0x7fb52c060ee0 legacy=0x7fb52c09cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb529ffb700 1 -- 172.21.15.61:0/678044524 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4169001324 0 0) 0x7fb50c003260 con 0x7fb52c060ee0 2022-07-20T03:12:50.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52affd700 1 -- 172.21.15.61:0/678044524 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb52c09e320 con 0x7fb52c09df40 2022-07-20T03:12:50.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb52affd700 1 --2- 172.21.15.61:0/678044524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c145680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fb524001b80 tx=0x7fb51c006d90).ready entity=mon.1 client_cookie=cced86fa48be54f5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb529ffb700 1 -- 172.21.15.61:0/678044524 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5240019e0 con 0x7fb52c09df40 2022-07-20T03:12:50.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb529ffb700 1 -- 172.21.15.61:0/678044524 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb51c000e90 con 0x7fb52c09df40 2022-07-20T03:12:50.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb529ffb700 1 -- 172.21.15.61:0/678044524 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb51c00e470 con 0x7fb52c09df40 2022-07-20T03:12:50.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- 172.21.15.61:0/678044524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 msgr2=0x7fb52c145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 --2- 172.21.15.61:0/678044524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c145680 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fb524001b80 tx=0x7fb51c006d90).stop 2022-07-20T03:12:50.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- 172.21.15.61:0/678044524 shutdown_connections 2022-07-20T03:12:50.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 --2- 172.21.15.61:0/678044524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c145680 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- 172.21.15.61:0/678044524 >> 172.21.15.61:0/678044524 conn(0x7fb52c0b5f30 msgr2=0x7fb52c0b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:50.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- 172.21.15.61:0/678044524 shutdown_connections 2022-07-20T03:12:50.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.815+0000 7fb531426700 1 -- 172.21.15.61:0/678044524 wait complete. 2022-07-20T03:12:50.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 Processor -- start 2022-07-20T03:12:50.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 -- start start 2022-07-20T03:12:50.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 0x7fb52c1439a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c09b0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 0x7fb52c09b5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0b20b0 con 0x7fb52c0b4d60 2022-07-20T03:12:50.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c09bab0 con 0x7fb52c09df40 2022-07-20T03:12:50.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb531426700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb52c0b06a0 con 0x7fb52c060ee0 2022-07-20T03:12:50.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 0x7fb52c09b5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52a7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c09b0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52affd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 0x7fb52c1439a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 0x7fb52c09b5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45866/0 (socket says 172.21.15.61:45866) 2022-07-20T03:12:50.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52a7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c09b0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55544/0 (socket says 172.21.15.61:55544) 2022-07-20T03:12:50.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52affd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 0x7fb52c1439a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:60766/0 (socket says 172.21.15.61:60766) 2022-07-20T03:12:50.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.819+0000 7fb52b7fe700 1 -- 172.21.15.61:0/2821417051 learned_addr learned my addr 172.21.15.61:0/2821417051 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:50.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52b7fe700 1 -- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 msgr2=0x7fb52c1439a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52b7fe700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 0x7fb52c1439a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52b7fe700 1 -- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 msgr2=0x7fb52c09b0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:50.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52b7fe700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c09b0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:50.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52b7fe700 1 -- 172.21.15.61:0/2821417051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb51c00b120 con 0x7fb52c0b4d60 2022-07-20T03:12:50.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52affd700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 0x7fb52c1439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:50.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb52b7fe700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 0x7fb52c09b5c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fb524001c40 tx=0x7fb5240021a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb524004ab0 con 0x7fb52c0b4d60 2022-07-20T03:12:50.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb524004c10 con 0x7fb52c0b4d60 2022-07-20T03:12:50.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb52c0b0940 con 0x7fb52c0b4d60 2022-07-20T03:12:50.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb524007170 con 0x7fb52c0b4d60 2022-07-20T03:12:50.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb52c155590 con 0x7fb52c0b4d60 2022-07-20T03:12:50.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.823+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb52c0574e0 con 0x7fb52c0b4d60 2022-07-20T03:12:50.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.827+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb52401b070 con 0x7fb52c0b4d60 2022-07-20T03:12:50.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.827+0000 7fb523fff700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb510053920 0x7fb510055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:50.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.827+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(57..57 src has 1..57) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb52406b450 con 0x7fb52c0b4d60 2022-07-20T03:12:50.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.827+0000 7fb52affd700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb510053920 0x7fb510055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:50.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.827+0000 7fb52affd700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb510053920 0x7fb510055de0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb51c01a000 tx=0x7fb51c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:50.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.831+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb524042680 con 0x7fb52c0b4d60 2022-07-20T03:12:50.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:50.955+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb52c061810 con 0x7fb52c0b4d60 2022-07-20T03:12:51.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.279+0000 7fb523fff700 1 -- 172.21.15.61:0/2821417051 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v58) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb524045ca0 con 0x7fb52c0b4d60 2022-07-20T03:12:51.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb510053920 msgr2=0x7fb510055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb510053920 0x7fb510055de0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7fb51c01a000 tx=0x7fb51c014000).stop 2022-07-20T03:12:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 msgr2=0x7fb52c09b5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 0x7fb52c09b5c0 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fb524001c40 tx=0x7fb5240021a0).stop 2022-07-20T03:12:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 shutdown_connections 2022-07-20T03:12:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb510053920 0x7fb510055de0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb52c09df40 0x7fb52c09b0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb52c0b4d60 0x7fb52c09b5c0 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 --2- 172.21.15.61:0/2821417051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb52c060ee0 0x7fb52c1439a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 >> 172.21.15.61:0/2821417051 conn(0x7fb52c0b5f30 msgr2=0x7fb52c144860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 shutdown_connections 2022-07-20T03:12:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.283+0000 7fb531426700 1 -- 172.21.15.61:0/2821417051 wait complete. 2022-07-20T03:12:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:12:51.306 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:12:51.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.415+0000 7f47555f2700 1 Processor -- start 2022-07-20T03:12:51.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.415+0000 7f47555f2700 1 -- start start 2022-07-20T03:12:51.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f47500b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500b3f10 con 0x7f47501461c0 2022-07-20T03:12:51.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f47500b3790 con 0x7f47500b2ec0 2022-07-20T03:12:51.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47500b4ea0 con 0x7f475005d160 2022-07-20T03:12:51.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f4754df1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f47501461c0 0x7f4750146650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42472/0 (socket says 172.21.15.61:42472) 2022-07-20T03:12:51.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f4754df1700 1 -- 172.21.15.61:0/3290118124 learned_addr learned my addr 172.21.15.61:0/3290118124 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:51.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474f7fe700 1 --2- 172.21.15.61:0/3290118124 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f47500b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4054628747 0 0) 0x7f47500b3790 con 0x7f47500b2ec0 2022-07-20T03:12:51.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4734003260 con 0x7f47500b2ec0 2022-07-20T03:12:51.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 742281677 0 0) 0x7f47500b3f10 con 0x7f47501461c0 2022-07-20T03:12:51.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f47500b3790 con 0x7f47501461c0 2022-07-20T03:12:51.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474f7fe700 1 -- 172.21.15.61:0/3290118124 >> v1:172.21.15.117:6789/0 conn(0x7f47500b2ec0 legacy=0x7f4750142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474f7fe700 1 -- 172.21.15.61:0/3290118124 >> v1:172.21.15.61:6789/0 conn(0x7f47501461c0 legacy=0x7f4750146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474f7fe700 1 -- 172.21.15.61:0/3290118124 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47500b2620 con 0x7f475005d160 2022-07-20T03:12:51.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474f7fe700 1 --2- 172.21.15.61:0/3290118124 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f47500b32a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4738002f70 tx=0x7f4738006df0).ready entity=mon.2 client_cookie=c1885a393798967b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:51.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f473800d5c0 con 0x7f475005d160 2022-07-20T03:12:51.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4738000e90 con 0x7f475005d160 2022-07-20T03:12:51.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f474effd700 1 -- 172.21.15.61:0/3290118124 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f473800d910 con 0x7f475005d160 2022-07-20T03:12:51.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- 172.21.15.61:0/3290118124 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 msgr2=0x7f47500b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 --2- 172.21.15.61:0/3290118124 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f47500b32a0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f4738002f70 tx=0x7f4738006df0).stop 2022-07-20T03:12:51.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- 172.21.15.61:0/3290118124 shutdown_connections 2022-07-20T03:12:51.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 --2- 172.21.15.61:0/3290118124 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f47500b32a0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- 172.21.15.61:0/3290118124 >> 172.21.15.61:0/3290118124 conn(0x7f4750066e20 msgr2=0x7f4750067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:51.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- 172.21.15.61:0/3290118124 shutdown_connections 2022-07-20T03:12:51.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.419+0000 7f47555f2700 1 -- 172.21.15.61:0/3290118124 wait complete. 2022-07-20T03:12:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 Processor -- start 2022-07-20T03:12:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- start start 2022-07-20T03:12:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f4750156210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 0x7f47500b5180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47501461c0 0x7f4750152860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750152420 con 0x7f47500b2ec0 2022-07-20T03:12:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4750145190 con 0x7f47501461c0 2022-07-20T03:12:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4750140080 con 0x7f475005d160 2022-07-20T03:12:51.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 0x7f47500b5180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 0x7f47500b5180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45876/0 (socket says 172.21.15.61:45876) 2022-07-20T03:12:51.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 -- 172.21.15.61:0/1885323829 learned_addr learned my addr 172.21.15.61:0/1885323829 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:51.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f4754df1700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47501461c0 0x7f4750152860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474ffff700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f4750156210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 -- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 msgr2=0x7f4750156210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f4750156210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 -- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47501461c0 msgr2=0x7f4750152860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47501461c0 0x7f4750152860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 -- 172.21.15.61:0/1885323829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f473800b180 con 0x7f47500b2ec0 2022-07-20T03:12:51.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f4754df1700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47501461c0 0x7f4750152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:51.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474ffff700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f4750156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:12:51.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474f7fe700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 0x7f47500b5180 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f4738002f70 tx=0x7f4738014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:51.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4738016070 con 0x7f47500b2ec0 2022-07-20T03:12:51.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4750141970 con 0x7f47500b2ec0 2022-07-20T03:12:51.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f47380139c0 con 0x7f47500b2ec0 2022-07-20T03:12:51.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f475013cba0 con 0x7f47500b2ec0 2022-07-20T03:12:51.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4738013d20 con 0x7f47500b2ec0 2022-07-20T03:12:51.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.423+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4750141c20 con 0x7f47500b2ec0 2022-07-20T03:12:51.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.427+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f473800f030 con 0x7f47500b2ec0 2022-07-20T03:12:51.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.427+0000 7f474d7fa700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4728053a40 0x7f4728055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.427+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(58..58 src has 1..58) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f473806cb80 con 0x7f47500b2ec0 2022-07-20T03:12:51.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.427+0000 7f474ffff700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4728053a40 0x7f4728055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.427+0000 7f474ffff700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4728053a40 0x7f4728055f00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4740008000 tx=0x7f4740009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:51.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.435+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4750141c20 con 0x7f47500b2ec0 2022-07-20T03:12:51.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.555+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f4750141c20 con 0x7f47500b2ec0 2022-07-20T03:12:51.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.643+0000 7f474d7fa700 1 -- 172.21.15.61:0/1885323829 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v59) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4750141c20 con 0x7f47500b2ec0 2022-07-20T03:12:51.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4728053a40 msgr2=0x7f4728055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4728053a40 0x7f4728055f00 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f4740008000 tx=0x7f4740009040).stop 2022-07-20T03:12:51.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 msgr2=0x7f47500b5180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 0x7f47500b5180 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7f4738002f70 tx=0x7f4738014000).stop 2022-07-20T03:12:51.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 shutdown_connections 2022-07-20T03:12:51.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4728053a40 0x7f4728055f00 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47501461c0 0x7f4750152860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f47500b2ec0 0x7f47500b5180 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 --2- 172.21.15.61:0/1885323829 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f475005d160 0x7f4750156210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.647+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 >> 172.21.15.61:0/1885323829 conn(0x7f4750066e20 msgr2=0x7f47501420e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:51.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.651+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 shutdown_connections 2022-07-20T03:12:51.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.651+0000 7f47555f2700 1 -- 172.21.15.61:0/1885323829 wait complete. 2022-07-20T03:12:51.689 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:12:51.690 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:12:51.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.823+0000 7f84905de700 1 Processor -- start 2022-07-20T03:12:51.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.823+0000 7f84905de700 1 -- start start 2022-07-20T03:12:51.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f84880ac180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84881437e0 0x7f848813f920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f84880b3a50 con 0x7f848813fe10 2022-07-20T03:12:51.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f848813d3b0 con 0x7f84881437e0 2022-07-20T03:12:51.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84880ac670 con 0x7f84880b3670 2022-07-20T03:12:51.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84881437e0 0x7f848813f920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84881437e0 0x7f848813f920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45886/0 (socket says 172.21.15.61:45886) 2022-07-20T03:12:51.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 -- 172.21.15.61:0/3505850546 learned_addr learned my addr 172.21.15.61:0/3505850546 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:51.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848e37a700 1 --2- 172.21.15.61:0/3505850546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f84880ac180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 -- 172.21.15.61:0/3505850546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 msgr2=0x7f84880ac180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:51.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 --2- 172.21.15.61:0/3505850546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f84880ac180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 -- 172.21.15.61:0/3505850546 >> v1:172.21.15.117:6789/0 conn(0x7f848813fe10 legacy=0x7f8488099300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 -- 172.21.15.61:0/3505850546 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84881430a0 con 0x7f84881437e0 2022-07-20T03:12:51.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848db79700 1 --2- 172.21.15.61:0/3505850546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84881437e0 0x7f848813f920 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8478002f70 tx=0x7f8478006eb0).ready entity=mon.0 client_cookie=7172b1b3fc1836a5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:51.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848d378700 1 -- 172.21.15.61:0/3505850546 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f847800f970 con 0x7f84881437e0 2022-07-20T03:12:51.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848d378700 1 -- 172.21.15.61:0/3505850546 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478000e90 con 0x7f84881437e0 2022-07-20T03:12:51.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f848d378700 1 -- 172.21.15.61:0/3505850546 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f847800fd00 con 0x7f84881437e0 2022-07-20T03:12:51.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- 172.21.15.61:0/3505850546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84881437e0 msgr2=0x7f848813f920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 --2- 172.21.15.61:0/3505850546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84881437e0 0x7f848813f920 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f8478002f70 tx=0x7f8478006eb0).stop 2022-07-20T03:12:51.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- 172.21.15.61:0/3505850546 shutdown_connections 2022-07-20T03:12:51.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 --2- 172.21.15.61:0/3505850546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84881437e0 0x7f848813f920 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 --2- 172.21.15.61:0/3505850546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f84880ac180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- 172.21.15.61:0/3505850546 >> 172.21.15.61:0/3505850546 conn(0x7f84880b6c50 msgr2=0x7f84880b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:51.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- 172.21.15.61:0/3505850546 shutdown_connections 2022-07-20T03:12:51.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.827+0000 7f84905de700 1 -- 172.21.15.61:0/3505850546 wait complete. 2022-07-20T03:12:51.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 Processor -- start 2022-07-20T03:12:51.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 -- start start 2022-07-20T03:12:51.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f848813eef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 0x7f848809b3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84881437e0 0x7f8488151b60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f848813c150 con 0x7f848813fe10 2022-07-20T03:12:51.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8488152050 con 0x7f84881437e0 2022-07-20T03:12:51.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f84905de700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84880b0440 con 0x7f84880b3670 2022-07-20T03:12:51.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 0x7f848809b3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 0x7f848809b3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45892/0 (socket says 172.21.15.61:45892) 2022-07-20T03:12:51.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 -- 172.21.15.61:0/142388087 learned_addr learned my addr 172.21.15.61:0/142388087 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:51.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848e37a700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f848813eef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848eb7b700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84881437e0 0x7f8488151b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 -- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 msgr2=0x7f848813eef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f848813eef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 -- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84881437e0 msgr2=0x7f8488151b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:51.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84881437e0 0x7f8488151b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:51.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.831+0000 7f848db79700 1 -- 172.21.15.61:0/142388087 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847800b1e0 con 0x7f848813fe10 2022-07-20T03:12:51.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f848db79700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 0x7f848809b3a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f8488152190 tx=0x7f8478017000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:51.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8478020070 con 0x7f848813fe10 2022-07-20T03:12:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8478010e50 con 0x7f848813fe10 2022-07-20T03:12:51.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f847801a670 con 0x7f848813fe10 2022-07-20T03:12:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84880aff00 con 0x7f848813fe10 2022-07-20T03:12:51.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f848814dad0 con 0x7f848813fe10 2022-07-20T03:12:51.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f848813ec30 con 0x7f848813fe10 2022-07-20T03:12:51.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f847801a7d0 con 0x7f848813fe10 2022-07-20T03:12:51.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84777fe700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8468053a60 0x7f8468055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:51.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.835+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(59..59 src has 1..59) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f847806ede0 con 0x7f848813fe10 2022-07-20T03:12:51.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.839+0000 7f848e37a700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8468053a60 0x7f8468055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:51.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.839+0000 7f848e37a700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8468053a60 0x7f8468055f20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f848400b000 tx=0x7f8484006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:51.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:51.843+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f848813ec30 con 0x7f848813fe10 2022-07-20T03:12:52.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.007+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f848813ec30 con 0x7f848813fe10 2022-07-20T03:12:52.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.019+0000 7f84777fe700 1 -- 172.21.15.61:0/142388087 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 11 and data pool 12 v34) v1 ==== 189+0+0 (secure 0 0 0) 0x7f848813ec30 con 0x7f848813fe10 2022-07-20T03:12:52.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.019+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8468053a60 msgr2=0x7f8468055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.019+0000 7f84905de700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8468053a60 0x7f8468055f20 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f848400b000 tx=0x7f8484006df0).stop 2022-07-20T03:12:52.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.019+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 msgr2=0x7f848809b3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.019+0000 7f84905de700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 0x7f848809b3a0 secure :-1 s=READY pgs=599 cs=0 l=1 rev1=1 rx=0x7f8488152190 tx=0x7f8478017000).stop 2022-07-20T03:12:52.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 shutdown_connections 2022-07-20T03:12:52.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8468053a60 0x7f8468055f20 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84881437e0 0x7f8488151b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f848813fe10 0x7f848809b3a0 unknown :-1 s=CLOSED pgs=599 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 --2- 172.21.15.61:0/142388087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84880b3670 0x7f848813eef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 >> 172.21.15.61:0/142388087 conn(0x7f84880b6c50 msgr2=0x7f84880b4450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:52.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 shutdown_connections 2022-07-20T03:12:52.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.023+0000 7f84905de700 1 -- 172.21.15.61:0/142388087 wait complete. 2022-07-20T03:12:52.045 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 11 and data pool 12 2022-07-20T03:12:52.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:12:52.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efecc5700 1 Processor -- start 2022-07-20T03:12:52.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efecc5700 1 -- start start 2022-07-20T03:12:52.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efecc5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9ef80adff0 0x7f9ef80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efecc5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef8147010 con 0x7f9ef80af8e0 2022-07-20T03:12:52.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efecc5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ef81472b0 con 0x7f9ef80b5650 2022-07-20T03:12:52.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efecc5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ef8146d40 con 0x7f9ef80adff0 2022-07-20T03:12:52.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efe4c4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9ef80af8e0 0x7f9ef805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42514/0 (socket says 172.21.15.61:42514) 2022-07-20T03:12:52.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efe4c4700 1 -- 172.21.15.61:0/2884098008 learned_addr learned my addr 172.21.15.61:0/2884098008 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:52.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.167+0000 7f9efd4c2700 1 --2- 172.21.15.61:0/2884098008 >> v2:172.21.15.117:3300/0 conn(0x7f9ef80adff0 0x7f9ef80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efccc1700 1 -- 172.21.15.61:0/2884098008 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4234255540 0 0) 0x7f9ef8147010 con 0x7f9ef80af8e0 2022-07-20T03:12:52.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efccc1700 1 -- 172.21.15.61:0/2884098008 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9ee0003260 con 0x7f9ef80af8e0 2022-07-20T03:12:52.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efd4c2700 1 -- 172.21.15.61:0/2884098008 >> v1:172.21.15.117:6789/0 conn(0x7f9ef80b5650 legacy=0x7f9ef8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efd4c2700 1 -- 172.21.15.61:0/2884098008 >> v1:172.21.15.61:6789/0 conn(0x7f9ef80af8e0 legacy=0x7f9ef805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efd4c2700 1 -- 172.21.15.61:0/2884098008 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef805c4a0 con 0x7f9ef80adff0 2022-07-20T03:12:52.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efd4c2700 1 --2- 172.21.15.61:0/2884098008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80adff0 0x7f9ef80b5a30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f9ef40032f0 tx=0x7f9ef40091a0).ready entity=mon.1 client_cookie=2255d199571087be server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efccc1700 1 -- 172.21.15.61:0/2884098008 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ef400b680 con 0x7f9ef80adff0 2022-07-20T03:12:52.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efccc1700 1 -- 172.21.15.61:0/2884098008 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef400b7e0 con 0x7f9ef80adff0 2022-07-20T03:12:52.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efccc1700 1 -- 172.21.15.61:0/2884098008 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ef400b950 con 0x7f9ef80adff0 2022-07-20T03:12:52.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 -- 172.21.15.61:0/2884098008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80adff0 msgr2=0x7f9ef80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 --2- 172.21.15.61:0/2884098008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80adff0 0x7f9ef80b5a30 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f9ef40032f0 tx=0x7f9ef40091a0).stop 2022-07-20T03:12:52.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 -- 172.21.15.61:0/2884098008 shutdown_connections 2022-07-20T03:12:52.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 --2- 172.21.15.61:0/2884098008 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80adff0 0x7f9ef80b5a30 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 -- 172.21.15.61:0/2884098008 >> 172.21.15.61:0/2884098008 conn(0x7f9ef80634e0 msgr2=0x7f9ef80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:52.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 -- 172.21.15.61:0/2884098008 shutdown_connections 2022-07-20T03:12:52.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 -- 172.21.15.61:0/2884098008 wait complete. 2022-07-20T03:12:52.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 Processor -- start 2022-07-20T03:12:52.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.171+0000 7f9efecc5700 1 -- start start 2022-07-20T03:12:52.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ef80adff0 0x7f9ef80643d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 0x7f9ef810e6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80b5650 0x7f9ef8146070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef8111680 con 0x7f9ef80af8e0 2022-07-20T03:12:52.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef810ebe0 con 0x7f9ef80b5650 2022-07-20T03:12:52.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ef810ee50 con 0x7f9ef80adff0 2022-07-20T03:12:52.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 0x7f9ef810e6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 0x7f9ef810e6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45920/0 (socket says 172.21.15.61:45920) 2022-07-20T03:12:52.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 -- 172.21.15.61:0/617737121 learned_addr learned my addr 172.21.15.61:0/617737121 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:52.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efe4c4700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80b5650 0x7f9ef8146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efdcc3700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ef80adff0 0x7f9ef80643d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 -- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ef80adff0 msgr2=0x7f9ef80643d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ef80adff0 0x7f9ef80643d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 -- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80b5650 msgr2=0x7f9ef8146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80b5650 0x7f9ef8146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 -- 172.21.15.61:0/617737121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ef4009420 con 0x7f9ef80af8e0 2022-07-20T03:12:52.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efdcc3700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ef80adff0 0x7f9ef80643d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:52.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efd4c2700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 0x7f9ef810e6c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f9ef40032f0 tx=0x7f9ef4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ef400b950 con 0x7f9ef80af8e0 2022-07-20T03:12:52.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ef4021670 con 0x7f9ef80af8e0 2022-07-20T03:12:52.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ef8159960 con 0x7f9ef80af8e0 2022-07-20T03:12:52.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ef400b680 con 0x7f9ef80af8e0 2022-07-20T03:12:52.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ef8146780 con 0x7f9ef80af8e0 2022-07-20T03:12:52.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ef8159b40 con 0x7f9ef80af8e0 2022-07-20T03:12:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.175+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9ef400f070 con 0x7f9ef80af8e0 2022-07-20T03:12:52.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.179+0000 7f9ee6ffd700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ed80539b0 0x7f9ed8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.179+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ef406d370 con 0x7f9ef80af8e0 2022-07-20T03:12:52.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.179+0000 7f9efdcc3700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ed80539b0 0x7f9ed8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.179+0000 7f9efdcc3700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ed80539b0 0x7f9ed8055e70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f9ee800b000 tx=0x7f9ee8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.187+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ef8159b40 con 0x7f9ef80af8e0 2022-07-20T03:12:52.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.315+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ef8159af0 con 0x7f9ef80af8e0 2022-07-20T03:12:52.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.315+0000 7f9ee6ffd700 1 -- 172.21.15.61:0/617737121 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f9ef40442d0 con 0x7f9ef80af8e0 2022-07-20T03:12:52.355 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:52.356 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":60,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:12:52.012658+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:12:51.013317+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T03:12:51.616869+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:12:52.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ed80539b0 msgr2=0x7f9ed8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ed80539b0 0x7f9ed8055e70 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f9ee800b000 tx=0x7f9ee8009040).stop 2022-07-20T03:12:52.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 msgr2=0x7f9ef810e6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 0x7f9ef810e6c0 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7f9ef40032f0 tx=0x7f9ef4016000).stop 2022-07-20T03:12:52.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 shutdown_connections 2022-07-20T03:12:52.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9ed80539b0 0x7f9ed8055e70 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ef80b5650 0x7f9ef8146070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ef80af8e0 0x7f9ef810e6c0 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 --2- 172.21.15.61:0/617737121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ef80adff0 0x7f9ef80643d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 >> 172.21.15.61:0/617737121 conn(0x7f9ef80634e0 msgr2=0x7f9ef8141820 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:52.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 shutdown_connections 2022-07-20T03:12:52.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.319+0000 7f9efecc5700 1 -- 172.21.15.61:0/617737121 wait complete. 2022-07-20T03:12:52.382 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:12:52.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 Processor -- start 2022-07-20T03:12:52.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 -- start start 2022-07-20T03:12:52.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c0b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5d7c066160 0x7f5d7c13df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d7c0b7f50 con 0x7f5d7c0b31e0 2022-07-20T03:12:52.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d7c0b2800 con 0x7f5d7c066160 2022-07-20T03:12:52.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8236f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c0b2aa0 con 0x7f5d7c0b7b70 2022-07-20T03:12:52.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8136d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c0b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d80b6c700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f5d7c0b31e0 0x7f5d7c13d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:51028/0 (socket says 172.21.15.61:51028) 2022-07-20T03:12:52.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d8136d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c0b1ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:60832/0 (socket says 172.21.15.61:60832) 2022-07-20T03:12:52.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d80b6c700 1 -- 172.21.15.61:0/4197347941 learned_addr learned my addr 172.21.15.61:0/4197347941 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:52.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.499+0000 7f5d81b6e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5d7c066160 0x7f5d7c13df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8136d700 1 -- 172.21.15.61:0/4197347941 >> v2:172.21.15.117:3300/0 conn(0x7f5d7c066160 msgr2=0x7f5d7c13df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:52.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8136d700 1 --2- 172.21.15.61:0/4197347941 >> v2:172.21.15.117:3300/0 conn(0x7f5d7c066160 0x7f5d7c13df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8136d700 1 -- 172.21.15.61:0/4197347941 >> v1:172.21.15.117:6789/0 conn(0x7f5d7c0b31e0 legacy=0x7f5d7c13d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8136d700 1 -- 172.21.15.61:0/4197347941 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7c0b2d40 con 0x7f5d7c0b7b70 2022-07-20T03:12:52.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8136d700 1 --2- 172.21.15.61:0/4197347941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c0b1ac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5d78002f70 tx=0x7f5d78006df0).ready entity=mon.2 client_cookie=eb89d69b1ead1d8b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d6bfff700 1 -- 172.21.15.61:0/4197347941 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7800d570 con 0x7f5d7c0b7b70 2022-07-20T03:12:52.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d6bfff700 1 -- 172.21.15.61:0/4197347941 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d78000e90 con 0x7f5d7c0b7b70 2022-07-20T03:12:52.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d6bfff700 1 -- 172.21.15.61:0/4197347941 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d7800d8c0 con 0x7f5d7c0b7b70 2022-07-20T03:12:52.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 -- 172.21.15.61:0/4197347941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 msgr2=0x7f5d7c0b1ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 --2- 172.21.15.61:0/4197347941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c0b1ac0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f5d78002f70 tx=0x7f5d78006df0).stop 2022-07-20T03:12:52.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 -- 172.21.15.61:0/4197347941 shutdown_connections 2022-07-20T03:12:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 --2- 172.21.15.61:0/4197347941 >> v2:172.21.15.117:3300/0 conn(0x7f5d7c066160 0x7f5d7c13df20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 --2- 172.21.15.61:0/4197347941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c0b1ac0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 -- 172.21.15.61:0/4197347941 >> 172.21.15.61:0/4197347941 conn(0x7f5d7c0656b0 msgr2=0x7f5d7c0b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:52.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 -- 172.21.15.61:0/4197347941 shutdown_connections 2022-07-20T03:12:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.503+0000 7f5d8236f700 1 -- 172.21.15.61:0/4197347941 wait complete. 2022-07-20T03:12:52.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 Processor -- start 2022-07-20T03:12:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 -- start start 2022-07-20T03:12:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 0x7f5d7c0adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0b31e0 0x7f5d7c155440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c155930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c061490 con 0x7f5d7c066160 2022-07-20T03:12:52.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c13a790 con 0x7f5d7c0b31e0 2022-07-20T03:12:52.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d7c13a9f0 con 0x7f5d7c0b7b70 2022-07-20T03:12:52.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 0x7f5d7c0adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 0x7f5d7c0adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45932/0 (socket says 172.21.15.61:45932) 2022-07-20T03:12:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 -- 172.21.15.61:0/2173910106 learned_addr learned my addr 172.21.15.61:0/2173910106 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:52.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d81b6e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c155930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d80b6c700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0b31e0 0x7f5d7c155440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 -- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 msgr2=0x7f5d7c155930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c155930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 -- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0b31e0 msgr2=0x7f5d7c155440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0b31e0 0x7f5d7c155440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 -- 172.21.15.61:0/2173910106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d7800b180 con 0x7f5d7c066160 2022-07-20T03:12:52.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d81b6e700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:52.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8136d700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 0x7f5d7c0adff0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f5d78002f70 tx=0x7f5d78014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d78016070 con 0x7f5d7c066160 2022-07-20T03:12:52.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d7800d8f0 con 0x7f5d7c066160 2022-07-20T03:12:52.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.507+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d7c13ac90 con 0x7f5d7c066160 2022-07-20T03:12:52.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.511+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d78013de0 con 0x7f5d7c066160 2022-07-20T03:12:52.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.511+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d7c13b3d0 con 0x7f5d7c066160 2022-07-20T03:12:52.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.511+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d7c0574e0 con 0x7f5d7c066160 2022-07-20T03:12:52.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.511+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5d78025030 con 0x7f5d7c066160 2022-07-20T03:12:52.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.515+0000 7f5d6a7fc700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d5c053ad0 0x7f5d5c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.515+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5d7806d0f0 con 0x7f5d7c066160 2022-07-20T03:12:52.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.515+0000 7f5d80b6c700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d5c053ad0 0x7f5d5c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.515+0000 7f5d80b6c700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d5c053ad0 0x7f5d5c055f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f5d6c00d000 tx=0x7f5d6c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.519+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d78044050 con 0x7f5d7c066160 2022-07-20T03:12:52.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.651+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d7c068630 con 0x7f5d7c066160 2022-07-20T03:12:52.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.651+0000 7f5d6a7fc700 1 -- 172.21.15.61:0/2173910106 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v60) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f5d78023020 con 0x7f5d7c066160 2022-07-20T03:12:52.660 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:52.661 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":60,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:12:52.012658+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:12:51.013317+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T03:12:51.616869+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:12:52.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.651+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d5c053ad0 msgr2=0x7f5d5c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.651+0000 7f5d8236f700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d5c053ad0 0x7f5d5c055f90 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f5d6c00d000 tx=0x7f5d6c009040).stop 2022-07-20T03:12:52.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.651+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 msgr2=0x7f5d7c0adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.651+0000 7f5d8236f700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 0x7f5d7c0adff0 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7f5d78002f70 tx=0x7f5d78014000).stop 2022-07-20T03:12:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 shutdown_connections 2022-07-20T03:12:52.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d5c053ad0 0x7f5d5c055f90 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d7c0b31e0 0x7f5d7c155440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d7c0b7b70 0x7f5d7c155930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 --2- 172.21.15.61:0/2173910106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d7c066160 0x7f5d7c0adff0 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 >> 172.21.15.61:0/2173910106 conn(0x7f5d7c0656b0 msgr2=0x7f5d7c0672b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 shutdown_connections 2022-07-20T03:12:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.655+0000 7f5d8236f700 1 -- 172.21.15.61:0/2173910106 wait complete. 2022-07-20T03:12:52.673 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:12:52.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.799+0000 7fab3c725700 1 Processor -- start 2022-07-20T03:12:52.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3c725700 1 -- start start 2022-07-20T03:12:52.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3c725700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 0x7fab340adb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3c725700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fab34146580 0x7fab34146990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3c725700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab340ad3e0 con 0x7fab34062ec0 2022-07-20T03:12:52.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3c725700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab340b3b10 con 0x7fab34146580 2022-07-20T03:12:52.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3c725700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab340ad680 con 0x7fab3405cf40 2022-07-20T03:12:52.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab39cc0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fab34062ec0 0x7fab34142b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42542/0 (socket says 172.21.15.61:42542) 2022-07-20T03:12:52.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab39cc0700 1 -- 172.21.15.61:0/2724213936 learned_addr learned my addr 172.21.15.61:0/2724213936 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:52.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3acc2700 1 --2- 172.21.15.61:0/2724213936 >> v2:172.21.15.117:3300/0 conn(0x7fab34146580 0x7fab34146990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 0x7fab340adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 773030411 0 0) 0x7fab340ad3e0 con 0x7fab34062ec0 2022-07-20T03:12:52.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab1c003260 con 0x7fab34062ec0 2022-07-20T03:12:52.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2948918719 0 0) 0x7fab1c003260 con 0x7fab34062ec0 2022-07-20T03:12:52.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 msgr2=0x7fab340adb20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:52.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 0x7fab340adb20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 >> v2:172.21.15.117:3300/0 conn(0x7fab34146580 msgr2=0x7fab34146990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:52.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 --2- 172.21.15.61:0/2724213936 >> v2:172.21.15.117:3300/0 conn(0x7fab34146580 0x7fab34146990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab34143290 con 0x7fab34062ec0 2022-07-20T03:12:52.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fab28001660 con 0x7fab34062ec0 2022-07-20T03:12:52.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 >> v1:172.21.15.61:6789/0 conn(0x7fab34062ec0 legacy=0x7fab34142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c003750 0x7fab1c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 0x7fab1c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab1c0049f0 0x7fab1c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab340b3b10 con 0x7fab1c003750 2022-07-20T03:12:52.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab340ad680 con 0x7fab1c0049f0 2022-07-20T03:12:52.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab1c00b750 con 0x7fab3405cf40 2022-07-20T03:12:52.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c003750 0x7fab1c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3acc2700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab1c0049f0 0x7fab1c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab39cc0700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 0x7fab1c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 msgr2=0x7fab1c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:52.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3405cf40 0x7fab1c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab1c0049f0 msgr2=0x7fab1c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:52.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab1c0049f0 0x7fab1c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2724213936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab1c003260 con 0x7fab1c003750 2022-07-20T03:12:52.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c003750 0x7fab1c003b70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fab3000c000 tx=0x7fab30006df0).ready entity=mon.0 client_cookie=e41874ebf2ea4c53 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3000e830 con 0x7fab1c003750 2022-07-20T03:12:52.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab3000e990 con 0x7fab1c003750 2022-07-20T03:12:52.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.803+0000 7fab394bf700 1 -- 172.21.15.61:0/2724213936 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3000ecf0 con 0x7fab1c003750 2022-07-20T03:12:52.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 -- 172.21.15.61:0/2724213936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c003750 msgr2=0x7fab1c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 --2- 172.21.15.61:0/2724213936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab1c003750 0x7fab1c003b70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fab3000c000 tx=0x7fab30006df0).stop 2022-07-20T03:12:52.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3acc2700 1 -- 172.21.15.61:0/2724213936 reap_dead start 2022-07-20T03:12:52.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 -- 172.21.15.61:0/2724213936 shutdown_connections 2022-07-20T03:12:52.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 -- 172.21.15.61:0/2724213936 >> 172.21.15.61:0/2724213936 conn(0x7fab340b7d30 msgr2=0x7fab340b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:52.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 -- 172.21.15.61:0/2724213936 shutdown_connections 2022-07-20T03:12:52.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 -- 172.21.15.61:0/2724213936 wait complete. 2022-07-20T03:12:52.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 Processor -- start 2022-07-20T03:12:52.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.807+0000 7fab3c725700 1 -- start start 2022-07-20T03:12:52.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 0x7fab34151f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3409ab90 0x7fab340609c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab34060eb0 0x7fab34155c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab340aea80 con 0x7fab34151b40 2022-07-20T03:12:52.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 0x7fab34151f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 0x7fab34151f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45952/0 (socket says 172.21.15.61:45952) 2022-07-20T03:12:52.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2852435926 learned_addr learned my addr 172.21.15.61:0/2852435926 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:52.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab3409b340 con 0x7fab34060eb0 2022-07-20T03:12:52.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab3409b0a0 con 0x7fab3409ab90 2022-07-20T03:12:52.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab39cc0700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3409ab90 0x7fab340609c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3acc2700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab34060eb0 0x7fab34155c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3409ab90 msgr2=0x7fab340609c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3409ab90 0x7fab340609c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab34060eb0 msgr2=0x7fab34155c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab34060eb0 0x7fab34155c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 -- 172.21.15.61:0/2852435926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab341557a0 con 0x7fab34151b40 2022-07-20T03:12:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3a4c1700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 0x7fab34151f20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fab3409a4a0 tx=0x7fab30019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab3001cd60 con 0x7fab34151b40 2022-07-20T03:12:52.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab30000e90 con 0x7fab34151b40 2022-07-20T03:12:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab3409b830 con 0x7fab34151b40 2022-07-20T03:12:52.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab300245e0 con 0x7fab34151b40 2022-07-20T03:12:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab34152630 con 0x7fab34151b40 2022-07-20T03:12:52.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.811+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab34064810 con 0x7fab34151b40 2022-07-20T03:12:52.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.815+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fab3002a020 con 0x7fab34151b40 2022-07-20T03:12:52.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.815+0000 7fab277fe700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab14053920 0x7fab14055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:52.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.815+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(60..60 src has 1..60) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fab3006d410 con 0x7fab34151b40 2022-07-20T03:12:52.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.815+0000 7fab39cc0700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab14053920 0x7fab14055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:52.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.815+0000 7fab39cc0700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab14053920 0x7fab14055de0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fab2800d000 tx=0x7fab28007040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:52.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.819+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab30047d00 con 0x7fab34151b40 2022-07-20T03:12:53.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:52.987+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fab340574e0 con 0x7fab34151b40 2022-07-20T03:12:53.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab277fe700 1 -- 172.21.15.61:0/2852435926 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v35) v1 ==== 118+0+0 (secure 0 0 0) 0x7fab3000be60 con 0x7fab34151b40 2022-07-20T03:12:53.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab14053920 msgr2=0x7fab14055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab14053920 0x7fab14055de0 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fab2800d000 tx=0x7fab28007040).stop 2022-07-20T03:12:53.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 msgr2=0x7fab34151f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 0x7fab34151f20 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fab3409a4a0 tx=0x7fab30019000).stop 2022-07-20T03:12:53.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 shutdown_connections 2022-07-20T03:12:53.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab14053920 0x7fab14055de0 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab34060eb0 0x7fab34155c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab3409ab90 0x7fab340609c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 --2- 172.21.15.61:0/2852435926 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab34151b40 0x7fab34151f20 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 >> 172.21.15.61:0/2852435926 conn(0x7fab340b7d30 msgr2=0x7fab340b24c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:53.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 shutdown_connections 2022-07-20T03:12:53.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.019+0000 7fab3c725700 1 -- 172.21.15.61:0/2852435926 wait complete. 2022-07-20T03:12:53.086 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:53.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.227+0000 7f1394877700 1 Processor -- start 2022-07-20T03:12:53.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.227+0000 7f1394877700 1 -- start start 2022-07-20T03:12:53.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f1394877700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f13900b1e00 0x7f139014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f1394877700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f139005d480 con 0x7f13900acef0 2022-07-20T03:12:53.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f1394877700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13900b21e0 con 0x7f139005c3f0 2022-07-20T03:12:53.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f1394877700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f139005cf40 con 0x7f13900b1e00 2022-07-20T03:12:53.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f13900acef0 0x7f1390146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42562/0 (socket says 172.21.15.61:42562) 2022-07-20T03:12:53.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138ed9d700 1 -- 172.21.15.61:0/2935768211 learned_addr learned my addr 172.21.15.61:0/2935768211 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:53.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138e59c700 1 --2- 172.21.15.61:0/2935768211 >> v2:172.21.15.117:3300/0 conn(0x7f13900b1e00 0x7f139014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1071853084 0 0) 0x7f139005d480 con 0x7f13900acef0 2022-07-20T03:12:53.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1374003260 con 0x7f13900acef0 2022-07-20T03:12:53.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3493426133 0 0) 0x7f13900b21e0 con 0x7f139005c3f0 2022-07-20T03:12:53.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f139005d480 con 0x7f139005c3f0 2022-07-20T03:12:53.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2400716881 0 0) 0x7f1374003260 con 0x7f13900acef0 2022-07-20T03:12:53.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 >> v2:172.21.15.117:3300/0 conn(0x7f13900b1e00 msgr2=0x7f139014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:53.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 --2- 172.21.15.61:0/2935768211 >> v2:172.21.15.117:3300/0 conn(0x7f13900b1e00 0x7f139014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 >> v1:172.21.15.117:6789/0 conn(0x7f139005c3f0 legacy=0x7f13900ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139005d1e0 con 0x7f13900acef0 2022-07-20T03:12:53.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f13880019e0 con 0x7f13900acef0 2022-07-20T03:12:53.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 >> v1:172.21.15.61:6789/0 conn(0x7f13900acef0 legacy=0x7f1390146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13740038e0 0x7f1374003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1374004500 0x7f1374004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1374004e70 0x7f137400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139005cf40 con 0x7f1374004500 2022-07-20T03:12:53.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f137400bd50 con 0x7f13740038e0 2022-07-20T03:12:53.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f137400bf10 con 0x7f1374004e70 2022-07-20T03:12:53.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1374004500 0x7f1374004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138e59c700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13740038e0 0x7f1374003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138ed9d700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1374004e70 0x7f137400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 -- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1374004e70 msgr2=0x7f137400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:53.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1374004e70 0x7f137400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 -- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13740038e0 msgr2=0x7f1374003cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:53.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13740038e0 0x7f1374003cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 -- 172.21.15.61:0/2935768211 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1374003260 con 0x7f1374004500 2022-07-20T03:12:53.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138ed9d700 1 -- 172.21.15.61:0/2935768211 reap_dead start 2022-07-20T03:12:53.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138dd9b700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1374004500 0x7f1374004980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f1380001110 tx=0x7f1380007a00).ready entity=mon.0 client_cookie=4b13fabd6d16c6e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:53.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1380007d10 con 0x7f1374004500 2022-07-20T03:12:53.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1380007070 con 0x7f1374004500 2022-07-20T03:12:53.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.231+0000 7f138d59a700 1 -- 172.21.15.61:0/2935768211 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f13800073d0 con 0x7f1374004500 2022-07-20T03:12:53.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- 172.21.15.61:0/2935768211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1374004500 msgr2=0x7f1374004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1374004500 0x7f1374004980 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7f1380001110 tx=0x7f1380007a00).stop 2022-07-20T03:12:53.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- 172.21.15.61:0/2935768211 shutdown_connections 2022-07-20T03:12:53.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 --2- 172.21.15.61:0/2935768211 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1374004500 0x7f1374004980 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- 172.21.15.61:0/2935768211 >> 172.21.15.61:0/2935768211 conn(0x7f13900611c0 msgr2=0x7f1390062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:53.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- 172.21.15.61:0/2935768211 shutdown_connections 2022-07-20T03:12:53.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- 172.21.15.61:0/2935768211 wait complete. 2022-07-20T03:12:53.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 Processor -- start 2022-07-20T03:12:53.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- start start 2022-07-20T03:12:53.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 0x7f13900b5850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f139009e080 0x7f139009e500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f139009e9f0 0x7f13901426b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f139014a9d0 con 0x7f139014a5f0 2022-07-20T03:12:53.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13900b3970 con 0x7f139009e080 2022-07-20T03:12:53.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f1394877700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13900b3c10 con 0x7f139009e9f0 2022-07-20T03:12:53.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 0x7f13900b5850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 0x7f13900b5850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45972/0 (socket says 172.21.15.61:45972) 2022-07-20T03:12:53.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 -- 172.21.15.61:0/3116816215 learned_addr learned my addr 172.21.15.61:0/3116816215 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:53.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138ed9d700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f139009e9f0 0x7f13901426b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138dd9b700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f139009e080 0x7f139009e500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 -- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f139009e9f0 msgr2=0x7f13901426b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f139009e9f0 0x7f13901426b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 -- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f139009e080 msgr2=0x7f139009e500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f139009e080 0x7f139009e500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 -- 172.21.15.61:0/3116816215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f139009ee70 con 0x7f139014a5f0 2022-07-20T03:12:53.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.235+0000 7f138e59c700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 0x7f13900b5850 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f138400b000 tx=0x7f1384006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:53.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1384011e70 con 0x7f139014a5f0 2022-07-20T03:12:53.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f138400f940 con 0x7f139014a5f0 2022-07-20T03:12:53.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13900b5010 con 0x7f139014a5f0 2022-07-20T03:12:53.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f138400fca0 con 0x7f139014a5f0 2022-07-20T03:12:53.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1390142ba0 con 0x7f139014a5f0 2022-07-20T03:12:53.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13900b04b0 con 0x7f139014a5f0 2022-07-20T03:12:53.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1384014440 con 0x7f139014a5f0 2022-07-20T03:12:53.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f137f7fe700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f136c053ad0 0x7f136c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f138406c690 con 0x7f139014a5f0 2022-07-20T03:12:53.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f138dd9b700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f136c053ad0 0x7f136c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.239+0000 7f138dd9b700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f136c053ad0 0x7f136c055f90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f138001b000 tx=0x7f1380015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:53.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.247+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1384043590 con 0x7f139014a5f0 2022-07-20T03:12:53.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13900574e0 con 0x7f139014a5f0 2022-07-20T03:12:53.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f137f7fe700 1 -- 172.21.15.61:0/3116816215 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f1384018020 con 0x7f139014a5f0 2022-07-20T03:12:53.418 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:53.418 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:12:53.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f136c053ad0 msgr2=0x7f136c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f136c053ad0 0x7f136c055f90 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f138001b000 tx=0x7f1380015000).stop 2022-07-20T03:12:53.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 msgr2=0x7f13900b5850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 0x7f13900b5850 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7f138400b000 tx=0x7f1384006df0).stop 2022-07-20T03:12:53.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 shutdown_connections 2022-07-20T03:12:53.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f136c053ad0 0x7f136c055f90 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f139009e080 0x7f139009e500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f139009e9f0 0x7f13901426b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 --2- 172.21.15.61:0/3116816215 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f139014a5f0 0x7f13900b5850 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 >> 172.21.15.61:0/3116816215 conn(0x7f1374004500 msgr2=0x7f139014b810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:53.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 shutdown_connections 2022-07-20T03:12:53.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.415+0000 7f1394877700 1 -- 172.21.15.61:0/3116816215 wait complete. 2022-07-20T03:12:53.479 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:12:53.512 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:12:53.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 Processor -- start 2022-07-20T03:12:53.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- start start 2022-07-20T03:12:53.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc1424e0 0x7f22dc0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22dc0aea10 0x7f22dc0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22dc1433d0 con 0x7f22dc0b0570 2022-07-20T03:12:53.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f22dc143b10 con 0x7f22dc0aea10 2022-07-20T03:12:53.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc143670 con 0x7f22dc1424e0 2022-07-20T03:12:53.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22dc0aea10 0x7f22dc0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22dc0aea10 0x7f22dc0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45984/0 (socket says 172.21.15.61:45984) 2022-07-20T03:12:53.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 -- 172.21.15.61:0/3786477585 learned_addr learned my addr 172.21.15.61:0/3786477585 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:53.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e1b4d700 1 --2- 172.21.15.61:0/3786477585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc1424e0 0x7f22dc0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 -- 172.21.15.61:0/3786477585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc1424e0 msgr2=0x7f22dc0b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:53.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 --2- 172.21.15.61:0/3786477585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc1424e0 0x7f22dc0b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 -- 172.21.15.61:0/3786477585 >> v1:172.21.15.117:6789/0 conn(0x7f22dc0b0570 legacy=0x7f22dc141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 -- 172.21.15.61:0/3786477585 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22dc0b1010 con 0x7f22dc0aea10 2022-07-20T03:12:53.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e2b4f700 1 --2- 172.21.15.61:0/3786477585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0aea10 0x7f22dc0ad350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f22d40032f0 tx=0x7f22d4009290).ready entity=mon.0 client_cookie=7a50606d7c95addc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:53.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e134c700 1 -- 172.21.15.61:0/3786477585 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22d400b8a0 con 0x7f22dc0aea10 2022-07-20T03:12:53.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e134c700 1 -- 172.21.15.61:0/3786477585 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22d400ba00 con 0x7f22dc0aea10 2022-07-20T03:12:53.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e134c700 1 -- 172.21.15.61:0/3786477585 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22d400bb70 con 0x7f22dc0aea10 2022-07-20T03:12:53.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- 172.21.15.61:0/3786477585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0aea10 msgr2=0x7f22dc0ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 --2- 172.21.15.61:0/3786477585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0aea10 0x7f22dc0ad350 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f22d40032f0 tx=0x7f22d4009290).stop 2022-07-20T03:12:53.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- 172.21.15.61:0/3786477585 shutdown_connections 2022-07-20T03:12:53.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 --2- 172.21.15.61:0/3786477585 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0aea10 0x7f22dc0ad350 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 --2- 172.21.15.61:0/3786477585 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc1424e0 0x7f22dc0b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.619+0000 7f22e3350700 1 -- 172.21.15.61:0/3786477585 >> 172.21.15.61:0/3786477585 conn(0x7f22dc0b9360 msgr2=0x7f22dc0b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:53.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.623+0000 7f22e3350700 1 -- 172.21.15.61:0/3786477585 shutdown_connections 2022-07-20T03:12:53.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.623+0000 7f22e3350700 1 -- 172.21.15.61:0/3786477585 wait complete. 2022-07-20T03:12:53.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.623+0000 7f22e3350700 1 Processor -- start 2022-07-20T03:12:53.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.623+0000 7f22e3350700 1 -- start start 2022-07-20T03:12:53.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc0aea10 0x7f22dc0b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 0x7f22dc140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22dc1424e0 0x7f22dc155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc13e9b0 con 0x7f22dc0b0570 2022-07-20T03:12:53.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0b4790 con 0x7f22dc1424e0 2022-07-20T03:12:53.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22dc0b49b0 con 0x7f22dc0aea10 2022-07-20T03:12:53.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 0x7f22dc140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 0x7f22dc140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45988/0 (socket says 172.21.15.61:45988) 2022-07-20T03:12:53.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 -- 172.21.15.61:0/2622967166 learned_addr learned my addr 172.21.15.61:0/2622967166 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:53.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e234e700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc0aea10 0x7f22dc0b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e2b4f700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22dc1424e0 0x7f22dc155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 -- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc0aea10 msgr2=0x7f22dc0b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc0aea10 0x7f22dc0b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 -- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22dc1424e0 msgr2=0x7f22dc155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22dc1424e0 0x7f22dc155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 -- 172.21.15.61:0/2622967166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22d4009530 con 0x7f22dc0b0570 2022-07-20T03:12:53.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e2b4f700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22dc1424e0 0x7f22dc155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:53.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e1b4d700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 0x7f22dc140070 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f22dc0b48d0 tx=0x7f22cc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:53.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22cc011780 con 0x7f22dc0b0570 2022-07-20T03:12:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f22cc0118e0 con 0x7f22dc0b0570 2022-07-20T03:12:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22dc0b4c50 con 0x7f22dc0b0570 2022-07-20T03:12:53.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22cc011c40 con 0x7f22dc0b0570 2022-07-20T03:12:53.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22dc1405e0 con 0x7f22dc0b0570 2022-07-20T03:12:53.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f22dc0ace30 con 0x7f22dc0b0570 2022-07-20T03:12:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.627+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f22cc012420 con 0x7f22dc0b0570 2022-07-20T03:12:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.631+0000 7f22cb7fe700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22bc0539d0 0x7f22bc055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.631+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f22cc06bcc0 con 0x7f22dc0b0570 2022-07-20T03:12:53.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.631+0000 7f22e234e700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22bc0539d0 0x7f22bc055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.631+0000 7f22e234e700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22bc0539d0 0x7f22bc055e90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f22d800d000 tx=0x7f22d8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:53.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.635+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f22cc072080 con 0x7f22dc0b0570 2022-07-20T03:12:53.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.763+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f22dc0574e0 con 0x7f22dc0b0570 2022-07-20T03:12:53.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.763+0000 7f22cb7fe700 1 -- 172.21.15.61:0/2622967166 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f22cc042ab0 con 0x7f22dc0b0570 2022-07-20T03:12:53.960 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:53.961 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":61,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:12:53.016664+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:12:51.013317+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T03:12:51.616869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:12:53.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22bc0539d0 msgr2=0x7f22bc055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22bc0539d0 0x7f22bc055e90 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7f22d800d000 tx=0x7f22d8007b90).stop 2022-07-20T03:12:53.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 msgr2=0x7f22dc140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 0x7f22dc140070 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7f22dc0b48d0 tx=0x7f22cc008040).stop 2022-07-20T03:12:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 shutdown_connections 2022-07-20T03:12:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f22bc0539d0 0x7f22bc055e90 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22dc1424e0 0x7f22dc155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22dc0b0570 0x7f22dc140070 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 --2- 172.21.15.61:0/2622967166 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22dc0aea10 0x7f22dc0b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:53.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 >> 172.21.15.61:0/2622967166 conn(0x7f22dc0b9360 msgr2=0x7f22dc0ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:53.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 shutdown_connections 2022-07-20T03:12:53.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:53.767+0000 7f22e3350700 1 -- 172.21.15.61:0/2622967166 wait complete. 2022-07-20T03:12:53.989 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:12:54.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 Processor -- start 2022-07-20T03:12:54.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 -- start start 2022-07-20T03:12:54.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f60ac067490 0x7f60ac143d70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f60ac05d1a0 0x7f60ac144260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60ac05c7a0 con 0x7f60ac0b1ae0 2022-07-20T03:12:54.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac067870 con 0x7f60ac067490 2022-07-20T03:12:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60b0ab4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f60ac05c9d0 con 0x7f60ac05d1a0 2022-07-20T03:12:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.239+0000 7f60ab7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f60ac067490 0x7f60ac143d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f60ac067490 0x7f60ac143d70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:45994/0 (socket says 172.21.15.61:45994) 2022-07-20T03:12:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 -- 172.21.15.61:0/78434438 learned_addr learned my addr 172.21.15.61:0/78434438 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:54.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60aaffd700 1 --2- 172.21.15.61:0/78434438 >> v2:172.21.15.117:3300/0 conn(0x7f60ac05d1a0 0x7f60ac144260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 -- 172.21.15.61:0/78434438 >> v2:172.21.15.117:3300/0 conn(0x7f60ac05d1a0 msgr2=0x7f60ac144260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:54.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 --2- 172.21.15.61:0/78434438 >> v2:172.21.15.117:3300/0 conn(0x7f60ac05d1a0 0x7f60ac144260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 -- 172.21.15.61:0/78434438 >> v1:172.21.15.117:6789/0 conn(0x7f60ac0b1ae0 legacy=0x7f60ac13f740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 -- 172.21.15.61:0/78434438 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac05cc70 con 0x7f60ac067490 2022-07-20T03:12:54.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60ab7fe700 1 --2- 172.21.15.61:0/78434438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac067490 0x7f60ac143d70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f60ac0b28f0 tx=0x7f609c006eb0).ready entity=mon.0 client_cookie=d0632eb6fd968c24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60aa7fc700 1 -- 172.21.15.61:0/78434438 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f609c00d610 con 0x7f60ac067490 2022-07-20T03:12:54.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60aa7fc700 1 -- 172.21.15.61:0/78434438 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c000e90 con 0x7f60ac067490 2022-07-20T03:12:54.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60aa7fc700 1 -- 172.21.15.61:0/78434438 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f609c00d9a0 con 0x7f60ac067490 2022-07-20T03:12:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 -- 172.21.15.61:0/78434438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac067490 msgr2=0x7f60ac143d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/78434438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac067490 0x7f60ac143d70 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f60ac0b28f0 tx=0x7f609c006eb0).stop 2022-07-20T03:12:54.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 -- 172.21.15.61:0/78434438 shutdown_connections 2022-07-20T03:12:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/78434438 >> v2:172.21.15.117:3300/0 conn(0x7f60ac05d1a0 0x7f60ac144260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/78434438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac067490 0x7f60ac143d70 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 -- 172.21.15.61:0/78434438 >> 172.21.15.61:0/78434438 conn(0x7f60ac064c00 msgr2=0x7f60ac064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:54.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 -- 172.21.15.61:0/78434438 shutdown_connections 2022-07-20T03:12:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.243+0000 7f60b0ab4700 1 -- 172.21.15.61:0/78434438 wait complete. 2022-07-20T03:12:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 Processor -- start 2022-07-20T03:12:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 -- start start 2022-07-20T03:12:54.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 0x7f60ac05f840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60ac067490 0x7f60ac143c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60ac0b1ae0 0x7f60ac13e010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac140f90 con 0x7f60ac05d1a0 2022-07-20T03:12:54.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac060750 con 0x7f60ac067490 2022-07-20T03:12:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60ac060c80 con 0x7f60ac0b1ae0 2022-07-20T03:12:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 0x7f60ac05f840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 0x7f60ac05f840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46002/0 (socket says 172.21.15.61:46002) 2022-07-20T03:12:54.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 -- 172.21.15.61:0/563809868 learned_addr learned my addr 172.21.15.61:0/563809868 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60aaffd700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60ac067490 0x7f60ac143c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60abfff700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60ac0b1ae0 0x7f60ac13e010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 -- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60ac0b1ae0 msgr2=0x7f60ac13e010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60ac0b1ae0 0x7f60ac13e010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 -- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60ac067490 msgr2=0x7f60ac143c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60ac067490 0x7f60ac143c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 -- 172.21.15.61:0/563809868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f609c00b1e0 con 0x7f60ac05d1a0 2022-07-20T03:12:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60aaffd700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60ac067490 0x7f60ac143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60ab7fe700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 0x7f60ac05f840 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f60ac0b28f0 tx=0x7f609c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f609c011860 con 0x7f60ac05d1a0 2022-07-20T03:12:54.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f609c0119c0 con 0x7f60ac05d1a0 2022-07-20T03:12:54.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60ac060f20 con 0x7f60ac05d1a0 2022-07-20T03:12:54.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.247+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f609c011d20 con 0x7f60ac05d1a0 2022-07-20T03:12:54.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60ac061280 con 0x7f60ac05d1a0 2022-07-20T03:12:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f609c025030 con 0x7f60ac05d1a0 2022-07-20T03:12:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60a8ff9700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6090053a40 0x7f6090055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f609c06ced0 con 0x7f60ac05d1a0 2022-07-20T03:12:54.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60ac152030 con 0x7f60ac05d1a0 2022-07-20T03:12:54.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60aaffd700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6090053a40 0x7f6090055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.251+0000 7f60aaffd700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6090053a40 0x7f6090055f00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f609400b000 tx=0x7f6094006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.259+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f609c023030 con 0x7f60ac05d1a0 2022-07-20T03:12:54.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.427+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f60ac0acb50 con 0x7f60ac05d1a0 2022-07-20T03:12:54.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.435+0000 7f60a8ff9700 1 -- 172.21.15.61:0/563809868 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v22) v1 ==== 184+0+0 (secure 0 0 0) 0x7f609c02d0e0 con 0x7f60ac05d1a0 2022-07-20T03:12:54.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6090053a40 msgr2=0x7f6090055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6090053a40 0x7f6090055f00 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7f609400b000 tx=0x7f6094006df0).stop 2022-07-20T03:12:54.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 msgr2=0x7f60ac05f840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 0x7f60ac05f840 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f60ac0b28f0 tx=0x7f609c015000).stop 2022-07-20T03:12:54.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 shutdown_connections 2022-07-20T03:12:54.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6090053a40 0x7f6090055f00 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60ac067490 0x7f60ac143c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60ac0b1ae0 0x7f60ac13e010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 --2- 172.21.15.61:0/563809868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60ac05d1a0 0x7f60ac05f840 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 >> 172.21.15.61:0/563809868 conn(0x7f60ac064c00 msgr2=0x7f60ac0b5040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:54.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 shutdown_connections 2022-07-20T03:12:54.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.439+0000 7f60b0ab4700 1 -- 172.21.15.61:0/563809868 wait complete. 2022-07-20T03:12:54.470 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:12:54.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:12:54.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 Processor -- start 2022-07-20T03:12:54.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 -- start start 2022-07-20T03:12:54.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b3cb0 0x7f45240aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f45240ad0e0 0x7f45241418f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45240ae280 con 0x7f45240b6800 2022-07-20T03:12:54.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45240b6be0 con 0x7f45240ad0e0 2022-07-20T03:12:54.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452b474700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45240ae4b0 con 0x7f45240b3cb0 2022-07-20T03:12:54.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452ac73700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f45240ad0e0 0x7f45241418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452a472700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f45240b6800 0x7f4524145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42606/0 (socket says 172.21.15.61:42606) 2022-07-20T03:12:54.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452a472700 1 -- 172.21.15.61:0/2162118790 learned_addr learned my addr 172.21.15.61:0/2162118790 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:54.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f4529c71700 1 --2- 172.21.15.61:0/2162118790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b3cb0 0x7f45240aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452ac73700 1 -- 172.21.15.61:0/2162118790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b3cb0 msgr2=0x7f45240aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:54.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452ac73700 1 --2- 172.21.15.61:0/2162118790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b3cb0 0x7f45240aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452ac73700 1 -- 172.21.15.61:0/2162118790 >> v1:172.21.15.61:6789/0 conn(0x7f45240b6800 legacy=0x7f4524145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452ac73700 1 -- 172.21.15.61:0/2162118790 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45240ae750 con 0x7f45240ad0e0 2022-07-20T03:12:54.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f452ac73700 1 --2- 172.21.15.61:0/2162118790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240ad0e0 0x7f45241418f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f451c0032f0 tx=0x7f451c0091d0).ready entity=mon.0 client_cookie=8045c3c8475c3eee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f4529470700 1 -- 172.21.15.61:0/2162118790 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f451c00b6c0 con 0x7f45240ad0e0 2022-07-20T03:12:54.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.595+0000 7f4529470700 1 -- 172.21.15.61:0/2162118790 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f451c00b820 con 0x7f45240ad0e0 2022-07-20T03:12:54.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f4529470700 1 -- 172.21.15.61:0/2162118790 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f451c00b990 con 0x7f45240ad0e0 2022-07-20T03:12:54.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 -- 172.21.15.61:0/2162118790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240ad0e0 msgr2=0x7f45241418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 --2- 172.21.15.61:0/2162118790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240ad0e0 0x7f45241418f0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f451c0032f0 tx=0x7f451c0091d0).stop 2022-07-20T03:12:54.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 -- 172.21.15.61:0/2162118790 shutdown_connections 2022-07-20T03:12:54.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 --2- 172.21.15.61:0/2162118790 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240ad0e0 0x7f45241418f0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 --2- 172.21.15.61:0/2162118790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b3cb0 0x7f45240aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 -- 172.21.15.61:0/2162118790 >> 172.21.15.61:0/2162118790 conn(0x7f45240b9dd0 msgr2=0x7f45240b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 -- 172.21.15.61:0/2162118790 shutdown_connections 2022-07-20T03:12:54.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 -- 172.21.15.61:0/2162118790 wait complete. 2022-07-20T03:12:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 Processor -- start 2022-07-20T03:12:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 -- start start 2022-07-20T03:12:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45240ad0e0 0x7f4524144120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.599+0000 7f452b474700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 0x7f45240af380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b6800 0x7f452413d7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4524144990 con 0x7f45240b3cb0 2022-07-20T03:12:54.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45240b0fd0 con 0x7f45240ad0e0 2022-07-20T03:12:54.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45240b1270 con 0x7f45240b6800 2022-07-20T03:12:54.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 0x7f45240af380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 0x7f45240af380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46014/0 (socket says 172.21.15.61:46014) 2022-07-20T03:12:54.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 -- 172.21.15.61:0/780418920 learned_addr learned my addr 172.21.15.61:0/780418920 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:54.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452a472700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45240ad0e0 0x7f4524144120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452ac73700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b6800 0x7f452413d7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 -- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b6800 msgr2=0x7f452413d7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b6800 0x7f452413d7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 -- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45240ad0e0 msgr2=0x7f4524144120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45240ad0e0 0x7f4524144120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 -- 172.21.15.61:0/780418920 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f451c009470 con 0x7f45240b3cb0 2022-07-20T03:12:54.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f4529c71700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 0x7f45240af380 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f452000c000 tx=0x7f4520006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4520010780 con 0x7f45240b3cb0 2022-07-20T03:12:54.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45200108e0 con 0x7f45240b3cb0 2022-07-20T03:12:54.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f45240b1510 con 0x7f45240b3cb0 2022-07-20T03:12:54.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4520010c40 con 0x7f45240b3cb0 2022-07-20T03:12:54.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45241457d0 con 0x7f45240b3cb0 2022-07-20T03:12:54.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.603+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f45240b5530 con 0x7f45240b3cb0 2022-07-20T03:12:54.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.607+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f452001c460 con 0x7f45240b3cb0 2022-07-20T03:12:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.607+0000 7f45177fe700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4504053a60 0x7f4504055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.607+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f452006c270 con 0x7f45240b3cb0 2022-07-20T03:12:54.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.607+0000 7f452a472700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4504053a60 0x7f4504055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.607+0000 7f452a472700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4504053a60 0x7f4504055f20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f451800d000 tx=0x7f4518000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.615+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45200430d0 con 0x7f45240b3cb0 2022-07-20T03:12:54.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.779+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f45240574e0 con 0x7f45240b3cb0 2022-07-20T03:12:54.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f45177fe700 1 -- 172.21.15.61:0/780418920 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v23) v1 ==== 184+0+0 (secure 0 0 0) 0x7f45200466f0 con 0x7f45240b3cb0 2022-07-20T03:12:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4504053a60 msgr2=0x7f4504055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4504053a60 0x7f4504055f20 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f451800d000 tx=0x7f4518000bc0).stop 2022-07-20T03:12:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 msgr2=0x7f45240af380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 0x7f45240af380 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f452000c000 tx=0x7f4520006df0).stop 2022-07-20T03:12:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 shutdown_connections 2022-07-20T03:12:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4504053a60 0x7f4504055f20 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45240b6800 0x7f452413d7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45240b3cb0 0x7f45240af380 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 --2- 172.21.15.61:0/780418920 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45240ad0e0 0x7f4524144120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 >> 172.21.15.61:0/780418920 conn(0x7f45240b9dd0 msgr2=0x7f4524143120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 shutdown_connections 2022-07-20T03:12:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.787+0000 7f452b474700 1 -- 172.21.15.61:0/780418920 wait complete. 2022-07-20T03:12:54.803 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:12:54.804 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:12:54.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feeae0e4700 1 Processor -- start 2022-07-20T03:12:54.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feeae0e4700 1 -- start start 2022-07-20T03:12:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feeae0e4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 0x7feea80b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feeae0e4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80b3e40 con 0x7feea80b2290 2022-07-20T03:12:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feeae0e4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feea80ba2d0 con 0x7feea80b9ef0 2022-07-20T03:12:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feeae0e4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea80b4070 con 0x7feea80b3650 2022-07-20T03:12:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feea80b2290 0x7feea8141d40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42624/0 (socket says 172.21.15.61:42624) 2022-07-20T03:12:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea7fff700 1 -- 172.21.15.61:0/3494351134 learned_addr learned my addr 172.21.15.61:0/3494351134 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7fee9effd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 0x7feea80b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2010811751 0 0) 0x7feea80b3e40 con 0x7feea80b2290 2022-07-20T03:12:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee88003260 con 0x7feea80b2290 2022-07-20T03:12:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 993893469 0 0) 0x7feea80ba2d0 con 0x7feea80b9ef0 2022-07-20T03:12:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feea80b3e40 con 0x7feea80b9ef0 2022-07-20T03:12:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 73905105 0 0) 0x7fee88003260 con 0x7feea80b2290 2022-07-20T03:12:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 msgr2=0x7feea80b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 0x7feea80b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 >> v1:172.21.15.117:6789/0 conn(0x7feea80b9ef0 legacy=0x7feea80b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea80b1df0 con 0x7feea80b2290 2022-07-20T03:12:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7feea00019e0 con 0x7feea80b2290 2022-07-20T03:12:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 >> v1:172.21.15.61:6789/0 conn(0x7feea80b2290 legacy=0x7feea8141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 0x7fee88003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7fee88004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880049f0 0x7fee88004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea80b4070 con 0x7fee88003e20 2022-07-20T03:12:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8800b800 con 0x7fee880049f0 2022-07-20T03:12:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee8800b9c0 con 0x7feea80b3650 2022-07-20T03:12:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7fee9effd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7fee88004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea77fe700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 0x7fee88003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.927+0000 7feea7fff700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880049f0 0x7fee88004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7fee9effd700 1 -- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 msgr2=0x7fee88003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:54.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7fee9effd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea80b3650 0x7fee88003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7fee9effd700 1 -- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880049f0 msgr2=0x7fee88004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7fee9effd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880049f0 0x7fee88004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7fee9effd700 1 -- 172.21.15.61:0/3494351134 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee88003260 con 0x7fee88003e20 2022-07-20T03:12:54.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7fee9effd700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7fee88004500 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fee9000c000 tx=0x7fee90006df0).ready entity=mon.0 client_cookie=892b7294e233b651 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee9000e830 con 0x7fee88003e20 2022-07-20T03:12:54.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee9000e990 con 0x7fee88003e20 2022-07-20T03:12:54.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feea6ffd700 1 -- 172.21.15.61:0/3494351134 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee9000ecf0 con 0x7fee88003e20 2022-07-20T03:12:54.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 -- 172.21.15.61:0/3494351134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 msgr2=0x7fee88004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 --2- 172.21.15.61:0/3494351134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7fee88004500 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fee9000c000 tx=0x7fee90006df0).stop 2022-07-20T03:12:54.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feea7fff700 1 -- 172.21.15.61:0/3494351134 reap_dead start 2022-07-20T03:12:54.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 -- 172.21.15.61:0/3494351134 shutdown_connections 2022-07-20T03:12:54.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 -- 172.21.15.61:0/3494351134 >> 172.21.15.61:0/3494351134 conn(0x7feea8067980 msgr2=0x7feea80b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:54.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 -- 172.21.15.61:0/3494351134 shutdown_connections 2022-07-20T03:12:54.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 -- 172.21.15.61:0/3494351134 wait complete. 2022-07-20T03:12:54.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 Processor -- start 2022-07-20T03:12:54.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.931+0000 7feeae0e4700 1 -- start start 2022-07-20T03:12:54.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7feea80b7ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea8136e00 0x7feea8137260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feea8151f20 0x7feea81523a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:54.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea80ad610 con 0x7fee88003e20 2022-07-20T03:12:54.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feea813ff20 con 0x7feea8151f20 2022-07-20T03:12:54.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feea813cdf0 con 0x7feea8136e00 2022-07-20T03:12:54.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7feea80b7ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7feea80b7ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46034/0 (socket says 172.21.15.61:46034) 2022-07-20T03:12:54.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 -- 172.21.15.61:0/4155419075 learned_addr learned my addr 172.21.15.61:0/4155419075 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:54.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7fee9effd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea8136e00 0x7feea8137260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea7fff700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feea8151f20 0x7feea81523a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:54.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 -- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea8136e00 msgr2=0x7feea8137260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea8136e00 0x7feea8137260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 -- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feea8151f20 msgr2=0x7feea81523a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:54.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feea8151f20 0x7feea81523a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:54.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 -- 172.21.15.61:0/4155419075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feea8137750 con 0x7fee88003e20 2022-07-20T03:12:54.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea77fe700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7feea80b7ef0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fee9800b000 tx=0x7fee98009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:54.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee98014620 con 0x7fee88003e20 2022-07-20T03:12:54.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee98014780 con 0x7fee88003e20 2022-07-20T03:12:54.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feea80b85c0 con 0x7fee88003e20 2022-07-20T03:12:54.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee98014ae0 con 0x7fee88003e20 2022-07-20T03:12:55.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feea8146f90 con 0x7fee88003e20 2022-07-20T03:12:55.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.935+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feea813cfb0 con 0x7fee88003e20 2022-07-20T03:12:55.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.939+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fee98014c40 con 0x7fee88003e20 2022-07-20T03:12:55.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.939+0000 7feea57fa700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee8c053af0 0x7fee8c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.939+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fee9806bfd0 con 0x7fee88003e20 2022-07-20T03:12:55.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.939+0000 7fee9effd700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee8c053af0 0x7fee8c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.939+0000 7fee9effd700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee8c053af0 0x7fee8c055fb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fee9001c000 tx=0x7fee90019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:55.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:54.947+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee98042e20 con 0x7fee88003e20 2022-07-20T03:12:55.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.115+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7feea80574e0 con 0x7fee88003e20 2022-07-20T03:12:55.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.123+0000 7feea57fa700 1 -- 172.21.15.61:0/4155419075 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v24) v1 ==== 184+0+0 (secure 0 0 0) 0x7fee98046440 con 0x7fee88003e20 2022-07-20T03:12:55.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee8c053af0 msgr2=0x7fee8c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee8c053af0 0x7fee8c055fb0 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fee9001c000 tx=0x7fee90019000).stop 2022-07-20T03:12:55.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 msgr2=0x7feea80b7ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7feea80b7ef0 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fee9800b000 tx=0x7fee98009040).stop 2022-07-20T03:12:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 shutdown_connections 2022-07-20T03:12:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee8c053af0 0x7fee8c055fb0 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feea8151f20 0x7feea81523a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feea8136e00 0x7feea8137260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 --2- 172.21.15.61:0/4155419075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88003e20 0x7feea80b7ef0 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 >> 172.21.15.61:0/4155419075 conn(0x7feea8067980 msgr2=0x7feea8142460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 shutdown_connections 2022-07-20T03:12:55.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.127+0000 7feeae0e4700 1 -- 172.21.15.61:0/4155419075 wait complete. 2022-07-20T03:12:55.140 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:12:55.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:55.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 Processor -- start 2022-07-20T03:12:55.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- start start 2022-07-20T03:12:55.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e40b1e00 0x7f18e414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f18e405c3f0 0x7f18e40ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f18e405d480 con 0x7f18e405bd90 2022-07-20T03:12:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18e40b21e0 con 0x7f18e405c3f0 2022-07-20T03:12:55.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e405cf40 con 0x7f18e40b1e00 2022-07-20T03:12:55.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f18e405bd90 0x7f18e4146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42644/0 (socket says 172.21.15.61:42644) 2022-07-20T03:12:55.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e3fff700 1 -- 172.21.15.61:0/4237619505 learned_addr learned my addr 172.21.15.61:0/4237619505 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:55.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18daffd700 1 --2- 172.21.15.61:0/4237619505 >> v2:172.21.15.117:3300/0 conn(0x7f18e405c3f0 0x7f18e40ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e37fe700 1 --2- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e40b1e00 0x7f18e414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e2ffd700 1 -- 172.21.15.61:0/4237619505 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2959237498 0 0) 0x7f18e405d480 con 0x7f18e405bd90 2022-07-20T03:12:55.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e2ffd700 1 -- 172.21.15.61:0/4237619505 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f18c4003260 con 0x7f18e405bd90 2022-07-20T03:12:55.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18daffd700 1 -- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e40b1e00 msgr2=0x7f18e414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:55.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18daffd700 1 --2- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e40b1e00 0x7f18e414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18daffd700 1 -- 172.21.15.61:0/4237619505 >> v1:172.21.15.61:6789/0 conn(0x7f18e405bd90 legacy=0x7f18e4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18daffd700 1 -- 172.21.15.61:0/4237619505 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18e405d1e0 con 0x7f18e405c3f0 2022-07-20T03:12:55.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18daffd700 1 --2- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e405c3f0 0x7f18e40ac7e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f18cc002f70 tx=0x7f18cc006df0).ready entity=mon.1 client_cookie=118d260e4d46bf59 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:55.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e2ffd700 1 -- 172.21.15.61:0/4237619505 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18cc00d4a0 con 0x7f18e405c3f0 2022-07-20T03:12:55.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e2ffd700 1 -- 172.21.15.61:0/4237619505 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18cc000e90 con 0x7f18e405c3f0 2022-07-20T03:12:55.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e2ffd700 1 -- 172.21.15.61:0/4237619505 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18cc00d7f0 con 0x7f18e405c3f0 2022-07-20T03:12:55.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e405c3f0 msgr2=0x7f18e40ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e405c3f0 0x7f18e40ac7e0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f18cc002f70 tx=0x7f18cc006df0).stop 2022-07-20T03:12:55.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- 172.21.15.61:0/4237619505 shutdown_connections 2022-07-20T03:12:55.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e405c3f0 0x7f18e40ac7e0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/4237619505 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e40b1e00 0x7f18e414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- 172.21.15.61:0/4237619505 >> 172.21.15.61:0/4237619505 conn(0x7f18e40611c0 msgr2=0x7f18e4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:55.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- 172.21.15.61:0/4237619505 shutdown_connections 2022-07-20T03:12:55.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.247+0000 7f18e9c5f700 1 -- 172.21.15.61:0/4237619505 wait complete. 2022-07-20T03:12:55.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 Processor -- start 2022-07-20T03:12:55.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 -- start start 2022-07-20T03:12:55.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 0x7f18e4149320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e40b1e00 0x7f18e4156500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e4063a80 0x7f18e415a120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e4066500 con 0x7f18e405c3f0 2022-07-20T03:12:55.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18e409e080 con 0x7f18e40b1e00 2022-07-20T03:12:55.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f18e4159a50 con 0x7f18e4063a80 2022-07-20T03:12:55.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 0x7f18e4149320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 0x7f18e4149320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46048/0 (socket says 172.21.15.61:46048) 2022-07-20T03:12:55.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 -- 172.21.15.61:0/1614274346 learned_addr learned my addr 172.21.15.61:0/1614274346 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:55.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18daffd700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e40b1e00 0x7f18e4156500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e3fff700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e4063a80 0x7f18e415a120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 -- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e4063a80 msgr2=0x7f18e415a120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e4063a80 0x7f18e415a120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 -- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e40b1e00 msgr2=0x7f18e4156500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e40b1e00 0x7f18e4156500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 -- 172.21.15.61:0/1614274346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18cc00b180 con 0x7f18e405c3f0 2022-07-20T03:12:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e3fff700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e4063a80 0x7f18e415a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e37fe700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 0x7f18e4149320 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f18d400c000 tx=0x7f18d4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18d4010780 con 0x7f18e405c3f0 2022-07-20T03:12:55.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f18d40108e0 con 0x7f18e405c3f0 2022-07-20T03:12:55.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18d4010c40 con 0x7f18e405c3f0 2022-07-20T03:12:55.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18e40642d0 con 0x7f18e405c3f0 2022-07-20T03:12:55.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.251+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18e4159c80 con 0x7f18e405c3f0 2022-07-20T03:12:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.255+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18e4159e80 con 0x7f18e405c3f0 2022-07-20T03:12:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.255+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f18d401c460 con 0x7f18e405c3f0 2022-07-20T03:12:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.255+0000 7f18e17fa700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18c8053a40 0x7f18c8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:55.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.255+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f18d406c250 con 0x7f18e405c3f0 2022-07-20T03:12:55.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.255+0000 7f18daffd700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18c8053a40 0x7f18c8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:55.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.255+0000 7f18daffd700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18c8053a40 0x7f18c8055f00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f18cc01a000 tx=0x7f18cc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:55.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.263+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f18e4159e80 con 0x7f18e405c3f0 2022-07-20T03:12:55.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f18e40b5c20 con 0x7f18e405c3f0 2022-07-20T03:12:55.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e17fa700 1 -- 172.21.15.61:0/1614274346 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f18d40430d0 con 0x7f18e405c3f0 2022-07-20T03:12:55.521 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:55.521 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:12:55.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18c8053a40 msgr2=0x7f18c8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18c8053a40 0x7f18c8055f00 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f18cc01a000 tx=0x7f18cc014000).stop 2022-07-20T03:12:55.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 msgr2=0x7f18e4149320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:55.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 0x7f18e4149320 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f18d400c000 tx=0x7f18d4006df0).stop 2022-07-20T03:12:55.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 shutdown_connections 2022-07-20T03:12:55.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18c8053a40 0x7f18c8055f00 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18e40b1e00 0x7f18e4156500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18e4063a80 0x7f18e415a120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 --2- 172.21.15.61:0/1614274346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18e405c3f0 0x7f18e4149320 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:55.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 >> 172.21.15.61:0/1614274346 conn(0x7f18e405bd90 msgr2=0x7f18e40ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:55.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 shutdown_connections 2022-07-20T03:12:55.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.423+0000 7f18e9c5f700 1 -- 172.21.15.61:0/1614274346 wait complete. 2022-07-20T03:12:55.541 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:12:55.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:12:52.013020+0000', 'modified': '2022-07-20T03:12:53.015972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5982}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5982': {'gid': 5982, 'name': 'a', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2218834214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2218834214}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2218834214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:12:55.754 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:12:55.755 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:12:55.891 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5982,"want_state":"up:active","state":"up:active","rank_uptime":3.8657312419999998,"mdsmap_epoch":35,"osdmap_epoch":61,"osdmap_epoch_barrier":60,"uptime":8.24791688} 2022-07-20T03:12:55.892 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5982, "mdsmap_epoch": 35, "osdmap_epoch": 61, "osdmap_epoch_barrier": 60, "rank_uptime": 3.865731242, "state": "up:active", "uptime": 8.24791688, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:12:55.893 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:12:56.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 Processor -- start 2022-07-20T03:12:56.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- start start 2022-07-20T03:12:56.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00ac260 0x7fc0d00ac640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc0d00b3620 0x7fc0d0096890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0d00b22d0 con 0x7fc0d0141200 2022-07-20T03:12:56.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc0d00b26d0 con 0x7fc0d00b3620 2022-07-20T03:12:56.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0140ac0 con 0x7fc0d00ac260 2022-07-20T03:12:56.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc0d00b3620 0x7fc0d0096890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc0d00b3620 0x7fc0d0096890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46058/0 (socket says 172.21.15.61:46058) 2022-07-20T03:12:56.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 -- 172.21.15.61:0/4055376389 learned_addr learned my addr 172.21.15.61:0/4055376389 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:56.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0cf7fe700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00ac260 0x7fc0d00ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 -- 172.21.15.61:0/4055376389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00ac260 msgr2=0x7fc0d00ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:56.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00ac260 0x7fc0d00ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 -- 172.21.15.61:0/4055376389 >> v1:172.21.15.117:6789/0 conn(0x7fc0d0141200 legacy=0x7fc0d01400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ce7fc700 1 -- 172.21.15.61:0/4055376389 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3222952938 0 0) 0x7fc0d00b22d0 con 0x7fc0d0141200 2022-07-20T03:12:56.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 -- 172.21.15.61:0/4055376389 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0d0140d60 con 0x7fc0d00b3620 2022-07-20T03:12:56.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0cf7fe700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00ac260 0x7fc0d00ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:12:56.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ceffd700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00b3620 0x7fc0d0096890 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fc0b8002f70 tx=0x7fc0b8006e50).ready entity=mon.0 client_cookie=1f68558df7f0c6ea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ce7fc700 1 -- 172.21.15.61:0/4055376389 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0b800d5c0 con 0x7fc0d00b3620 2022-07-20T03:12:56.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ce7fc700 1 -- 172.21.15.61:0/4055376389 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0b8000e90 con 0x7fc0d00b3620 2022-07-20T03:12:56.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0ce7fc700 1 -- 172.21.15.61:0/4055376389 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0b800d910 con 0x7fc0d00b3620 2022-07-20T03:12:56.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4055376389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00b3620 msgr2=0x7fc0d0096890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00b3620 0x7fc0d0096890 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7fc0b8002f70 tx=0x7fc0b8006e50).stop 2022-07-20T03:12:56.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4055376389 shutdown_connections 2022-07-20T03:12:56.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00b3620 0x7fc0d0096890 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4055376389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00ac260 0x7fc0d00ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4055376389 >> 172.21.15.61:0/4055376389 conn(0x7fc0d00b47d0 msgr2=0x7fc0d005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:56.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4055376389 shutdown_connections 2022-07-20T03:12:56.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:55.999+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4055376389 wait complete. 2022-07-20T03:12:56.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.003+0000 7fc0d5ece700 1 Processor -- start 2022-07-20T03:12:56.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.003+0000 7fc0d5ece700 1 -- start start 2022-07-20T03:12:56.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.003+0000 7fc0d5ece700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 0x7fc0d009b220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.003+0000 7fc0d5ece700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00b3620 0x7fc0d0062fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0d0141200 0x7fc0d009bc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d0099070 con 0x7fc0d00ac260 2022-07-20T03:12:56.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d009b860 con 0x7fc0d0141200 2022-07-20T03:12:56.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0d00ae6a0 con 0x7fc0d00b3620 2022-07-20T03:12:56.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 0x7fc0d009b220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 0x7fc0d009b220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46062/0 (socket says 172.21.15.61:46062) 2022-07-20T03:12:56.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 -- 172.21.15.61:0/4172110289 learned_addr learned my addr 172.21.15.61:0/4172110289 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:56.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0ceffd700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00b3620 0x7fc0d0062fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cffff700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0d0141200 0x7fc0d009bc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 -- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00b3620 msgr2=0x7fc0d0062fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00b3620 0x7fc0d0062fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 -- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0d0141200 msgr2=0x7fc0d009bc90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0d0141200 0x7fc0d009bc90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 -- 172.21.15.61:0/4172110289 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0c0009470 con 0x7fc0d00ac260 2022-07-20T03:12:56.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0cf7fe700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 0x7fc0d009b220 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fc0c000c000 tx=0x7fc0c00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c000c850 con 0x7fc0d00ac260 2022-07-20T03:12:56.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0c000c9b0 con 0x7fc0d00ac260 2022-07-20T03:12:56.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0b800b1e0 con 0x7fc0d00ac260 2022-07-20T03:12:56.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0c000cd10 con 0x7fc0d00ac260 2022-07-20T03:12:56.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc0d009ccb0 con 0x7fc0d00ac260 2022-07-20T03:12:56.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.007+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0d01407d0 con 0x7fc0d00ac260 2022-07-20T03:12:56.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.011+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc0c0022070 con 0x7fc0d00ac260 2022-07-20T03:12:56.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.011+0000 7fc0ccff9700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0b4053920 0x7fc0b4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.011+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc0c006c5d0 con 0x7fc0d00ac260 2022-07-20T03:12:56.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.011+0000 7fc0ceffd700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0b4053920 0x7fc0b4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.011+0000 7fc0ceffd700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0b4053920 0x7fc0b4055de0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc0b801a000 tx=0x7fc0b8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.015+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0c0043530 con 0x7fc0d00ac260 2022-07-20T03:12:56.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.143+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc0d00574e0 con 0x7fc0d00ac260 2022-07-20T03:12:56.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.147+0000 7fc0ccff9700 1 -- 172.21.15.61:0/4172110289 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v61) v1 ==== 74+0+16089 (secure 0 0 0) 0x7fc0c0046b50 con 0x7fc0d00ac260 2022-07-20T03:12:56.149 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:56.149 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":61,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:12:53.016664+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":11,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:12:51.013317+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"60","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":12,"pool_name":"cephfs_data","create_time":"2022-07-20T03:12:51.616869+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"61","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:12:56.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0b4053920 msgr2=0x7fc0b4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0b4053920 0x7fc0b4055de0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7fc0b801a000 tx=0x7fc0b8014000).stop 2022-07-20T03:12:56.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 msgr2=0x7fc0d009b220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 0x7fc0d009b220 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fc0c000c000 tx=0x7fc0c00091d0).stop 2022-07-20T03:12:56.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 shutdown_connections 2022-07-20T03:12:56.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0b4053920 0x7fc0b4055de0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc0d0141200 0x7fc0d009bc90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0d00b3620 0x7fc0d0062fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 --2- 172.21.15.61:0/4172110289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc0d00ac260 0x7fc0d009b220 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 >> 172.21.15.61:0/4172110289 conn(0x7fc0d00b47d0 msgr2=0x7fc0d0096890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:56.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 shutdown_connections 2022-07-20T03:12:56.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.151+0000 7fc0d5ece700 1 -- 172.21.15.61:0/4172110289 wait complete. 2022-07-20T03:12:56.171 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 11, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:12:51.013317+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '60', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 12, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:12:51.616869+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '61', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:12:56.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:12:56.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 Processor -- start 2022-07-20T03:12:56.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- start start 2022-07-20T03:12:56.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f176c0b23b0 0x7f176c05c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b0580 0x7f176c0b1bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f176c146190 0x7f176c146620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c067f70 con 0x7f176c146190 2022-07-20T03:12:56.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f176c0ad0e0 con 0x7f176c0b23b0 2022-07-20T03:12:56.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c0b16a0 con 0x7f176c0b0580 2022-07-20T03:12:56.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176affd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f176c0b23b0 0x7f176c05c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176affd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f176c0b23b0 0x7f176c05c0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:55750/0 (socket says 172.21.15.61:55750) 2022-07-20T03:12:56.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f176c146190 0x7f176c146620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176affd700 1 -- 172.21.15.61:0/3075386391 learned_addr learned my addr 172.21.15.61:0/3075386391 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:56.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176a7fc700 1 --2- 172.21.15.61:0/3075386391 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b0580 0x7f176c0b1bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 -- 172.21.15.61:0/3075386391 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b0580 msgr2=0x7f176c0b1bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:56.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 --2- 172.21.15.61:0/3075386391 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b0580 0x7f176c0b1bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 -- 172.21.15.61:0/3075386391 >> v2:172.21.15.117:3300/0 conn(0x7f176c0b23b0 msgr2=0x7f176c05c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:56.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 --2- 172.21.15.61:0/3075386391 >> v2:172.21.15.117:3300/0 conn(0x7f176c0b23b0 0x7f176c05c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 -- 172.21.15.61:0/3075386391 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f176c0ae660 con 0x7f176c146190 2022-07-20T03:12:56.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176b7fe700 1 --2- 172.21.15.61:0/3075386391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c146620 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f17640032f0 tx=0x7f17640091d0).ready entity=mon.0 client_cookie=2259df9eeb6965d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f1769ffb700 1 -- 172.21.15.61:0/3075386391 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f176400b600 con 0x7f176c146190 2022-07-20T03:12:56.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f1769ffb700 1 -- 172.21.15.61:0/3075386391 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f176400b760 con 0x7f176c146190 2022-07-20T03:12:56.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f1769ffb700 1 -- 172.21.15.61:0/3075386391 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f176400b8d0 con 0x7f176c146190 2022-07-20T03:12:56.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- 172.21.15.61:0/3075386391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 msgr2=0x7f176c146620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- 172.21.15.61:0/3075386391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c146620 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7f17640032f0 tx=0x7f17640091d0).stop 2022-07-20T03:12:56.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- 172.21.15.61:0/3075386391 shutdown_connections 2022-07-20T03:12:56.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- 172.21.15.61:0/3075386391 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c146620 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- 172.21.15.61:0/3075386391 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b0580 0x7f176c0b1bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 --2- 172.21.15.61:0/3075386391 >> v2:172.21.15.117:3300/0 conn(0x7f176c0b23b0 0x7f176c05c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- 172.21.15.61:0/3075386391 >> 172.21.15.61:0/3075386391 conn(0x7f176c0632d0 msgr2=0x7f176c0636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:56.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- 172.21.15.61:0/3075386391 shutdown_connections 2022-07-20T03:12:56.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.303+0000 7f176bfff700 1 -- 172.21.15.61:0/3075386391 wait complete. 2022-07-20T03:12:56.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 Processor -- start 2022-07-20T03:12:56.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 -- start start 2022-07-20T03:12:56.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f176c0b0580 0x7f176c0670e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b23b0 0x7f176c155ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c1154a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c115200 con 0x7f176c146190 2022-07-20T03:12:56.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f176c1555a0 con 0x7f176c0b0580 2022-07-20T03:12:56.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f176c155840 con 0x7f176c0b23b0 2022-07-20T03:12:56.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c1154a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c1154a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46080/0 (socket says 172.21.15.61:46080) 2022-07-20T03:12:56.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 -- 172.21.15.61:0/1098857947 learned_addr learned my addr 172.21.15.61:0/1098857947 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176a7fc700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b23b0 0x7f176c155ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176affd700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f176c0b0580 0x7f176c0670e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 -- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b23b0 msgr2=0x7f176c155ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b23b0 0x7f176c155ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 -- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f176c0b0580 msgr2=0x7f176c0670e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f176c0b0580 0x7f176c0670e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 -- 172.21.15.61:0/1098857947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1764009470 con 0x7f176c146190 2022-07-20T03:12:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176affd700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f176c0b0580 0x7f176c0670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.307+0000 7f176b7fe700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c1154a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f17640032f0 tx=0x7f1764016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f176400b600 con 0x7f176c146190 2022-07-20T03:12:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1764011ca0 con 0x7f176c146190 2022-07-20T03:12:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f176c1370f0 con 0x7f176c146190 2022-07-20T03:12:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f176401f3f0 con 0x7f176c146190 2022-07-20T03:12:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f176c115990 con 0x7f176c146190 2022-07-20T03:12:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f176c155a00 con 0x7f176c146190 2022-07-20T03:12:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1764026030 con 0x7f176c146190 2022-07-20T03:12:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f1763fff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17500539b0 0x7f1750055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f176406c9b0 con 0x7f176c146190 2022-07-20T03:12:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f176affd700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17500539b0 0x7f1750055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.311+0000 7f176affd700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17500539b0 0x7f1750055e70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f1754008000 tx=0x7f1754008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.319+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1764047340 con 0x7f176c146190 2022-07-20T03:12:56.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.443+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f176c0ae900 con 0x7f176c146190 2022-07-20T03:12:56.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.443+0000 7f1763fff700 1 -- 172.21.15.61:0/1098857947 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v61) v1 ==== 110+0+10 (secure 0 0 0) 0x7f17640240b0 con 0x7f176c146190 2022-07-20T03:12:56.447 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:12:56.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.443+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17500539b0 msgr2=0x7f1750055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.443+0000 7f176bfff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17500539b0 0x7f1750055e70 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f1754008000 tx=0x7f1754008040).stop 2022-07-20T03:12:56.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.443+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 msgr2=0x7f176c1154a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.443+0000 7f176bfff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c1154a0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f17640032f0 tx=0x7f1764016000).stop 2022-07-20T03:12:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 shutdown_connections 2022-07-20T03:12:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17500539b0 0x7f1750055e70 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f176c146190 0x7f176c1154a0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f176c0b23b0 0x7f176c155ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 --2- 172.21.15.61:0/1098857947 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f176c0b0580 0x7f176c0670e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 >> 172.21.15.61:0/1098857947 conn(0x7f176c0632d0 msgr2=0x7f176c0adad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:56.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 shutdown_connections 2022-07-20T03:12:56.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.447+0000 7f176bfff700 1 -- 172.21.15.61:0/1098857947 wait complete. 2022-07-20T03:12:56.569 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:12:56.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 Processor -- start 2022-07-20T03:12:56.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- start start 2022-07-20T03:12:56.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41462a0 0x7f12a4146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12a40988b0 0x7f12a4146b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12a40afdb0 con 0x7f12a41422c0 2022-07-20T03:12:56.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12a40b0050 con 0x7f12a40988b0 2022-07-20T03:12:56.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a405fcd0 con 0x7f12a41462a0 2022-07-20T03:12:56.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12a40988b0 0x7f12a4146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a2d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f12a41422c0 0x7f12a4060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42684/0 (socket says 172.21.15.61:42684) 2022-07-20T03:12:56.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a2d9d700 1 -- 172.21.15.61:0/3195318885 learned_addr learned my addr 172.21.15.61:0/3195318885 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:56.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a259c700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41462a0 0x7f12a4146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a1d9b700 1 -- 172.21.15.61:0/3195318885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41462a0 msgr2=0x7f12a4146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:56.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a1d9b700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41462a0 0x7f12a4146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a1d9b700 1 -- 172.21.15.61:0/3195318885 >> v1:172.21.15.61:6789/0 conn(0x7f12a41422c0 legacy=0x7f12a4060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a1d9b700 1 -- 172.21.15.61:0/3195318885 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a40afae0 con 0x7f12a40988b0 2022-07-20T03:12:56.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a259c700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41462a0 0x7f12a4146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:56.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a1d9b700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a40988b0 0x7f12a4146b70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f1298002f70 tx=0x7f1298006df0).ready entity=mon.0 client_cookie=9d4d7fb6a0c3fccc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a159a700 1 -- 172.21.15.61:0/3195318885 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f129800d570 con 0x7f12a40988b0 2022-07-20T03:12:56.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a159a700 1 -- 172.21.15.61:0/3195318885 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1298000e90 con 0x7f12a40988b0 2022-07-20T03:12:56.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a159a700 1 -- 172.21.15.61:0/3195318885 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f129800d8c0 con 0x7f12a40988b0 2022-07-20T03:12:56.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3195318885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a40988b0 msgr2=0x7f12a4146b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a40988b0 0x7f12a4146b70 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f1298002f70 tx=0x7f1298006df0).stop 2022-07-20T03:12:56.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3195318885 shutdown_connections 2022-07-20T03:12:56.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a40988b0 0x7f12a4146b70 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3195318885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41462a0 0x7f12a4146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3195318885 >> 172.21.15.61:0/3195318885 conn(0x7f12a40b5f00 msgr2=0x7f12a40b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:56.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3195318885 shutdown_connections 2022-07-20T03:12:56.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.695+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3195318885 wait complete. 2022-07-20T03:12:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 Processor -- start 2022-07-20T03:12:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 -- start start 2022-07-20T03:12:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12a40988b0 0x7f12a41402e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41422c0 0x7f12a409c560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 0x7f12a40b1cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a4149150 con 0x7f12a41462a0 2022-07-20T03:12:56.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12a41493f0 con 0x7f12a40988b0 2022-07-20T03:12:56.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a8bf0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f12a40b1810 con 0x7f12a41422c0 2022-07-20T03:12:56.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 0x7f12a40b1cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 0x7f12a40b1cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46094/0 (socket says 172.21.15.61:46094) 2022-07-20T03:12:56.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a259c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12a40988b0 0x7f12a41402e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a1d9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41422c0 0x7f12a409c560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 -- 172.21.15.61:0/3422827026 learned_addr learned my addr 172.21.15.61:0/3422827026 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:56.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 -- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41422c0 msgr2=0x7f12a409c560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41422c0 0x7f12a409c560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 -- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12a40988b0 msgr2=0x7f12a41402e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12a40988b0 0x7f12a41402e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 -- 172.21.15.61:0/3422827026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129800b180 con 0x7f12a41462a0 2022-07-20T03:12:56.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.699+0000 7f12a2d9d700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 0x7f12a40b1cb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f12a4060050 tx=0x7f129c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f129c005ba0 con 0x7f12a41462a0 2022-07-20T03:12:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f129c005d00 con 0x7f12a41462a0 2022-07-20T03:12:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12a409ca50 con 0x7f12a41462a0 2022-07-20T03:12:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f129c0072c0 con 0x7f12a41462a0 2022-07-20T03:12:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12a4148cb0 con 0x7f12a41462a0 2022-07-20T03:12:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12a40b19d0 con 0x7f12a41462a0 2022-07-20T03:12:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f129c010070 con 0x7f12a41462a0 2022-07-20T03:12:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12937fe700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1280053a40 0x7f1280055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:56.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f129c06b630 con 0x7f12a41462a0 2022-07-20T03:12:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12a259c700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1280053a40 0x7f1280055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.703+0000 7f12a259c700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1280053a40 0x7f1280055f00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f129400c000 tx=0x7f1294008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.715+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f129c042590 con 0x7f12a41462a0 2022-07-20T03:12:56.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.847+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f12a40574e0 con 0x7f12a41462a0 2022-07-20T03:12:56.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.847+0000 7f12937fe700 1 -- 172.21.15.61:0/3422827026 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v61) v1 ==== 92+0+10 (secure 0 0 0) 0x7f129c018020 con 0x7f12a41462a0 2022-07-20T03:12:56.851 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:12:56.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1280053a40 msgr2=0x7f1280055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1280053a40 0x7f1280055f00 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f129400c000 tx=0x7f1294008040).stop 2022-07-20T03:12:56.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 msgr2=0x7f12a40b1cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:56.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 0x7f12a40b1cb0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f12a4060050 tx=0x7f129c007f40).stop 2022-07-20T03:12:56.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 shutdown_connections 2022-07-20T03:12:56.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1280053a40 0x7f1280055f00 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12a41462a0 0x7f12a40b1cb0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12a41422c0 0x7f12a409c560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 --2- 172.21.15.61:0/3422827026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12a40988b0 0x7f12a41402e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:56.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 >> 172.21.15.61:0/3422827026 conn(0x7f12a40b5f00 msgr2=0x7f12a4142730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:56.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 shutdown_connections 2022-07-20T03:12:56.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.851+0000 7f12a8bf0700 1 -- 172.21.15.61:0/3422827026 wait complete. 2022-07-20T03:12:56.867 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:12:57.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33d46b700 1 Processor -- start 2022-07-20T03:12:57.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33d46b700 1 -- start start 2022-07-20T03:12:57.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33d46b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff3380b02e0 0x7ff3381431d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33d46b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3380ad380 con 0x7ff3380acfa0 2022-07-20T03:12:57.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33d46b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff3380af470 con 0x7ff3380b1030 2022-07-20T03:12:57.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33d46b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff33813f030 con 0x7ff3380b02e0 2022-07-20T03:12:57.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff337fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff3380acfa0 0x7ff338142ac0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42698/0 (socket says 172.21.15.61:42698) 2022-07-20T03:12:57.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff337fff700 1 -- 172.21.15.61:0/917983380 learned_addr learned my addr 172.21.15.61:0/917983380 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:57.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:56.999+0000 7ff33cc6a700 1 --2- 172.21.15.61:0/917983380 >> v2:172.21.15.117:3300/0 conn(0x7ff3380b02e0 0x7ff3381431d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff336ffd700 1 -- 172.21.15.61:0/917983380 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1967777741 0 0) 0x7ff3380ad380 con 0x7ff3380acfa0 2022-07-20T03:12:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff336ffd700 1 -- 172.21.15.61:0/917983380 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff318003260 con 0x7ff3380acfa0 2022-07-20T03:12:57.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33cc6a700 1 -- 172.21.15.61:0/917983380 >> v1:172.21.15.117:6789/0 conn(0x7ff3380b1030 legacy=0x7ff3380ae4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33cc6a700 1 -- 172.21.15.61:0/917983380 >> v1:172.21.15.61:6789/0 conn(0x7ff3380acfa0 legacy=0x7ff338142ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33cc6a700 1 -- 172.21.15.61:0/917983380 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3380b1630 con 0x7ff3380b02e0 2022-07-20T03:12:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33cc6a700 1 --2- 172.21.15.61:0/917983380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b02e0 0x7ff3381431d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff3300032f0 tx=0x7ff33000a370).ready entity=mon.1 client_cookie=c6ddb5d67026affb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff336ffd700 1 -- 172.21.15.61:0/917983380 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff33000cba0 con 0x7ff3380b02e0 2022-07-20T03:12:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff336ffd700 1 -- 172.21.15.61:0/917983380 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff33000cd00 con 0x7ff3380b02e0 2022-07-20T03:12:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff336ffd700 1 -- 172.21.15.61:0/917983380 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff33000f660 con 0x7ff3380b02e0 2022-07-20T03:12:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 -- 172.21.15.61:0/917983380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b02e0 msgr2=0x7ff3381431d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 --2- 172.21.15.61:0/917983380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b02e0 0x7ff3381431d0 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7ff3300032f0 tx=0x7ff33000a370).stop 2022-07-20T03:12:57.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 -- 172.21.15.61:0/917983380 shutdown_connections 2022-07-20T03:12:57.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 --2- 172.21.15.61:0/917983380 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b02e0 0x7ff3381431d0 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 -- 172.21.15.61:0/917983380 >> 172.21.15.61:0/917983380 conn(0x7ff3380632d0 msgr2=0x7ff3380636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:57.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 -- 172.21.15.61:0/917983380 shutdown_connections 2022-07-20T03:12:57.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.003+0000 7ff33d46b700 1 -- 172.21.15.61:0/917983380 wait complete. 2022-07-20T03:12:57.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 Processor -- start 2022-07-20T03:12:57.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 -- start start 2022-07-20T03:12:57.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 0x7ff338111360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3380b02e0 0x7ff3380b3950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b1030 0x7ff3380642c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33810f160 con 0x7ff3380acfa0 2022-07-20T03:12:57.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff33810f400 con 0x7ff3380b1030 2022-07-20T03:12:57.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33d46b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff338065cd0 con 0x7ff3380b02e0 2022-07-20T03:12:57.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 0x7ff338111360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 0x7ff338111360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46104/0 (socket says 172.21.15.61:46104) 2022-07-20T03:12:57.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 -- 172.21.15.61:0/2321330383 learned_addr learned my addr 172.21.15.61:0/2321330383 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:57.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff3377fe700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3380b02e0 0x7ff3380b3950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff33cc6a700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b1030 0x7ff3380642c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 -- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3380b02e0 msgr2=0x7ff3380b3950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3380b02e0 0x7ff3380b3950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 -- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b1030 msgr2=0x7ff3380642c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b1030 0x7ff3380642c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 -- 172.21.15.61:0/2321330383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff33000a610 con 0x7ff3380acfa0 2022-07-20T03:12:57.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.007+0000 7ff337fff700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 0x7ff338111360 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7ff320001110 tx=0x7ff320007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff3200070d0 con 0x7ff3380acfa0 2022-07-20T03:12:57.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff320007230 con 0x7ff3380acfa0 2022-07-20T03:12:57.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3380647b0 con 0x7ff3380acfa0 2022-07-20T03:12:57.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff320005610 con 0x7ff3380acfa0 2022-07-20T03:12:57.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff33810e9d0 con 0x7ff3380acfa0 2022-07-20T03:12:57.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff338065e90 con 0x7ff3380acfa0 2022-07-20T03:12:57.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff320010070 con 0x7ff3380acfa0 2022-07-20T03:12:57.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3357fa700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff31c053a60 0x7ff31c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff32006b400 con 0x7ff3380acfa0 2022-07-20T03:12:57.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.011+0000 7ff3377fe700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff31c053a60 0x7ff31c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.015+0000 7ff3377fe700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff31c053a60 0x7ff31c055f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff32800d000 tx=0x7ff328007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.019+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff320042360 con 0x7ff3380acfa0 2022-07-20T03:12:57.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff338069a20 con 0x7ff3380acfa0 2022-07-20T03:12:57.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff3357fa700 1 -- 172.21.15.61:0/2321330383 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v61) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff320045980 con 0x7ff3380acfa0 2022-07-20T03:12:57.209 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:12:57.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff31c053a60 msgr2=0x7ff31c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff31c053a60 0x7ff31c055f20 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7ff32800d000 tx=0x7ff328007b90).stop 2022-07-20T03:12:57.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 msgr2=0x7ff338111360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 0x7ff338111360 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7ff320001110 tx=0x7ff320007ad0).stop 2022-07-20T03:12:57.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 shutdown_connections 2022-07-20T03:12:57.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff31c053a60 0x7ff31c055f20 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3380b1030 0x7ff3380642c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3380b02e0 0x7ff3380b3950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 --2- 172.21.15.61:0/2321330383 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3380acfa0 0x7ff338111360 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 >> 172.21.15.61:0/2321330383 conn(0x7ff3380632d0 msgr2=0x7ff33813fb60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:57.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 shutdown_connections 2022-07-20T03:12:57.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.151+0000 7ff33d46b700 1 -- 172.21.15.61:0/2321330383 wait complete. 2022-07-20T03:12:57.337 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:12:57.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 Processor -- start 2022-07-20T03:12:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 -- start start 2022-07-20T03:12:57.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d680b2210 0x7f9d680b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680ac670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d680624e0 con 0x7f9d68062090 2022-07-20T03:12:57.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d6805fbc0 con 0x7f9d680b2210 2022-07-20T03:12:57.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6cf6a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d6805fe60 con 0x7f9d680ac240 2022-07-20T03:12:57.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d65d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9d68062090 0x7f9d681467b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42712/0 (socket says 172.21.15.61:42712) 2022-07-20T03:12:57.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d65d9b700 1 -- 172.21.15.61:0/2190671284 learned_addr learned my addr 172.21.15.61:0/2190671284 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:57.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6659c700 1 --2- 172.21.15.61:0/2190671284 >> v2:172.21.15.117:3300/0 conn(0x7f9d680b2210 0x7f9d680b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d66d9d700 1 --2- 172.21.15.61:0/2190671284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680ac670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6559a700 1 -- 172.21.15.61:0/2190671284 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2836796646 0 0) 0x7f9d50001110 con 0x7f9d68062090 2022-07-20T03:12:57.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6559a700 1 -- 172.21.15.61:0/2190671284 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d48003260 con 0x7f9d68062090 2022-07-20T03:12:57.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d66d9d700 1 -- 172.21.15.61:0/2190671284 >> v2:172.21.15.117:3300/0 conn(0x7f9d680b2210 msgr2=0x7f9d680b25f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:57.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d66d9d700 1 --2- 172.21.15.61:0/2190671284 >> v2:172.21.15.117:3300/0 conn(0x7f9d680b2210 0x7f9d680b25f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d66d9d700 1 -- 172.21.15.61:0/2190671284 >> v1:172.21.15.61:6789/0 conn(0x7f9d68062090 legacy=0x7f9d681467b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d66d9d700 1 -- 172.21.15.61:0/2190671284 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d68060100 con 0x7f9d680ac240 2022-07-20T03:12:57.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d66d9d700 1 --2- 172.21.15.61:0/2190671284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680ac670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9d60002f70 tx=0x7f9d60006df0).ready entity=mon.2 client_cookie=9faacef50f49e2ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.463+0000 7f9d6559a700 1 -- 172.21.15.61:0/2190671284 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d6000d4a0 con 0x7f9d680ac240 2022-07-20T03:12:57.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6559a700 1 -- 172.21.15.61:0/2190671284 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d60000e90 con 0x7f9d680ac240 2022-07-20T03:12:57.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6559a700 1 -- 172.21.15.61:0/2190671284 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d6000d7f0 con 0x7f9d680ac240 2022-07-20T03:12:57.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2190671284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 msgr2=0x7f9d680ac670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2190671284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680ac670 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f9d60002f70 tx=0x7f9d60006df0).stop 2022-07-20T03:12:57.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2190671284 shutdown_connections 2022-07-20T03:12:57.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2190671284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680ac670 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2190671284 >> v2:172.21.15.117:3300/0 conn(0x7f9d680b2210 0x7f9d680b25f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2190671284 >> 172.21.15.61:0/2190671284 conn(0x7f9d680b36c0 msgr2=0x7f9d680b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:57.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2190671284 shutdown_connections 2022-07-20T03:12:57.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2190671284 wait complete. 2022-07-20T03:12:57.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 Processor -- start 2022-07-20T03:12:57.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.467+0000 7f9d6cf6a700 1 -- start start 2022-07-20T03:12:57.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 0x7f9d68159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680b0bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d680b2210 0x7f9d680994e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d680975e0 con 0x7f9d68062090 2022-07-20T03:12:57.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d68097810 con 0x7f9d680b2210 2022-07-20T03:12:57.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d68097ab0 con 0x7f9d680ac240 2022-07-20T03:12:57.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 0x7f9d68159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 0x7f9d68159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46118/0 (socket says 172.21.15.61:46118) 2022-07-20T03:12:57.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 -- 172.21.15.61:0/2321408106 learned_addr learned my addr 172.21.15.61:0/2321408106 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:57.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d66d9d700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d680b2210 0x7f9d680994e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d65d9b700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680b0bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 -- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 msgr2=0x7f9d680b0bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680b0bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 -- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d680b2210 msgr2=0x7f9d680994e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d680b2210 0x7f9d680994e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 -- 172.21.15.61:0/2321408106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d6000b180 con 0x7f9d68062090 2022-07-20T03:12:57.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d66d9d700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d680b2210 0x7f9d680994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:12:57.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6659c700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 0x7f9d68159ac0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f9d5800c000 tx=0x7f9d580091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d58009560 con 0x7f9d68062090 2022-07-20T03:12:57.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d5800c850 con 0x7f9d68062090 2022-07-20T03:12:57.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d680b10b0 con 0x7f9d68062090 2022-07-20T03:12:57.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d5800cbb0 con 0x7f9d68062090 2022-07-20T03:12:57.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d6809c010 con 0x7f9d68062090 2022-07-20T03:12:57.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d68098a40 con 0x7f9d68062090 2022-07-20T03:12:57.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.471+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d58022070 con 0x7f9d68062090 2022-07-20T03:12:57.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.475+0000 7f9d5f7fe700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d4c0539b0 0x7f9d4c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.475+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9d5806c650 con 0x7f9d68062090 2022-07-20T03:12:57.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.475+0000 7f9d65d9b700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d4c0539b0 0x7f9d4c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.475+0000 7f9d65d9b700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d4c0539b0 0x7f9d4c055e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9d5000b000 tx=0x7f9d50009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.483+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d68098a40 con 0x7f9d68062090 2022-07-20T03:12:57.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.611+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f9d68098a40 con 0x7f9d68062090 2022-07-20T03:12:57.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d5f7fe700 1 -- 172.21.15.61:0/2321408106 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v61) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9d68098a40 con 0x7f9d68062090 2022-07-20T03:12:57.623 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:12:57.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d4c0539b0 msgr2=0x7f9d4c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d4c0539b0 0x7f9d4c055e70 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f9d5000b000 tx=0x7f9d50009040).stop 2022-07-20T03:12:57.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 msgr2=0x7f9d68159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 0x7f9d68159ac0 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f9d5800c000 tx=0x7f9d580091d0).stop 2022-07-20T03:12:57.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 shutdown_connections 2022-07-20T03:12:57.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d4c0539b0 0x7f9d4c055e70 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d680b2210 0x7f9d680994e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d680ac240 0x7f9d680b0bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 --2- 172.21.15.61:0/2321408106 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d68062090 0x7f9d68159ac0 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 >> 172.21.15.61:0/2321408106 conn(0x7f9d680b36c0 msgr2=0x7f9d68149ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:57.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 shutdown_connections 2022-07-20T03:12:57.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.615+0000 7f9d6cf6a700 1 -- 172.21.15.61:0/2321408106 wait complete. 2022-07-20T03:12:57.636 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:12:57.637 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:12:57.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 Processor -- start 2022-07-20T03:12:57.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- start start 2022-07-20T03:12:57.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fade80aea20 0x7fade8140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80675f0 0x7fade8140f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fade80aee00 con 0x7fade8141470 2022-07-20T03:12:57.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fade8144470 con 0x7fade80aea20 2022-07-20T03:12:57.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade8144710 con 0x7fade80675f0 2022-07-20T03:12:57.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadecc05700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fade8141470 0x7fade813a110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42728/0 (socket says 172.21.15.61:42728) 2022-07-20T03:12:57.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadecc05700 1 -- 172.21.15.61:0/2723457891 learned_addr learned my addr 172.21.15.61:0/2723457891 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:12:57.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7faddf7fe700 1 --2- 172.21.15.61:0/2723457891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80675f0 0x7fade8140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fade7fff700 1 --2- 172.21.15.61:0/2723457891 >> v2:172.21.15.117:3300/0 conn(0x7fade80aea20 0x7fade8140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7faddf7fe700 1 -- 172.21.15.61:0/2723457891 >> v2:172.21.15.117:3300/0 conn(0x7fade80aea20 msgr2=0x7fade8140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:12:57.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fade77fe700 1 -- 172.21.15.61:0/2723457891 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 834906397 0 0) 0x7fade80aee00 con 0x7fade8141470 2022-07-20T03:12:57.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7faddf7fe700 1 --2- 172.21.15.61:0/2723457891 >> v2:172.21.15.117:3300/0 conn(0x7fade80aea20 0x7fade8140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7faddf7fe700 1 -- 172.21.15.61:0/2723457891 >> v1:172.21.15.61:6789/0 conn(0x7fade8141470 legacy=0x7fade813a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7faddf7fe700 1 -- 172.21.15.61:0/2723457891 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fade81449b0 con 0x7fade80675f0 2022-07-20T03:12:57.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7faddf7fe700 1 --2- 172.21.15.61:0/2723457891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80675f0 0x7fade8140f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fade8063ee0 tx=0x7fadd0006df0).ready entity=mon.2 client_cookie=96da6fdd6274fc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fade77fe700 1 -- 172.21.15.61:0/2723457891 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd000d570 con 0x7fade80675f0 2022-07-20T03:12:57.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fade77fe700 1 -- 172.21.15.61:0/2723457891 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0000e90 con 0x7fade80675f0 2022-07-20T03:12:57.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fade77fe700 1 -- 172.21.15.61:0/2723457891 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd000d8c0 con 0x7fade80675f0 2022-07-20T03:12:57.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- 172.21.15.61:0/2723457891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80675f0 msgr2=0x7fade8140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 --2- 172.21.15.61:0/2723457891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80675f0 0x7fade8140f80 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fade8063ee0 tx=0x7fadd0006df0).stop 2022-07-20T03:12:57.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- 172.21.15.61:0/2723457891 shutdown_connections 2022-07-20T03:12:57.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 --2- 172.21.15.61:0/2723457891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80675f0 0x7fade8140f80 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 --2- 172.21.15.61:0/2723457891 >> v2:172.21.15.117:3300/0 conn(0x7fade80aea20 0x7fade8140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- 172.21.15.61:0/2723457891 >> 172.21.15.61:0/2723457891 conn(0x7fade80611c0 msgr2=0x7fade8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:57.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- 172.21.15.61:0/2723457891 shutdown_connections 2022-07-20T03:12:57.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.739+0000 7fadee668700 1 -- 172.21.15.61:0/2723457891 wait complete. 2022-07-20T03:12:57.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 Processor -- start 2022-07-20T03:12:57.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 -- start start 2022-07-20T03:12:57.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80aea20 0x7fade813dc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 0x7fade8153650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade813a5e0 0x7fade8153b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade813a340 con 0x7fade8141470 2022-07-20T03:12:57.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fade813e6f0 con 0x7fade813a5e0 2022-07-20T03:12:57.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fade813e990 con 0x7fade80aea20 2022-07-20T03:12:57.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 0x7fade8153650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 0x7fade8153650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46134/0 (socket says 172.21.15.61:46134) 2022-07-20T03:12:57.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 -- 172.21.15.61:0/1508499978 learned_addr learned my addr 172.21.15.61:0/1508499978 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:12:57.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fade7fff700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80aea20 0x7fade813dc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadecc05700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade813a5e0 0x7fade8153b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 -- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80aea20 msgr2=0x7fade813dc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80aea20 0x7fade813dc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 -- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade813a5e0 msgr2=0x7fade8153b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade813a5e0 0x7fade8153b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 -- 172.21.15.61:0/1508499978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fadd000b180 con 0x7fade8141470 2022-07-20T03:12:57.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7faddf7fe700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 0x7fade8153650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fade8063ee0 tx=0x7fadd0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd0016070 con 0x7fade8141470 2022-07-20T03:12:57.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fade8150a70 con 0x7fade8141470 2022-07-20T03:12:57.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fadd0013a30 con 0x7fade8141470 2022-07-20T03:12:57.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fade813ab50 con 0x7fade8141470 2022-07-20T03:12:57.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.743+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fadd0013d90 con 0x7fade8141470 2022-07-20T03:12:57.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.747+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fade8144e50 con 0x7fade8141470 2022-07-20T03:12:57.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.747+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fadd000f030 con 0x7fade8141470 2022-07-20T03:12:57.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.747+0000 7fade5ffb700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fadcc053940 0x7fadcc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:12:57.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.747+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fadd006cba0 con 0x7fade8141470 2022-07-20T03:12:57.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.747+0000 7fade7fff700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fadcc053940 0x7fadcc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:12:57.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.747+0000 7fade7fff700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fadcc053940 0x7fadcc055e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fadd800b000 tx=0x7fadd8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:12:57.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.751+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fadd0043b10 con 0x7fade8141470 2022-07-20T03:12:57.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.915+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fade80574e0 con 0x7fade8141470 2022-07-20T03:12:57.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fade5ffb700 1 -- 172.21.15.61:0/1508499978 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fadd0023030 con 0x7fade8141470 2022-07-20T03:12:57.936 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:12:57.936 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:12:57.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fadcc053940 msgr2=0x7fadcc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fadcc053940 0x7fadcc055e00 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7fadd800b000 tx=0x7fadd8006df0).stop 2022-07-20T03:12:57.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 msgr2=0x7fade8153650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:12:57.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 0x7fade8153650 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7fade8063ee0 tx=0x7fadd0014000).stop 2022-07-20T03:12:57.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 shutdown_connections 2022-07-20T03:12:57.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fadcc053940 0x7fadcc055e00 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fade813a5e0 0x7fade8153b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fade8141470 0x7fade8153650 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 --2- 172.21.15.61:0/1508499978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fade80aea20 0x7fade813dc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:12:57.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 >> 172.21.15.61:0/1508499978 conn(0x7fade80675f0 msgr2=0x7fade80afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:12:57.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 shutdown_connections 2022-07-20T03:12:57.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:12:57.919+0000 7fadee668700 1 -- 172.21.15.61:0/1508499978 wait complete. 2022-07-20T03:12:57.961 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:12:58.169 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 35, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:12:52.013020+0000', 'modified': '2022-07-20T03:12:53.015972+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 5982}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_5982': {'gid': 5982, 'name': 'a', 'rank': 0, 'incarnation': 34, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2218834214', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2218834214}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2218834214}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [12], 'metadata_pool': 11, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:12:58.178 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:12:58.178 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:12:58.317 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":5982,"want_state":"up:active","state":"up:active","rank_uptime":6.2919946050000002,"mdsmap_epoch":35,"osdmap_epoch":61,"osdmap_epoch_barrier":60,"uptime":10.67418827} 2022-07-20T03:12:58.318 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 5982, "mdsmap_epoch": 35, "osdmap_epoch": 61, "osdmap_epoch_barrier": 60, "rank_uptime": 6.291994605, "state": "up:active", "uptime": 10.67418827, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:12:58.319 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:12:58.319 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:12:58.320 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:12:58.321 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.321 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:58.336 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:12:58.400 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:12:58.401 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:12:58.436 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:12:58.437 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:12:58.439 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.440 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:12:58.499 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:12:58.504 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:58.508 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:12:58.538 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:12:58.539 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:12:58.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:12:58.584+0000 7fdffe04e080 -1 init, newargv = 0x55adcbf91260 newargc=9 2022-07-20T03:12:58.598 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22819]: starting ceph client 2022-07-20T03:12:58.600 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:12:58.615 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:12:58.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.624 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:12:58.626 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22819]: starting fuse 2022-07-20T03:12:58.640 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:12:58.641 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:12:58.654 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.655 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:12:58.658 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:12:58.659 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:12:58.659 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:12:58.660 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:12:58.661 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:58.662 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:12:58.662 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:12:58.669 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:12:58.670 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:58.671 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:12:58.671 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:12:58.678 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:12:58.678 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:12:58.683 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:58.684 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:12:58.684 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:12:58.685 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:12:58.685 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:12:58.686 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:12:58.687 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:58.687 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:12:58.688 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:12:58.774 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22819.asok 2022-07-20T03:12:58.775 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22819.asok 2022-07-20T03:12:58.776 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.779 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22819.asok status 2022-07-20T03:12:58.876 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:12:58.877 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:12:58.878 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:12:58.878 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:12:58.878 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:12:58.879 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:12:58.879 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:12:58.880 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22819", 2022-07-20T03:12:58.880 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:12:58.881 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:12:58.881 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:12:58.882 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:12:58.882 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6135, 2022-07-20T03:12:58.883 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:12:58.884 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:12:58.884 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:12:58.885 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6135 2022-07-20T03:12:58.898 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:12:58.899 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:12:58.900 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:12:58.900 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:12:58.901 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3026002431 2022-07-20T03:12:58.901 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:12:58.902 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:12:58.910 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:12:58.911 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:12:58.911 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:12:58.912 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 3026002431 2022-07-20T03:12:58.912 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:12:58.913 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6135 172.21.15.124:0/3026002431", 2022-07-20T03:12:58.913 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/3026002431", 2022-07-20T03:12:58.914 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:12:58.918 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 35, 2022-07-20T03:12:58.918 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 61, 2022-07-20T03:12:58.919 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:12:58.919 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:12:58.920 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:12:58.920 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:12:58.926 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.927 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:58.928 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:58.946 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:12:58.947 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:12:58.948 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.949 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:58.978 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:12:58.979 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:12:58.982 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:12:59.030 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:12:59.032 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:12:59.033 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:12:59.034 DEBUG:teuthology.orchestra.run.smithi124:> import sys 2022-07-20T03:12:59.036 DEBUG:teuthology.orchestra.run.smithi124:> import time 2022-07-20T03:12:59.036 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:12:59.037 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:59.038 DEBUG:teuthology.orchestra.run.smithi124:> n = 1000 2022-07-20T03:12:59.040 DEBUG:teuthology.orchestra.run.smithi124:> abs_path = "/home/ubuntu/cephtest/mnt.0/dc-dir/dc-file" 2022-07-20T03:12:59.041 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:59.042 DEBUG:teuthology.orchestra.run.smithi124:> if not os.path.exists(os.path.dirname(abs_path)): 2022-07-20T03:12:59.043 DEBUG:teuthology.orchestra.run.smithi124:> os.makedirs(os.path.dirname(abs_path)) 2022-07-20T03:12:59.044 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:12:59.055 DEBUG:teuthology.orchestra.run.smithi124:> for i in range(0, n): 2022-07-20T03:12:59.056 DEBUG:teuthology.orchestra.run.smithi124:> fname = "{0}_{1}".format(abs_path, i) 2022-07-20T03:12:59.056 DEBUG:teuthology.orchestra.run.smithi124:> with open(fname, '"'"'w'"'"') as f: 2022-07-20T03:12:59.057 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content'"'"') 2022-07-20T03:12:59.061 DEBUG:teuthology.orchestra.run.smithi124:> if True: 2022-07-20T03:12:59.061 DEBUG:teuthology.orchestra.run.smithi124:> f.flush() 2022-07-20T03:12:59.062 DEBUG:teuthology.orchestra.run.smithi124:> os.fsync(f.fileno()) 2022-07-20T03:12:59.063 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:13:06.050 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:06.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:06.057 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:06.059 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:06.070 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:08.750 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:08.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 Processor -- start 2022-07-20T03:13:08.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- start start 2022-07-20T03:13:08.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e580b5650 0x7f5e580b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:08.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580af8e0 0x7f5e5805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:08.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e58147010 con 0x7f5e580adff0 2022-07-20T03:13:08.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e581472b0 con 0x7f5e580b5650 2022-07-20T03:13:08.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58146d40 con 0x7f5e580af8e0 2022-07-20T03:13:08.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e580b5650 0x7f5e580b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:08.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e580b5650 0x7f5e580b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46140/0 (socket says 172.21.15.61:46140) 2022-07-20T03:13:08.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 -- 172.21.15.61:0/1056707348 learned_addr learned my addr 172.21.15.61:0/1056707348 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:08.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e577fe700 1 --1- 172.21.15.61:0/1056707348 >> v1:172.21.15.61:6789/0 conn(0x7f5e580adff0 0x7f5e58147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42740/0 (socket says 172.21.15.61:42740) 2022-07-20T03:13:08.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5ccfc700 1 --2- 172.21.15.61:0/1056707348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580af8e0 0x7f5e5805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:08.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 -- 172.21.15.61:0/1056707348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580af8e0 msgr2=0x7f5e5805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:08.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 --2- 172.21.15.61:0/1056707348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580af8e0 0x7f5e5805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:08.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 -- 172.21.15.61:0/1056707348 >> v1:172.21.15.61:6789/0 conn(0x7f5e580adff0 legacy=0x7f5e58147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:08.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 -- 172.21.15.61:0/1056707348 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e580acae0 con 0x7f5e580b5650 2022-07-20T03:13:08.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e57fff700 1 --2- 172.21.15.61:0/1056707348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580b5650 0x7f5e580b5a30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f5e40002f70 tx=0x7f5e40006eb0).ready entity=mon.0 client_cookie=72bdb8004003c02 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:08.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e56ffd700 1 -- 172.21.15.61:0/1056707348 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e4000d5c0 con 0x7f5e580b5650 2022-07-20T03:13:08.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e56ffd700 1 -- 172.21.15.61:0/1056707348 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e40000e90 con 0x7f5e580b5650 2022-07-20T03:13:08.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e56ffd700 1 -- 172.21.15.61:0/1056707348 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e4000d950 con 0x7f5e580b5650 2022-07-20T03:13:08.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1056707348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580b5650 msgr2=0x7f5e580b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:08.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1056707348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580b5650 0x7f5e580b5a30 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7f5e40002f70 tx=0x7f5e40006eb0).stop 2022-07-20T03:13:08.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1056707348 shutdown_connections 2022-07-20T03:13:08.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1056707348 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580af8e0 0x7f5e5805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:08.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1056707348 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580b5650 0x7f5e580b5a30 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:08.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.863+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1056707348 >> 172.21.15.61:0/1056707348 conn(0x7f5e580634e0 msgr2=0x7f5e580b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:08.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1056707348 shutdown_connections 2022-07-20T03:13:08.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1056707348 wait complete. 2022-07-20T03:13:08.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 Processor -- start 2022-07-20T03:13:08.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 -- start start 2022-07-20T03:13:08.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580adff0 0x7f5e580652b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:08.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 0x7f5e58111ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:08.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e580b5650 0x7f5e581565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:08.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e580677c0 con 0x7f5e580af8e0 2022-07-20T03:13:08.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58064b70 con 0x7f5e580b5650 2022-07-20T03:13:08.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5d4fd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e58064e10 con 0x7f5e580adff0 2022-07-20T03:13:08.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e577fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 0x7f5e58111ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:08.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e57fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580adff0 0x7f5e580652b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:08.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e5ccfc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e580b5650 0x7f5e581565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:08.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e57fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580adff0 0x7f5e580652b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:32818/0 (socket says 172.21.15.61:32818) 2022-07-20T03:13:08.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e57fff700 1 -- 172.21.15.61:0/1374794254 learned_addr learned my addr 172.21.15.61:0/1374794254 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:08.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.867+0000 7f5e577fe700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 0x7f5e58111ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46148/0 (socket says 172.21.15.61:46148) 2022-07-20T03:13:08.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e577fe700 1 -- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580adff0 msgr2=0x7f5e580652b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:08.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e577fe700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580adff0 0x7f5e580652b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:08.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e577fe700 1 -- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e580b5650 msgr2=0x7f5e581565a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:08.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e577fe700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e580b5650 0x7f5e581565a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:08.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e577fe700 1 -- 172.21.15.61:0/1374794254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e4000b1e0 con 0x7f5e580af8e0 2022-07-20T03:13:08.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e5ccfc700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e580b5650 0x7f5e581565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:08.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e577fe700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 0x7f5e58111ab0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f5e58060b70 tx=0x7f5e48007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:08.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e480072e0 con 0x7f5e580af8e0 2022-07-20T03:13:08.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e4800cba0 con 0x7f5e580af8e0 2022-07-20T03:13:08.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e58110c20 con 0x7f5e580af8e0 2022-07-20T03:13:08.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e480054e0 con 0x7f5e580af8e0 2022-07-20T03:13:08.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e58064350 con 0x7f5e580af8e0 2022-07-20T03:13:08.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e58067620 con 0x7f5e580af8e0 2022-07-20T03:13:08.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.871+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5e48010070 con 0x7f5e580af8e0 2022-07-20T03:13:08.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.875+0000 7f5e557fa700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e30053ad0 0x7f5e30055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:08.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.875+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5e4806b450 con 0x7f5e580af8e0 2022-07-20T03:13:08.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.875+0000 7f5e57fff700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e30053ad0 0x7f5e30055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:08.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.875+0000 7f5e57fff700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e30053ad0 0x7f5e30055f90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f5e4001c000 tx=0x7f5e40015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:08.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:08.883+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e58067620 con 0x7f5e580af8e0 2022-07-20T03:13:09.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.043+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5e58067620 con 0x7f5e580af8e0 2022-07-20T03:13:09.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.043+0000 7f5e557fa700 1 -- 172.21.15.61:0/1374794254 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f5e58067620 con 0x7f5e580af8e0 2022-07-20T03:13:09.050 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:09.050 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:13:09.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e30053ad0 msgr2=0x7f5e30055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e30053ad0 0x7f5e30055f90 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f5e4001c000 tx=0x7f5e40015000).stop 2022-07-20T03:13:09.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 msgr2=0x7f5e58111ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 0x7f5e58111ab0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7f5e58060b70 tx=0x7f5e48007ab0).stop 2022-07-20T03:13:09.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 shutdown_connections 2022-07-20T03:13:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5e30053ad0 0x7f5e30055f90 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e580b5650 0x7f5e581565a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e580af8e0 0x7f5e58111ab0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 --2- 172.21.15.61:0/1374794254 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e580adff0 0x7f5e580652b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 >> 172.21.15.61:0/1374794254 conn(0x7f5e580634e0 msgr2=0x7f5e580b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 shutdown_connections 2022-07-20T03:13:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.047+0000 7f5e5d4fd700 1 -- 172.21.15.61:0/1374794254 wait complete. 2022-07-20T03:13:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:13:09.059 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_caps 20 2022-07-20T03:13:09.160 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_caps = '20' (not observed, change may require restart) "} 2022-07-20T03:13:09.161 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_caps = '20' (not observed, change may require restart) " } 2022-07-20T03:13:09.162 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:09.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.263+0000 7fbaccaea700 1 Processor -- start 2022-07-20T03:13:09.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.263+0000 7fbaccaea700 1 -- start start 2022-07-20T03:13:09.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbac805c3f0 0x7fbac814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbac80acef0 0x7fbac8146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbac805d480 con 0x7fbac80b1e00 2022-07-20T03:13:09.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac80b21e0 con 0x7fbac805c3f0 2022-07-20T03:13:09.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbac805cf40 con 0x7fbac80acef0 2022-07-20T03:13:09.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbac805c3f0 0x7fbac814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbac805c3f0 0x7fbac814a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46154/0 (socket says 172.21.15.61:46154) 2022-07-20T03:13:09.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 -- 172.21.15.61:0/302624950 learned_addr learned my addr 172.21.15.61:0/302624950 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:09.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac6d9d700 1 --2- 172.21.15.61:0/302624950 >> v2:172.21.15.117:3300/0 conn(0x7fbac80acef0 0x7fbac8146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 -- 172.21.15.61:0/302624950 >> v2:172.21.15.117:3300/0 conn(0x7fbac80acef0 msgr2=0x7fbac8146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:09.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 --2- 172.21.15.61:0/302624950 >> v2:172.21.15.117:3300/0 conn(0x7fbac80acef0 0x7fbac8146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 -- 172.21.15.61:0/302624950 >> v1:172.21.15.117:6789/0 conn(0x7fbac80b1e00 legacy=0x7fbac80ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:09.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 -- 172.21.15.61:0/302624950 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbac805d1e0 con 0x7fbac805c3f0 2022-07-20T03:13:09.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac5d9b700 1 --2- 172.21.15.61:0/302624950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac805c3f0 0x7fbac814a6d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fbab8002f70 tx=0x7fbab8006eb0).ready entity=mon.0 client_cookie=a9233c516c0fdebe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:09.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac559a700 1 -- 172.21.15.61:0/302624950 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab800d610 con 0x7fbac805c3f0 2022-07-20T03:13:09.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac559a700 1 -- 172.21.15.61:0/302624950 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab8000e90 con 0x7fbac805c3f0 2022-07-20T03:13:09.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbac559a700 1 -- 172.21.15.61:0/302624950 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab800d9a0 con 0x7fbac805c3f0 2022-07-20T03:13:09.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- 172.21.15.61:0/302624950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac805c3f0 msgr2=0x7fbac814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 --2- 172.21.15.61:0/302624950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac805c3f0 0x7fbac814a6d0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7fbab8002f70 tx=0x7fbab8006eb0).stop 2022-07-20T03:13:09.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- 172.21.15.61:0/302624950 shutdown_connections 2022-07-20T03:13:09.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 --2- 172.21.15.61:0/302624950 >> v2:172.21.15.117:3300/0 conn(0x7fbac80acef0 0x7fbac8146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 --2- 172.21.15.61:0/302624950 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac805c3f0 0x7fbac814a6d0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- 172.21.15.61:0/302624950 >> 172.21.15.61:0/302624950 conn(0x7fbac80611c0 msgr2=0x7fbac8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:09.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- 172.21.15.61:0/302624950 shutdown_connections 2022-07-20T03:13:09.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- 172.21.15.61:0/302624950 wait complete. 2022-07-20T03:13:09.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 Processor -- start 2022-07-20T03:13:09.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.267+0000 7fbaccaea700 1 -- start start 2022-07-20T03:13:09.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbac80acef0 0x7fbac8159ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 0x7fbac80b5310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbac80a1850 0x7fbac8065260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac8144110 con 0x7fbac80b1e00 2022-07-20T03:13:09.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80a13f0 con 0x7fbac80acef0 2022-07-20T03:13:09.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbac80a0e50 con 0x7fbac80a1850 2022-07-20T03:13:09.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 0x7fbac80b5310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 0x7fbac80b5310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46162/0 (socket says 172.21.15.61:46162) 2022-07-20T03:13:09.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 -- 172.21.15.61:0/1974954076 learned_addr learned my addr 172.21.15.61:0/1974954076 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:09.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac6d9d700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbac80a1850 0x7fbac8065260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac659c700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbac80acef0 0x7fbac8159ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 -- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbac80a1850 msgr2=0x7fbac8065260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbac80a1850 0x7fbac8065260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 -- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbac80acef0 msgr2=0x7fbac8159ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbac80acef0 0x7fbac8159ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 -- 172.21.15.61:0/1974954076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbab800b1e0 con 0x7fbac80b1e00 2022-07-20T03:13:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbac5d9b700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 0x7fbac80b5310 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fbab8002f70 tx=0x7fbab8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab80067e0 con 0x7fbac80b1e00 2022-07-20T03:13:09.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbab800d290 con 0x7fbac80b1e00 2022-07-20T03:13:09.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbac80a10f0 con 0x7fbac80b1e00 2022-07-20T03:13:09.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbab8011d70 con 0x7fbac80b1e00 2022-07-20T03:13:09.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbac80a09f0 con 0x7fbac80b1e00 2022-07-20T03:13:09.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.271+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbac80a15b0 con 0x7fbac80b1e00 2022-07-20T03:13:09.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.275+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbab8025030 con 0x7fbac80b1e00 2022-07-20T03:13:09.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.275+0000 7fbab77fe700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbaa4053a40 0x7fbaa4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.275+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbab806cda0 con 0x7fbac80b1e00 2022-07-20T03:13:09.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.275+0000 7fbac659c700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbaa4053a40 0x7fbaa4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.275+0000 7fbac659c700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbaa4053a40 0x7fbaa4055f00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fbabc00d000 tx=0x7fbabc007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:09.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.279+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbac80a15b0 con 0x7fbac80b1e00 2022-07-20T03:13:09.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fbac80a15b0 con 0x7fbac80b1e00 2022-07-20T03:13:09.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbab77fe700 1 -- 172.21.15.61:0/1974954076 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fbac80a15b0 con 0x7fbac80b1e00 2022-07-20T03:13:09.444 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:09.444 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:13:09.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbaa4053a40 msgr2=0x7fbaa4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbaa4053a40 0x7fbaa4055f00 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7fbabc00d000 tx=0x7fbabc007b70).stop 2022-07-20T03:13:09.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 msgr2=0x7fbac80b5310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 0x7fbac80b5310 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7fbab8002f70 tx=0x7fbab8015000).stop 2022-07-20T03:13:09.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 shutdown_connections 2022-07-20T03:13:09.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbaa4053a40 0x7fbaa4055f00 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbac80a1850 0x7fbac8065260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbac80b1e00 0x7fbac80b5310 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 --2- 172.21.15.61:0/1974954076 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbac80acef0 0x7fbac8159ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 >> 172.21.15.61:0/1974954076 conn(0x7fbac805c3f0 msgr2=0x7fbac80afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:09.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 shutdown_connections 2022-07-20T03:13:09.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.443+0000 7fbaccaea700 1 -- 172.21.15.61:0/1974954076 wait complete. 2022-07-20T03:13:09.449 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:13:09.458 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_recall_max_decay_threshold 400 2022-07-20T03:13:09.575 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_recall_max_decay_threshold = '400' (not observed, change may require restart) "} 2022-07-20T03:13:09.576 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_recall_max_decay_threshold = '400' (not observed, change may require restart) " } 2022-07-20T03:13:09.577 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:09.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 Processor -- start 2022-07-20T03:13:09.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- start start 2022-07-20T03:13:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86ac0b25d0 0x7f86ac143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac1406b0 con 0x7f86ac0b21f0 2022-07-20T03:13:09.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86ac0ab430 con 0x7f86ac0ac860 2022-07-20T03:13:09.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86ac0ab6d0 con 0x7f86ac0b25d0 2022-07-20T03:13:09.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b098d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f86ac0b21f0 0x7f86ac142f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42766/0 (socket says 172.21.15.61:42766) 2022-07-20T03:13:09.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b098d700 1 -- 172.21.15.61:0/221929675 learned_addr learned my addr 172.21.15.61:0/221929675 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:09.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b118e700 1 --2- 172.21.15.61:0/221929675 >> v2:172.21.15.61:3300/0 conn(0x7f86ac0b25d0 0x7f86ac143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86ab7fe700 1 -- 172.21.15.61:0/221929675 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1615263828 0 0) 0x7f86ac1406b0 con 0x7f86ac0b21f0 2022-07-20T03:13:09.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86ab7fe700 1 -- 172.21.15.61:0/221929675 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8698003260 con 0x7f86ac0b21f0 2022-07-20T03:13:09.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b118e700 1 -- 172.21.15.61:0/221929675 >> v1:172.21.15.117:6789/0 conn(0x7f86ac0ac860 legacy=0x7f86ac13ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b118e700 1 -- 172.21.15.61:0/221929675 >> v1:172.21.15.61:6789/0 conn(0x7f86ac0b21f0 legacy=0x7f86ac142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b118e700 1 -- 172.21.15.61:0/221929675 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86ac0ab970 con 0x7f86ac0b25d0 2022-07-20T03:13:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b118e700 1 --2- 172.21.15.61:0/221929675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0b25d0 0x7f86ac143690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f86a4002f70 tx=0x7f86a4006df0).ready entity=mon.0 client_cookie=b97573bbb62f73c5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:09.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86ab7fe700 1 -- 172.21.15.61:0/221929675 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a400d610 con 0x7f86ac0b25d0 2022-07-20T03:13:09.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86ab7fe700 1 -- 172.21.15.61:0/221929675 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a4000e90 con 0x7f86ac0b25d0 2022-07-20T03:13:09.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86ab7fe700 1 -- 172.21.15.61:0/221929675 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a400d960 con 0x7f86ac0b25d0 2022-07-20T03:13:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- 172.21.15.61:0/221929675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0b25d0 msgr2=0x7f86ac143690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/221929675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0b25d0 0x7f86ac143690 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7f86a4002f70 tx=0x7f86a4006df0).stop 2022-07-20T03:13:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- 172.21.15.61:0/221929675 shutdown_connections 2022-07-20T03:13:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/221929675 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0b25d0 0x7f86ac143690 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- 172.21.15.61:0/221929675 >> 172.21.15.61:0/221929675 conn(0x7f86ac0b3690 msgr2=0x7f86ac0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- 172.21.15.61:0/221929675 shutdown_connections 2022-07-20T03:13:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.687+0000 7f86b2bf1700 1 -- 172.21.15.61:0/221929675 wait complete. 2022-07-20T03:13:09.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 Processor -- start 2022-07-20T03:13:09.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 -- start start 2022-07-20T03:13:09.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 0x7f86ac143a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86ac0b21f0 0x7f86ac155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86ac0b25d0 0x7f86ac155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac136240 con 0x7f86ac0ac860 2022-07-20T03:13:09.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0b5680 con 0x7f86ac0b21f0 2022-07-20T03:13:09.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86ac0b5920 con 0x7f86ac0b25d0 2022-07-20T03:13:09.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 0x7f86ac143a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 0x7f86ac143a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46174/0 (socket says 172.21.15.61:46174) 2022-07-20T03:13:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 -- 172.21.15.61:0/1776846643 learned_addr learned my addr 172.21.15.61:0/1776846643 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86abfff700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86ac0b21f0 0x7f86ac155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b118e700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86ac0b25d0 0x7f86ac155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 -- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86ac0b25d0 msgr2=0x7f86ac155f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86ac0b25d0 0x7f86ac155f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 -- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86ac0b21f0 msgr2=0x7f86ac155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86ac0b21f0 0x7f86ac155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 -- 172.21.15.61:0/1776846643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86a400b180 con 0x7f86ac0ac860 2022-07-20T03:13:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b118e700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86ac0b25d0 0x7f86ac155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b098d700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 0x7f86ac143a10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f86a0001110 tx=0x7f86a0007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a000ce20 con 0x7f86ac0ac860 2022-07-20T03:13:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86a00072e0 con 0x7f86ac0ac860 2022-07-20T03:13:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86ac05fa00 con 0x7f86ac0ac860 2022-07-20T03:13:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86a0005800 con 0x7f86ac0ac860 2022-07-20T03:13:09.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.691+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86ac141460 con 0x7f86ac0ac860 2022-07-20T03:13:09.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.695+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f86a0010070 con 0x7f86ac0ac860 2022-07-20T03:13:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.695+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86ac0ab0b0 con 0x7f86ac0ac860 2022-07-20T03:13:09.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.695+0000 7f86a9ffb700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f868c053920 0x7f868c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.695+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f86a006b6a0 con 0x7f86ac0ac860 2022-07-20T03:13:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.695+0000 7f86abfff700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f868c053920 0x7f868c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:09.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.695+0000 7f86abfff700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f868c053920 0x7f868c055de0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f869c00d000 tx=0x7f869c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:09.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.699+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86a00424f0 con 0x7f86ac0ac860 2022-07-20T03:13:09.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86ac0574e0 con 0x7f86ac0ac860 2022-07-20T03:13:09.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86a9ffb700 1 -- 172.21.15.61:0/1776846643 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f86a0018030 con 0x7f86ac0ac860 2022-07-20T03:13:09.884 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:09.884 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:13:09.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f868c053920 msgr2=0x7f868c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f868c053920 0x7f868c055de0 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f869c00d000 tx=0x7f869c009040).stop 2022-07-20T03:13:09.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 msgr2=0x7f86ac143a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:09.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 0x7f86ac143a10 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f86a0001110 tx=0x7f86a0007ad0).stop 2022-07-20T03:13:09.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 shutdown_connections 2022-07-20T03:13:09.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f868c053920 0x7f868c055de0 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86ac0b21f0 0x7f86ac155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86ac0b25d0 0x7f86ac155f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 --2- 172.21.15.61:0/1776846643 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86ac0ac860 0x7f86ac143a10 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:09.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 >> 172.21.15.61:0/1776846643 conn(0x7f86ac0b3690 msgr2=0x7f86ac140070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:09.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.859+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 shutdown_connections 2022-07-20T03:13:09.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:09.863+0000 7f86b2bf1700 1 -- 172.21.15.61:0/1776846643 wait complete. 2022-07-20T03:13:09.899 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:13:09.899 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell mds.a cache drop 10 2022-07-20T03:13:10.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.011+0000 7efdfd086700 1 Processor -- start 2022-07-20T03:13:10.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.011+0000 7efdfd086700 1 -- start start 2022-07-20T03:13:10.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf80ab180 0x7efdf8061ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7efdf8142720 0x7efdf8142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf805f840 con 0x7efdf80b34d0 2022-07-20T03:13:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf8142b90 con 0x7efdf8142720 2022-07-20T03:13:10.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf805fa70 con 0x7efdf80ab180 2022-07-20T03:13:10.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf6d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efdf80b34d0 0x7efdf8142010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42780/0 (socket says 172.21.15.61:42780) 2022-07-20T03:13:10.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf6d9d700 1 -- 172.21.15.61:0/3519299843 learned_addr learned my addr 172.21.15.61:0/3519299843 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:10.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf759e700 1 --2- 172.21.15.61:0/3519299843 >> v2:172.21.15.117:3300/0 conn(0x7efdf8142720 0x7efdf8142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf659c700 1 --2- 172.21.15.61:0/3519299843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf80ab180 0x7efdf8061ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf5d9b700 1 -- 172.21.15.61:0/3519299843 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1780980970 0 0) 0x7efdf805f840 con 0x7efdf80b34d0 2022-07-20T03:13:10.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf5d9b700 1 -- 172.21.15.61:0/3519299843 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efddc003260 con 0x7efdf80b34d0 2022-07-20T03:13:10.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf659c700 1 -- 172.21.15.61:0/3519299843 >> v2:172.21.15.117:3300/0 conn(0x7efdf8142720 msgr2=0x7efdf8142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf659c700 1 --2- 172.21.15.61:0/3519299843 >> v2:172.21.15.117:3300/0 conn(0x7efdf8142720 0x7efdf8142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf659c700 1 -- 172.21.15.61:0/3519299843 >> v1:172.21.15.61:6789/0 conn(0x7efdf80b34d0 legacy=0x7efdf8142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf659c700 1 -- 172.21.15.61:0/3519299843 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf805fd10 con 0x7efdf80ab180 2022-07-20T03:13:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf759e700 1 --2- 172.21.15.61:0/3519299843 >> v2:172.21.15.117:3300/0 conn(0x7efdf8142720 0x7efdf8142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:10.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf659c700 1 --2- 172.21.15.61:0/3519299843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf80ab180 0x7efdf8061ba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efdec002f70 tx=0x7efdec006df0).ready entity=mon.2 client_cookie=bb583adf4da43077 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf5d9b700 1 -- 172.21.15.61:0/3519299843 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec00d4a0 con 0x7efdf80ab180 2022-07-20T03:13:10.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf5d9b700 1 -- 172.21.15.61:0/3519299843 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec000e90 con 0x7efdf80ab180 2022-07-20T03:13:10.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdf5d9b700 1 -- 172.21.15.61:0/3519299843 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec00d7f0 con 0x7efdf80ab180 2022-07-20T03:13:10.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- 172.21.15.61:0/3519299843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf80ab180 msgr2=0x7efdf8061ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 --2- 172.21.15.61:0/3519299843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf80ab180 0x7efdf8061ba0 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7efdec002f70 tx=0x7efdec006df0).stop 2022-07-20T03:13:10.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- 172.21.15.61:0/3519299843 shutdown_connections 2022-07-20T03:13:10.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 --2- 172.21.15.61:0/3519299843 >> v2:172.21.15.117:3300/0 conn(0x7efdf8142720 0x7efdf8142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 --2- 172.21.15.61:0/3519299843 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf80ab180 0x7efdf8061ba0 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- 172.21.15.61:0/3519299843 >> 172.21.15.61:0/3519299843 conn(0x7efdf80b6c50 msgr2=0x7efdf80b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- 172.21.15.61:0/3519299843 shutdown_connections 2022-07-20T03:13:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.015+0000 7efdfd086700 1 -- 172.21.15.61:0/3519299843 wait complete. 2022-07-20T03:13:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 Processor -- start 2022-07-20T03:13:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 -- start start 2022-07-20T03:13:10.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf80ab180 0x7efdf8099970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf80b34d0 0x7efdf809b8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf8142720 0x7efdf809bda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf8152ac0 con 0x7efdf80ab180 2022-07-20T03:13:10.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf8152d60 con 0x7efdf80b34d0 2022-07-20T03:13:10.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdfd086700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf8152710 con 0x7efdf8142720 2022-07-20T03:13:10.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf80ab180 0x7efdf8099970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf659c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf80b34d0 0x7efdf809b8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf80ab180 0x7efdf8099970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46188/0 (socket says 172.21.15.61:46188) 2022-07-20T03:13:10.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf6d9d700 1 -- 172.21.15.61:0/1607442121 learned_addr learned my addr 172.21.15.61:0/1607442121 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:10.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf759e700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf8142720 0x7efdf809bda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf659c700 1 -- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf8142720 msgr2=0x7efdf809bda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf659c700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf8142720 0x7efdf809bda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf659c700 1 -- 172.21.15.61:0/1607442121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf80ab180 msgr2=0x7efdf8099970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf659c700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf80ab180 0x7efdf8099970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.019+0000 7efdf659c700 1 -- 172.21.15.61:0/1607442121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf0009470 con 0x7efdf80b34d0 2022-07-20T03:13:10.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efdf659c700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf80b34d0 0x7efdf809b8b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7efdec002f70 tx=0x7efdec014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efde7fff700 1 -- 172.21.15.61:0/1607442121 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec016070 con 0x7efdf80b34d0 2022-07-20T03:13:10.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efde7fff700 1 -- 172.21.15.61:0/1607442121 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec0138d0 con 0x7efdf80b34d0 2022-07-20T03:13:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efde7fff700 1 -- 172.21.15.61:0/1607442121 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec013c30 con 0x7efdf80b34d0 2022-07-20T03:13:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efdec00b180 con 0x7efdf80b34d0 2022-07-20T03:13:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf809a330 con 0x7efdf80b34d0 2022-07-20T03:13:10.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.023+0000 7efde7fff700 1 -- 172.21.15.61:0/1607442121 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efdec00f030 con 0x7efdf80b34d0 2022-07-20T03:13:10.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efde7fff700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053a40 0x7efdd4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efde7fff700 1 -- 172.21.15.61:0/1607442121 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdec06ca80 con 0x7efdf80b34d0 2022-07-20T03:13:10.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053a40 0x7efdd4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053a40 0x7efdd4055f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7efde800d000 tx=0x7efde8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 Processor -- start 2022-07-20T03:13:10.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 -- start start 2022-07-20T03:13:10.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd8c970 0xd8cd50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd51c50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd52140 0xd503f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xcc3210 con 0xd88790 2022-07-20T03:13:10.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xca3450 con 0xd52140 2022-07-20T03:13:10.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xd88bd0 con 0xd8c970 2022-07-20T03:13:10.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd51c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd51c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46198/0 (socket says 172.21.15.61:46198) 2022-07-20T03:13:10.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 -- 172.21.15.61:0/2047164350 learned_addr learned my addr 172.21.15.61:0/2047164350 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:10.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd8c970 0xd8cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf759e700 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd52140 0xd503f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 -- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd8c970 msgr2=0xd8cd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd8c970 0xd8cd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 -- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd52140 msgr2=0xd503f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd52140 0xd503f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 -- 172.21.15.61:0/2047164350 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xcfaad0 con 0xd88790 2022-07-20T03:13:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.027+0000 7efdf659c700 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd51c50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7efdec06cc90 tx=0x7efdec010000).ready entity=mon.0 client_cookie=4d1ce446132b23ee server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efdfd086700 1 -- 172.21.15.61:0/2047164350 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec045a40 con 0xd88790 2022-07-20T03:13:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efdfd086700 1 -- 172.21.15.61:0/2047164350 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec046b90 con 0xd88790 2022-07-20T03:13:10.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efdfd086700 1 -- 172.21.15.61:0/2047164350 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec046600 con 0xd88790 2022-07-20T03:13:10.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 -- 172.21.15.61:0/2047164350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 msgr2=0xd51c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd51c50 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7efdec06cc90 tx=0x7efdec010000).stop 2022-07-20T03:13:10.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 -- 172.21.15.61:0/2047164350 shutdown_connections 2022-07-20T03:13:10.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd52140 0xd503f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd51c50 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 --2- 172.21.15.61:0/2047164350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd8c970 0xd8cd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 -- 172.21.15.61:0/2047164350 >> 172.21.15.61:0/2047164350 conn(0xcda8f0 msgr2=0xd8c260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:10.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 -- 172.21.15.61:0/2047164350 shutdown_connections 2022-07-20T03:13:10.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.031+0000 7efe072dd740 1 -- 172.21.15.61:0/2047164350 wait complete. 2022-07-20T03:13:10.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- start start 2022-07-20T03:13:10.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 Processor -- start 2022-07-20T03:13:10.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd52140 0xd59f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd96d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd4ce90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xd98040 con 0xd88790 2022-07-20T03:13:10.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xd58ec0 con 0xd8c970 2022-07-20T03:13:10.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xd5a430 con 0xd52140 2022-07-20T03:13:10.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 10 client.0 ll_register_callbacks cb 0xd0a0b0 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:13:10.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd96d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd96d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46202/0 (socket says 172.21.15.61:46202) 2022-07-20T03:13:10.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 -- 172.21.15.61:0/2053589706 learned_addr learned my addr 172.21.15.61:0/2053589706 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:10.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf759e700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd4ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd52140 0xd59f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 -- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd52140 msgr2=0xd59f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd52140 0xd59f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 -- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 msgr2=0xd4ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd4ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 -- 172.21.15.61:0/2053589706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf809a330 con 0xd88790 2022-07-20T03:13:10.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf659c700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd96d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0xd5a5c0 tx=0x7efdec028000).ready entity=mon.0 client_cookie=238a1b56f1ea13e7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdd1ffb700 10 client.0 ms_handle_connect on v2:172.21.15.61:3300/0 2022-07-20T03:13:10.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdd1ffb700 1 -- 172.21.15.61:0/2053589706 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec046d00 con 0xd88790 2022-07-20T03:13:10.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdd1ffb700 1 -- 172.21.15.61:0/2053589706 <== mon.0 v2:172.21.15.61:3300/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdec03bc90 con 0xd88790 2022-07-20T03:13:10.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0xcb91c0 con 0xd88790 2022-07-20T03:13:10.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdd1ffb700 1 -- 172.21.15.61:0/2053589706 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdec046e60 con 0xd88790 2022-07-20T03:13:10.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 10 client.6171 fetch_fsmap learned FSMap version 35 2022-07-20T03:13:10.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0xcb27b0 con 0xd88790 2022-07-20T03:13:10.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdd1ffb700 1 -- 172.21.15.61:0/2053589706 <== mon.0 v2:172.21.15.61:3300/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0xcb91c0 con 0xd88790 2022-07-20T03:13:10.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdd1ffb700 1 -- 172.21.15.61:0/2053589706 <== mon.0 v2:172.21.15.61:3300/0 5 ==== fsmap(e 35) v1 ==== 1247+0+0 (secure 0 0 0) 0x7efdec0478d0 con 0xd88790 2022-07-20T03:13:10.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 10 client.6171 fetch_fsmap finished waiting for FSMap version 35 2022-07-20T03:13:10.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 10 client.6171 resolve_mds: resolved ID 'a' to GID 5982 2022-07-20T03:13:10.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 20 client.6171 populate_metadata read hostname 'smithi061' 2022-07-20T03:13:10.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xd4d380 0xd5c240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 4 client.6171 mds_command: new command op to 5982 tid=0[{"prefix": "get_command_descriptions"}] 2022-07-20T03:13:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 --> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0xcb1240 con 0xd4d380 2022-07-20T03:13:10.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.035+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xd4d380 0xd5c240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efdd1ffb700 10 client.6171 ms_handle_connect on v2:172.21.15.61:6832/2218834214 2022-07-20T03:13:10.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xd4d380 0xd5c240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=9835d7140ca1cf74 server_cookie=fdd143b174253c4c in_seq=0 out_seq=0 2022-07-20T03:13:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efdd1ffb700 1 -- 172.21.15.61:0/2053589706 <== mds.0 v2:172.21.15.61:6832/2218834214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+17002 (crc 0 0 0) 0xcb1240 con 0xd4d380 2022-07-20T03:13:10.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efdd1ffb700 10 client.6171 handle_command_reply: tid=0 2022-07-20T03:13:10.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 client.6171 shutdown 2022-07-20T03:13:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 msgr2=0xd96d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd96d80 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0xd5a5c0 tx=0x7efdec028000).stop 2022-07-20T03:13:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 shutdown_connections 2022-07-20T03:13:10.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xd4d380 0xd5c240 crc :-1 s=READY pgs=12 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd4ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd88790 0xd96d80 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 --2- 172.21.15.61:0/2053589706 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd52140 0xd59f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efdd1ffb700 0 client.6171 ms_handle_reset on v2:172.21.15.61:6832/2218834214 2022-07-20T03:13:10.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 >> 172.21.15.61:0/2053589706 conn(0xcda8f0 msgr2=0xd8c260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:10.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 shutdown_connections 2022-07-20T03:13:10.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 1 -- 172.21.15.61:0/2053589706 wait complete. 2022-07-20T03:13:10.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.039+0000 7efe072dd740 20 client.6171 trim_cache size 0 max 16384 2022-07-20T03:13:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 Processor -- start 2022-07-20T03:13:10.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 -- start start 2022-07-20T03:13:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xd53c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd86b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd87000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xd19890 con 0xd52140 2022-07-20T03:13:10.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xd4df70 con 0xd8c970 2022-07-20T03:13:10.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xd4dde0 con 0xd88790 2022-07-20T03:13:10.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xd53c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xd53c00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46212/0 (socket says 172.21.15.61:46212) 2022-07-20T03:13:10.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 -- 172.21.15.61:0/2034360472 learned_addr learned my addr 172.21.15.61:0/2034360472 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf659c700 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd86b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf759e700 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd87000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 -- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 msgr2=0xd86b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd86b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 -- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 msgr2=0xd87000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd87000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 -- 172.21.15.61:0/2034360472 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0xcbb3a0 con 0xd52140 2022-07-20T03:13:10.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.051+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xd53c00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7efde8014000 tx=0x7efde800a000).ready entity=mon.0 client_cookie=11afb18a04a615e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdd3fff700 1 -- 172.21.15.61:0/2034360472 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde80240e0 con 0xd52140 2022-07-20T03:13:10.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdd3fff700 1 -- 172.21.15.61:0/2034360472 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde80138e0 con 0xd52140 2022-07-20T03:13:10.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdd3fff700 1 -- 172.21.15.61:0/2034360472 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde8013c40 con 0xd52140 2022-07-20T03:13:10.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- 172.21.15.61:0/2034360472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 msgr2=0xd53c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:10.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xd53c00 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7efde8014000 tx=0x7efde800a000).stop 2022-07-20T03:13:10.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- 172.21.15.61:0/2034360472 shutdown_connections 2022-07-20T03:13:10.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xd87000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd86b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- 172.21.15.61:0/2034360472 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xd53c00 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- 172.21.15.61:0/2034360472 >> 172.21.15.61:0/2034360472 conn(0xcda8f0 msgr2=0xd974a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:10.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- 172.21.15.61:0/2034360472 shutdown_connections 2022-07-20T03:13:10.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- 172.21.15.61:0/2034360472 wait complete. 2022-07-20T03:13:10.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- start start 2022-07-20T03:13:10.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 Processor -- start 2022-07-20T03:13:10.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xded3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd502e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xdedad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0xd53b80 con 0xd52140 2022-07-20T03:13:10.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0xded8c0 con 0xd8c970 2022-07-20T03:13:10.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0xdeb100 con 0xd88790 2022-07-20T03:13:10.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efe072dd740 10 client.0 ll_register_callbacks cb 0xd5b520 invalidate_ino_cb 0 invalidate_dentry_cb 0 switch_interrupt_cb 0 remount_cb 0 2022-07-20T03:13:10.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xded3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xded3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46218/0 (socket says 172.21.15.61:46218) 2022-07-20T03:13:10.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd502e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf6d9d700 1 -- 172.21.15.61:0/3520979631 learned_addr learned my addr 172.21.15.61:0/3520979631 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:10.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf759e700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xdedad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 -- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 msgr2=0xdedad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xdedad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 -- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 msgr2=0xded3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:10.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xded3d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:10.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 -- 172.21.15.61:0/3520979631 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efddc003260 con 0xd88790 2022-07-20T03:13:10.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xded3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:10.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdf659c700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd502e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0xd53800 tx=0x7efdec039000).ready entity=mon.2 client_cookie=d96da0cbf2fc4e7c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:10.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.055+0000 7efdd2ffd700 10 client.0 ms_handle_connect on v2:172.21.15.117:3301/0 2022-07-20T03:13:10.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdd2ffd700 1 -- 172.21.15.61:0/3520979631 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec0471e0 con 0xd88790 2022-07-20T03:13:10.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdd2ffd700 1 -- 172.21.15.61:0/3520979631 <== mon.2 v2:172.21.15.117:3301/0 2 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efdec047db0 con 0xd88790 2022-07-20T03:13:10.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_get_version(what=fsmap handle=1) v1 -- 0x7efdec046b90 con 0xd88790 2022-07-20T03:13:10.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdd2ffd700 1 -- 172.21.15.61:0/3520979631 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_get_version_reply(handle=1 version=35) v2 ==== 24+0+0 (secure 0 0 0) 0x7efdec046d00 con 0xd88790 2022-07-20T03:13:10.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 10 client.5579 fetch_fsmap learned FSMap version 35 2022-07-20T03:13:10.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,fsmap=35,monmap=2+}) v3 -- 0xcb7d90 con 0xd88790 2022-07-20T03:13:10.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdd2ffd700 1 -- 172.21.15.61:0/3520979631 <== mon.2 v2:172.21.15.117:3301/0 4 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec046b90 con 0xd88790 2022-07-20T03:13:10.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdd2ffd700 1 -- 172.21.15.61:0/3520979631 <== mon.2 v2:172.21.15.117:3301/0 5 ==== fsmap(e 35) v1 ==== 1247+0+0 (secure 0 0 0) 0x7efdec0478d0 con 0xd88790 2022-07-20T03:13:10.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 10 client.5579 fetch_fsmap finished waiting for FSMap version 35 2022-07-20T03:13:10.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 10 client.5579 resolve_mds: resolved ID 'a' to GID 5982 2022-07-20T03:13:10.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 20 client.5579 populate_metadata read hostname 'smithi061' 2022-07-20T03:13:10.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xda73f0 0xdedfc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:10.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 4 client.5579 mds_command: new command op to 5982 tid=0[{"prefix": "cache drop", "timeout": 10}] 2022-07-20T03:13:10.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 --> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] -- command(tid 0: {"prefix": "cache drop", "timeout": 10}) v1 -- 0xba65e0 con 0xda73f0 2022-07-20T03:13:10.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xda73f0 0xdedfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:10.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdf6d9d700 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xda73f0 0xdedfc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).ready entity=mds.0 client_cookie=75bb6b99703e67e1 server_cookie=77b84d7deaef61e in_seq=0 out_seq=0 2022-07-20T03:13:10.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:10.059+0000 7efdd2ffd700 10 client.5579 ms_handle_connect on v2:172.21.15.61:6832/2218834214 2022-07-20T03:13:20.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdd2ffd700 1 -- 172.21.15.61:0/3520979631 <== mds.0 v2:172.21.15.61:6832/2218834214 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+400 (crc 0 0 0) 0xba65e0 con 0xda73f0 2022-07-20T03:13:20.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdd2ffd700 10 client.5579 handle_command_reply: tid=0 2022-07-20T03:13:20.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 client.5579 shutdown 2022-07-20T03:13:20.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 msgr2=0xd502e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:20.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd502e0 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0xd53800 tx=0x7efdec039000).stop 2022-07-20T03:13:20.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 shutdown_connections 2022-07-20T03:13:20.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:6832/2218834214,v1:172.21.15.61:6833/2218834214] conn(0xda73f0 0xdedfc0 crc :-1 s=READY pgs=13 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0xd8c970 0xdedad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0xd88790 0xd502e0 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 --2- 172.21.15.61:0/3520979631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0xd52140 0xded3d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdd2ffd700 0 client.5579 ms_handle_reset on v2:172.21.15.61:6832/2218834214 2022-07-20T03:13:20.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 >> 172.21.15.61:0/3520979631 conn(0xcda8f0 msgr2=0xd97390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:20.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 shutdown_connections 2022-07-20T03:13:20.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 1 -- 172.21.15.61:0/3520979631 wait complete. 2022-07-20T03:13:20.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efe072dd740 20 client.5579 trim_cache size 0 max 16384 2022-07-20T03:13:20.303 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:13:20.304 INFO:teuthology.orchestra.run.smithi061.stdout: "client_recall": { 2022-07-20T03:13:20.304 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": -110, 2022-07-20T03:13:20.305 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "(110) Connection timed out", 2022-07-20T03:13:20.305 INFO:teuthology.orchestra.run.smithi061.stdout: "recalled": 240 2022-07-20T03:13:20.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:20.309 INFO:teuthology.orchestra.run.smithi061.stdout: "flush_journal": { 2022-07-20T03:13:20.309 INFO:teuthology.orchestra.run.smithi061.stdout: "return_code": 0, 2022-07-20T03:13:20.310 INFO:teuthology.orchestra.run.smithi061.stdout: "message": "" 2022-07-20T03:13:20.310 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:20.310 INFO:teuthology.orchestra.run.smithi061.stdout: "trim_cache": { 2022-07-20T03:13:20.311 INFO:teuthology.orchestra.run.smithi061.stdout: "trimmed": 220 2022-07-20T03:13:20.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:20.313 INFO:teuthology.orchestra.run.smithi061.stdout: "cache": { 2022-07-20T03:13:20.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": { 2022-07-20T03:13:20.313 INFO:teuthology.orchestra.run.smithi061.stdout: "items": 15448, 2022-07-20T03:13:20.314 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes": 2428464 2022-07-20T03:13:20.314 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:20.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:20.315 INFO:teuthology.orchestra.run.smithi061.stdout: "duration": 10.231867012 2022-07-20T03:13:20.316 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:13:20.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053a40 msgr2=0x7efdd4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:20.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053a40 0x7efdd4055f00 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7efde800d000 tx=0x7efde8009040).stop 2022-07-20T03:13:20.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf80b34d0 msgr2=0x7efdf809b8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:20.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf80b34d0 0x7efdf809b8b0 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7efdec002f70 tx=0x7efdec014000).stop 2022-07-20T03:13:20.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 shutdown_connections 2022-07-20T03:13:20.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053a40 0x7efdd4055f00 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf80b34d0 0x7efdf809b8b0 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf8142720 0x7efdf809bda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 --2- 172.21.15.61:0/1607442121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf80ab180 0x7efdf8099970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:20.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 >> 172.21.15.61:0/1607442121 conn(0x7efdf80b6c50 msgr2=0x7efdf8140180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:20.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 shutdown_connections 2022-07-20T03:13:20.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:20.294+0000 7efdfd086700 1 -- 172.21.15.61:0/1607442121 wait complete. 2022-07-20T03:13:20.344 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:13:20.350 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:20.379 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:20.403 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:20.432 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:20.453 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:20.474 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:13:20.475 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:20.476 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:20.476 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:20.485 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:13:20.487 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:13:20.487 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:13:20.488 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:13:20.489 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:20.505 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22819]: fuse finished with error 0 and tester_r 0 2022-07-20T03:13:20.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:20.508 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:20.508 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:20.571 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:13:20.571 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:13:20.608 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:13:20.610 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:20.674 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:20.676 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:20.676 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:20.709 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:20.733 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:13:20.734 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:13:20.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:13:20.736 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:20.786 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:13:20.788 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:20.789 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:20.790 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:20.791 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:13:20.809 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:20.811 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:13:20.811 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:13:20.812 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:13:20.813 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:13:20.863 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:13:20.865 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:20.870 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:13:21.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.014+0000 7fd7399f0700 1 Processor -- start 2022-07-20T03:13:21.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.014+0000 7fd7399f0700 1 -- start start 2022-07-20T03:13:21.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.014+0000 7fd7399f0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340acef0 0x7fd73414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.014+0000 7fd7399f0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd73405d480 con 0x7fd73405c3f0 2022-07-20T03:13:21.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.014+0000 7fd7399f0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7340b21e0 con 0x7fd7340b1e00 2022-07-20T03:13:21.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.014+0000 7fd7399f0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd73405cf40 con 0x7fd7340acef0 2022-07-20T03:13:21.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7327fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd73405c3f0 0x7fd734146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42824/0 (socket says 172.21.15.61:42824) 2022-07-20T03:13:21.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7327fc700 1 -- 172.21.15.61:0/2655515949 learned_addr learned my addr 172.21.15.61:0/2655515949 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:21.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340acef0 0x7fd73414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3248481376 0 0) 0x7fd73405d480 con 0x7fd73405c3f0 2022-07-20T03:13:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd714003260 con 0x7fd73405c3f0 2022-07-20T03:13:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3787260566 0 0) 0x7fd7340b21e0 con 0x7fd7340b1e00 2022-07-20T03:13:21.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd73405d480 con 0x7fd7340b1e00 2022-07-20T03:13:21.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1266197171 0 0) 0x7fd714003260 con 0x7fd73405c3f0 2022-07-20T03:13:21.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340acef0 msgr2=0x7fd73414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:21.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340acef0 0x7fd73414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 >> v1:172.21.15.117:6789/0 conn(0x7fd7340b1e00 legacy=0x7fd7340ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73405d1e0 con 0x7fd73405c3f0 2022-07-20T03:13:21.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fd71c001680 con 0x7fd73405c3f0 2022-07-20T03:13:21.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 >> v1:172.21.15.61:6789/0 conn(0x7fd73405c3f0 legacy=0x7fd734146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd714003910 0x7fd714003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd714004500 0x7fd714004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd714004e70 0x7fd71400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd73405cf40 con 0x7fd714003910 2022-07-20T03:13:21.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd71400bdd0 con 0x7fd714004e70 2022-07-20T03:13:21.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd71400bf90 con 0x7fd714004500 2022-07-20T03:13:21.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd732ffd700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd714003910 0x7fd714003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd714004e70 0x7fd71400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7327fc700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd714004500 0x7fd714004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 -- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd714004500 msgr2=0x7fd714004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:21.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd714004500 0x7fd714004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 -- 172.21.15.61:0/2655515949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd714003910 msgr2=0x7fd714003d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:21.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd714003910 0x7fd714003d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 -- 172.21.15.61:0/2655515949 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd714003260 con 0x7fd714004e70 2022-07-20T03:13:21.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd732ffd700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd714003910 0x7fd714003d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:21.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd714004e70 0x7fd71400b750 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fd7140054d0 tx=0x7fd72c0091d0).ready entity=mon.1 client_cookie=71bd4de6bf9776b0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:21.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd72c00e640 con 0x7fd714004e70 2022-07-20T03:13:21.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd72c00e7a0 con 0x7fd714004e70 2022-07-20T03:13:21.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd731ffb700 1 -- 172.21.15.61:0/2655515949 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd72c00eb00 con 0x7fd714004e70 2022-07-20T03:13:21.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7399f0700 1 -- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd714004e70 msgr2=0x7fd71400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2655515949 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd714004e70 0x7fd71400b750 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fd7140054d0 tx=0x7fd72c0091d0).stop 2022-07-20T03:13:21.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7337fe700 1 -- 172.21.15.61:0/2655515949 reap_dead start 2022-07-20T03:13:21.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7399f0700 1 -- 172.21.15.61:0/2655515949 shutdown_connections 2022-07-20T03:13:21.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7399f0700 1 -- 172.21.15.61:0/2655515949 >> 172.21.15.61:0/2655515949 conn(0x7fd7340611c0 msgr2=0x7fd734062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:21.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7399f0700 1 -- 172.21.15.61:0/2655515949 shutdown_connections 2022-07-20T03:13:21.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.018+0000 7fd7399f0700 1 -- 172.21.15.61:0/2655515949 wait complete. 2022-07-20T03:13:21.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 Processor -- start 2022-07-20T03:13:21.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 -- start start 2022-07-20T03:13:21.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 0x7fd734143170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd73414a8d0 0x7fd734143660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd73409fbd0 0x7fd7340a0080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7340ada20 con 0x7fd73414a8d0 2022-07-20T03:13:21.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd734148d20 con 0x7fd73409fbd0 2022-07-20T03:13:21.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd734148fc0 con 0x7fd7340ad640 2022-07-20T03:13:21.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd73414a8d0 0x7fd734143660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd732ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 0x7fd734143170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd732ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 0x7fd734143170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:32910/0 (socket says 172.21.15.61:32910) 2022-07-20T03:13:21.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd732ffd700 1 -- 172.21.15.61:0/2037870010 learned_addr learned my addr 172.21.15.61:0/2037870010 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:21.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7337fe700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd73409fbd0 0x7fd7340a0080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 -- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 msgr2=0x7fd734143170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 0x7fd734143170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 -- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd73409fbd0 msgr2=0x7fd7340a0080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd73409fbd0 0x7fd7340a0080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 -- 172.21.15.61:0/2037870010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd73414a060 con 0x7fd73414a8d0 2022-07-20T03:13:21.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd732ffd700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 0x7fd734143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:21.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7327fc700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd73414a8d0 0x7fd734143660 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fd71c00d000 tx=0x7fd7340ac1a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:21.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd71c012540 con 0x7fd73414a8d0 2022-07-20T03:13:21.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd71c0126a0 con 0x7fd73414a8d0 2022-07-20T03:13:21.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd73414a370 con 0x7fd73414a8d0 2022-07-20T03:13:21.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd71c012a00 con 0x7fd73414a8d0 2022-07-20T03:13:21.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.022+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd734143b50 con 0x7fd73414a8d0 2022-07-20T03:13:21.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.026+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd734149180 con 0x7fd73414a8d0 2022-07-20T03:13:21.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.026+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd71c012b60 con 0x7fd73414a8d0 2022-07-20T03:13:21.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.026+0000 7fd72bfff700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd718053a40 0x7fd718055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:21.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.026+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd71c06b640 con 0x7fd73414a8d0 2022-07-20T03:13:21.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.026+0000 7fd732ffd700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd718053a40 0x7fd718055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:21.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.030+0000 7fd732ffd700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd718053a40 0x7fd718055f00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fd724007580 tx=0x7fd724007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:21.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.034+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd71c0425a0 con 0x7fd73414a8d0 2022-07-20T03:13:21.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.158+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fd7340b04b0 con 0x7fd73414a8d0 2022-07-20T03:13:21.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd72bfff700 1 -- 172.21.15.61:0/2037870010 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v61) v1 ==== 81+0+33696 (secure 0 0 0) 0x7fd71c019020 con 0x7fd73414a8d0 2022-07-20T03:13:21.205 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:21.210 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:13:21.211 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 61, 2022-07-20T03:13:21.211 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:13:21.211 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:13:21.212 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:12:53.016664+0000", 2022-07-20T03:13:21.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:13:21.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:13:21.214 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:13:21.215 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:13:21.215 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:13:21.216 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:13:21.216 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:13:21.216 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:13:21.217 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:13:21.217 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:21.217 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:13:21.218 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:13:21.218 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:13:21.219 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:13:21.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:13:21.220 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 12, 2022-07-20T03:13:21.220 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:13:21.221 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:13:21.221 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:13:21.221 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:13:21.222 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:13:21.222 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.223 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:13:21.223 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:13:21.225 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:13:21.225 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:13:21.226 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:13:21.226 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:13:21.227 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:13:21.227 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:13:21.227 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:13:21.228 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:13:21.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:13:21.229 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:13:21.229 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:13:21.230 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:13:21.230 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:13:21.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:13:21.231 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:13:21.232 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:13:21.232 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:13:21.234 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:13:21.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:13:21.236 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:13:21.236 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:13:21.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:13:21.242 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:13:21.243 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:13:21.243 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:13:21.244 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:13:21.244 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:13:21.245 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:13:21.245 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:13:21.246 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:13:21.254 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:13:21.254 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:13:21.255 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:13:21.255 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:13:21.256 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:13:21.256 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:13:21.257 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:13:21.257 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:13:21.257 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:13:21.258 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:13:21.259 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:13:21.260 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:13:21.260 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:13:21.260 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:13:21.261 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:13:21.261 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:13:21.262 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:13:21.262 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:13:21.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.264 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:13:21.264 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:13:21.265 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:13:21.265 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:13:21.266 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:13:21.266 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:13:21.267 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:13:21.267 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:13:21.269 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:13:21.269 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:13:21.270 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:13:21.270 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:13:21.271 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:13:21.271 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.272 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:13:21.272 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:13:21.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.273 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.275 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:13:21.275 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:13:21.276 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:13:21.276 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:13:21.277 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:13:21.277 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:13:21.278 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:13:21.279 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:13:21.279 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:13:21.279 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:13:21.280 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:13:21.280 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:13:21.281 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:13:21.283 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:13:21.283 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:13:21.285 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:13:21.285 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:13:21.286 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:13:21.286 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:13:21.287 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:13:21.287 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:13:21.288 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:13:21.288 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:13:21.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.289 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:13:21.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:13:21.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:13:21.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:13:21.291 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:13:21.292 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:13:21.292 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:13:21.292 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:13:21.293 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:13:21.293 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:13:21.293 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:13:21.294 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:13:21.295 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:13:21.296 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:13:21.296 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:13:21.297 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:13:21.297 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:13:21.297 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:13:21.298 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:13:21.298 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:13:21.299 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:13:21.299 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:13:21.300 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:13:21.300 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:13:21.301 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:13:21.301 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:13:21.302 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:13:21.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.303 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:13:21.303 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:13:21.304 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:13:21.304 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:13:21.304 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:13:21.305 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:13:21.305 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:13:21.315 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:13:21.321 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:13:21.322 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:13:21.343 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:13:21.345 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:13:21.345 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:13:21.346 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:13:21.346 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.347 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.347 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.347 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 11, 2022-07-20T03:13:21.348 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:13:21.348 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:12:51.013317+0000", 2022-07-20T03:13:21.348 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:13:21.349 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:13:21.349 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:13:21.356 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:13:21.358 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:13:21.358 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:13:21.359 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:13:21.359 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:13:21.360 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:13:21.360 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:13:21.360 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:13:21.361 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:13:21.361 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:13:21.361 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:13:21.362 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:13:21.362 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:13:21.363 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:13:21.363 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:13:21.364 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:13:21.365 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:13:21.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.365 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "60", 2022-07-20T03:13:21.366 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:13:21.366 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:13:21.367 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:13:21.367 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:13:21.368 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:13:21.370 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:13:21.378 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:13:21.379 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:13:21.379 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:13:21.380 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:13:21.380 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:13:21.381 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:13:21.381 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:13:21.382 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:13:21.382 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:13:21.383 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:13:21.383 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:13:21.394 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:13:21.394 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:13:21.395 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:13:21.395 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:13:21.396 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:13:21.396 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:13:21.397 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:13:21.397 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:13:21.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:13:21.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.401 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:13:21.401 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:13:21.402 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:13:21.402 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:13:21.403 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:13:21.403 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:13:21.403 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:13:21.404 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:13:21.404 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:13:21.405 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:13:21.405 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:13:21.405 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:13:21.406 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:13:21.406 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:13:21.407 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:13:21.408 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.409 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:13:21.409 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:13:21.410 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:13:21.410 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.411 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.411 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.412 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 12, 2022-07-20T03:13:21.413 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:13:21.418 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:12:51.616869+0000", 2022-07-20T03:13:21.419 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:13:21.419 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:13:21.419 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:13:21.420 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:13:21.421 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:13:21.422 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:13:21.422 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:13:21.423 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:13:21.423 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:13:21.428 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:13:21.429 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:13:21.429 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:13:21.430 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:13:21.430 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:13:21.430 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:13:21.431 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:13:21.431 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:13:21.431 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:13:21.432 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:13:21.432 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:13:21.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.433 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "61", 2022-07-20T03:13:21.434 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:13:21.435 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:13:21.435 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:13:21.435 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:13:21.436 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:13:21.436 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:13:21.437 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:13:21.437 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:13:21.438 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:13:21.439 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:13:21.439 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:13:21.440 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:13:21.440 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:13:21.441 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:13:21.441 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:13:21.442 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:13:21.442 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:13:21.450 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:13:21.450 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:13:21.451 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:13:21.451 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:13:21.452 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:13:21.453 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:13:21.454 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:13:21.454 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:13:21.456 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:13:21.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.457 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:13:21.457 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:13:21.458 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:13:21.458 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:13:21.459 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:13:21.459 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:13:21.460 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:13:21.460 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:13:21.461 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:13:21.461 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:13:21.462 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:13:21.462 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:13:21.463 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:13:21.463 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:13:21.463 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:13:21.464 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.464 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.465 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.465 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:21.468 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:13:21.469 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.469 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:13:21.470 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:13:21.470 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.471 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.471 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.472 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.472 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.472 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.473 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.473 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:21.474 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.474 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.475 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.475 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.476 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.476 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.477 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:13:21.477 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.477 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.478 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.478 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.479 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:13:21.479 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.480 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.480 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.481 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.481 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.483 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.483 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.484 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.484 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:13:21.485 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.485 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.486 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.487 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.487 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:13:21.488 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.488 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.489 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.489 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.490 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.490 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.490 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.491 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.491 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:13:21.491 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.491 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.492 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.492 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.492 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:13:21.492 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.493 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.493 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.493 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.493 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.494 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.494 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.494 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.494 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:13:21.494 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.495 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.495 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.495 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.495 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:13:21.496 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:21.496 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.496 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.496 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.497 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:13:21.497 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:13:21.497 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:13:21.497 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:13:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.498 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.499 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.499 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.499 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:13:21.499 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:13:21.500 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.500 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.500 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.500 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.500 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.501 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.501 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.501 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:21.501 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.502 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.502 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.502 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.503 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.503 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.503 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:13:21.503 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.504 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.504 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.504 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.504 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:13:21.505 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.505 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.505 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.505 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.505 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.506 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.506 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.506 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.507 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:13:21.507 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.507 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.508 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.508 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:13:21.508 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.508 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.509 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.509 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.509 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.509 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.509 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.510 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.510 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:13:21.510 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.510 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.511 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.512 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.512 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:13:21.512 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.512 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.513 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.513 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.513 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.513 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.513 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.514 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.514 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:13:21.514 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.514 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.515 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.515 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.515 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:13:21.515 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:21.516 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.516 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.516 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.516 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:13:21.516 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:13:21.517 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:13:21.517 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:13:21.517 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.517 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.518 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.518 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.519 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.519 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.519 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:13:21.519 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:13:21.520 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.520 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.520 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.520 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.521 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.521 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.521 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.521 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 58, 2022-07-20T03:13:21.522 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.522 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.522 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.522 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.522 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.523 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.523 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:13:21.523 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.523 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.524 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.524 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.524 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:13:21.525 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.525 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.525 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.525 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.526 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.526 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.526 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.526 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.526 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:13:21.527 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.527 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.527 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.527 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.528 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:13:21.528 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.528 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.529 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.530 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.530 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:13:21.531 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.532 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.532 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.532 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:13:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.534 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:13:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.535 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:13:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.536 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:13:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:13:21.537 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:13:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:13:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.538 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.539 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:13:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:13:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.540 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.541 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-20T03:13:21.542 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.542 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.542 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.543 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.543 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.543 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.543 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:13:21.544 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.544 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.544 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.544 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:13:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.546 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.546 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.546 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:13:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.548 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.548 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:13:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.550 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:13:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.551 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:13:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.552 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.553 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.554 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.554 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:13:21.554 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.555 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.555 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.555 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.556 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:13:21.556 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:21.557 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.558 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.559 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:13:21.559 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:13:21.560 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:13:21.560 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:13:21.561 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.561 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.562 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.562 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.563 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.563 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.564 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:13:21.564 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:13:21.564 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.565 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.565 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.566 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.566 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.567 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.567 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.568 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:21.568 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.569 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.569 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.570 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.570 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.571 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.571 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:13:21.572 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.572 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.573 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.573 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.574 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:13:21.574 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.575 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.575 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.576 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.576 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.577 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.578 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.578 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.579 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:13:21.579 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.580 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.580 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.581 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.581 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:13:21.582 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.582 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.583 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.583 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.585 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.586 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.586 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.587 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.587 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:13:21.588 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.588 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.589 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.589 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.590 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:13:21.610 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.611 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.612 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.612 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.612 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.613 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.613 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.615 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.615 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:13:21.616 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.616 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.616 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.617 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.617 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:13:21.618 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:21.618 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.625 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.625 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.626 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:13:21.626 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:13:21.627 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:13:21.627 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:13:21.628 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.628 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.629 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.629 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.632 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.632 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.633 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:13:21.633 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:13:21.634 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.634 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.635 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.635 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.635 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.636 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.637 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.637 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 58, 2022-07-20T03:13:21.638 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.639 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.639 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.639 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.640 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.640 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.641 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:13:21.641 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.642 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.643 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.643 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.644 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:13:21.644 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.645 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.645 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.646 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.646 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.647 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.647 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.648 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.648 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:13:21.649 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.649 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.651 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.651 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.652 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:13:21.652 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.653 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.654 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.654 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.655 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.655 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.656 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.656 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.656 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:13:21.657 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.657 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.658 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.658 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.659 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:13:21.659 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.660 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.660 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.661 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.662 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.662 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.663 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.664 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.665 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:13:21.665 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.666 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.666 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.667 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.667 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:13:21.668 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:21.668 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.669 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.669 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.669 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:13:21.670 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:13:21.670 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:13:21.671 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:13:21.673 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.673 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.674 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.676 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.677 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.689 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.690 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:13:21.690 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:13:21.691 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.691 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.692 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.692 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.692 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.692 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.692 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.693 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:21.693 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.693 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.693 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.694 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.694 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.694 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.694 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:13:21.696 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.696 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.696 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.696 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.696 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:13:21.697 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.697 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.697 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.697 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.698 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.698 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.699 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.699 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.699 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:13:21.699 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.700 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.700 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.700 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.700 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:13:21.701 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.701 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.703 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.703 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.703 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.704 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.705 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.705 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:13:21.705 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.706 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.706 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.706 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.706 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:13:21.707 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.707 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.707 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.707 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.707 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.708 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.709 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.709 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:13:21.709 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.710 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.710 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.710 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.711 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:13:21.711 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:21.712 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.712 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.713 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.713 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:13:21.714 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:13:21.715 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:13:21.715 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:13:21.715 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.716 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.716 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.717 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.717 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.718 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.718 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:13:21.718 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:13:21.719 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:21.719 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:21.720 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:21.720 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:21.720 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:21.720 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:21.721 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:21.721 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 58, 2022-07-20T03:13:21.721 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:21.722 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:21.728 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:21.729 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.729 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.739 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.740 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:13:21.744 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.744 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.745 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.747 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.747 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:13:21.748 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.748 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.749 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.749 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.750 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:21.750 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.750 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.751 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.751 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:13:21.752 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.753 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.753 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.753 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:13:21.754 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.755 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.755 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.756 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.757 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:21.757 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.758 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.758 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.759 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:13:21.759 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.760 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.761 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.761 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:13:21.763 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.763 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.764 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.764 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.766 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:21.766 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:21.768 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.768 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:21.768 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:13:21.769 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.769 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.772 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.773 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:21.774 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:13:21.775 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:21.776 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.776 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.776 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.777 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:13:21.779 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:13:21.780 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:13:21.781 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:13:21.782 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:21.783 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:21.783 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:21.784 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:21.784 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.785 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:21.805 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:13:21.805 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.805 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:13:21.806 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.806 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.806 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.806 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.807 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.807 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:13:21.807 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.807 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.808 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.808 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:13:21.808 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.808 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.808 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.809 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.809 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.809 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:13:21.809 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.810 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.810 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.810 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:13:21.811 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.811 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.811 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.811 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.811 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.812 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:13:21.812 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.812 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.812 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.813 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:13:21.813 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.813 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.813 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.814 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.814 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.814 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:13:21.814 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.815 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.815 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.815 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:13:21.815 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.816 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.816 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.816 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.816 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:13:21.835 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.835 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.835 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.836 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:13:21.836 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.836 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.836 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.837 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.837 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.842 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:13:21.842 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.843 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.843 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.844 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:13:21.844 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.845 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.845 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.847 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.848 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.848 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:13:21.849 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.849 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.849 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:21.850 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:13:21.850 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:21.850 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:21.851 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:21.851 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:21.852 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:21.852 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:13:21.853 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:21.853 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.853 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:21.854 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:13:21.854 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:13:21.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:13:21.855 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:13:21.855 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:13:21.856 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:13:21.858 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:13:21.872 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:13:21.872 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:13:21.873 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:13:21.873 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:13:21.874 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:13:21.875 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:13:21.875 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:21.876 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:21.876 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:13:21.877 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:13:21.877 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:13:21.877 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:13:21.878 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:13:21.879 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:13:21.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd718053a40 msgr2=0x7fd718055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd718053a40 0x7fd718055f00 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fd724007580 tx=0x7fd724007a00).stop 2022-07-20T03:13:21.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd73414a8d0 msgr2=0x7fd734143660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:21.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd73414a8d0 0x7fd734143660 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7fd71c00d000 tx=0x7fd7340ac1a0).stop 2022-07-20T03:13:21.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 shutdown_connections 2022-07-20T03:13:21.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd718053a40 0x7fd718055f00 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd73409fbd0 0x7fd7340a0080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd73414a8d0 0x7fd734143660 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 --2- 172.21.15.61:0/2037870010 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7340ad640 0x7fd734143170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:21.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 >> 172.21.15.61:0/2037870010 conn(0x7fd7341441f0 msgr2=0x7fd7340b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:21.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 shutdown_connections 2022-07-20T03:13:21.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:21.162+0000 7fd7399f0700 1 -- 172.21.15.61:0/2037870010 wait complete. 2022-07-20T03:13:21.911 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:22.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08f59e700 1 Processor -- start 2022-07-20T03:13:22.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08f59e700 1 -- start start 2022-07-20T03:13:22.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900af8e0 0x7fe0900b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe090147010 con 0x7fe0900adff0 2022-07-20T03:13:22.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08f59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0901472b0 con 0x7fe0900b5650 2022-07-20T03:13:22.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe090146d40 con 0x7fe0900af8e0 2022-07-20T03:13:22.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08dd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe0900adff0 0x7fe09005bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42846/0 (socket says 172.21.15.61:42846) 2022-07-20T03:13:22.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08dd9b700 1 -- 172.21.15.61:0/2615408195 learned_addr learned my addr 172.21.15.61:0/2615408195 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:22.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2615408195 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900af8e0 0x7fe0900b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2364702745 0 0) 0x7fe090147010 con 0x7fe0900adff0 2022-07-20T03:13:22.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe070003260 con 0x7fe0900adff0 2022-07-20T03:13:22.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3179290025 0 0) 0x7fe0901472b0 con 0x7fe0900b5650 2022-07-20T03:13:22.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe090147010 con 0x7fe0900b5650 2022-07-20T03:13:22.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08ed9d700 1 -- 172.21.15.61:0/2615408195 >> v1:172.21.15.117:6789/0 conn(0x7fe0900b5650 legacy=0x7fe090147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.022+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2491097616 0 0) 0x7fe070003260 con 0x7fe0900adff0 2022-07-20T03:13:22.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08ed9d700 1 -- 172.21.15.61:0/2615408195 >> v1:172.21.15.61:6789/0 conn(0x7fe0900adff0 legacy=0x7fe09005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08ed9d700 1 -- 172.21.15.61:0/2615408195 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe09005c4a0 con 0x7fe0900af8e0 2022-07-20T03:13:22.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2615408195 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900af8e0 0x7fe0900b5a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe078001290 tx=0x7fe080006df0).ready entity=mon.2 client_cookie=73cbd2ffc5e5bd06 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:22.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0780017e0 con 0x7fe0900af8e0 2022-07-20T03:13:22.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe080000e90 con 0x7fe0900af8e0 2022-07-20T03:13:22.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08d59a700 1 -- 172.21.15.61:0/2615408195 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe080012460 con 0x7fe0900af8e0 2022-07-20T03:13:22.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 -- 172.21.15.61:0/2615408195 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900af8e0 msgr2=0x7fe0900b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2615408195 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900af8e0 0x7fe0900b5a30 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fe078001290 tx=0x7fe080006df0).stop 2022-07-20T03:13:22.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 -- 172.21.15.61:0/2615408195 shutdown_connections 2022-07-20T03:13:22.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2615408195 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900af8e0 0x7fe0900b5a30 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 -- 172.21.15.61:0/2615408195 >> 172.21.15.61:0/2615408195 conn(0x7fe0900634e0 msgr2=0x7fe0900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:22.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 -- 172.21.15.61:0/2615408195 shutdown_connections 2022-07-20T03:13:22.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 -- 172.21.15.61:0/2615408195 wait complete. 2022-07-20T03:13:22.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 Processor -- start 2022-07-20T03:13:22.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.026+0000 7fe08f59e700 1 -- start start 2022-07-20T03:13:22.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0900adff0 0x7fe09014a690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 0x7fe09010e130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900b5650 0x7fe09010f940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe09010e620 con 0x7fe0900af8e0 2022-07-20T03:13:22.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe090064c50 con 0x7fe0900adff0 2022-07-20T03:13:22.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe09010fe30 con 0x7fe0900b5650 2022-07-20T03:13:22.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 0x7fe09010e130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 0x7fe09010e130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46254/0 (socket says 172.21.15.61:46254) 2022-07-20T03:13:22.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 -- 172.21.15.61:0/2836553945 learned_addr learned my addr 172.21.15.61:0/2836553945 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:22.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08ed9d700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900b5650 0x7fe09010f940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08e59c700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0900adff0 0x7fe09014a690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 -- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900b5650 msgr2=0x7fe09010f940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900b5650 0x7fe09010f940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 -- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0900adff0 msgr2=0x7fe09014a690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0900adff0 0x7fe09014a690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 -- 172.21.15.61:0/2836553945 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe08000b180 con 0x7fe0900af8e0 2022-07-20T03:13:22.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08dd9b700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 0x7fe09010e130 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fe078001160 tx=0x7fe078002ea0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:22.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0780070d0 con 0x7fe0900af8e0 2022-07-20T03:13:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe078007230 con 0x7fe0900af8e0 2022-07-20T03:13:22.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0901100c0 con 0x7fe0900af8e0 2022-07-20T03:13:22.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0780073a0 con 0x7fe0900af8e0 2022-07-20T03:13:22.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.030+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe090110d60 con 0x7fe0900af8e0 2022-07-20T03:13:22.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.034+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe090064df0 con 0x7fe0900af8e0 2022-07-20T03:13:22.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.034+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe078010070 con 0x7fe0900af8e0 2022-07-20T03:13:22.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.034+0000 7fe0877fe700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0740539b0 0x7fe074055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.034+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe07806b6f0 con 0x7fe0900af8e0 2022-07-20T03:13:22.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.034+0000 7fe08e59c700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0740539b0 0x7fe074055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.034+0000 7fe08e59c700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0740539b0 0x7fe074055e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fe08800d000 tx=0x7fe088007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:22.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.042+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe090064df0 con 0x7fe0900af8e0 2022-07-20T03:13:22.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.202+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe090064df0 con 0x7fe0900af8e0 2022-07-20T03:13:22.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.206+0000 7fe0877fe700 1 -- 172.21.15.61:0/2836553945 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 35 v35) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fe090064df0 con 0x7fe0900af8e0 2022-07-20T03:13:22.207 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:22.208 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":35,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33}],"filesystems":[{"mdsmap":{"epoch":35,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:12:53.015972+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":5982},"failed":[],"damaged":[],"stopped":[],"info":{"gid_5982":{"gid":5982,"name":"a","rank":0,"incarnation":34,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2218834214","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2218834214},{"type":"v1","addr":"172.21.15.61:6833","nonce":2218834214}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:13:22.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.206+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0740539b0 msgr2=0x7fe074055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.206+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0740539b0 0x7fe074055e70 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fe08800d000 tx=0x7fe088007da0).stop 2022-07-20T03:13:22.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.206+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 msgr2=0x7fe09010e130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.206+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 0x7fe09010e130 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fe078001160 tx=0x7fe078002ea0).stop 2022-07-20T03:13:22.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 shutdown_connections 2022-07-20T03:13:22.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0740539b0 0x7fe074055e70 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0900b5650 0x7fe09010f940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0900af8e0 0x7fe09010e130 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 --2- 172.21.15.61:0/2836553945 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0900adff0 0x7fe09014a690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 >> 172.21.15.61:0/2836553945 conn(0x7fe0900634e0 msgr2=0x7fe090146070 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:22.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 shutdown_connections 2022-07-20T03:13:22.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.210+0000 7fe08f59e700 1 -- 172.21.15.61:0/2836553945 wait complete. 2022-07-20T03:13:22.242 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 35 2022-07-20T03:13:22.246 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:13:22.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 Processor -- start 2022-07-20T03:13:22.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 -- start start 2022-07-20T03:13:22.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f47480b5990 0x7f47480ae050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480af8e0 0x7f4748146960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4748142ef0 0x7f47481433a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4748146e50 con 0x7f4748142ef0 2022-07-20T03:13:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4748142a90 con 0x7f47480b5990 2022-07-20T03:13:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474ec10700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f47480b5d70 con 0x7f47480af8e0 2022-07-20T03:13:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4748142ef0 0x7f47481433a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4748142ef0 0x7f47481433a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46264/0 (socket says 172.21.15.61:46264) 2022-07-20T03:13:22.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 -- 172.21.15.61:0/1963401392 learned_addr learned my addr 172.21.15.61:0/1963401392 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:22.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474d40d700 1 --2- 172.21.15.61:0/1963401392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480af8e0 0x7f4748146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474dc0e700 1 --2- 172.21.15.61:0/1963401392 >> v2:172.21.15.117:3300/0 conn(0x7f47480b5990 0x7f47480ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 -- 172.21.15.61:0/1963401392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480af8e0 msgr2=0x7f4748146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 --2- 172.21.15.61:0/1963401392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480af8e0 0x7f4748146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 -- 172.21.15.61:0/1963401392 >> v2:172.21.15.117:3300/0 conn(0x7f47480b5990 msgr2=0x7f47480ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:22.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 --2- 172.21.15.61:0/1963401392 >> v2:172.21.15.117:3300/0 conn(0x7f47480b5990 0x7f47480ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 -- 172.21.15.61:0/1963401392 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f47480afdf0 con 0x7f4748142ef0 2022-07-20T03:13:22.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.362+0000 7f474e40f700 1 --2- 172.21.15.61:0/1963401392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f47481433a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f4740002f70 tx=0x7f4740006eb0).ready entity=mon.0 client_cookie=c4b5a459cf9ea8a2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:22.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474cc0c700 1 -- 172.21.15.61:0/1963401392 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f474000d570 con 0x7f4748142ef0 2022-07-20T03:13:22.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474cc0c700 1 -- 172.21.15.61:0/1963401392 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4740000e90 con 0x7f4748142ef0 2022-07-20T03:13:22.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474cc0c700 1 -- 172.21.15.61:0/1963401392 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f474000d900 con 0x7f4748142ef0 2022-07-20T03:13:22.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 -- 172.21.15.61:0/1963401392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 msgr2=0x7f47481433a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 --2- 172.21.15.61:0/1963401392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f47481433a0 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f4740002f70 tx=0x7f4740006eb0).stop 2022-07-20T03:13:22.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 -- 172.21.15.61:0/1963401392 shutdown_connections 2022-07-20T03:13:22.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 --2- 172.21.15.61:0/1963401392 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f47481433a0 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 --2- 172.21.15.61:0/1963401392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480af8e0 0x7f4748146960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 --2- 172.21.15.61:0/1963401392 >> v2:172.21.15.117:3300/0 conn(0x7f47480b5990 0x7f47480ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 -- 172.21.15.61:0/1963401392 >> 172.21.15.61:0/1963401392 conn(0x7f4748063480 msgr2=0x7f47480b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:22.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 -- 172.21.15.61:0/1963401392 shutdown_connections 2022-07-20T03:13:22.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.366+0000 7f474ec10700 1 -- 172.21.15.61:0/1963401392 wait complete. 2022-07-20T03:13:22.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 Processor -- start 2022-07-20T03:13:22.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 -- start start 2022-07-20T03:13:22.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47480af8e0 0x7f4748113b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480b5990 0x7f47480642a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f4748064790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f47480674a0 con 0x7f4748142ef0 2022-07-20T03:13:22.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4748066f60 con 0x7f47480af8e0 2022-07-20T03:13:22.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4748067200 con 0x7f47480b5990 2022-07-20T03:13:22.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474e40f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f4748064790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47480af8e0 0x7f4748113b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474e40f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f4748064790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46270/0 (socket says 172.21.15.61:46270) 2022-07-20T03:13:22.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474e40f700 1 -- 172.21.15.61:0/3960568685 learned_addr learned my addr 172.21.15.61:0/3960568685 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:22.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474d40d700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480b5990 0x7f47480642a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 -- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480b5990 msgr2=0x7f47480642a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480b5990 0x7f47480642a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 -- 172.21.15.61:0/3960568685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 msgr2=0x7f4748064790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:22.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f4748064790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 -- 172.21.15.61:0/3960568685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f474000b1e0 con 0x7f47480af8e0 2022-07-20T03:13:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474dc0e700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47480af8e0 0x7f4748113b70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f473c00a000 tx=0x7f473c008040).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f473c006430 con 0x7f47480af8e0 2022-07-20T03:13:22.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f473c006590 con 0x7f47480af8e0 2022-07-20T03:13:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4748153200 con 0x7f47480af8e0 2022-07-20T03:13:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f473c0058e0 con 0x7f47480af8e0 2022-07-20T03:13:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.370+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4748114060 con 0x7f47480af8e0 2022-07-20T03:13:22.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.374+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f473c005a40 con 0x7f47480af8e0 2022-07-20T03:13:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.374+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f47480b0090 con 0x7f47480af8e0 2022-07-20T03:13:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.374+0000 7f473affd700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f47280539b0 0x7f4728055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.374+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(61..61 src has 1..61) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f473c06bda0 con 0x7f47480af8e0 2022-07-20T03:13:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.374+0000 7f474d40d700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f47280539b0 0x7f4728055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:22.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.374+0000 7f474d40d700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f47280539b0 0x7f4728055e70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f474400b000 tx=0x7f47440091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:22.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.386+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f473c042b70 con 0x7f47480af8e0 2022-07-20T03:13:22.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:22.546+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f47480574e0 con 0x7f47480af8e0 2022-07-20T03:13:22.638 INFO:tasks.ceph.mds.a.smithi061.stderr:2022-07-20T03:13:22.638+0000 7fa735acf700 -1 MDSIOContextBase: failed with -108, restarting... 2022-07-20T03:13:22.665 INFO:tasks.ceph.mds.a.smithi061.stderr: -1> 2022-07-20T03:13:22.638+0000 7fa735acf700 -1 MDSIOContextBase: failed with -108, restarting... 2022-07-20T03:13:22.685 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:13:23.206 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:13:23.206+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:13:23.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.222+0000 7f473affd700 1 -- 172.21.15.61:0/3960568685 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v36) v1 ==== 161+0+0 (secure 0 0 0) 0x7f473c046190 con 0x7f47480af8e0 2022-07-20T03:13:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f47280539b0 msgr2=0x7f4728055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f47280539b0 0x7f4728055e70 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f474400b000 tx=0x7f47440091d0).stop 2022-07-20T03:13:23.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47480af8e0 msgr2=0x7f4748113b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47480af8e0 0x7f4748113b70 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f473c00a000 tx=0x7f473c008040).stop 2022-07-20T03:13:23.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 shutdown_connections 2022-07-20T03:13:23.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f47280539b0 0x7f4728055e70 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4748142ef0 0x7f4748064790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f47480b5990 0x7f47480642a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 --2- 172.21.15.61:0/3960568685 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f47480af8e0 0x7f4748113b70 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 >> 172.21.15.61:0/3960568685 conn(0x7f4748063480 msgr2=0x7f47480b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:23.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 shutdown_connections 2022-07-20T03:13:23.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.226+0000 7f474ec10700 1 -- 172.21.15.61:0/3960568685 wait complete. 2022-07-20T03:13:23.276 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:13:23.277 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:23.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 Processor -- start 2022-07-20T03:13:23.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 -- start start 2022-07-20T03:13:23.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa2200b1ef0 0x7fa220142f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2200ac3a0 0x7fa220143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa220060980 con 0x7fa220143960 2022-07-20T03:13:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2200b22d0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.406+0000 7fa22622f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa220098810 con 0x7fa2200b1ef0 2022-07-20T03:13:23.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2200ac3a0 0x7fa220143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2200ac3a0 0x7fa220143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46288/0 (socket says 172.21.15.61:46288) 2022-07-20T03:13:23.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 -- 172.21.15.61:0/2360174161 learned_addr learned my addr 172.21.15.61:0/2360174161 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:23.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22522d700 1 --2- 172.21.15.61:0/2360174161 >> v2:172.21.15.117:3300/0 conn(0x7fa2200b1ef0 0x7fa220142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 -- 172.21.15.61:0/2360174161 >> v2:172.21.15.117:3300/0 conn(0x7fa2200b1ef0 msgr2=0x7fa220142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:23.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 --2- 172.21.15.61:0/2360174161 >> v2:172.21.15.117:3300/0 conn(0x7fa2200b1ef0 0x7fa220142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 -- 172.21.15.61:0/2360174161 >> v1:172.21.15.117:6789/0 conn(0x7fa220143960 legacy=0x7fa2200ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 -- 172.21.15.61:0/2360174161 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa220060bb0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa224a2c700 1 --2- 172.21.15.61:0/2360174161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220143470 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7fa210002f70 tx=0x7fa210006eb0).ready entity=mon.0 client_cookie=a4fd89c9eecee8d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa217fff700 1 -- 172.21.15.61:0/2360174161 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21000d610 con 0x7fa2200ac3a0 2022-07-20T03:13:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa217fff700 1 -- 172.21.15.61:0/2360174161 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa210000e90 con 0x7fa2200ac3a0 2022-07-20T03:13:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa217fff700 1 -- 172.21.15.61:0/2360174161 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21000d9a0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 -- 172.21.15.61:0/2360174161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 msgr2=0x7fa220143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 --2- 172.21.15.61:0/2360174161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220143470 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7fa210002f70 tx=0x7fa210006eb0).stop 2022-07-20T03:13:23.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 -- 172.21.15.61:0/2360174161 shutdown_connections 2022-07-20T03:13:23.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 --2- 172.21.15.61:0/2360174161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220143470 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 --2- 172.21.15.61:0/2360174161 >> v2:172.21.15.117:3300/0 conn(0x7fa2200b1ef0 0x7fa220142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 -- 172.21.15.61:0/2360174161 >> 172.21.15.61:0/2360174161 conn(0x7fa2200b2ca0 msgr2=0x7fa2200b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 -- 172.21.15.61:0/2360174161 shutdown_connections 2022-07-20T03:13:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.410+0000 7fa22622f700 1 -- 172.21.15.61:0/2360174161 wait complete. 2022-07-20T03:13:23.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 Processor -- start 2022-07-20T03:13:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- start start 2022-07-20T03:13:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220099cb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2200b1ef0 0x7fa220141850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa220143960 0x7fa220141d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200ad640 con 0x7fa2200ac3a0 2022-07-20T03:13:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa22009a1a0 con 0x7fa2200b1ef0 2022-07-20T03:13:23.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa2200ad830 con 0x7fa220143960 2022-07-20T03:13:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220099cb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220099cb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46294/0 (socket says 172.21.15.61:46294) 2022-07-20T03:13:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 -- 172.21.15.61:0/2345011898 learned_addr learned my addr 172.21.15.61:0/2345011898 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa224a2c700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2200b1ef0 0x7fa220141850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa225a2e700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa220143960 0x7fa220141d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 -- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa220143960 msgr2=0x7fa220141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa220143960 0x7fa220141d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 -- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2200b1ef0 msgr2=0x7fa220141850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2200b1ef0 0x7fa220141850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 -- 172.21.15.61:0/2345011898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa21000b1e0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa224a2c700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2200b1ef0 0x7fa220141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22522d700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220099cb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7fa21c00a000 tx=0x7fa21c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:23.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21c011780 con 0x7fa2200ac3a0 2022-07-20T03:13:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa21c0118e0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2200adc30 con 0x7fa2200ac3a0 2022-07-20T03:13:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa21c011c40 con 0x7fa2200ac3a0 2022-07-20T03:13:23.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa22009a780 con 0x7fa2200ac3a0 2022-07-20T03:13:23.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa220062270 con 0x7fa2200ac3a0 2022-07-20T03:13:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.414+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa21c01c460 con 0x7fa2200ac3a0 2022-07-20T03:13:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.418+0000 7fa2167fc700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053ad0 0x7fa208055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.418+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fa21c06c3b0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.418+0000 7fa224a2c700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053ad0 0x7fa208055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.418+0000 7fa224a2c700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053ad0 0x7fa208055f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa21001a000 tx=0x7fa210015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:23.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.422+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa21c043260 con 0x7fa2200ac3a0 2022-07-20T03:13:23.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.585+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa2200574e0 con 0x7fa2200ac3a0 2022-07-20T03:13:23.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.589+0000 7fa2167fc700 1 -- 172.21.15.61:0/2345011898 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 36 v36) v1 ==== 94+0+2135 (secure 0 0 0) 0x7fa21c046880 con 0x7fa2200ac3a0 2022-07-20T03:13:23.592 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:23.593 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":36,"default_fscid":5,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2549659477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2549659477},{"type":"v1","addr":"172.21.15.61:6833","nonce":2549659477}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":36}],"filesystems":[{"mdsmap":{"epoch":36,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:12:52.013020+0000","modified":"2022-07-20T03:13:23.208785+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":62,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[12],"metadata_pool":11,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":5}]} 2022-07-20T03:13:23.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.589+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053ad0 msgr2=0x7fa208055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.589+0000 7fa22622f700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053ad0 0x7fa208055f90 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fa21001a000 tx=0x7fa210015000).stop 2022-07-20T03:13:23.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.589+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 msgr2=0x7fa220099cb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.589+0000 7fa22622f700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220099cb0 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7fa21c00a000 tx=0x7fa21c008040).stop 2022-07-20T03:13:23.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 shutdown_connections 2022-07-20T03:13:23.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053ad0 0x7fa208055f90 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2200b1ef0 0x7fa220141850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa220143960 0x7fa220141d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 --2- 172.21.15.61:0/2345011898 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2200ac3a0 0x7fa220099cb0 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 >> 172.21.15.61:0/2345011898 conn(0x7fa2200b2ca0 msgr2=0x7fa2200ab390 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:23.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 shutdown_connections 2022-07-20T03:13:23.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.593+0000 7fa22622f700 1 -- 172.21.15.61:0/2345011898 wait complete. 2022-07-20T03:13:23.597 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 36 2022-07-20T03:13:23.611 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:13:23.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 Processor -- start 2022-07-20T03:13:23.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 -- start start 2022-07-20T03:13:23.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b680ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0b680af8c0 0x7f0b680ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0b680b0b80 con 0x7f0b680aeae0 2022-07-20T03:13:23.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0b6805d4d0 con 0x7f0b680af8c0 2022-07-20T03:13:23.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b67fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b680b5d50 con 0x7f0b680b5970 2022-07-20T03:13:23.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b677fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0b680aeae0 0x7f0b6805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42904/0 (socket says 172.21.15.61:42904) 2022-07-20T03:13:23.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b677fe700 1 -- 172.21.15.61:0/494086246 learned_addr learned my addr 172.21.15.61:0/494086246 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:23.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b667fc700 1 --2- 172.21.15.61:0/494086246 >> v2:172.21.15.117:3300/0 conn(0x7f0b680af8c0 0x7f0b680ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b66ffd700 1 --2- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b680ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b65ffb700 1 -- 172.21.15.61:0/494086246 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 93156204 0 0) 0x7f0b680b0b80 con 0x7f0b680aeae0 2022-07-20T03:13:23.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b65ffb700 1 -- 172.21.15.61:0/494086246 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0b48003260 con 0x7f0b680aeae0 2022-07-20T03:13:23.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b667fc700 1 -- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 msgr2=0x7f0b680ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:23.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b667fc700 1 --2- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b680ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b667fc700 1 -- 172.21.15.61:0/494086246 >> v1:172.21.15.61:6789/0 conn(0x7f0b680aeae0 legacy=0x7f0b6805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.725+0000 7f0b667fc700 1 -- 172.21.15.61:0/494086246 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b680b0db0 con 0x7f0b680af8c0 2022-07-20T03:13:23.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 --2- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680af8c0 0x7f0b680ae5f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f0b500032f0 tx=0x7f0b500091d0).ready entity=mon.1 client_cookie=e4df2b105b8f5f86 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:23.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b65ffb700 1 -- 172.21.15.61:0/494086246 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b5000b600 con 0x7f0b680af8c0 2022-07-20T03:13:23.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b65ffb700 1 -- 172.21.15.61:0/494086246 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b5000b760 con 0x7f0b680af8c0 2022-07-20T03:13:23.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b65ffb700 1 -- 172.21.15.61:0/494086246 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b5000b8d0 con 0x7f0b680af8c0 2022-07-20T03:13:23.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680af8c0 msgr2=0x7f0b680ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 --2- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680af8c0 0x7f0b680ae5f0 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f0b500032f0 tx=0x7f0b500091d0).stop 2022-07-20T03:13:23.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/494086246 shutdown_connections 2022-07-20T03:13:23.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 --2- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680af8c0 0x7f0b680ae5f0 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 --2- 172.21.15.61:0/494086246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b680ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/494086246 >> 172.21.15.61:0/494086246 conn(0x7f0b68063490 msgr2=0x7f0b680b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:23.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/494086246 shutdown_connections 2022-07-20T03:13:23.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/494086246 wait complete. 2022-07-20T03:13:23.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 Processor -- start 2022-07-20T03:13:23.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- start start 2022-07-20T03:13:23.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680aeae0 0x7f0b68159f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 0x7f0b680b3910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b68066a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68111490 con 0x7f0b680af8c0 2022-07-20T03:13:23.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0b68148750 con 0x7f0b680aeae0 2022-07-20T03:13:23.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0b6814bc00 con 0x7f0b680b5970 2022-07-20T03:13:23.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 0x7f0b680b3910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 0x7f0b680b3910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46310/0 (socket says 172.21.15.61:46310) 2022-07-20T03:13:23.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 -- 172.21.15.61:0/3317671596 learned_addr learned my addr 172.21.15.61:0/3317671596 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:23.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b66ffd700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680aeae0 0x7f0b68159f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b677fe700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b68066a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 -- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 msgr2=0x7f0b68066a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b68066a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 -- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680aeae0 msgr2=0x7f0b68159f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:23.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680aeae0 0x7f0b68159f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:23.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 -- 172.21.15.61:0/3317671596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0b50009470 con 0x7f0b680af8c0 2022-07-20T03:13:23.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b66ffd700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680aeae0 0x7f0b68159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:23.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b667fc700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 0x7f0b680b3910 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f0b500032f0 tx=0x7f0b50016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:23.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b50011cc0 con 0x7f0b680af8c0 2022-07-20T03:13:23.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0b6815a400 con 0x7f0b680af8c0 2022-07-20T03:13:23.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.729+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0b680b3e00 con 0x7f0b680af8c0 2022-07-20T03:13:23.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0b50011e20 con 0x7f0b680af8c0 2022-07-20T03:13:23.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0b5001f5e0 con 0x7f0b680af8c0 2022-07-20T03:13:23.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0b6814ba30 con 0x7f0b680af8c0 2022-07-20T03:13:23.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0b50026030 con 0x7f0b680af8c0 2022-07-20T03:13:23.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b5ffff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0b4c053a40 0x7f0b4c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:23.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f0b5006c9d0 con 0x7f0b680af8c0 2022-07-20T03:13:23.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b66ffd700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0b4c053a40 0x7f0b4c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:23.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.733+0000 7f0b66ffd700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0b4c053a40 0x7f0b4c055f00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f0b5800b000 tx=0x7f0b58008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:23.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.741+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0b50043930 con 0x7f0b680af8c0 2022-07-20T03:13:23.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:23.901+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f0b680acae0 con 0x7f0b680af8c0 2022-07-20T03:13:24.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.721+0000 7f0b5ffff700 1 -- 172.21.15.61:0/3317671596 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v37) v1 ==== 104+0+0 (secure 0 0 0) 0x7f0b500240a0 con 0x7f0b680af8c0 2022-07-20T03:13:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0b4c053a40 msgr2=0x7f0b4c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0b4c053a40 0x7f0b4c055f00 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f0b5800b000 tx=0x7f0b58008040).stop 2022-07-20T03:13:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 msgr2=0x7f0b680b3910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 0x7f0b680b3910 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f0b500032f0 tx=0x7f0b50016000).stop 2022-07-20T03:13:24.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 shutdown_connections 2022-07-20T03:13:24.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0b4c053a40 0x7f0b4c055f00 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0b680b5970 0x7f0b68066a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0b680af8c0 0x7f0b680b3910 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 --2- 172.21.15.61:0/3317671596 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0b680aeae0 0x7f0b68159f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 >> 172.21.15.61:0/3317671596 conn(0x7f0b68063490 msgr2=0x7f0b680ae020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:24.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 shutdown_connections 2022-07-20T03:13:24.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.725+0000 7f0b67fff700 1 -- 172.21.15.61:0/3317671596 wait complete. 2022-07-20T03:13:24.740 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:13:24.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.873+0000 7fe031caf700 1 Processor -- start 2022-07-20T03:13:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.873+0000 7fe031caf700 1 -- start start 2022-07-20T03:13:24.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe02c0af200 0x7fe02c13e800 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 0x7fe02c13ff00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe02c13f390 con 0x7fe02c063b20 2022-07-20T03:13:24.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe02c13f1a0 con 0x7fe02c0af200 2022-07-20T03:13:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0b0960 con 0x7fe02c13ecf0 2022-07-20T03:13:24.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe02c063b20 0x7fe02c13f7f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42916/0 (socket says 172.21.15.61:42916) 2022-07-20T03:13:24.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02affd700 1 -- 172.21.15.61:0/4097434624 learned_addr learned my addr 172.21.15.61:0/4097434624 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:24.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02b7fe700 1 --2- 172.21.15.61:0/4097434624 >> v2:172.21.15.117:3300/0 conn(0x7fe02c0af200 0x7fe02c13e800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 0x7fe02c13ff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1639411417 0 0) 0x7fe02c13f390 con 0x7fe02c063b20 2022-07-20T03:13:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe00c003260 con 0x7fe02c063b20 2022-07-20T03:13:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 312896162 0 0) 0x7fe00c003260 con 0x7fe02c063b20 2022-07-20T03:13:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 msgr2=0x7fe02c13ff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 0x7fe02c13ff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 >> v2:172.21.15.117:3300/0 conn(0x7fe02c0af200 msgr2=0x7fe02c13e800 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:24.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 --2- 172.21.15.61:0/4097434624 >> v2:172.21.15.117:3300/0 conn(0x7fe02c0af200 0x7fe02c13e800 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c0b06c0 con 0x7fe02c063b20 2022-07-20T03:13:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fe01c001a00 con 0x7fe02c063b20 2022-07-20T03:13:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02b7fe700 1 --2- 172.21.15.61:0/4097434624 >> v2:172.21.15.117:3300/0 conn(0x7fe02c0af200 0x7fe02c13e800 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:24.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 >> v1:172.21.15.61:6789/0 conn(0x7fe02c063b20 legacy=0x7fe02c13f7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 0x7fe00c003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c003c80 0x7fe00c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe00c0049f0 0x7fe00c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c0b0960 con 0x7fe00c0049f0 2022-07-20T03:13:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c00b750 con 0x7fe00c003c80 2022-07-20T03:13:24.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe00c00b890 con 0x7fe02c13ecf0 2022-07-20T03:13:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02b7fe700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 0x7fe00c003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe00c0049f0 0x7fe00c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02affd700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c003c80 0x7fe00c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 -- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 msgr2=0x7fe00c003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:24.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c13ecf0 0x7fe00c003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 -- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c003c80 msgr2=0x7fe00c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:24.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c003c80 0x7fe00c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 -- 172.21.15.61:0/4097434624 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe00c003260 con 0x7fe00c0049f0 2022-07-20T03:13:24.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe00c0049f0 0x7fe00c004e70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fe00c0054d0 tx=0x7fe024006e80).ready entity=mon.0 client_cookie=c17f53fb16cb0e86 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:24.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe02400f750 con 0x7fe00c0049f0 2022-07-20T03:13:24.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe02400f8b0 con 0x7fe00c0049f0 2022-07-20T03:13:24.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02a7fc700 1 -- 172.21.15.61:0/4097434624 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe02400fc10 con 0x7fe00c0049f0 2022-07-20T03:13:24.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 -- 172.21.15.61:0/4097434624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe00c0049f0 msgr2=0x7fe00c004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:24.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe031caf700 1 --2- 172.21.15.61:0/4097434624 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe00c0049f0 0x7fe00c004e70 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fe00c0054d0 tx=0x7fe024006e80).stop 2022-07-20T03:13:24.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.877+0000 7fe02bfff700 1 -- 172.21.15.61:0/4097434624 reap_dead start 2022-07-20T03:13:24.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- 172.21.15.61:0/4097434624 shutdown_connections 2022-07-20T03:13:24.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- 172.21.15.61:0/4097434624 >> 172.21.15.61:0/4097434624 conn(0x7fe02c0635b0 msgr2=0x7fe02c0b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:24.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- 172.21.15.61:0/4097434624 shutdown_connections 2022-07-20T03:13:24.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- 172.21.15.61:0/4097434624 wait complete. 2022-07-20T03:13:24.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 Processor -- start 2022-07-20T03:13:24.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- start start 2022-07-20T03:13:24.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c0049f0 0x7fe02c10f5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c113200 0x7fe02c1104b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 0x7fe02c116150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c1115d0 con 0x7fe02c115ca0 2022-07-20T03:13:24.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c115840 con 0x7fe00c0049f0 2022-07-20T03:13:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe02c116640 con 0x7fe02c113200 2022-07-20T03:13:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 0x7fe02c116150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 0x7fe02c116150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46332/0 (socket says 172.21.15.61:46332) 2022-07-20T03:13:24.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 -- 172.21.15.61:0/2519730525 learned_addr learned my addr 172.21.15.61:0/2519730525 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02affd700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c113200 0x7fe02c1104b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02b7fe700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c0049f0 0x7fe02c10f5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 -- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c113200 msgr2=0x7fe02c1104b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c113200 0x7fe02c1104b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 -- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c0049f0 msgr2=0x7fe02c10f5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c0049f0 0x7fe02c10f5a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:24.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 -- 172.21.15.61:0/2519730525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe02c1120f0 con 0x7fe02c115ca0 2022-07-20T03:13:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe02bfff700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 0x7fe02c116150 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fe02c0622d0 tx=0x7fe024006e80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe024020070 con 0x7fe02c115ca0 2022-07-20T03:13:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe024012d10 con 0x7fe02c115ca0 2022-07-20T03:13:24.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe02400f750 con 0x7fe02c115ca0 2022-07-20T03:13:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe02c112390 con 0x7fe02c115ca0 2022-07-20T03:13:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.881+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe02c1126f0 con 0x7fe02c115ca0 2022-07-20T03:13:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.885+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe02402c070 con 0x7fe02c115ca0 2022-07-20T03:13:24.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.885+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe02c0574e0 con 0x7fe02c115ca0 2022-07-20T03:13:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.885+0000 7fe028ff9700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053a60 0x7fe010055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.885+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(62..62 src has 1..62) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe02406e0e0 con 0x7fe02c115ca0 2022-07-20T03:13:24.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.885+0000 7fe02b7fe700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053a60 0x7fe010055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:24.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.885+0000 7fe02b7fe700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053a60 0x7fe010055f20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fe01400b000 tx=0x7fe014006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:24.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:24.889+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe024044ff0 con 0x7fe02c115ca0 2022-07-20T03:13:25.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.017+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fe02c13e020 con 0x7fe02c115ca0 2022-07-20T03:13:25.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.725+0000 7fe028ff9700 1 -- 172.21.15.61:0/2519730525 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v63) v1 ==== 185+0+0 (secure 0 0 0) 0x7fe024048610 con 0x7fe02c115ca0 2022-07-20T03:13:25.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053a60 msgr2=0x7fe010055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:25.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053a60 0x7fe010055f20 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fe01400b000 tx=0x7fe014006df0).stop 2022-07-20T03:13:25.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 msgr2=0x7fe02c116150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:25.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 0x7fe02c116150 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7fe02c0622d0 tx=0x7fe024006e80).stop 2022-07-20T03:13:25.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 shutdown_connections 2022-07-20T03:13:25.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe010053a60 0x7fe010055f20 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe02c115ca0 0x7fe02c116150 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe02c113200 0x7fe02c1104b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 --2- 172.21.15.61:0/2519730525 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe00c0049f0 0x7fe02c10f5a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 >> 172.21.15.61:0/2519730525 conn(0x7fe02c0635b0 msgr2=0x7fe02c0b50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:25.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 shutdown_connections 2022-07-20T03:13:25.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.729+0000 7fe031caf700 1 -- 172.21.15.61:0/2519730525 wait complete. 2022-07-20T03:13:25.742 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:13:25.748 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:13:25.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.889+0000 7f4920fcc700 1 Processor -- start 2022-07-20T03:13:25.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.889+0000 7f4920fcc700 1 -- start start 2022-07-20T03:13:25.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c0ac560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:25.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f491c0b4af0 0x7f491c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:25.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f491c0677d0 con 0x7f491c0afc90 2022-07-20T03:13:25.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f491c0b65e0 con 0x7f491c0b4af0 2022-07-20T03:13:25.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491c0aca50 con 0x7f491c0b6200 2022-07-20T03:13:25.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f491c0b4af0 0x7f491c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:25.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f491c0b4af0 0x7f491c13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46338/0 (socket says 172.21.15.61:46338) 2022-07-20T03:13:25.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 -- 172.21.15.61:0/4013613533 learned_addr learned my addr 172.21.15.61:0/4013613533 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:25.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f491bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f491c0afc90 0x7f491c0b11e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42938/0 (socket says 172.21.15.61:42938) 2022-07-20T03:13:25.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f491b7fe700 1 --2- 172.21.15.61:0/4013613533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c0ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:25.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 -- 172.21.15.61:0/4013613533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 msgr2=0x7f491c0ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:25.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 --2- 172.21.15.61:0/4013613533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c0ac560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 -- 172.21.15.61:0/4013613533 >> v1:172.21.15.61:6789/0 conn(0x7f491c0afc90 legacy=0x7f491c0b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:25.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f491affd700 1 -- 172.21.15.61:0/4013613533 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 708036274 0 0) 0x7f491c0677d0 con 0x7f491c0afc90 2022-07-20T03:13:25.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 -- 172.21.15.61:0/4013613533 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f491c067530 con 0x7f491c0b4af0 2022-07-20T03:13:25.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4913fff700 1 --2- 172.21.15.61:0/4013613533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0b4af0 0x7f491c13f950 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f4904002f70 tx=0x7f4904006f40).ready entity=mon.0 client_cookie=ac09e64e4bb93a34 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:25.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f491affd700 1 -- 172.21.15.61:0/4013613533 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490400dc70 con 0x7f491c0b4af0 2022-07-20T03:13:25.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f491affd700 1 -- 172.21.15.61:0/4013613533 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4904000e90 con 0x7f491c0b4af0 2022-07-20T03:13:25.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f491affd700 1 -- 172.21.15.61:0/4013613533 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490400be90 con 0x7f491c0b4af0 2022-07-20T03:13:25.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- 172.21.15.61:0/4013613533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0b4af0 msgr2=0x7f491c13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:25.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 --2- 172.21.15.61:0/4013613533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0b4af0 0x7f491c13f950 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7f4904002f70 tx=0x7f4904006f40).stop 2022-07-20T03:13:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- 172.21.15.61:0/4013613533 shutdown_connections 2022-07-20T03:13:25.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 --2- 172.21.15.61:0/4013613533 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0b4af0 0x7f491c13f950 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 --2- 172.21.15.61:0/4013613533 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c0ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- 172.21.15.61:0/4013613533 >> 172.21.15.61:0/4013613533 conn(0x7f491c063d10 msgr2=0x7f491c0b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:25.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- 172.21.15.61:0/4013613533 shutdown_connections 2022-07-20T03:13:25.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.893+0000 7f4920fcc700 1 -- 172.21.15.61:0/4013613533 wait complete. 2022-07-20T03:13:25.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 Processor -- start 2022-07-20T03:13:25.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 -- start start 2022-07-20T03:13:25.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 0x7f491c13bcc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:25.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f491c0b4af0 0x7f491c0b1600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:25.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c1358c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:25.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c1427b0 con 0x7f491c0afc90 2022-07-20T03:13:25.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f491c142e80 con 0x7f491c0b4af0 2022-07-20T03:13:25.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4920fcc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f491c143b40 con 0x7f491c0b6200 2022-07-20T03:13:25.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f491b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 0x7f491c13bcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:25.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f491b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 0x7f491c13bcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46342/0 (socket says 172.21.15.61:46342) 2022-07-20T03:13:25.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f491b7fe700 1 -- 172.21.15.61:0/3225508984 learned_addr learned my addr 172.21.15.61:0/3225508984 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:25.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f491bfff700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c1358c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:25.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.897+0000 7f4913fff700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f491c0b4af0 0x7f491c0b1600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:25.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f491b7fe700 1 -- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 msgr2=0x7f491c1358c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:25.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f491b7fe700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c1358c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f491b7fe700 1 -- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f491c0b4af0 msgr2=0x7f491c0b1600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:25.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f491b7fe700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f491c0b4af0 0x7f491c0b1600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:25.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f491b7fe700 1 -- 172.21.15.61:0/3225508984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f490400b460 con 0x7f491c0afc90 2022-07-20T03:13:25.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f4913fff700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f491c0b4af0 0x7f491c0b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:25.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f491b7fe700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 0x7f491c13bcc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f490c0030b0 tx=0x7f490c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:25.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490c011780 con 0x7f491c0afc90 2022-07-20T03:13:25.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f490c0118e0 con 0x7f491c0afc90 2022-07-20T03:13:25.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f491c1429e0 con 0x7f491c0afc90 2022-07-20T03:13:25.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f490c011c40 con 0x7f491c0afc90 2022-07-20T03:13:25.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f491c1437c0 con 0x7f491c0afc90 2022-07-20T03:13:25.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f491c1439b0 con 0x7f491c0afc90 2022-07-20T03:13:25.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f490c012420 con 0x7f491c0afc90 2022-07-20T03:13:25.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f49197fa700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4900053920 0x7f4900055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:25.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(63..63 src has 1..63) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f490c06bb80 con 0x7f491c0afc90 2022-07-20T03:13:25.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.901+0000 7f4913fff700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4900053920 0x7f4900055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:25.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.905+0000 7f4913fff700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4900053920 0x7f4900055de0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f490401c000 tx=0x7f491c05c3d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:25.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:25.909+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f490c042b40 con 0x7f491c0afc90 2022-07-20T03:13:26.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:26.041+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f491c0b5220 con 0x7f491c0afc90 2022-07-20T03:13:27.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.045+0000 7f49197fa700 1 -- 172.21.15.61:0/3225508984 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v64) v1 ==== 173+0+0 (secure 0 0 0) 0x7f490c046160 con 0x7f491c0afc90 2022-07-20T03:13:27.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4900053920 msgr2=0x7f4900055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4900053920 0x7f4900055de0 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f490401c000 tx=0x7f491c05c3d0).stop 2022-07-20T03:13:27.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 msgr2=0x7f491c13bcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 0x7f491c13bcc0 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7f490c0030b0 tx=0x7f490c008040).stop 2022-07-20T03:13:27.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 shutdown_connections 2022-07-20T03:13:27.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4900053920 0x7f4900055de0 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f491c0b4af0 0x7f491c0b1600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f491c0b6200 0x7f491c1358c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 --2- 172.21.15.61:0/3225508984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f491c0afc90 0x7f491c13bcc0 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 >> 172.21.15.61:0/3225508984 conn(0x7f491c063d10 msgr2=0x7f491c0b24f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:27.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 shutdown_connections 2022-07-20T03:13:27.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.053+0000 7f4920fcc700 1 -- 172.21.15.61:0/3225508984 wait complete. 2022-07-20T03:13:27.072 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:13:27.074 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:13:27.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c7acc700 1 Processor -- start 2022-07-20T03:13:27.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c7acc700 1 -- start start 2022-07-20T03:13:27.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c7acc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa3c01461d0 0x7fa3c0142690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c7acc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c005bfa0 con 0x7fa3c00b7ca0 2022-07-20T03:13:27.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c7acc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3c00b8080 con 0x7fa3c0066610 2022-07-20T03:13:27.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c7acc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3c00b2840 con 0x7fa3c01461d0 2022-07-20T03:13:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c5868700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa3c00b7ca0 0x7fa3c0065f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42952/0 (socket says 172.21.15.61:42952) 2022-07-20T03:13:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c5868700 1 -- 172.21.15.61:0/388441374 learned_addr learned my addr 172.21.15.61:0/388441374 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c6069700 1 --2- 172.21.15.61:0/388441374 >> v2:172.21.15.117:3300/0 conn(0x7fa3c01461d0 0x7fa3c0142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c4866700 1 -- 172.21.15.61:0/388441374 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3435688097 0 0) 0x7fa3c005bfa0 con 0x7fa3c00b7ca0 2022-07-20T03:13:27.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c4866700 1 -- 172.21.15.61:0/388441374 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa3a8003260 con 0x7fa3c00b7ca0 2022-07-20T03:13:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c6069700 1 -- 172.21.15.61:0/388441374 >> v1:172.21.15.117:6789/0 conn(0x7fa3c0066610 legacy=0x7fa3c0141f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c6069700 1 -- 172.21.15.61:0/388441374 >> v1:172.21.15.61:6789/0 conn(0x7fa3c00b7ca0 legacy=0x7fa3c0065f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c6069700 1 -- 172.21.15.61:0/388441374 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3c00b23c0 con 0x7fa3c01461d0 2022-07-20T03:13:27.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.201+0000 7fa3c6069700 1 --2- 172.21.15.61:0/388441374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c01461d0 0x7fa3c0142690 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fa3b80032f0 tx=0x7fa3b80091d0).ready entity=mon.1 client_cookie=582db75be20c75a4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c4866700 1 -- 172.21.15.61:0/388441374 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b800b6c0 con 0x7fa3c01461d0 2022-07-20T03:13:27.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c4866700 1 -- 172.21.15.61:0/388441374 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b800b820 con 0x7fa3c01461d0 2022-07-20T03:13:27.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c4866700 1 -- 172.21.15.61:0/388441374 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b800b990 con 0x7fa3c01461d0 2022-07-20T03:13:27.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 -- 172.21.15.61:0/388441374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c01461d0 msgr2=0x7fa3c0142690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/388441374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c01461d0 0x7fa3c0142690 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fa3b80032f0 tx=0x7fa3b80091d0).stop 2022-07-20T03:13:27.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 -- 172.21.15.61:0/388441374 shutdown_connections 2022-07-20T03:13:27.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/388441374 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c01461d0 0x7fa3c0142690 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 -- 172.21.15.61:0/388441374 >> 172.21.15.61:0/388441374 conn(0x7fa3c0065760 msgr2=0x7fa3c00b7250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:27.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 -- 172.21.15.61:0/388441374 shutdown_connections 2022-07-20T03:13:27.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.205+0000 7fa3c7acc700 1 -- 172.21.15.61:0/388441374 wait complete. 2022-07-20T03:13:27.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 Processor -- start 2022-07-20T03:13:27.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 -- start start 2022-07-20T03:13:27.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c0066610 0x7fa3c013cea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3c00b7ca0 0x7fa3c0145620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 0x7fa3c00b0370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c013df30 con 0x7fa3c01461d0 2022-07-20T03:13:27.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c013dbf0 con 0x7fa3c0066610 2022-07-20T03:13:27.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa3c013e160 con 0x7fa3c00b7ca0 2022-07-20T03:13:27.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 0x7fa3c00b0370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 0x7fa3c00b0370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46360/0 (socket says 172.21.15.61:46360) 2022-07-20T03:13:27.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 -- 172.21.15.61:0/848030217 learned_addr learned my addr 172.21.15.61:0/848030217 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:27.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c5868700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c0066610 0x7fa3c013cea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c5067700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3c00b7ca0 0x7fa3c0145620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 -- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3c00b7ca0 msgr2=0x7fa3c0145620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3c00b7ca0 0x7fa3c0145620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 -- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c0066610 msgr2=0x7fa3c013cea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c0066610 0x7fa3c013cea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 -- 172.21.15.61:0/848030217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3b8009470 con 0x7fa3c01461d0 2022-07-20T03:13:27.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c6069700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 0x7fa3c00b0370 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7fa3b80032f0 tx=0x7fa3b8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:27.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b800b6c0 con 0x7fa3c01461d0 2022-07-20T03:13:27.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3c013d870 con 0x7fa3c01461d0 2022-07-20T03:13:27.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3b8021670 con 0x7fa3c01461d0 2022-07-20T03:13:27.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3c00b0a00 con 0x7fa3c01461d0 2022-07-20T03:13:27.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.209+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa3b80217e0 con 0x7fa3c01461d0 2022-07-20T03:13:27.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.213+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3c013e320 con 0x7fa3c01461d0 2022-07-20T03:13:27.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.213+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa3b800f070 con 0x7fa3c01461d0 2022-07-20T03:13:27.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.213+0000 7fa3aeffd700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa3a0053ad0 0x7fa3a0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.213+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa3b806d050 con 0x7fa3c01461d0 2022-07-20T03:13:27.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.213+0000 7fa3c5868700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa3a0053ad0 0x7fa3a0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.213+0000 7fa3c5868700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa3a0053ad0 0x7fa3a0055f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa3b000d000 tx=0x7fa3b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:27.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.217+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa3b8044290 con 0x7fa3c01461d0 2022-07-20T03:13:27.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.345+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7fa3c00574e0 con 0x7fa3c01461d0 2022-07-20T03:13:27.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.349+0000 7fa3aeffd700 1 -- 172.21.15.61:0/848030217 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v9) v1 ==== 104+0+0 (secure 0 0 0) 0x7fa3b8015070 con 0x7fa3c01461d0 2022-07-20T03:13:27.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa3a0053ad0 msgr2=0x7fa3a0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa3a0053ad0 0x7fa3a0055f90 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fa3b000d000 tx=0x7fa3b0009040).stop 2022-07-20T03:13:27.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 msgr2=0x7fa3c00b0370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 0x7fa3c00b0370 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7fa3b80032f0 tx=0x7fa3b8016000).stop 2022-07-20T03:13:27.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 shutdown_connections 2022-07-20T03:13:27.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa3a0053ad0 0x7fa3a0055f90 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3c01461d0 0x7fa3c00b0370 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3c00b7ca0 0x7fa3c0145620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 --2- 172.21.15.61:0/848030217 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3c0066610 0x7fa3c013cea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 >> 172.21.15.61:0/848030217 conn(0x7fa3c0065760 msgr2=0x7fa3c01436c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 shutdown_connections 2022-07-20T03:13:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.353+0000 7fa3c7acc700 1 -- 172.21.15.61:0/848030217 wait complete. 2022-07-20T03:13:27.427 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout' 2022-07-20T03:13:27.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 Processor -- start 2022-07-20T03:13:27.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- start start 2022-07-20T03:13:27.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2d8c060c00 0x7f2d8c09cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2d8c061410 0x7f2d8c146330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2d8c05d4b0 con 0x7f2d8c146820 2022-07-20T03:13:27.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c09e590 con 0x7f2d8c061410 2022-07-20T03:13:27.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2d8c060fe0 con 0x7f2d8c060c00 2022-07-20T03:13:27.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2d8c061410 0x7f2d8c146330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2d8c061410 0x7f2d8c146330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46364/0 (socket says 172.21.15.61:46364) 2022-07-20T03:13:27.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 -- 172.21.15.61:0/1859889726 learned_addr learned my addr 172.21.15.61:0/1859889726 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:27.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d917e4700 1 --2- 172.21.15.61:0/1859889726 >> v2:172.21.15.117:3300/0 conn(0x7f2d8c060c00 0x7f2d8c09cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 -- 172.21.15.61:0/1859889726 >> v2:172.21.15.117:3300/0 conn(0x7f2d8c060c00 msgr2=0x7f2d8c09cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:27.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 --2- 172.21.15.61:0/1859889726 >> v2:172.21.15.117:3300/0 conn(0x7f2d8c060c00 0x7f2d8c09cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 -- 172.21.15.61:0/1859889726 >> v1:172.21.15.117:6789/0 conn(0x7f2d8c146820 legacy=0x7f2d8c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 -- 172.21.15.61:0/1859889726 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d8c0b39c0 con 0x7f2d8c061410 2022-07-20T03:13:27.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d90fe3700 1 --2- 172.21.15.61:0/1859889726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c061410 0x7f2d8c146330 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f2d7c002f70 tx=0x7f2d7c006eb0).ready entity=mon.0 client_cookie=3ab64b38c130a900 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:27.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d83fff700 1 -- 172.21.15.61:0/1859889726 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d7c00d5c0 con 0x7f2d8c061410 2022-07-20T03:13:27.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d83fff700 1 -- 172.21.15.61:0/1859889726 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d7c000e90 con 0x7f2d8c061410 2022-07-20T03:13:27.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d83fff700 1 -- 172.21.15.61:0/1859889726 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d7c00d950 con 0x7f2d8c061410 2022-07-20T03:13:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- 172.21.15.61:0/1859889726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c061410 msgr2=0x7f2d8c146330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 --2- 172.21.15.61:0/1859889726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c061410 0x7f2d8c146330 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7f2d7c002f70 tx=0x7f2d7c006eb0).stop 2022-07-20T03:13:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- 172.21.15.61:0/1859889726 shutdown_connections 2022-07-20T03:13:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 --2- 172.21.15.61:0/1859889726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c061410 0x7f2d8c146330 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 --2- 172.21.15.61:0/1859889726 >> v2:172.21.15.117:3300/0 conn(0x7f2d8c060c00 0x7f2d8c09cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- 172.21.15.61:0/1859889726 >> 172.21.15.61:0/1859889726 conn(0x7f2d8c0b5f30 msgr2=0x7f2d8c0b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:27.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- 172.21.15.61:0/1859889726 shutdown_connections 2022-07-20T03:13:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.577+0000 7f2d93a48700 1 -- 172.21.15.61:0/1859889726 wait complete. 2022-07-20T03:13:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 Processor -- start 2022-07-20T03:13:27.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 -- start start 2022-07-20T03:13:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 0x7f2d8c14a5a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c061410 0x7f2d8c144eb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d8c146820 0x7f2d8c1453a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c09b300 con 0x7f2d8c060c00 2022-07-20T03:13:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c09b5a0 con 0x7f2d8c061410 2022-07-20T03:13:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d93a48700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2d8c0b1630 con 0x7f2d8c146820 2022-07-20T03:13:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d917e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 0x7f2d8c14a5a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d917e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 0x7f2d8c14a5a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46370/0 (socket says 172.21.15.61:46370) 2022-07-20T03:13:27.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d917e4700 1 -- 172.21.15.61:0/703048005 learned_addr learned my addr 172.21.15.61:0/703048005 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d90fe3700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c061410 0x7f2d8c144eb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.581+0000 7f2d91fe5700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d8c146820 0x7f2d8c1453a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d917e4700 1 -- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d8c146820 msgr2=0x7f2d8c1453a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d917e4700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d8c146820 0x7f2d8c1453a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d917e4700 1 -- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c061410 msgr2=0x7f2d8c144eb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:27.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d917e4700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c061410 0x7f2d8c144eb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:27.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d917e4700 1 -- 172.21.15.61:0/703048005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2d7c00b1e0 con 0x7f2d8c060c00 2022-07-20T03:13:27.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d917e4700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 0x7f2d8c14a5a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f2d8800c000 tx=0x7f2d88006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:27.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d88010780 con 0x7f2d8c060c00 2022-07-20T03:13:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2d880108e0 con 0x7f2d8c060c00 2022-07-20T03:13:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2d88010c40 con 0x7f2d8c060c00 2022-07-20T03:13:27.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2d8c145890 con 0x7f2d8c060c00 2022-07-20T03:13:27.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2d8c0b1c70 con 0x7f2d8c060c00 2022-07-20T03:13:27.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2d8c09de60 con 0x7f2d8c060c00 2022-07-20T03:13:27.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.585+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2d8801c460 con 0x7f2d8c060c00 2022-07-20T03:13:27.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.589+0000 7f2d827fc700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2d74053ad0 0x7f2d74055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:27.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.589+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f2d8806bff0 con 0x7f2d8c060c00 2022-07-20T03:13:27.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.589+0000 7f2d90fe3700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2d74053ad0 0x7f2d74055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.589+0000 7f2d90fe3700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2d74053ad0 0x7f2d74055f90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f2d7c01c000 tx=0x7f2d7c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:27.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.593+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2d88043130 con 0x7f2d8c060c00 2022-07-20T03:13:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:27.765+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]} v 0) v1 -- 0x7f2d8c0574e0 con 0x7f2d8c060c00 2022-07-20T03:13:28.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.057+0000 7f2d827fc700 1 -- 172.21.15.61:0/703048005 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestCacheDrop.test_drop_cache_command_timeout"]}]=0 v279) v1 ==== 147+0+0 (secure 0 0 0) 0x7f2d88046750 con 0x7f2d8c060c00 2022-07-20T03:13:28.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2d74053ad0 msgr2=0x7f2d74055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:28.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2d74053ad0 0x7f2d74055f90 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f2d7c01c000 tx=0x7f2d7c015000).stop 2022-07-20T03:13:28.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 msgr2=0x7f2d8c14a5a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:28.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 0x7f2d8c14a5a0 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7f2d8800c000 tx=0x7f2d88006df0).stop 2022-07-20T03:13:28.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 shutdown_connections 2022-07-20T03:13:28.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2d74053ad0 0x7f2d74055f90 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2d8c061410 0x7f2d8c144eb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2d8c146820 0x7f2d8c1453a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 --2- 172.21.15.61:0/703048005 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2d8c060c00 0x7f2d8c14a5a0 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 >> 172.21.15.61:0/703048005 conn(0x7f2d8c0b5f30 msgr2=0x7f2d8c0b8480 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:28.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 shutdown_connections 2022-07-20T03:13:28.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.061+0000 7f2d93a48700 1 -- 172.21.15.61:0/703048005 wait complete. 2022-07-20T03:13:28.091 INFO:tasks.cephfs_test_runner:test_drop_cache_command_timeout (tasks.cephfs.test_misc.TestCacheDrop) ... ok 2022-07-20T03:13:28.092 INFO:tasks.cephfs_test_runner:Starting test: test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:13:28.093 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-20T03:13:28.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 Processor -- start 2022-07-20T03:13:28.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- start start 2022-07-20T03:13:28.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c0b1e00 0x7fd59c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:28.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd59c05c3f0 0x7fd59c0ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:28.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd59c05d480 con 0x7fd59c05bd90 2022-07-20T03:13:28.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd59c0b21e0 con 0x7fd59c05c3f0 2022-07-20T03:13:28.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c05cf40 con 0x7fd59c0b1e00 2022-07-20T03:13:28.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd59c05c3f0 0x7fd59c0ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:28.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd59c05c3f0 0x7fd59c0ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46380/0 (socket says 172.21.15.61:46380) 2022-07-20T03:13:28.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 -- 172.21.15.61:0/2347189040 learned_addr learned my addr 172.21.15.61:0/2347189040 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:28.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59b7fe700 1 --2- 172.21.15.61:0/2347189040 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c0b1e00 0x7fd59c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:28.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 -- 172.21.15.61:0/2347189040 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c0b1e00 msgr2=0x7fd59c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:28.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 --2- 172.21.15.61:0/2347189040 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c0b1e00 0x7fd59c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 -- 172.21.15.61:0/2347189040 >> v1:172.21.15.117:6789/0 conn(0x7fd59c05bd90 legacy=0x7fd59c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:28.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 -- 172.21.15.61:0/2347189040 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd59c05d1e0 con 0x7fd59c05c3f0 2022-07-20T03:13:28.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59a7fc700 1 -- 172.21.15.61:0/2347189040 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1901798009 0 0) 0x7fd59c05d480 con 0x7fd59c05bd90 2022-07-20T03:13:28.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59affd700 1 --2- 172.21.15.61:0/2347189040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c05c3f0 0x7fd59c0ac7e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fd58c0032f0 tx=0x7fd58c009290).ready entity=mon.0 client_cookie=39ee649d3c666cc0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:28.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59a7fc700 1 -- 172.21.15.61:0/2347189040 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c00b8a0 con 0x7fd59c05c3f0 2022-07-20T03:13:28.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59a7fc700 1 -- 172.21.15.61:0/2347189040 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c00ba00 con 0x7fd59c05c3f0 2022-07-20T03:13:28.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd59a7fc700 1 -- 172.21.15.61:0/2347189040 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c00bb70 con 0x7fd59c05c3f0 2022-07-20T03:13:28.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- 172.21.15.61:0/2347189040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c05c3f0 msgr2=0x7fd59c0ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:28.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/2347189040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c05c3f0 0x7fd59c0ac7e0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7fd58c0032f0 tx=0x7fd58c009290).stop 2022-07-20T03:13:28.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- 172.21.15.61:0/2347189040 shutdown_connections 2022-07-20T03:13:28.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/2347189040 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c05c3f0 0x7fd59c0ac7e0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/2347189040 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c0b1e00 0x7fd59c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- 172.21.15.61:0/2347189040 >> 172.21.15.61:0/2347189040 conn(0x7fd59c0611c0 msgr2=0x7fd59c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:28.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- 172.21.15.61:0/2347189040 shutdown_connections 2022-07-20T03:13:28.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.257+0000 7fd5a1d90700 1 -- 172.21.15.61:0/2347189040 wait complete. 2022-07-20T03:13:28.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 Processor -- start 2022-07-20T03:13:28.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 -- start start 2022-07-20T03:13:28.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c05c3f0 0x7fd59c149b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:28.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 0x7fd59c063e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:28.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c065e20 0x7fd59c064340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:28.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c065340 con 0x7fd59c065e20 2022-07-20T03:13:28.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b59b0 con 0x7fd59c0b1e00 2022-07-20T03:13:28.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd5a1d90700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd59c0b5c50 con 0x7fd59c05c3f0 2022-07-20T03:13:28.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c05c3f0 0x7fd59c149b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:28.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 0x7fd59c063e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:28.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c065e20 0x7fd59c064340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:28.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 0x7fd59c063e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56066/0 (socket says 172.21.15.61:56066) 2022-07-20T03:13:28.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 -- 172.21.15.61:0/1426964118 learned_addr learned my addr 172.21.15.61:0/1426964118 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:28.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c05c3f0 0x7fd59c149b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33056/0 (socket says 172.21.15.61:33056) 2022-07-20T03:13:28.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 -- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c05c3f0 msgr2=0x7fd59c149b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:28.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c05c3f0 0x7fd59c149b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 -- 172.21.15.61:0/1426964118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c065e20 msgr2=0x7fd59c064340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:28.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c065e20 0x7fd59c064340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:28.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 -- 172.21.15.61:0/1426964118 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd58c009530 con 0x7fd59c0b1e00 2022-07-20T03:13:28.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59bfff700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c065e20 0x7fd59c064340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:28.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.261+0000 7fd59affd700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 0x7fd59c063e50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd58c0032f0 tx=0x7fd58c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:28.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c00bb70 con 0x7fd59c0b1e00 2022-07-20T03:13:28.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd58c021630 con 0x7fd59c0b1e00 2022-07-20T03:13:28.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd59c063a80 con 0x7fd59c0b1e00 2022-07-20T03:13:28.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd58c00b8a0 con 0x7fd59c0b1e00 2022-07-20T03:13:28.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd59c0649d0 con 0x7fd59c0b1e00 2022-07-20T03:13:28.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd59c0b4ff0 con 0x7fd59c0b1e00 2022-07-20T03:13:28.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd58c027020 con 0x7fd59c0b1e00 2022-07-20T03:13:28.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd598ff9700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd574053ad0 0x7fd574055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:28.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.265+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd58c06d1b0 con 0x7fd59c0b1e00 2022-07-20T03:13:28.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.269+0000 7fd59b7fe700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd574053ad0 0x7fd574055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:28.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.269+0000 7fd59b7fe700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd574053ad0 0x7fd574055f90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fd58400b000 tx=0x7fd584006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:28.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.277+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd58c0443f0 con 0x7fd59c0b1e00 2022-07-20T03:13:28.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:28.449+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7fd59c0b5e10 con 0x7fd59c0b1e00 2022-07-20T03:13:29.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.061+0000 7fd598ff9700 1 -- 172.21.15.61:0/1426964118 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v280) v1 ==== 142+0+0 (secure 0 0 0) 0x7fd58c015020 con 0x7fd59c0b1e00 2022-07-20T03:13:29.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd574053ad0 msgr2=0x7fd574055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd574053ad0 0x7fd574055f90 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7fd58400b000 tx=0x7fd584006df0).stop 2022-07-20T03:13:29.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 msgr2=0x7fd59c063e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 0x7fd59c063e50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fd58c0032f0 tx=0x7fd58c016000).stop 2022-07-20T03:13:29.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 shutdown_connections 2022-07-20T03:13:29.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd574053ad0 0x7fd574055f90 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd59c0b1e00 0x7fd59c063e50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd59c065e20 0x7fd59c064340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 --2- 172.21.15.61:0/1426964118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd59c05c3f0 0x7fd59c149b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 >> 172.21.15.61:0/1426964118 conn(0x7fd59c05bd90 msgr2=0x7fd59c1461e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:29.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 shutdown_connections 2022-07-20T03:13:29.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.065+0000 7fd5a1d90700 1 -- 172.21.15.61:0/1426964118 wait complete. 2022-07-20T03:13:29.100 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:13:29.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 Processor -- start 2022-07-20T03:13:29.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 -- start start 2022-07-20T03:13:29.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f441462a0 0x7f9f44146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:29.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f440988b0 0x7f9f44146b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:29.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f440afdb0 con 0x7f9f441422c0 2022-07-20T03:13:29.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f440b0050 con 0x7f9f440988b0 2022-07-20T03:13:29.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4405fcd0 con 0x7f9f441462a0 2022-07-20T03:13:29.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f48b58700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9f441422c0 0x7f9f44060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:42994/0 (socket says 172.21.15.61:42994) 2022-07-20T03:13:29.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f437fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f440988b0 0x7f9f44146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:29.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f48b58700 1 -- 172.21.15.61:0/3308069294 learned_addr learned my addr 172.21.15.61:0/3308069294 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:29.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f43fff700 1 --2- 172.21.15.61:0/3308069294 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f441462a0 0x7f9f44146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:29.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f437fe700 1 -- 172.21.15.61:0/3308069294 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f441462a0 msgr2=0x7f9f44146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:29.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f437fe700 1 --2- 172.21.15.61:0/3308069294 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f441462a0 0x7f9f44146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f437fe700 1 -- 172.21.15.61:0/3308069294 >> v1:172.21.15.61:6789/0 conn(0x7f9f441422c0 legacy=0x7f9f44060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f437fe700 1 -- 172.21.15.61:0/3308069294 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f440afae0 con 0x7f9f440988b0 2022-07-20T03:13:29.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f437fe700 1 --2- 172.21.15.61:0/3308069294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f440988b0 0x7f9f44146b70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f9f38002f70 tx=0x7f9f38006df0).ready entity=mon.0 client_cookie=5a3f49bf7909b306 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:29.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f42ffd700 1 -- 172.21.15.61:0/3308069294 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3800d570 con 0x7f9f440988b0 2022-07-20T03:13:29.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f42ffd700 1 -- 172.21.15.61:0/3308069294 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f38000e90 con 0x7f9f440988b0 2022-07-20T03:13:29.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f42ffd700 1 -- 172.21.15.61:0/3308069294 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3800d8c0 con 0x7f9f440988b0 2022-07-20T03:13:29.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/3308069294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f440988b0 msgr2=0x7f9f44146b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.265+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/3308069294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f440988b0 0x7f9f44146b70 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f9f38002f70 tx=0x7f9f38006df0).stop 2022-07-20T03:13:29.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/3308069294 shutdown_connections 2022-07-20T03:13:29.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/3308069294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f440988b0 0x7f9f44146b70 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/3308069294 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f441462a0 0x7f9f44146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/3308069294 >> 172.21.15.61:0/3308069294 conn(0x7f9f440b5f00 msgr2=0x7f9f440b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:29.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/3308069294 shutdown_connections 2022-07-20T03:13:29.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/3308069294 wait complete. 2022-07-20T03:13:29.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 Processor -- start 2022-07-20T03:13:29.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- start start 2022-07-20T03:13:29.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f440988b0 0x7f9f4409a900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:29.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f441422c0 0x7f9f4409e4f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:29.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 0x7f9f4409ed20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:29.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4409e9e0 con 0x7f9f441462a0 2022-07-20T03:13:29.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f44099f20 con 0x7f9f441422c0 2022-07-20T03:13:29.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f4a5bb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f4409a1c0 con 0x7f9f440988b0 2022-07-20T03:13:29.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 0x7f9f4409ed20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:29.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 0x7f9f4409ed20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46402/0 (socket says 172.21.15.61:46402) 2022-07-20T03:13:29.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 -- 172.21.15.61:0/960201887 learned_addr learned my addr 172.21.15.61:0/960201887 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:29.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f43fff700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f440988b0 0x7f9f4409a900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:29.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f437fe700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f441422c0 0x7f9f4409e4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:29.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 -- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f440988b0 msgr2=0x7f9f4409a900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f440988b0 0x7f9f4409a900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 -- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f441422c0 msgr2=0x7f9f4409e4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f441422c0 0x7f9f4409e4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 -- 172.21.15.61:0/960201887 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f3800b180 con 0x7f9f441462a0 2022-07-20T03:13:29.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f437fe700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f441422c0 0x7f9f4409e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:29.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f48b58700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 0x7f9f4409ed20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f9f44060050 tx=0x7f9f3c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:29.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.269+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c005ba0 con 0x7f9f441462a0 2022-07-20T03:13:29.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f3c005d00 con 0x7f9f441462a0 2022-07-20T03:13:29.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f4409a460 con 0x7f9f441462a0 2022-07-20T03:13:29.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f3c0072c0 con 0x7f9f441462a0 2022-07-20T03:13:29.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f4409e130 con 0x7f9f441462a0 2022-07-20T03:13:29.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f4409a5f0 con 0x7f9f441462a0 2022-07-20T03:13:29.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f3c01a060 con 0x7f9f441462a0 2022-07-20T03:13:29.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f417fa700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f240539b0 0x7f9f24055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:29.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9f3c06b7a0 con 0x7f9f441462a0 2022-07-20T03:13:29.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.273+0000 7f9f43fff700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f240539b0 0x7f9f24055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:29.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.277+0000 7f9f43fff700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f240539b0 0x7f9f24055e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f9f3400b000 tx=0x7f9f34006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:29.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.277+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f3c0428d0 con 0x7f9f441462a0 2022-07-20T03:13:29.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.405+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9f44067ea0 con 0x7f9f441462a0 2022-07-20T03:13:29.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f417fa700 1 -- 172.21.15.61:0/960201887 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v10)=0 v10) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9f3c018070 con 0x7f9f441462a0 2022-07-20T03:13:29.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f240539b0 msgr2=0x7f9f24055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f240539b0 0x7f9f24055e70 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f9f3400b000 tx=0x7f9f34006df0).stop 2022-07-20T03:13:29.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 msgr2=0x7f9f4409ed20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:29.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 0x7f9f4409ed20 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7f9f44060050 tx=0x7f9f3c007f40).stop 2022-07-20T03:13:29.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 shutdown_connections 2022-07-20T03:13:29.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f240539b0 0x7f9f24055e70 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f441422c0 0x7f9f4409e4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f441462a0 0x7f9f4409ed20 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 --2- 172.21.15.61:0/960201887 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f440988b0 0x7f9f4409a900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:29.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 >> 172.21.15.61:0/960201887 conn(0x7f9f440b5f00 msgr2=0x7f9f440b62e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:29.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 shutdown_connections 2022-07-20T03:13:29.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:29.413+0000 7f9f4a5bb700 1 -- 172.21.15.61:0/960201887 wait complete. 2022-07-20T03:13:29.503 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:13:29.504 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:29.534 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:29.584 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:29.671 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:29.731 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:13:29.799 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:13:29.800 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:29.801 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:29.809 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:29.896 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:29.906 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:13:29.907 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:13:29.908 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:29.909 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:29.938 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:29.955 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:29.968 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:13:29.968 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:13:29.969 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:30.011 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:30.012 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:13:30.067 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:13:30.068 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:13:30.068 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:13:30.069 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:13:30.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 Processor -- start 2022-07-20T03:13:30.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- start start 2022-07-20T03:13:30.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff0340b20b0 0x7ff034147bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:30.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340ae440 con 0x7ff034146ca0 2022-07-20T03:13:30.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0340ad7c0 con 0x7ff0341480a0 2022-07-20T03:13:30.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff0340ada60 con 0x7ff0340b20b0 2022-07-20T03:13:30.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff038bb7700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff034146ca0 0x7ff03405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43008/0 (socket says 172.21.15.61:43008) 2022-07-20T03:13:30.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff038bb7700 1 -- 172.21.15.61:0/983066199 learned_addr learned my addr 172.21.15.61:0/983066199 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:30.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff033fff700 1 --2- 172.21.15.61:0/983066199 >> v2:172.21.15.117:3300/0 conn(0x7ff0340b20b0 0x7ff034147bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:30.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff032ffd700 1 -- 172.21.15.61:0/983066199 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2299333123 0 0) 0x7ff0340ae440 con 0x7ff034146ca0 2022-07-20T03:13:30.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff033fff700 1 -- 172.21.15.61:0/983066199 >> v1:172.21.15.117:6789/0 conn(0x7ff0341480a0 legacy=0x7ff034146590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:30.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff033fff700 1 -- 172.21.15.61:0/983066199 >> v1:172.21.15.61:6789/0 conn(0x7ff034146ca0 legacy=0x7ff03405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:30.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff033fff700 1 -- 172.21.15.61:0/983066199 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0340add00 con 0x7ff0340b20b0 2022-07-20T03:13:30.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff033fff700 1 --2- 172.21.15.61:0/983066199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0340b20b0 0x7ff034147bb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff01c002f70 tx=0x7ff01c006df0).ready entity=mon.1 client_cookie=51dcf4bd3b1d11d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:30.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff032ffd700 1 -- 172.21.15.61:0/983066199 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff01c00d610 con 0x7ff0340b20b0 2022-07-20T03:13:30.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff032ffd700 1 -- 172.21.15.61:0/983066199 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff01c000e90 con 0x7ff0340b20b0 2022-07-20T03:13:30.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff032ffd700 1 -- 172.21.15.61:0/983066199 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff01c00d960 con 0x7ff0340b20b0 2022-07-20T03:13:30.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- 172.21.15.61:0/983066199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0340b20b0 msgr2=0x7ff034147bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:30.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 --2- 172.21.15.61:0/983066199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0340b20b0 0x7ff034147bb0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7ff01c002f70 tx=0x7ff01c006df0).stop 2022-07-20T03:13:30.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- 172.21.15.61:0/983066199 shutdown_connections 2022-07-20T03:13:30.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 --2- 172.21.15.61:0/983066199 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0340b20b0 0x7ff034147bb0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:30.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- 172.21.15.61:0/983066199 >> 172.21.15.61:0/983066199 conn(0x7ff0340b72c0 msgr2=0x7ff0340b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:30.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.229+0000 7ff03a61a700 1 -- 172.21.15.61:0/983066199 shutdown_connections 2022-07-20T03:13:30.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 -- 172.21.15.61:0/983066199 wait complete. 2022-07-20T03:13:30.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 Processor -- start 2022-07-20T03:13:30.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 -- start start 2022-07-20T03:13:30.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0340b20b0 0x7ff034113c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:30.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff034146ca0 0x7ff0341485c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:30.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 0x7ff034148ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:30.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0340b0c50 con 0x7ff0341480a0 2022-07-20T03:13:30.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff034114480 con 0x7ff034146ca0 2022-07-20T03:13:30.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff03a61a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff034156860 con 0x7ff0340b20b0 2022-07-20T03:13:30.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.233+0000 7ff038bb7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 0x7ff034148ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:30.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff033fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0340b20b0 0x7ff034113c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:30.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 0x7ff034148ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46416/0 (socket says 172.21.15.61:46416) 2022-07-20T03:13:30.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 -- 172.21.15.61:0/1027331343 learned_addr learned my addr 172.21.15.61:0/1027331343 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:30.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff0337fe700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff034146ca0 0x7ff0341485c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:30.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 -- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0340b20b0 msgr2=0x7ff034113c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:30.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0340b20b0 0x7ff034113c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:30.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 -- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff034146ca0 msgr2=0x7ff0341485c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:30.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff034146ca0 0x7ff0341485c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:30.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 -- 172.21.15.61:0/1027331343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff01c00b180 con 0x7ff0341480a0 2022-07-20T03:13:30.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff033fff700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0340b20b0 0x7ff034113c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:30.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff038bb7700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 0x7ff034148ab0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7ff024000ed0 tx=0x7ff024007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff024007110 con 0x7ff0341480a0 2022-07-20T03:13:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff024007270 con 0x7ff0341480a0 2022-07-20T03:13:30.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff024005030 con 0x7ff0341480a0 2022-07-20T03:13:30.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff034156b00 con 0x7ff0341480a0 2022-07-20T03:13:30.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0341141c0 con 0x7ff0341480a0 2022-07-20T03:13:30.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.237+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff034156fa0 con 0x7ff0341480a0 2022-07-20T03:13:30.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.241+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff024005190 con 0x7ff0341480a0 2022-07-20T03:13:30.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.241+0000 7ff0317fa700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff00c053ad0 0x7ff00c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:30.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.241+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff02406b390 con 0x7ff0341480a0 2022-07-20T03:13:30.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.241+0000 7ff033fff700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff00c053ad0 0x7ff00c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:30.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.241+0000 7ff033fff700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff00c053ad0 0x7ff00c055f90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff01c01c000 tx=0x7ff01c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:30.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.245+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff034156fa0 con 0x7ff0341480a0 2022-07-20T03:13:30.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.369+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7ff034149140 con 0x7ff0341480a0 2022-07-20T03:13:30.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff0317fa700 1 -- 172.21.15.61:0/1027331343 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v64) v1 ==== 81+0+28876 (secure 0 0 0) 0x7ff0240425d0 con 0x7ff0341480a0 2022-07-20T03:13:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:30.396 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:13:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 64, 2022-07-20T03:13:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:13:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:13:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:13:27.041057+0000", 2022-07-20T03:13:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:13:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:13:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:13:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:13:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:13:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:13:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:13:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:13:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:13:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:13:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:13:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:13:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:13:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:13:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 12, 2022-07-20T03:13:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:13:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:13:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:13:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:13:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:13:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:13:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:13:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:13:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:13:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:13:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:13:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:13:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:13:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:13:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:13:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:13:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:13:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:13:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:13:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:13:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:13:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:13:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:13:30.414 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:13:30.415 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:13:30.416 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:13:30.416 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:13:30.417 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:13:30.417 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.418 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:13:30.420 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:13:30.420 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:13:30.420 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:13:30.421 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:13:30.421 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:13:30.422 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:13:30.422 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:13:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:13:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:13:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:13:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:13:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:13:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:13:30.425 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:13:30.446 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:13:30.447 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:13:30.447 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:13:30.448 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:13:30.448 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:13:30.449 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:13:30.451 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:13:30.451 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:13:30.452 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:13:30.452 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:13:30.452 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:13:30.453 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:13:30.453 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.454 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:13:30.454 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:13:30.455 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:13:30.456 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:13:30.457 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:13:30.457 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:13:30.457 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:13:30.504 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:13:30.505 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:13:30.505 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:13:30.505 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:13:30.505 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:13:30.506 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:13:30.506 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.506 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:13:30.507 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:13:30.507 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.507 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.508 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.508 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:13:30.508 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:13:30.509 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:13:30.509 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:13:30.509 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:13:30.509 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:13:30.510 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:13:30.510 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:13:30.510 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:13:30.511 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:13:30.511 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:13:30.511 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:13:30.511 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:13:30.512 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:13:30.512 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:13:30.512 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:13:30.512 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:13:30.513 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:13:30.513 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:13:30.513 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:13:30.519 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:13:30.520 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:13:30.520 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:13:30.520 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.520 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:13:30.521 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:13:30.521 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:13:30.521 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:13:30.532 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:13:30.539 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:13:30.540 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:13:30.540 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:13:30.541 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:13:30.542 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:13:30.542 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:13:30.543 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:13:30.543 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:13:30.545 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:13:30.545 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:13:30.546 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:13:30.546 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:13:30.546 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:13:30.547 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:13:30.547 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:13:30.548 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:13:30.549 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:13:30.553 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:13:30.554 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:13:30.554 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:13:30.566 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:13:30.567 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:13:30.567 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.568 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:13:30.568 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:13:30.568 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:13:30.569 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:13:30.569 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:13:30.569 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:13:30.570 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:13:30.571 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:13:30.571 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:13:30.572 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:13:30.572 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:13:30.572 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:13:30.615 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:13:30.615 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:13:30.616 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.616 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.617 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:30.652 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:13:30.682 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.682 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:13:30.691 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:13:30.692 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:30.692 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:30.693 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:30.722 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:30.740 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:30.740 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:30.741 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:30.741 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:30.742 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:30.742 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:30.742 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:30.743 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.743 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.744 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.744 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:13:30.744 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.745 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.745 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.758 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.759 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:13:30.759 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.759 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.773 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.774 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.774 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:30.775 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.775 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.776 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.776 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:13:30.777 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.777 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.777 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.794 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.794 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:13:30.795 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.796 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.796 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.797 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.797 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:30.797 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.797 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.797 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.798 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:13:30.798 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.798 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.798 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.799 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.799 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:13:30.799 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.799 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.800 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.800 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.800 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:30.800 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.801 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.801 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:13:30.801 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.801 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.802 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.802 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.802 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:13:30.803 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:13:30.803 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.803 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.803 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.828 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:13:30.828 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:13:30.828 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:13:30.828 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:13:30.829 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:30.829 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:30.829 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:30.829 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.830 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.830 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.831 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:13:30.831 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:13:30.832 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:30.832 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:30.833 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:30.833 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:30.833 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:30.834 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:30.834 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:30.835 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:30.835 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:30.835 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:30.835 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:30.836 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.836 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.836 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.836 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:13:30.836 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.837 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.837 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.838 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.838 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:13:30.838 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.839 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.839 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.839 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.839 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:30.840 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.840 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.840 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.840 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:13:30.841 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.841 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.841 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.841 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.842 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:13:30.842 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.842 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.842 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.843 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.843 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:30.844 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.844 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.845 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.845 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:13:30.846 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.847 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.848 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:13:30.849 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.850 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.850 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.851 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:30.852 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.852 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.853 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.853 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:13:30.854 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.854 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.854 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.855 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.855 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:13:30.855 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:13:30.855 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.856 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.856 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.856 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:13:30.856 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:13:30.857 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:13:30.857 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:13:30.857 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:30.857 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:30.858 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:30.858 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.858 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.858 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.859 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:13:30.859 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:13:30.859 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:30.860 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:30.860 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:30.860 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:30.861 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:30.861 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:30.861 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:30.861 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 58, 2022-07-20T03:13:30.862 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:30.862 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:30.862 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:30.863 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.863 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.863 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.863 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:13:30.864 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.864 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.864 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.864 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.865 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:13:30.865 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.865 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.865 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.866 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.866 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:30.866 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.866 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.866 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.867 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:13:30.867 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.867 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.867 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.868 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.868 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:13:30.868 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.877 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.877 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.878 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:30.878 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.878 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.879 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:13:30.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.880 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.881 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:13:30.881 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.916 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.916 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.916 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.916 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:30.917 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.917 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.917 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:13:30.917 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.918 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.918 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:13:30.919 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:13:30.919 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.920 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.920 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.920 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:13:30.921 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:13:30.921 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:13:30.921 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:13:30.921 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:30.922 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:30.925 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:30.925 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.925 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.926 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.926 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:13:30.926 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:13:30.926 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:30.927 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:30.927 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:30.927 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:30.927 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:30.928 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:30.928 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:30.928 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 49, 2022-07-20T03:13:30.928 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:30.928 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:30.929 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:30.943 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.943 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.944 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.944 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:13:30.945 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.945 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.952 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.953 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.953 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:13:30.953 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.953 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.954 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.954 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.954 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:30.954 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.955 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.955 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.955 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:13:30.955 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.956 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.956 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.956 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.956 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:13:30.957 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.957 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.957 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.957 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.958 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:30.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.958 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.958 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.959 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:13:30.959 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.959 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.959 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.960 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.960 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:13:30.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.961 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.961 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.961 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.962 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:30.962 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.962 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.962 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.963 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:13:30.963 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.963 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.963 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.964 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.964 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:13:30.964 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:13:30.964 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.965 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.965 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.965 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:13:30.965 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:13:30.966 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:13:30.966 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:13:30.966 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:30.968 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:30.968 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:30.968 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.968 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.969 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.974 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:13:30.974 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:13:30.975 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:30.975 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:30.975 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:30.975 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:30.976 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:30.976 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:30.976 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:30.976 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:30.977 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:30.977 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:30.977 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:30.977 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.977 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.978 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.978 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:13:30.979 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:30.979 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.979 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.980 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.980 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:13:30.989 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:30.990 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.990 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:30.990 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.990 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:30.991 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:30.992 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.992 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:30.992 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:13:30.993 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:30.993 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:30.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:30.994 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:30.994 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:13:30.995 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:30.995 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:30.997 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.004 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:31.005 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.005 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.005 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:13:31.006 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:31.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.007 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.007 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.007 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:13:31.008 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:31.008 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.008 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.008 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.009 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:31.009 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.010 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:13:31.010 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:31.015 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.017 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:13:31.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:13:31.017 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.018 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.018 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.019 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:13:31.027 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:13:31.027 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:13:31.027 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:13:31.028 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:31.028 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:31.028 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:31.035 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.035 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:13:31.036 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:13:31.036 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:31.036 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:31.036 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:31.037 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:31.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:31.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:31.037 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:31.038 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 58, 2022-07-20T03:13:31.042 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:31.043 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:31.043 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:31.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:13:31.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:13:31.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.046 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.046 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.047 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.047 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:31.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:13:31.048 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.049 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:13:31.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.050 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.050 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.051 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:31.051 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.051 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:13:31.053 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.054 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.054 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.054 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:13:31.054 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.055 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.055 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.055 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.056 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:31.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:13:31.057 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.058 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.058 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:13:31.059 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:13:31.059 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.059 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.060 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:13:31.060 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:13:31.061 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:13:31.061 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:13:31.062 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:31.062 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:31.062 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:31.062 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.063 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.063 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:13:31.063 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:13:31.074 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:31.075 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:31.075 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:31.075 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:31.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:31.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:31.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:31.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:13:31.076 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:31.077 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:31.077 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:31.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.077 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.106 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.110 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:13:31.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.111 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:13:31.112 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.112 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.112 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.112 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.113 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:31.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.113 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.113 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:13:31.115 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.121 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.122 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:13:31.122 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.122 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.123 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:31.123 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.123 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:13:31.124 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.124 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.125 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.125 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:13:31.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.137 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.138 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.138 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.138 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:31.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.139 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:13:31.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.140 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.140 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.141 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:13:31.141 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:13:31.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.141 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.142 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:13:31.142 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:13:31.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:13:31.143 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:13:31.143 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:31.144 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:31.144 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:31.144 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.145 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.145 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:13:31.145 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:13:31.146 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:13:31.146 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:13:31.146 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:13:31.146 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:13:31.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:13:31.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:13:31.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:13:31.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 58, 2022-07-20T03:13:31.147 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:13:31.148 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:13:31.148 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:13:31.148 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.149 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:13:31.149 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.150 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.150 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.150 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:13:31.151 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.151 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.152 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.152 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:13:31.153 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.154 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:13:31.155 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.155 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.155 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.156 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:13:31.156 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.156 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.156 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.157 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:13:31.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.157 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.157 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.158 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:13:31.158 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.158 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.158 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.159 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.159 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:13:31.160 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.161 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.169 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.170 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:13:31.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:13:31.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:13:31.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:13:31.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.173 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.173 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:13:31.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:13:31.198 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:13:31.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.199 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.207 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:13:31.207 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:13:31.207 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:13:31.208 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:13:31.208 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:13:31.208 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:13:31.208 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:13:31.209 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:31.209 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.209 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:31.210 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:13:31.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.210 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:13:31.211 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.211 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.212 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.212 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.212 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:13:31.213 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.214 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:13:31.215 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.215 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.215 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.215 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.215 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:13:31.216 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.217 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:13:31.217 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.217 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.217 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.217 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.276 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.277 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:13:31.277 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.279 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:13:31.279 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.280 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.296 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.297 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.297 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.297 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:13:31.297 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.298 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.298 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:13:31.298 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.299 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.299 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.299 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.299 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.300 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:13:31.300 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.300 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.300 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:13:31.301 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.301 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.301 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.301 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.302 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.302 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:13:31.302 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.303 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.303 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:13:31.303 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.303 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.304 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.304 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.304 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:13:31.305 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:31.305 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:13:31.305 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:13:31.306 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:13:31.306 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:13:31.306 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:13:31.307 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:13:31.307 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:13:31.307 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:13:31.307 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.307 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:13:31.308 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:13:31.308 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:13:31.323 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:13:31.324 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:13:31.324 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:13:31.324 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2218834214": "2022-07-21T03:13:22.551162+0000", 2022-07-20T03:13:31.325 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2218834214": "2022-07-21T03:13:22.551162+0000" 2022-07-20T03:13:31.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.325 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:13:31.325 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:13:31.326 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:13:31.326 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:13:31.326 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:13:31.326 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:13:31.327 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:13:31.327 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:13:31.327 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:31.327 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:31.328 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:13:31.328 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:13:31.328 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:13:31.328 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:13:31.338 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:13:31.339 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:13:31.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff00c053ad0 msgr2=0x7ff00c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff00c053ad0 0x7ff00c055f90 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7ff01c01c000 tx=0x7ff01c015000).stop 2022-07-20T03:13:31.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 msgr2=0x7ff034148ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 0x7ff034148ab0 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7ff024000ed0 tx=0x7ff024007a40).stop 2022-07-20T03:13:31.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 shutdown_connections 2022-07-20T03:13:31.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff00c053ad0 0x7ff00c055f90 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff034146ca0 0x7ff0341485c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0341480a0 0x7ff034148ab0 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 --2- 172.21.15.61:0/1027331343 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0340b20b0 0x7ff034113c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 >> 172.21.15.61:0/1027331343 conn(0x7ff0340b72c0 msgr2=0x7ff0341441d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:31.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 shutdown_connections 2022-07-20T03:13:31.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:30.373+0000 7ff03a61a700 1 -- 172.21.15.61:0/1027331343 wait complete. 2022-07-20T03:13:31.391 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:31.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 Processor -- start 2022-07-20T03:13:31.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 -- start start 2022-07-20T03:13:31.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe4c40b5650 0x7fe4c40b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c405bd90 0x7fe4c4142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4c40aea30 con 0x7fe4c40b1200 2022-07-20T03:13:31.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe4c4142970 con 0x7fe4c40b5650 2022-07-20T03:13:31.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.505+0000 7fe4c3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c40aec60 con 0x7fe4c405bd90 2022-07-20T03:13:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c37fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c405bd90 0x7fe4c4142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe4c40b5650 0x7fe4c40b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe4c40b5650 0x7fe4c40b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56100/0 (socket says 172.21.15.61:56100) 2022-07-20T03:13:31.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4147090075 learned_addr learned my addr 172.21.15.61:0/4147090075 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c405bd90 msgr2=0x7fe4c4142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 --2- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c405bd90 0x7fe4c4142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4147090075 >> v1:172.21.15.117:6789/0 conn(0x7fe4c40b1200 legacy=0x7fe4c405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4147090075 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c40af180 con 0x7fe4c40b5650 2022-07-20T03:13:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c2ffd700 1 --2- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c40b5650 0x7fe4c40b5a30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe4bc0032f0 tx=0x7fe4bc009290).ready entity=mon.1 client_cookie=7691574da5b1a6c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c1ffb700 1 -- 172.21.15.61:0/4147090075 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4bc00b8a0 con 0x7fe4c40b5650 2022-07-20T03:13:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c1ffb700 1 -- 172.21.15.61:0/4147090075 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc00ba00 con 0x7fe4c40b5650 2022-07-20T03:13:31.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c1ffb700 1 -- 172.21.15.61:0/4147090075 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4bc00bb70 con 0x7fe4c40b5650 2022-07-20T03:13:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c40b5650 msgr2=0x7fe4c40b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c40b5650 0x7fe4c40b5a30 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7fe4bc0032f0 tx=0x7fe4bc009290).stop 2022-07-20T03:13:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4147090075 shutdown_connections 2022-07-20T03:13:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c405bd90 0x7fe4c4142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4147090075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c40b5650 0x7fe4c40b5a30 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4147090075 >> 172.21.15.61:0/4147090075 conn(0x7fe4c40634e0 msgr2=0x7fe4c40b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4147090075 shutdown_connections 2022-07-20T03:13:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.509+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4147090075 wait complete. 2022-07-20T03:13:31.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 Processor -- start 2022-07-20T03:13:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 -- start start 2022-07-20T03:13:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 0x7fe4c413dad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c40b1200 0x7fe4c4144690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4c40b5650 0x7fe4c4140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c4142540 con 0x7fe4c40b5650 2022-07-20T03:13:31.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c4063b80 con 0x7fe4c405bd90 2022-07-20T03:13:31.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe4c413eb60 con 0x7fe4c40b1200 2022-07-20T03:13:31.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 0x7fe4c413dad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 0x7fe4c413dad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56106/0 (socket says 172.21.15.61:56106) 2022-07-20T03:13:31.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4205029999 learned_addr learned my addr 172.21.15.61:0/4205029999 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:31.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c27fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c40b1200 0x7fe4c4144690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c37fe700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4c40b5650 0x7fe4c4140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c40b1200 msgr2=0x7fe4c4144690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c40b1200 0x7fe4c4144690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4205029999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4c40b5650 msgr2=0x7fe4c4140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4c40b5650 0x7fe4c4140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 -- 172.21.15.61:0/4205029999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4bc009530 con 0x7fe4c405bd90 2022-07-20T03:13:31.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c27fc700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c40b1200 0x7fe4c4144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:31.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c2ffd700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 0x7fe4c413dad0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fe4bc0032f0 tx=0x7fe4bc016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:31.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4bc00b8a0 con 0x7fe4c405bd90 2022-07-20T03:13:31.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe4bc021440 con 0x7fe4c405bd90 2022-07-20T03:13:31.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4c4140a30 con 0x7fe4c405bd90 2022-07-20T03:13:31.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe4bc0217a0 con 0x7fe4c405bd90 2022-07-20T03:13:31.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.513+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4c413d0f0 con 0x7fe4c405bd90 2022-07-20T03:13:31.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.517+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4c4063d80 con 0x7fe4c405bd90 2022-07-20T03:13:31.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.517+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe4bc029070 con 0x7fe4c405bd90 2022-07-20T03:13:31.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.517+0000 7fe4bbfff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe4a8053ad0 0x7fe4a8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.517+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe4bc06ce10 con 0x7fe4c405bd90 2022-07-20T03:13:31.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.517+0000 7fe4c27fc700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe4a8053ad0 0x7fe4a8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.517+0000 7fe4c27fc700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe4a8053ad0 0x7fe4a8055f90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fe4ac00d000 tx=0x7fe4ac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:31.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.525+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe4c4063d80 con 0x7fe4c405bd90 2022-07-20T03:13:31.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe4c4063d50 con 0x7fe4c405bd90 2022-07-20T03:13:31.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4bbfff700 1 -- 172.21.15.61:0/4205029999 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1212 (secure 0 0 0) 0x7fe4bc015030 con 0x7fe4c405bd90 2022-07-20T03:13:31.727 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:31.728 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2549659477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2549659477},{"type":"v1","addr":"172.21.15.61:6833","nonce":2549659477}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":36}],"filesystems":[]} 2022-07-20T03:13:31.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe4a8053ad0 msgr2=0x7fe4a8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe4a8053ad0 0x7fe4a8055f90 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7fe4ac00d000 tx=0x7fe4ac009040).stop 2022-07-20T03:13:31.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 msgr2=0x7fe4c413dad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:31.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 0x7fe4c413dad0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7fe4bc0032f0 tx=0x7fe4bc016000).stop 2022-07-20T03:13:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 shutdown_connections 2022-07-20T03:13:31.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe4a8053ad0 0x7fe4a8055f90 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4c40b5650 0x7fe4c4140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4c40b1200 0x7fe4c4144690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 --2- 172.21.15.61:0/4205029999 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe4c405bd90 0x7fe4c413dad0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 >> 172.21.15.61:0/4205029999 conn(0x7fe4c40634e0 msgr2=0x7fe4c4148470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:31.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 shutdown_connections 2022-07-20T03:13:31.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.693+0000 7fe4c3fff700 1 -- 172.21.15.61:0/4205029999 wait complete. 2022-07-20T03:13:31.802 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 37 2022-07-20T03:13:31.803 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:31.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.921+0000 7f3dd359e700 1 Processor -- start 2022-07-20T03:13:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- start start 2022-07-20T03:13:31.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40b0570 0x7f3dd40b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3dd40aea10 0x7f3dd4141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:31.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3dd4142750 con 0x7f3dd41422c0 2022-07-20T03:13:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3dd4143b10 con 0x7f3dd40aea10 2022-07-20T03:13:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd41433d0 con 0x7f3dd40b0570 2022-07-20T03:13:31.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd259c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40b0570 0x7f3dd40b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3dd40aea10 0x7f3dd4141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:31.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd2d9d700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f3dd41422c0 0x7f3dd40ad350 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:51540/0 (socket says 172.21.15.61:51540) 2022-07-20T03:13:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3dd40aea10 0x7f3dd4141dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56114/0 (socket says 172.21.15.61:56114) 2022-07-20T03:13:31.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd2d9d700 1 -- 172.21.15.61:0/1490050463 learned_addr learned my addr 172.21.15.61:0/1490050463 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:31.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40b0570 msgr2=0x7f3dd40b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:31.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 --2- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40b0570 0x7f3dd40b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:31.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/1490050463 >> v1:172.21.15.117:6789/0 conn(0x7f3dd41422c0 legacy=0x7f3dd40ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/1490050463 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dd4143670 con 0x7f3dd40aea10 2022-07-20T03:13:32.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd259c700 1 --2- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40b0570 0x7f3dd40b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:32.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd1d9b700 1 --2- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd40aea10 0x7f3dd4141dd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3dc8002f70 tx=0x7f3dc8006df0).ready entity=mon.1 client_cookie=a2f579e6b858e479 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:32.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd159a700 1 -- 172.21.15.61:0/1490050463 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc800d570 con 0x7f3dd40aea10 2022-07-20T03:13:32.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd159a700 1 -- 172.21.15.61:0/1490050463 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc8000e90 con 0x7f3dd40aea10 2022-07-20T03:13:32.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd159a700 1 -- 172.21.15.61:0/1490050463 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc800d8c0 con 0x7f3dd40aea10 2022-07-20T03:13:32.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd40aea10 msgr2=0x7f3dd4141dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 --2- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd40aea10 0x7f3dd4141dd0 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f3dc8002f70 tx=0x7f3dc8006df0).stop 2022-07-20T03:13:32.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- 172.21.15.61:0/1490050463 shutdown_connections 2022-07-20T03:13:32.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 --2- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd40aea10 0x7f3dd4141dd0 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 --2- 172.21.15.61:0/1490050463 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40b0570 0x7f3dd40b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.925+0000 7f3dd359e700 1 -- 172.21.15.61:0/1490050463 >> 172.21.15.61:0/1490050463 conn(0x7f3dd40b9360 msgr2=0x7f3dd40b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 -- 172.21.15.61:0/1490050463 shutdown_connections 2022-07-20T03:13:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 -- 172.21.15.61:0/1490050463 wait complete. 2022-07-20T03:13:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 Processor -- start 2022-07-20T03:13:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 -- start start 2022-07-20T03:13:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40aea10 0x7f3dd40b4dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 0x7f3dd4114af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd41422c0 0x7f3dd4112550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd4112fe0 con 0x7f3dd40b0570 2022-07-20T03:13:32.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b4780 con 0x7f3dd41422c0 2022-07-20T03:13:32.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.929+0000 7f3dd359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3dd40b4a20 con 0x7f3dd40aea10 2022-07-20T03:13:32.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 0x7f3dd4114af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 0x7f3dd4114af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46440/0 (socket says 172.21.15.61:46440) 2022-07-20T03:13:32.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/4016470650 learned_addr learned my addr 172.21.15.61:0/4016470650 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:32.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd2d9d700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd41422c0 0x7f3dd4112550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd259c700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40aea10 0x7f3dd40b4dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40aea10 msgr2=0x7f3dd40b4dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40aea10 0x7f3dd40b4dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd41422c0 msgr2=0x7f3dd4112550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd41422c0 0x7f3dd4112550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 -- 172.21.15.61:0/4016470650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3dc400b180 con 0x7f3dd40b0570 2022-07-20T03:13:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd1d9b700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 0x7f3dd4114af0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f3dc8002f70 tx=0x7f3dc8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:32.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc8016070 con 0x7f3dd40b0570 2022-07-20T03:13:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3dc800d8f0 con 0x7f3dd40b0570 2022-07-20T03:13:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3dc800b180 con 0x7f3dd40b0570 2022-07-20T03:13:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3dc8013de0 con 0x7f3dd40b0570 2022-07-20T03:13:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3dd4112c60 con 0x7f3dd40b0570 2022-07-20T03:13:32.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3dd40b4be0 con 0x7f3dd40b0570 2022-07-20T03:13:32.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.933+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3dc8025030 con 0x7f3dd40b0570 2022-07-20T03:13:32.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.937+0000 7f3dc37fe700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3db0053a00 0x7f3db0055ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.937+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(64..64 src has 1..64) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f3dc806cbd0 con 0x7f3dd40b0570 2022-07-20T03:13:32.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.937+0000 7f3dd259c700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3db0053a00 0x7f3db0055ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.937+0000 7f3dd259c700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3db0053a00 0x7f3db0055ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3dc400b000 tx=0x7f3dc4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:32.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:31.941+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3dc8043e10 con 0x7f3dd40b0570 2022-07-20T03:13:32.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.101+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f3dd40b1010 con 0x7f3dd40b0570 2022-07-20T03:13:32.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.101+0000 7f3dc37fe700 1 -- 172.21.15.61:0/4016470650 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 37 v37) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f3dc8023020 con 0x7f3dd40b0570 2022-07-20T03:13:32.114 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:32.114 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":37,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5420,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/3619012033","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":3619012033},{"type":"v1","addr":"172.21.15.117:6835","nonce":3619012033}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":33},{"gid":6201,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2549659477","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2549659477},{"type":"v1","addr":"172.21.15.61:6833","nonce":2549659477}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":36}],"filesystems":[]} 2022-07-20T03:13:32.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3db0053a00 msgr2=0x7f3db0055ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3db0053a00 0x7f3db0055ec0 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f3dc400b000 tx=0x7f3dc4006df0).stop 2022-07-20T03:13:32.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 msgr2=0x7f3dd4114af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 0x7f3dd4114af0 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7f3dc8002f70 tx=0x7f3dc8014000).stop 2022-07-20T03:13:32.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 shutdown_connections 2022-07-20T03:13:32.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3db0053a00 0x7f3db0055ec0 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3dd41422c0 0x7f3dd4112550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3dd40b0570 0x7f3dd4114af0 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 --2- 172.21.15.61:0/4016470650 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3dd40aea10 0x7f3dd40b4dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 >> 172.21.15.61:0/4016470650 conn(0x7f3dd40b9360 msgr2=0x7f3dd40b8bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:32.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 shutdown_connections 2022-07-20T03:13:32.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.105+0000 7f3dd359e700 1 -- 172.21.15.61:0/4016470650 wait complete. 2022-07-20T03:13:32.141 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 37 2022-07-20T03:13:32.142 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:13:32.142 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:13:32.187 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:13:32.187 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:13:32.188 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:13:32.188 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:13:32.189 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:13:32.208 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:13:32.223 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:13:32.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:13:32.225 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:13:32.226 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:13:32.236 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:13:32.237 DEBUG:teuthology.parallel:result is None 2022-07-20T03:13:32.282 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:13:32.283 DEBUG:teuthology.parallel:result is None 2022-07-20T03:13:32.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:13:32.341 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:13:32.355 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:13:32.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 Processor -- start 2022-07-20T03:13:32.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 -- start start 2022-07-20T03:13:32.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea940acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fea94146770 0x7fea9414c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea940b46e0 con 0x7fea940b4f80 2022-07-20T03:13:32.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea940ad990 con 0x7fea94146770 2022-07-20T03:13:32.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a880700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b4b10 con 0x7fea940b5860 2022-07-20T03:13:32.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea940acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea940acae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33132/0 (socket says 172.21.15.61:33132) 2022-07-20T03:13:32.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 -- 172.21.15.61:0/1918005726 learned_addr learned my addr 172.21.15.61:0/1918005726 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:32.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a07f700 1 --2- 172.21.15.61:0/1918005726 >> v2:172.21.15.117:3300/0 conn(0x7fea94146770 0x7fea9414c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 -- 172.21.15.61:0/1918005726 >> v2:172.21.15.117:3300/0 conn(0x7fea94146770 msgr2=0x7fea9414c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 --2- 172.21.15.61:0/1918005726 >> v2:172.21.15.117:3300/0 conn(0x7fea94146770 0x7fea9414c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 -- 172.21.15.61:0/1918005726 >> v1:172.21.15.117:6789/0 conn(0x7fea940b4f80 legacy=0x7fea94146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 -- 172.21.15.61:0/1918005726 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea940ad3d0 con 0x7fea940b5860 2022-07-20T03:13:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9a07f700 1 --2- 172.21.15.61:0/1918005726 >> v2:172.21.15.117:3300/0 conn(0x7fea94146770 0x7fea9414c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9987e700 1 --2- 172.21.15.61:0/1918005726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea940acae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fea90002f70 tx=0x7fea90006eb0).ready entity=mon.2 client_cookie=a7a159e085a49588 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9887c700 1 -- 172.21.15.61:0/1918005726 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9000d570 con 0x7fea940b5860 2022-07-20T03:13:32.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9887c700 1 -- 172.21.15.61:0/1918005726 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea90000e90 con 0x7fea940b5860 2022-07-20T03:13:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.405+0000 7fea9887c700 1 -- 172.21.15.61:0/1918005726 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea9000d900 con 0x7fea940b5860 2022-07-20T03:13:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 -- 172.21.15.61:0/1918005726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 msgr2=0x7fea940acae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 --2- 172.21.15.61:0/1918005726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea940acae0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fea90002f70 tx=0x7fea90006eb0).stop 2022-07-20T03:13:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 -- 172.21.15.61:0/1918005726 shutdown_connections 2022-07-20T03:13:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 --2- 172.21.15.61:0/1918005726 >> v2:172.21.15.117:3300/0 conn(0x7fea94146770 0x7fea9414c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 --2- 172.21.15.61:0/1918005726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea940acae0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 -- 172.21.15.61:0/1918005726 >> 172.21.15.61:0/1918005726 conn(0x7fea940b5e10 msgr2=0x7fea940b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:32.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 -- 172.21.15.61:0/1918005726 shutdown_connections 2022-07-20T03:13:32.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 -- 172.21.15.61:0/1918005726 wait complete. 2022-07-20T03:13:32.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 Processor -- start 2022-07-20T03:13:32.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.409+0000 7fea9a880700 1 -- start start 2022-07-20T03:13:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea940b4f80 0x7fea94112930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea94151ed0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 0x7fea941523c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b3110 con 0x7fea94146770 2022-07-20T03:13:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea94115240 con 0x7fea940b4f80 2022-07-20T03:13:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea940b1e10 con 0x7fea940b5860 2022-07-20T03:13:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 0x7fea941523c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 0x7fea941523c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46470/0 (socket says 172.21.15.61:46470) 2022-07-20T03:13:32.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 -- 172.21.15.61:0/21679709 learned_addr learned my addr 172.21.15.61:0/21679709 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:32.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9987e700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea940b4f80 0x7fea94112930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9907d700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea94151ed0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 -- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 msgr2=0x7fea94151ed0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea94151ed0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 -- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea940b4f80 msgr2=0x7fea94112930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea940b4f80 0x7fea94112930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 -- 172.21.15.61:0/21679709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea8c009470 con 0x7fea94146770 2022-07-20T03:13:32.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9987e700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea940b4f80 0x7fea94112930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a07f700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 0x7fea941523c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fea8c00c000 tx=0x7fea8c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea8c00c850 con 0x7fea94146770 2022-07-20T03:13:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea8c00c9b0 con 0x7fea94146770 2022-07-20T03:13:32.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea8c00cd10 con 0x7fea94146770 2022-07-20T03:13:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea9000b1e0 con 0x7fea94146770 2022-07-20T03:13:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea940b1bb0 con 0x7fea94146770 2022-07-20T03:13:32.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.413+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea940b2060 con 0x7fea94146770 2022-07-20T03:13:32.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.417+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fea8c022070 con 0x7fea94146770 2022-07-20T03:13:32.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.417+0000 7fea82ffd700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea74053a40 0x7fea74055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:32.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.417+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(66..66 src has 1..66) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fea8c06c3d0 con 0x7fea94146770 2022-07-20T03:13:32.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.417+0000 7fea9987e700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea74053a40 0x7fea74055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:32.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.417+0000 7fea9987e700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea74053a40 0x7fea74055f00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fea9001a000 tx=0x7fea90015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:32.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.421+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea8c043610 con 0x7fea94146770 2022-07-20T03:13:32.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:32.561+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fea940af0f0 con 0x7fea94146770 2022-07-20T03:13:33.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.401+0000 7fea82ffd700 1 -- 172.21.15.61:0/21679709 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v67) v1 ==== 97+0+0 (secure 0 0 0) 0x7fea8c046c30 con 0x7fea94146770 2022-07-20T03:13:33.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea74053a40 msgr2=0x7fea74055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:33.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea74053a40 0x7fea74055f00 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fea9001a000 tx=0x7fea90015000).stop 2022-07-20T03:13:33.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 msgr2=0x7fea941523c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:33.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 0x7fea941523c0 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7fea8c00c000 tx=0x7fea8c0091d0).stop 2022-07-20T03:13:33.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 shutdown_connections 2022-07-20T03:13:33.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea74053a40 0x7fea74055f00 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea94146770 0x7fea941523c0 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea940b5860 0x7fea94151ed0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 --2- 172.21.15.61:0/21679709 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea940b4f80 0x7fea94112930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 >> 172.21.15.61:0/21679709 conn(0x7fea940b5e10 msgr2=0x7fea940ae530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:33.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 shutdown_connections 2022-07-20T03:13:33.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.405+0000 7fea9a880700 1 -- 172.21.15.61:0/21679709 wait complete. 2022-07-20T03:13:33.420 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:13:33.421 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:13:33.428 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:13:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 Processor -- start 2022-07-20T03:13:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 -- start start 2022-07-20T03:13:33.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4b700aea20 0x7f4b70140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b700b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b700aee00 con 0x7f4b700675f0 2022-07-20T03:13:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b70144470 con 0x7f4b700aea20 2022-07-20T03:13:33.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.537+0000 7f4b775d3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70144710 con 0x7f4b700b1a70 2022-07-20T03:13:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b7536f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4b700aea20 0x7f4b70140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b74b6e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4b700675f0 0x7f4b70140f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43074/0 (socket says 172.21.15.61:43074) 2022-07-20T03:13:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b74b6e700 1 -- 172.21.15.61:0/1757092509 learned_addr learned my addr 172.21.15.61:0/1757092509 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b75b70700 1 --2- 172.21.15.61:0/1757092509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b700b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b67fff700 1 -- 172.21.15.61:0/1757092509 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2116065646 0 0) 0x7f4b700aee00 con 0x7f4b700675f0 2022-07-20T03:13:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b67fff700 1 -- 172.21.15.61:0/1757092509 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b54003260 con 0x7f4b700675f0 2022-07-20T03:13:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b7536f700 1 -- 172.21.15.61:0/1757092509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 msgr2=0x7f4b700b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b7536f700 1 --2- 172.21.15.61:0/1757092509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b700b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b7536f700 1 -- 172.21.15.61:0/1757092509 >> v1:172.21.15.61:6789/0 conn(0x7f4b700675f0 legacy=0x7f4b70140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b7536f700 1 -- 172.21.15.61:0/1757092509 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b701449b0 con 0x7f4b700aea20 2022-07-20T03:13:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b7536f700 1 --2- 172.21.15.61:0/1757092509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b70140a90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f4b700b1750 tx=0x7f4b60006df0).ready entity=mon.0 client_cookie=d7eba93281b72e11 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b67fff700 1 -- 172.21.15.61:0/1757092509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b6000d570 con 0x7f4b700aea20 2022-07-20T03:13:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b67fff700 1 -- 172.21.15.61:0/1757092509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60000e90 con 0x7f4b700aea20 2022-07-20T03:13:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b67fff700 1 -- 172.21.15.61:0/1757092509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b6000d8c0 con 0x7f4b700aea20 2022-07-20T03:13:33.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 -- 172.21.15.61:0/1757092509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 msgr2=0x7f4b70140a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:33.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1757092509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b70140a90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f4b700b1750 tx=0x7f4b60006df0).stop 2022-07-20T03:13:33.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 -- 172.21.15.61:0/1757092509 shutdown_connections 2022-07-20T03:13:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1757092509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b700b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1757092509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b70140a90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 -- 172.21.15.61:0/1757092509 >> 172.21.15.61:0/1757092509 conn(0x7f4b700611c0 msgr2=0x7f4b70062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 -- 172.21.15.61:0/1757092509 shutdown_connections 2022-07-20T03:13:33.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 -- 172.21.15.61:0/1757092509 wait complete. 2022-07-20T03:13:33.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 Processor -- start 2022-07-20T03:13:33.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.541+0000 7f4b775d3700 1 -- start start 2022-07-20T03:13:33.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b7013b390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:33.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b7013b880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:33.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b7013f140 0x7f4b701538a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:33.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701381f0 con 0x7f4b700aea20 2022-07-20T03:13:33.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b70138420 con 0x7f4b7013f140 2022-07-20T03:13:33.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b701386c0 con 0x7f4b700b1a70 2022-07-20T03:13:33.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b7013b390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:33.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b7013b390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46482/0 (socket says 172.21.15.61:46482) 2022-07-20T03:13:33.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 -- 172.21.15.61:0/1264817246 learned_addr learned my addr 172.21.15.61:0/1264817246 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:33.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b74b6e700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b7013b880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:33.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b75b70700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b7013f140 0x7f4b701538a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:33.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 -- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 msgr2=0x7f4b7013b880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:33.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b7013b880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 -- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b7013f140 msgr2=0x7f4b701538a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:33.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b7013f140 0x7f4b701538a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:33.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 -- 172.21.15.61:0/1264817246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b6000b180 con 0x7f4b700aea20 2022-07-20T03:13:33.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b7536f700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b7013b390 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f4b700b1750 tx=0x7f4b60014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:33.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b60013de0 con 0x7f4b700aea20 2022-07-20T03:13:33.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b60002f70 con 0x7f4b700aea20 2022-07-20T03:13:33.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b70138ab0 con 0x7f4b700aea20 2022-07-20T03:13:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b6001e5f0 con 0x7f4b700aea20 2022-07-20T03:13:33.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.545+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b70153fb0 con 0x7f4b700aea20 2022-07-20T03:13:33.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.549+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b70144e50 con 0x7f4b700aea20 2022-07-20T03:13:33.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.549+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4b60025030 con 0x7f4b700aea20 2022-07-20T03:13:33.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.549+0000 7f4b667fc700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b58053920 0x7f4b58055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:33.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.549+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4b6006c950 con 0x7f4b700aea20 2022-07-20T03:13:33.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.549+0000 7f4b74b6e700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b58053920 0x7f4b58055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:33.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.549+0000 7f4b74b6e700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b58053920 0x7f4b58055de0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4b6c00d000 tx=0x7f4b6c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:33.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.553+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b60043c20 con 0x7f4b700aea20 2022-07-20T03:13:33.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.721+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f4b70138880 con 0x7f4b700aea20 2022-07-20T03:13:33.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.721+0000 7f4b667fc700 1 -- 172.21.15.61:0/1264817246 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v24) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f4b60047200 con 0x7f4b700aea20 2022-07-20T03:13:33.745 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:33.745 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:13:33.746 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:13:33.746 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.746 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:13:33.747 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:13:33.747 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.748 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:13:33.806 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.807 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.807 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.808 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.808 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.809 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:13:33.810 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:13:33.810 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:13:33.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.812 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.812 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.813 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.813 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.814 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:13:33.816 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:13:33.816 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.818 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.818 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.819 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.819 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.820 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.820 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.821 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:13:33.843 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:13:33.844 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.844 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.845 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.845 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.846 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.846 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.846 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.847 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:13:33.847 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:13:33.848 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.848 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.849 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.859 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.859 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.860 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.860 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.860 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:13:33.861 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:13:33.861 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.864 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.864 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.870 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.871 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.871 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.872 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.872 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:13:33.872 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:13:33.873 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:33.873 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:33.873 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:33.874 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:33.874 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:33.875 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:33.875 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:33.876 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:13:33.876 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:13:34.011 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.044 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:34.044 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:34.045 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:34.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.047 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:13:34.047 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:13:34.047 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.070 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:34.071 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:34.071 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:34.072 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.109 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:13:34.109 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:13:34.138 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.138 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:34.139 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:34.139 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:34.139 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.145 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:13:34.146 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:13:34.147 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.147 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:13:34.148 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:13:34.149 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:13:34.149 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.150 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.150 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.151 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:13:34.151 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:13:34.151 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.152 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:13:34.152 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:13:34.167 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:13:34.168 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.169 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.170 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:13:34.170 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:13:34.170 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.171 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:13:34.171 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:13:34.172 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:13:34.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.173 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.174 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:13:34.180 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:13:34.180 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.181 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:13:34.181 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:13:34.184 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:13:34.185 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:34.185 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.185 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.193 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.194 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:13:34.201 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:13:34.202 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.202 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:13:34.202 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:13:34.203 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:34.203 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.204 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:34.204 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:34.205 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:13:34.205 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:13:34.206 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:13:34.206 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:13:34.206 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:13:34.227 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:13:34.227 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.228 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:34.229 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:13:34.230 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:13:34.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b58053920 msgr2=0x7f4b58055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b58053920 0x7f4b58055de0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f4b6c00d000 tx=0x7f4b6c009040).stop 2022-07-20T03:13:34.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 msgr2=0x7f4b7013b390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b7013b390 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f4b700b1750 tx=0x7f4b60014000).stop 2022-07-20T03:13:34.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 shutdown_connections 2022-07-20T03:13:34.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b58053920 0x7f4b58055de0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b7013f140 0x7f4b701538a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b700b1a70 0x7f4b7013b880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 --2- 172.21.15.61:0/1264817246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b700aea20 0x7f4b7013b390 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 >> 172.21.15.61:0/1264817246 conn(0x7f4b700675f0 msgr2=0x7f4b701395e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 shutdown_connections 2022-07-20T03:13:34.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:33.725+0000 7f4b775d3700 1 -- 172.21.15.61:0/1264817246 wait complete. 2022-07-20T03:13:34.290 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:13:34.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 Processor -- start 2022-07-20T03:13:34.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 -- start start 2022-07-20T03:13:34.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc3d00b1ef0 0x7fc3d0062040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc3d0061760 0x7fc3d00ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3d00b22d0 con 0x7fc3d0098ae0 2022-07-20T03:13:34.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00abe50 con 0x7fc3d0061760 2022-07-20T03:13:34.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d71dd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3d00ac0f0 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc3d0061760 0x7fc3d00ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc3d0061760 0x7fc3d00ac830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46492/0 (socket says 172.21.15.61:46492) 2022-07-20T03:13:34.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 -- 172.21.15.61:0/4222021136 learned_addr learned my addr 172.21.15.61:0/4222021136 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:34.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d59da700 1 --1- 172.21.15.61:0/4222021136 >> v1:172.21.15.61:6789/0 conn(0x7fc3d0098ae0 0x7fc3d0061050 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43092/0 (socket says 172.21.15.61:43092) 2022-07-20T03:13:34.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d61db700 1 --2- 172.21.15.61:0/4222021136 >> v2:172.21.15.117:3300/0 conn(0x7fc3d00b1ef0 0x7fc3d0062040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 -- 172.21.15.61:0/4222021136 >> v2:172.21.15.117:3300/0 conn(0x7fc3d00b1ef0 msgr2=0x7fc3d0062040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:34.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 --2- 172.21.15.61:0/4222021136 >> v2:172.21.15.117:3300/0 conn(0x7fc3d00b1ef0 0x7fc3d0062040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 -- 172.21.15.61:0/4222021136 >> v1:172.21.15.61:6789/0 conn(0x7fc3d0098ae0 legacy=0x7fc3d0061050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d51d9700 1 -- 172.21.15.61:0/4222021136 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2575276992 0 0) 0x7fc3d00b22d0 con 0x7fc3d0098ae0 2022-07-20T03:13:34.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 -- 172.21.15.61:0/4222021136 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3d00ac390 con 0x7fc3d0061760 2022-07-20T03:13:34.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.401+0000 7fc3d69dc700 1 --2- 172.21.15.61:0/4222021136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d0061760 0x7fc3d00ac830 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fc3c0002f70 tx=0x7fc3c0006eb0).ready entity=mon.0 client_cookie=a88f3d7f05e27dcd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:34.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d51d9700 1 -- 172.21.15.61:0/4222021136 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3c000d5c0 con 0x7fc3d0061760 2022-07-20T03:13:34.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d51d9700 1 -- 172.21.15.61:0/4222021136 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0000e90 con 0x7fc3d0061760 2022-07-20T03:13:34.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d51d9700 1 -- 172.21.15.61:0/4222021136 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3c000d910 con 0x7fc3d0061760 2022-07-20T03:13:34.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 -- 172.21.15.61:0/4222021136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d0061760 msgr2=0x7fc3d00ac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/4222021136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d0061760 0x7fc3d00ac830 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7fc3c0002f70 tx=0x7fc3c0006eb0).stop 2022-07-20T03:13:34.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 -- 172.21.15.61:0/4222021136 shutdown_connections 2022-07-20T03:13:34.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/4222021136 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d0061760 0x7fc3d00ac830 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/4222021136 >> v2:172.21.15.117:3300/0 conn(0x7fc3d00b1ef0 0x7fc3d0062040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 -- 172.21.15.61:0/4222021136 >> 172.21.15.61:0/4222021136 conn(0x7fc3d00b2ca0 msgr2=0x7fc3d00b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:34.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 -- 172.21.15.61:0/4222021136 shutdown_connections 2022-07-20T03:13:34.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 -- 172.21.15.61:0/4222021136 wait complete. 2022-07-20T03:13:34.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 Processor -- start 2022-07-20T03:13:34.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.405+0000 7fc3d71dd700 1 -- start start 2022-07-20T03:13:34.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3d0061760 0x7fc3d0143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d0098ae0 0x7fc3d0157ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 0x7fc3d014a2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d015aa80 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d0149ba0 con 0x7fc3d0098ae0 2022-07-20T03:13:34.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3d0149e40 con 0x7fc3d0061760 2022-07-20T03:13:34.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 0x7fc3d014a2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 0x7fc3d014a2e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46500/0 (socket says 172.21.15.61:46500) 2022-07-20T03:13:34.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 -- 172.21.15.61:0/3156503874 learned_addr learned my addr 172.21.15.61:0/3156503874 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:34.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d59da700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d0098ae0 0x7fc3d0157ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d61db700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3d0061760 0x7fc3d0143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 -- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3d0061760 msgr2=0x7fc3d0143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3d0061760 0x7fc3d0143e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 -- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d0098ae0 msgr2=0x7fc3d0157ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d0098ae0 0x7fc3d0157ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 -- 172.21.15.61:0/3156503874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3c000b1e0 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d69dc700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 0x7fc3d014a2e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fc3c0002f70 tx=0x7fc3c0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:34.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3c000d940 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc3c0011a20 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3d0148600 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3d015a5e0 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.409+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3c0011d80 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.413+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3d015a8c0 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.413+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc3c001a040 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.413+0000 7fc3cf7fe700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3b0053920 0x7fc3b0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.413+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc3c006c990 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.413+0000 7fc3d61db700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3b0053920 0x7fc3b0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.413+0000 7fc3d61db700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3b0053920 0x7fc3b0055de0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fc3c800b000 tx=0x7fc3c80091a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:34.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.417+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc3c0043b10 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.545+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc3d014a000 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3cf7fe700 1 -- 172.21.15.61:0/3156503874 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v67) v1 ==== 74+0+13230 (secure 0 0 0) 0x7fc3c0023020 con 0x7fc3d00b1ef0 2022-07-20T03:13:34.549 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:34.550 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":67,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:13:33.394324+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":12,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":49,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":58,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:13:34.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3b0053920 msgr2=0x7fc3b0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3b0053920 0x7fc3b0055de0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7fc3c800b000 tx=0x7fc3c80091a0).stop 2022-07-20T03:13:34.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 msgr2=0x7fc3d014a2e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 0x7fc3d014a2e0 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7fc3c0002f70 tx=0x7fc3c0015000).stop 2022-07-20T03:13:34.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 shutdown_connections 2022-07-20T03:13:34.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3b0053920 0x7fc3b0055de0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3d0098ae0 0x7fc3d0157ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3d00b1ef0 0x7fc3d014a2e0 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 --2- 172.21.15.61:0/3156503874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3d0061760 0x7fc3d0143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 >> 172.21.15.61:0/3156503874 conn(0x7fc3d00b2ca0 msgr2=0x7fc3d0145740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:34.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 shutdown_connections 2022-07-20T03:13:34.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.549+0000 7fc3d71dd700 1 -- 172.21.15.61:0/3156503874 wait complete. 2022-07-20T03:13:34.574 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:13:34.575 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:13:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 Processor -- start 2022-07-20T03:13:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- start start 2022-07-20T03:13:34.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f28080b9da0 0x7f28081468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f28080b3cf0 0x7f2808142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28080ba180 con 0x7f2808142f40 2022-07-20T03:13:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28080ad5b0 con 0x7f28080b3cf0 2022-07-20T03:13:34.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28081461b0 con 0x7f28080b9da0 2022-07-20T03:13:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f28080b3cf0 0x7f2808142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f28080b3cf0 0x7f2808142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46504/0 (socket says 172.21.15.61:46504) 2022-07-20T03:13:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 -- 172.21.15.61:0/3673422526 learned_addr learned my addr 172.21.15.61:0/3673422526 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:34.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280cdfb700 1 --2- 172.21.15.61:0/3673422526 >> v2:172.21.15.117:3300/0 conn(0x7f28080b9da0 0x7f28081468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 -- 172.21.15.61:0/3673422526 >> v2:172.21.15.117:3300/0 conn(0x7f28080b9da0 msgr2=0x7f28081468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:34.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 --2- 172.21.15.61:0/3673422526 >> v2:172.21.15.117:3300/0 conn(0x7f28080b9da0 0x7f28081468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 -- 172.21.15.61:0/3673422526 >> v1:172.21.15.117:6789/0 conn(0x7f2808142f40 legacy=0x7f280805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 -- 172.21.15.61:0/3673422526 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2808146450 con 0x7f28080b3cf0 2022-07-20T03:13:34.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f2807fff700 1 --2- 172.21.15.61:0/3673422526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b3cf0 0x7f2808142a50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f27f4002f70 tx=0x7f27f4006eb0).ready entity=mon.0 client_cookie=5bbfe2c23682786f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:34.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f28077fe700 1 -- 172.21.15.61:0/3673422526 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27f400d5c0 con 0x7f28080b3cf0 2022-07-20T03:13:34.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f28077fe700 1 -- 172.21.15.61:0/3673422526 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27f4000e90 con 0x7f28080b3cf0 2022-07-20T03:13:34.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f28077fe700 1 -- 172.21.15.61:0/3673422526 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27f400d950 con 0x7f28080b3cf0 2022-07-20T03:13:34.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- 172.21.15.61:0/3673422526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b3cf0 msgr2=0x7f2808142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 --2- 172.21.15.61:0/3673422526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b3cf0 0x7f2808142a50 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7f27f4002f70 tx=0x7f27f4006eb0).stop 2022-07-20T03:13:34.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- 172.21.15.61:0/3673422526 shutdown_connections 2022-07-20T03:13:34.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 --2- 172.21.15.61:0/3673422526 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b3cf0 0x7f2808142a50 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 --2- 172.21.15.61:0/3673422526 >> v2:172.21.15.117:3300/0 conn(0x7f28080b9da0 0x7f28081468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- 172.21.15.61:0/3673422526 >> 172.21.15.61:0/3673422526 conn(0x7f28080678c0 msgr2=0x7f28080b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:34.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- 172.21.15.61:0/3673422526 shutdown_connections 2022-07-20T03:13:34.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.713+0000 7f280ddfd700 1 -- 172.21.15.61:0/3673422526 wait complete. 2022-07-20T03:13:34.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 Processor -- start 2022-07-20T03:13:34.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 -- start start 2022-07-20T03:13:34.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28080b3cf0 0x7f2808151fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b9da0 0x7f28080b8530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2808142f40 0x7f28080b2450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080b19a0 con 0x7f28080b9da0 2022-07-20T03:13:34.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28080afdb0 con 0x7f28080b3cf0 2022-07-20T03:13:34.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280ddfd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28080b1040 con 0x7f2808142f40 2022-07-20T03:13:34.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f2807fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b9da0 0x7f28080b8530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f2807fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b9da0 0x7f28080b8530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46510/0 (socket says 172.21.15.61:46510) 2022-07-20T03:13:34.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f2807fff700 1 -- 172.21.15.61:0/1681674888 learned_addr learned my addr 172.21.15.61:0/1681674888 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:34.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28080b3cf0 0x7f2808151fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280d5fc700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2808142f40 0x7f28080b2450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 -- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2808142f40 msgr2=0x7f28080b2450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2808142f40 0x7f28080b2450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 -- 172.21.15.61:0/1681674888 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b9da0 msgr2=0x7f28080b8530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b9da0 0x7f28080b8530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 -- 172.21.15.61:0/1681674888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27f400b1e0 con 0x7f28080b3cf0 2022-07-20T03:13:34.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f280cdfb700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28080b3cf0 0x7f2808151fa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f27fc00c000 tx=0x7f27fc006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:34.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.717+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27fc010780 con 0x7f28080b3cf0 2022-07-20T03:13:34.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28080af1a0 con 0x7f28080b3cf0 2022-07-20T03:13:34.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f27fc0108e0 con 0x7f28080b3cf0 2022-07-20T03:13:34.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f27fc010c40 con 0x7f28080b3cf0 2022-07-20T03:13:34.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2808112540 con 0x7f28080b3cf0 2022-07-20T03:13:34.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28080afef0 con 0x7f28080b3cf0 2022-07-20T03:13:34.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f27fc01b410 con 0x7f28080b3cf0 2022-07-20T03:13:34.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f2805ffb700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f27e8053a40 0x7f27e8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:34.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f27fc06bc60 con 0x7f28080b3cf0 2022-07-20T03:13:34.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.721+0000 7f2807fff700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f27e8053a40 0x7f27e8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:34.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.725+0000 7f2807fff700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f27e8053a40 0x7f27e8055f00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f27f401c000 tx=0x7f27f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:34.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.729+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f27fc042df0 con 0x7f28080b3cf0 2022-07-20T03:13:34.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.861+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f28080af6d0 con 0x7f28080b3cf0 2022-07-20T03:13:34.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.861+0000 7f2805ffb700 1 -- 172.21.15.61:0/1681674888 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v67) v1 ==== 110+0+10 (secure 0 0 0) 0x7f27fc046410 con 0x7f28080b3cf0 2022-07-20T03:13:34.871 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:13:34.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f27e8053a40 msgr2=0x7f27e8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f27e8053a40 0x7f27e8055f00 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f27f401c000 tx=0x7f27f4015000).stop 2022-07-20T03:13:34.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28080b3cf0 msgr2=0x7f2808151fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:34.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28080b3cf0 0x7f2808151fa0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f27fc00c000 tx=0x7f27fc006df0).stop 2022-07-20T03:13:34.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 shutdown_connections 2022-07-20T03:13:34.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f27e8053a40 0x7f27e8055f00 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2808142f40 0x7f28080b2450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28080b9da0 0x7f28080b8530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 --2- 172.21.15.61:0/1681674888 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28080b3cf0 0x7f2808151fa0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:34.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 >> 172.21.15.61:0/1681674888 conn(0x7f28080678c0 msgr2=0x7f28080b4410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:34.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 shutdown_connections 2022-07-20T03:13:34.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:34.865+0000 7f280ddfd700 1 -- 172.21.15.61:0/1681674888 wait complete. 2022-07-20T03:13:34.898 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:13:35.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f312b6db700 1 Processor -- start 2022-07-20T03:13:35.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f312b6db700 1 -- start start 2022-07-20T03:13:35.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f312b6db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f3124146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f312b6db700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240afdb0 con 0x7f3124146b70 2022-07-20T03:13:35.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f312b6db700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f31240b0050 con 0x7f31240988b0 2022-07-20T03:13:35.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f312b6db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312405fcd0 con 0x7f31241462a0 2022-07-20T03:13:35.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3129c78700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3124146b70 0x7f3124060170 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43120/0 (socket says 172.21.15.61:43120) 2022-07-20T03:13:35.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3129c78700 1 -- 172.21.15.61:0/528464837 learned_addr learned my addr 172.21.15.61:0/528464837 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:35.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3129477700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f3124146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1369660532 0 0) 0x7f31240afdb0 con 0x7f3124146b70 2022-07-20T03:13:35.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f310c003260 con 0x7f3124146b70 2022-07-20T03:13:35.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2503286617 0 0) 0x7f310c003260 con 0x7f3124146b70 2022-07-20T03:13:35.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 msgr2=0x7f3124146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f3124146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 >> v1:172.21.15.117:6789/0 conn(0x7f31240988b0 legacy=0x7f31241422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f31240afae0 con 0x7f3124146b70 2022-07-20T03:13:35.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f311c0019e0 con 0x7f3124146b70 2022-07-20T03:13:35.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 >> v1:172.21.15.61:6789/0 conn(0x7f3124146b70 legacy=0x7f3124060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f310c003780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f310c003cb0 0x7f310c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c0049f0 0x7f310c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312405fcd0 con 0x7f310c003cb0 2022-07-20T03:13:35.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f310c00b800 con 0x7f310c0049f0 2022-07-20T03:13:35.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f310c00b9c0 con 0x7f31241462a0 2022-07-20T03:13:35.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3128c76700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f310c003cb0 0x7f310c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3129477700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f310c003780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.041+0000 7f3129c78700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c0049f0 0x7f310c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3128c76700 1 -- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 msgr2=0x7f310c003780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3128c76700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f310c003780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3128c76700 1 -- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c0049f0 msgr2=0x7f310c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3128c76700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c0049f0 0x7f310c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3128c76700 1 -- 172.21.15.61:0/528464837 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f310c003260 con 0x7f310c003cb0 2022-07-20T03:13:35.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3129477700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f31241462a0 0x7f310c003780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:35.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3128c76700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f310c003cb0 0x7f310c004500 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f3124066650 tx=0x7f3114007a00).ready entity=mon.0 client_cookie=240621b88d07d115 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3114007d10 con 0x7f310c003cb0 2022-07-20T03:13:35.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114007030 con 0x7f310c003cb0 2022-07-20T03:13:35.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3113fff700 1 -- 172.21.15.61:0/528464837 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3114007390 con 0x7f310c003cb0 2022-07-20T03:13:35.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 -- 172.21.15.61:0/528464837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f310c003cb0 msgr2=0x7f310c004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 --2- 172.21.15.61:0/528464837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f310c003cb0 0x7f310c004500 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7f3124066650 tx=0x7f3114007a00).stop 2022-07-20T03:13:35.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f3129c78700 1 -- 172.21.15.61:0/528464837 reap_dead start 2022-07-20T03:13:35.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 -- 172.21.15.61:0/528464837 shutdown_connections 2022-07-20T03:13:35.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 -- 172.21.15.61:0/528464837 >> 172.21.15.61:0/528464837 conn(0x7f31240b5f00 msgr2=0x7f31240b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:35.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 -- 172.21.15.61:0/528464837 shutdown_connections 2022-07-20T03:13:35.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 -- 172.21.15.61:0/528464837 wait complete. 2022-07-20T03:13:35.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 Processor -- start 2022-07-20T03:13:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.045+0000 7f312b6db700 1 -- start start 2022-07-20T03:13:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c003cb0 0x7f3124140dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 0x7f3124155d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3124156230 0x7f3124147980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3124145850 con 0x7f3124155670 2022-07-20T03:13:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f312409d680 con 0x7f310c003cb0 2022-07-20T03:13:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f312409d920 con 0x7f3124156230 2022-07-20T03:13:35.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 0x7f3124155d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 0x7f3124155d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46530/0 (socket says 172.21.15.61:46530) 2022-07-20T03:13:35.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 -- 172.21.15.61:0/4072700139 learned_addr learned my addr 172.21.15.61:0/4072700139 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:35.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3129477700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c003cb0 0x7f3124140dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3129c78700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3124156230 0x7f3124147980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 -- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3124156230 msgr2=0x7f3124147980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3124156230 0x7f3124147980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 -- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c003cb0 msgr2=0x7f3124140dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c003cb0 0x7f3124140dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 -- 172.21.15.61:0/4072700139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3124153240 con 0x7f3124155670 2022-07-20T03:13:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f3128c76700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 0x7f3124155d40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f3124066650 tx=0x7f3114015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3114020640 con 0x7f3124155670 2022-07-20T03:13:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3114006e90 con 0x7f3124155670 2022-07-20T03:13:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3124144660 con 0x7f3124155670 2022-07-20T03:13:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3114020990 con 0x7f3124155670 2022-07-20T03:13:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.049+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31241454d0 con 0x7f3124155670 2022-07-20T03:13:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.053+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f312409d4e0 con 0x7f3124155670 2022-07-20T03:13:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.053+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f311401d070 con 0x7f3124155670 2022-07-20T03:13:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.053+0000 7f31127fc700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3104053a60 0x7f3104055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.053+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f311406cb20 con 0x7f3124155670 2022-07-20T03:13:35.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.053+0000 7f3129477700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3104053a60 0x7f3104055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.053+0000 7f3129477700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3104053a60 0x7f3104055f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f312000b000 tx=0x7f3120006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.061+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f312409d4e0 con 0x7f3124155670 2022-07-20T03:13:35.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f312409d4e0 con 0x7f3124155670 2022-07-20T03:13:35.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f31127fc700 1 -- 172.21.15.61:0/4072700139 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v67) v1 ==== 92+0+10 (secure 0 0 0) 0x7f312409d4e0 con 0x7f3124155670 2022-07-20T03:13:35.265 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:35.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3104053a60 msgr2=0x7f3104055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3104053a60 0x7f3104055f20 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f312000b000 tx=0x7f3120006df0).stop 2022-07-20T03:13:35.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 msgr2=0x7f3124155d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 0x7f3124155d40 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7f3124066650 tx=0x7f3114015000).stop 2022-07-20T03:13:35.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 shutdown_connections 2022-07-20T03:13:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3104053a60 0x7f3104055f20 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3124156230 0x7f3124147980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3124155670 0x7f3124155d40 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 --2- 172.21.15.61:0/4072700139 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f310c003cb0 0x7f3124140dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.189+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 >> 172.21.15.61:0/4072700139 conn(0x7f31240b5f00 msgr2=0x7f31240b8b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:35.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.193+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 shutdown_connections 2022-07-20T03:13:35.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.193+0000 7f312b6db700 1 -- 172.21.15.61:0/4072700139 wait complete. 2022-07-20T03:13:35.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:13:35.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35efea8700 1 Processor -- start 2022-07-20T03:13:35.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35efea8700 1 -- start start 2022-07-20T03:13:35.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35efea8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 0x7f35e80ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35efea8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e805ff30 con 0x7f35e80b1620 2022-07-20T03:13:35.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35efea8700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f35e80b1a00 con 0x7f35e80616b0 2022-07-20T03:13:35.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35efea8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e80ad500 con 0x7f35e8061b40 2022-07-20T03:13:35.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f35e80b1620 0x7f35e80acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43136/0 (socket says 172.21.15.61:43136) 2022-07-20T03:13:35.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 -- 172.21.15.61:0/2661626530 learned_addr learned my addr 172.21.15.61:0/2661626530 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:35.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ee445700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 0x7f35e80ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1239740481 0 0) 0x7f35e805ff30 con 0x7f35e80b1620 2022-07-20T03:13:35.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35c8003260 con 0x7f35e80b1620 2022-07-20T03:13:35.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222527257 0 0) 0x7f35e80b1a00 con 0x7f35e80616b0 2022-07-20T03:13:35.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f35e805ff30 con 0x7f35e80616b0 2022-07-20T03:13:35.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3722469737 0 0) 0x7f35c8003260 con 0x7f35e80b1620 2022-07-20T03:13:35.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 msgr2=0x7f35e80ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 0x7f35e80ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 >> v1:172.21.15.117:6789/0 conn(0x7f35e80616b0 legacy=0x7f35e80aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e805c0d0 con 0x7f35e80b1620 2022-07-20T03:13:35.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f35d8001680 con 0x7f35e80b1620 2022-07-20T03:13:35.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 >> v1:172.21.15.61:6789/0 conn(0x7f35e80b1620 legacy=0x7f35e80acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35c8003910 0x7f35c8003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 0x7f35c8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c80049f0 0x7f35c8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e80ad500 con 0x7f35c8003910 2022-07-20T03:13:35.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35c800b780 con 0x7f35c80049f0 2022-07-20T03:13:35.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35c800b940 con 0x7f35e8061b40 2022-07-20T03:13:35.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35c8003910 0x7f35c8003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ed443700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 0x7f35c8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35ee445700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c80049f0 0x7f35c8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 -- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 msgr2=0x7f35c8004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8061b40 0x7f35c8004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 -- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c80049f0 msgr2=0x7f35c8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c80049f0 0x7f35c8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 -- 172.21.15.61:0/2661626530 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35c8003260 con 0x7f35c8003910 2022-07-20T03:13:35.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.429+0000 7f35edc44700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35c8003910 0x7f35c8003d10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f35d8001450 tx=0x7f35e805fe20).ready entity=mon.0 client_cookie=27570c946a9a4c14 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d8007170 con 0x7f35c8003910 2022-07-20T03:13:35.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35d80072d0 con 0x7f35c8003910 2022-07-20T03:13:35.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35ecc42700 1 -- 172.21.15.61:0/2661626530 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35d800c8f0 con 0x7f35c8003910 2022-07-20T03:13:35.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- 172.21.15.61:0/2661626530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35c8003910 msgr2=0x7f35c8003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 --2- 172.21.15.61:0/2661626530 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35c8003910 0x7f35c8003d10 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f35d8001450 tx=0x7f35e805fe20).stop 2022-07-20T03:13:35.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35ee445700 1 -- 172.21.15.61:0/2661626530 reap_dead start 2022-07-20T03:13:35.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- 172.21.15.61:0/2661626530 shutdown_connections 2022-07-20T03:13:35.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- 172.21.15.61:0/2661626530 >> 172.21.15.61:0/2661626530 conn(0x7f35e80b23d0 msgr2=0x7f35e80b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:35.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- 172.21.15.61:0/2661626530 shutdown_connections 2022-07-20T03:13:35.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- 172.21.15.61:0/2661626530 wait complete. 2022-07-20T03:13:35.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 Processor -- start 2022-07-20T03:13:35.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- start start 2022-07-20T03:13:35.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c8003910 0x7f35e814a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 0x7f35e81497e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8063f10 0x7f35e8149cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8157340 con 0x7f35e8064b80 2022-07-20T03:13:35.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f35e81562f0 con 0x7f35c8003910 2022-07-20T03:13:35.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35efea8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f35e8063600 con 0x7f35e8063f10 2022-07-20T03:13:35.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.433+0000 7f35ed443700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 0x7f35e81497e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 0x7f35e81497e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46552/0 (socket says 172.21.15.61:46552) 2022-07-20T03:13:35.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 -- 172.21.15.61:0/3165951428 learned_addr learned my addr 172.21.15.61:0/3165951428 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:35.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35edc44700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c8003910 0x7f35e814a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ee445700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8063f10 0x7f35e8149cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 -- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8063f10 msgr2=0x7f35e8149cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8063f10 0x7f35e8149cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 -- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c8003910 msgr2=0x7f35e814a850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c8003910 0x7f35e814a850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 -- 172.21.15.61:0/3165951428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f35e80638a0 con 0x7f35e8064b80 2022-07-20T03:13:35.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35ed443700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 0x7f35e81497e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f35e000d000 tx=0x7f35e0007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e0017070 con 0x7f35e8064b80 2022-07-20T03:13:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f35e0002210 con 0x7f35e8064b80 2022-07-20T03:13:35.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f35e8063b40 con 0x7f35e8064b80 2022-07-20T03:13:35.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f35e0002570 con 0x7f35e8064b80 2022-07-20T03:13:35.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f35e814a240 con 0x7f35e8064b80 2022-07-20T03:13:35.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f35e805be30 con 0x7f35e8064b80 2022-07-20T03:13:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.437+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f35e00017a0 con 0x7f35e8064b80 2022-07-20T03:13:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.441+0000 7f35e6ffd700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f35cc053a60 0x7f35cc055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.441+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f35e006bd50 con 0x7f35e8064b80 2022-07-20T03:13:35.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.441+0000 7f35edc44700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f35cc053a60 0x7f35cc055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.441+0000 7f35edc44700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f35cc053a60 0x7f35cc055f20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f35d8016000 tx=0x7f35d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.445+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f35e0042fe0 con 0x7f35e8064b80 2022-07-20T03:13:35.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.577+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f35e8063cd0 con 0x7f35e8064b80 2022-07-20T03:13:35.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.577+0000 7f35e6ffd700 1 -- 172.21.15.61:0/3165951428 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v67) v1 ==== 84+0+143 (secure 0 0 0) 0x7f35e0046600 con 0x7f35e8064b80 2022-07-20T03:13:35.603 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:35.604 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:13:35.604 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:35.605 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:13:35.605 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:13:35.606 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:13:35.621 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:13:35.622 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:13:35.622 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:13:35.623 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:13:35.623 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:13:35.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f35cc053a60 msgr2=0x7f35cc055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f35cc053a60 0x7f35cc055f20 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f35d8016000 tx=0x7f35d8015000).stop 2022-07-20T03:13:35.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 msgr2=0x7f35e81497e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 0x7f35e81497e0 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f35e000d000 tx=0x7f35e0007d80).stop 2022-07-20T03:13:35.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 shutdown_connections 2022-07-20T03:13:35.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f35cc053a60 0x7f35cc055f20 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f35e8063f10 0x7f35e8149cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f35e8064b80 0x7f35e81497e0 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 --2- 172.21.15.61:0/3165951428 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f35c8003910 0x7f35e814a850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 >> 172.21.15.61:0/3165951428 conn(0x7f35e80b23d0 msgr2=0x7f35e80ac050 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 shutdown_connections 2022-07-20T03:13:35.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.581+0000 7f35efea8700 1 -- 172.21.15.61:0/3165951428 wait complete. 2022-07-20T03:13:35.657 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:13:35.661 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:13:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.805+0000 7fb41df68700 1 Processor -- start 2022-07-20T03:13:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.805+0000 7fb41df68700 1 -- start start 2022-07-20T03:13:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb41df68700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb4180b4d60 0x7fb418145d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb41df68700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4180607e0 con 0x7fb41809df40 2022-07-20T03:13:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb41df68700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb418060a80 con 0x7fb418060ee0 2022-07-20T03:13:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb41df68700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb418060d20 con 0x7fb4180b4d60 2022-07-20T03:13:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4177fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb41809df40 0x7fb418145680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43156/0 (socket says 172.21.15.61:43156) 2022-07-20T03:13:35.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4177fe700 1 -- 172.21.15.61:0/3607301509 learned_addr learned my addr 172.21.15.61:0/3607301509 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:35.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb416ffd700 1 --2- 172.21.15.61:0/3607301509 >> v2:172.21.15.117:3300/0 conn(0x7fb4180b4d60 0x7fb418145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 11774987 0 0) 0x7fb4180607e0 con 0x7fb41809df40 2022-07-20T03:13:35.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb404003260 con 0x7fb41809df40 2022-07-20T03:13:35.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 640743085 0 0) 0x7fb418060a80 con 0x7fb418060ee0 2022-07-20T03:13:35.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4180607e0 con 0x7fb418060ee0 2022-07-20T03:13:35.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3918162540 0 0) 0x7fb404003260 con 0x7fb41809df40 2022-07-20T03:13:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 >> v2:172.21.15.117:3300/0 conn(0x7fb4180b4d60 msgr2=0x7fb418145d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 --2- 172.21.15.61:0/3607301509 >> v2:172.21.15.117:3300/0 conn(0x7fb4180b4d60 0x7fb418145d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 >> v1:172.21.15.117:6789/0 conn(0x7fb418060ee0 legacy=0x7fb41809cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41809e320 con 0x7fb41809df40 2022-07-20T03:13:35.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fb40c0017e0 con 0x7fb41809df40 2022-07-20T03:13:35.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 >> v1:172.21.15.61:6789/0 conn(0x7fb41809df40 legacy=0x7fb418145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4040038e0 0x7fb404003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb404004500 0x7fb404004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb404004e70 0x7fb40400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418060d20 con 0x7fb4040038e0 2022-07-20T03:13:35.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb40400bcf0 con 0x7fb404004e70 2022-07-20T03:13:35.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb40400beb0 con 0x7fb404004500 2022-07-20T03:13:35.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4177fe700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4040038e0 0x7fb404003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb404004e70 0x7fb40400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb416ffd700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb404004500 0x7fb404004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 -- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb404004500 msgr2=0x7fb404004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb404004500 0x7fb404004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 -- 172.21.15.61:0/3607301509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4040038e0 msgr2=0x7fb404003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:35.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4040038e0 0x7fb404003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 -- 172.21.15.61:0/3607301509 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb404003260 con 0x7fb404004e70 2022-07-20T03:13:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb4177fe700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4040038e0 0x7fb404003ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 -- 172.21.15.61:0/3607301509 reap_dead start 2022-07-20T03:13:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.809+0000 7fb417fff700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb404004e70 0x7fb40400b750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb4040054d0 tx=0x7fb410007d80).ready entity=mon.1 client_cookie=7af2f70b5ea8122f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb410006890 con 0x7fb404004e70 2022-07-20T03:13:35.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4100069f0 con 0x7fb404004e70 2022-07-20T03:13:35.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb4167fc700 1 -- 172.21.15.61:0/3607301509 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb410006d50 con 0x7fb404004e70 2022-07-20T03:13:35.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 -- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb404004e70 msgr2=0x7fb40400b750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb404004e70 0x7fb40400b750 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb4040054d0 tx=0x7fb410007d80).stop 2022-07-20T03:13:35.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 -- 172.21.15.61:0/3607301509 shutdown_connections 2022-07-20T03:13:35.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 --2- 172.21.15.61:0/3607301509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb404004e70 0x7fb40400b750 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 -- 172.21.15.61:0/3607301509 >> 172.21.15.61:0/3607301509 conn(0x7fb4180b5f30 msgr2=0x7fb4180b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:35.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 -- 172.21.15.61:0/3607301509 shutdown_connections 2022-07-20T03:13:35.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 -- 172.21.15.61:0/3607301509 wait complete. 2022-07-20T03:13:35.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 Processor -- start 2022-07-20T03:13:35.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.813+0000 7fb41df68700 1 -- start start 2022-07-20T03:13:35.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 0x7fb418142760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb418140b80 0x7fb418141000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180ae450 0x7fb4180ae8d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180b17d0 con 0x7fb404004e70 2022-07-20T03:13:35.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180af650 con 0x7fb4180ae450 2022-07-20T03:13:35.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4180b0190 con 0x7fb418140b80 2022-07-20T03:13:35.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 0x7fb418142760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 0x7fb418142760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46570/0 (socket says 172.21.15.61:46570) 2022-07-20T03:13:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 -- 172.21.15.61:0/921147270 learned_addr learned my addr 172.21.15.61:0/921147270 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb417fff700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180ae450 0x7fb4180ae8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb416ffd700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb418140b80 0x7fb418141000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 -- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb418140b80 msgr2=0x7fb418141000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb418140b80 0x7fb418141000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 -- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180ae450 msgr2=0x7fb4180ae8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:35.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180ae450 0x7fb4180ae8d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:35.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 -- 172.21.15.61:0/921147270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4180b0430 con 0x7fb404004e70 2022-07-20T03:13:35.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb417fff700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180ae450 0x7fb4180ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:35.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb416ffd700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb418140b80 0x7fb418141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:35.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb4177fe700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 0x7fb418142760 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fb40c00d000 tx=0x7fb40c009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb40c0072c0 con 0x7fb404004e70 2022-07-20T03:13:35.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb40c0057c0 con 0x7fb404004e70 2022-07-20T03:13:35.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4180b1450 con 0x7fb404004e70 2022-07-20T03:13:35.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb40c005b20 con 0x7fb404004e70 2022-07-20T03:13:35.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4180b0990 con 0x7fb404004e70 2022-07-20T03:13:35.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.817+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4180574e0 con 0x7fb404004e70 2022-07-20T03:13:35.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.821+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb40c01a030 con 0x7fb404004e70 2022-07-20T03:13:35.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.821+0000 7fb414ff9700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053a60 0x7fb3f8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:35.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.821+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(67..67 src has 1..67) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb40c06b8d0 con 0x7fb404004e70 2022-07-20T03:13:35.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.821+0000 7fb416ffd700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053a60 0x7fb3f8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:35.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.821+0000 7fb416ffd700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053a60 0x7fb3f8055f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fb40800b000 tx=0x7fb408008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:35.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.825+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb40c042b60 con 0x7fb404004e70 2022-07-20T03:13:35.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:35.953+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb4180b1640 con 0x7fb404004e70 2022-07-20T03:13:36.126 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.117+0000 7fb414ff9700 1 -- 172.21.15.61:0/921147270 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v68) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb40c046180 con 0x7fb404004e70 2022-07-20T03:13:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053a60 msgr2=0x7fb3f8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053a60 0x7fb3f8055f20 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fb40800b000 tx=0x7fb408008040).stop 2022-07-20T03:13:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 msgr2=0x7fb418142760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:36.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 0x7fb418142760 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fb40c00d000 tx=0x7fb40c009040).stop 2022-07-20T03:13:36.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 shutdown_connections 2022-07-20T03:13:36.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053a60 0x7fb3f8055f20 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180ae450 0x7fb4180ae8d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb418140b80 0x7fb418141000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 --2- 172.21.15.61:0/921147270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb404004e70 0x7fb418142760 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 >> 172.21.15.61:0/921147270 conn(0x7fb4180b5f30 msgr2=0x7fb418145990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:36.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 shutdown_connections 2022-07-20T03:13:36.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.121+0000 7fb41df68700 1 -- 172.21.15.61:0/921147270 wait complete. 2022-07-20T03:13:36.137 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:13:36.151 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:13:36.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:36.164 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:36.196 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:36.221 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:13:36.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.305+0000 7fed3906e700 1 Processor -- start 2022-07-20T03:13:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.305+0000 7fed3906e700 1 -- start start 2022-07-20T03:13:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed34140490 0x7fed340afa40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fed34141280 0x7fed3405cda0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fed34060fa0 con 0x7fed3405d290 2022-07-20T03:13:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fed34140b40 con 0x7fed34141280 2022-07-20T03:13:36.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed34140de0 con 0x7fed34140490 2022-07-20T03:13:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed32d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed34140490 0x7fed340afa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3259c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fed34141280 0x7fed3405cda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:36.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3359e700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fed3405d290 0x7fed3405fa50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:51684/0 (socket says 172.21.15.61:51684) 2022-07-20T03:13:36.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3359e700 1 -- 172.21.15.61:0/1322940880 learned_addr learned my addr 172.21.15.61:0/1322940880 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:36.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed32d9d700 1 -- 172.21.15.61:0/1322940880 >> v2:172.21.15.117:3300/0 conn(0x7fed34141280 msgr2=0x7fed3405cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:36.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed32d9d700 1 --2- 172.21.15.61:0/1322940880 >> v2:172.21.15.117:3300/0 conn(0x7fed34141280 0x7fed3405cda0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed32d9d700 1 -- 172.21.15.61:0/1322940880 >> v1:172.21.15.117:6789/0 conn(0x7fed3405d290 legacy=0x7fed3405fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:36.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed31d9b700 1 -- 172.21.15.61:0/1322940880 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2186495430 0 0) 0x7fed34060fa0 con 0x7fed3405d290 2022-07-20T03:13:36.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed32d9d700 1 -- 172.21.15.61:0/1322940880 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed340b02e0 con 0x7fed34140490 2022-07-20T03:13:36.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3259c700 1 --2- 172.21.15.61:0/1322940880 >> v2:172.21.15.117:3300/0 conn(0x7fed34141280 0x7fed3405cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:36.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed32d9d700 1 --2- 172.21.15.61:0/1322940880 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed34140490 0x7fed340afa40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fed1c002f70 tx=0x7fed1c006df0).ready entity=mon.2 client_cookie=f7b89b1312bb5cba server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:36.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed31d9b700 1 -- 172.21.15.61:0/1322940880 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed1c00d570 con 0x7fed34140490 2022-07-20T03:13:36.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed31d9b700 1 -- 172.21.15.61:0/1322940880 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed1c000e90 con 0x7fed34140490 2022-07-20T03:13:36.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed31d9b700 1 -- 172.21.15.61:0/1322940880 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed1c00d8c0 con 0x7fed34140490 2022-07-20T03:13:36.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 -- 172.21.15.61:0/1322940880 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed34140490 msgr2=0x7fed340afa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:36.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.309+0000 7fed3906e700 1 --2- 172.21.15.61:0/1322940880 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed34140490 0x7fed340afa40 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fed1c002f70 tx=0x7fed1c006df0).stop 2022-07-20T03:13:36.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- 172.21.15.61:0/1322940880 shutdown_connections 2022-07-20T03:13:36.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 --2- 172.21.15.61:0/1322940880 >> v2:172.21.15.117:3300/0 conn(0x7fed34141280 0x7fed3405cda0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 --2- 172.21.15.61:0/1322940880 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed34140490 0x7fed340afa40 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- 172.21.15.61:0/1322940880 >> 172.21.15.61:0/1322940880 conn(0x7fed340b5b70 msgr2=0x7fed340b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:36.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- 172.21.15.61:0/1322940880 shutdown_connections 2022-07-20T03:13:36.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- 172.21.15.61:0/1322940880 wait complete. 2022-07-20T03:13:36.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 Processor -- start 2022-07-20T03:13:36.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- start start 2022-07-20T03:13:36.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3405d290 0x7fed34150980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:36.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed34140490 0x7fed340ad1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:36.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed34141280 0x7fed340ad6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:36.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed340abaf0 con 0x7fed34140490 2022-07-20T03:13:36.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed34067230 con 0x7fed34141280 2022-07-20T03:13:36.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3906e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fed340674d0 con 0x7fed3405d290 2022-07-20T03:13:36.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed34140490 0x7fed340ad1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:36.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed34141280 0x7fed340ad6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:36.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed34140490 0x7fed340ad1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46588/0 (socket says 172.21.15.61:46588) 2022-07-20T03:13:36.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3259c700 1 -- 172.21.15.61:0/641896355 learned_addr learned my addr 172.21.15.61:0/641896355 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:36.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed32d9d700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3405d290 0x7fed34150980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:36.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3359e700 1 -- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3405d290 msgr2=0x7fed34150980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:36.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.313+0000 7fed3359e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3405d290 0x7fed34150980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3359e700 1 -- 172.21.15.61:0/641896355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed34140490 msgr2=0x7fed340ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:36.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3359e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed34140490 0x7fed340ad1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:36.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3359e700 1 -- 172.21.15.61:0/641896355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed2400b180 con 0x7fed34141280 2022-07-20T03:13:36.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3359e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed34141280 0x7fed340ad6e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fed2c001250 tx=0x7fed2c007d00).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:36.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c005ba0 con 0x7fed34141280 2022-07-20T03:13:36.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fed2c005d00 con 0x7fed34141280 2022-07-20T03:13:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed1c00b180 con 0x7fed34141280 2022-07-20T03:13:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fed2c004a90 con 0x7fed34141280 2022-07-20T03:13:36.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed34065c00 con 0x7fed34141280 2022-07-20T03:13:36.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed34060160 con 0x7fed34141280 2022-07-20T03:13:36.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fed2c004bf0 con 0x7fed34141280 2022-07-20T03:13:36.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed2bfff700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed0c053a40 0x7fed0c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:36.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.317+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(68..68 src has 1..68) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fed2c06b7a0 con 0x7fed34141280 2022-07-20T03:13:36.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.321+0000 7fed32d9d700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed0c053a40 0x7fed0c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:36.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.321+0000 7fed32d9d700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed0c053a40 0x7fed0c055f00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fed1c01a000 tx=0x7fed1c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:36.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.325+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fed2c042870 con 0x7fed34141280 2022-07-20T03:13:36.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:36.457+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fed340574e0 con 0x7fed34141280 2022-07-20T03:13:37.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.125+0000 7fed2bfff700 1 -- 172.21.15.61:0/641896355 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v69) v1 ==== 125+0+0 (secure 0 0 0) 0x7fed2c045e90 con 0x7fed34141280 2022-07-20T03:13:37.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed0c053a40 msgr2=0x7fed0c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed0c053a40 0x7fed0c055f00 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fed1c01a000 tx=0x7fed1c014000).stop 2022-07-20T03:13:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed34141280 msgr2=0x7fed340ad6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed34141280 0x7fed340ad6e0 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fed2c001250 tx=0x7fed2c007d00).stop 2022-07-20T03:13:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 shutdown_connections 2022-07-20T03:13:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fed0c053a40 0x7fed0c055f00 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fed34141280 0x7fed340ad6e0 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fed34140490 0x7fed340ad1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 --2- 172.21.15.61:0/641896355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fed3405d290 0x7fed34150980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.129+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 >> 172.21.15.61:0/641896355 conn(0x7fed340b5b70 msgr2=0x7fed3409a4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.133+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 shutdown_connections 2022-07-20T03:13:37.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.133+0000 7fed3906e700 1 -- 172.21.15.61:0/641896355 wait complete. 2022-07-20T03:13:37.137 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:13:37.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:13:37.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.253+0000 7f6b1a091700 1 Processor -- start 2022-07-20T03:13:37.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.253+0000 7f6b1a091700 1 -- start start 2022-07-20T03:13:37.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6b140ab470 0x7f6b14061a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14143080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6b14061370 con 0x7f6b140b3560 2022-07-20T03:13:37.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6b14146ae0 con 0x7f6b140ab470 2022-07-20T03:13:37.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b140b3940 con 0x7f6b14061f30 2022-07-20T03:13:37.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6b140ab470 0x7f6b14061a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6b140ab470 0x7f6b14061a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46594/0 (socket says 172.21.15.61:46594) 2022-07-20T03:13:37.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 -- 172.21.15.61:0/2331065883 learned_addr learned my addr 172.21.15.61:0/2331065883 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:37.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b19890700 1 --2- 172.21.15.61:0/2331065883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 -- 172.21.15.61:0/2331065883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 msgr2=0x7f6b14143080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:37.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 --2- 172.21.15.61:0/2331065883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14143080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 -- 172.21.15.61:0/2331065883 >> v1:172.21.15.117:6789/0 conn(0x7f6b140b3560 legacy=0x7f6b141463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 -- 172.21.15.61:0/2331065883 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b14061010 con 0x7f6b140ab470 2022-07-20T03:13:37.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1888e700 1 --2- 172.21.15.61:0/2331065883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b14061a40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f6b04002f70 tx=0x7f6b04006eb0).ready entity=mon.0 client_cookie=3c980742cfaa23d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:37.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b03fff700 1 -- 172.21.15.61:0/2331065883 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b0400d570 con 0x7f6b140ab470 2022-07-20T03:13:37.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b03fff700 1 -- 172.21.15.61:0/2331065883 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b04000e90 con 0x7f6b140ab470 2022-07-20T03:13:37.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b03fff700 1 -- 172.21.15.61:0/2331065883 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b0400d900 con 0x7f6b140ab470 2022-07-20T03:13:37.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- 172.21.15.61:0/2331065883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 msgr2=0x7f6b14061a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2331065883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b14061a40 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f6b04002f70 tx=0x7f6b04006eb0).stop 2022-07-20T03:13:37.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- 172.21.15.61:0/2331065883 shutdown_connections 2022-07-20T03:13:37.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2331065883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14143080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2331065883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b14061a40 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- 172.21.15.61:0/2331065883 >> 172.21.15.61:0/2331065883 conn(0x7f6b140b6b20 msgr2=0x7f6b140b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- 172.21.15.61:0/2331065883 shutdown_connections 2022-07-20T03:13:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.257+0000 7f6b1a091700 1 -- 172.21.15.61:0/2331065883 wait complete. 2022-07-20T03:13:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 Processor -- start 2022-07-20T03:13:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 -- start start 2022-07-20T03:13:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14155840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b1413f820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b140b3560 0x7f6b1413fd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14065600 con 0x7f6b140ab470 2022-07-20T03:13:37.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6b14140220 con 0x7f6b140b3560 2022-07-20T03:13:37.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1a091700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6b141404c0 con 0x7f6b14061f30 2022-07-20T03:13:37.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1888e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b1413f820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1908f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14155840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1908f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14155840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33270/0 (socket says 172.21.15.61:33270) 2022-07-20T03:13:37.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1908f700 1 -- 172.21.15.61:0/2299641425 learned_addr learned my addr 172.21.15.61:0/2299641425 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:37.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b19890700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b140b3560 0x7f6b1413fd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1888e700 1 -- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 msgr2=0x7f6b14155840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1888e700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14155840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1888e700 1 -- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b140b3560 msgr2=0x7f6b1413fd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1888e700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b140b3560 0x7f6b1413fd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.261+0000 7f6b1888e700 1 -- 172.21.15.61:0/2299641425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6b0400b1e0 con 0x7f6b140ab470 2022-07-20T03:13:37.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b1888e700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b1413f820 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f6b04002f70 tx=0x7f6b04015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:37.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b04011760 con 0x7f6b140ab470 2022-07-20T03:13:37.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6b040118c0 con 0x7f6b140ab470 2022-07-20T03:13:37.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6b14156020 con 0x7f6b140ab470 2022-07-20T03:13:37.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6b04011c20 con 0x7f6b140ab470 2022-07-20T03:13:37.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6b14155d30 con 0x7f6b140ab470 2022-07-20T03:13:37.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6b14064e40 con 0x7f6b140ab470 2022-07-20T03:13:37.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6b04025030 con 0x7f6b140ab470 2022-07-20T03:13:37.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b027fc700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6af4053af0 0x7f6af4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(69..69 src has 1..69) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f6b0406ce60 con 0x7f6b140ab470 2022-07-20T03:13:37.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.265+0000 7f6b1908f700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6af4053af0 0x7f6af4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.269+0000 7f6b1908f700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6af4053af0 0x7f6af4055fb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f6b1000d000 tx=0x7f6b10007c50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:37.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.273+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6b14064e40 con 0x7f6b140ab470 2022-07-20T03:13:37.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.437+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f6b14064e40 con 0x7f6b140ab470 2022-07-20T03:13:37.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.449+0000 7f6b027fc700 1 -- 172.21.15.61:0/2299641425 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 13 and data pool 14 v39) v1 ==== 189+0+0 (secure 0 0 0) 0x7f6b14064e40 con 0x7f6b140ab470 2022-07-20T03:13:37.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6af4053af0 msgr2=0x7f6af4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6af4053af0 0x7f6af4055fb0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7f6b1000d000 tx=0x7f6b10007c50).stop 2022-07-20T03:13:37.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 msgr2=0x7f6b1413f820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b1413f820 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f6b04002f70 tx=0x7f6b04015000).stop 2022-07-20T03:13:37.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 shutdown_connections 2022-07-20T03:13:37.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6af4053af0 0x7f6af4055fb0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6b140b3560 0x7f6b1413fd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6b140ab470 0x7f6b1413f820 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 --2- 172.21.15.61:0/2299641425 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6b14061f30 0x7f6b14155840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 >> 172.21.15.61:0/2299641425 conn(0x7f6b140b6b20 msgr2=0x7f6b14141210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:37.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 shutdown_connections 2022-07-20T03:13:37.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.457+0000 7f6b1a091700 1 -- 172.21.15.61:0/2299641425 wait complete. 2022-07-20T03:13:37.465 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 13 and data pool 14 2022-07-20T03:13:37.509 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:13:37.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 Processor -- start 2022-07-20T03:13:37.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 -- start start 2022-07-20T03:13:37.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb7601424e0 0x7fb7600b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600aea10 0x7fb7600ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb7601433d0 con 0x7fb7600b0570 2022-07-20T03:13:37.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb760143b10 con 0x7fb7601424e0 2022-07-20T03:13:37.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb76574c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb760143670 con 0x7fb7600aea10 2022-07-20T03:13:37.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb7600b0570 0x7fb760141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43214/0 (socket says 172.21.15.61:43214) 2022-07-20T03:13:37.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75ffff700 1 -- 172.21.15.61:0/206375394 learned_addr learned my addr 172.21.15.61:0/206375394 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:37.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb764f4b700 1 --2- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600aea10 0x7fb7600ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75f7fe700 1 --2- 172.21.15.61:0/206375394 >> v2:172.21.15.117:3300/0 conn(0x7fb7601424e0 0x7fb7600b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75effd700 1 -- 172.21.15.61:0/206375394 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1103000483 0 0) 0x7fb7601433d0 con 0x7fb7600b0570 2022-07-20T03:13:37.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75effd700 1 -- 172.21.15.61:0/206375394 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb740003260 con 0x7fb7600b0570 2022-07-20T03:13:37.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75f7fe700 1 -- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600aea10 msgr2=0x7fb7600ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:37.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75f7fe700 1 --2- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600aea10 0x7fb7600ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75f7fe700 1 -- 172.21.15.61:0/206375394 >> v1:172.21.15.61:6789/0 conn(0x7fb7600b0570 legacy=0x7fb760141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75f7fe700 1 -- 172.21.15.61:0/206375394 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7600b1010 con 0x7fb7601424e0 2022-07-20T03:13:37.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75f7fe700 1 --2- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb7600b0950 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fb7580032f0 tx=0x7fb7580091d0).ready entity=mon.1 client_cookie=b2c3edc6ef525515 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:37.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75effd700 1 -- 172.21.15.61:0/206375394 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75800b600 con 0x7fb7601424e0 2022-07-20T03:13:37.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75effd700 1 -- 172.21.15.61:0/206375394 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75800b760 con 0x7fb7601424e0 2022-07-20T03:13:37.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.645+0000 7fb75effd700 1 -- 172.21.15.61:0/206375394 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75800b8d0 con 0x7fb7601424e0 2022-07-20T03:13:37.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 -- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 msgr2=0x7fb7600b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 --2- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb7600b0950 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fb7580032f0 tx=0x7fb7580091d0).stop 2022-07-20T03:13:37.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 -- 172.21.15.61:0/206375394 shutdown_connections 2022-07-20T03:13:37.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 --2- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600aea10 0x7fb7600ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 --2- 172.21.15.61:0/206375394 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb7600b0950 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 -- 172.21.15.61:0/206375394 >> 172.21.15.61:0/206375394 conn(0x7fb7600b9360 msgr2=0x7fb7600b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 -- 172.21.15.61:0/206375394 shutdown_connections 2022-07-20T03:13:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 -- 172.21.15.61:0/206375394 wait complete. 2022-07-20T03:13:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 Processor -- start 2022-07-20T03:13:37.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.649+0000 7fb76574c700 1 -- start start 2022-07-20T03:13:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 0x7fb7600b1390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600b0570 0x7fb7600b47d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb76013eb70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600b39e0 con 0x7fb7600aea10 2022-07-20T03:13:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600b5100 con 0x7fb7601424e0 2022-07-20T03:13:37.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb7600b1d90 con 0x7fb7600b0570 2022-07-20T03:13:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 0x7fb7600b1390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 0x7fb7600b1390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46622/0 (socket says 172.21.15.61:46622) 2022-07-20T03:13:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 -- 172.21.15.61:0/592810777 learned_addr learned my addr 172.21.15.61:0/592810777 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:37.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb764f4b700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb76013eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75f7fe700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600b0570 0x7fb7600b47d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 -- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600b0570 msgr2=0x7fb7600b47d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600b0570 0x7fb7600b47d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 -- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 msgr2=0x7fb76013eb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb76013eb70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 -- 172.21.15.61:0/592810777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb758009470 con 0x7fb7600aea10 2022-07-20T03:13:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb764f4b700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb76013eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:37.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75ffff700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 0x7fb7600b1390 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fb748001110 tx=0x7fb748007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:37.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7480070d0 con 0x7fb7600aea10 2022-07-20T03:13:37.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb748007230 con 0x7fb7600aea10 2022-07-20T03:13:37.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb748005610 con 0x7fb7600aea10 2022-07-20T03:13:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb7600b2090 con 0x7fb7600aea10 2022-07-20T03:13:37.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb760146610 con 0x7fb7600aea10 2022-07-20T03:13:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.653+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb7600ace30 con 0x7fb7600aea10 2022-07-20T03:13:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.657+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb748010070 con 0x7fb7600aea10 2022-07-20T03:13:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.657+0000 7fb75d7fa700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb744053ad0 0x7fb744055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.657+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb74806b310 con 0x7fb7600aea10 2022-07-20T03:13:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.657+0000 7fb75f7fe700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb744053ad0 0x7fb744055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.657+0000 7fb75f7fe700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb744053ad0 0x7fb744055f90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb75801c000 tx=0x7fb758016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:37.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.661+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb748042270 con 0x7fb7600aea10 2022-07-20T03:13:37.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.789+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb7600574e0 con 0x7fb7600aea10 2022-07-20T03:13:37.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb75d7fa700 1 -- 172.21.15.61:0/592810777 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16102 (secure 0 0 0) 0x7fb748018070 con 0x7fb7600aea10 2022-07-20T03:13:37.805 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:37.805 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":70,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:13:37.441552+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:13:36.009693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T03:13:36.517295+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:13:37.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb744053ad0 msgr2=0x7fb744055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb744053ad0 0x7fb744055f90 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb75801c000 tx=0x7fb758016000).stop 2022-07-20T03:13:37.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 msgr2=0x7fb7600b1390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:37.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 0x7fb7600b1390 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7fb748001110 tx=0x7fb748007aa0).stop 2022-07-20T03:13:37.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 shutdown_connections 2022-07-20T03:13:37.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb744053ad0 0x7fb744055f90 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb7601424e0 0x7fb76013eb70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb7600b0570 0x7fb7600b47d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 --2- 172.21.15.61:0/592810777 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb7600aea10 0x7fb7600b1390 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:37.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 >> 172.21.15.61:0/592810777 conn(0x7fb7600b9360 msgr2=0x7fb760141130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:37.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 shutdown_connections 2022-07-20T03:13:37.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.793+0000 7fb76574c700 1 -- 172.21.15.61:0/592810777 wait complete. 2022-07-20T03:13:37.823 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:13:37.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 Processor -- start 2022-07-20T03:13:37.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- start start 2022-07-20T03:13:37.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37980605a0 0x7f379813f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:37.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3798140940 con 0x7f37980ab180 2022-07-20T03:13:37.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37980b22d0 con 0x7f37980b1ef0 2022-07-20T03:13:37.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f379805fbc0 con 0x7f37980605a0 2022-07-20T03:13:37.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37980605a0 0x7f379813f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:37.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37980605a0 0x7f379813f980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46630/0 (socket says 172.21.15.61:46630) 2022-07-20T03:13:38.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 -- 172.21.15.61:0/2705176291 learned_addr learned my addr 172.21.15.61:0/2705176291 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:38.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 -- 172.21.15.61:0/2705176291 >> v1:172.21.15.117:6789/0 conn(0x7f37980b1ef0 legacy=0x7f37980abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 -- 172.21.15.61:0/2705176291 >> v1:172.21.15.61:6789/0 conn(0x7f37980ab180 legacy=0x7f37980ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 -- 172.21.15.61:0/2705176291 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379805fe60 con 0x7f37980605a0 2022-07-20T03:13:38.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f37967fc700 1 --2- 172.21.15.61:0/2705176291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980605a0 0x7f379813f980 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f37900032f0 tx=0x7f379000b530).ready entity=mon.0 client_cookie=50ffd2ba52cb7eb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3795ffb700 1 -- 172.21.15.61:0/2705176291 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3790010660 con 0x7f37980605a0 2022-07-20T03:13:38.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3795ffb700 1 -- 172.21.15.61:0/2705176291 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37900107c0 con 0x7f37980605a0 2022-07-20T03:13:38.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3795ffb700 1 -- 172.21.15.61:0/2705176291 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3790010b20 con 0x7f37980605a0 2022-07-20T03:13:38.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- 172.21.15.61:0/2705176291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980605a0 msgr2=0x7f379813f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 --2- 172.21.15.61:0/2705176291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980605a0 0x7f379813f980 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f37900032f0 tx=0x7f379000b530).stop 2022-07-20T03:13:38.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- 172.21.15.61:0/2705176291 shutdown_connections 2022-07-20T03:13:38.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 --2- 172.21.15.61:0/2705176291 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980605a0 0x7f379813f980 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- 172.21.15.61:0/2705176291 >> 172.21.15.61:0/2705176291 conn(0x7f37980b2ca0 msgr2=0x7f37980b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:38.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- 172.21.15.61:0/2705176291 shutdown_connections 2022-07-20T03:13:38.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.933+0000 7f3797fff700 1 -- 172.21.15.61:0/2705176291 wait complete. 2022-07-20T03:13:38.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 Processor -- start 2022-07-20T03:13:38.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- start start 2022-07-20T03:13:38.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37980605a0 0x7f379814ea50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 0x7f37980b0010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37980b1ef0 0x7f3798064380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3798064870 con 0x7f37980ab180 2022-07-20T03:13:38.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f379814e070 con 0x7f37980b1ef0 2022-07-20T03:13:38.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f379814e310 con 0x7f37980605a0 2022-07-20T03:13:38.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 0x7f37980b0010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 0x7f37980b0010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46638/0 (socket says 172.21.15.61:46638) 2022-07-20T03:13:38.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 -- 172.21.15.61:0/4054784346 learned_addr learned my addr 172.21.15.61:0/4054784346 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:38.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3796ffd700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37980605a0 0x7f379814ea50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37977fe700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37980b1ef0 0x7f3798064380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 -- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37980605a0 msgr2=0x7f379814ea50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37980605a0 0x7f379814ea50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 -- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37980b1ef0 msgr2=0x7f3798064380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37980b1ef0 0x7f3798064380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 -- 172.21.15.61:0/4054784346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f379000b7d0 con 0x7f37980ab180 2022-07-20T03:13:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3796ffd700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37980605a0 0x7f379814ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f37967fc700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 0x7f37980b0010 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f37900032f0 tx=0x7f3798062870).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379001d070 con 0x7f37980ab180 2022-07-20T03:13:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790018b80 con 0x7f37980ab180 2022-07-20T03:13:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f379814e5b0 con 0x7f37980ab180 2022-07-20T03:13:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379000f3d0 con 0x7f37980ab180 2022-07-20T03:13:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3798141760 con 0x7f37980ab180 2022-07-20T03:13:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.937+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f379814e740 con 0x7f37980ab180 2022-07-20T03:13:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.941+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f379000f530 con 0x7f37980ab180 2022-07-20T03:13:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.941+0000 7f378ffff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f377c053920 0x7f377c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.941+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f379006d7b0 con 0x7f37980ab180 2022-07-20T03:13:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.941+0000 7f3796ffd700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f377c053920 0x7f377c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.941+0000 7f3796ffd700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f377c053920 0x7f377c055de0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f378000d000 tx=0x7f3780009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:37.949+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f37900446b0 con 0x7f37980ab180 2022-07-20T03:13:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.077+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3798060100 con 0x7f37980ab180 2022-07-20T03:13:38.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.077+0000 7f378ffff700 1 -- 172.21.15.61:0/4054784346 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v70) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f3790047cd0 con 0x7f37980ab180 2022-07-20T03:13:38.110 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:38.111 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":70,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:13:37.441552+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:13:36.009693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T03:13:36.517295+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:13:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f377c053920 msgr2=0x7f377c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f377c053920 0x7f377c055de0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f378000d000 tx=0x7f3780009040).stop 2022-07-20T03:13:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 msgr2=0x7f37980b0010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 0x7f37980b0010 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f37900032f0 tx=0x7f3798062870).stop 2022-07-20T03:13:38.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 shutdown_connections 2022-07-20T03:13:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f377c053920 0x7f377c055de0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37980b1ef0 0x7f3798064380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37980ab180 0x7f37980b0010 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 --2- 172.21.15.61:0/4054784346 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37980605a0 0x7f379814ea50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 >> 172.21.15.61:0/4054784346 conn(0x7f37980b2ca0 msgr2=0x7f379813ee70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:38.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 shutdown_connections 2022-07-20T03:13:38.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.081+0000 7f3797fff700 1 -- 172.21.15.61:0/4054784346 wait complete. 2022-07-20T03:13:38.121 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:13:38.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.237+0000 7f67b7fff700 1 Processor -- start 2022-07-20T03:13:38.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.237+0000 7f67b7fff700 1 -- start start 2022-07-20T03:13:38.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f67b813dcc0 0x7f67b8064ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f67b805f840 0x7f67b81408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f67b80af580 con 0x7f67b80b0b90 2022-07-20T03:13:38.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b805ff80 con 0x7f67b813dcc0 2022-07-20T03:13:38.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f67b805c360 con 0x7f67b805f840 2022-07-20T03:13:38.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f67b813dcc0 0x7f67b8064ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f67b813dcc0 0x7f67b8064ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46644/0 (socket says 172.21.15.61:46644) 2022-07-20T03:13:38.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 -- 172.21.15.61:0/922070281 learned_addr learned my addr 172.21.15.61:0/922070281 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:38.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b67fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f67b80b0b90 0x7f67b80aee70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43244/0 (socket says 172.21.15.61:43244) 2022-07-20T03:13:38.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b77fe700 1 --2- 172.21.15.61:0/922070281 >> v2:172.21.15.117:3300/0 conn(0x7f67b805f840 0x7f67b81408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 -- 172.21.15.61:0/922070281 >> v2:172.21.15.117:3300/0 conn(0x7f67b805f840 msgr2=0x7f67b81408d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:38.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 --2- 172.21.15.61:0/922070281 >> v2:172.21.15.117:3300/0 conn(0x7f67b805f840 0x7f67b81408d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 -- 172.21.15.61:0/922070281 >> v1:172.21.15.61:6789/0 conn(0x7f67b80b0b90 legacy=0x7f67b80aee70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 -- 172.21.15.61:0/922070281 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b8064380 con 0x7f67b813dcc0 2022-07-20T03:13:38.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b77fe700 1 --2- 172.21.15.61:0/922070281 >> v2:172.21.15.117:3300/0 conn(0x7f67b805f840 0x7f67b81408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:38.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b6ffd700 1 --2- 172.21.15.61:0/922070281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b8064ac0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f67a8002f70 tx=0x7f67a8006eb0).ready entity=mon.0 client_cookie=8d03be452d74ab22 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b5ffb700 1 -- 172.21.15.61:0/922070281 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67a800d610 con 0x7f67b813dcc0 2022-07-20T03:13:38.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b5ffb700 1 -- 172.21.15.61:0/922070281 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67a8000e90 con 0x7f67b813dcc0 2022-07-20T03:13:38.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b5ffb700 1 -- 172.21.15.61:0/922070281 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67a800d9a0 con 0x7f67b813dcc0 2022-07-20T03:13:38.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- 172.21.15.61:0/922070281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 msgr2=0x7f67b8064ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 --2- 172.21.15.61:0/922070281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b8064ac0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7f67a8002f70 tx=0x7f67a8006eb0).stop 2022-07-20T03:13:38.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- 172.21.15.61:0/922070281 shutdown_connections 2022-07-20T03:13:38.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 --2- 172.21.15.61:0/922070281 >> v2:172.21.15.117:3300/0 conn(0x7f67b805f840 0x7f67b81408d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 --2- 172.21.15.61:0/922070281 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b8064ac0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- 172.21.15.61:0/922070281 >> 172.21.15.61:0/922070281 conn(0x7f67b80632d0 msgr2=0x7f67b80636b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:38.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- 172.21.15.61:0/922070281 shutdown_connections 2022-07-20T03:13:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.241+0000 7f67b7fff700 1 -- 172.21.15.61:0/922070281 wait complete. 2022-07-20T03:13:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 Processor -- start 2022-07-20T03:13:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 -- start start 2022-07-20T03:13:38.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b805f840 0x7f67b8150000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67b80b0b90 0x7f67b813f370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b813b140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b813ffc0 con 0x7f67b813dcc0 2022-07-20T03:13:38.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67b80664c0 con 0x7f67b805f840 2022-07-20T03:13:38.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.245+0000 7f67b7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f67b810e9d0 con 0x7f67b80b0b90 2022-07-20T03:13:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b813b140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b6ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b805f840 0x7f67b8150000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b813b140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46652/0 (socket says 172.21.15.61:46652) 2022-07-20T03:13:38.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 -- 172.21.15.61:0/965962382 learned_addr learned my addr 172.21.15.61:0/965962382 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:38.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b67fc700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67b80b0b90 0x7f67b813f370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 -- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67b80b0b90 msgr2=0x7f67b813f370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67b80b0b90 0x7f67b813f370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 -- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b805f840 msgr2=0x7f67b8150000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b805f840 0x7f67b8150000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 -- 172.21.15.61:0/965962382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67a800b1e0 con 0x7f67b813dcc0 2022-07-20T03:13:38.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b67fc700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67b80b0b90 0x7f67b813f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:38.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b77fe700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b813b140 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f67b0007000 tx=0x7f67b0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b0015070 con 0x7f67b813dcc0 2022-07-20T03:13:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f67b0010580 con 0x7f67b813dcc0 2022-07-20T03:13:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67b8065f40 con 0x7f67b813dcc0 2022-07-20T03:13:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f67b00108e0 con 0x7f67b813dcc0 2022-07-20T03:13:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67b8066c00 con 0x7f67b813dcc0 2022-07-20T03:13:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.249+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67b810ec50 con 0x7f67b813dcc0 2022-07-20T03:13:38.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.253+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f67b0010a40 con 0x7f67b813dcc0 2022-07-20T03:13:38.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.253+0000 7f67a7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6794053a40 0x7f6794055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.253+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(70..70 src has 1..70) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f67b006c180 con 0x7f67b813dcc0 2022-07-20T03:13:38.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.253+0000 7f67b6ffd700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6794053a40 0x7f6794055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.253+0000 7f67b6ffd700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6794053a40 0x7f6794055f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f67a801c000 tx=0x7f67a8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.257+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f67b00430e0 con 0x7f67b813dcc0 2022-07-20T03:13:38.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.417+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f67b8064620 con 0x7f67b813dcc0 2022-07-20T03:13:38.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.449+0000 7f67a7fff700 1 -- 172.21.15.61:0/965962382 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v40) v1 ==== 118+0+0 (secure 0 0 0) 0x7f67b0046700 con 0x7f67b813dcc0 2022-07-20T03:13:38.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.449+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6794053a40 msgr2=0x7f6794055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.449+0000 7f67b7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6794053a40 0x7f6794055f00 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f67a801c000 tx=0x7f67a8015000).stop 2022-07-20T03:13:38.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.449+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 msgr2=0x7f67b813b140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.449+0000 7f67b7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b813b140 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f67b0007000 tx=0x7f67b0008040).stop 2022-07-20T03:13:38.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.449+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 shutdown_connections 2022-07-20T03:13:38.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6794053a40 0x7f6794055f00 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f67b813dcc0 0x7f67b813b140 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f67b80b0b90 0x7f67b813f370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 --2- 172.21.15.61:0/965962382 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f67b805f840 0x7f67b8150000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 >> 172.21.15.61:0/965962382 conn(0x7f67b80632d0 msgr2=0x7f67b80b0180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:38.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 shutdown_connections 2022-07-20T03:13:38.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.453+0000 7f67b7fff700 1 -- 172.21.15.61:0/965962382 wait complete. 2022-07-20T03:13:38.469 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:38.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 Processor -- start 2022-07-20T03:13:38.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- start start 2022-07-20T03:13:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f63c40acef0 0x7f63c414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c405d480 con 0x7f63c40b1e00 2022-07-20T03:13:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63c40b21e0 con 0x7f63c405c3f0 2022-07-20T03:13:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f63c405cf40 con 0x7f63c40acef0 2022-07-20T03:13:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c99ef700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f63c40b1e00 0x7f63c40ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43256/0 (socket says 172.21.15.61:43256) 2022-07-20T03:13:38.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c99ef700 1 -- 172.21.15.61:0/551442371 learned_addr learned my addr 172.21.15.61:0/551442371 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:38.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63ca1f0700 1 --2- 172.21.15.61:0/551442371 >> v2:172.21.15.61:3300/0 conn(0x7f63c40acef0 0x7f63c414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c89ed700 1 -- 172.21.15.61:0/551442371 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3731582149 0 0) 0x7f63c405d480 con 0x7f63c40b1e00 2022-07-20T03:13:38.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c89ed700 1 -- 172.21.15.61:0/551442371 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f63ac003260 con 0x7f63c40b1e00 2022-07-20T03:13:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c89ed700 1 -- 172.21.15.61:0/551442371 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1452349351 0 0) 0x7f63c40b21e0 con 0x7f63c405c3f0 2022-07-20T03:13:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63ca1f0700 1 -- 172.21.15.61:0/551442371 >> v1:172.21.15.117:6789/0 conn(0x7f63c405c3f0 legacy=0x7f63c4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63ca1f0700 1 -- 172.21.15.61:0/551442371 >> v1:172.21.15.61:6789/0 conn(0x7f63c40b1e00 legacy=0x7f63c40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63ca1f0700 1 -- 172.21.15.61:0/551442371 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c405d1e0 con 0x7f63c40acef0 2022-07-20T03:13:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63ca1f0700 1 --2- 172.21.15.61:0/551442371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40acef0 0x7f63c414a6d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f63c0001800 tx=0x7f63bc0091d0).ready entity=mon.0 client_cookie=523e1b358de198ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c89ed700 1 -- 172.21.15.61:0/551442371 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63c0001660 con 0x7f63c40acef0 2022-07-20T03:13:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c89ed700 1 -- 172.21.15.61:0/551442371 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63bc00b580 con 0x7f63c40acef0 2022-07-20T03:13:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63c89ed700 1 -- 172.21.15.61:0/551442371 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63bc00b6f0 con 0x7f63c40acef0 2022-07-20T03:13:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- 172.21.15.61:0/551442371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40acef0 msgr2=0x7f63c414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 --2- 172.21.15.61:0/551442371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40acef0 0x7f63c414a6d0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f63c0001800 tx=0x7f63bc0091d0).stop 2022-07-20T03:13:38.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- 172.21.15.61:0/551442371 shutdown_connections 2022-07-20T03:13:38.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 --2- 172.21.15.61:0/551442371 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40acef0 0x7f63c414a6d0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- 172.21.15.61:0/551442371 >> 172.21.15.61:0/551442371 conn(0x7f63c40611c0 msgr2=0x7f63c4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:38.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- 172.21.15.61:0/551442371 shutdown_connections 2022-07-20T03:13:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.633+0000 7f63cbc53700 1 -- 172.21.15.61:0/551442371 wait complete. 2022-07-20T03:13:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 Processor -- start 2022-07-20T03:13:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 -- start start 2022-07-20T03:13:38.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63c40acef0 0x7f63c4143310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 0x7f63c41474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c40a13b0 0x7f63c41424c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c409fc20 con 0x7f63c40b1e00 2022-07-20T03:13:38.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40a1b30 con 0x7f63c40a13b0 2022-07-20T03:13:38.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63c40a1dd0 con 0x7f63c40acef0 2022-07-20T03:13:38.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 0x7f63c41474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c99ef700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63c40acef0 0x7f63c4143310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63ca1f0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c40a13b0 0x7f63c41424c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 0x7f63c41474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46666/0 (socket says 172.21.15.61:46666) 2022-07-20T03:13:38.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 -- 172.21.15.61:0/1933245103 learned_addr learned my addr 172.21.15.61:0/1933245103 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:38.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 -- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63c40acef0 msgr2=0x7f63c4143310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63c40acef0 0x7f63c4143310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 -- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c40a13b0 msgr2=0x7f63c41424c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c40a13b0 0x7f63c41424c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 -- 172.21.15.61:0/1933245103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63bc009470 con 0x7f63c40b1e00 2022-07-20T03:13:38.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63c91ee700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 0x7f63c41474b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f63b4000ed0 tx=0x7f63b4007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63b4012400 con 0x7f63c40b1e00 2022-07-20T03:13:38.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f63b4012560 con 0x7f63c40b1e00 2022-07-20T03:13:38.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f63b40128c0 con 0x7f63c40b1e00 2022-07-20T03:13:38.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63c414a790 con 0x7f63c40b1e00 2022-07-20T03:13:38.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.637+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63c4149e00 con 0x7f63c40b1e00 2022-07-20T03:13:38.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.641+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63c40a1790 con 0x7f63c40b1e00 2022-07-20T03:13:38.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.641+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f63b4019030 con 0x7f63c40b1e00 2022-07-20T03:13:38.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.641+0000 7f63b2ffd700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63a4053a40 0x7f63a4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:38.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.641+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f63b406b0e0 con 0x7f63c40b1e00 2022-07-20T03:13:38.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.641+0000 7f63c99ef700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63a4053a40 0x7f63a4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:38.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.641+0000 7f63c99ef700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63a4053a40 0x7f63a4055f00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f63c000d000 tx=0x7f63c0000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:38.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.649+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63c40a1790 con 0x7f63c40b1e00 2022-07-20T03:13:38.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.809+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f63c40a1790 con 0x7f63c40b1e00 2022-07-20T03:13:38.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63b2ffd700 1 -- 172.21.15.61:0/1933245103 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f63c40a1790 con 0x7f63c40b1e00 2022-07-20T03:13:38.819 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:38.820 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:13:38.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63a4053a40 msgr2=0x7f63a4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63a4053a40 0x7f63a4055f00 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f63c000d000 tx=0x7f63c0000bc0).stop 2022-07-20T03:13:38.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 msgr2=0x7f63c41474b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:38.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 0x7f63c41474b0 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7f63b4000ed0 tx=0x7f63b4007a90).stop 2022-07-20T03:13:38.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 shutdown_connections 2022-07-20T03:13:38.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63a4053a40 0x7f63a4055f00 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63c40a13b0 0x7f63c41424c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63c40b1e00 0x7f63c41474b0 unknown :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 --2- 172.21.15.61:0/1933245103 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f63c40acef0 0x7f63c4143310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:38.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 >> 172.21.15.61:0/1933245103 conn(0x7f63c405c3f0 msgr2=0x7f63c4144de0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:38.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 shutdown_connections 2022-07-20T03:13:38.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.813+0000 7f63cbc53700 1 -- 172.21.15.61:0/1933245103 wait complete. 2022-07-20T03:13:38.831 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:13:38.856 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:13:39.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 Processor -- start 2022-07-20T03:13:39.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- start start 2022-07-20T03:13:39.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9ca80b5650 0x7f9ca80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9ca8147010 con 0x7f9ca80af8e0 2022-07-20T03:13:39.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9ca81472b0 con 0x7f9ca80b5650 2022-07-20T03:13:39.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8146d40 con 0x7f9ca80adff0 2022-07-20T03:13:39.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cad814700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9ca80af8e0 0x7f9ca805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43274/0 (socket says 172.21.15.61:43274) 2022-07-20T03:13:39.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cad814700 1 -- 172.21.15.61:0/2742037644 learned_addr learned my addr 172.21.15.61:0/2742037644 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:39.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cad013700 1 --2- 172.21.15.61:0/2742037644 >> v2:172.21.15.61:3300/0 conn(0x7f9ca80b5650 0x7f9ca80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cac812700 1 --2- 172.21.15.61:0/2742037644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9c97fff700 1 -- 172.21.15.61:0/2742037644 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 710270852 0 0) 0x7f9ca8147010 con 0x7f9ca80af8e0 2022-07-20T03:13:39.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9c97fff700 1 -- 172.21.15.61:0/2742037644 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c90003260 con 0x7f9ca80af8e0 2022-07-20T03:13:39.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cac812700 1 -- 172.21.15.61:0/2742037644 >> v2:172.21.15.61:3300/0 conn(0x7f9ca80b5650 msgr2=0x7f9ca80b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:39.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cac812700 1 --2- 172.21.15.61:0/2742037644 >> v2:172.21.15.61:3300/0 conn(0x7f9ca80b5650 0x7f9ca80b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cac812700 1 -- 172.21.15.61:0/2742037644 >> v1:172.21.15.61:6789/0 conn(0x7f9ca80af8e0 legacy=0x7f9ca805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cac812700 1 -- 172.21.15.61:0/2742037644 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ca8147c40 con 0x7f9ca80adff0 2022-07-20T03:13:39.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cac812700 1 --2- 172.21.15.61:0/2742037644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8147750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9c98002f70 tx=0x7f9c98006d90).ready entity=mon.2 client_cookie=77cdba8de7f05a2e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9c97fff700 1 -- 172.21.15.61:0/2742037644 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c9800e590 con 0x7f9ca80adff0 2022-07-20T03:13:39.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9c97fff700 1 -- 172.21.15.61:0/2742037644 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c98000e90 con 0x7f9ca80adff0 2022-07-20T03:13:39.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9c97fff700 1 -- 172.21.15.61:0/2742037644 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c9800e8e0 con 0x7f9ca80adff0 2022-07-20T03:13:39.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- 172.21.15.61:0/2742037644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 msgr2=0x7f9ca8147750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 --2- 172.21.15.61:0/2742037644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8147750 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f9c98002f70 tx=0x7f9c98006d90).stop 2022-07-20T03:13:39.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- 172.21.15.61:0/2742037644 shutdown_connections 2022-07-20T03:13:39.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 --2- 172.21.15.61:0/2742037644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8147750 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 --2- 172.21.15.61:0/2742037644 >> v2:172.21.15.61:3300/0 conn(0x7f9ca80b5650 0x7f9ca80b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- 172.21.15.61:0/2742037644 >> 172.21.15.61:0/2742037644 conn(0x7f9ca80634e0 msgr2=0x7f9ca80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- 172.21.15.61:0/2742037644 shutdown_connections 2022-07-20T03:13:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.973+0000 7f9cae015700 1 -- 172.21.15.61:0/2742037644 wait complete. 2022-07-20T03:13:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 Processor -- start 2022-07-20T03:13:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 -- start start 2022-07-20T03:13:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8156fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca80af8e0 0x7f9ca8148720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ca80b5650 0x7f9ca8144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca81565d0 con 0x7f9ca80b5650 2022-07-20T03:13:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8156870 con 0x7f9ca80af8e0 2022-07-20T03:13:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9ca8156b10 con 0x7f9ca80adff0 2022-07-20T03:13:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8156fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad814700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ca80b5650 0x7f9ca8144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8156fb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33350/0 (socket says 172.21.15.61:33350) 2022-07-20T03:13:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 -- 172.21.15.61:0/2526069058 learned_addr learned my addr 172.21.15.61:0/2526069058 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cac812700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca80af8e0 0x7f9ca8148720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 -- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca80af8e0 msgr2=0x7f9ca8148720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca80af8e0 0x7f9ca8148720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 -- 172.21.15.61:0/2526069058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ca80b5650 msgr2=0x7f9ca8144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ca80b5650 0x7f9ca8144ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 -- 172.21.15.61:0/2526069058 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9800b120 con 0x7f9ca80adff0 2022-07-20T03:13:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad814700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ca80b5650 0x7f9ca8144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cad013700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8156fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9ca400c000 tx=0x7f9ca4006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca4011750 con 0x7f9ca80adff0 2022-07-20T03:13:39.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ca40118b0 con 0x7f9ca80adff0 2022-07-20T03:13:39.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ca8148c10 con 0x7f9ca80adff0 2022-07-20T03:13:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ca4011c10 con 0x7f9ca80adff0 2022-07-20T03:13:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.977+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ca81561e0 con 0x7f9ca80adff0 2022-07-20T03:13:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.981+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ca8156cd0 con 0x7f9ca80adff0 2022-07-20T03:13:39.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.981+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9ca4011d70 con 0x7f9ca80adff0 2022-07-20T03:13:39.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.981+0000 7f9c967fc700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c880539d0 0x7f9c88055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.981+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9ca406c350 con 0x7f9ca80adff0 2022-07-20T03:13:39.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.981+0000 7f9cac812700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c880539d0 0x7f9c88055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.981+0000 7f9cac812700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c880539d0 0x7f9c88055e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f9c9801a000 tx=0x7f9c98014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:38.993+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9ca40431b0 con 0x7f9ca80adff0 2022-07-20T03:13:39.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.121+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9ca805c4a0 con 0x7f9ca80adff0 2022-07-20T03:13:39.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.121+0000 7f9c967fc700 1 -- 172.21.15.61:0/2526069058 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f9ca40467d0 con 0x7f9ca80adff0 2022-07-20T03:13:39.165 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:39.166 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":71,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:13:38.445600+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:13:36.009693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T03:13:36.517295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:13:39.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c880539d0 msgr2=0x7f9c88055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c880539d0 0x7f9c88055e90 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f9c9801a000 tx=0x7f9c98014000).stop 2022-07-20T03:13:39.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 msgr2=0x7f9ca8156fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8156fb0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f9ca400c000 tx=0x7f9ca4006df0).stop 2022-07-20T03:13:39.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 shutdown_connections 2022-07-20T03:13:39.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c880539d0 0x7f9c88055e90 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9ca80af8e0 0x7f9ca8148720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9ca80b5650 0x7f9ca8144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 --2- 172.21.15.61:0/2526069058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9ca80adff0 0x7f9ca8156fb0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 >> 172.21.15.61:0/2526069058 conn(0x7f9ca80634e0 msgr2=0x7f9ca80b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 shutdown_connections 2022-07-20T03:13:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.125+0000 7f9cae015700 1 -- 172.21.15.61:0/2526069058 wait complete. 2022-07-20T03:13:39.188 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:13:39.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 Processor -- start 2022-07-20T03:13:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 -- start start 2022-07-20T03:13:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1d6c05cf40 0x7f1d6c146a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c0b5830 0x7f1d6c142c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d6c143120 0x7f1d6c0af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b0130 con 0x7f1d6c143120 2022-07-20T03:13:39.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1d6c1462d0 con 0x7f1d6c05cf40 2022-07-20T03:13:39.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d72efc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c146570 con 0x7f1d6c0b5830 2022-07-20T03:13:39.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d6c143120 0x7f1d6c0af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1d6c143120 0x7f1d6c0af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46690/0 (socket says 172.21.15.61:46690) 2022-07-20T03:13:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 -- 172.21.15.61:0/3778212474 learned_addr learned my addr 172.21.15.61:0/3778212474 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d6bfff700 1 --2- 172.21.15.61:0/3778212474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c0b5830 0x7f1d6c142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d70c98700 1 --2- 172.21.15.61:0/3778212474 >> v2:172.21.15.117:3300/0 conn(0x7f1d6c05cf40 0x7f1d6c146a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 -- 172.21.15.61:0/3778212474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c0b5830 msgr2=0x7f1d6c142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 --2- 172.21.15.61:0/3778212474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c0b5830 0x7f1d6c142c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 -- 172.21.15.61:0/3778212474 >> v2:172.21.15.117:3300/0 conn(0x7f1d6c05cf40 msgr2=0x7f1d6c146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:39.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 --2- 172.21.15.61:0/3778212474 >> v2:172.21.15.117:3300/0 conn(0x7f1d6c05cf40 0x7f1d6c146a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 -- 172.21.15.61:0/3778212474 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d6c061280 con 0x7f1d6c143120 2022-07-20T03:13:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d71499700 1 --2- 172.21.15.61:0/3778212474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c143120 0x7f1d6c0af2c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f1d640032f0 tx=0x7f1d64009290).ready entity=mon.0 client_cookie=6c67fa598b2be1d0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d6b7fe700 1 -- 172.21.15.61:0/3778212474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6400b680 con 0x7f1d6c143120 2022-07-20T03:13:39.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d6b7fe700 1 -- 172.21.15.61:0/3778212474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d6400b7e0 con 0x7f1d6c143120 2022-07-20T03:13:39.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.321+0000 7f1d6b7fe700 1 -- 172.21.15.61:0/3778212474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d6400b950 con 0x7f1d6c143120 2022-07-20T03:13:39.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 -- 172.21.15.61:0/3778212474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c143120 msgr2=0x7f1d6c0af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 --2- 172.21.15.61:0/3778212474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c143120 0x7f1d6c0af2c0 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f1d640032f0 tx=0x7f1d64009290).stop 2022-07-20T03:13:39.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 -- 172.21.15.61:0/3778212474 shutdown_connections 2022-07-20T03:13:39.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 --2- 172.21.15.61:0/3778212474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c143120 0x7f1d6c0af2c0 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 --2- 172.21.15.61:0/3778212474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c0b5830 0x7f1d6c142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 --2- 172.21.15.61:0/3778212474 >> v2:172.21.15.117:3300/0 conn(0x7f1d6c05cf40 0x7f1d6c146a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 -- 172.21.15.61:0/3778212474 >> 172.21.15.61:0/3778212474 conn(0x7f1d6c0b69e0 msgr2=0x7f1d6c066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:39.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 -- 172.21.15.61:0/3778212474 shutdown_connections 2022-07-20T03:13:39.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 -- 172.21.15.61:0/3778212474 wait complete. 2022-07-20T03:13:39.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 Processor -- start 2022-07-20T03:13:39.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.325+0000 7f1d72efc700 1 -- start start 2022-07-20T03:13:39.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d6c05cf40 0x7f1d6c0b2780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 0x7f1d6c13c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c143120 0x7f1d6c13cba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b18f0 con 0x7f1d6c0b5830 2022-07-20T03:13:39.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b3220 con 0x7f1d6c05cf40 2022-07-20T03:13:39.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1d6c0b36c0 con 0x7f1d6c143120 2022-07-20T03:13:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 0x7f1d6c13c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 0x7f1d6c13c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46694/0 (socket says 172.21.15.61:46694) 2022-07-20T03:13:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 -- 172.21.15.61:0/1392382113 learned_addr learned my addr 172.21.15.61:0/1392382113 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d71499700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c143120 0x7f1d6c13cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d70c98700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d6c05cf40 0x7f1d6c0b2780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 -- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c143120 msgr2=0x7f1d6c13cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c143120 0x7f1d6c13cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 -- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d6c05cf40 msgr2=0x7f1d6c0b2780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d6c05cf40 0x7f1d6c0b2780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 -- 172.21.15.61:0/1392382113 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d64009530 con 0x7f1d6c0b5830 2022-07-20T03:13:39.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d6bfff700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 0x7f1d6c13c6b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f1d6000c000 tx=0x7f1d60006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d60010780 con 0x7f1d6c0b5830 2022-07-20T03:13:39.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d600108e0 con 0x7f1d6c0b5830 2022-07-20T03:13:39.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d6c13d090 con 0x7f1d6c0b5830 2022-07-20T03:13:39.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d60010c40 con 0x7f1d6c0b5830 2022-07-20T03:13:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.329+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d6c09f3b0 con 0x7f1d6c0b5830 2022-07-20T03:13:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.333+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d6c0b1a30 con 0x7f1d6c0b5830 2022-07-20T03:13:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.333+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1d6001c460 con 0x7f1d6c0b5830 2022-07-20T03:13:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.333+0000 7f1d69ffb700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d4c053920 0x7f1d4c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.333+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1d6006c340 con 0x7f1d6c0b5830 2022-07-20T03:13:39.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.333+0000 7f1d70c98700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d4c053920 0x7f1d4c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.333+0000 7f1d70c98700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d4c053920 0x7f1d4c055de0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f1d5c00b000 tx=0x7f1d5c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.337+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d600431a0 con 0x7f1d6c0b5830 2022-07-20T03:13:39.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.501+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f1d6c05d320 con 0x7f1d6c0b5830 2022-07-20T03:13:39.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d69ffb700 1 -- 172.21.15.61:0/1392382113 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v25) v1 ==== 184+0+0 (secure 0 0 0) 0x7f1d600467c0 con 0x7f1d6c0b5830 2022-07-20T03:13:39.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d4c053920 msgr2=0x7f1d4c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d4c053920 0x7f1d4c055de0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f1d5c00b000 tx=0x7f1d5c006df0).stop 2022-07-20T03:13:39.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 msgr2=0x7f1d6c13c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 0x7f1d6c13c6b0 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7f1d6000c000 tx=0x7f1d60006df0).stop 2022-07-20T03:13:39.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 shutdown_connections 2022-07-20T03:13:39.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d4c053920 0x7f1d4c055de0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1d6c143120 0x7f1d6c13cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1d6c0b5830 0x7f1d6c13c6b0 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 --2- 172.21.15.61:0/1392382113 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1d6c05cf40 0x7f1d6c0b2780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 >> 172.21.15.61:0/1392382113 conn(0x7f1d6c0b69e0 msgr2=0x7f1d6c0af280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:39.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.509+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 shutdown_connections 2022-07-20T03:13:39.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.513+0000 7f1d72efc700 1 -- 172.21.15.61:0/1392382113 wait complete. 2022-07-20T03:13:39.523 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:13:39.550 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:13:39.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 Processor -- start 2022-07-20T03:13:39.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 -- start start 2022-07-20T03:13:39.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d6c0b3cb0 0x7f6d6c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6d6c0ae280 con 0x7f6d6c0b6800 2022-07-20T03:13:39.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6d6c0b6be0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d722ba700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c0ae4b0 con 0x7f6d6c0ad0e0 2022-07-20T03:13:39.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d6c0b3cb0 0x7f6d6c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6d6c0b3cb0 0x7f6d6c0aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46702/0 (socket says 172.21.15.61:46702) 2022-07-20T03:13:39.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 -- 172.21.15.61:0/19437765 learned_addr learned my addr 172.21.15.61:0/19437765 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:39.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d71ab9700 1 --2- 172.21.15.61:0/19437765 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 -- 172.21.15.61:0/19437765 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 msgr2=0x7f6d6c1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:39.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 --2- 172.21.15.61:0/19437765 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c1418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 -- 172.21.15.61:0/19437765 >> v1:172.21.15.117:6789/0 conn(0x7f6d6c0b6800 legacy=0x7f6d6c145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 -- 172.21.15.61:0/19437765 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d6c0ae750 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d70ab7700 1 --2- 172.21.15.61:0/19437765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c0aebf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f6d5c002f70 tx=0x7f6d5c006eb0).ready entity=mon.0 client_cookie=6fb238a5fc830ccd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d63fff700 1 -- 172.21.15.61:0/19437765 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c00d570 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d63fff700 1 -- 172.21.15.61:0/19437765 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c000e90 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.725+0000 7f6d63fff700 1 -- 172.21.15.61:0/19437765 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c00d900 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 -- 172.21.15.61:0/19437765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 msgr2=0x7f6d6c0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 --2- 172.21.15.61:0/19437765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c0aebf0 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7f6d5c002f70 tx=0x7f6d5c006eb0).stop 2022-07-20T03:13:39.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 -- 172.21.15.61:0/19437765 shutdown_connections 2022-07-20T03:13:39.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 --2- 172.21.15.61:0/19437765 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 --2- 172.21.15.61:0/19437765 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c0aebf0 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 -- 172.21.15.61:0/19437765 >> 172.21.15.61:0/19437765 conn(0x7f6d6c0b9dd0 msgr2=0x7f6d6c0b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:39.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 -- 172.21.15.61:0/19437765 shutdown_connections 2022-07-20T03:13:39.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 -- 172.21.15.61:0/19437765 wait complete. 2022-07-20T03:13:39.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 Processor -- start 2022-07-20T03:13:39.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.729+0000 7f6d722ba700 1 -- start start 2022-07-20T03:13:39.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c113750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d6c0b6800 0x7f6d6c0b0170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c13f9a0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c156030 con 0x7f6d6c0b6800 2022-07-20T03:13:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6d6c1562d0 con 0x7f6d6c0ad0e0 2022-07-20T03:13:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d712b8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c113750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d712b8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c113750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33378/0 (socket says 172.21.15.61:33378) 2022-07-20T03:13:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d712b8700 1 -- 172.21.15.61:0/278148784 learned_addr learned my addr 172.21.15.61:0/278148784 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:39.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d71ab9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d6c0b6800 0x7f6d6c0b0170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 -- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 msgr2=0x7f6d6c113750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c113750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 -- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d6c0b6800 msgr2=0x7f6d6c0b0170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d6c0b6800 0x7f6d6c0b0170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 -- 172.21.15.61:0/278148784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6d5c00b1e0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d70ab7700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c155680 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f6d5c002f70 tx=0x7f6d5c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c011760 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6d5c0118c0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6d6c0b1be0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6d5c011c20 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.733+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6d6c155b70 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.737+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6d6c0b2d40 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.737+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6d5c01a040 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.737+0000 7f6d627fc700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d54053af0 0x7f6d54055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:39.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.737+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6d5c06cb50 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.737+0000 7f6d712b8700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d54053af0 0x7f6d54055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:39.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.737+0000 7f6d712b8700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d54053af0 0x7f6d54055fb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6d6800d000 tx=0x7f6d68009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:39.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.741+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6d5c043bf0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.905+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f6d6c0b5530 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d627fc700 1 -- 172.21.15.61:0/278148784 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v26) v1 ==== 184+0+0 (secure 0 0 0) 0x7f6d5c0230e0 con 0x7f6d6c0b3cb0 2022-07-20T03:13:39.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d54053af0 msgr2=0x7f6d54055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d54053af0 0x7f6d54055fb0 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6d6800d000 tx=0x7f6d68009040).stop 2022-07-20T03:13:39.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 msgr2=0x7f6d6c155680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:39.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c155680 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f6d5c002f70 tx=0x7f6d5c015000).stop 2022-07-20T03:13:39.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 shutdown_connections 2022-07-20T03:13:39.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d54053af0 0x7f6d54055fb0 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6d6c0b6800 0x7f6d6c0b0170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6d6c0b3cb0 0x7f6d6c155680 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 --2- 172.21.15.61:0/278148784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6d6c0ad0e0 0x7f6d6c113750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:39.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 >> 172.21.15.61:0/278148784 conn(0x7f6d6c0b9dd0 msgr2=0x7f6d6c140290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:39.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 shutdown_connections 2022-07-20T03:13:39.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:39.913+0000 7f6d722ba700 1 -- 172.21.15.61:0/278148784 wait complete. 2022-07-20T03:13:39.927 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:13:39.929 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:13:40.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d18dc700 1 Processor -- start 2022-07-20T03:13:40.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d18dc700 1 -- start start 2022-07-20T03:13:40.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d18dc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f38cc14f9d0 0x7f38cc151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d18dc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc0b0b80 con 0x7f38cc0af8c0 2022-07-20T03:13:40.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d18dc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f38cc05d4d0 con 0x7f38cc0b5970 2022-07-20T03:13:40.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d18dc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f38cc0b5d50 con 0x7f38cc14f9d0 2022-07-20T03:13:40.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f38cc0af8c0 0x7f38cc0ae810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43314/0 (socket says 172.21.15.61:43314) 2022-07-20T03:13:40.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cbfff700 1 -- 172.21.15.61:0/223337806 learned_addr learned my addr 172.21.15.61:0/223337806 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:40.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d10db700 1 --2- 172.21.15.61:0/223337806 >> v2:172.21.15.61:3300/0 conn(0x7f38cc14f9d0 0x7f38cc151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cb7fe700 1 -- 172.21.15.61:0/223337806 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2499635460 0 0) 0x7f38cc0b0b80 con 0x7f38cc0af8c0 2022-07-20T03:13:40.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cb7fe700 1 -- 172.21.15.61:0/223337806 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f38b8003260 con 0x7f38cc0af8c0 2022-07-20T03:13:40.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d10db700 1 -- 172.21.15.61:0/223337806 >> v1:172.21.15.117:6789/0 conn(0x7f38cc0b5970 legacy=0x7f38cc0ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cb7fe700 1 -- 172.21.15.61:0/223337806 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 899922125 0 0) 0x7f38b8003260 con 0x7f38cc0af8c0 2022-07-20T03:13:40.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d10db700 1 -- 172.21.15.61:0/223337806 >> v1:172.21.15.61:6789/0 conn(0x7f38cc0af8c0 legacy=0x7f38cc0ae810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d10db700 1 -- 172.21.15.61:0/223337806 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38cc0b0db0 con 0x7f38cc14f9d0 2022-07-20T03:13:40.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38d10db700 1 --2- 172.21.15.61:0/223337806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc14f9d0 0x7f38cc151dc0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f38c0001800 tx=0x7f38c40091d0).ready entity=mon.0 client_cookie=a5e7a32e7d78ae4f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:40.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cb7fe700 1 -- 172.21.15.61:0/223337806 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c0001660 con 0x7f38cc14f9d0 2022-07-20T03:13:40.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cb7fe700 1 -- 172.21.15.61:0/223337806 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38c400b580 con 0x7f38cc14f9d0 2022-07-20T03:13:40.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.065+0000 7f38cb7fe700 1 -- 172.21.15.61:0/223337806 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38c400b6f0 con 0x7f38cc14f9d0 2022-07-20T03:13:40.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 -- 172.21.15.61:0/223337806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc14f9d0 msgr2=0x7f38cc151dc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 --2- 172.21.15.61:0/223337806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc14f9d0 0x7f38cc151dc0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f38c0001800 tx=0x7f38c40091d0).stop 2022-07-20T03:13:40.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 -- 172.21.15.61:0/223337806 shutdown_connections 2022-07-20T03:13:40.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 --2- 172.21.15.61:0/223337806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc14f9d0 0x7f38cc151dc0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 -- 172.21.15.61:0/223337806 >> 172.21.15.61:0/223337806 conn(0x7f38cc063490 msgr2=0x7f38cc0b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:40.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 -- 172.21.15.61:0/223337806 shutdown_connections 2022-07-20T03:13:40.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 -- 172.21.15.61:0/223337806 wait complete. 2022-07-20T03:13:40.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 Processor -- start 2022-07-20T03:13:40.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 -- start start 2022-07-20T03:13:40.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 0x7f38cc111980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.069+0000 7f38d18dc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38cc0b5970 0x7f38cc138f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38cc14f9d0 0x7f38cc143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc110010 con 0x7f38cc0af8c0 2022-07-20T03:13:40.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc145cf0 con 0x7f38cc0b5970 2022-07-20T03:13:40.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc145f90 con 0x7f38cc14f9d0 2022-07-20T03:13:40.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 0x7f38cc111980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38cbfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38cc0b5970 0x7f38cc138f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 0x7f38cc111980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46722/0 (socket says 172.21.15.61:46722) 2022-07-20T03:13:40.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 -- 172.21.15.61:0/4265634619 learned_addr learned my addr 172.21.15.61:0/4265634619 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:40.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d10db700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38cc14f9d0 0x7f38cc143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 -- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38cc14f9d0 msgr2=0x7f38cc143980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38cc14f9d0 0x7f38cc143980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 -- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38cc0b5970 msgr2=0x7f38cc138f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38cc0b5970 0x7f38cc138f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 -- 172.21.15.61:0/4265634619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c4009470 con 0x7f38cc0af8c0 2022-07-20T03:13:40.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d08da700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 0x7f38cc111980 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f38cc0b5f80 tx=0x7f38bc007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:40.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38bc012400 con 0x7f38cc0af8c0 2022-07-20T03:13:40.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f38bc012560 con 0x7f38cc0af8c0 2022-07-20T03:13:40.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc146230 con 0x7f38cc0af8c0 2022-07-20T03:13:40.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f38bc0128c0 con 0x7f38cc0af8c0 2022-07-20T03:13:40.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc066f60 con 0x7f38cc0af8c0 2022-07-20T03:13:40.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.073+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc0acae0 con 0x7f38cc0af8c0 2022-07-20T03:13:40.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.077+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f38bc019030 con 0x7f38cc0af8c0 2022-07-20T03:13:40.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.077+0000 7f38c9ffb700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f38ac053ad0 0x7f38ac055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.077+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f38bc06b280 con 0x7f38cc0af8c0 2022-07-20T03:13:40.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.077+0000 7f38cbfff700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f38ac053ad0 0x7f38ac055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.077+0000 7f38cbfff700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f38ac053ad0 0x7f38ac055f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f38c000d000 tx=0x7f38c0000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:40.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.081+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f38bc0420d0 con 0x7f38cc0af8c0 2022-07-20T03:13:40.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.249+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f38cc0574e0 con 0x7f38cc0af8c0 2022-07-20T03:13:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38c9ffb700 1 -- 172.21.15.61:0/4265634619 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v27) v1 ==== 184+0+0 (secure 0 0 0) 0x7f38bc0456f0 con 0x7f38cc0af8c0 2022-07-20T03:13:40.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f38ac053ad0 msgr2=0x7f38ac055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f38ac053ad0 0x7f38ac055f90 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7f38c000d000 tx=0x7f38c0000bc0).stop 2022-07-20T03:13:40.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 msgr2=0x7f38cc111980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 0x7f38cc111980 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f38cc0b5f80 tx=0x7f38bc007c50).stop 2022-07-20T03:13:40.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 shutdown_connections 2022-07-20T03:13:40.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f38ac053ad0 0x7f38ac055f90 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f38cc0b5970 0x7f38cc138f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f38cc14f9d0 0x7f38cc143980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 --2- 172.21.15.61:0/4265634619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f38cc0af8c0 0x7f38cc111980 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 >> 172.21.15.61:0/4265634619 conn(0x7f38cc063490 msgr2=0x7f38cc05c500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:40.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 shutdown_connections 2022-07-20T03:13:40.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.257+0000 7f38d18dc700 1 -- 172.21.15.61:0/4265634619 wait complete. 2022-07-20T03:13:40.270 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:13:40.275 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:40.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 Processor -- start 2022-07-20T03:13:40.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- start start 2022-07-20T03:13:40.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f43840b21f0 0x7f43840b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43840ac860 0x7f4384142f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4384143470 0x7f43841438e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4384143dd0 con 0x7f43840ac860 2022-07-20T03:13:40.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43840ab430 con 0x7f43840b21f0 2022-07-20T03:13:40.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f43840ab6d0 con 0x7f4384143470 2022-07-20T03:13:40.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43840ac860 0x7f4384142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f43840ac860 0x7f4384142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46728/0 (socket says 172.21.15.61:46728) 2022-07-20T03:13:40.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 -- 172.21.15.61:0/1815225901 learned_addr learned my addr 172.21.15.61:0/1815225901 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:40.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f4382ffd700 1 --2- 172.21.15.61:0/1815225901 >> v2:172.21.15.117:3300/0 conn(0x7f43840b21f0 0x7f43840b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43837fe700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4384143470 0x7f43841438e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 -- 172.21.15.61:0/1815225901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4384143470 msgr2=0x7f43841438e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:40.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4384143470 0x7f43841438e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 -- 172.21.15.61:0/1815225901 >> v2:172.21.15.117:3300/0 conn(0x7f43840b21f0 msgr2=0x7f43840b25d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:40.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 --2- 172.21.15.61:0/1815225901 >> v2:172.21.15.117:3300/0 conn(0x7f43840b21f0 0x7f43840b25d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 -- 172.21.15.61:0/1815225901 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43840ab970 con 0x7f43840ac860 2022-07-20T03:13:40.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f4382ffd700 1 --2- 172.21.15.61:0/1815225901 >> v2:172.21.15.117:3300/0 conn(0x7f43840b21f0 0x7f43840b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:40.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43837fe700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4384143470 0x7f43841438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:40.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43827fc700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43840ac860 0x7f4384142f80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f4374002f70 tx=0x7f4374006eb0).ready entity=mon.0 client_cookie=e65bb96d066a1eef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:40.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f4381ffb700 1 -- 172.21.15.61:0/1815225901 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f437400d570 con 0x7f43840ac860 2022-07-20T03:13:40.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f4381ffb700 1 -- 172.21.15.61:0/1815225901 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4374000e90 con 0x7f43840ac860 2022-07-20T03:13:40.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f4381ffb700 1 -- 172.21.15.61:0/1815225901 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f437400d900 con 0x7f43840ac860 2022-07-20T03:13:40.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- 172.21.15.61:0/1815225901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43840ac860 msgr2=0x7f4384142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43840ac860 0x7f4384142f80 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7f4374002f70 tx=0x7f4374006eb0).stop 2022-07-20T03:13:40.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- 172.21.15.61:0/1815225901 shutdown_connections 2022-07-20T03:13:40.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4384143470 0x7f43841438e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- 172.21.15.61:0/1815225901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f43840ac860 0x7f4384142f80 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 --2- 172.21.15.61:0/1815225901 >> v2:172.21.15.117:3300/0 conn(0x7f43840b21f0 0x7f43840b25d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- 172.21.15.61:0/1815225901 >> 172.21.15.61:0/1815225901 conn(0x7f43840b3690 msgr2=0x7f43840b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:40.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- 172.21.15.61:0/1815225901 shutdown_connections 2022-07-20T03:13:40.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.380+0000 7f43894db700 1 -- 172.21.15.61:0/1815225901 wait complete. 2022-07-20T03:13:40.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 Processor -- start 2022-07-20T03:13:40.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 -- start start 2022-07-20T03:13:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43840ac860 0x7f43840af9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43840b21f0 0x7f438413ce90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 0x7f438413d380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4384157750 con 0x7f4384143470 2022-07-20T03:13:40.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f438413dbb0 con 0x7f43840b21f0 2022-07-20T03:13:40.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4384157010 con 0x7f43840ac860 2022-07-20T03:13:40.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 0x7f438413d380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 0x7f438413d380 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46736/0 (socket says 172.21.15.61:46736) 2022-07-20T03:13:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 -- 172.21.15.61:0/2584281781 learned_addr learned my addr 172.21.15.61:0/2584281781 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f4382ffd700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43840ac860 0x7f43840af9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43827fc700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43840b21f0 0x7f438413ce90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 -- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43840ac860 msgr2=0x7f43840af9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43840ac860 0x7f43840af9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 -- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43840b21f0 msgr2=0x7f438413ce90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43840b21f0 0x7f438413ce90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 -- 172.21.15.61:0/2584281781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f437400b1e0 con 0x7f4384143470 2022-07-20T03:13:40.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43827fc700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43840b21f0 0x7f438413ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:40.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43837fe700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 0x7f438413d380 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f437c007000 tx=0x7f437c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:40.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f437c00e510 con 0x7f4384143470 2022-07-20T03:13:40.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f437c00e670 con 0x7f4384143470 2022-07-20T03:13:40.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43841572b0 con 0x7f4384143470 2022-07-20T03:13:40.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f437c00e9d0 con 0x7f4384143470 2022-07-20T03:13:40.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.384+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f438413d870 con 0x7f4384143470 2022-07-20T03:13:40.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.388+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4384157440 con 0x7f4384143470 2022-07-20T03:13:40.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.388+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f437c019030 con 0x7f4384143470 2022-07-20T03:13:40.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.388+0000 7f437bfff700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f435c053920 0x7f435c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:40.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.388+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f437c06bde0 con 0x7f4384143470 2022-07-20T03:13:40.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.388+0000 7f4382ffd700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f435c053920 0x7f435c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:40.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.388+0000 7f4382ffd700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f435c053920 0x7f435c055de0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f436c00c000 tx=0x7f436c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:40.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.396+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f437c042ca0 con 0x7f4384143470 2022-07-20T03:13:40.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.560+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f43840ab0b0 con 0x7f4384143470 2022-07-20T03:13:40.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f437bfff700 1 -- 172.21.15.61:0/2584281781 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f437c0462c0 con 0x7f4384143470 2022-07-20T03:13:40.565 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:40.565 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:13:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f435c053920 msgr2=0x7f435c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f435c053920 0x7f435c055de0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7f436c00c000 tx=0x7f436c008040).stop 2022-07-20T03:13:40.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 msgr2=0x7f438413d380 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 0x7f438413d380 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f437c007000 tx=0x7f437c008040).stop 2022-07-20T03:13:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 shutdown_connections 2022-07-20T03:13:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f435c053920 0x7f435c055de0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f43840b21f0 0x7f438413ce90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4384143470 0x7f438413d380 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 --2- 172.21.15.61:0/2584281781 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43840ac860 0x7f43840af9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 >> 172.21.15.61:0/2584281781 conn(0x7f43840b3690 msgr2=0x7f43841403c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 shutdown_connections 2022-07-20T03:13:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:40.564+0000 7f43894db700 1 -- 172.21.15.61:0/2584281781 wait complete. 2022-07-20T03:13:40.571 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:13:40.781 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:13:37.441965+0000', 'modified': '2022-07-20T03:13:38.444883+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6285': {'gid': 6285, 'name': 'a', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/460461689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 460461689}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 460461689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:13:40.782 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:13:40.782 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:13:40.928 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6285,"want_state":"up:active","state":"up:active","rank_uptime":3.4752368499999999,"mdsmap_epoch":40,"osdmap_epoch":71,"osdmap_epoch_barrier":70,"uptime":8.5608497589999999} 2022-07-20T03:13:40.929 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6285, "mdsmap_epoch": 40, "osdmap_epoch": 71, "osdmap_epoch_barrier": 70, "rank_uptime": 3.47523685, "state": "up:active", "uptime": 8.560849759, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:13:40.930 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:13:41.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 Processor -- start 2022-07-20T03:13:41.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 -- start start 2022-07-20T03:13:41.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd8040b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd8040aea10 0x7fd804141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8041422c0 0x7fd8040ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd804143b10 con 0x7fd8041422c0 2022-07-20T03:13:41.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd804142750 con 0x7fd8040aea10 2022-07-20T03:13:41.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80a02d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8041433d0 con 0x7fd8040b0570 2022-07-20T03:13:41.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8041422c0 0x7fd8040ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd8041422c0 0x7fd8040ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46746/0 (socket says 172.21.15.61:46746) 2022-07-20T03:13:41.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 -- 172.21.15.61:0/2574922792 learned_addr learned my addr 172.21.15.61:0/2574922792 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:41.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80882a700 1 --2- 172.21.15.61:0/2574922792 >> v2:172.21.15.117:3300/0 conn(0x7fd8040aea10 0x7fd804141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80902b700 1 --2- 172.21.15.61:0/2574922792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd8040b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 -- 172.21.15.61:0/2574922792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 msgr2=0x7fd8040b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:41.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 --2- 172.21.15.61:0/2574922792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd8040b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 -- 172.21.15.61:0/2574922792 >> v2:172.21.15.117:3300/0 conn(0x7fd8040aea10 msgr2=0x7fd804141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:41.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 --2- 172.21.15.61:0/2574922792 >> v2:172.21.15.117:3300/0 conn(0x7fd8040aea10 0x7fd804141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 -- 172.21.15.61:0/2574922792 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd804143670 con 0x7fd8041422c0 2022-07-20T03:13:41.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd80982c700 1 --2- 172.21.15.61:0/2574922792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8041422c0 0x7fd8040ad350 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fd7fc0032f0 tx=0x7fd7fc009290).ready entity=mon.0 client_cookie=bb1801505d9f5fdf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd7fbfff700 1 -- 172.21.15.61:0/2574922792 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7fc00b870 con 0x7fd8041422c0 2022-07-20T03:13:41.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd7fbfff700 1 -- 172.21.15.61:0/2574922792 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7fc00b9d0 con 0x7fd8041422c0 2022-07-20T03:13:41.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.036+0000 7fd7fbfff700 1 -- 172.21.15.61:0/2574922792 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7fc00bb40 con 0x7fd8041422c0 2022-07-20T03:13:41.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- 172.21.15.61:0/2574922792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8041422c0 msgr2=0x7fd8040ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- 172.21.15.61:0/2574922792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8041422c0 0x7fd8040ad350 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7fd7fc0032f0 tx=0x7fd7fc009290).stop 2022-07-20T03:13:41.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- 172.21.15.61:0/2574922792 shutdown_connections 2022-07-20T03:13:41.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- 172.21.15.61:0/2574922792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8041422c0 0x7fd8040ad350 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- 172.21.15.61:0/2574922792 >> v2:172.21.15.117:3300/0 conn(0x7fd8040aea10 0x7fd804141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- 172.21.15.61:0/2574922792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd8040b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- 172.21.15.61:0/2574922792 >> 172.21.15.61:0/2574922792 conn(0x7fd8040b9360 msgr2=0x7fd8040b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- 172.21.15.61:0/2574922792 shutdown_connections 2022-07-20T03:13:41.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- 172.21.15.61:0/2574922792 wait complete. 2022-07-20T03:13:41.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 Processor -- start 2022-07-20T03:13:41.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- start start 2022-07-20T03:13:41.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 0x7fd8040b2480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd80413ea40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8041422c0 0x7fd80413ef30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8041469a0 con 0x7fd8040aea10 2022-07-20T03:13:41.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8040b2970 con 0x7fd8041422c0 2022-07-20T03:13:41.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80a02d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd8040b2bb0 con 0x7fd8040b0570 2022-07-20T03:13:41.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 0x7fd8040b2480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 0x7fd8040b2480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46748/0 (socket says 172.21.15.61:46748) 2022-07-20T03:13:41.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 -- 172.21.15.61:0/3312310332 learned_addr learned my addr 172.21.15.61:0/3312310332 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:41.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80882a700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd80413ea40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80982c700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8041422c0 0x7fd80413ef30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 -- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 msgr2=0x7fd80413ea40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd80413ea40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 -- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8041422c0 msgr2=0x7fd80413ef30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8041422c0 0x7fd80413ef30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 -- 172.21.15.61:0/3312310332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7fc009530 con 0x7fd8040aea10 2022-07-20T03:13:41.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.040+0000 7fd80902b700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 0x7fd8040b2480 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fd8000030b0 tx=0x7fd800008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd800011780 con 0x7fd8040aea10 2022-07-20T03:13:41.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd8000118e0 con 0x7fd8040aea10 2022-07-20T03:13:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd804146bd0 con 0x7fd8040aea10 2022-07-20T03:13:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd800011c40 con 0x7fd8040aea10 2022-07-20T03:13:41.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8041465a0 con 0x7fd8040aea10 2022-07-20T03:13:41.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7e0003410 con 0x7fd8040aea10 2022-07-20T03:13:41.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd80001c460 con 0x7fd8040aea10 2022-07-20T03:13:41.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd7fa7fc700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7ec053a40 0x7fd7ec055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd80006c090 con 0x7fd8040aea10 2022-07-20T03:13:41.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd80882a700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7ec053a40 0x7fd7ec055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.044+0000 7fd80882a700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7ec053a40 0x7fd7ec055f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fd7f400b000 tx=0x7fd7f4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.052+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd800042e80 con 0x7fd8040aea10 2022-07-20T03:13:41.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.172+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd7e00036c0 con 0x7fd8040aea10 2022-07-20T03:13:41.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.176+0000 7fd7fa7fc700 1 -- 172.21.15.61:0/3312310332 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16089 (secure 0 0 0) 0x7fd8000464a0 con 0x7fd8040aea10 2022-07-20T03:13:41.183 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:41.184 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":71,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:13:38.445600+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:13:36.009693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T03:13:36.517295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:13:41.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.176+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7ec053a40 msgr2=0x7fd7ec055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.176+0000 7fd80a02d700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7ec053a40 0x7fd7ec055f00 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fd7f400b000 tx=0x7fd7f4006df0).stop 2022-07-20T03:13:41.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.176+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 msgr2=0x7fd8040b2480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.176+0000 7fd80a02d700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 0x7fd8040b2480 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fd8000030b0 tx=0x7fd800008040).stop 2022-07-20T03:13:41.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 shutdown_connections 2022-07-20T03:13:41.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7ec053a40 0x7fd7ec055f00 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd8041422c0 0x7fd80413ef30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd8040b0570 0x7fd80413ea40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 --2- 172.21.15.61:0/3312310332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd8040aea10 0x7fd8040b2480 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 >> 172.21.15.61:0/3312310332 conn(0x7fd8040b9360 msgr2=0x7fd8040acee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 shutdown_connections 2022-07-20T03:13:41.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.180+0000 7fd80a02d700 1 -- 172.21.15.61:0/3312310332 wait complete. 2022-07-20T03:13:41.201 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:13:36.009693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:13:36.517295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:13:41.208 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:13:41.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.320+0000 7f54c9bf0700 1 Processor -- start 2022-07-20T03:13:41.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.320+0000 7f54c9bf0700 1 -- start start 2022-07-20T03:13:41.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f54c413f7d0 0x7f54c40ab9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f54c40ac7e0 0x7f54c413e720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f54c40ac5e0 con 0x7f54c40b1d40 2022-07-20T03:13:41.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c40b2120 con 0x7f54c40ac7e0 2022-07-20T03:13:41.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f54c4060830 con 0x7f54c413f7d0 2022-07-20T03:13:41.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f54c40ac7e0 0x7f54c413e720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f54c40ac7e0 0x7f54c413e720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46760/0 (socket says 172.21.15.61:46760) 2022-07-20T03:13:41.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 -- 172.21.15.61:0/2300043789 learned_addr learned my addr 172.21.15.61:0/2300043789 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:41.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c3fff700 1 --2- 172.21.15.61:0/2300043789 >> v2:172.21.15.117:3300/0 conn(0x7f54c413f7d0 0x7f54c40ab9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 -- 172.21.15.61:0/2300043789 >> v2:172.21.15.117:3300/0 conn(0x7f54c413f7d0 msgr2=0x7f54c40ab9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:41.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 --2- 172.21.15.61:0/2300043789 >> v2:172.21.15.117:3300/0 conn(0x7f54c413f7d0 0x7f54c40ab9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 -- 172.21.15.61:0/2300043789 >> v1:172.21.15.117:6789/0 conn(0x7f54c40b1d40 legacy=0x7f54c40ab2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 -- 172.21.15.61:0/2300043789 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54c4060ad0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c93ef700 1 --2- 172.21.15.61:0/2300043789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c413e720 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f54bc0032f0 tx=0x7f54bc00b530).ready entity=mon.0 client_cookie=8f66eee020000b63 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c37fe700 1 -- 172.21.15.61:0/2300043789 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54bc00ddf0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c37fe700 1 -- 172.21.15.61:0/2300043789 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54bc010470 con 0x7f54c40ac7e0 2022-07-20T03:13:41.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c37fe700 1 -- 172.21.15.61:0/2300043789 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54bc0107d0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2300043789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 msgr2=0x7f54c413e720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2300043789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c413e720 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7f54bc0032f0 tx=0x7f54bc00b530).stop 2022-07-20T03:13:41.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2300043789 shutdown_connections 2022-07-20T03:13:41.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2300043789 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c413e720 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2300043789 >> v2:172.21.15.117:3300/0 conn(0x7f54c413f7d0 0x7f54c40ab9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2300043789 >> 172.21.15.61:0/2300043789 conn(0x7f54c40b2af0 msgr2=0x7f54c40b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2300043789 shutdown_connections 2022-07-20T03:13:41.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.324+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2300043789 wait complete. 2022-07-20T03:13:41.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 Processor -- start 2022-07-20T03:13:41.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 -- start start 2022-07-20T03:13:41.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c41454a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c40b1d40 0x7f54c4150e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c413f7d0 0x7f54c4151350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c40afd10 con 0x7f54c40ac7e0 2022-07-20T03:13:41.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54c414e440 con 0x7f54c40b1d40 2022-07-20T03:13:41.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f54c414e6e0 con 0x7f54c413f7d0 2022-07-20T03:13:41.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c41454a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c41454a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46762/0 (socket says 172.21.15.61:46762) 2022-07-20T03:13:41.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 -- 172.21.15.61:0/2111436118 learned_addr learned my addr 172.21.15.61:0/2111436118 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:41.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c93ef700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c413f7d0 0x7f54c4151350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c3fff700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c40b1d40 0x7f54c4150e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 -- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c413f7d0 msgr2=0x7f54c4151350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c413f7d0 0x7f54c4151350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 -- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c40b1d40 msgr2=0x7f54c4150e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c40b1d40 0x7f54c4150e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 -- 172.21.15.61:0/2111436118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f54bc00b7d0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c8bee700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c41454a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f54c40610c0 tx=0x7f54b8007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54b800ce30 con 0x7f54c40ac7e0 2022-07-20T03:13:41.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f54b800f470 con 0x7f54c40ac7e0 2022-07-20T03:13:41.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54c40ad4d0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f54b800f7d0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.328+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54c4145f60 con 0x7f54c40ac7e0 2022-07-20T03:13:41.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.332+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54c4060d70 con 0x7f54c40ac7e0 2022-07-20T03:13:41.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.332+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f54b8019030 con 0x7f54c40ac7e0 2022-07-20T03:13:41.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.332+0000 7f54c1ffb700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a4053920 0x7f54a4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.332+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f54b806b270 con 0x7f54c40ac7e0 2022-07-20T03:13:41.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.332+0000 7f54c3fff700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a4053920 0x7f54a4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.336+0000 7f54c3fff700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a4053920 0x7f54a4055de0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f54b000b000 tx=0x7f54b0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.340+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f54b80421d0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.464+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f54c40574e0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.464+0000 7f54c1ffb700 1 -- 172.21.15.61:0/2111436118 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v71) v1 ==== 110+0+10 (secure 0 0 0) 0x7f54b80457f0 con 0x7f54c40ac7e0 2022-07-20T03:13:41.490 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:13:41.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a4053920 msgr2=0x7f54a4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a4053920 0x7f54a4055de0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f54b000b000 tx=0x7f54b0008040).stop 2022-07-20T03:13:41.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 msgr2=0x7f54c41454a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c41454a0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7f54c40610c0 tx=0x7f54b8007bf0).stop 2022-07-20T03:13:41.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 shutdown_connections 2022-07-20T03:13:41.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f54a4053920 0x7f54a4055de0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f54c40b1d40 0x7f54c4150e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f54c413f7d0 0x7f54c4151350 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 --2- 172.21.15.61:0/2111436118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f54c40ac7e0 0x7f54c41454a0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 >> 172.21.15.61:0/2111436118 conn(0x7f54c40b2af0 msgr2=0x7f54c40b3e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 shutdown_connections 2022-07-20T03:13:41.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.468+0000 7f54c9bf0700 1 -- 172.21.15.61:0/2111436118 wait complete. 2022-07-20T03:13:41.520 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:13:41.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 Processor -- start 2022-07-20T03:13:41.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 -- start start 2022-07-20T03:13:41.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f89400b2ec0 0x7f89400b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f894005d160 0x7f8940142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89400b3f10 con 0x7f8940142700 2022-07-20T03:13:41.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8940142b90 con 0x7f89400b2ec0 2022-07-20T03:13:41.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.636+0000 7f8945c19700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f89400b4ea0 con 0x7f894005d160 2022-07-20T03:13:41.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f89400b2ec0 0x7f89400b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f89400b2ec0 0x7f89400b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46770/0 (socket says 172.21.15.61:46770) 2022-07-20T03:13:41.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 -- 172.21.15.61:0/3968156241 learned_addr learned my addr 172.21.15.61:0/3968156241 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:41.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f893ffff700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f894005d160 0x7f8940142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 -- 172.21.15.61:0/3968156241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f894005d160 msgr2=0x7f8940142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:41.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f894005d160 0x7f8940142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 -- 172.21.15.61:0/3968156241 >> v1:172.21.15.117:6789/0 conn(0x7f8940142700 legacy=0x7f89401461c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 -- 172.21.15.61:0/3968156241 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89400b3790 con 0x7f89400b2ec0 2022-07-20T03:13:41.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f893ffff700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f894005d160 0x7f8940142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:41.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8944c17700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b2ec0 0x7f89400b32a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f89400b4fe0 tx=0x7f8930009290).ready entity=mon.0 client_cookie=93d7ae279a78f11c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f893f7fe700 1 -- 172.21.15.61:0/3968156241 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f893000b860 con 0x7f89400b2ec0 2022-07-20T03:13:41.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f893f7fe700 1 -- 172.21.15.61:0/3968156241 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893000b9c0 con 0x7f89400b2ec0 2022-07-20T03:13:41.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f893f7fe700 1 -- 172.21.15.61:0/3968156241 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f893000bb30 con 0x7f89400b2ec0 2022-07-20T03:13:41.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 -- 172.21.15.61:0/3968156241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b2ec0 msgr2=0x7f89400b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b2ec0 0x7f89400b32a0 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f89400b4fe0 tx=0x7f8930009290).stop 2022-07-20T03:13:41.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 -- 172.21.15.61:0/3968156241 shutdown_connections 2022-07-20T03:13:41.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f894005d160 0x7f8940142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 --2- 172.21.15.61:0/3968156241 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b2ec0 0x7f89400b32a0 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 -- 172.21.15.61:0/3968156241 >> 172.21.15.61:0/3968156241 conn(0x7f8940066e20 msgr2=0x7f8940067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 -- 172.21.15.61:0/3968156241 shutdown_connections 2022-07-20T03:13:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 -- 172.21.15.61:0/3968156241 wait complete. 2022-07-20T03:13:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 Processor -- start 2022-07-20T03:13:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.640+0000 7f8945c19700 1 -- start start 2022-07-20T03:13:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 0x7f8940140b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400b2ec0 0x7f894013bb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8940142700 0x7f894013c080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89401124c0 con 0x7f894005d160 2022-07-20T03:13:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89401126f0 con 0x7f8940142700 2022-07-20T03:13:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8940112990 con 0x7f89400b2ec0 2022-07-20T03:13:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 0x7f8940140b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 0x7f8940140b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46776/0 (socket says 172.21.15.61:46776) 2022-07-20T03:13:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 -- 172.21.15.61:0/2101561070 learned_addr learned my addr 172.21.15.61:0/2101561070 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f893ffff700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400b2ec0 0x7f894013bb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945418700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8940142700 0x7f894013c080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 -- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400b2ec0 msgr2=0x7f894013bb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400b2ec0 0x7f894013bb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 -- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8940142700 msgr2=0x7f894013c080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8940142700 0x7f894013c080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 -- 172.21.15.61:0/2101561070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8930009530 con 0x7f894005d160 2022-07-20T03:13:41.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8944c17700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 0x7f8940140b70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f89400b4fe0 tx=0x7f8930016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89300032f0 con 0x7f894005d160 2022-07-20T03:13:41.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f893000bda0 con 0x7f894005d160 2022-07-20T03:13:41.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f893001f610 con 0x7f894005d160 2022-07-20T03:13:41.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8940112c30 con 0x7f894005d160 2022-07-20T03:13:41.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89401130d0 con 0x7f894005d160 2022-07-20T03:13:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.644+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8940112dc0 con 0x7f894005d160 2022-07-20T03:13:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.648+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f893000f070 con 0x7f894005d160 2022-07-20T03:13:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.648+0000 7f893dffb700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.648+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f893006c6c0 con 0x7f894005d160 2022-07-20T03:13:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.648+0000 7f893ffff700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.648+0000 7f893ffff700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f892800c000 tx=0x7f8928008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.652+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8930043620 con 0x7f894005d160 2022-07-20T03:13:41.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f89400b2620 con 0x7f894005d160 2022-07-20T03:13:41.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f893dffb700 1 -- 172.21.15.61:0/2101561070 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7f89300240a0 con 0x7f894005d160 2022-07-20T03:13:41.805 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:41.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 msgr2=0x7f8924055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f892800c000 tx=0x7f8928008040).stop 2022-07-20T03:13:41.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 msgr2=0x7f8940140b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 0x7f8940140b70 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f89400b4fe0 tx=0x7f8930016000).stop 2022-07-20T03:13:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 shutdown_connections 2022-07-20T03:13:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8940142700 0x7f894013c080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400b2ec0 0x7f894013bb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 --2- 172.21.15.61:0/2101561070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f894005d160 0x7f8940140b70 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 >> 172.21.15.61:0/2101561070 conn(0x7f8940066e20 msgr2=0x7f894013cc80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 shutdown_connections 2022-07-20T03:13:41.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.784+0000 7f8945c19700 1 -- 172.21.15.61:0/2101561070 wait complete. 2022-07-20T03:13:41.812 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:13:41.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.920+0000 7f7bacda7700 1 Processor -- start 2022-07-20T03:13:41.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- start start 2022-07-20T03:13:41.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7ba8147ab0 0x7f7ba80b0770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba805cc70 con 0x7f7ba80af940 2022-07-20T03:13:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ba80b5dd0 con 0x7f7ba80b59f0 2022-07-20T03:13:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ba805c760 con 0x7f7ba8147ab0 2022-07-20T03:13:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7ba80af940 0x7f7ba80b0060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43384/0 (socket says 172.21.15.61:43384) 2022-07-20T03:13:41.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba6ffd700 1 -- 172.21.15.61:0/3205773978 learned_addr learned my addr 172.21.15.61:0/3205773978 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:41.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba7fff700 1 --2- 172.21.15.61:0/3205773978 >> v2:172.21.15.117:3300/0 conn(0x7f7ba8147ab0 0x7f7ba80b0770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:41.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba67fc700 1 -- 172.21.15.61:0/3205773978 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2354363670 0 0) 0x7f7ba805cc70 con 0x7f7ba80af940 2022-07-20T03:13:41.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba67fc700 1 -- 172.21.15.61:0/3205773978 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7b88003260 con 0x7f7ba80af940 2022-07-20T03:13:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba7fff700 1 -- 172.21.15.61:0/3205773978 >> v1:172.21.15.117:6789/0 conn(0x7f7ba80b59f0 legacy=0x7f7ba81473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba7fff700 1 -- 172.21.15.61:0/3205773978 >> v1:172.21.15.61:6789/0 conn(0x7f7ba80af940 legacy=0x7f7ba80b0060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba7fff700 1 -- 172.21.15.61:0/3205773978 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba805c9d0 con 0x7f7ba8147ab0 2022-07-20T03:13:41.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba7fff700 1 --2- 172.21.15.61:0/3205773978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba8147ab0 0x7f7ba80b0770 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7ba0002f70 tx=0x7f7ba0006df0).ready entity=mon.1 client_cookie=150c39add70d0d8a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:41.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba67fc700 1 -- 172.21.15.61:0/3205773978 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba000d610 con 0x7f7ba8147ab0 2022-07-20T03:13:41.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba67fc700 1 -- 172.21.15.61:0/3205773978 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba0000e90 con 0x7f7ba8147ab0 2022-07-20T03:13:41.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7ba67fc700 1 -- 172.21.15.61:0/3205773978 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba000d960 con 0x7f7ba8147ab0 2022-07-20T03:13:41.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- 172.21.15.61:0/3205773978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba8147ab0 msgr2=0x7f7ba80b0770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:41.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 --2- 172.21.15.61:0/3205773978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba8147ab0 0x7f7ba80b0770 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7ba0002f70 tx=0x7f7ba0006df0).stop 2022-07-20T03:13:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- 172.21.15.61:0/3205773978 shutdown_connections 2022-07-20T03:13:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 --2- 172.21.15.61:0/3205773978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba8147ab0 0x7f7ba80b0770 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:41.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- 172.21.15.61:0/3205773978 >> 172.21.15.61:0/3205773978 conn(0x7f7ba8063480 msgr2=0x7f7ba80b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:41.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- 172.21.15.61:0/3205773978 shutdown_connections 2022-07-20T03:13:41.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.924+0000 7f7bacda7700 1 -- 172.21.15.61:0/3205773978 wait complete. 2022-07-20T03:13:41.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 Processor -- start 2022-07-20T03:13:41.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 -- start start 2022-07-20T03:13:41.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba80af940 0x7f7ba8143a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:41.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ba80b59f0 0x7f7ba8159c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 0x7f7ba8156350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8144570 con 0x7f7ba8147ab0 2022-07-20T03:13:42.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba81433a0 con 0x7f7ba80af940 2022-07-20T03:13:42.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ba8141af0 con 0x7f7ba80b59f0 2022-07-20T03:13:42.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 0x7f7ba8156350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 0x7f7ba8156350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46794/0 (socket says 172.21.15.61:46794) 2022-07-20T03:13:42.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 -- 172.21.15.61:0/4193324427 learned_addr learned my addr 172.21.15.61:0/4193324427 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:42.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba77fe700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba80af940 0x7f7ba8143a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba6ffd700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ba80b59f0 0x7f7ba8159c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 -- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ba80b59f0 msgr2=0x7f7ba8159c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ba80b59f0 0x7f7ba8159c60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 -- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba80af940 msgr2=0x7f7ba8143a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba80af940 0x7f7ba8143a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 -- 172.21.15.61:0/4193324427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ba000b180 con 0x7f7ba8147ab0 2022-07-20T03:13:42.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba7fff700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 0x7f7ba8156350 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f7ba8144380 tx=0x7f7ba0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba0016070 con 0x7f7ba8147ab0 2022-07-20T03:13:42.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ba000d290 con 0x7f7ba8147ab0 2022-07-20T03:13:42.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ba815a6b0 con 0x7f7ba8147ab0 2022-07-20T03:13:42.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ba0013c60 con 0x7f7ba8147ab0 2022-07-20T03:13:42.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.928+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ba815a370 con 0x7f7ba8147ab0 2022-07-20T03:13:42.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.932+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ba8142130 con 0x7f7ba8147ab0 2022-07-20T03:13:42.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.932+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7ba0025030 con 0x7f7ba8147ab0 2022-07-20T03:13:42.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.932+0000 7f7ba4ff9700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b8c053ad0 0x7f7b8c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.932+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7ba006cef0 con 0x7f7ba8147ab0 2022-07-20T03:13:42.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.932+0000 7f7ba77fe700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b8c053ad0 0x7f7b8c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.932+0000 7f7ba77fe700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b8c053ad0 0x7f7b8c055f90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7b9800d000 tx=0x7f7b98007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:41.936+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7ba8142130 con 0x7f7ba8147ab0 2022-07-20T03:13:42.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f7ba8142130 con 0x7f7ba8147ab0 2022-07-20T03:13:42.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7ba4ff9700 1 -- 172.21.15.61:0/4193324427 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+10 (secure 0 0 0) 0x7f7ba8142130 con 0x7f7ba8147ab0 2022-07-20T03:13:42.070 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:42.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b8c053ad0 msgr2=0x7f7b8c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b8c053ad0 0x7f7b8c055f90 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7b9800d000 tx=0x7f7b98007f10).stop 2022-07-20T03:13:42.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 msgr2=0x7f7ba8156350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 0x7f7ba8156350 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f7ba8144380 tx=0x7f7ba0014000).stop 2022-07-20T03:13:42.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 shutdown_connections 2022-07-20T03:13:42.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b8c053ad0 0x7f7b8c055f90 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ba8147ab0 0x7f7ba8156350 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ba80b59f0 0x7f7ba8159c60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.064+0000 7f7bacda7700 1 --2- 172.21.15.61:0/4193324427 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ba80af940 0x7f7ba8143a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.068+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 >> 172.21.15.61:0/4193324427 conn(0x7f7ba8063480 msgr2=0x7f7ba813e4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:42.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.068+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 shutdown_connections 2022-07-20T03:13:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.068+0000 7f7bacda7700 1 -- 172.21.15.61:0/4193324427 wait complete. 2022-07-20T03:13:42.117 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:13:42.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 Processor -- start 2022-07-20T03:13:42.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 -- start start 2022-07-20T03:13:42.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea38142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea38143060 0x7fea3809b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea38147030 con 0x7fea3805d0e0 2022-07-20T03:13:42.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea3805d4c0 con 0x7fea38143060 2022-07-20T03:13:42.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3e4c8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea381461e0 con 0x7fea380b5960 2022-07-20T03:13:42.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea38143060 0x7fea3809b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea38143060 0x7fea3809b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46802/0 (socket says 172.21.15.61:46802) 2022-07-20T03:13:42.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea37fff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fea3805d0e0 0x7fea38146920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:51900/0 (socket says 172.21.15.61:51900) 2022-07-20T03:13:42.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea377fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea38142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 -- 172.21.15.61:0/600120893 learned_addr learned my addr 172.21.15.61:0/600120893 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:42.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 -- 172.21.15.61:0/600120893 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 msgr2=0x7fea38142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:42.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 --2- 172.21.15.61:0/600120893 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea38142b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea36ffd700 1 -- 172.21.15.61:0/600120893 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1297181989 0 0) 0x7fea38147030 con 0x7fea3805d0e0 2022-07-20T03:13:42.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 -- 172.21.15.61:0/600120893 >> v1:172.21.15.117:6789/0 conn(0x7fea3805d0e0 legacy=0x7fea38146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 -- 172.21.15.61:0/600120893 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea38146480 con 0x7fea38143060 2022-07-20T03:13:42.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.248+0000 7fea3ca65700 1 --2- 172.21.15.61:0/600120893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b6d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fea300032f0 tx=0x7fea30009290).ready entity=mon.0 client_cookie=e8fad9c8b5f84e92 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea36ffd700 1 -- 172.21.15.61:0/600120893 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea3000b8a0 con 0x7fea38143060 2022-07-20T03:13:42.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea36ffd700 1 -- 172.21.15.61:0/600120893 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea3000ba00 con 0x7fea38143060 2022-07-20T03:13:42.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea36ffd700 1 -- 172.21.15.61:0/600120893 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea3000bb70 con 0x7fea38143060 2022-07-20T03:13:42.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- 172.21.15.61:0/600120893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 msgr2=0x7fea3809b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/600120893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b6d0 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fea300032f0 tx=0x7fea30009290).stop 2022-07-20T03:13:42.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- 172.21.15.61:0/600120893 shutdown_connections 2022-07-20T03:13:42.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/600120893 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b6d0 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/600120893 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea38142b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- 172.21.15.61:0/600120893 >> 172.21.15.61:0/600120893 conn(0x7fea380b6a20 msgr2=0x7fea38066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:42.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- 172.21.15.61:0/600120893 shutdown_connections 2022-07-20T03:13:42.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- 172.21.15.61:0/600120893 wait complete. 2022-07-20T03:13:42.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 Processor -- start 2022-07-20T03:13:42.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- start start 2022-07-20T03:13:42.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea3805d0e0 0x7fea380b2fd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea3809ae10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3809eb20 con 0x7fea38143060 2022-07-20T03:13:42.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea3809ed50 con 0x7fea3805d0e0 2022-07-20T03:13:42.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3e4c8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea3808f6b0 con 0x7fea380b5960 2022-07-20T03:13:42.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46808/0 (socket says 172.21.15.61:46808) 2022-07-20T03:13:42.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 -- 172.21.15.61:0/429551320 learned_addr learned my addr 172.21.15.61:0/429551320 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:42.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea37fff700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea3805d0e0 0x7fea380b2fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea377fe700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea3809ae10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 -- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 msgr2=0x7fea3809ae10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea3809ae10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 -- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea3805d0e0 msgr2=0x7fea380b2fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea3805d0e0 0x7fea380b2fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.252+0000 7fea3ca65700 1 -- 172.21.15.61:0/429551320 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea30009530 con 0x7fea38143060 2022-07-20T03:13:42.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea3ca65700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b300 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fea300032f0 tx=0x7fea30016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea3000bb70 con 0x7fea38143060 2022-07-20T03:13:42.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea30021630 con 0x7fea38143060 2022-07-20T03:13:42.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea380b34c0 con 0x7fea38143060 2022-07-20T03:13:42.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea3000b8a0 con 0x7fea38143060 2022-07-20T03:13:42.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea3809e720 con 0x7fea38143060 2022-07-20T03:13:42.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea3808f870 con 0x7fea38143060 2022-07-20T03:13:42.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fea30028070 con 0x7fea38143060 2022-07-20T03:13:42.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.256+0000 7fea357fa700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea180539b0 0x7fea18055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.260+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fea3006cdc0 con 0x7fea38143060 2022-07-20T03:13:42.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.260+0000 7fea37fff700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea180539b0 0x7fea18055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.260+0000 7fea37fff700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea180539b0 0x7fea18055e70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fea2800d000 tx=0x7fea280079e0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.264+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea30043d80 con 0x7fea38143060 2022-07-20T03:13:42.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fea3809e8e0 con 0x7fea38143060 2022-07-20T03:13:42.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea357fa700 1 -- 172.21.15.61:0/429551320 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+10 (secure 0 0 0) 0x7fea30015030 con 0x7fea38143060 2022-07-20T03:13:42.394 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:42.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea180539b0 msgr2=0x7fea18055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea180539b0 0x7fea18055e70 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fea2800d000 tx=0x7fea280079e0).stop 2022-07-20T03:13:42.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 msgr2=0x7fea3809b300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b300 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7fea300032f0 tx=0x7fea30016000).stop 2022-07-20T03:13:42.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 shutdown_connections 2022-07-20T03:13:42.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea180539b0 0x7fea18055e70 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea38143060 0x7fea3809b300 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea380b5960 0x7fea3809ae10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 --2- 172.21.15.61:0/429551320 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea3805d0e0 0x7fea380b2fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 >> 172.21.15.61:0/429551320 conn(0x7fea380b6a20 msgr2=0x7fea38146840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:42.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 shutdown_connections 2022-07-20T03:13:42.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.392+0000 7fea3e4c8700 1 -- 172.21.15.61:0/429551320 wait complete. 2022-07-20T03:13:42.413 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:13:42.415 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:42.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.524+0000 7f6dc4f9e700 1 Processor -- start 2022-07-20T03:13:42.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.524+0000 7f6dc4f9e700 1 -- start start 2022-07-20T03:13:42.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.524+0000 7f6dc4f9e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc00b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6dc00aea10 0x7f6dc00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6dc01433d0 con 0x7f6dc00b0570 2022-07-20T03:13:42.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6dc0143b10 con 0x7f6dc00aea10 2022-07-20T03:13:42.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc0143670 con 0x7f6dc01424e0 2022-07-20T03:13:42.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6dc00aea10 0x7f6dc00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbeffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc00b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6dc00aea10 0x7f6dc00ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56496/0 (socket says 172.21.15.61:56496) 2022-07-20T03:13:42.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 -- 172.21.15.61:0/2698395264 learned_addr learned my addr 172.21.15.61:0/2698395264 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 -- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 msgr2=0x7f6dc00b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:42.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 --2- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc00b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 -- 172.21.15.61:0/2698395264 >> v1:172.21.15.117:6789/0 conn(0x7f6dc00b0570 legacy=0x7f6dc0141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 -- 172.21.15.61:0/2698395264 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dc00b1010 con 0x7f6dc00aea10 2022-07-20T03:13:42.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbffff700 1 --2- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00ad350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6db80032f0 tx=0x7f6db8009290).ready entity=mon.1 client_cookie=676a71475cbdc5d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbe7fc700 1 -- 172.21.15.61:0/2698395264 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6db800b8a0 con 0x7f6dc00aea10 2022-07-20T03:13:42.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbe7fc700 1 -- 172.21.15.61:0/2698395264 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db800ba00 con 0x7f6dc00aea10 2022-07-20T03:13:42.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dbe7fc700 1 -- 172.21.15.61:0/2698395264 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6db800bb70 con 0x7f6dc00aea10 2022-07-20T03:13:42.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 msgr2=0x7f6dc00ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00ad350 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f6db80032f0 tx=0x7f6db8009290).stop 2022-07-20T03:13:42.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/2698395264 shutdown_connections 2022-07-20T03:13:42.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00ad350 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/2698395264 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc00b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/2698395264 >> 172.21.15.61:0/2698395264 conn(0x7f6dc00b9360 msgr2=0x7f6dc00b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:42.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/2698395264 shutdown_connections 2022-07-20T03:13:42.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.528+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/2698395264 wait complete. 2022-07-20T03:13:42.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 Processor -- start 2022-07-20T03:13:42.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 -- start start 2022-07-20T03:13:42.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00b42a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 0x7f6dc0140070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc0155aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc013e9b0 con 0x7f6dc00b0570 2022-07-20T03:13:42.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00b4790 con 0x7f6dc00aea10 2022-07-20T03:13:42.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6dc00b49b0 con 0x7f6dc01424e0 2022-07-20T03:13:42.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 0x7f6dc0140070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbf7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00b42a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 0x7f6dc0140070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46820/0 (socket says 172.21.15.61:46820) 2022-07-20T03:13:42.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 -- 172.21.15.61:0/3382360813 learned_addr learned my addr 172.21.15.61:0/3382360813 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:42.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbffff700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc0155aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 -- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 msgr2=0x7f6dc0155aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc0155aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 -- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 msgr2=0x7f6dc00b42a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00b42a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 -- 172.21.15.61:0/3382360813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db8009530 con 0x7f6dc00b0570 2022-07-20T03:13:42.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbf7fe700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:42.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbeffd700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 0x7f6dc0140070 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f6dc00b48d0 tx=0x7f6db0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6db0010780 con 0x7f6dc00b0570 2022-07-20T03:13:42.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6db00108e0 con 0x7f6dc00b0570 2022-07-20T03:13:42.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6dc00b4c50 con 0x7f6dc00b0570 2022-07-20T03:13:42.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6db0010c40 con 0x7f6dc00b0570 2022-07-20T03:13:42.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.532+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6dc01405e0 con 0x7f6dc00b0570 2022-07-20T03:13:42.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.536+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6db001c460 con 0x7f6dc00b0570 2022-07-20T03:13:42.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.536+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6dc00ace30 con 0x7f6dc00b0570 2022-07-20T03:13:42.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.536+0000 7f6dbcff9700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6da00539d0 0x7f6da0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:42.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.536+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6db006c140 con 0x7f6dc00b0570 2022-07-20T03:13:42.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.536+0000 7f6dbf7fe700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6da00539d0 0x7f6da0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:42.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.536+0000 7f6dbf7fe700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6da00539d0 0x7f6da0055e90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f6db400d000 tx=0x7f6db4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:42.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.540+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6db0043040 con 0x7f6dc00b0570 2022-07-20T03:13:42.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.700+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6dc00574e0 con 0x7f6dc00b0570 2022-07-20T03:13:42.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dbcff9700 1 -- 172.21.15.61:0/3382360813 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f6db0046660 con 0x7f6dc00b0570 2022-07-20T03:13:42.787 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:42.787 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:13:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6da00539d0 msgr2=0x7f6da0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6da00539d0 0x7f6da0055e90 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7f6db400d000 tx=0x7f6db4009040).stop 2022-07-20T03:13:42.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 msgr2=0x7f6dc0140070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 0x7f6dc0140070 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f6dc00b48d0 tx=0x7f6db0006df0).stop 2022-07-20T03:13:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 shutdown_connections 2022-07-20T03:13:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6da00539d0 0x7f6da0055e90 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6dc01424e0 0x7f6dc0155aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6dc00b0570 0x7f6dc0140070 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 --2- 172.21.15.61:0/3382360813 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6dc00aea10 0x7f6dc00b42a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 >> 172.21.15.61:0/3382360813 conn(0x7f6dc00b9360 msgr2=0x7f6dc00ae2d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 shutdown_connections 2022-07-20T03:13:42.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:42.704+0000 7f6dc4f9e700 1 -- 172.21.15.61:0/3382360813 wait complete. 2022-07-20T03:13:42.797 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:13:43.014 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:13:37.441965+0000', 'modified': '2022-07-20T03:13:38.444883+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6285': {'gid': 6285, 'name': 'a', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/460461689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 460461689}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 460461689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:13:43.015 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:13:43.017 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:13:43.137 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6285,"want_state":"up:active","state":"up:active","rank_uptime":5.68325192,"mdsmap_epoch":40,"osdmap_epoch":71,"osdmap_epoch_barrier":70,"uptime":10.768864306999999} 2022-07-20T03:13:43.138 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6285, "mdsmap_epoch": 40, "osdmap_epoch": 71, "osdmap_epoch_barrier": 70, "rank_uptime": 5.68325192, "state": "up:active", "uptime": 10.768864307, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:13:43.140 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:13:43.141 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:13:43.142 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:13:43.168 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.169 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:43.211 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:13:43.312 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:13:43.313 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:13:43.360 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:13:43.363 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:13:43.364 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.365 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:13:43.383 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:13:43.384 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:43.431 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:13:43.454 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:13:43.455 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:13:43.506 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:13:43.505+0000 7f9e5f89b080 -1 init, newargv = 0x55976d8423b0 newargc=9 2022-07-20T03:13:43.507 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22913]: starting ceph client 2022-07-20T03:13:43.514 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:13:43.516 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22913]: starting fuse 2022-07-20T03:13:43.517 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:13:43.522 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.523 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:13:43.592 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:13:43.593 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:13:43.595 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.596 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:13:43.596 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:13:43.596 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:13:43.603 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:13:43.604 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:13:43.604 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:13:43.605 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:13:43.605 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:13:43.605 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:13:43.606 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:13:43.609 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:13:43.610 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:13:43.611 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:13:43.611 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:13:43.611 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:13:43.612 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:13:43.612 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:13:43.613 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:13:43.613 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:13:43.644 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:13:43.644 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:13:43.644 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:13:43.645 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:13:43.715 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22913.asok 2022-07-20T03:13:43.731 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22913.asok 2022-07-20T03:13:43.741 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.742 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22913.asok status 2022-07-20T03:13:43.885 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:13:43.885 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:13:43.886 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:13:43.898 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:13:43.910 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:13:43.911 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:13:43.911 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:13:43.912 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22913", 2022-07-20T03:13:43.912 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:13:43.913 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:13:43.913 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:13:43.926 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:13:43.927 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6433, 2022-07-20T03:13:43.927 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:13:43.927 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:13:43.927 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:13:43.928 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6433 2022-07-20T03:13:43.928 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:13:43.928 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:13:43.928 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:13:43.929 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:13:43.929 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2780278373 2022-07-20T03:13:43.929 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:13:43.929 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:13:43.930 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:13:43.930 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:13:43.930 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:13:43.931 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2780278373 2022-07-20T03:13:43.931 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:13:43.931 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6433 172.21.15.124:0/2780278373", 2022-07-20T03:13:43.932 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/2780278373", 2022-07-20T03:13:43.932 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:13:43.932 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 40, 2022-07-20T03:13:43.932 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 71, 2022-07-20T03:13:43.933 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:13:43.933 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:13:43.933 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:13:43.933 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:13:43.950 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.951 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:43.951 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:43.981 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:13:43.981 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:13:43.991 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:43.992 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:44.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:13:44.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.156+0000 7f8896299700 1 Processor -- start 2022-07-20T03:13:44.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.156+0000 7f8896299700 1 -- start start 2022-07-20T03:13:44.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900b5650 0x7f88900b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f88900af8e0 0x7f889005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8890147010 con 0x7f88900adff0 2022-07-20T03:13:44.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f88901472b0 con 0x7f88900af8e0 2022-07-20T03:13:44.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8890146d40 con 0x7f88900b5650 2022-07-20T03:13:44.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8894a96700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f88900adff0 0x7f8890147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43428/0 (socket says 172.21.15.61:43428) 2022-07-20T03:13:44.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8894a96700 1 -- 172.21.15.61:0/4073220577 learned_addr learned my addr 172.21.15.61:0/4073220577 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:44.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895a98700 1 --2- 172.21.15.61:0/4073220577 >> v2:172.21.15.61:3300/0 conn(0x7f88900af8e0 0x7f889005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895297700 1 --2- 172.21.15.61:0/4073220577 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900b5650 0x7f88900b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8887fff700 1 -- 172.21.15.61:0/4073220577 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1420307846 0 0) 0x7f8890147010 con 0x7f88900adff0 2022-07-20T03:13:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8887fff700 1 -- 172.21.15.61:0/4073220577 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8874003260 con 0x7f88900adff0 2022-07-20T03:13:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895297700 1 -- 172.21.15.61:0/4073220577 >> v2:172.21.15.61:3300/0 conn(0x7f88900af8e0 msgr2=0x7f889005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:44.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895297700 1 --2- 172.21.15.61:0/4073220577 >> v2:172.21.15.61:3300/0 conn(0x7f88900af8e0 0x7f889005bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895297700 1 -- 172.21.15.61:0/4073220577 >> v1:172.21.15.61:6789/0 conn(0x7f88900adff0 legacy=0x7f8890147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895297700 1 -- 172.21.15.61:0/4073220577 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88900acae0 con 0x7f88900b5650 2022-07-20T03:13:44.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8895297700 1 --2- 172.21.15.61:0/4073220577 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900b5650 0x7f88900b5a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8880002f70 tx=0x7f8880006df0).ready entity=mon.2 client_cookie=7a22d323204c22 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8887fff700 1 -- 172.21.15.61:0/4073220577 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888000d4a0 con 0x7f88900b5650 2022-07-20T03:13:44.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8887fff700 1 -- 172.21.15.61:0/4073220577 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8880000e90 con 0x7f88900b5650 2022-07-20T03:13:44.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8887fff700 1 -- 172.21.15.61:0/4073220577 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f888000d7f0 con 0x7f88900b5650 2022-07-20T03:13:44.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- 172.21.15.61:0/4073220577 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900b5650 msgr2=0x7f88900b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 --2- 172.21.15.61:0/4073220577 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900b5650 0x7f88900b5a30 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f8880002f70 tx=0x7f8880006df0).stop 2022-07-20T03:13:44.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- 172.21.15.61:0/4073220577 shutdown_connections 2022-07-20T03:13:44.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 --2- 172.21.15.61:0/4073220577 >> v2:172.21.15.61:3300/0 conn(0x7f88900af8e0 0x7f889005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 --2- 172.21.15.61:0/4073220577 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900b5650 0x7f88900b5a30 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- 172.21.15.61:0/4073220577 >> 172.21.15.61:0/4073220577 conn(0x7f88900634e0 msgr2=0x7f88900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:44.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- 172.21.15.61:0/4073220577 shutdown_connections 2022-07-20T03:13:44.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.160+0000 7f8896299700 1 -- 172.21.15.61:0/4073220577 wait complete. 2022-07-20T03:13:44.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 Processor -- start 2022-07-20T03:13:44.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 -- start start 2022-07-20T03:13:44.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88900adff0 0x7f8890148de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900af8e0 0x7f889010e360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 0x7f8890159df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88900659f0 con 0x7f88900b5650 2022-07-20T03:13:44.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88901596b0 con 0x7f88900adff0 2022-07-20T03:13:44.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8896299700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8890159950 con 0x7f88900af8e0 2022-07-20T03:13:44.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 0x7f8890159df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 0x7f8890159df0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46836/0 (socket says 172.21.15.61:46836) 2022-07-20T03:13:44.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 -- 172.21.15.61:0/1317984308 learned_addr learned my addr 172.21.15.61:0/1317984308 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:44.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895297700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88900adff0 0x7f8890148de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8894a96700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900af8e0 0x7f889010e360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 -- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900af8e0 msgr2=0x7f889010e360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900af8e0 0x7f889010e360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 -- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88900adff0 msgr2=0x7f8890148de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88900adff0 0x7f8890148de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 -- 172.21.15.61:0/1317984308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f888000b180 con 0x7f88900b5650 2022-07-20T03:13:44.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.164+0000 7f8895a98700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 0x7f8890159df0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f888800c000 tx=0x7f8888006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.168+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8888011750 con 0x7f88900b5650 2022-07-20T03:13:44.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.168+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f88880118b0 con 0x7f88900b5650 2022-07-20T03:13:44.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.168+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88900656e0 con 0x7f88900b5650 2022-07-20T03:13:44.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.168+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8888011c10 con 0x7f88900b5650 2022-07-20T03:13:44.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.168+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f889010ff70 con 0x7f88900b5650 2022-07-20T03:13:44.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.168+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8890159b10 con 0x7f88900b5650 2022-07-20T03:13:44.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.172+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8888011d70 con 0x7f88900b5650 2022-07-20T03:13:44.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.172+0000 7f88867fc700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8878053a60 0x7f8878055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.172+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f888806d100 con 0x7f88900b5650 2022-07-20T03:13:44.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.172+0000 7f8895297700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8878053a60 0x7f8878055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.172+0000 7f8895297700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8878053a60 0x7f8878055f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f888001a000 tx=0x7f8880014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.180+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8888043f50 con 0x7f88900b5650 2022-07-20T03:13:44.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.304+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f889005c280 con 0x7f88900b5650 2022-07-20T03:13:44.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f88867fc700 1 -- 172.21.15.61:0/1317984308 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v71) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f8888047570 con 0x7f88900b5650 2022-07-20T03:13:44.324 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:44.324 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":71,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:13:38.445600+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:13:36.009693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T03:13:36.517295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:13:44.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8878053a60 msgr2=0x7f8878055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8878053a60 0x7f8878055f20 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f888001a000 tx=0x7f8880014000).stop 2022-07-20T03:13:44.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 msgr2=0x7f8890159df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 0x7f8890159df0 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f888800c000 tx=0x7f8888006df0).stop 2022-07-20T03:13:44.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 shutdown_connections 2022-07-20T03:13:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8878053a60 0x7f8878055f20 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f88900b5650 0x7f8890159df0 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f88900af8e0 0x7f889010e360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 --2- 172.21.15.61:0/1317984308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f88900adff0 0x7f8890148de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 >> 172.21.15.61:0/1317984308 conn(0x7f88900634e0 msgr2=0x7f8890146600 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 shutdown_connections 2022-07-20T03:13:44.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.308+0000 7f8896299700 1 -- 172.21.15.61:0/1317984308 wait complete. 2022-07-20T03:13:44.341 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:13:36.009693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:13:36.517295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:13:44.342 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:13:44.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecf743700 1 Processor -- start 2022-07-20T03:13:44.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecf743700 1 -- start start 2022-07-20T03:13:44.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecf743700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ec80b3ed0 0x7f3ec8143530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecf743700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec80b6680 con 0x7f3ec8143a20 2022-07-20T03:13:44.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecf743700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ec81465c0 con 0x7f3ec80b62a0 2022-07-20T03:13:44.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecf743700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ec80b4af0 con 0x7f3ec80b3ed0 2022-07-20T03:13:44.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecccde700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ec80b3ed0 0x7f3ec8143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecdce0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3ec8143a20 0x7f3ec813fcc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43442/0 (socket says 172.21.15.61:43442) 2022-07-20T03:13:44.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecdce0700 1 -- 172.21.15.61:0/2047899637 learned_addr learned my addr 172.21.15.61:0/2047899637 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:44.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecccde700 1 -- 172.21.15.61:0/2047899637 >> v1:172.21.15.117:6789/0 conn(0x7f3ec80b62a0 legacy=0x7f3ec80b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecccde700 1 -- 172.21.15.61:0/2047899637 >> v1:172.21.15.61:6789/0 conn(0x7f3ec8143a20 legacy=0x7f3ec813fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.500+0000 7f3ecccde700 1 -- 172.21.15.61:0/2047899637 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8142df0 con 0x7f3ec80b3ed0 2022-07-20T03:13:44.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecccde700 1 --2- 172.21.15.61:0/2047899637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8143530 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f3ebc002f70 tx=0x7f3ebc006df0).ready entity=mon.0 client_cookie=3e909cf61620463b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ec7fff700 1 -- 172.21.15.61:0/2047899637 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ebc00d610 con 0x7f3ec80b3ed0 2022-07-20T03:13:44.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ec7fff700 1 -- 172.21.15.61:0/2047899637 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ebc000e90 con 0x7f3ec80b3ed0 2022-07-20T03:13:44.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ec7fff700 1 -- 172.21.15.61:0/2047899637 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ebc00d960 con 0x7f3ec80b3ed0 2022-07-20T03:13:44.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 -- 172.21.15.61:0/2047899637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec80b3ed0 msgr2=0x7f3ec8143530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 --2- 172.21.15.61:0/2047899637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8143530 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f3ebc002f70 tx=0x7f3ebc006df0).stop 2022-07-20T03:13:44.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 -- 172.21.15.61:0/2047899637 shutdown_connections 2022-07-20T03:13:44.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 --2- 172.21.15.61:0/2047899637 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8143530 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 -- 172.21.15.61:0/2047899637 >> 172.21.15.61:0/2047899637 conn(0x7f3ec8066f00 msgr2=0x7f3ec80672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:44.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 -- 172.21.15.61:0/2047899637 shutdown_connections 2022-07-20T03:13:44.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.504+0000 7f3ecf743700 1 -- 172.21.15.61:0/2047899637 wait complete. 2022-07-20T03:13:44.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 Processor -- start 2022-07-20T03:13:44.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 -- start start 2022-07-20T03:13:44.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8152b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ec80b62a0 0x7f3ec8147ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec8143a20 0x7f3ec8141550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80b9540 con 0x7f3ec8143a20 2022-07-20T03:13:44.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80b9770 con 0x7f3ec80b3ed0 2022-07-20T03:13:44.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ec80b9a10 con 0x7f3ec80b62a0 2022-07-20T03:13:44.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec8143a20 0x7f3ec8141550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecd4df700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8152b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecd4df700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8152b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56526/0 (socket says 172.21.15.61:56526) 2022-07-20T03:13:44.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecd4df700 1 -- 172.21.15.61:0/1084145644 learned_addr learned my addr 172.21.15.61:0/1084145644 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:44.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecccde700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ec80b62a0 0x7f3ec8147ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 -- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ec80b62a0 msgr2=0x7f3ec8147ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ec80b62a0 0x7f3ec8147ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 -- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 msgr2=0x7f3ec8152b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8152b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 -- 172.21.15.61:0/1084145644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ebc00b180 con 0x7f3ec8143a20 2022-07-20T03:13:44.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecd4df700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:44.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecdce0700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec8143a20 0x7f3ec8141550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f3ec80b3b70 tx=0x7f3ec0007ef0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ec000ce40 con 0x7f3ec8143a20 2022-07-20T03:13:44.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3ec0005ba0 con 0x7f3ec8143a20 2022-07-20T03:13:44.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ec80b9cb0 con 0x7f3ec8143a20 2022-07-20T03:13:44.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.508+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3ec00070d0 con 0x7f3ec8143a20 2022-07-20T03:13:44.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.512+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ec8142770 con 0x7f3ec8143a20 2022-07-20T03:13:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.512+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ec8143090 con 0x7f3ec8143a20 2022-07-20T03:13:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.512+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3ec001a060 con 0x7f3ec8143a20 2022-07-20T03:13:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.512+0000 7f3ec67fc700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ea8053a40 0x7f3ea8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.512+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3ec006b790 con 0x7f3ec8143a20 2022-07-20T03:13:44.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.512+0000 7f3ecd4df700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ea8053a40 0x7f3ea8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.516+0000 7f3ecd4df700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ea8053a40 0x7f3ea8055f00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f3eb800d000 tx=0x7f3eb8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.520+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3ec0042670 con 0x7f3ec8143a20 2022-07-20T03:13:44.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f3ec80574e0 con 0x7f3ec8143a20 2022-07-20T03:13:44.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ec67fc700 1 -- 172.21.15.61:0/1084145644 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v71) v1 ==== 110+0+10 (secure 0 0 0) 0x7f3ec0018020 con 0x7f3ec8143a20 2022-07-20T03:13:44.651 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:13:44.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ea8053a40 msgr2=0x7f3ea8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ea8053a40 0x7f3ea8055f00 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f3eb800d000 tx=0x7f3eb8009040).stop 2022-07-20T03:13:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec8143a20 msgr2=0x7f3ec8141550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec8143a20 0x7f3ec8141550 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f3ec80b3b70 tx=0x7f3ec0007ef0).stop 2022-07-20T03:13:44.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 shutdown_connections 2022-07-20T03:13:44.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3ea8053a40 0x7f3ea8055f00 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ec8143a20 0x7f3ec8141550 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ec80b62a0 0x7f3ec8147ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 --2- 172.21.15.61:0/1084145644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ec80b3ed0 0x7f3ec8152b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 >> 172.21.15.61:0/1084145644 conn(0x7f3ec8066f00 msgr2=0x7f3ec80b43e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:44.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 shutdown_connections 2022-07-20T03:13:44.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.648+0000 7f3ecf743700 1 -- 172.21.15.61:0/1084145644 wait complete. 2022-07-20T03:13:44.663 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:13:44.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.764+0000 7fe5ca27b700 1 Processor -- start 2022-07-20T03:13:44.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.764+0000 7fe5ca27b700 1 -- start start 2022-07-20T03:13:44.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b59d0 0x7fe5c40b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5c414ada0 0x7fe5c40b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe5c40b14f0 0x7fe5c405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c4148080 con 0x7fe5c414ada0 2022-07-20T03:13:44.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe5c40afbc0 con 0x7fe5c40b14f0 2022-07-20T03:13:44.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c40b00e0 con 0x7fe5c40b59d0 2022-07-20T03:13:44.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5c414ada0 0x7fe5c40b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe5c414ada0 0x7fe5c40b0cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46854/0 (socket says 172.21.15.61:46854) 2022-07-20T03:13:44.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 -- 172.21.15.61:0/955261206 learned_addr learned my addr 172.21.15.61:0/955261206 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:44.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c9a7a700 1 --2- 172.21.15.61:0/955261206 >> v2:172.21.15.117:3300/0 conn(0x7fe5c40b14f0 0x7fe5c405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 -- 172.21.15.61:0/955261206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b59d0 msgr2=0x7fe5c40b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:44.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c9279700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b59d0 0x7fe5c40b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b59d0 0x7fe5c40b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 -- 172.21.15.61:0/955261206 >> v2:172.21.15.117:3300/0 conn(0x7fe5c40b14f0 msgr2=0x7fe5c405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 --2- 172.21.15.61:0/955261206 >> v2:172.21.15.117:3300/0 conn(0x7fe5c40b14f0 0x7fe5c405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 -- 172.21.15.61:0/955261206 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c40b5db0 con 0x7fe5c414ada0 2022-07-20T03:13:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c9a7a700 1 --2- 172.21.15.61:0/955261206 >> v2:172.21.15.117:3300/0 conn(0x7fe5c40b14f0 0x7fe5c405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:44.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c9279700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b59d0 0x7fe5c40b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:44.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5c8a78700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c414ada0 0x7fe5c40b0cd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fe5c0002f70 tx=0x7fe5c0006eb0).ready entity=mon.0 client_cookie=27fada95ec5e9dab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5bbfff700 1 -- 172.21.15.61:0/955261206 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5c000d570 con 0x7fe5c414ada0 2022-07-20T03:13:44.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5bbfff700 1 -- 172.21.15.61:0/955261206 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c0000e90 con 0x7fe5c414ada0 2022-07-20T03:13:44.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5bbfff700 1 -- 172.21.15.61:0/955261206 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5c000d900 con 0x7fe5c414ada0 2022-07-20T03:13:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- 172.21.15.61:0/955261206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c414ada0 msgr2=0x7fe5c40b0cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c414ada0 0x7fe5c40b0cd0 secure :-1 s=READY pgs=735 cs=0 l=1 rev1=1 rx=0x7fe5c0002f70 tx=0x7fe5c0006eb0).stop 2022-07-20T03:13:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- 172.21.15.61:0/955261206 shutdown_connections 2022-07-20T03:13:44.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/955261206 >> v2:172.21.15.117:3300/0 conn(0x7fe5c40b14f0 0x7fe5c405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c414ada0 0x7fe5c40b0cd0 unknown :-1 s=CLOSED pgs=735 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/955261206 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b59d0 0x7fe5c40b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- 172.21.15.61:0/955261206 >> 172.21.15.61:0/955261206 conn(0x7fe5c40634c0 msgr2=0x7fe5c40b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:44.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- 172.21.15.61:0/955261206 shutdown_connections 2022-07-20T03:13:44.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- 172.21.15.61:0/955261206 wait complete. 2022-07-20T03:13:44.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 Processor -- start 2022-07-20T03:13:44.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.768+0000 7fe5ca27b700 1 -- start start 2022-07-20T03:13:44.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b14f0 0x7fe5c40b3ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 0x7fe5c41493e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5c414ada0 0x7fe5c414bcf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4156e40 con 0x7fe5c40b59d0 2022-07-20T03:13:44.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c4148f30 con 0x7fe5c414ada0 2022-07-20T03:13:44.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe5c414b5b0 con 0x7fe5c40b14f0 2022-07-20T03:13:44.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 0x7fe5c41493e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 0x7fe5c41493e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46860/0 (socket says 172.21.15.61:46860) 2022-07-20T03:13:44.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 -- 172.21.15.61:0/1683611701 learned_addr learned my addr 172.21.15.61:0/1683611701 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:44.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c9279700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b14f0 0x7fe5c40b3ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c9a7a700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5c414ada0 0x7fe5c414bcf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 -- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b14f0 msgr2=0x7fe5c40b3ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b14f0 0x7fe5c40b3ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 -- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5c414ada0 msgr2=0x7fe5c414bcf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5c414ada0 0x7fe5c414bcf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 -- 172.21.15.61:0/1683611701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5c000b1e0 con 0x7fe5c40b59d0 2022-07-20T03:13:44.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c9a7a700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5c414ada0 0x7fe5c414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:44.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c9279700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b14f0 0x7fe5c40b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:13:44.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5c8a78700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 0x7fe5c41493e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fe5c0002f70 tx=0x7fe5c0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5c0011990 con 0x7fe5c40b59d0 2022-07-20T03:13:44.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe5c414b850 con 0x7fe5c40b59d0 2022-07-20T03:13:44.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe5c0011af0 con 0x7fe5c40b59d0 2022-07-20T03:13:44.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe5c0011e50 con 0x7fe5c40b59d0 2022-07-20T03:13:44.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe5c4148720 con 0x7fe5c40b59d0 2022-07-20T03:13:44.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe5c414b9e0 con 0x7fe5c40b59d0 2022-07-20T03:13:44.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.772+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe5c0025030 con 0x7fe5c40b59d0 2022-07-20T03:13:44.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.776+0000 7fe5ba7fc700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5ac053af0 0x7fe5ac055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:44.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.776+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe5c006d2c0 con 0x7fe5c40b59d0 2022-07-20T03:13:44.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.776+0000 7fe5c9279700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5ac053af0 0x7fe5ac055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:44.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.776+0000 7fe5c9279700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5ac053af0 0x7fe5ac055fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe5b4008000 tx=0x7fe5b4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:44.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.780+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe5c0044110 con 0x7fe5c40b59d0 2022-07-20T03:13:44.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.904+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fe5c40b0340 con 0x7fe5c40b59d0 2022-07-20T03:13:44.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ba7fc700 1 -- 172.21.15.61:0/1683611701 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v71) v1 ==== 92+0+10 (secure 0 0 0) 0x7fe5c0023020 con 0x7fe5c40b59d0 2022-07-20T03:13:44.911 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:44.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5ac053af0 msgr2=0x7fe5ac055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5ac053af0 0x7fe5ac055fb0 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fe5b4008000 tx=0x7fe5b4008040).stop 2022-07-20T03:13:44.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 msgr2=0x7fe5c41493e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:44.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 0x7fe5c41493e0 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7fe5c0002f70 tx=0x7fe5c0015000).stop 2022-07-20T03:13:44.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 shutdown_connections 2022-07-20T03:13:44.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe5ac053af0 0x7fe5ac055fb0 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe5c414ada0 0x7fe5c414bcf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe5c40b59d0 0x7fe5c41493e0 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 --2- 172.21.15.61:0/1683611701 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe5c40b14f0 0x7fe5c40b3ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:44.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 >> 172.21.15.61:0/1683611701 conn(0x7fe5c40634c0 msgr2=0x7fe5c40adc30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:44.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 shutdown_connections 2022-07-20T03:13:44.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:44.908+0000 7fe5ca27b700 1 -- 172.21.15.61:0/1683611701 wait complete. 2022-07-20T03:13:44.926 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:13:45.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 Processor -- start 2022-07-20T03:13:45.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 -- start start 2022-07-20T03:13:45.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b9da0 0x7f5aa81468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5aa81419a0 0x7f5aa8141e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5aa80ba180 con 0x7f5aa80b3cf0 2022-07-20T03:13:45.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5aa80ad5b0 con 0x7f5aa81419a0 2022-07-20T03:13:45.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aae051700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa81461b0 con 0x7f5aa80b9da0 2022-07-20T03:13:45.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5aa81419a0 0x7f5aa8141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5aa81419a0 0x7f5aa8141e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46870/0 (socket says 172.21.15.61:46870) 2022-07-20T03:13:45.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 -- 172.21.15.61:0/190415243 learned_addr learned my addr 172.21.15.61:0/190415243 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:45.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad04f700 1 --2- 172.21.15.61:0/190415243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b9da0 0x7f5aa81468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 -- 172.21.15.61:0/190415243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b9da0 msgr2=0x7f5aa81468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:45.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 --2- 172.21.15.61:0/190415243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b9da0 0x7f5aa81468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 -- 172.21.15.61:0/190415243 >> v1:172.21.15.117:6789/0 conn(0x7f5aa80b3cf0 legacy=0x7f5aa8142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 -- 172.21.15.61:0/190415243 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa8146450 con 0x7f5aa81419a0 2022-07-20T03:13:45.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.036+0000 7f5aad850700 1 --2- 172.21.15.61:0/190415243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa81419a0 0x7f5aa8141e50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f5aa00032f0 tx=0x7f5aa0009290).ready entity=mon.0 client_cookie=5496c38519a2f0cf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5a9ffff700 1 -- 172.21.15.61:0/190415243 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aa000b8a0 con 0x7f5aa81419a0 2022-07-20T03:13:45.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5a9ffff700 1 -- 172.21.15.61:0/190415243 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5aa000ba00 con 0x7f5aa81419a0 2022-07-20T03:13:45.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5a9ffff700 1 -- 172.21.15.61:0/190415243 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5aa000bb70 con 0x7f5aa81419a0 2022-07-20T03:13:45.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 -- 172.21.15.61:0/190415243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa81419a0 msgr2=0x7f5aa8141e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 --2- 172.21.15.61:0/190415243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa81419a0 0x7f5aa8141e50 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f5aa00032f0 tx=0x7f5aa0009290).stop 2022-07-20T03:13:45.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 -- 172.21.15.61:0/190415243 shutdown_connections 2022-07-20T03:13:45.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 --2- 172.21.15.61:0/190415243 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa81419a0 0x7f5aa8141e50 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 --2- 172.21.15.61:0/190415243 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b9da0 0x7f5aa81468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 -- 172.21.15.61:0/190415243 >> 172.21.15.61:0/190415243 conn(0x7f5aa80678c0 msgr2=0x7f5aa80b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:45.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 -- 172.21.15.61:0/190415243 shutdown_connections 2022-07-20T03:13:45.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 -- 172.21.15.61:0/190415243 wait complete. 2022-07-20T03:13:45.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 Processor -- start 2022-07-20T03:13:45.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.040+0000 7f5aae051700 1 -- start start 2022-07-20T03:13:45.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b3cf0 0x7f5aa80b6760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 0x7f5aa81142d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5aa81419a0 0x7f5aa8151c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa80b2fd0 con 0x7f5aa80b9da0 2022-07-20T03:13:45.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8113b90 con 0x7f5aa81419a0 2022-07-20T03:13:45.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5aa8113e30 con 0x7f5aa80b3cf0 2022-07-20T03:13:45.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 0x7f5aa81142d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aad850700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5aa81419a0 0x7f5aa8151c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aad04f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b3cf0 0x7f5aa80b6760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 0x7f5aa81142d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46876/0 (socket says 172.21.15.61:46876) 2022-07-20T03:13:45.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 -- 172.21.15.61:0/1034673358 learned_addr learned my addr 172.21.15.61:0/1034673358 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:45.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 -- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b3cf0 msgr2=0x7f5aa80b6760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b3cf0 0x7f5aa80b6760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 -- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5aa81419a0 msgr2=0x7f5aa8151c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5aa81419a0 0x7f5aa8151c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 -- 172.21.15.61:0/1034673358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5aa0009530 con 0x7f5aa80b9da0 2022-07-20T03:13:45.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aad850700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5aa81419a0 0x7f5aa8151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:13:45.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aac84e700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 0x7f5aa81142d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f5aa80af020 tx=0x7f5a98007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a98012400 con 0x7f5aa80b9da0 2022-07-20T03:13:45.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a98012560 con 0x7f5aa80b9da0 2022-07-20T03:13:45.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5aa80b1750 con 0x7f5aa80b9da0 2022-07-20T03:13:45.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a980128c0 con 0x7f5aa80b9da0 2022-07-20T03:13:45.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.044+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5aa80b25f0 con 0x7f5aa80b9da0 2022-07-20T03:13:45.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.048+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5aa8114960 con 0x7f5aa80b9da0 2022-07-20T03:13:45.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.048+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5a98019030 con 0x7f5aa80b9da0 2022-07-20T03:13:45.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.048+0000 7f5a9e7fc700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a90053af0 0x7f5a90055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.048+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a9806b2f0 con 0x7f5aa80b9da0 2022-07-20T03:13:45.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.048+0000 7f5aad04f700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a90053af0 0x7f5a90055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.048+0000 7f5aad04f700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a90053af0 0x7f5a90055fb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5aa400b000 tx=0x7f5aa4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.056+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5aa8114960 con 0x7f5aa80b9da0 2022-07-20T03:13:45.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.180+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f5aa8114960 con 0x7f5aa80b9da0 2022-07-20T03:13:45.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.180+0000 7f5a9e7fc700 1 -- 172.21.15.61:0/1034673358 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v71) v1 ==== 104+0+10 (secure 0 0 0) 0x7f5aa8114960 con 0x7f5aa80b9da0 2022-07-20T03:13:45.184 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:45.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a90053af0 msgr2=0x7f5a90055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a90053af0 0x7f5a90055fb0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f5aa400b000 tx=0x7f5aa4006df0).stop 2022-07-20T03:13:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 msgr2=0x7f5aa81142d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 0x7f5aa81142d0 secure :-1 s=READY pgs=738 cs=0 l=1 rev1=1 rx=0x7f5aa80af020 tx=0x7f5a98007c20).stop 2022-07-20T03:13:45.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 shutdown_connections 2022-07-20T03:13:45.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a90053af0 0x7f5a90055fb0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5aa81419a0 0x7f5aa8151c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5aa80b9da0 0x7f5aa81142d0 unknown :-1 s=CLOSED pgs=738 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 --2- 172.21.15.61:0/1034673358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5aa80b3cf0 0x7f5aa80b6760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 >> 172.21.15.61:0/1034673358 conn(0x7f5aa80678c0 msgr2=0x7f5aa80b9330 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:45.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 shutdown_connections 2022-07-20T03:13:45.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.184+0000 7f5aae051700 1 -- 172.21.15.61:0/1034673358 wait complete. 2022-07-20T03:13:45.197 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:13:45.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 Processor -- start 2022-07-20T03:13:45.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 -- start start 2022-07-20T03:13:45.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f51000b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51001461c0 0x7f5100146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f51000b3f10 con 0x7f51000b2ec0 2022-07-20T03:13:45.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51000b4ea0 con 0x7f51001461c0 2022-07-20T03:13:45.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50ff59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51000b3790 con 0x7f510005d160 2022-07-20T03:13:45.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51001461c0 0x7f5100146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51001461c0 0x7f5100146650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46886/0 (socket says 172.21.15.61:46886) 2022-07-20T03:13:45.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 -- 172.21.15.61:0/2223380236 learned_addr learned my addr 172.21.15.61:0/2223380236 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:45.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fdd9b700 1 --2- 172.21.15.61:0/2223380236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f51000b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 -- 172.21.15.61:0/2223380236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 msgr2=0x7f51000b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:45.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 --2- 172.21.15.61:0/2223380236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f51000b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 -- 172.21.15.61:0/2223380236 >> v1:172.21.15.117:6789/0 conn(0x7f51000b2ec0 legacy=0x7f5100142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 -- 172.21.15.61:0/2223380236 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51000b2620 con 0x7f51001461c0 2022-07-20T03:13:45.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fed9d700 1 --2- 172.21.15.61:0/2223380236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51001461c0 0x7f5100146650 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f50f80032f0 tx=0x7f50f8009290).ready entity=mon.0 client_cookie=42b38a5a27b35708 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fd59a700 1 -- 172.21.15.61:0/2223380236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f800b8a0 con 0x7f51001461c0 2022-07-20T03:13:45.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fd59a700 1 -- 172.21.15.61:0/2223380236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50f800ba00 con 0x7f51001461c0 2022-07-20T03:13:45.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.348+0000 7f50fd59a700 1 -- 172.21.15.61:0/2223380236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50f800bb70 con 0x7f51001461c0 2022-07-20T03:13:45.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 -- 172.21.15.61:0/2223380236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51001461c0 msgr2=0x7f5100146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 --2- 172.21.15.61:0/2223380236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51001461c0 0x7f5100146650 secure :-1 s=READY pgs=739 cs=0 l=1 rev1=1 rx=0x7f50f80032f0 tx=0x7f50f8009290).stop 2022-07-20T03:13:45.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 -- 172.21.15.61:0/2223380236 shutdown_connections 2022-07-20T03:13:45.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 --2- 172.21.15.61:0/2223380236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51001461c0 0x7f5100146650 unknown :-1 s=CLOSED pgs=739 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 --2- 172.21.15.61:0/2223380236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f51000b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 -- 172.21.15.61:0/2223380236 >> 172.21.15.61:0/2223380236 conn(0x7f5100066e20 msgr2=0x7f5100067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:45.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 -- 172.21.15.61:0/2223380236 shutdown_connections 2022-07-20T03:13:45.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 -- 172.21.15.61:0/2223380236 wait complete. 2022-07-20T03:13:45.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 Processor -- start 2022-07-20T03:13:45.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.352+0000 7f50ff59e700 1 -- start start 2022-07-20T03:13:45.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f5100155ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 0x7f510013d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51001461c0 0x7f51000ad410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51001398f0 con 0x7f51000b2ec0 2022-07-20T03:13:45.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f510013a2f0 con 0x7f51001461c0 2022-07-20T03:13:45.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f51000acff0 con 0x7f510005d160 2022-07-20T03:13:45.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 0x7f510013d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fe59c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f5100155ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 0x7f510013d6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46890/0 (socket says 172.21.15.61:46890) 2022-07-20T03:13:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 -- 172.21.15.61:0/538522731 learned_addr learned my addr 172.21.15.61:0/538522731 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:45.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fed9d700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51001461c0 0x7f51000ad410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 -- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 msgr2=0x7f5100155ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f5100155ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 -- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51001461c0 msgr2=0x7f51000ad410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51001461c0 0x7f51000ad410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 -- 172.21.15.61:0/538522731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50f8009530 con 0x7f51000b2ec0 2022-07-20T03:13:45.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50fdd9b700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 0x7f510013d6d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f510013a430 tx=0x7f50e8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50e8010780 con 0x7f51000b2ec0 2022-07-20T03:13:45.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f50e80108e0 con 0x7f51000b2ec0 2022-07-20T03:13:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51000b8090 con 0x7f51000b2ec0 2022-07-20T03:13:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f50e8010c40 con 0x7f51000b2ec0 2022-07-20T03:13:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f51000b85d0 con 0x7f51000b2ec0 2022-07-20T03:13:45.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.356+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51000b28c0 con 0x7f51000b2ec0 2022-07-20T03:13:45.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.360+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f50e801c460 con 0x7f51000b2ec0 2022-07-20T03:13:45.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.360+0000 7f50f77fe700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f50e4053a40 0x7f50e4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.360+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f50e806c250 con 0x7f51000b2ec0 2022-07-20T03:13:45.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.360+0000 7f50fe59c700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f50e4053a40 0x7f50e4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.360+0000 7f50fe59c700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f50e4053a40 0x7f50e4055f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f50f000d000 tx=0x7f50f0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.368+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f50e8043150 con 0x7f51000b2ec0 2022-07-20T03:13:45.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.496+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f51000574e0 con 0x7f51000b2ec0 2022-07-20T03:13:45.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.496+0000 7f50f77fe700 1 -- 172.21.15.61:0/538522731 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v71) v1 ==== 100+0+10 (secure 0 0 0) 0x7f50e8046770 con 0x7f51000b2ec0 2022-07-20T03:13:45.498 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:13:45.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f50e4053a40 msgr2=0x7f50e4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f50e4053a40 0x7f50e4055f00 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f50f000d000 tx=0x7f50f0007b90).stop 2022-07-20T03:13:45.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 msgr2=0x7f510013d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 0x7f510013d6d0 secure :-1 s=READY pgs=740 cs=0 l=1 rev1=1 rx=0x7f510013a430 tx=0x7f50e8006df0).stop 2022-07-20T03:13:45.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 shutdown_connections 2022-07-20T03:13:45.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f50e4053a40 0x7f50e4055f00 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51001461c0 0x7f51000ad410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51000b2ec0 0x7f510013d6d0 unknown :-1 s=CLOSED pgs=740 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 --2- 172.21.15.61:0/538522731 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f510005d160 0x7f5100155ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 >> 172.21.15.61:0/538522731 conn(0x7f5100066e20 msgr2=0x7f51001465d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:45.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 shutdown_connections 2022-07-20T03:13:45.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.500+0000 7f50ff59e700 1 -- 172.21.15.61:0/538522731 wait complete. 2022-07-20T03:13:45.517 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:13:45.528 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:45.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.632+0000 7fedd4efd700 1 Processor -- start 2022-07-20T03:13:45.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.632+0000 7fedd4efd700 1 -- start start 2022-07-20T03:13:45.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fedd013c790 0x7fedd00b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 0x7fedd00b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fedd0142ba0 con 0x7fedd00b9f00 2022-07-20T03:13:45.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fedd01429b0 con 0x7fedd013c790 2022-07-20T03:13:45.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd0146330 con 0x7fedd00b3ca0 2022-07-20T03:13:45.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 0x7fedd00b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedcf7fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fedd013c790 0x7fedd00b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 0x7fedd00b9a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33568/0 (socket says 172.21.15.61:33568) 2022-07-20T03:13:45.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 -- 172.21.15.61:0/2521202737 learned_addr learned my addr 172.21.15.61:0/2521202737 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:45.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedcffff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fedd00b9f00 0x7fedd0143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:52002/0 (socket says 172.21.15.61:52002) 2022-07-20T03:13:45.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedce7fc700 1 -- 172.21.15.61:0/2521202737 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2139704016 0 0) 0x7fedd0142ba0 con 0x7fedd00b9f00 2022-07-20T03:13:45.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 -- 172.21.15.61:0/2521202737 >> v2:172.21.15.117:3300/0 conn(0x7fedd013c790 msgr2=0x7fedd00b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:45.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 --2- 172.21.15.61:0/2521202737 >> v2:172.21.15.117:3300/0 conn(0x7fedd013c790 0x7fedd00b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 -- 172.21.15.61:0/2521202737 >> v1:172.21.15.117:6789/0 conn(0x7fedd00b9f00 legacy=0x7fedd0143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 -- 172.21.15.61:0/2521202737 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedd00b4c00 con 0x7fedd00b3ca0 2022-07-20T03:13:45.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedceffd700 1 --2- 172.21.15.61:0/2521202737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 0x7fedd00b9a10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fedc0002f70 tx=0x7fedc0006eb0).ready entity=mon.2 client_cookie=687de2d79f798c38 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedce7fc700 1 -- 172.21.15.61:0/2521202737 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedc000d570 con 0x7fedd00b3ca0 2022-07-20T03:13:45.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedce7fc700 1 -- 172.21.15.61:0/2521202737 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc0000e90 con 0x7fedd00b3ca0 2022-07-20T03:13:45.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedce7fc700 1 -- 172.21.15.61:0/2521202737 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedc000d8c0 con 0x7fedd00b3ca0 2022-07-20T03:13:45.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- 172.21.15.61:0/2521202737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 msgr2=0x7fedd00b9a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2521202737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 0x7fedd00b9a10 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fedc0002f70 tx=0x7fedc0006eb0).stop 2022-07-20T03:13:45.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- 172.21.15.61:0/2521202737 shutdown_connections 2022-07-20T03:13:45.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2521202737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b3ca0 0x7fedd00b9a10 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2521202737 >> v2:172.21.15.117:3300/0 conn(0x7fedd013c790 0x7fedd00b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- 172.21.15.61:0/2521202737 >> 172.21.15.61:0/2521202737 conn(0x7fedd00678b0 msgr2=0x7fedd00b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:45.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- 172.21.15.61:0/2521202737 shutdown_connections 2022-07-20T03:13:45.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.636+0000 7fedd4efd700 1 -- 172.21.15.61:0/2521202737 wait complete. 2022-07-20T03:13:45.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 Processor -- start 2022-07-20T03:13:45.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 -- start start 2022-07-20T03:13:45.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 0x7fedd00b2980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b9f00 0x7fedd00b2440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd013c790 0x7fedd0144b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00b0070 con 0x7fedd00b3ca0 2022-07-20T03:13:45.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00b0540 con 0x7fedd013c790 2022-07-20T03:13:45.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fedd00b02a0 con 0x7fedd00b9f00 2022-07-20T03:13:45.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 0x7fedd00b2980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 0x7fedd00b2980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46902/0 (socket says 172.21.15.61:46902) 2022-07-20T03:13:45.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 -- 172.21.15.61:0/2182747247 learned_addr learned my addr 172.21.15.61:0/2182747247 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:45.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcffff700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd013c790 0x7fedd0144b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedceffd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b9f00 0x7fedd00b2440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 -- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b9f00 msgr2=0x7fedd00b2440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b9f00 0x7fedd00b2440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 -- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd013c790 msgr2=0x7fedd0144b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd013c790 0x7fedd0144b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 -- 172.21.15.61:0/2182747247 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fedc000b1e0 con 0x7fedd00b3ca0 2022-07-20T03:13:45.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedcf7fe700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 0x7fedd00b2980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fedc400c000 tx=0x7fedc4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedc4011750 con 0x7fedd00b3ca0 2022-07-20T03:13:45.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fedc40118b0 con 0x7fedd00b3ca0 2022-07-20T03:13:45.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fedd00ae0c0 con 0x7fedd00b3ca0 2022-07-20T03:13:45.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fedc4011c10 con 0x7fedd00b3ca0 2022-07-20T03:13:45.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.640+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fedd0155460 con 0x7fedd00b3ca0 2022-07-20T03:13:45.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.644+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fedd0146090 con 0x7fedd00b3ca0 2022-07-20T03:13:45.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.644+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fedc4011d70 con 0x7fedd00b3ca0 2022-07-20T03:13:45.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.644+0000 7fedccff9700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fedb0053920 0x7fedb0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:45.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.644+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fedc406c370 con 0x7fedd00b3ca0 2022-07-20T03:13:45.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.644+0000 7fedceffd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fedb0053920 0x7fedb0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.648+0000 7fedceffd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fedb0053920 0x7fedb0055de0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fedc001c000 tx=0x7fedc0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.652+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fedc4043160 con 0x7fedd00b3ca0 2022-07-20T03:13:45.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.812+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fedd00574e0 con 0x7fedd00b3ca0 2022-07-20T03:13:45.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedccff9700 1 -- 172.21.15.61:0/2182747247 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7fedc4046780 con 0x7fedd00b3ca0 2022-07-20T03:13:45.825 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:45.826 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:13:45.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fedb0053920 msgr2=0x7fedb0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fedb0053920 0x7fedb0055de0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7fedc001c000 tx=0x7fedc0015000).stop 2022-07-20T03:13:45.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 msgr2=0x7fedd00b2980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:45.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 0x7fedd00b2980 secure :-1 s=READY pgs=741 cs=0 l=1 rev1=1 rx=0x7fedc400c000 tx=0x7fedc4006df0).stop 2022-07-20T03:13:45.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 shutdown_connections 2022-07-20T03:13:45.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fedb0053920 0x7fedb0055de0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fedd013c790 0x7fedd0144b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fedd00b9f00 0x7fedd00b2440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 --2- 172.21.15.61:0/2182747247 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fedd00b3ca0 0x7fedd00b2980 unknown :-1 s=CLOSED pgs=741 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 >> 172.21.15.61:0/2182747247 conn(0x7fedd00678b0 msgr2=0x7fedd00b7560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:45.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 shutdown_connections 2022-07-20T03:13:45.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:45.816+0000 7fedd4efd700 1 -- 172.21.15.61:0/2182747247 wait complete. 2022-07-20T03:13:45.885 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:13:46.105 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:13:37.441965+0000', 'modified': '2022-07-20T03:13:38.444883+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6285': {'gid': 6285, 'name': 'a', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/460461689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 460461689}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 460461689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:13:46.106 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:13:46.107 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:13:46.232 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6285,"want_state":"up:active","state":"up:active","rank_uptime":8.7755166490000001,"mdsmap_epoch":40,"osdmap_epoch":71,"osdmap_epoch_barrier":70,"uptime":13.861129420999999} 2022-07-20T03:13:46.232 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6285, "mdsmap_epoch": 40, "osdmap_epoch": 71, "osdmap_epoch_barrier": 70, "rank_uptime": 8.775516649, "state": "up:active", "uptime": 13.861129421, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:13:46.233 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:13:46.233 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:13:46.234 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:13:46.234 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:46.235 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:46.249 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:13:46.323 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:13:46.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:13:46.391 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:13:46.400 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:13:46.401 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:46.403 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:13:46.469 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:13:46.494 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:46.507 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:13:46.547 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:13:46.548 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:13:46.592 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:13:46.572+0000 7fc712c16080 -1 init, newargv = 0x5654a68bb400 newargc=9ceph-fuse[ 2022-07-20T03:13:46.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:22431]: starting ceph client 2022-07-20T03:13:46.593 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22431]: starting fuse 2022-07-20T03:13:46.604 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:13:46.607 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:13:46.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:46.609 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:13:46.654 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:13:46.655 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:13:46.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:46.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:13:46.658 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:13:46.660 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:13:46.661 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:13:46.661 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:13:46.662 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:13:46.663 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:13:46.664 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:13:46.665 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:13:46.666 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:13:46.667 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:13:46.668 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:13:46.669 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:13:46.670 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:13:46.671 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:13:46.672 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:13:46.673 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:13:46.674 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:13:46.675 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:13:46.676 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:13:46.730 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:13:46.731 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:13:46.733 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:13:46.785 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22431.asok 2022-07-20T03:13:46.799 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22431.asok 2022-07-20T03:13:46.801 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:46.802 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22431.asok status 2022-07-20T03:13:46.959 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:13:46.963 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:13:46.963 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:13:46.963 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:13:46.963 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:13:46.964 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:13:46.964 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:13:46.964 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22431", 2022-07-20T03:13:46.964 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:13:46.964 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:13:46.967 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:13:46.967 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:13:46.968 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 5807, 2022-07-20T03:13:46.968 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:13:46.969 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:13:46.969 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:13:46.969 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 5807 2022-07-20T03:13:46.970 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:13:46.970 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:13:46.970 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:13:46.971 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:13:46.971 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2426976566 2022-07-20T03:13:46.972 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:13:46.972 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:13:46.972 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:13:46.973 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:13:46.997 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:13:46.998 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2426976566 2022-07-20T03:13:46.998 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:13:46.999 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.5807 172.21.15.163:0/2426976566", 2022-07-20T03:13:46.999 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/2426976566", 2022-07-20T03:13:47.000 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:13:47.000 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 40, 2022-07-20T03:13:47.000 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 71, 2022-07-20T03:13:47.001 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:13:47.001 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:13:47.035 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:13:47.035 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:13:47.037 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:47.038 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:47.039 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:47.057 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:13:47.058 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:13:47.059 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:47.060 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:47.126 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:47.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 Processor -- start 2022-07-20T03:13:47.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 -- start start 2022-07-20T03:13:47.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e80b08c0 0x7f79e8145b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f79e8060d80 0x7f79e80adf50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e80b1d10 con 0x7f79e80b61d0 2022-07-20T03:13:47.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80ad480 con 0x7f79e80b08c0 2022-07-20T03:13:47.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e7fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80b65b0 con 0x7f79e8060d80 2022-07-20T03:13:47.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e80b08c0 0x7f79e8145b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f79e80b61d0 0x7f79e8145400 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43508/0 (socket says 172.21.15.61:43508) 2022-07-20T03:13:47.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e6ffd700 1 -- 172.21.15.61:0/836471002 learned_addr learned my addr 172.21.15.61:0/836471002 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:47.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.244+0000 7f79e77fe700 1 --2- 172.21.15.61:0/836471002 >> v2:172.21.15.117:3300/0 conn(0x7f79e8060d80 0x7f79e80adf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e67fc700 1 -- 172.21.15.61:0/836471002 >> v2:172.21.15.117:3300/0 conn(0x7f79e8060d80 msgr2=0x7f79e80adf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:47.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e67fc700 1 --2- 172.21.15.61:0/836471002 >> v2:172.21.15.117:3300/0 conn(0x7f79e8060d80 0x7f79e80adf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e67fc700 1 -- 172.21.15.61:0/836471002 >> v1:172.21.15.61:6789/0 conn(0x7f79e80b61d0 legacy=0x7f79e8145400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e67fc700 1 -- 172.21.15.61:0/836471002 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e80b1f70 con 0x7f79e80b08c0 2022-07-20T03:13:47.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e5ffb700 1 -- 172.21.15.61:0/836471002 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2448392843 0 0) 0x7f79e80b1d10 con 0x7f79e80b61d0 2022-07-20T03:13:47.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e67fc700 1 --2- 172.21.15.61:0/836471002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b08c0 0x7f79e8145b10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f79d8002f70 tx=0x7f79d8006df0).ready entity=mon.0 client_cookie=734bc10097aaca27 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:47.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e5ffb700 1 -- 172.21.15.61:0/836471002 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79d800d5c0 con 0x7f79e80b08c0 2022-07-20T03:13:47.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e5ffb700 1 -- 172.21.15.61:0/836471002 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79d8000e90 con 0x7f79e80b08c0 2022-07-20T03:13:47.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e5ffb700 1 -- 172.21.15.61:0/836471002 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79d800d910 con 0x7f79e80b08c0 2022-07-20T03:13:47.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 -- 172.21.15.61:0/836471002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b08c0 msgr2=0x7f79e8145b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 --2- 172.21.15.61:0/836471002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b08c0 0x7f79e8145b10 secure :-1 s=READY pgs=743 cs=0 l=1 rev1=1 rx=0x7f79d8002f70 tx=0x7f79d8006df0).stop 2022-07-20T03:13:47.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 -- 172.21.15.61:0/836471002 shutdown_connections 2022-07-20T03:13:47.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 --2- 172.21.15.61:0/836471002 >> v2:172.21.15.117:3300/0 conn(0x7f79e8060d80 0x7f79e80adf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 --2- 172.21.15.61:0/836471002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b08c0 0x7f79e8145b10 unknown :-1 s=CLOSED pgs=743 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 -- 172.21.15.61:0/836471002 >> 172.21.15.61:0/836471002 conn(0x7f79e8063cf0 msgr2=0x7f79e80b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:47.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 -- 172.21.15.61:0/836471002 shutdown_connections 2022-07-20T03:13:47.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 -- 172.21.15.61:0/836471002 wait complete. 2022-07-20T03:13:47.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 Processor -- start 2022-07-20T03:13:47.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.248+0000 7f79e7fff700 1 -- start start 2022-07-20T03:13:47.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 0x7f79e81440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e80b08c0 0x7f79e813d4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b61d0 0x7f79e813d9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e813fe00 con 0x7f79e8060d80 2022-07-20T03:13:47.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e813fb60 con 0x7f79e80b08c0 2022-07-20T03:13:47.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e81431b0 con 0x7f79e80b61d0 2022-07-20T03:13:47.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 0x7f79e81440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 0x7f79e81440c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46916/0 (socket says 172.21.15.61:46916) 2022-07-20T03:13:47.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 -- 172.21.15.61:0/370579205 learned_addr learned my addr 172.21.15.61:0/370579205 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:47.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e67fc700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e80b08c0 0x7f79e813d4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e77fe700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b61d0 0x7f79e813d9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 -- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b61d0 msgr2=0x7f79e813d9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b61d0 0x7f79e813d9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 -- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e80b08c0 msgr2=0x7f79e813d4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e80b08c0 0x7f79e813d4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 -- 172.21.15.61:0/370579205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79d800b180 con 0x7f79e8060d80 2022-07-20T03:13:47.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e67fc700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e80b08c0 0x7f79e813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:47.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e6ffd700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 0x7f79e81440c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f79dc000ed0 tx=0x7f79dc007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:47.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79dc00ce20 con 0x7f79e8060d80 2022-07-20T03:13:47.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79dc0072e0 con 0x7f79e8060d80 2022-07-20T03:13:47.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e8144d10 con 0x7f79e8060d80 2022-07-20T03:13:47.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79dc005610 con 0x7f79e8060d80 2022-07-20T03:13:47.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e8143650 con 0x7f79e8060d80 2022-07-20T03:13:47.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.252+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e8143370 con 0x7f79e8060d80 2022-07-20T03:13:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.256+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f79dc01a060 con 0x7f79e8060d80 2022-07-20T03:13:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.256+0000 7f79d7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c4053a40 0x7f79c4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.256+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f79dc06b740 con 0x7f79e8060d80 2022-07-20T03:13:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.256+0000 7f79e67fc700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c4053a40 0x7f79c4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.256+0000 7f79e67fc700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c4053a40 0x7f79c4055f00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f79d8018000 tx=0x7f79d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.260+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79dc042590 con 0x7f79e8060d80 2022-07-20T03:13:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.420+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f79e80574e0 con 0x7f79e8060d80 2022-07-20T03:13:47.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79d7fff700 1 -- 172.21.15.61:0/370579205 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f79dc018070 con 0x7f79e8060d80 2022-07-20T03:13:47.458 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:47.476 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:13:47.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c4053a40 msgr2=0x7f79c4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c4053a40 0x7f79c4055f00 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f79d8018000 tx=0x7f79d8014000).stop 2022-07-20T03:13:47.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 msgr2=0x7f79e81440c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 0x7f79e81440c0 secure :-1 s=READY pgs=744 cs=0 l=1 rev1=1 rx=0x7f79dc000ed0 tx=0x7f79dc007ce0).stop 2022-07-20T03:13:47.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 shutdown_connections 2022-07-20T03:13:47.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c4053a40 0x7f79c4055f00 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e80b08c0 0x7f79e813d4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b61d0 0x7f79e813d9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 --2- 172.21.15.61:0/370579205 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e8060d80 0x7f79e81440c0 unknown :-1 s=CLOSED pgs=744 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 >> 172.21.15.61:0/370579205 conn(0x7f79e8063cf0 msgr2=0x7f79e80adab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 shutdown_connections 2022-07-20T03:13:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.424+0000 7f79e7fff700 1 -- 172.21.15.61:0/370579205 wait complete. 2022-07-20T03:13:47.537 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:13:47.539 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:13:47.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.648+0000 7f0aeb0e0700 1 Processor -- start 2022-07-20T03:13:47.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.648+0000 7f0aeb0e0700 1 -- start start 2022-07-20T03:13:47.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0aeb0e0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0ae405c3f0 0x7f0ae414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0aeb0e0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae405d480 con 0x7f0ae40b1e00 2022-07-20T03:13:47.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0aeb0e0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0ae40b21e0 con 0x7f0ae40acef0 2022-07-20T03:13:47.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0aeb0e0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0ae405cf40 con 0x7f0ae405c3f0 2022-07-20T03:13:47.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0ae40b1e00 0x7f0ae40ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43522/0 (socket says 172.21.15.61:43522) 2022-07-20T03:13:47.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2413236883 learned_addr learned my addr 172.21.15.61:0/2413236883 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:13:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae3fff700 1 --2- 172.21.15.61:0/2413236883 >> v2:172.21.15.117:3300/0 conn(0x7f0ae405c3f0 0x7f0ae414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420146884 0 0) 0x7f0ae405d480 con 0x7f0ae40b1e00 2022-07-20T03:13:47.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0ad0003260 con 0x7f0ae40b1e00 2022-07-20T03:13:47.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1411255180 0 0) 0x7f0ad0003260 con 0x7f0ae40b1e00 2022-07-20T03:13:47.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 >> v2:172.21.15.117:3300/0 conn(0x7f0ae405c3f0 msgr2=0x7f0ae414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:47.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 --2- 172.21.15.61:0/2413236883 >> v2:172.21.15.117:3300/0 conn(0x7f0ae405c3f0 0x7f0ae414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 >> v1:172.21.15.117:6789/0 conn(0x7f0ae40acef0 legacy=0x7f0ae4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae405d1e0 con 0x7f0ae40b1e00 2022-07-20T03:13:47.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f0adc0019e0 con 0x7f0ae40b1e00 2022-07-20T03:13:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 >> v1:172.21.15.61:6789/0 conn(0x7f0ae40b1e00 legacy=0x7f0ae40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad0003750 0x7f0ad0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad0004500 0x7f0ad0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad0004ec0 0x7f0ad000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae405cf40 con 0x7f0ad0003750 2022-07-20T03:13:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad000be50 con 0x7f0ad0004500 2022-07-20T03:13:47.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ad000c010 con 0x7f0ad0004ec0 2022-07-20T03:13:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad0003750 0x7f0ad0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae3fff700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad0004500 0x7f0ad0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae967d700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad0004ec0 0x7f0ad000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad0004ec0 msgr2=0x7f0ad000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ad0004ec0 0x7f0ad000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad0004500 msgr2=0x7f0ad0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:13:47.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ad0004500 0x7f0ad0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae967d700 1 -- 172.21.15.61:0/2413236883 reap_dead start 2022-07-20T03:13:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2413236883 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ad0003260 con 0x7f0ad0003750 2022-07-20T03:13:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad0003750 0x7f0ad0003b70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f0adc0016f0 tx=0x7f0ae40ac4d0).ready entity=mon.0 client_cookie=7e2ec9b3eabf33b5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:47.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0adc00cc40 con 0x7f0ad0003750 2022-07-20T03:13:47.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0adc00cda0 con 0x7f0ad0003750 2022-07-20T03:13:47.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.652+0000 7f0ae37fe700 1 -- 172.21.15.61:0/2413236883 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0adc005b80 con 0x7f0ad0003750 2022-07-20T03:13:47.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2413236883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad0003750 msgr2=0x7f0ad0003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad0003750 0x7f0ad0003b70 secure :-1 s=READY pgs=746 cs=0 l=1 rev1=1 rx=0x7f0adc0016f0 tx=0x7f0ae40ac4d0).stop 2022-07-20T03:13:47.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2413236883 shutdown_connections 2022-07-20T03:13:47.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2413236883 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ad0003750 0x7f0ad0003b70 unknown :-1 s=CLOSED pgs=746 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2413236883 >> 172.21.15.61:0/2413236883 conn(0x7f0ae40611c0 msgr2=0x7f0ae4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:47.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2413236883 shutdown_connections 2022-07-20T03:13:47.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2413236883 wait complete. 2022-07-20T03:13:47.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 Processor -- start 2022-07-20T03:13:47.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- start start 2022-07-20T03:13:47.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 0x7f0ae41432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ae4065ec0 0x7f0ae4064220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4064710 0x7f0ae4142690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae4063c10 con 0x7f0ae414aa00 2022-07-20T03:13:47.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae414ade0 con 0x7f0ae4064710 2022-07-20T03:13:47.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.656+0000 7f0aeb0e0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0ae40b5ac0 con 0x7f0ae4065ec0 2022-07-20T03:13:47.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 0x7f0ae41432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 0x7f0ae41432a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46936/0 (socket says 172.21.15.61:46936) 2022-07-20T03:13:47.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2005058660 learned_addr learned my addr 172.21.15.61:0/2005058660 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:13:47.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae3fff700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ae4065ec0 0x7f0ae4064220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae967d700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4064710 0x7f0ae4142690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ae4065ec0 msgr2=0x7f0ae4064220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ae4065ec0 0x7f0ae4064220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4064710 msgr2=0x7f0ae4142690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4064710 0x7f0ae4142690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 -- 172.21.15.61:0/2005058660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ae414d370 con 0x7f0ae414aa00 2022-07-20T03:13:47.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae967d700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4064710 0x7f0ae4142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:47.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae3fff700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ae4065ec0 0x7f0ae4064220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:13:47.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae8e7c700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 0x7f0ae41432a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f0adc0016f0 tx=0x7f0adc013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:47.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0adc001210 con 0x7f0ae414aa00 2022-07-20T03:13:47.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0adc004ab0 con 0x7f0ae414aa00 2022-07-20T03:13:47.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ae4143790 con 0x7f0ae414aa00 2022-07-20T03:13:47.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0adc004e10 con 0x7f0ae414aa00 2022-07-20T03:13:47.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ae4142b80 con 0x7f0ae414aa00 2022-07-20T03:13:47.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ae40b5c30 con 0x7f0ae414aa00 2022-07-20T03:13:47.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.660+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0adc025070 con 0x7f0ae414aa00 2022-07-20T03:13:47.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.664+0000 7f0ae1ffb700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ac4053ad0 0x7f0ac4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:13:47.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.664+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(71..71 src has 1..71) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0adc06c450 con 0x7f0ae414aa00 2022-07-20T03:13:47.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.664+0000 7f0ae3fff700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ac4053ad0 0x7f0ac4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:13:47.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.664+0000 7f0ae3fff700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ac4053ad0 0x7f0ac4055f90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f0ad800b000 tx=0x7f0ad8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:13:47.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.668+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0ae40b5c30 con 0x7f0ae414aa00 2022-07-20T03:13:47.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0ae40b5c30 con 0x7f0ae414aa00 2022-07-20T03:13:47.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0ae1ffb700 1 -- 172.21.15.61:0/2005058660 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f0ae40b5c30 con 0x7f0ae414aa00 2022-07-20T03:13:47.877 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:13:47.887 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:13:47.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ac4053ad0 msgr2=0x7f0ac4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ac4053ad0 0x7f0ac4055f90 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f0ad800b000 tx=0x7f0ad8008040).stop 2022-07-20T03:13:47.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 msgr2=0x7f0ae41432a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:13:47.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 0x7f0ae41432a0 secure :-1 s=READY pgs=747 cs=0 l=1 rev1=1 rx=0x7f0adc0016f0 tx=0x7f0adc013000).stop 2022-07-20T03:13:47.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 shutdown_connections 2022-07-20T03:13:47.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ac4053ad0 0x7f0ac4055f90 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0ae4064710 0x7f0ae4142690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0ae4065ec0 0x7f0ae4064220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 --2- 172.21.15.61:0/2005058660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0ae414aa00 0x7f0ae41432a0 unknown :-1 s=CLOSED pgs=747 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:13:47.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 >> 172.21.15.61:0/2005058660 conn(0x7f0ad0003750 msgr2=0x7f0ae4147340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:13:47.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 shutdown_connections 2022-07-20T03:13:47.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:13:47.832+0000 7f0aeb0e0700 1 -- 172.21.15.61:0/2005058660 wait complete. 2022-07-20T03:13:47.917 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:13:47.921 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json config set mds_cap_revoke_eviction_timeout 30.0 2022-07-20T03:13:48.033 INFO:teuthology.orchestra.run.smithi061.stdout:{"success":"mds_cap_revoke_eviction_timeout = '30.000000' "} 2022-07-20T03:13:48.034 DEBUG:tasks.cephfs.filesystem:_json_asok output { "success": "mds_cap_revoke_eviction_timeout = '30.000000' " } 2022-07-20T03:13:48.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:48.036 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:48.037 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:13:48.082 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:13:48.083 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:13:48.086 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:13:48.086 DEBUG:teuthology.orchestra.run.smithi124:> import time 2022-07-20T03:13:48.087 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:13:48.087 DEBUG:teuthology.orchestra.run.smithi124:> with open("/home/ubuntu/cephtest/mnt.0/background_file", '"'"'w'"'"') as f: 2022-07-20T03:13:48.088 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content'"'"') 2022-07-20T03:13:48.088 DEBUG:teuthology.orchestra.run.smithi124:> f.flush() 2022-07-20T03:13:48.088 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content2'"'"') 2022-07-20T03:13:48.089 DEBUG:teuthology.orchestra.run.smithi124:> while True: 2022-07-20T03:13:48.089 DEBUG:teuthology.orchestra.run.smithi124:> time.sleep(1) 2022-07-20T03:13:48.090 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:13:48.127 DEBUG:teuthology.orchestra.run.smithi124:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-20T03:13:48.144 INFO:teuthology.orchestra.run.smithi124.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/background_file': No such file or directory 2022-07-20T03:13:48.146 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-20T03:13:49.148 DEBUG:teuthology.orchestra.run.smithi124:> sudo ls /home/ubuntu/cephtest/mnt.0/background_file 2022-07-20T03:13:49.183 INFO:teuthology.orchestra.run.smithi124.stdout:/home/ubuntu/cephtest/mnt.0/background_file 2022-07-20T03:13:49.184 DEBUG:tasks.cephfs.mount:File background_file became visible from 0 after 1s 2022-07-20T03:13:49.185 DEBUG:teuthology.orchestra.run.smithi163:> sudo ls /home/ubuntu/cephtest/mnt.1/background_file 2022-07-20T03:13:49.233 INFO:teuthology.orchestra.run.smithi163.stdout:/home/ubuntu/cephtest/mnt.1/background_file 2022-07-20T03:13:49.235 DEBUG:tasks.cephfs.mount:File background_file became visible from 1 after 0s 2022-07-20T03:13:49.236 INFO:tasks.cephfs.fuse_mount:Killing ceph-fuse connection on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:13:49.253 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:13:49.254 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:13:49.255 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:13:49.304 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:13:49.305 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:13:49.306 DEBUG:teuthology.orchestra.run.smithi163:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:13:49.307 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:13:49.307 DEBUG:teuthology.orchestra.run.smithi163:> import time 2022-07-20T03:13:49.308 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:13:49.309 DEBUG:teuthology.orchestra.run.smithi163:> fd = os.open("/home/ubuntu/cephtest/mnt.1/background_file", os.O_RDWR | os.O_CREAT, 0o644) 2022-07-20T03:13:49.310 DEBUG:teuthology.orchestra.run.smithi163:> try: 2022-07-20T03:13:49.311 DEBUG:teuthology.orchestra.run.smithi163:> while True: 2022-07-20T03:13:49.312 DEBUG:teuthology.orchestra.run.smithi163:> os.write(fd, b'"'"'content'"'"') 2022-07-20T03:13:49.313 DEBUG:teuthology.orchestra.run.smithi163:> time.sleep(1) 2022-07-20T03:13:49.329 DEBUG:teuthology.orchestra.run.smithi163:> if not False: 2022-07-20T03:13:49.332 DEBUG:teuthology.orchestra.run.smithi163:> break 2022-07-20T03:13:49.333 DEBUG:teuthology.orchestra.run.smithi163:> except IOError as e: 2022-07-20T03:13:49.335 DEBUG:teuthology.orchestra.run.smithi163:> pass 2022-07-20T03:13:49.336 DEBUG:teuthology.orchestra.run.smithi163:> os.close(fd) 2022-07-20T03:13:49.336 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:14:06.283 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:06.331 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:06.345 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:06.353 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:06.389 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:28.523 INFO:tasks.cephfs.test_misc:cap_waiter waited 39.12944054603577s 2022-07-20T03:14:28.524 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist ls --format=json 2022-07-20T03:14:28.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.634+0000 7f4c8a37b700 1 Processor -- start 2022-07-20T03:14:28.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.634+0000 7f4c8a37b700 1 -- start start 2022-07-20T03:14:28.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c84060ca0 0x7f4c84061080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:28.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4c84097a80 0x7f4c8414ab80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c8414a750 con 0x7f4c840afb80 2022-07-20T03:14:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c84148430 con 0x7f4c84097a80 2022-07-20T03:14:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c840b2840 con 0x7f4c84060ca0 2022-07-20T03:14:28.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c88918700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4c840afb80 0x7f4c8414b070 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43548/0 (socket says 172.21.15.61:43548) 2022-07-20T03:14:28.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c88918700 1 -- 172.21.15.61:0/1981575695 learned_addr learned my addr 172.21.15.61:0/1981575695 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:28.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c7bfff700 1 --2- 172.21.15.61:0/1981575695 >> v2:172.21.15.117:3300/0 conn(0x7f4c84097a80 0x7f4c8414ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:28.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c83fff700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c84060ca0 0x7f4c84061080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:28.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c837fe700 1 -- 172.21.15.61:0/1981575695 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2249941628 0 0) 0x7f4c8414a750 con 0x7f4c840afb80 2022-07-20T03:14:28.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c837fe700 1 -- 172.21.15.61:0/1981575695 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c68003260 con 0x7f4c840afb80 2022-07-20T03:14:28.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c7bfff700 1 -- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c84060ca0 msgr2=0x7f4c84061080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:28.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c7bfff700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c84060ca0 0x7f4c84061080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c7bfff700 1 -- 172.21.15.61:0/1981575695 >> v1:172.21.15.61:6789/0 conn(0x7f4c840afb80 legacy=0x7f4c8414b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:28.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c7bfff700 1 -- 172.21.15.61:0/1981575695 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c840ac110 con 0x7f4c84097a80 2022-07-20T03:14:28.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c83fff700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c84060ca0 0x7f4c84061080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:28.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c7bfff700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414ab80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f4c74002f70 tx=0x7f4c74006df0).ready entity=mon.1 client_cookie=2ce697d97a5ce2ba server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:28.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c837fe700 1 -- 172.21.15.61:0/1981575695 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c7400d4a0 con 0x7f4c84097a80 2022-07-20T03:14:28.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c837fe700 1 -- 172.21.15.61:0/1981575695 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c74000e90 con 0x7f4c84097a80 2022-07-20T03:14:28.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c837fe700 1 -- 172.21.15.61:0/1981575695 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c7400d7f0 con 0x7f4c84097a80 2022-07-20T03:14:28.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 msgr2=0x7f4c8414ab80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:28.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414ab80 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7f4c74002f70 tx=0x7f4c74006df0).stop 2022-07-20T03:14:28.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1981575695 shutdown_connections 2022-07-20T03:14:28.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414ab80 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1981575695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c84060ca0 0x7f4c84061080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1981575695 >> 172.21.15.61:0/1981575695 conn(0x7f4c840b5cb0 msgr2=0x7f4c840b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:28.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1981575695 shutdown_connections 2022-07-20T03:14:28.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.638+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1981575695 wait complete. 2022-07-20T03:14:28.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 Processor -- start 2022-07-20T03:14:28.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 -- start start 2022-07-20T03:14:28.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 0x7f4c840b0c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:28.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414b100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:28.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c840afb80 0x7f4c8414b5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:28.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c841419c0 con 0x7f4c84060ca0 2022-07-20T03:14:28.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8414aaa0 con 0x7f4c84097a80 2022-07-20T03:14:28.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c8a37b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8414c920 con 0x7f4c840afb80 2022-07-20T03:14:28.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 0x7f4c840b0c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 0x7f4c840b0c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46952/0 (socket says 172.21.15.61:46952) 2022-07-20T03:14:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c83fff700 1 -- 172.21.15.61:0/1189142961 learned_addr learned my addr 172.21.15.61:0/1189142961 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c7bfff700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414b100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.642+0000 7f4c88918700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c840afb80 0x7f4c8414b5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c83fff700 1 -- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c840afb80 msgr2=0x7f4c8414b5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c83fff700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c840afb80 0x7f4c8414b5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c83fff700 1 -- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 msgr2=0x7f4c8414b100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:28.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c83fff700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414b100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c83fff700 1 -- 172.21.15.61:0/1189142961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c6c00b180 con 0x7f4c84060ca0 2022-07-20T03:14:28.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c83fff700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 0x7f4c840b0c50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f4c6c00c000 tx=0x7f4c6c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:28.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c6c00c7e0 con 0x7f4c84060ca0 2022-07-20T03:14:28.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c6c00c940 con 0x7f4c84060ca0 2022-07-20T03:14:28.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c7400b180 con 0x7f4c84060ca0 2022-07-20T03:14:28.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c6c00cca0 con 0x7f4c84060ca0 2022-07-20T03:14:28.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8414cf50 con 0x7f4c84060ca0 2022-07-20T03:14:28.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c840abe40 con 0x7f4c84060ca0 2022-07-20T03:14:28.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4c6c01c460 con 0x7f4c84060ca0 2022-07-20T03:14:28.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c81ffb700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c5c053a40 0x7f4c5c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:28.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f4c6c06c510 con 0x7f4c84060ca0 2022-07-20T03:14:28.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.646+0000 7f4c7bfff700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c5c053a40 0x7f4c5c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:28.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.650+0000 7f4c7bfff700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c5c053a40 0x7f4c5c055f00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4c7401a000 tx=0x7f4c74014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:28.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.654+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c6c0432d0 con 0x7f4c84060ca0 2022-07-20T03:14:28.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.790+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist ls", "format": "json"} v 0) v1 -- 0x7f4c840574e0 con 0x7f4c84060ca0 2022-07-20T03:14:28.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.790+0000 7f4c81ffb700 1 -- 172.21.15.61:0/1189142961 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist ls", "format": "json"}]=0 listed 1 entries v72) v1 ==== 98+0+81 (secure 0 0 0) 0x7f4c6c0468f0 con 0x7f4c84060ca0 2022-07-20T03:14:28.837 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:28.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c5c053a40 msgr2=0x7f4c5c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:28.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c5c053a40 0x7f4c5c055f00 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f4c7401a000 tx=0x7f4c74014000).stop 2022-07-20T03:14:28.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 msgr2=0x7f4c840b0c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:28.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 0x7f4c840b0c50 secure :-1 s=READY pgs=749 cs=0 l=1 rev1=1 rx=0x7f4c6c00c000 tx=0x7f4c6c006df0).stop 2022-07-20T03:14:28.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 shutdown_connections 2022-07-20T03:14:28.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c5c053a40 0x7f4c5c055f00 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c84097a80 0x7f4c8414b100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c840afb80 0x7f4c8414b5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 --2- 172.21.15.61:0/1189142961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c84060ca0 0x7f4c840b0c50 unknown :-1 s=CLOSED pgs=749 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:28.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 >> 172.21.15.61:0/1189142961 conn(0x7f4c840b5cb0 msgr2=0x7f4c84144490 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:28.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 shutdown_connections 2022-07-20T03:14:28.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:28.794+0000 7f4c8a37b700 1 -- 172.21.15.61:0/1189142961 wait complete. 2022-07-20T03:14:28.871 INFO:teuthology.orchestra.run.smithi061.stderr:listed 1 entries 2022-07-20T03:14:28.872 INFO:teuthology.orchestra.run.smithi061.stdout:[{"addr":"172.21.15.124:0/2780278373","until":"2022-07-20T04:14:22.364604+0000"}] 2022-07-20T03:14:28.925 INFO:tasks.cephfs.fuse_mount:Cleaning up killed ceph-fuse connection 2022-07-20T03:14:28.942 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:28.961 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:28.961 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:28.982 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:28.984 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Transport endpoint is not connected 2022-07-20T03:14:29.011 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T03:14:29.012 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:14:29.013 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:14:29.032 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:29.068 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:29.069 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:29.069 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:29.121 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:14:29.122 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:14:29.123 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:29.124 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:29.178 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:29.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 Processor -- start 2022-07-20T03:14:29.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- start start 2022-07-20T03:14:29.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f854c0adff0 0x7f854c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c147010 con 0x7f854c0af8e0 2022-07-20T03:14:29.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f854c1472b0 con 0x7f854c0b5650 2022-07-20T03:14:29.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f854c146d40 con 0x7f854c0adff0 2022-07-20T03:14:29.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854ad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f854c0af8e0 0x7f854c05bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43562/0 (socket says 172.21.15.61:43562) 2022-07-20T03:14:29.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854ad9d700 1 -- 172.21.15.61:0/1330642272 learned_addr learned my addr 172.21.15.61:0/1330642272 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:29.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 --2- 172.21.15.61:0/1330642272 >> v2:172.21.15.117:3300/0 conn(0x7f854c0adff0 0x7f854c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854959a700 1 -- 172.21.15.61:0/1330642272 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 357336680 0 0) 0x7f854c147010 con 0x7f854c0af8e0 2022-07-20T03:14:29.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854959a700 1 -- 172.21.15.61:0/1330642272 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8530003260 con 0x7f854c0af8e0 2022-07-20T03:14:29.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/1330642272 >> v1:172.21.15.117:6789/0 conn(0x7f854c0b5650 legacy=0x7f854c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/1330642272 >> v1:172.21.15.61:6789/0 conn(0x7f854c0af8e0 legacy=0x7f854c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/1330642272 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f854c05c4a0 con 0x7f854c0adff0 2022-07-20T03:14:29.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 --2- 172.21.15.61:0/1330642272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0adff0 0x7f854c0b5a30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f853c002f70 tx=0x7f853c006d90).ready entity=mon.1 client_cookie=b5a06a3c16be30d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:29.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854959a700 1 -- 172.21.15.61:0/1330642272 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853c00d4a0 con 0x7f854c0adff0 2022-07-20T03:14:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854959a700 1 -- 172.21.15.61:0/1330642272 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c000e90 con 0x7f854c0adff0 2022-07-20T03:14:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854959a700 1 -- 172.21.15.61:0/1330642272 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853c00d7f0 con 0x7f854c0adff0 2022-07-20T03:14:29.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- 172.21.15.61:0/1330642272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0adff0 msgr2=0x7f854c0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 --2- 172.21.15.61:0/1330642272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0adff0 0x7f854c0b5a30 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f853c002f70 tx=0x7f853c006d90).stop 2022-07-20T03:14:29.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- 172.21.15.61:0/1330642272 shutdown_connections 2022-07-20T03:14:29.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 --2- 172.21.15.61:0/1330642272 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0adff0 0x7f854c0b5a30 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- 172.21.15.61:0/1330642272 >> 172.21.15.61:0/1330642272 conn(0x7f854c0634e0 msgr2=0x7f854c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:29.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- 172.21.15.61:0/1330642272 shutdown_connections 2022-07-20T03:14:29.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- 172.21.15.61:0/1330642272 wait complete. 2022-07-20T03:14:29.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 Processor -- start 2022-07-20T03:14:29.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- start start 2022-07-20T03:14:29.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f854c0adff0 0x7f854c064240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 0x7f854c10ed40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0b5650 0x7f854c10f230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c143730 con 0x7f854c0af8e0 2022-07-20T03:14:29.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f854c110ae0 con 0x7f854c0b5650 2022-07-20T03:14:29.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f854c110d80 con 0x7f854c0adff0 2022-07-20T03:14:29.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 0x7f854c10ed40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 0x7f854c10ed40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46968/0 (socket says 172.21.15.61:46968) 2022-07-20T03:14:29.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/225137511 learned_addr learned my addr 172.21.15.61:0/225137511 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:29.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854ad9d700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0b5650 0x7f854c10f230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f854a59c700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f854c0adff0 0x7f854c064240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f854c0adff0 msgr2=0x7f854c064240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f854c0adff0 0x7f854c064240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0b5650 msgr2=0x7f854c10f230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0b5650 0x7f854c10f230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.290+0000 7f8549d9b700 1 -- 172.21.15.61:0/225137511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f853c00b120 con 0x7f854c0af8e0 2022-07-20T03:14:29.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f854ad9d700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0b5650 0x7f854c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:29.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f8549d9b700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 0x7f854c10ed40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f853c002f70 tx=0x7f853c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:29.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853c020070 con 0x7f854c0af8e0 2022-07-20T03:14:29.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f853c011a20 con 0x7f854c0af8e0 2022-07-20T03:14:29.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f853c011d80 con 0x7f854c0af8e0 2022-07-20T03:14:29.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f854c111020 con 0x7f854c0af8e0 2022-07-20T03:14:29.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f854c063990 con 0x7f854c0af8e0 2022-07-20T03:14:29.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f854c10eb40 con 0x7f854c0af8e0 2022-07-20T03:14:29.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f853c00f070 con 0x7f854c0af8e0 2022-07-20T03:14:29.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f853b7fe700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8528053af0 0x7f8528055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.294+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f853c06c970 con 0x7f854c0af8e0 2022-07-20T03:14:29.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.298+0000 7f854a59c700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8528053af0 0x7f8528055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.298+0000 7f854a59c700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8528053af0 0x7f8528055fb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f854000b000 tx=0x7f8540009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:29.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.302+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f853c043790 con 0x7f854c0af8e0 2022-07-20T03:14:29.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.430+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f854c0574e0 con 0x7f854c0af8e0 2022-07-20T03:14:29.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.430+0000 7f853b7fe700 1 -- 172.21.15.61:0/225137511 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v72) v1 ==== 74+0+16151 (secure 0 0 0) 0x7f853c006920 con 0x7f854c0af8e0 2022-07-20T03:14:29.535 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:29.536 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:23.218756+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":13,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:13:36.009693+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"70","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":14,"pool_name":"cephfs_data","create_time":"2022-07-20T03:13:36.517295+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"71","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.124:0/2780278373":"2022-07-20T04:14:22.364604+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:29.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.430+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8528053af0 msgr2=0x7f8528055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.430+0000 7f854b59e700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8528053af0 0x7f8528055fb0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f854000b000 tx=0x7f8540009040).stop 2022-07-20T03:14:29.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.430+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 msgr2=0x7f854c10ed40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.430+0000 7f854b59e700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 0x7f854c10ed40 secure :-1 s=READY pgs=751 cs=0 l=1 rev1=1 rx=0x7f853c002f70 tx=0x7f853c014000).stop 2022-07-20T03:14:29.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 shutdown_connections 2022-07-20T03:14:29.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8528053af0 0x7f8528055fb0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f854c0b5650 0x7f854c10f230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f854c0af8e0 0x7f854c10ed40 unknown :-1 s=CLOSED pgs=751 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 --2- 172.21.15.61:0/225137511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f854c0adff0 0x7f854c064240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 >> 172.21.15.61:0/225137511 conn(0x7f854c0634e0 msgr2=0x7f854c1460a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:29.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 shutdown_connections 2022-07-20T03:14:29.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.434+0000 7f854b59e700 1 -- 172.21.15.61:0/225137511 wait complete. 2022-07-20T03:14:29.616 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 13, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:13:36.009693+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '70', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 14, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:13:36.517295+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '71', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:14:29.617 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:14:29.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 Processor -- start 2022-07-20T03:14:29.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- start start 2022-07-20T03:14:29.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdab40b59f0 0x7fdab405c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab405d2f0 con 0x7fdab40af940 2022-07-20T03:14:29.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdab40b5dd0 con 0x7fdab40addd0 2022-07-20T03:14:29.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdab405ccd0 con 0x7fdab40b59f0 2022-07-20T03:14:29.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdab40af940 0x7fdab405c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43574/0 (socket says 172.21.15.61:43574) 2022-07-20T03:14:29.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdab40b59f0 0x7fdab405c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 -- 172.21.15.61:0/764229607 learned_addr learned my addr 172.21.15.61:0/764229607 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:29.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2161226401 0 0) 0x7fdab405d2f0 con 0x7fdab40af940 2022-07-20T03:14:29.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda94003260 con 0x7fdab40af940 2022-07-20T03:14:29.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3802757572 0 0) 0x7fda94003260 con 0x7fdab40af940 2022-07-20T03:14:29.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 >> v2:172.21.15.61:3300/0 conn(0x7fdab40b59f0 msgr2=0x7fdab405c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 --2- 172.21.15.61:0/764229607 >> v2:172.21.15.61:3300/0 conn(0x7fdab40b59f0 0x7fdab405c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 >> v1:172.21.15.117:6789/0 conn(0x7fdab40addd0 legacy=0x7fdab4142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab41460b0 con 0x7fdab40af940 2022-07-20T03:14:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fda9c001660 con 0x7fdab40af940 2022-07-20T03:14:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab3fff700 1 --2- 172.21.15.61:0/764229607 >> v2:172.21.15.61:3300/0 conn(0x7fdab40b59f0 0x7fdab405c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 >> v1:172.21.15.61:6789/0 conn(0x7fdab40af940 legacy=0x7fdab405c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda94003750 0x7fda94003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fda94004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda94004e70 0x7fda9400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9400bc40 con 0x7fda94004500 2022-07-20T03:14:29.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda9400bdb0 con 0x7fda94004e70 2022-07-20T03:14:29.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda9400bf70 con 0x7fda94003750 2022-07-20T03:14:29.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fda94004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab88f0700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda94004e70 0x7fda9400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 -- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda94003750 msgr2=0x7fda94003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:29.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda94003750 0x7fda94003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T03:14:29.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 -- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda94004e70 msgr2=0x7fda9400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:29.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda94004e70 0x7fda9400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 -- 172.21.15.61:0/764229607 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda94003260 con 0x7fda94004500 2022-07-20T03:14:29.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab88f0700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda94004e70 0x7fda9400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:29.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab88f0700 1 -- 172.21.15.61:0/764229607 reap_dead start 2022-07-20T03:14:29.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab37fe700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fda94004980 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fda9c001370 tx=0x7fdab40b2cc0).ready entity=mon.0 client_cookie=fb4573832c338264 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:29.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c0071a0 con 0x7fda94004500 2022-07-20T03:14:29.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda9c007300 con 0x7fda94004500 2022-07-20T03:14:29.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab2ffd700 1 -- 172.21.15.61:0/764229607 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda9c00cbb0 con 0x7fda94004500 2022-07-20T03:14:29.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- 172.21.15.61:0/764229607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 msgr2=0x7fda94004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fda94004980 secure :-1 s=READY pgs=753 cs=0 l=1 rev1=1 rx=0x7fda9c001370 tx=0x7fdab40b2cc0).stop 2022-07-20T03:14:29.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- 172.21.15.61:0/764229607 shutdown_connections 2022-07-20T03:14:29.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 --2- 172.21.15.61:0/764229607 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fda94004980 unknown :-1 s=CLOSED pgs=753 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.742+0000 7fdab90f1700 1 -- 172.21.15.61:0/764229607 >> 172.21.15.61:0/764229607 conn(0x7fdab4063480 msgr2=0x7fdab40b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:29.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- 172.21.15.61:0/764229607 shutdown_connections 2022-07-20T03:14:29.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- 172.21.15.61:0/764229607 wait complete. 2022-07-20T03:14:29.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 Processor -- start 2022-07-20T03:14:29.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- start start 2022-07-20T03:14:29.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fdab40b32b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdab4066160 0x7fdab41594a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdab4064b80 0x7fdab4159990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab4065580 con 0x7fda94004500 2022-07-20T03:14:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdab40642a0 con 0x7fdab4066160 2022-07-20T03:14:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdab4065120 con 0x7fdab4064b80 2022-07-20T03:14:29.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fdab40b32b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fdab40b32b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46988/0 (socket says 172.21.15.61:46988) 2022-07-20T03:14:29.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 -- 172.21.15.61:0/1348127358 learned_addr learned my addr 172.21.15.61:0/1348127358 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:29.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab37fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdab4066160 0x7fdab41594a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab88f0700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdab4064b80 0x7fdab4159990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 -- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdab4064b80 msgr2=0x7fdab4159990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdab4064b80 0x7fdab4159990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 -- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdab4066160 msgr2=0x7fdab41594a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdab4066160 0x7fdab41594a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 -- 172.21.15.61:0/1348127358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdab40644d0 con 0x7fda94004500 2022-07-20T03:14:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab3fff700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fdab40b32b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7fdaa400c000 tx=0x7fdaa4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:29.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdaa4016070 con 0x7fda94004500 2022-07-20T03:14:29.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdaa4000e90 con 0x7fda94004500 2022-07-20T03:14:29.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdab4063eb0 con 0x7fda94004500 2022-07-20T03:14:29.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdaa4010970 con 0x7fda94004500 2022-07-20T03:14:29.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.746+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdab415a0a0 con 0x7fda94004500 2022-07-20T03:14:29.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.750+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdab405cfe0 con 0x7fda94004500 2022-07-20T03:14:29.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.750+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdaa4010ad0 con 0x7fda94004500 2022-07-20T03:14:29.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.750+0000 7fdab17fa700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda98053ad0 0x7fda98055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:29.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.750+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fdaa406c0a0 con 0x7fda94004500 2022-07-20T03:14:29.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.750+0000 7fdab37fe700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda98053ad0 0x7fda98055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:29.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.750+0000 7fdab37fe700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda98053ad0 0x7fda98055f90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fda9c018000 tx=0x7fda9c013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:29.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.758+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdaa4042ec0 con 0x7fda94004500 2022-07-20T03:14:29.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fdab40574e0 con 0x7fda94004500 2022-07-20T03:14:29.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab17fa700 1 -- 172.21.15.61:0/1348127358 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v72) v1 ==== 110+0+10 (secure 0 0 0) 0x7fdaa40464e0 con 0x7fda94004500 2022-07-20T03:14:30.000 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:14:30.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda98053ad0 msgr2=0x7fda98055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda98053ad0 0x7fda98055f90 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fda9c018000 tx=0x7fda9c013000).stop 2022-07-20T03:14:30.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 msgr2=0x7fdab40b32b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fdab40b32b0 secure :-1 s=READY pgs=754 cs=0 l=1 rev1=1 rx=0x7fdaa400c000 tx=0x7fdaa4006df0).stop 2022-07-20T03:14:30.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 shutdown_connections 2022-07-20T03:14:30.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda98053ad0 0x7fda98055f90 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdab4066160 0x7fdab41594a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdab4064b80 0x7fdab4159990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 --2- 172.21.15.61:0/1348127358 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda94004500 0x7fdab40b32b0 unknown :-1 s=CLOSED pgs=754 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 >> 172.21.15.61:0/1348127358 conn(0x7fdab4063480 msgr2=0x7fdab4144920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 shutdown_connections 2022-07-20T03:14:30.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:29.882+0000 7fdab90f1700 1 -- 172.21.15.61:0/1348127358 wait complete. 2022-07-20T03:14:30.054 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:14:30.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 Processor -- start 2022-07-20T03:14:30.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 -- start start 2022-07-20T03:14:30.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f12940ac520 0x7f1294142d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12940b1df0 0x7f1294143200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f12940b1ae0 con 0x7f12940b12c0 2022-07-20T03:14:30.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12941436f0 con 0x7f12940b1df0 2022-07-20T03:14:30.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f129b10f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f12940ad980 con 0x7f12940ac520 2022-07-20T03:14:30.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12940b1df0 0x7f1294143200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f12940b1df0 0x7f1294143200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:46998/0 (socket says 172.21.15.61:46998) 2022-07-20T03:14:30.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 -- 172.21.15.61:0/3874991497 learned_addr learned my addr 172.21.15.61:0/3874991497 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:30.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f1298eab700 1 --2- 172.21.15.61:0/3874991497 >> v2:172.21.15.117:3300/0 conn(0x7f12940ac520 0x7f1294142d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 -- 172.21.15.61:0/3874991497 >> v2:172.21.15.117:3300/0 conn(0x7f12940ac520 msgr2=0x7f1294142d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:30.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 --2- 172.21.15.61:0/3874991497 >> v2:172.21.15.117:3300/0 conn(0x7f12940ac520 0x7f1294142d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 -- 172.21.15.61:0/3874991497 >> v1:172.21.15.117:6789/0 conn(0x7f12940b12c0 legacy=0x7f129405f840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 -- 172.21.15.61:0/3874991497 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12940b1840 con 0x7f12940b1df0 2022-07-20T03:14:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12996ac700 1 --2- 172.21.15.61:0/3874991497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294143200 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f128c0032f0 tx=0x7f128c00b530).ready entity=mon.0 client_cookie=b1444485accee21c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12937fe700 1 -- 172.21.15.61:0/3874991497 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f128c00ddf0 con 0x7f12940b1df0 2022-07-20T03:14:30.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12937fe700 1 -- 172.21.15.61:0/3874991497 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c010470 con 0x7f12940b1df0 2022-07-20T03:14:30.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.170+0000 7f12937fe700 1 -- 172.21.15.61:0/3874991497 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f128c0107d0 con 0x7f12940b1df0 2022-07-20T03:14:30.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- 172.21.15.61:0/3874991497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 msgr2=0x7f1294143200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 --2- 172.21.15.61:0/3874991497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294143200 secure :-1 s=READY pgs=755 cs=0 l=1 rev1=1 rx=0x7f128c0032f0 tx=0x7f128c00b530).stop 2022-07-20T03:14:30.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- 172.21.15.61:0/3874991497 shutdown_connections 2022-07-20T03:14:30.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 --2- 172.21.15.61:0/3874991497 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294143200 unknown :-1 s=CLOSED pgs=755 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 --2- 172.21.15.61:0/3874991497 >> v2:172.21.15.117:3300/0 conn(0x7f12940ac520 0x7f1294142d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- 172.21.15.61:0/3874991497 >> 172.21.15.61:0/3874991497 conn(0x7f12940b23b0 msgr2=0x7f12940b2790 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- 172.21.15.61:0/3874991497 shutdown_connections 2022-07-20T03:14:30.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- 172.21.15.61:0/3874991497 wait complete. 2022-07-20T03:14:30.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 Processor -- start 2022-07-20T03:14:30.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- start start 2022-07-20T03:14:30.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12940ac520 0x7f129413b990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12940b12c0 0x7f1294139220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294139710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12940af650 con 0x7f12940b1df0 2022-07-20T03:14:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129413a2d0 con 0x7f12940ac520 2022-07-20T03:14:30.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f129b10f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129413b040 con 0x7f12940b12c0 2022-07-20T03:14:30.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f12996ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294139710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f1298eab700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12940ac520 0x7f129413b990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f12996ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294139710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47004/0 (socket says 172.21.15.61:47004) 2022-07-20T03:14:30.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f12996ac700 1 -- 172.21.15.61:0/605677549 learned_addr learned my addr 172.21.15.61:0/605677549 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:30.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.174+0000 7f1293fff700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12940b12c0 0x7f1294139220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f12996ac700 1 -- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12940b12c0 msgr2=0x7f1294139220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f12996ac700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12940b12c0 0x7f1294139220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f12996ac700 1 -- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12940ac520 msgr2=0x7f129413b990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f12996ac700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12940ac520 0x7f129413b990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f12996ac700 1 -- 172.21.15.61:0/605677549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c00b7d0 con 0x7f12940b1df0 2022-07-20T03:14:30.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f12996ac700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294139710 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f128c0032f0 tx=0x7f1294061990).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f128c01d070 con 0x7f12940b1df0 2022-07-20T03:14:30.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c018800 con 0x7f12940b1df0 2022-07-20T03:14:30.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12940aece0 con 0x7f12940b1df0 2022-07-20T03:14:30.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f128c018b60 con 0x7f12940b1df0 2022-07-20T03:14:30.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129409b050 con 0x7f12940b1df0 2022-07-20T03:14:30.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129413b180 con 0x7f12940b1df0 2022-07-20T03:14:30.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f128c018cc0 con 0x7f12940b1df0 2022-07-20T03:14:30.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1291ffb700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1278053a40 0x7f1278055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f128c06d4a0 con 0x7f12940b1df0 2022-07-20T03:14:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1298eab700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1278053a40 0x7f1278055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.178+0000 7f1298eab700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1278053a40 0x7f1278055f00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f128400b000 tx=0x7f1284006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.186+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f128c044480 con 0x7f12940b1df0 2022-07-20T03:14:30.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f129413a950 con 0x7f12940b1df0 2022-07-20T03:14:30.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f1291ffb700 1 -- 172.21.15.61:0/605677549 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v72) v1 ==== 92+0+10 (secure 0 0 0) 0x7f128c047aa0 con 0x7f12940b1df0 2022-07-20T03:14:30.365 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:30.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1278053a40 msgr2=0x7f1278055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1278053a40 0x7f1278055f00 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f128400b000 tx=0x7f1284006df0).stop 2022-07-20T03:14:30.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 msgr2=0x7f1294139710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294139710 secure :-1 s=READY pgs=756 cs=0 l=1 rev1=1 rx=0x7f128c0032f0 tx=0x7f1294061990).stop 2022-07-20T03:14:30.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 shutdown_connections 2022-07-20T03:14:30.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1278053a40 0x7f1278055f00 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f12940b1df0 0x7f1294139710 unknown :-1 s=CLOSED pgs=756 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f12940b12c0 0x7f1294139220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 --2- 172.21.15.61:0/605677549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f12940ac520 0x7f129413b990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 >> 172.21.15.61:0/605677549 conn(0x7f12940b23b0 msgr2=0x7f129413e3f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 shutdown_connections 2022-07-20T03:14:30.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.314+0000 7f129b10f700 1 -- 172.21.15.61:0/605677549 wait complete. 2022-07-20T03:14:30.396 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:14:30.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.526+0000 7fb806ce8700 1 Processor -- start 2022-07-20T03:14:30.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.526+0000 7fb806ce8700 1 -- start start 2022-07-20T03:14:30.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb80009df40 0x7fb800145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb800060ee0 0x7fb80009cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8000607e0 con 0x7fb8000b4d60 2022-07-20T03:14:30.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb800060d20 con 0x7fb800060ee0 2022-07-20T03:14:30.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb800060a80 con 0x7fb80009df40 2022-07-20T03:14:30.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb800060ee0 0x7fb80009cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb800060ee0 0x7fb80009cba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47010/0 (socket says 172.21.15.61:47010) 2022-07-20T03:14:30.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 -- 172.21.15.61:0/110125524 learned_addr learned my addr 172.21.15.61:0/110125524 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:30.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb7fffff700 1 --1- 172.21.15.61:0/110125524 >> v1:172.21.15.61:6789/0 conn(0x7fb8000b4d60 0x7fb800145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43608/0 (socket says 172.21.15.61:43608) 2022-07-20T03:14:30.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb804a84700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb80009df40 0x7fb800145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 -- 172.21.15.61:0/110125524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb80009df40 msgr2=0x7fb800145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:30.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb80009df40 0x7fb800145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 -- 172.21.15.61:0/110125524 >> v1:172.21.15.61:6789/0 conn(0x7fb8000b4d60 legacy=0x7fb800145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 -- 172.21.15.61:0/110125524 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb80009e320 con 0x7fb800060ee0 2022-07-20T03:14:30.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb804a84700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb80009df40 0x7fb800145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:30.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb805285700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb80009cba0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fb7f80032f0 tx=0x7fb7f8009290).ready entity=mon.0 client_cookie=a0080da26150882b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb7ff7fe700 1 -- 172.21.15.61:0/110125524 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7f800b8a0 con 0x7fb800060ee0 2022-07-20T03:14:30.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb7ff7fe700 1 -- 172.21.15.61:0/110125524 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f800ba00 con 0x7fb800060ee0 2022-07-20T03:14:30.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb7ff7fe700 1 -- 172.21.15.61:0/110125524 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7f800bb70 con 0x7fb800060ee0 2022-07-20T03:14:30.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- 172.21.15.61:0/110125524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 msgr2=0x7fb80009cba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb80009cba0 secure :-1 s=READY pgs=758 cs=0 l=1 rev1=1 rx=0x7fb7f80032f0 tx=0x7fb7f8009290).stop 2022-07-20T03:14:30.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- 172.21.15.61:0/110125524 shutdown_connections 2022-07-20T03:14:30.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb80009cba0 unknown :-1 s=CLOSED pgs=758 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 --2- 172.21.15.61:0/110125524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb80009df40 0x7fb800145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- 172.21.15.61:0/110125524 >> 172.21.15.61:0/110125524 conn(0x7fb8000b5f30 msgr2=0x7fb8000b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- 172.21.15.61:0/110125524 shutdown_connections 2022-07-20T03:14:30.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.530+0000 7fb806ce8700 1 -- 172.21.15.61:0/110125524 wait complete. 2022-07-20T03:14:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 Processor -- start 2022-07-20T03:14:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 -- start start 2022-07-20T03:14:30.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb800152210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80009df40 0x7fb80008f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8000b4d60 0x7fb800090600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8000b28b0 con 0x7fb800060ee0 2022-07-20T03:14:30.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb800092d00 con 0x7fb80009df40 2022-07-20T03:14:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb806ce8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb800092f30 con 0x7fb8000b4d60 2022-07-20T03:14:30.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb804a84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb800152210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb804a84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb800152210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47014/0 (socket says 172.21.15.61:47014) 2022-07-20T03:14:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb804a84700 1 -- 172.21.15.61:0/2670329270 learned_addr learned my addr 172.21.15.61:0/2670329270 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb7fffff700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80009df40 0x7fb80008f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.534+0000 7fb805285700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8000b4d60 0x7fb800090600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb804a84700 1 -- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8000b4d60 msgr2=0x7fb800090600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb804a84700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8000b4d60 0x7fb800090600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb804a84700 1 -- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80009df40 msgr2=0x7fb80008f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb804a84700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80009df40 0x7fb80008f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb804a84700 1 -- 172.21.15.61:0/2670329270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8009530 con 0x7fb800060ee0 2022-07-20T03:14:30.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb804a84700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb800152210 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fb7f00068e0 tx=0x7fb7f0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7f0011780 con 0x7fb800060ee0 2022-07-20T03:14:30.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb7f00118e0 con 0x7fb800060ee0 2022-07-20T03:14:30.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb800152700 con 0x7fb800060ee0 2022-07-20T03:14:30.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb7f0011c40 con 0x7fb800060ee0 2022-07-20T03:14:30.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb80008fe40 con 0x7fb800060ee0 2022-07-20T03:14:30.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb800093130 con 0x7fb800060ee0 2022-07-20T03:14:30.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb7f0012420 con 0x7fb800060ee0 2022-07-20T03:14:30.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fdffb700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7e4053920 0x7fb7e4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fb7f006be40 con 0x7fb800060ee0 2022-07-20T03:14:30.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.538+0000 7fb7fffff700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7e4053920 0x7fb7e4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.542+0000 7fb7fffff700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7e4053920 0x7fb7e4055de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fb7e800d000 tx=0x7fb7e8007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.546+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7f0042c70 con 0x7fb800060ee0 2022-07-20T03:14:30.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.670+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8000b2d00 con 0x7fb800060ee0 2022-07-20T03:14:30.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.670+0000 7fb7fdffb700 1 -- 172.21.15.61:0/2670329270 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v72) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb7f0046290 con 0x7fb800060ee0 2022-07-20T03:14:30.680 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:30.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7e4053920 msgr2=0x7fb7e4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7e4053920 0x7fb7e4055de0 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7fb7e800d000 tx=0x7fb7e8007b80).stop 2022-07-20T03:14:30.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 msgr2=0x7fb800152210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb800152210 secure :-1 s=READY pgs=759 cs=0 l=1 rev1=1 rx=0x7fb7f00068e0 tx=0x7fb7f0008040).stop 2022-07-20T03:14:30.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 shutdown_connections 2022-07-20T03:14:30.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb7e4053920 0x7fb7e4055de0 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb80009df40 0x7fb80008f950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8000b4d60 0x7fb800090600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 --2- 172.21.15.61:0/2670329270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb800060ee0 0x7fb800152210 unknown :-1 s=CLOSED pgs=759 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 >> 172.21.15.61:0/2670329270 conn(0x7fb8000b5f30 msgr2=0x7fb800148a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 shutdown_connections 2022-07-20T03:14:30.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.674+0000 7fb806ce8700 1 -- 172.21.15.61:0/2670329270 wait complete. 2022-07-20T03:14:30.704 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:14:30.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f924c94a700 1 Processor -- start 2022-07-20T03:14:30.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f924c94a700 1 -- start start 2022-07-20T03:14:30.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f924c94a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92480af920 0x7f92480b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f924c94a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92480adf70 con 0x7f92480b59d0 2022-07-20T03:14:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f924c94a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f92480b5db0 con 0x7f924805d0c0 2022-07-20T03:14:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f924c94a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9248146300 con 0x7f92480af920 2022-07-20T03:14:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f92477fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f92480b59d0 0x7f92480ae3d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43620/0 (socket says 172.21.15.61:43620) 2022-07-20T03:14:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f92477fe700 1 -- 172.21.15.61:0/1110543908 learned_addr learned my addr 172.21.15.61:0/1110543908 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.818+0000 7f9246ffd700 1 --2- 172.21.15.61:0/1110543908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92480af920 0x7f92480b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f92467fc700 1 -- 172.21.15.61:0/1110543908 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2388922076 0 0) 0x7f92480adf70 con 0x7f92480b59d0 2022-07-20T03:14:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f92467fc700 1 -- 172.21.15.61:0/1110543908 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9228003260 con 0x7f92480b59d0 2022-07-20T03:14:30.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f9246ffd700 1 -- 172.21.15.61:0/1110543908 >> v1:172.21.15.117:6789/0 conn(0x7f924805d0c0 legacy=0x7f9248141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f9246ffd700 1 -- 172.21.15.61:0/1110543908 >> v1:172.21.15.61:6789/0 conn(0x7f92480b59d0 legacy=0x7f92480ae3d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f92467fc700 1 -- 172.21.15.61:0/1110543908 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1955673234 0 0) 0x7f9228003260 con 0x7f92480b59d0 2022-07-20T03:14:30.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f9246ffd700 1 -- 172.21.15.61:0/1110543908 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9248146060 con 0x7f92480af920 2022-07-20T03:14:30.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f9246ffd700 1 --2- 172.21.15.61:0/1110543908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92480af920 0x7f92480b0040 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9238001820 tx=0x7f92300091d0).ready entity=mon.2 client_cookie=a947d0a0ae872ab9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f92467fc700 1 -- 172.21.15.61:0/1110543908 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9238001680 con 0x7f92480af920 2022-07-20T03:14:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f92467fc700 1 -- 172.21.15.61:0/1110543908 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f923000b5b0 con 0x7f92480af920 2022-07-20T03:14:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f92467fc700 1 -- 172.21.15.61:0/1110543908 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f923000b720 con 0x7f92480af920 2022-07-20T03:14:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 -- 172.21.15.61:0/1110543908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92480af920 msgr2=0x7f92480b0040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 --2- 172.21.15.61:0/1110543908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92480af920 0x7f92480b0040 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f9238001820 tx=0x7f92300091d0).stop 2022-07-20T03:14:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 -- 172.21.15.61:0/1110543908 shutdown_connections 2022-07-20T03:14:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 --2- 172.21.15.61:0/1110543908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92480af920 0x7f92480b0040 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 -- 172.21.15.61:0/1110543908 >> 172.21.15.61:0/1110543908 conn(0x7f92480634c0 msgr2=0x7f92480b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 -- 172.21.15.61:0/1110543908 shutdown_connections 2022-07-20T03:14:30.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.822+0000 7f924c94a700 1 -- 172.21.15.61:0/1110543908 wait complete. 2022-07-20T03:14:30.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 Processor -- start 2022-07-20T03:14:30.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 -- start start 2022-07-20T03:14:30.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f924805d0c0 0x7f92481529d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92480af920 0x7f9248144490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 0x7f9248065b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9248152290 con 0x7f92480b59d0 2022-07-20T03:14:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9248152530 con 0x7f92480af920 2022-07-20T03:14:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f924c94a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9248152ec0 con 0x7f924805d0c0 2022-07-20T03:14:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 0x7f9248065b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 0x7f9248065b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47032/0 (socket says 172.21.15.61:47032) 2022-07-20T03:14:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 -- 172.21.15.61:0/2468007132 learned_addr learned my addr 172.21.15.61:0/2468007132 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9246ffd700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92480af920 0x7f9248144490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f92477fe700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f924805d0c0 0x7f92481529d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 -- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f924805d0c0 msgr2=0x7f92481529d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f924805d0c0 0x7f92481529d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 -- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92480af920 msgr2=0x7f9248144490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92480af920 0x7f9248144490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.826+0000 7f9247fff700 1 -- 172.21.15.61:0/2468007132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9230009470 con 0x7f92480b59d0 2022-07-20T03:14:30.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9247fff700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 0x7f9248065b50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f92480ae2b0 tx=0x7f9240007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9240012400 con 0x7f92480b59d0 2022-07-20T03:14:30.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9240012560 con 0x7f92480b59d0 2022-07-20T03:14:30.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9248144980 con 0x7f92480b59d0 2022-07-20T03:14:30.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f92400128c0 con 0x7f92480b59d0 2022-07-20T03:14:30.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9248145230 con 0x7f92480b59d0 2022-07-20T03:14:30.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9248112fd0 con 0x7f92480b59d0 2022-07-20T03:14:30.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9240019030 con 0x7f92480b59d0 2022-07-20T03:14:30.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9244ff9700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f922c053920 0x7f922c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:30.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f924006b1b0 con 0x7f92480b59d0 2022-07-20T03:14:30.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f92477fe700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f922c053920 0x7f922c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:30.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.830+0000 7f92477fe700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f922c053920 0x7f922c055de0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f923800b000 tx=0x7f9238000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:30.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.838+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f92400420e0 con 0x7f92480b59d0 2022-07-20T03:14:30.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.966+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f92481526f0 con 0x7f92480b59d0 2022-07-20T03:14:30.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.966+0000 7f9244ff9700 1 -- 172.21.15.61:0/2468007132 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v72) v1 ==== 100+0+10 (secure 0 0 0) 0x7f9240045700 con 0x7f92480b59d0 2022-07-20T03:14:30.978 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:30.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f922c053920 msgr2=0x7f922c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f922c053920 0x7f922c055de0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f923800b000 tx=0x7f9238000bc0).stop 2022-07-20T03:14:30.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 msgr2=0x7f9248065b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:30.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 0x7f9248065b50 secure :-1 s=READY pgs=761 cs=0 l=1 rev1=1 rx=0x7f92480ae2b0 tx=0x7f9240007b90).stop 2022-07-20T03:14:30.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 shutdown_connections 2022-07-20T03:14:30.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f922c053920 0x7f922c055de0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92480af920 0x7f9248144490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92480b59d0 0x7f9248065b50 unknown :-1 s=CLOSED pgs=761 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 --2- 172.21.15.61:0/2468007132 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f924805d0c0 0x7f92481529d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:30.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 >> 172.21.15.61:0/2468007132 conn(0x7f92480634c0 msgr2=0x7f92481429b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:30.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 shutdown_connections 2022-07-20T03:14:30.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:30.970+0000 7f924c94a700 1 -- 172.21.15.61:0/2468007132 wait complete. 2022-07-20T03:14:30.989 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:14:30.990 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:31.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 Processor -- start 2022-07-20T03:14:31.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 -- start start 2022-07-20T03:14:31.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf413fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:31.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1bf40aecf0 0x7f1bf4140020 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:31.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1bf40b5d70 con 0x7f1bf40aff80 2022-07-20T03:14:31.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1bf40651c0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40b0690 con 0x7f1bf40b5990 2022-07-20T03:14:31.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1bf40aecf0 0x7f1bf4140020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:31.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1bf40aecf0 0x7f1bf4140020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47038/0 (socket says 172.21.15.61:47038) 2022-07-20T03:14:31.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 -- 172.21.15.61:0/3087257933 learned_addr learned my addr 172.21.15.61:0/3087257933 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:31.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf2ffd700 1 --2- 172.21.15.61:0/3087257933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf413fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:31.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 -- 172.21.15.61:0/3087257933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 msgr2=0x7f1bf413fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:31.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 --2- 172.21.15.61:0/3087257933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf413fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 -- 172.21.15.61:0/3087257933 >> v1:172.21.15.117:6789/0 conn(0x7f1bf40aff80 legacy=0x7f1bf405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:31.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 -- 172.21.15.61:0/3087257933 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf40b0930 con 0x7f1bf40aecf0 2022-07-20T03:14:31.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bebfff700 1 --2- 172.21.15.61:0/3087257933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4140020 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7f1bdc002f70 tx=0x7f1bdc006e50).ready entity=mon.0 client_cookie=bc184031c7def295 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:31.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.094+0000 7f1bf27fc700 1 -- 172.21.15.61:0/3087257933 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bdc00d5c0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf27fc700 1 -- 172.21.15.61:0/3087257933 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1bdc000e90 con 0x7f1bf40aecf0 2022-07-20T03:14:31.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf27fc700 1 -- 172.21.15.61:0/3087257933 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1bdc00d950 con 0x7f1bf40aecf0 2022-07-20T03:14:31.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- 172.21.15.61:0/3087257933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 msgr2=0x7f1bf4140020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:31.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/3087257933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4140020 secure :-1 s=READY pgs=762 cs=0 l=1 rev1=1 rx=0x7f1bdc002f70 tx=0x7f1bdc006e50).stop 2022-07-20T03:14:31.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- 172.21.15.61:0/3087257933 shutdown_connections 2022-07-20T03:14:31.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/3087257933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4140020 unknown :-1 s=CLOSED pgs=762 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/3087257933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf413fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- 172.21.15.61:0/3087257933 >> 172.21.15.61:0/3087257933 conn(0x7f1bf40634e0 msgr2=0x7f1bf40b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:31.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- 172.21.15.61:0/3087257933 shutdown_connections 2022-07-20T03:14:31.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- 172.21.15.61:0/3087257933 wait complete. 2022-07-20T03:14:31.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 Processor -- start 2022-07-20T03:14:31.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- start start 2022-07-20T03:14:31.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4143510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:31.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bf40aff80 0x7f1bf4114c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:31.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf4144390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:31.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf40ae390 con 0x7f1bf40aecf0 2022-07-20T03:14:31.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf413c7b0 con 0x7f1bf40aff80 2022-07-20T03:14:31.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1bf41144f0 con 0x7f1bf40b5990 2022-07-20T03:14:31.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4143510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4143510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47042/0 (socket says 172.21.15.61:47042) 2022-07-20T03:14:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf2ffd700 1 -- 172.21.15.61:0/1877759806 learned_addr learned my addr 172.21.15.61:0/1877759806 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:31.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bebfff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bf40aff80 0x7f1bf4114c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:31.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.098+0000 7f1bf37fe700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf4144390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:31.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf2ffd700 1 -- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 msgr2=0x7f1bf4144390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:31.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf2ffd700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf4144390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf2ffd700 1 -- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bf40aff80 msgr2=0x7f1bf4114c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:31.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf2ffd700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bf40aff80 0x7f1bf4114c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf2ffd700 1 -- 172.21.15.61:0/1877759806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bdc00b1e0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf2ffd700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4143510 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f1be400c000 tx=0x7f1be4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:31.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be4011750 con 0x7f1bf40aecf0 2022-07-20T03:14:31.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1be40118b0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1bf4114790 con 0x7f1bf40aecf0 2022-07-20T03:14:31.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1be4011c10 con 0x7f1bf40aecf0 2022-07-20T03:14:31.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1bf4115120 con 0x7f1bf40aecf0 2022-07-20T03:14:31.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1bf40574e0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1be4011d70 con 0x7f1bf40aecf0 2022-07-20T03:14:31.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf0ff9700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bd8053a40 0x7f1bd8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:31.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.102+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f1be406c2b0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.106+0000 7f1bebfff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bd8053a40 0x7f1bd8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:31.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.106+0000 7f1bebfff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bd8053a40 0x7f1bd8055f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f1bdc01a000 tx=0x7f1bdc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:31.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.110+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1be4043180 con 0x7f1bf40aecf0 2022-07-20T03:14:31.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.270+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1bf4114920 con 0x7f1bf40aecf0 2022-07-20T03:14:31.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.270+0000 7f1bf0ff9700 1 -- 172.21.15.61:0/1877759806 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f1be40467a0 con 0x7f1bf40aecf0 2022-07-20T03:14:31.354 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:31.355 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:14:31.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bd8053a40 msgr2=0x7f1bd8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:31.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bd8053a40 0x7f1bd8055f00 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f1bdc01a000 tx=0x7f1bdc014000).stop 2022-07-20T03:14:31.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 msgr2=0x7f1bf4143510 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:31.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4143510 secure :-1 s=READY pgs=763 cs=0 l=1 rev1=1 rx=0x7f1be400c000 tx=0x7f1be4006df0).stop 2022-07-20T03:14:31.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 shutdown_connections 2022-07-20T03:14:31.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bd8053a40 0x7f1bd8055f00 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1bf40aff80 0x7f1bf4114c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1bf40b5990 0x7f1bf4144390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 --2- 172.21.15.61:0/1877759806 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1bf40aecf0 0x7f1bf4143510 unknown :-1 s=CLOSED pgs=763 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:31.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 >> 172.21.15.61:0/1877759806 conn(0x7f1bf40634e0 msgr2=0x7f1bf405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:31.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 shutdown_connections 2022-07-20T03:14:31.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:31.274+0000 7f1bf3fff700 1 -- 172.21.15.61:0/1877759806 wait complete. 2022-07-20T03:14:31.393 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:14:31.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 40, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:13:37.441965+0000', 'modified': '2022-07-20T03:13:38.444883+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6285}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6285': {'gid': 6285, 'name': 'a', 'rank': 0, 'incarnation': 39, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/460461689', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 460461689}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 460461689}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [14], 'metadata_pool': 13, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:14:31.614 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:14:31.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:14:31.742 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6285,"want_state":"up:active","state":"up:active","rank_uptime":54.285947223999997,"mdsmap_epoch":40,"osdmap_epoch":72,"osdmap_epoch_barrier":72,"uptime":59.371560743000003} 2022-07-20T03:14:31.744 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6285, "mdsmap_epoch": 40, "osdmap_epoch": 72, "osdmap_epoch_barrier": 72, "rank_uptime": 54.285947224, "state": "up:active", "uptime": 59.371560743, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:14:31.744 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:14:31.745 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:14:31.745 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:14:31.745 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:31.746 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:31.769 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:14:31.831 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:14:31.832 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:14:31.847 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:14:31.848 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:14:31.850 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:31.851 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:14:31.924 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:14:31.938 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:31.954 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:14:31.994 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:14:31.995 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:14:32.020 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:14:32.011+0000 7f22c0fa3080 -1 init, newargv = 0x55d9b83e22c0 newargc=9ceph-fuse[ 2022-07-20T03:14:32.021 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:22994]: starting ceph client 2022-07-20T03:14:32.040 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22994]: starting fuse 2022-07-20T03:14:32.062 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:14:32.063 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:14:32.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.065 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:14:32.113 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:14:32.114 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:14:32.115 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.116 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:14:32.116 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:14:32.117 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:14:32.118 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:14:32.119 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:14:32.119 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:32.120 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:14:32.121 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:14:32.122 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:14:32.122 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:32.123 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:14:32.124 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:14:32.124 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:14:32.125 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:14:32.126 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:32.127 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:14:32.127 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:14:32.128 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:14:32.129 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:14:32.130 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:14:32.130 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:32.131 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:14:32.132 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:14:32.178 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.22994.asok 2022-07-20T03:14:32.197 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.22994.asok 2022-07-20T03:14:32.198 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.198 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.22994.asok status 2022-07-20T03:14:32.311 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:14:32.311 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:14:32.312 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:14:32.312 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:14:32.312 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:14:32.312 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:14:32.313 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:14:32.313 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "22994", 2022-07-20T03:14:32.313 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:14:32.322 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:32.323 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:14:32.324 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:14:32.325 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6541, 2022-07-20T03:14:32.325 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:14:32.326 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:14:32.327 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:14:32.356 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6541 2022-07-20T03:14:32.356 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:32.357 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:14:32.357 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:14:32.357 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:14:32.357 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2442581068 2022-07-20T03:14:32.358 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:14:32.358 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:32.358 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:14:32.358 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:14:32.359 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:14:32.359 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2442581068 2022-07-20T03:14:32.359 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:32.359 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6541 172.21.15.124:0/2442581068", 2022-07-20T03:14:32.359 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/2442581068", 2022-07-20T03:14:32.360 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:14:32.360 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 40, 2022-07-20T03:14:32.360 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 72, 2022-07-20T03:14:32.379 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:14:32.380 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:14:32.380 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:14:32.380 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:14:32.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.381 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:32.382 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:32.393 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:14:32.394 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:14:32.395 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.396 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:32.426 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:14:32.427 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:32.454 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:32.490 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:32.529 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:32.748 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:32.771 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:14:32.772 INFO:tasks.cephfs.mount:Terminating background process 2022-07-20T03:14:32.773 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.774 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:32.775 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:32.786 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:14:32.788 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:14:32.789 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:14:32.790 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:14:32.792 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:32.851 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[22994]: fuse finished with error 0 and tester_r 0 2022-07-20T03:14:32.853 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:32.854 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:32.855 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:32.867 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:14:32.868 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:14:32.986 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:32.987 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:33.006 INFO:tasks.cephfs.mount:Terminating background process 2022-07-20T03:14:33.007 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:33.008 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:33.009 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:33.030 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:14:33.031 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:14:33.032 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:14:33.040 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:14:33.041 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:33.088 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22431]: fuse finished with error 0 and tester_r 0 2022-07-20T03:14:33.104 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:33.106 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:33.107 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:33.122 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:14:33.141 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:14:33.267 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:33.268 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:33.334 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:33.335 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:33.336 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:14:33.364 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:33.399 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:14:33.411 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:14:33.412 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:33.413 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:14:33.436 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:33.438 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:14:33.439 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:14:33.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 Processor -- start 2022-07-20T03:14:33.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 -- start start 2022-07-20T03:14:33.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80b5650 0x7fddd80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fddd80adff0 0x7fddd8147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fddd80af8e0 0x7fddd805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd8147010 con 0x7fddd80adff0 2022-07-20T03:14:33.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fddd81472b0 con 0x7fddd80af8e0 2022-07-20T03:14:33.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddddd84700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd8146d40 con 0x7fddd80b5650 2022-07-20T03:14:33.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fddd80adff0 0x7fddd8147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fddd80adff0 0x7fddd8147750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47050/0 (socket says 172.21.15.61:47050) 2022-07-20T03:14:33.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 -- 172.21.15.61:0/2601783032 learned_addr learned my addr 172.21.15.61:0/2601783032 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:33.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fdddd583700 1 --2- 172.21.15.61:0/2601783032 >> v2:172.21.15.117:3300/0 conn(0x7fddd80af8e0 0x7fddd805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fdddcd82700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80b5650 0x7fddd80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 -- 172.21.15.61:0/2601783032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80b5650 msgr2=0x7fddd80b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:33.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80b5650 0x7fddd80b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 -- 172.21.15.61:0/2601783032 >> v2:172.21.15.117:3300/0 conn(0x7fddd80af8e0 msgr2=0x7fddd805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:33.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 --2- 172.21.15.61:0/2601783032 >> v2:172.21.15.117:3300/0 conn(0x7fddd80af8e0 0x7fddd805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 -- 172.21.15.61:0/2601783032 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddd8147c40 con 0x7fddd80adff0 2022-07-20T03:14:33.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fdddcd82700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80b5650 0x7fddd80b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:33.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd7fff700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8147750 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7fddc8002f70 tx=0x7fddc8006e50).ready entity=mon.0 client_cookie=a766facf161f25bd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:33.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd77fe700 1 -- 172.21.15.61:0/2601783032 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddc800d570 con 0x7fddd80adff0 2022-07-20T03:14:33.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd77fe700 1 -- 172.21.15.61:0/2601783032 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddc8000e90 con 0x7fddd80adff0 2022-07-20T03:14:33.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.562+0000 7fddd77fe700 1 -- 172.21.15.61:0/2601783032 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddc800d900 con 0x7fddd80adff0 2022-07-20T03:14:33.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- 172.21.15.61:0/2601783032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 msgr2=0x7fddd8147750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:33.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8147750 secure :-1 s=READY pgs=764 cs=0 l=1 rev1=1 rx=0x7fddc8002f70 tx=0x7fddc8006e50).stop 2022-07-20T03:14:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- 172.21.15.61:0/2601783032 shutdown_connections 2022-07-20T03:14:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- 172.21.15.61:0/2601783032 >> v2:172.21.15.117:3300/0 conn(0x7fddd80af8e0 0x7fddd805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8147750 unknown :-1 s=CLOSED pgs=764 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- 172.21.15.61:0/2601783032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80b5650 0x7fddd80b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- 172.21.15.61:0/2601783032 >> 172.21.15.61:0/2601783032 conn(0x7fddd80634e0 msgr2=0x7fddd80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:33.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- 172.21.15.61:0/2601783032 shutdown_connections 2022-07-20T03:14:33.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- 172.21.15.61:0/2601783032 wait complete. 2022-07-20T03:14:33.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 Processor -- start 2022-07-20T03:14:33.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- start start 2022-07-20T03:14:33.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8111070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80af8e0 0x7fddd814a650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddd80b5650 0x7fddd8156830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd810fc60 con 0x7fddd80adff0 2022-07-20T03:14:33.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fddd81563d0 con 0x7fddd80b5650 2022-07-20T03:14:33.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fddddd84700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fddd814b010 con 0x7fddd80af8e0 2022-07-20T03:14:33.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.566+0000 7fdddcd82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8111070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8111070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47056/0 (socket says 172.21.15.61:47056) 2022-07-20T03:14:33.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 -- 172.21.15.61:0/233359714 learned_addr learned my addr 172.21.15.61:0/233359714 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:33.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddd583700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddd80b5650 0x7fddd8156830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddd7fff700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80af8e0 0x7fddd814a650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 -- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80af8e0 msgr2=0x7fddd814a650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:33.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80af8e0 0x7fddd814a650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 -- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddd80b5650 msgr2=0x7fddd8156830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:33.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddd80b5650 0x7fddd8156830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:33.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 -- 172.21.15.61:0/233359714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fddc800b1e0 con 0x7fddd80adff0 2022-07-20T03:14:33.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fdddcd82700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8111070 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fddcc007000 tx=0x7fddcc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:33.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddcc00ce80 con 0x7fddd80adff0 2022-07-20T03:14:33.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fddcc010580 con 0x7fddd80adff0 2022-07-20T03:14:33.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fddcc0108e0 con 0x7fddd80adff0 2022-07-20T03:14:33.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fddd810f860 con 0x7fddd80adff0 2022-07-20T03:14:33.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fddd8156ec0 con 0x7fddd80adff0 2022-07-20T03:14:33.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.570+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fddd810f9f0 con 0x7fddd80adff0 2022-07-20T03:14:33.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.574+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fddcc010a40 con 0x7fddd80adff0 2022-07-20T03:14:33.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.574+0000 7fddd5ffb700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fddb8053a60 0x7fddb8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:33.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.574+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7fddcc06bbe0 con 0x7fddd80adff0 2022-07-20T03:14:33.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.574+0000 7fddd7fff700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fddb8053a60 0x7fddb8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:33.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.574+0000 7fddd7fff700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fddb8053a60 0x7fddb8055f20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fddc801c000 tx=0x7fddc8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:33.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.578+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fddcc042b10 con 0x7fddd80adff0 2022-07-20T03:14:33.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.706+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fddd80574e0 con 0x7fddd80adff0 2022-07-20T03:14:33.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddd5ffb700 1 -- 172.21.15.61:0/233359714 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v72) v1 ==== 81+0+33773 (secure 0 0 0) 0x7fddcc046130 con 0x7fddd80adff0 2022-07-20T03:14:33.713 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:33.714 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:14:33.714 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 72, 2022-07-20T03:14:33.715 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:14:33.715 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:14:33.716 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:14:23.218756+0000", 2022-07-20T03:14:33.716 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:14:33.717 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:14:33.717 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:14:33.718 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:14:33.719 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:14:33.719 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:14:33.720 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:14:33.720 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:14:33.721 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:14:33.722 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:33.722 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:14:33.723 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:14:33.723 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:14:33.724 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:14:33.724 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:14:33.748 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 14, 2022-07-20T03:14:33.748 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:14:33.748 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:14:33.749 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:14:33.749 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:14:33.750 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:14:33.750 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.751 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:14:33.751 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:14:33.763 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:14:33.763 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:33.764 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:33.764 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:33.765 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:33.765 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:33.766 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:33.766 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:33.767 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:33.767 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:14:33.768 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:14:33.768 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:14:33.768 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:14:33.769 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:14:33.769 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:33.770 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:33.770 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:33.771 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:33.772 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:33.772 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:33.773 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:33.773 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.773 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:14:33.774 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:33.774 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:33.775 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:33.775 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:33.775 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:33.776 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:33.776 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:33.777 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:33.777 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:33.777 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:33.779 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:33.779 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:33.779 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:33.780 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:33.780 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:33.781 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:33.781 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:33.781 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:33.782 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:33.783 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:33.783 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:33.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:33.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:33.785 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:33.785 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:33.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:33.802 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.802 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:33.803 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:33.803 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:33.804 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:33.804 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:33.805 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:33.805 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:33.806 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:33.806 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:33.807 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:33.807 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:33.807 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:14:33.808 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:14:33.808 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.809 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:33.809 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:14:33.809 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.810 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.810 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:14:33.811 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:14:33.812 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:14:33.812 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:33.813 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:33.813 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:33.814 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:33.815 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:33.815 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:33.816 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:33.816 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:33.816 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:33.817 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:33.817 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:33.818 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:33.819 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:33.819 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:33.820 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:33.820 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:33.826 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:33.826 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:33.827 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:33.827 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:33.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.828 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:14:33.828 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:33.837 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:33.837 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:33.838 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:33.838 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:33.839 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:33.839 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:33.840 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:33.841 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:33.841 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:33.843 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:33.843 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:33.844 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:33.844 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:33.844 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:33.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:33.845 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:33.846 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:33.849 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:33.850 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:33.850 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:33.851 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:33.851 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:33.852 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:33.852 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:33.853 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:33.853 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.853 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:33.854 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:33.854 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:33.855 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:33.855 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:33.856 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:33.856 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:33.857 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:33.857 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:33.858 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:33.858 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:33.859 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:14:33.860 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:33.860 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:14:33.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.861 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.861 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.862 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 13, 2022-07-20T03:14:33.862 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:14:33.863 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:13:36.009693+0000", 2022-07-20T03:14:33.863 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:33.864 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:33.864 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:33.865 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:33.865 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:33.865 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:33.866 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:33.866 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:33.867 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:33.867 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:33.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:33.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:33.869 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:33.869 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:33.869 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:33.870 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:33.871 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:33.871 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:33.872 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:33.872 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:33.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.873 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "70", 2022-07-20T03:14:33.874 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:33.874 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:33.875 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:33.875 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:33.875 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:33.876 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:33.876 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:33.877 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:33.877 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:33.878 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:33.878 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:33.878 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:33.879 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:33.879 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:33.880 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:33.880 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:33.881 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:33.881 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:33.881 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:33.883 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:33.884 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:33.890 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:33.891 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:33.891 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:33.892 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:33.898 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:33.899 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.899 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:33.900 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:33.900 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:33.901 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:33.901 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:33.902 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:33.902 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:33.903 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:33.904 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:33.904 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:33.904 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:33.905 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:14:33.905 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:14:33.906 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:14:33.906 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:14:33.907 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.908 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:33.908 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:14:33.909 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:14:33.909 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.910 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.910 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.911 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.911 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 14, 2022-07-20T03:14:33.911 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:14:33.912 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:13:36.517295+0000", 2022-07-20T03:14:33.912 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:33.913 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:33.913 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:33.913 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:33.914 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:33.914 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:33.915 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:33.915 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:33.915 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:33.916 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:33.916 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:33.917 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:33.917 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:33.917 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:33.918 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:33.924 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:33.924 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:33.925 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:33.925 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:33.925 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:33.926 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.926 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "71", 2022-07-20T03:14:33.927 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:33.927 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:33.928 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:33.928 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:33.928 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:33.929 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:33.929 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:33.930 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:33.930 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:33.931 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:33.931 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:33.932 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:33.932 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:33.932 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:33.935 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:33.935 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:33.936 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:33.936 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:33.937 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:33.937 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:33.938 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:33.938 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:33.939 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:33.939 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:33.939 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:33.940 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:33.940 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.941 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:33.941 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:33.942 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:33.942 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:33.942 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:33.943 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:33.948 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:33.948 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:33.949 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:33.949 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:33.949 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:33.950 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:14:33.950 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:33.951 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:14:33.951 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:14:33.952 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.952 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.953 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.953 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:33.954 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:14:33.955 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.955 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:14:33.956 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:14:33.956 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:33.957 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:33.957 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:33.963 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:33.963 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:33.964 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:33.964 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:33.965 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:33.967 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:33.967 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:33.968 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:33.968 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:33.969 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.969 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:33.969 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:14:33.970 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:33.970 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.971 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:33.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:14:33.972 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:33.973 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.973 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:33.973 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.974 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:33.974 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:33.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.975 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:33.995 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:14:33.996 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:33.996 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:33.997 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:33.997 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:33.998 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:14:33.998 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:33.999 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:33.999 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.000 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.000 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.001 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.001 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.001 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:14:34.003 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:34.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.005 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:14:34.006 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:34.006 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.007 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.008 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.008 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.009 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:14:34.010 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:34.010 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.012 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.012 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.012 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:14:34.013 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:34.013 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.014 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.014 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.015 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:14:34.017 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:14:34.017 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:14:34.018 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:14:34.018 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.019 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.019 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.019 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.020 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.020 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.021 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:14:34.021 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:14:34.021 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.032 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.032 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.033 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.033 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.034 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.034 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.035 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:14:34.035 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.036 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.037 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.039 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:14:34.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:14:34.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.043 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.044 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.044 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.046 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:14:34.047 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:14:34.050 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.051 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.052 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.052 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.052 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.053 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.053 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.054 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.054 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:14:34.055 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.055 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.055 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.056 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:14:34.056 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.057 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.057 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.058 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.060 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:14:34.061 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.062 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:14:34.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:34.063 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.063 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.064 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:14:34.065 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:14:34.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:14:34.066 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:14:34.066 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.067 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.067 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.068 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.069 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:14:34.070 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:14:34.071 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.071 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.072 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.072 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.073 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.073 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.073 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.074 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:34.074 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.075 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.075 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.076 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.077 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:14:34.078 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.086 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:14:34.088 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.089 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.090 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.091 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.092 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.093 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:14:34.093 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.094 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.094 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.094 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.095 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:14:34.095 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.096 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.096 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.097 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.097 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.098 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.098 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.099 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:14:34.100 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.100 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.101 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.102 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:14:34.103 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.103 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.104 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.104 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.105 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.106 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:14:34.107 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.107 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.107 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:14:34.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:34.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.112 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.113 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:14:34.114 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:14:34.114 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:14:34.115 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:14:34.116 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.117 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.117 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.118 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.118 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.120 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:14:34.121 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:14:34.121 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.134 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.135 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.135 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.136 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.136 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.137 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.171 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:34.172 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.172 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.173 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:14:34.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.176 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.176 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:14:34.177 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.177 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.177 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.177 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.178 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.179 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:14:34.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.180 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.180 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:14:34.191 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.191 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.191 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.192 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.192 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.193 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.205 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:14:34.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.206 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.206 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:14:34.207 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.207 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.207 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.208 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:14:34.209 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:14:34.211 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:34.211 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.211 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.212 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:14:34.212 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:14:34.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:14:34.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:14:34.213 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.213 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.213 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.214 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.215 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:14:34.216 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:14:34.216 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.217 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.217 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.217 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.217 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.218 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.218 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.218 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-20T03:14:34.218 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.218 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.219 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.219 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.219 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:14:34.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:14:34.221 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.221 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.221 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.227 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.227 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.228 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:14:34.228 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:14:34.230 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.230 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.230 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.231 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.231 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:14:34.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.233 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.233 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:14:34.234 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.234 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.234 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.235 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:14:34.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.236 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:14:34.237 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:34.237 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.237 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.238 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.238 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:14:34.238 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:14:34.238 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:14:34.239 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:14:34.239 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.239 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.239 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.240 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.240 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:14:34.241 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:14:34.241 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.241 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.241 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.249 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.250 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.250 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.250 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:34.251 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.251 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.251 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.252 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:14:34.253 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.253 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:14:34.254 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.263 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.264 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.264 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.266 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.266 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:14:34.267 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.267 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.268 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:14:34.268 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.268 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.268 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.286 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.286 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.287 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:14:34.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.288 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:14:34.288 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.289 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.289 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.289 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.289 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.290 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:14:34.305 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.306 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.306 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:14:34.307 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:34.307 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.308 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.309 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:14:34.309 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:14:34.345 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:14:34.346 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:14:34.346 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.347 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.347 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.348 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.349 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:14:34.350 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:14:34.363 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.371 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.372 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:34.373 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:14:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.375 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.376 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.376 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:14:34.376 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.376 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.377 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.377 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.377 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.378 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.378 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:14:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.379 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:14:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.380 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:14:34.382 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.383 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:14:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.384 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.385 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.385 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.385 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.385 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.386 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:14:34.386 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.386 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:14:34.387 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.388 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:14:34.389 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:14:34.389 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:14:34.393 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:14:34.393 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.393 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.393 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.394 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.394 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:14:34.395 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:14:34.395 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:34.395 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:34.395 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:34.396 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-20T03:14:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:34.397 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.398 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:14:34.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.399 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:14:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.403 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.403 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:34.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.404 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.405 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:14:34.406 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.407 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.407 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.407 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.409 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:14:34.409 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.409 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.410 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.411 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.412 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:34.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.413 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.415 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:14:34.415 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:14:34.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.429 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.430 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.431 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:34.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:34.431 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:34.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:14:34.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.433 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:34.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:14:34.434 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:34.434 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.434 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.435 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.435 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:14:34.435 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:14:34.435 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:14:34.436 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:14:34.436 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:34.436 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:34.436 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:34.436 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:34.437 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.437 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:34.437 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:14:34.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.438 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:14:34.438 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.438 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.438 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.439 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.439 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.439 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:14:34.439 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.440 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.440 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.440 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:14:34.440 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.440 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.441 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.441 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.441 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.441 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:14:34.442 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.442 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.443 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.443 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:14:34.443 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.443 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.443 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.444 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.451 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.451 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:14:34.452 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.452 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:14:34.453 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.453 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.453 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.453 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.454 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.454 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:14:34.454 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.454 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.455 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:14:34.455 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.455 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.456 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.456 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.458 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.495 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:14:34.495 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.495 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.496 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.496 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:14:34.496 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.497 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.498 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.498 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.498 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.498 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:14:34.499 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.499 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.500 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.500 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:14:34.500 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.501 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.501 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.501 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.501 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.502 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:14:34.502 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.502 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.502 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:34.503 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:14:34.503 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:34.503 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:34.503 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:34.504 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:34.504 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:34.504 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:14:34.504 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:34.505 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.505 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:34.505 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:14:34.506 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:14:34.506 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:14:34.506 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:14:34.506 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:14:34.507 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.124:0/2780278373": "2022-07-20T04:14:22.364604+0000" 2022-07-20T03:14:34.507 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.507 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:14:34.508 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:14:34.508 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:14:34.509 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:14:34.509 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:14:34.509 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:14:34.510 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:14:34.510 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:14:34.510 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:34.511 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:34.511 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:14:34.511 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:14:34.511 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:14:34.512 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:14:34.512 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:14:34.512 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:14:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fddb8053a60 msgr2=0x7fddb8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fddb8053a60 0x7fddb8055f20 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fddc801c000 tx=0x7fddc8015000).stop 2022-07-20T03:14:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 msgr2=0x7fddd8111070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8111070 secure :-1 s=READY pgs=765 cs=0 l=1 rev1=1 rx=0x7fddcc007000 tx=0x7fddcc008040).stop 2022-07-20T03:14:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 shutdown_connections 2022-07-20T03:14:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fddb8053a60 0x7fddb8055f20 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fddd80b5650 0x7fddd8156830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fddd80af8e0 0x7fddd814a650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 --2- 172.21.15.61:0/233359714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fddd80adff0 0x7fddd8111070 unknown :-1 s=CLOSED pgs=765 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 >> 172.21.15.61:0/233359714 conn(0x7fddd80634e0 msgr2=0x7fddd80b4f10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 shutdown_connections 2022-07-20T03:14:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:33.710+0000 7fddddd84700 1 -- 172.21.15.61:0/233359714 wait complete. 2022-07-20T03:14:34.519 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:34.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 Processor -- start 2022-07-20T03:14:34.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- start start 2022-07-20T03:14:34.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340ad770 0x7f0d340add50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340adb50 con 0x7f0d340acea0 2022-07-20T03:14:34.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0d340ae380 con 0x7f0d341405a0 2022-07-20T03:14:34.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d340b2920 con 0x7f0d340ad770 2022-07-20T03:14:34.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d2b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0d340acea0 0x7f0d3413fe90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43664/0 (socket says 172.21.15.61:43664) 2022-07-20T03:14:34.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d2b7fe700 1 -- 172.21.15.61:0/3822302405 learned_addr learned my addr 172.21.15.61:0/3822302405 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:34.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d33fff700 1 --2- 172.21.15.61:0/3822302405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340ad770 0x7f0d340add50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d337fe700 1 -- 172.21.15.61:0/3822302405 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2819151857 0 0) 0x7f0d340adb50 con 0x7f0d340acea0 2022-07-20T03:14:34.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d337fe700 1 -- 172.21.15.61:0/3822302405 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0d14003260 con 0x7f0d340acea0 2022-07-20T03:14:34.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d33fff700 1 -- 172.21.15.61:0/3822302405 >> v1:172.21.15.117:6789/0 conn(0x7f0d341405a0 legacy=0x7f0d3405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d33fff700 1 -- 172.21.15.61:0/3822302405 >> v1:172.21.15.61:6789/0 conn(0x7f0d340acea0 legacy=0x7f0d3413fe90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d33fff700 1 -- 172.21.15.61:0/3822302405 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d34141590 con 0x7f0d340ad770 2022-07-20T03:14:34.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d33fff700 1 --2- 172.21.15.61:0/3822302405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340ad770 0x7f0d340add50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0d24002f70 tx=0x7f0d24006df0).ready entity=mon.2 client_cookie=4f61ad7f6e05bd61 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:34.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d337fe700 1 -- 172.21.15.61:0/3822302405 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2400d570 con 0x7f0d340ad770 2022-07-20T03:14:34.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d337fe700 1 -- 172.21.15.61:0/3822302405 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d24000e90 con 0x7f0d340ad770 2022-07-20T03:14:34.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d337fe700 1 -- 172.21.15.61:0/3822302405 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2400d8c0 con 0x7f0d340ad770 2022-07-20T03:14:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- 172.21.15.61:0/3822302405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340ad770 msgr2=0x7f0d340add50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 --2- 172.21.15.61:0/3822302405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340ad770 0x7f0d340add50 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f0d24002f70 tx=0x7f0d24006df0).stop 2022-07-20T03:14:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- 172.21.15.61:0/3822302405 shutdown_connections 2022-07-20T03:14:34.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 --2- 172.21.15.61:0/3822302405 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340ad770 0x7f0d340add50 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- 172.21.15.61:0/3822302405 >> 172.21.15.61:0/3822302405 conn(0x7f0d340b72c0 msgr2=0x7f0d340b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:34.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- 172.21.15.61:0/3822302405 shutdown_connections 2022-07-20T03:14:34.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.634+0000 7f0d3a320700 1 -- 172.21.15.61:0/3822302405 wait complete. 2022-07-20T03:14:34.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 Processor -- start 2022-07-20T03:14:34.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 -- start start 2022-07-20T03:14:34.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340acea0 0x7f0d34067020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d340ad770 0x7f0d34141fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 0x7f0d340afbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d34143290 con 0x7f0d341405a0 2022-07-20T03:14:34.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d341114e0 con 0x7f0d340ad770 2022-07-20T03:14:34.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d3a320700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d340af480 con 0x7f0d340acea0 2022-07-20T03:14:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 0x7f0d340afbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d2b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d340ad770 0x7f0d34141fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d33fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340acea0 0x7f0d34067020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 0x7f0d340afbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47074/0 (socket says 172.21.15.61:47074) 2022-07-20T03:14:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 -- 172.21.15.61:0/4129172016 learned_addr learned my addr 172.21.15.61:0/4129172016 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d33fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340acea0 0x7f0d34067020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33742/0 (socket says 172.21.15.61:33742) 2022-07-20T03:14:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 -- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340acea0 msgr2=0x7f0d34067020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340acea0 0x7f0d34067020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 -- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d340ad770 msgr2=0x7f0d34141fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d340ad770 0x7f0d34141fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 -- 172.21.15.61:0/4129172016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d2400b180 con 0x7f0d341405a0 2022-07-20T03:14:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d388bd700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 0x7f0d340afbc0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f0d34064890 tx=0x7f0d2c007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.638+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2c00f470 con 0x7f0d341405a0 2022-07-20T03:14:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d2c00f5d0 con 0x7f0d341405a0 2022-07-20T03:14:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d340af720 con 0x7f0d341405a0 2022-07-20T03:14:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d2c00f930 con 0x7f0d341405a0 2022-07-20T03:14:34.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d34111160 con 0x7f0d341405a0 2022-07-20T03:14:34.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d340af8b0 con 0x7f0d341405a0 2022-07-20T03:14:34.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0d2c019030 con 0x7f0d341405a0 2022-07-20T03:14:34.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d31ffb700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d18053a40 0x7f0d18055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f0d2c06b600 con 0x7f0d341405a0 2022-07-20T03:14:34.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.642+0000 7f0d33fff700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d18053a40 0x7f0d18055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.646+0000 7f0d33fff700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d18053a40 0x7f0d18055f00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f0d2401a000 tx=0x7f0d24014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.650+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d2c0423c0 con 0x7f0d341405a0 2022-07-20T03:14:34.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.814+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d340574e0 con 0x7f0d341405a0 2022-07-20T03:14:34.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.814+0000 7f0d31ffb700 1 -- 172.21.15.61:0/4129172016 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 40 v40) v1 ==== 94+0+2137 (secure 0 0 0) 0x7f0d2c0459e0 con 0x7f0d341405a0 2022-07-20T03:14:34.827 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:34.829 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":40,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":40,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:13:38.444883+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6285},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6285":{"gid":6285,"name":"a","rank":0,"incarnation":39,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/460461689","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":460461689},{"type":"v1","addr":"172.21.15.61:6833","nonce":460461689}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:14:34.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.814+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d18053a40 msgr2=0x7f0d18055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.814+0000 7f0d3a320700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d18053a40 0x7f0d18055f00 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f0d2401a000 tx=0x7f0d24014000).stop 2022-07-20T03:14:34.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.814+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 msgr2=0x7f0d340afbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.814+0000 7f0d3a320700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 0x7f0d340afbc0 secure :-1 s=READY pgs=767 cs=0 l=1 rev1=1 rx=0x7f0d34064890 tx=0x7f0d2c007f40).stop 2022-07-20T03:14:34.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 shutdown_connections 2022-07-20T03:14:34.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d18053a40 0x7f0d18055f00 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d340ad770 0x7f0d34141fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d341405a0 0x7f0d340afbc0 unknown :-1 s=CLOSED pgs=767 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 --2- 172.21.15.61:0/4129172016 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d340acea0 0x7f0d34067020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 >> 172.21.15.61:0/4129172016 conn(0x7f0d340b72c0 msgr2=0x7f0d3405cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:34.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 shutdown_connections 2022-07-20T03:14:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.818+0000 7f0d3a320700 1 -- 172.21.15.61:0/4129172016 wait complete. 2022-07-20T03:14:34.844 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 40 2022-07-20T03:14:34.844 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:14:34.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 Processor -- start 2022-07-20T03:14:34.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 -- start start 2022-07-20T03:14:34.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2c8005cda0 0x7f2c8005d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2c800b11b0 0x7f2c8005fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c80147090 con 0x7f2c80060bb0 2022-07-20T03:14:34.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c80099080 con 0x7f2c800b11b0 2022-07-20T03:14:34.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c881c7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c801472c0 con 0x7f2c8005cda0 2022-07-20T03:14:34.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.958+0000 7f2c85762700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2c800b11b0 0x7f2c8005fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c86764700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2c80060bb0 0x7f2c80147760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43680/0 (socket says 172.21.15.61:43680) 2022-07-20T03:14:34.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85762700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2c800b11b0 0x7f2c8005fa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47080/0 (socket says 172.21.15.61:47080) 2022-07-20T03:14:34.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c86764700 1 -- 172.21.15.61:0/2023345467 learned_addr learned my addr 172.21.15.61:0/2023345467 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:34.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85f63700 1 --2- 172.21.15.61:0/2023345467 >> v2:172.21.15.117:3300/0 conn(0x7f2c8005cda0 0x7f2c8005d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85762700 1 -- 172.21.15.61:0/2023345467 >> v2:172.21.15.117:3300/0 conn(0x7f2c8005cda0 msgr2=0x7f2c8005d180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:34.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85762700 1 --2- 172.21.15.61:0/2023345467 >> v2:172.21.15.117:3300/0 conn(0x7f2c8005cda0 0x7f2c8005d180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85762700 1 -- 172.21.15.61:0/2023345467 >> v1:172.21.15.61:6789/0 conn(0x7f2c80060bb0 legacy=0x7f2c80147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85762700 1 -- 172.21.15.61:0/2023345467 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c80147e70 con 0x7f2c800b11b0 2022-07-20T03:14:34.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c84f61700 1 -- 172.21.15.61:0/2023345467 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 214705610 0 0) 0x7f2c80147090 con 0x7f2c80060bb0 2022-07-20T03:14:34.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c85762700 1 --2- 172.21.15.61:0/2023345467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c800b11b0 0x7f2c8005fa50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f2c780032f0 tx=0x7f2c780091d0).ready entity=mon.0 client_cookie=669f29fc9c8009ff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:34.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c84f61700 1 -- 172.21.15.61:0/2023345467 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c7800b680 con 0x7f2c800b11b0 2022-07-20T03:14:34.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c84f61700 1 -- 172.21.15.61:0/2023345467 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c7800b7e0 con 0x7f2c800b11b0 2022-07-20T03:14:34.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c84f61700 1 -- 172.21.15.61:0/2023345467 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c7800b950 con 0x7f2c800b11b0 2022-07-20T03:14:34.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 -- 172.21.15.61:0/2023345467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c800b11b0 msgr2=0x7f2c8005fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2023345467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c800b11b0 0x7f2c8005fa50 secure :-1 s=READY pgs=769 cs=0 l=1 rev1=1 rx=0x7f2c780032f0 tx=0x7f2c780091d0).stop 2022-07-20T03:14:34.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 -- 172.21.15.61:0/2023345467 shutdown_connections 2022-07-20T03:14:34.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2023345467 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c800b11b0 0x7f2c8005fa50 unknown :-1 s=CLOSED pgs=769 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2023345467 >> v2:172.21.15.117:3300/0 conn(0x7f2c8005cda0 0x7f2c8005d180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 -- 172.21.15.61:0/2023345467 >> 172.21.15.61:0/2023345467 conn(0x7f2c800b5bb0 msgr2=0x7f2c800b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:34.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 -- 172.21.15.61:0/2023345467 shutdown_connections 2022-07-20T03:14:34.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.962+0000 7f2c881c7700 1 -- 172.21.15.61:0/2023345467 wait complete. 2022-07-20T03:14:34.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 Processor -- start 2022-07-20T03:14:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 -- start start 2022-07-20T03:14:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c8005cda0 0x7f2c80142a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 0x7f2c800aac70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c800b11b0 0x7f2c80156900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80067180 con 0x7f2c80060bb0 2022-07-20T03:14:34.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80066760 con 0x7f2c800b11b0 2022-07-20T03:14:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c881c7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c80143700 con 0x7f2c8005cda0 2022-07-20T03:14:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c85762700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 0x7f2c800aac70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c85f63700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c8005cda0 0x7f2c80142a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c85762700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 0x7f2c800aac70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47086/0 (socket says 172.21.15.61:47086) 2022-07-20T03:14:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c85762700 1 -- 172.21.15.61:0/2561381976 learned_addr learned my addr 172.21.15.61:0/2561381976 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.966+0000 7f2c86764700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c800b11b0 0x7f2c80156900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:34.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c85762700 1 -- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c8005cda0 msgr2=0x7f2c80142a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c85762700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c8005cda0 0x7f2c80142a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c85762700 1 -- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c800b11b0 msgr2=0x7f2c80156900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c85762700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c800b11b0 0x7f2c80156900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:34.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c85762700 1 -- 172.21.15.61:0/2561381976 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c78009470 con 0x7f2c80060bb0 2022-07-20T03:14:34.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c85762700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 0x7f2c800aac70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f2c80143ba0 tx=0x7f2c78016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:35.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c7800b680 con 0x7f2c80060bb0 2022-07-20T03:14:35.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c78021630 con 0x7f2c80060bb0 2022-07-20T03:14:35.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c78021990 con 0x7f2c80060bb0 2022-07-20T03:14:35.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c80066990 con 0x7f2c80060bb0 2022-07-20T03:14:35.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c800aa4e0 con 0x7f2c80060bb0 2022-07-20T03:14:35.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c800574e0 con 0x7f2c80060bb0 2022-07-20T03:14:35.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.970+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2c78028070 con 0x7f2c80060bb0 2022-07-20T03:14:35.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.974+0000 7f2c6effd700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c60053a40 0x7f2c60055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:35.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.974+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(72..72 src has 1..72) v4 ==== 6116+0+0 (secure 0 0 0) 0x7f2c7806cec0 con 0x7f2c80060bb0 2022-07-20T03:14:35.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.974+0000 7f2c85f63700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c60053a40 0x7f2c60055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:35.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.974+0000 7f2c85f63700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c60053a40 0x7f2c60055f00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f2c7000b000 tx=0x7f2c70006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:35.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:34.978+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c78043d20 con 0x7f2c80060bb0 2022-07-20T03:14:35.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:35.142+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f2c800611c0 con 0x7f2c80060bb0 2022-07-20T03:14:36.061 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:14:36.058+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:14:36.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.066+0000 7f2c6effd700 1 -- 172.21.15.61:0/2561381976 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v41) v1 ==== 161+0+0 (secure 0 0 0) 0x7f2c78015070 con 0x7f2c80060bb0 2022-07-20T03:14:36.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c60053a40 msgr2=0x7f2c60055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c60053a40 0x7f2c60055f00 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f2c7000b000 tx=0x7f2c70006df0).stop 2022-07-20T03:14:36.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 msgr2=0x7f2c800aac70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 0x7f2c800aac70 secure :-1 s=READY pgs=770 cs=0 l=1 rev1=1 rx=0x7f2c80143ba0 tx=0x7f2c78016000).stop 2022-07-20T03:14:36.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 shutdown_connections 2022-07-20T03:14:36.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c60053a40 0x7f2c60055f00 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c800b11b0 0x7f2c80156900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c80060bb0 0x7f2c800aac70 unknown :-1 s=CLOSED pgs=770 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 --2- 172.21.15.61:0/2561381976 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c8005cda0 0x7f2c80142a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 >> 172.21.15.61:0/2561381976 conn(0x7f2c800b5bb0 msgr2=0x7f2c8009a5c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:36.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 shutdown_connections 2022-07-20T03:14:36.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.070+0000 7f2c881c7700 1 -- 172.21.15.61:0/2561381976 wait complete. 2022-07-20T03:14:36.078 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:14:36.097 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:36.103 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:14:36.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.198+0000 7f7d6bfff700 1 Processor -- start 2022-07-20T03:14:36.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.198+0000 7f7d6bfff700 1 -- start start 2022-07-20T03:14:36.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c1427a0 0x7f7d6c061130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c142c20 con 0x7f7d6c0b1ef0 2022-07-20T03:14:36.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d6c0b22d0 con 0x7f7d6c061620 2022-07-20T03:14:36.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c1472a0 con 0x7f7d6c1427a0 2022-07-20T03:14:36.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6affd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7d6c0b1ef0 0x7f7d6c142090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43710/0 (socket says 172.21.15.61:43710) 2022-07-20T03:14:36.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6affd700 1 -- 172.21.15.61:0/1669049525 learned_addr learned my addr 172.21.15.61:0/1669049525 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:36.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d627fc700 1 --2- 172.21.15.61:0/1669049525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c1427a0 0x7f7d6c061130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1248281326 0 0) 0x7f7d6c0b22d0 con 0x7f7d6c061620 2022-07-20T03:14:36.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d4c003260 con 0x7f7d6c061620 2022-07-20T03:14:36.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1133281042 0 0) 0x7f7d6c142c20 con 0x7f7d6c0b1ef0 2022-07-20T03:14:36.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d6c0b22d0 con 0x7f7d6c0b1ef0 2022-07-20T03:14:36.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d627fc700 1 -- 172.21.15.61:0/1669049525 >> v1:172.21.15.117:6789/0 conn(0x7f7d6c061620 legacy=0x7f7d6c061a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d627fc700 1 -- 172.21.15.61:0/1669049525 >> v1:172.21.15.61:6789/0 conn(0x7f7d6c0b1ef0 legacy=0x7f7d6c142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d627fc700 1 -- 172.21.15.61:0/1669049525 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d6c0ac0c0 con 0x7f7d6c1427a0 2022-07-20T03:14:36.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d627fc700 1 --2- 172.21.15.61:0/1669049525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c1427a0 0x7f7d6c061130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7d54002f70 tx=0x7f7d54006df0).ready entity=mon.2 client_cookie=9c529ba6f589df2a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:36.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5400d5c0 con 0x7f7d6c1427a0 2022-07-20T03:14:36.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d54000e90 con 0x7f7d6c1427a0 2022-07-20T03:14:36.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6a7fc700 1 -- 172.21.15.61:0/1669049525 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5400d910 con 0x7f7d6c1427a0 2022-07-20T03:14:36.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- 172.21.15.61:0/1669049525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c1427a0 msgr2=0x7f7d6c061130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/1669049525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c1427a0 0x7f7d6c061130 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f7d54002f70 tx=0x7f7d54006df0).stop 2022-07-20T03:14:36.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- 172.21.15.61:0/1669049525 shutdown_connections 2022-07-20T03:14:36.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/1669049525 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c1427a0 0x7f7d6c061130 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- 172.21.15.61:0/1669049525 >> 172.21.15.61:0/1669049525 conn(0x7f7d6c0b2ca0 msgr2=0x7f7d6c0b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:36.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- 172.21.15.61:0/1669049525 shutdown_connections 2022-07-20T03:14:36.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.202+0000 7f7d6bfff700 1 -- 172.21.15.61:0/1669049525 wait complete. 2022-07-20T03:14:36.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 Processor -- start 2022-07-20T03:14:36.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 -- start start 2022-07-20T03:14:36.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d6c061620 0x7f7d6c0afe30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c0b1ef0 0x7f7d6c09a530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 0x7f7d6c09aa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0af3c0 con 0x7f7d6c1427a0 2022-07-20T03:14:36.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0ae3c0 con 0x7f7d6c061620 2022-07-20T03:14:36.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d6c0ae660 con 0x7f7d6c0b1ef0 2022-07-20T03:14:36.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 0x7f7d6c09aa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 0x7f7d6c09aa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47116/0 (socket says 172.21.15.61:47116) 2022-07-20T03:14:36.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6b7fe700 1 -- 172.21.15.61:0/3882338867 learned_addr learned my addr 172.21.15.61:0/3882338867 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:36.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d6c061620 0x7f7d6c0afe30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d627fc700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c0b1ef0 0x7f7d6c09a530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 -- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c0b1ef0 msgr2=0x7f7d6c09a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c0b1ef0 0x7f7d6c09a530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 -- 172.21.15.61:0/3882338867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 msgr2=0x7f7d6c09aa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 0x7f7d6c09aa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 -- 172.21.15.61:0/3882338867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d5400b180 con 0x7f7d6c061620 2022-07-20T03:14:36.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6b7fe700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 0x7f7d6c09aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:36.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6affd700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d6c061620 0x7f7d6c0afe30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7d5c004870 tx=0x7f7d5c0079c0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:36.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5c00ce20 con 0x7f7d6c061620 2022-07-20T03:14:36.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d5c0070d0 con 0x7f7d6c061620 2022-07-20T03:14:36.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d6c0af7f0 con 0x7f7d6c061620 2022-07-20T03:14:36.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d5c007430 con 0x7f7d6c061620 2022-07-20T03:14:36.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.206+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d6c0adf20 con 0x7f7d6c061620 2022-07-20T03:14:36.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.210+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d6c0af530 con 0x7f7d6c061620 2022-07-20T03:14:36.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.210+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7d5c01a060 con 0x7f7d6c061620 2022-07-20T03:14:36.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.210+0000 7f7d68ff9700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d50053a60 0x7f7d50055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.210+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f7d5c06b5b0 con 0x7f7d6c061620 2022-07-20T03:14:36.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.210+0000 7f7d627fc700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d50053a60 0x7f7d50055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.210+0000 7f7d627fc700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d50053a60 0x7f7d50055f20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f7d54018000 tx=0x7f7d54014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:36.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.218+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d6c0af530 con 0x7f7d6c061620 2022-07-20T03:14:36.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.382+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f7d6c0af530 con 0x7f7d6c061620 2022-07-20T03:14:36.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d68ff9700 1 -- 172.21.15.61:0/3882338867 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 41 v41) v1 ==== 94+0+1779 (secure 0 0 0) 0x7f7d6c0af530 con 0x7f7d6c061620 2022-07-20T03:14:36.387 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:36.387 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":41,"default_fscid":6,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38}],"filesystems":[{"mdsmap":{"epoch":41,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:13:37.441965+0000","modified":"2022-07-20T03:14:36.061605+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":73,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[14],"metadata_pool":13,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":6}]} 2022-07-20T03:14:36.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d50053a60 msgr2=0x7f7d50055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d50053a60 0x7f7d50055f20 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f7d54018000 tx=0x7f7d54014000).stop 2022-07-20T03:14:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d6c061620 msgr2=0x7f7d6c0afe30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d6c061620 0x7f7d6c0afe30 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f7d5c004870 tx=0x7f7d5c0079c0).stop 2022-07-20T03:14:36.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 shutdown_connections 2022-07-20T03:14:36.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d50053a60 0x7f7d50055f20 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d6c1427a0 0x7f7d6c09aa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d6c0b1ef0 0x7f7d6c09a530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 --2- 172.21.15.61:0/3882338867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d6c061620 0x7f7d6c0afe30 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 >> 172.21.15.61:0/3882338867 conn(0x7f7d6c0b2ca0 msgr2=0x7f7d6c140470 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:36.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 shutdown_connections 2022-07-20T03:14:36.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.386+0000 7f7d6bfff700 1 -- 172.21.15.61:0/3882338867 wait complete. 2022-07-20T03:14:36.458 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 41 2022-07-20T03:14:36.464 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:14:36.483 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:36.517 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:36.522 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:36.544 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:36.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:14:36.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 Processor -- start 2022-07-20T03:14:36.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 -- start start 2022-07-20T03:14:36.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f170013e410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f17000af300 0x7f170013e900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17000aff10 0x7f170013f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17000b17b0 con 0x7f17000aff10 2022-07-20T03:14:36.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17000b09a0 con 0x7f17000af300 2022-07-20T03:14:36.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.598+0000 7f1704b66700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f170013f090 con 0x7f17000ace90 2022-07-20T03:14:36.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17000aff10 0x7f170013f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17000aff10 0x7f170013f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47124/0 (socket says 172.21.15.61:47124) 2022-07-20T03:14:36.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 -- 172.21.15.61:0/370249936 learned_addr learned my addr 172.21.15.61:0/370249936 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:36.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16f6ffd700 1 --2- 172.21.15.61:0/370249936 >> v2:172.21.15.117:3300/0 conn(0x7f17000af300 0x7f170013e900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16ff7fe700 1 --2- 172.21.15.61:0/370249936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f170013e410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 -- 172.21.15.61:0/370249936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 msgr2=0x7f170013e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:36.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 --2- 172.21.15.61:0/370249936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f170013e410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 -- 172.21.15.61:0/370249936 >> v2:172.21.15.117:3300/0 conn(0x7f17000af300 msgr2=0x7f170013e900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:36.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 --2- 172.21.15.61:0/370249936 >> v2:172.21.15.117:3300/0 conn(0x7f17000af300 0x7f170013e900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 -- 172.21.15.61:0/370249936 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f170013edf0 con 0x7f17000aff10 2022-07-20T03:14:36.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16fffff700 1 --2- 172.21.15.61:0/370249936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000aff10 0x7f170013f530 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f16f80032f0 tx=0x7f16f8009290).ready entity=mon.0 client_cookie=8cc8a41bddfc200c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:36.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16feffd700 1 -- 172.21.15.61:0/370249936 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16f800b870 con 0x7f17000aff10 2022-07-20T03:14:36.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16feffd700 1 -- 172.21.15.61:0/370249936 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16f800b9d0 con 0x7f17000aff10 2022-07-20T03:14:36.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f16feffd700 1 -- 172.21.15.61:0/370249936 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16f800bb40 con 0x7f17000aff10 2022-07-20T03:14:36.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 -- 172.21.15.61:0/370249936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000aff10 msgr2=0x7f170013f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 --2- 172.21.15.61:0/370249936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000aff10 0x7f170013f530 secure :-1 s=READY pgs=775 cs=0 l=1 rev1=1 rx=0x7f16f80032f0 tx=0x7f16f8009290).stop 2022-07-20T03:14:36.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 -- 172.21.15.61:0/370249936 shutdown_connections 2022-07-20T03:14:36.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 --2- 172.21.15.61:0/370249936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000aff10 0x7f170013f530 unknown :-1 s=CLOSED pgs=775 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 --2- 172.21.15.61:0/370249936 >> v2:172.21.15.117:3300/0 conn(0x7f17000af300 0x7f170013e900 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 --2- 172.21.15.61:0/370249936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f170013e410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 -- 172.21.15.61:0/370249936 >> 172.21.15.61:0/370249936 conn(0x7f1700063490 msgr2=0x7f17000b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:36.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 -- 172.21.15.61:0/370249936 shutdown_connections 2022-07-20T03:14:36.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.602+0000 7f1704b66700 1 -- 172.21.15.61:0/370249936 wait complete. 2022-07-20T03:14:36.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 Processor -- start 2022-07-20T03:14:36.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 -- start start 2022-07-20T03:14:36.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f17001414e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 0x7f17001333b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17000aff10 0x7f1700063e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17001406e0 con 0x7f17000af300 2022-07-20T03:14:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1700132c70 con 0x7f17000aff10 2022-07-20T03:14:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f1704b66700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1700132f10 con 0x7f17000ace90 2022-07-20T03:14:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 0x7f17001333b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 0x7f17001333b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47132/0 (socket says 172.21.15.61:47132) 2022-07-20T03:14:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16ff7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f17001414e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16fffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17000aff10 0x7f1700063e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 -- 172.21.15.61:0/775459026 learned_addr learned my addr 172.21.15.61:0/775459026 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:36.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 -- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 msgr2=0x7f17001414e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f17001414e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 -- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17000aff10 msgr2=0x7f1700063e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:36.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17000aff10 0x7f1700063e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:36.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 -- 172.21.15.61:0/775459026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16f8009530 con 0x7f17000af300 2022-07-20T03:14:36.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.606+0000 7f16f6ffd700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 0x7f17001333b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f16e80030b0 tx=0x7f16e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:36.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16e8011780 con 0x7f17000af300 2022-07-20T03:14:36.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16e80118e0 con 0x7f17000af300 2022-07-20T03:14:36.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17001425f0 con 0x7f17000af300 2022-07-20T03:14:36.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16e8011c40 con 0x7f17000af300 2022-07-20T03:14:36.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17000668a0 con 0x7f17000af300 2022-07-20T03:14:36.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17001330d0 con 0x7f17000af300 2022-07-20T03:14:36.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f16e8012420 con 0x7f17000af300 2022-07-20T03:14:36.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16fd7fa700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f16e4053920 0x7f16e4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:36.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f16e806bca0 con 0x7f17000af300 2022-07-20T03:14:36.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.610+0000 7f16ff7fe700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f16e4053920 0x7f16e4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:36.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.614+0000 7f16ff7fe700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f16e4053920 0x7f16e4055de0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f16f000c000 tx=0x7f16f0008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:36.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.622+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16e8070070 con 0x7f17000af300 2022-07-20T03:14:36.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:36.786+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f17000b0c40 con 0x7f17000af300 2022-07-20T03:14:37.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.070+0000 7f16fd7fa700 1 -- 172.21.15.61:0/775459026 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v42) v1 ==== 104+0+0 (secure 0 0 0) 0x7f16e8042b20 con 0x7f17000af300 2022-07-20T03:14:37.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f16e4053920 msgr2=0x7f16e4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:37.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f16e4053920 0x7f16e4055de0 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f16f000c000 tx=0x7f16f0008040).stop 2022-07-20T03:14:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 msgr2=0x7f17001333b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 0x7f17001333b0 secure :-1 s=READY pgs=776 cs=0 l=1 rev1=1 rx=0x7f16e80030b0 tx=0x7f16e8008040).stop 2022-07-20T03:14:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 shutdown_connections 2022-07-20T03:14:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f16e4053920 0x7f16e4055de0 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17000aff10 0x7f1700063e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17000af300 0x7f17001333b0 unknown :-1 s=CLOSED pgs=776 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 --2- 172.21.15.61:0/775459026 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17000ace90 0x7f17001414e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 >> 172.21.15.61:0/775459026 conn(0x7f1700063490 msgr2=0x7f17000b4110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:37.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 shutdown_connections 2022-07-20T03:14:37.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.074+0000 7f1704b66700 1 -- 172.21.15.61:0/775459026 wait complete. 2022-07-20T03:14:37.196 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:14:37.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.322+0000 7f66b5c01700 1 Processor -- start 2022-07-20T03:14:37.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.322+0000 7f66b5c01700 1 -- start start 2022-07-20T03:14:37.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b009cb10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:37.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f66b005cda0 0x7f66b0146a40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:37.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f66b0060fe0 con 0x7f66b0061410 2022-07-20T03:14:37.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66b009e590 con 0x7f66b005cda0 2022-07-20T03:14:37.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b00b39c0 con 0x7f66b0060c00 2022-07-20T03:14:37.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f66b005cda0 0x7f66b0146a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:37.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f66b005cda0 0x7f66b0146a40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47142/0 (socket says 172.21.15.61:47142) 2022-07-20T03:14:37.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 -- 172.21.15.61:0/2202408017 learned_addr learned my addr 172.21.15.61:0/2202408017 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:37.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66af7fe700 1 --2- 172.21.15.61:0/2202408017 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b009cb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:37.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 -- 172.21.15.61:0/2202408017 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 msgr2=0x7f66b009cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:37.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 --2- 172.21.15.61:0/2202408017 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b009cb10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 -- 172.21.15.61:0/2202408017 >> v1:172.21.15.117:6789/0 conn(0x7f66b0061410 legacy=0x7f66b0146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:37.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 -- 172.21.15.61:0/2202408017 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66b009dbc0 con 0x7f66b005cda0 2022-07-20T03:14:37.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66affff700 1 --2- 172.21.15.61:0/2202408017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b005cda0 0x7f66b0146a40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f66a80032f0 tx=0x7f66a8009290).ready entity=mon.0 client_cookie=5c0d87148809ee72 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:37.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66ae7fc700 1 -- 172.21.15.61:0/2202408017 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a800b8a0 con 0x7f66b005cda0 2022-07-20T03:14:37.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66ae7fc700 1 -- 172.21.15.61:0/2202408017 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a800ba00 con 0x7f66b005cda0 2022-07-20T03:14:37.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66ae7fc700 1 -- 172.21.15.61:0/2202408017 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a800bb70 con 0x7f66b005cda0 2022-07-20T03:14:37.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- 172.21.15.61:0/2202408017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b005cda0 msgr2=0x7f66b0146a40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:37.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 --2- 172.21.15.61:0/2202408017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b005cda0 0x7f66b0146a40 secure :-1 s=READY pgs=777 cs=0 l=1 rev1=1 rx=0x7f66a80032f0 tx=0x7f66a8009290).stop 2022-07-20T03:14:37.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- 172.21.15.61:0/2202408017 shutdown_connections 2022-07-20T03:14:37.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 --2- 172.21.15.61:0/2202408017 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b005cda0 0x7f66b0146a40 unknown :-1 s=CLOSED pgs=777 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 --2- 172.21.15.61:0/2202408017 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b009cb10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- 172.21.15.61:0/2202408017 >> 172.21.15.61:0/2202408017 conn(0x7f66b00b5f30 msgr2=0x7f66b00b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:37.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- 172.21.15.61:0/2202408017 shutdown_connections 2022-07-20T03:14:37.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.326+0000 7f66b5c01700 1 -- 172.21.15.61:0/2202408017 wait complete. 2022-07-20T03:14:37.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 Processor -- start 2022-07-20T03:14:37.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 -- start start 2022-07-20T03:14:37.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66b005cda0 0x7f66b008f860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:37.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b0143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:37.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 0x7f66b009ab90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:37.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b01521e0 con 0x7f66b0061410 2022-07-20T03:14:37.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0152480 con 0x7f66b005cda0 2022-07-20T03:14:37.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f66b0152720 con 0x7f66b0060c00 2022-07-20T03:14:37.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 0x7f66b009ab90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:37.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 0x7f66b009ab90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47150/0 (socket says 172.21.15.61:47150) 2022-07-20T03:14:37.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 -- 172.21.15.61:0/4200755867 learned_addr learned my addr 172.21.15.61:0/4200755867 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:37.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66af7fe700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66b005cda0 0x7f66b008f860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:37.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66aeffd700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b0143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:37.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 -- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 msgr2=0x7f66b0143210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:37.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b0143210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 -- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66b005cda0 msgr2=0x7f66b008f860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:37.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66b005cda0 0x7f66b008f860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:37.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 -- 172.21.15.61:0/4200755867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66a8009530 con 0x7f66b0061410 2022-07-20T03:14:37.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66affff700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 0x7f66b009ab90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f66a80032f0 tx=0x7f66a8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:37.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a800bb70 con 0x7f66b0061410 2022-07-20T03:14:37.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66a8021630 con 0x7f66b0061410 2022-07-20T03:14:37.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66b0151ef0 con 0x7f66b0061410 2022-07-20T03:14:37.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f66a800b8a0 con 0x7f66b0061410 2022-07-20T03:14:37.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.330+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66b0144c50 con 0x7f66b0061410 2022-07-20T03:14:37.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.334+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66b01528e0 con 0x7f66b0061410 2022-07-20T03:14:37.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.334+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f66a8027020 con 0x7f66b0061410 2022-07-20T03:14:37.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.334+0000 7f66acff9700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66900539b0 0x7f6690055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:37.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.334+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(73..73 src has 1..73) v4 ==== 6202+0+0 (secure 0 0 0) 0x7f66a806d1f0 con 0x7f66b0061410 2022-07-20T03:14:37.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.334+0000 7f66af7fe700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66900539b0 0x7f6690055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:37.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.334+0000 7f66af7fe700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66900539b0 0x7f6690055e70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f66a400c000 tx=0x7f66a4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:37.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.338+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f66a80440d0 con 0x7f66b0061410 2022-07-20T03:14:37.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:37.462+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f66b009de60 con 0x7f66b0061410 2022-07-20T03:14:38.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.266+0000 7f66acff9700 1 -- 172.21.15.61:0/4200755867 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v74) v1 ==== 185+0+0 (secure 0 0 0) 0x7f66a8015020 con 0x7f66b0061410 2022-07-20T03:14:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66900539b0 msgr2=0x7f6690055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66900539b0 0x7f6690055e70 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f66a400c000 tx=0x7f66a4008040).stop 2022-07-20T03:14:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 msgr2=0x7f66b009ab90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 0x7f66b009ab90 secure :-1 s=READY pgs=778 cs=0 l=1 rev1=1 rx=0x7f66a80032f0 tx=0x7f66a8016000).stop 2022-07-20T03:14:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 shutdown_connections 2022-07-20T03:14:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66900539b0 0x7f6690055e70 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f66b0061410 0x7f66b009ab90 unknown :-1 s=CLOSED pgs=778 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f66b0060c00 0x7f66b0143210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 --2- 172.21.15.61:0/4200755867 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66b005cda0 0x7f66b008f860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 >> 172.21.15.61:0/4200755867 conn(0x7f66b00b5f30 msgr2=0x7f66b00b86b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 shutdown_connections 2022-07-20T03:14:38.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.270+0000 7f66b5c01700 1 -- 172.21.15.61:0/4200755867 wait complete. 2022-07-20T03:14:38.289 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:14:38.291 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:14:38.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.410+0000 7f47311cd700 1 Processor -- start 2022-07-20T03:14:38.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.410+0000 7f47311cd700 1 -- start start 2022-07-20T03:14:38.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.410+0000 7f47311cd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c14d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:38.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.410+0000 7f47311cd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c066b10 con 0x7f472c0b6840 2022-07-20T03:14:38.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.410+0000 7f47311cd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f472c0b6c20 con 0x7f472c0b3cf0 2022-07-20T03:14:38.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.410+0000 7f47311cd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b5c90 con 0x7f472c0b4bc0 2022-07-20T03:14:38.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f472c0b6840 0x7f472c0b5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43752/0 (socket says 172.21.15.61:43752) 2022-07-20T03:14:38.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472bfff700 1 -- 172.21.15.61:0/3092534382 learned_addr learned my addr 172.21.15.61:0/3092534382 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:38.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472b7fe700 1 --1- 172.21.15.61:0/3092534382 >> v1:172.21.15.117:6789/0 conn(0x7f472c0b3cf0 0x7f472c146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:52258/0 (socket says 172.21.15.61:52258) 2022-07-20T03:14:38.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47309cc700 1 --2- 172.21.15.61:0/3092534382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c14d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:38.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1286622683 0 0) 0x7f472c0b6c20 con 0x7f472c0b3cf0 2022-07-20T03:14:38.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f470c003260 con 0x7f472c0b3cf0 2022-07-20T03:14:38.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1968429447 0 0) 0x7f472c066b10 con 0x7f472c0b6840 2022-07-20T03:14:38.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f472c0b6c20 con 0x7f472c0b6840 2022-07-20T03:14:38.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47309cc700 1 -- 172.21.15.61:0/3092534382 >> v1:172.21.15.117:6789/0 conn(0x7f472c0b3cf0 legacy=0x7f472c146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47309cc700 1 -- 172.21.15.61:0/3092534382 >> v1:172.21.15.61:6789/0 conn(0x7f472c0b6840 legacy=0x7f472c0b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47309cc700 1 -- 172.21.15.61:0/3092534382 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f472c0aead0 con 0x7f472c0b4bc0 2022-07-20T03:14:38.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47309cc700 1 --2- 172.21.15.61:0/3092534382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c14d610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f47240032f0 tx=0x7f47240091d0).ready entity=mon.2 client_cookie=b802916081253155 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:38.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f472400b640 con 0x7f472c0b4bc0 2022-07-20T03:14:38.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f472400b7a0 con 0x7f472c0b4bc0 2022-07-20T03:14:38.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f472affd700 1 -- 172.21.15.61:0/3092534382 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f472400b910 con 0x7f472c0b4bc0 2022-07-20T03:14:38.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 -- 172.21.15.61:0/3092534382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 msgr2=0x7f472c14d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 --2- 172.21.15.61:0/3092534382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c14d610 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f47240032f0 tx=0x7f47240091d0).stop 2022-07-20T03:14:38.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 -- 172.21.15.61:0/3092534382 shutdown_connections 2022-07-20T03:14:38.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 --2- 172.21.15.61:0/3092534382 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c14d610 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 -- 172.21.15.61:0/3092534382 >> 172.21.15.61:0/3092534382 conn(0x7f472c0b9e10 msgr2=0x7f472c0b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:38.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 -- 172.21.15.61:0/3092534382 shutdown_connections 2022-07-20T03:14:38.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.414+0000 7f47311cd700 1 -- 172.21.15.61:0/3092534382 wait complete. 2022-07-20T03:14:38.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 Processor -- start 2022-07-20T03:14:38.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 -- start start 2022-07-20T03:14:38.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f472c0b3cf0 0x7f472c13d490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:38.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c149c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:38.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f472c0b6840 0x7f472c140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0af390 con 0x7f472c0b6840 2022-07-20T03:14:38.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0af0f0 con 0x7f472c0b3cf0 2022-07-20T03:14:38.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f472c0b2b70 con 0x7f472c0b4bc0 2022-07-20T03:14:38.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f472c0b3cf0 0x7f472c13d490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:38.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47309cc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f472c0b6840 0x7f472c140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:38.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c149c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:38.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f472c0b3cf0 0x7f472c13d490 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:56842/0 (socket says 172.21.15.61:56842) 2022-07-20T03:14:38.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472bfff700 1 -- 172.21.15.61:0/1763278873 learned_addr learned my addr 172.21.15.61:0/1763278873 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:38.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 -- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f472c0b3cf0 msgr2=0x7f472c13d490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f472c0b3cf0 0x7f472c13d490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 -- 172.21.15.61:0/1763278873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f472c0b6840 msgr2=0x7f472c140da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:38.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f472c0b6840 0x7f472c140da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:38.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 -- 172.21.15.61:0/1763278873 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4724009470 con 0x7f472c0b4bc0 2022-07-20T03:14:38.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47309cc700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f472c0b6840 0x7f472c140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:38.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f472b7fe700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c149c90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f471c001110 tx=0x7f471c0079f0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:38.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c00cdb0 con 0x7f472c0b4bc0 2022-07-20T03:14:38.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471c0070d0 con 0x7f472c0b4bc0 2022-07-20T03:14:38.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f472c0b2e50 con 0x7f472c0b4bc0 2022-07-20T03:14:38.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471c005550 con 0x7f472c0b4bc0 2022-07-20T03:14:38.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.418+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f472c13f550 con 0x7f472c0b4bc0 2022-07-20T03:14:38.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.422+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f472c0574e0 con 0x7f472c0b4bc0 2022-07-20T03:14:38.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.422+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f471c010070 con 0x7f472c0b4bc0 2022-07-20T03:14:38.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.422+0000 7f47297fa700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4710053920 0x7f4710055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:38.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.422+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(74..74 src has 1..74) v4 ==== 5764+0+0 (secure 0 0 0) 0x7f471c06b250 con 0x7f472c0b4bc0 2022-07-20T03:14:38.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.422+0000 7f472bfff700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4710053920 0x7f4710055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:38.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.422+0000 7f472bfff700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4710053920 0x7f4710055de0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f471400d000 tx=0x7f4714009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:38.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.434+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f471c0421d0 con 0x7f472c0b4bc0 2022-07-20T03:14:38.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:38.562+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f472c13f940 con 0x7f472c0b4bc0 2022-07-20T03:14:39.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.278+0000 7f47297fa700 1 -- 172.21.15.61:0/1763278873 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v75) v1 ==== 173+0+0 (secure 0 0 0) 0x7f471c0457f0 con 0x7f472c0b4bc0 2022-07-20T03:14:39.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4710053920 msgr2=0x7f4710055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4710053920 0x7f4710055de0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f471400d000 tx=0x7f4714009040).stop 2022-07-20T03:14:39.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 msgr2=0x7f472c149c90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c149c90 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f471c001110 tx=0x7f471c0079f0).stop 2022-07-20T03:14:39.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 shutdown_connections 2022-07-20T03:14:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4710053920 0x7f4710055de0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f472c0b6840 0x7f472c140da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f472c0b4bc0 0x7f472c149c90 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 --2- 172.21.15.61:0/1763278873 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f472c0b3cf0 0x7f472c13d490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 >> 172.21.15.61:0/1763278873 conn(0x7f472c0b9e10 msgr2=0x7f472c148130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 shutdown_connections 2022-07-20T03:14:39.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.282+0000 7f47311cd700 1 -- 172.21.15.61:0/1763278873 wait complete. 2022-07-20T03:14:39.319 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:14:39.319 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:14:39.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 Processor -- start 2022-07-20T03:14:39.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 -- start start 2022-07-20T03:14:39.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc0b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24dc146f20 0x7f24dc0b1040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24dc143970 con 0x7f24dc094b30 2022-07-20T03:14:39.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24dc095a00 con 0x7f24dc146f20 2022-07-20T03:14:39.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24e1edd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc093de0 con 0x7f24dc05c960 2022-07-20T03:14:39.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.446+0000 7f24daffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24dc146f20 0x7f24dc0b1040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24dbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f24dc094b30 0x7f24dc142250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43770/0 (socket says 172.21.15.61:43770) 2022-07-20T03:14:39.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24dbfff700 1 -- 172.21.15.61:0/885863045 learned_addr learned my addr 172.21.15.61:0/885863045 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:39.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24db7fe700 1 --2- 172.21.15.61:0/885863045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc0b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24daffd700 1 -- 172.21.15.61:0/885863045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 msgr2=0x7f24dc0b0b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:39.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24daffd700 1 --2- 172.21.15.61:0/885863045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc0b0b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24daffd700 1 -- 172.21.15.61:0/885863045 >> v1:172.21.15.61:6789/0 conn(0x7f24dc094b30 legacy=0x7f24dc142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24daffd700 1 -- 172.21.15.61:0/885863045 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24dc05cd40 con 0x7f24dc146f20 2022-07-20T03:14:39.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24daffd700 1 --2- 172.21.15.61:0/885863045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc146f20 0x7f24dc0b1040 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f24dc0b6150 tx=0x7f24cc006df0).ready entity=mon.0 client_cookie=34a59e8177fccf3c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:39.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24da7fc700 1 -- 172.21.15.61:0/885863045 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24cc00d570 con 0x7f24dc146f20 2022-07-20T03:14:39.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24da7fc700 1 -- 172.21.15.61:0/885863045 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24cc000e90 con 0x7f24dc146f20 2022-07-20T03:14:39.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24da7fc700 1 -- 172.21.15.61:0/885863045 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24cc00d8c0 con 0x7f24dc146f20 2022-07-20T03:14:39.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 -- 172.21.15.61:0/885863045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc146f20 msgr2=0x7f24dc0b1040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 --2- 172.21.15.61:0/885863045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc146f20 0x7f24dc0b1040 secure :-1 s=READY pgs=781 cs=0 l=1 rev1=1 rx=0x7f24dc0b6150 tx=0x7f24cc006df0).stop 2022-07-20T03:14:39.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 -- 172.21.15.61:0/885863045 shutdown_connections 2022-07-20T03:14:39.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 --2- 172.21.15.61:0/885863045 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc146f20 0x7f24dc0b1040 unknown :-1 s=CLOSED pgs=781 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 --2- 172.21.15.61:0/885863045 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc0b0b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 -- 172.21.15.61:0/885863045 >> 172.21.15.61:0/885863045 conn(0x7f24dc0b2c50 msgr2=0x7f24dc0b3030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:39.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 -- 172.21.15.61:0/885863045 shutdown_connections 2022-07-20T03:14:39.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 -- 172.21.15.61:0/885863045 wait complete. 2022-07-20T03:14:39.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.450+0000 7f24e1edd700 1 Processor -- start 2022-07-20T03:14:39.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 -- start start 2022-07-20T03:14:39.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc065ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc094b30 0x7f24dc066160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24dc146f20 0x7f24dc0673e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc0ad3e0 con 0x7f24dc094b30 2022-07-20T03:14:39.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc064ce0 con 0x7f24dc146f20 2022-07-20T03:14:39.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24dc142490 con 0x7f24dc05c960 2022-07-20T03:14:39.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc065ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc065ad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33846/0 (socket says 172.21.15.61:33846) 2022-07-20T03:14:39.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 -- 172.21.15.61:0/2633329663 learned_addr learned my addr 172.21.15.61:0/2633329663 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:39.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24dbfff700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24dc146f20 0x7f24dc0673e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24daffd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc094b30 0x7f24dc066160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 -- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24dc146f20 msgr2=0x7f24dc0673e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24dc146f20 0x7f24dc0673e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 -- 172.21.15.61:0/2633329663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc094b30 msgr2=0x7f24dc066160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc094b30 0x7f24dc066160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 -- 172.21.15.61:0/2633329663 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24cc00b180 con 0x7f24dc05c960 2022-07-20T03:14:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24daffd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc094b30 0x7f24dc066160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24db7fe700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc065ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f24dc1423b0 tx=0x7f24c4006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:39.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24c4010780 con 0x7f24dc05c960 2022-07-20T03:14:39.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24c40108e0 con 0x7f24dc05c960 2022-07-20T03:14:39.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24dc142730 con 0x7f24dc05c960 2022-07-20T03:14:39.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24c4010c40 con 0x7f24dc05c960 2022-07-20T03:14:39.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.454+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24dc0678d0 con 0x7f24dc05c960 2022-07-20T03:14:39.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.458+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24dc05ffd0 con 0x7f24dc05c960 2022-07-20T03:14:39.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.458+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f24c401c460 con 0x7f24dc05c960 2022-07-20T03:14:39.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.458+0000 7f24d8ff9700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c0053a40 0x7f24c0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.458+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f24c406c0e0 con 0x7f24dc05c960 2022-07-20T03:14:39.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.458+0000 7f24daffd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c0053a40 0x7f24c0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.458+0000 7f24daffd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c0053a40 0x7f24c0055f00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f24cc01a000 tx=0x7f24cc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:39.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.466+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24dc05ffd0 con 0x7f24dc05c960 2022-07-20T03:14:39.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.602+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f24dc05ffd0 con 0x7f24dc05c960 2022-07-20T03:14:39.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.610+0000 7f24d8ff9700 1 -- 172.21.15.61:0/2633329663 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v11) v1 ==== 104+0+0 (secure 0 0 0) 0x7f24dc05ffd0 con 0x7f24dc05c960 2022-07-20T03:14:39.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c0053a40 msgr2=0x7f24c0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c0053a40 0x7f24c0055f00 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f24cc01a000 tx=0x7f24cc014000).stop 2022-07-20T03:14:39.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 msgr2=0x7f24dc065ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc065ad0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f24dc1423b0 tx=0x7f24c4006df0).stop 2022-07-20T03:14:39.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 shutdown_connections 2022-07-20T03:14:39.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f24c0053a40 0x7f24c0055f00 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24dc146f20 0x7f24dc0673e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24dc094b30 0x7f24dc066160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 --2- 172.21.15.61:0/2633329663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24dc05c960 0x7f24dc065ad0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 >> 172.21.15.61:0/2633329663 conn(0x7f24dc0b2c50 msgr2=0x7f24dc0b4ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:39.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 shutdown_connections 2022-07-20T03:14:39.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.614+0000 7f24e1edd700 1 -- 172.21.15.61:0/2633329663 wait complete. 2022-07-20T03:14:39.642 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder' 2022-07-20T03:14:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.750+0000 7f6ecd228700 1 Processor -- start 2022-07-20T03:14:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.750+0000 7f6ecd228700 1 -- start start 2022-07-20T03:14:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 0x7f6ec80aefb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6ec80ace90 0x7f6ec80b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6ec81473e0 con 0x7f6ec80b11c0 2022-07-20T03:14:39.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6ec80b5f10 con 0x7f6ec80ace90 2022-07-20T03:14:39.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80afc40 con 0x7f6ec80b5b30 2022-07-20T03:14:39.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec759e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6ec80b11c0 0x7f6ec814a760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43782/0 (socket says 172.21.15.61:43782) 2022-07-20T03:14:39.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec759e700 1 -- 172.21.15.61:0/3930352077 learned_addr learned my addr 172.21.15.61:0/3930352077 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:39.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 --2- 172.21.15.61:0/3930352077 >> v2:172.21.15.117:3300/0 conn(0x7f6ec80ace90 0x7f6ec80b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec6d9d700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 0x7f6ec80aefb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1403532111 0 0) 0x7f6eb0001110 con 0x7f6ec80b11c0 2022-07-20T03:14:39.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6ea8003260 con 0x7f6ec80b11c0 2022-07-20T03:14:39.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3734294349 0 0) 0x7f6ea8003260 con 0x7f6ec80b11c0 2022-07-20T03:14:39.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 msgr2=0x7f6ec80aefb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:39.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 0x7f6ec80aefb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 >> v2:172.21.15.117:3300/0 conn(0x7f6ec80ace90 msgr2=0x7f6ec80b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:39.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 --2- 172.21.15.61:0/3930352077 >> v2:172.21.15.117:3300/0 conn(0x7f6ec80ace90 0x7f6ec80b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec80afee0 con 0x7f6ec80b11c0 2022-07-20T03:14:39.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f6eb00017f0 con 0x7f6ec80b11c0 2022-07-20T03:14:39.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 >> v1:172.21.15.61:6789/0 conn(0x7f6ec80b11c0 legacy=0x7f6ec814a760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 0x7f6ea8003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ea8003c80 0x7f6ea8004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ea80049f0 0x7f6ea8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80b5f10 con 0x7f6ea8003c80 2022-07-20T03:14:39.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80afc40 con 0x7f6ea80049f0 2022-07-20T03:14:39.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ea800b750 con 0x7f6ec80b5b30 2022-07-20T03:14:39.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ea8003c80 0x7f6ea8004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec759e700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ea80049f0 0x7f6ea8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 -- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 msgr2=0x7f6ea8003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:39.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ec80b5b30 0x7f6ea8003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).stop 2022-07-20T03:14:39.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 -- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ea80049f0 msgr2=0x7f6ea8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:39.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ea80049f0 0x7f6ea8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 -- 172.21.15.61:0/3930352077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ea8003260 con 0x7f6ea8003c80 2022-07-20T03:14:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec759e700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ea80049f0 0x7f6ea8004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:39.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec659c700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ea8003c80 0x7f6ea8004500 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6ea80054d0 tx=0x7f6ec00091d0).ready entity=mon.0 client_cookie=1b8b3880b428b632 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:39.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec000e7e0 con 0x7f6ea8003c80 2022-07-20T03:14:39.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec000e940 con 0x7f6ea8003c80 2022-07-20T03:14:39.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec5d9b700 1 -- 172.21.15.61:0/3930352077 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec000eca0 con 0x7f6ea8003c80 2022-07-20T03:14:39.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- 172.21.15.61:0/3930352077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ea8003c80 msgr2=0x7f6ea8004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 --2- 172.21.15.61:0/3930352077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ea8003c80 0x7f6ea8004500 secure :-1 s=READY pgs=783 cs=0 l=1 rev1=1 rx=0x7f6ea80054d0 tx=0x7f6ec00091d0).stop 2022-07-20T03:14:39.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ec759e700 1 -- 172.21.15.61:0/3930352077 reap_dead start 2022-07-20T03:14:39.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- 172.21.15.61:0/3930352077 shutdown_connections 2022-07-20T03:14:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- 172.21.15.61:0/3930352077 >> 172.21.15.61:0/3930352077 conn(0x7f6ec80635b0 msgr2=0x7f6ec80b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- 172.21.15.61:0/3930352077 shutdown_connections 2022-07-20T03:14:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.754+0000 7f6ecd228700 1 -- 172.21.15.61:0/3930352077 wait complete. 2022-07-20T03:14:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.758+0000 7f6ecd228700 1 Processor -- start 2022-07-20T03:14:39.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.758+0000 7f6ecd228700 1 -- start start 2022-07-20T03:14:39.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.758+0000 7f6ecd228700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ea8003c80 0x7f6ec8111290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.758+0000 7f6ecd228700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 0x7f6ec81567f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec8067230 0x7f6ec8156ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec81107d0 con 0x7f6ec8111780 2022-07-20T03:14:39.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec80660f0 con 0x7f6ec8067230 2022-07-20T03:14:39.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6ec8066530 con 0x7f6ea8003c80 2022-07-20T03:14:39.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 0x7f6ec81567f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 0x7f6ec81567f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47196/0 (socket says 172.21.15.61:47196) 2022-07-20T03:14:39.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 -- 172.21.15.61:0/841422518 learned_addr learned my addr 172.21.15.61:0/841422518 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec6d9d700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ea8003c80 0x7f6ec8111290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec759e700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec8067230 0x7f6ec8156ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 -- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ea8003c80 msgr2=0x7f6ec8111290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ea8003c80 0x7f6ec8111290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 -- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec8067230 msgr2=0x7f6ec8156ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec8067230 0x7f6ec8156ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 -- 172.21.15.61:0/841422518 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ec8111d00 con 0x7f6ec8111780 2022-07-20T03:14:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec6d9d700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ea8003c80 0x7f6ec8111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec759e700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec8067230 0x7f6ec8156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec659c700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 0x7f6ec81567f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6ec8066230 tx=0x7f6ec001a000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec0009600 con 0x7f6ec8111780 2022-07-20T03:14:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ec001b410 con 0x7f6ec8111780 2022-07-20T03:14:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ec001dea0 con 0x7f6ec8111780 2022-07-20T03:14:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ec8111fa0 con 0x7f6ec8111780 2022-07-20T03:14:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6ec81571d0 con 0x7f6ec8111780 2022-07-20T03:14:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6ec80ad490 con 0x7f6ec8111780 2022-07-20T03:14:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6ec002b020 con 0x7f6ec8111780 2022-07-20T03:14:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ebffff700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eac053a60 0x7f6eac055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f6ec006d990 con 0x7f6ec8111780 2022-07-20T03:14:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.762+0000 7f6ec6d9d700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eac053a60 0x7f6eac055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.766+0000 7f6ec6d9d700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eac053a60 0x7f6eac055f20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f6eb800b000 tx=0x7f6eb8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.774+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ec0044ba0 con 0x7f6ec8111780 2022-07-20T03:14:39.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:39.938+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]} v 0) v1 -- 0x7f6ec80574e0 con 0x7f6ec8111780 2022-07-20T03:14:40.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.278+0000 7f6ebffff700 1 -- 172.21.15.61:0/841422518 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_cap_revoke_nonresponder"]}]=0 v331) v1 ==== 139+0+0 (secure 0 0 0) 0x7f6ec0016020 con 0x7f6ec8111780 2022-07-20T03:14:40.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eac053a60 msgr2=0x7f6eac055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:40.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eac053a60 0x7f6eac055f20 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f6eb800b000 tx=0x7f6eb8008040).stop 2022-07-20T03:14:40.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 msgr2=0x7f6ec81567f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:40.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 0x7f6ec81567f0 secure :-1 s=READY pgs=784 cs=0 l=1 rev1=1 rx=0x7f6ec8066230 tx=0x7f6ec001a000).stop 2022-07-20T03:14:40.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 shutdown_connections 2022-07-20T03:14:40.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6eac053a60 0x7f6eac055f20 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6ec8067230 0x7f6ec8156ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6ec8111780 0x7f6ec81567f0 unknown :-1 s=CLOSED pgs=784 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 --2- 172.21.15.61:0/841422518 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6ea8003c80 0x7f6ec8111290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 >> 172.21.15.61:0/841422518 conn(0x7f6ec80635b0 msgr2=0x7f6ec80b45a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:40.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 shutdown_connections 2022-07-20T03:14:40.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.282+0000 7f6ecd228700 1 -- 172.21.15.61:0/841422518 wait complete. 2022-07-20T03:14:40.308 INFO:tasks.cephfs_test_runner:test_cap_revoke_nonresponder (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:14:40.309 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:14:40.309 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-20T03:14:40.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.430+0000 7f616bfff700 1 Processor -- start 2022-07-20T03:14:40.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.430+0000 7f616bfff700 1 -- start start 2022-07-20T03:14:40.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f616c0b0890 0x7f616c0ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:40.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f616c0b0350 0x7f616c13e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:40.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f616c0aabe0 con 0x7f616c0b4b10 2022-07-20T03:14:40.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0b4ef0 con 0x7f616c0b0890 2022-07-20T03:14:40.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f616c0aae10 con 0x7f616c0b0350 2022-07-20T03:14:40.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f616c0b0890 0x7f616c0ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:40.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f616c0b0890 0x7f616c0ac1e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47204/0 (socket says 172.21.15.61:47204) 2022-07-20T03:14:40.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 -- 172.21.15.61:0/825447873 learned_addr learned my addr 172.21.15.61:0/825447873 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:40.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616affd700 1 --1- 172.21.15.61:0/825447873 >> v1:172.21.15.117:6789/0 conn(0x7f616c0b4b10 0x7f616c0af8b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:52304/0 (socket says 172.21.15.61:52304) 2022-07-20T03:14:40.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616b7fe700 1 --2- 172.21.15.61:0/825447873 >> v2:172.21.15.117:3300/0 conn(0x7f616c0b0350 0x7f616c13e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:40.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 -- 172.21.15.61:0/825447873 >> v2:172.21.15.117:3300/0 conn(0x7f616c0b0350 msgr2=0x7f616c13e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:40.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 --2- 172.21.15.61:0/825447873 >> v2:172.21.15.117:3300/0 conn(0x7f616c0b0350 0x7f616c13e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 -- 172.21.15.61:0/825447873 >> v1:172.21.15.117:6789/0 conn(0x7f616c0b4b10 legacy=0x7f616c0af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:40.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 -- 172.21.15.61:0/825447873 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f616c0ab0b0 con 0x7f616c0b0890 2022-07-20T03:14:40.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f61627fc700 1 --2- 172.21.15.61:0/825447873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b0890 0x7f616c0ac1e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f6154002f70 tx=0x7f6154006f40).ready entity=mon.0 client_cookie=f7f0d21567218b2d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:40.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616a7fc700 1 -- 172.21.15.61:0/825447873 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f615400da80 con 0x7f616c0b0890 2022-07-20T03:14:40.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616a7fc700 1 -- 172.21.15.61:0/825447873 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6154000e90 con 0x7f616c0b0890 2022-07-20T03:14:40.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616a7fc700 1 -- 172.21.15.61:0/825447873 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f615400de10 con 0x7f616c0b0890 2022-07-20T03:14:40.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 -- 172.21.15.61:0/825447873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b0890 msgr2=0x7f616c0ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:40.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 --2- 172.21.15.61:0/825447873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b0890 0x7f616c0ac1e0 secure :-1 s=READY pgs=785 cs=0 l=1 rev1=1 rx=0x7f6154002f70 tx=0x7f6154006f40).stop 2022-07-20T03:14:40.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 -- 172.21.15.61:0/825447873 shutdown_connections 2022-07-20T03:14:40.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 --2- 172.21.15.61:0/825447873 >> v2:172.21.15.117:3300/0 conn(0x7f616c0b0350 0x7f616c13e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.434+0000 7f616bfff700 1 --2- 172.21.15.61:0/825447873 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b0890 0x7f616c0ac1e0 unknown :-1 s=CLOSED pgs=785 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- 172.21.15.61:0/825447873 >> 172.21.15.61:0/825447873 conn(0x7f616c0632b0 msgr2=0x7f616c063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:40.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- 172.21.15.61:0/825447873 shutdown_connections 2022-07-20T03:14:40.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- 172.21.15.61:0/825447873 wait complete. 2022-07-20T03:14:40.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 Processor -- start 2022-07-20T03:14:40.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- start start 2022-07-20T03:14:40.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f616c0b0350 0x7f616c0b5f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:40.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f616c0b0890 0x7f616c0ae1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:40.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 0x7f616c0ae6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:40.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c141f00 con 0x7f616c0b4b10 2022-07-20T03:14:40.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f616c060880 con 0x7f616c0b0890 2022-07-20T03:14:40.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f616c0603e0 con 0x7f616c0b0350 2022-07-20T03:14:40.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 0x7f616c0ae6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:40.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f61627fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f616c0b0890 0x7f616c0ae1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:40.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 0x7f616c0ae6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47212/0 (socket says 172.21.15.61:47212) 2022-07-20T03:14:40.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 -- 172.21.15.61:0/703643202 learned_addr learned my addr 172.21.15.61:0/703643202 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:40.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616affd700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f616c0b0350 0x7f616c0b5f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:40.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 -- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f616c0b0350 msgr2=0x7f616c0b5f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:40.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f616c0b0350 0x7f616c0b5f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 -- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f616c0b0890 msgr2=0x7f616c0ae1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:40.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f616c0b0890 0x7f616c0ae1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:40.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 -- 172.21.15.61:0/703643202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f615400b460 con 0x7f616c0b4b10 2022-07-20T03:14:40.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f616b7fe700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 0x7f616c0ae6e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f6164006e00 tx=0x7f61640091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:40.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.438+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f616400c640 con 0x7f616c0b4b10 2022-07-20T03:14:40.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f616400c7a0 con 0x7f616c0b4b10 2022-07-20T03:14:40.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f616c138af0 con 0x7f616c0b4b10 2022-07-20T03:14:40.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f616400cb00 con 0x7f616c0b4b10 2022-07-20T03:14:40.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f616c0ad250 con 0x7f616c0b4b10 2022-07-20T03:14:40.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f616c060a80 con 0x7f616c0b4b10 2022-07-20T03:14:40.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f616400cc60 con 0x7f616c0b4b10 2022-07-20T03:14:40.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f6168ff9700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6150053a60 0x7f6150055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:40.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f616406be20 con 0x7f616c0b4b10 2022-07-20T03:14:40.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.442+0000 7f616affd700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6150053a60 0x7f6150055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:40.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.446+0000 7f616affd700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6150053a60 0x7f6150055f20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f615c00d000 tx=0x7f615c007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:40.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.450+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6164042f90 con 0x7f616c0b4b10 2022-07-20T03:14:40.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:40.614+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7f616c0605a0 con 0x7f616c0b4b10 2022-07-20T03:14:41.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.286+0000 7f6168ff9700 1 -- 172.21.15.61:0/703643202 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v332) v1 ==== 129+0+0 (secure 0 0 0) 0x7f61640465b0 con 0x7f616c0b4b10 2022-07-20T03:14:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6150053a60 msgr2=0x7f6150055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6150053a60 0x7f6150055f20 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f615c00d000 tx=0x7f615c007b80).stop 2022-07-20T03:14:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 msgr2=0x7f616c0ae6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 0x7f616c0ae6e0 secure :-1 s=READY pgs=786 cs=0 l=1 rev1=1 rx=0x7f6164006e00 tx=0x7f61640091d0).stop 2022-07-20T03:14:41.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 shutdown_connections 2022-07-20T03:14:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6150053a60 0x7f6150055f20 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f616c0b0890 0x7f616c0ae1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f616c0b4b10 0x7f616c0ae6e0 unknown :-1 s=CLOSED pgs=786 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 --2- 172.21.15.61:0/703643202 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f616c0b0350 0x7f616c0b5f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 >> 172.21.15.61:0/703643202 conn(0x7f616c0632b0 msgr2=0x7f616c0b2210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:41.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 shutdown_connections 2022-07-20T03:14:41.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.290+0000 7f616bfff700 1 -- 172.21.15.61:0/703643202 wait complete. 2022-07-20T03:14:41.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:14:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.454+0000 7f28a56c8700 1 Processor -- start 2022-07-20T03:14:41.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.454+0000 7f28a56c8700 1 -- start start 2022-07-20T03:14:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f28a56c8700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f28a00b5960 0x7f28a0142b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f28a56c8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a0143060 0x7f28a009b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:41.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f28a56c8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f28a0147030 con 0x7f28a005d0e0 2022-07-20T03:14:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f28a56c8700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f28a005d4c0 con 0x7f28a00b5960 2022-07-20T03:14:41.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f28a56c8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a01461e0 con 0x7f28a0143060 2022-07-20T03:14:41.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f28a005d0e0 0x7f28a0146920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43814/0 (socket says 172.21.15.61:43814) 2022-07-20T03:14:41.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289effd700 1 -- 172.21.15.61:0/2491298470 learned_addr learned my addr 172.21.15.61:0/2491298470 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:41.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289f7fe700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a0143060 0x7f28a009b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:41.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289e7fc700 1 --2- 172.21.15.61:0/2491298470 >> v2:172.21.15.117:3300/0 conn(0x7f28a00b5960 0x7f28a0142b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:41.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289dffb700 1 -- 172.21.15.61:0/2491298470 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115238799 0 0) 0x7f28a0147030 con 0x7f28a005d0e0 2022-07-20T03:14:41.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289dffb700 1 -- 172.21.15.61:0/2491298470 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2884003260 con 0x7f28a005d0e0 2022-07-20T03:14:41.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289e7fc700 1 -- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a0143060 msgr2=0x7f28a009b6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:41.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289e7fc700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a0143060 0x7f28a009b6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289e7fc700 1 -- 172.21.15.61:0/2491298470 >> v1:172.21.15.61:6789/0 conn(0x7f28a005d0e0 legacy=0x7f28a0146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289e7fc700 1 -- 172.21.15.61:0/2491298470 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28a0146480 con 0x7f28a00b5960 2022-07-20T03:14:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289f7fe700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a0143060 0x7f28a009b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289e7fc700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a00b5960 0x7f28a0142b70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2894002f70 tx=0x7f2894006df0).ready entity=mon.1 client_cookie=115fc1ba85d0c2e2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289dffb700 1 -- 172.21.15.61:0/2491298470 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f289400d4a0 con 0x7f28a00b5960 2022-07-20T03:14:41.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289dffb700 1 -- 172.21.15.61:0/2491298470 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2894000e90 con 0x7f28a00b5960 2022-07-20T03:14:41.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.458+0000 7f289dffb700 1 -- 172.21.15.61:0/2491298470 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f289400d7f0 con 0x7f28a00b5960 2022-07-20T03:14:41.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a00b5960 msgr2=0x7f28a0142b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a00b5960 0x7f28a0142b70 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f2894002f70 tx=0x7f2894006df0).stop 2022-07-20T03:14:41.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- 172.21.15.61:0/2491298470 shutdown_connections 2022-07-20T03:14:41.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a0143060 0x7f28a009b6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 --2- 172.21.15.61:0/2491298470 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a00b5960 0x7f28a0142b70 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- 172.21.15.61:0/2491298470 >> 172.21.15.61:0/2491298470 conn(0x7f28a00b6a20 msgr2=0x7f28a0066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- 172.21.15.61:0/2491298470 shutdown_connections 2022-07-20T03:14:41.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- 172.21.15.61:0/2491298470 wait complete. 2022-07-20T03:14:41.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 Processor -- start 2022-07-20T03:14:41.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- start start 2022-07-20T03:14:41.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 0x7f28a0152660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:41.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a00b5960 0x7f28a00b1d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:41.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a0143060 0x7f28a00b2260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a00b1be0 con 0x7f28a005d0e0 2022-07-20T03:14:41.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0140410 con 0x7f28a0143060 2022-07-20T03:14:41.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f28a56c8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f28a0151f20 con 0x7f28a00b5960 2022-07-20T03:14:41.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f289effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 0x7f28a0152660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:41.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f289e7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a00b5960 0x7f28a00b1d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:41.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f289effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 0x7f28a0152660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47222/0 (socket says 172.21.15.61:47222) 2022-07-20T03:14:41.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f289f7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a0143060 0x7f28a00b2260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:41.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.462+0000 7f289effd700 1 -- 172.21.15.61:0/3333517612 learned_addr learned my addr 172.21.15.61:0/3333517612 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:41.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289effd700 1 -- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a00b5960 msgr2=0x7f28a00b1d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289effd700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a00b5960 0x7f28a00b1d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289effd700 1 -- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a0143060 msgr2=0x7f28a00b2260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289effd700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a0143060 0x7f28a00b2260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289effd700 1 -- 172.21.15.61:0/3333517612 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2898009470 con 0x7f28a005d0e0 2022-07-20T03:14:41.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289f7fe700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a0143060 0x7f28a00b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:41.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289e7fc700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a00b5960 0x7f28a00b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:41.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f289effd700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 0x7f28a0152660 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f2890001110 tx=0x7f2890007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:41.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f289000ce20 con 0x7f28a005d0e0 2022-07-20T03:14:41.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f28900072e0 con 0x7f28a005d0e0 2022-07-20T03:14:41.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f289400b180 con 0x7f28a005d0e0 2022-07-20T03:14:41.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2890005800 con 0x7f28a005d0e0 2022-07-20T03:14:41.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28a0152b50 con 0x7f28a005d0e0 2022-07-20T03:14:41.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28a00b01d0 con 0x7f28a005d0e0 2022-07-20T03:14:41.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2890010070 con 0x7f28a005d0e0 2022-07-20T03:14:41.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.466+0000 7f288ffff700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f287c0539b0 0x7f287c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:41.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.470+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f289006b280 con 0x7f28a005d0e0 2022-07-20T03:14:41.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.470+0000 7f289e7fc700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f287c0539b0 0x7f287c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:41.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.470+0000 7f289e7fc700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f287c0539b0 0x7f287c055e70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f289401a000 tx=0x7f2894014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:41.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.470+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2890042490 con 0x7f28a005d0e0 2022-07-20T03:14:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.602+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f28a00574e0 con 0x7f28a005d0e0 2022-07-20T03:14:41.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.610+0000 7f288ffff700 1 -- 172.21.15.61:0/3333517612 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v12)=0 v12) v1 ==== 122+0+0 (secure 0 0 0) 0x7f2890045ab0 con 0x7f28a005d0e0 2022-07-20T03:14:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f287c0539b0 msgr2=0x7f287c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f287c0539b0 0x7f287c055e70 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f289401a000 tx=0x7f2894014000).stop 2022-07-20T03:14:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 msgr2=0x7f28a0152660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:41.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 0x7f28a0152660 secure :-1 s=READY pgs=788 cs=0 l=1 rev1=1 rx=0x7f2890001110 tx=0x7f2890007ad0).stop 2022-07-20T03:14:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 shutdown_connections 2022-07-20T03:14:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f287c0539b0 0x7f287c055e70 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f28a0143060 0x7f28a00b2260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f28a00b5960 0x7f28a00b1d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 --2- 172.21.15.61:0/3333517612 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f28a005d0e0 0x7f28a0152660 unknown :-1 s=CLOSED pgs=788 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:41.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 >> 172.21.15.61:0/3333517612 conn(0x7f28a00b6a20 msgr2=0x7f28a009b8e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:41.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 shutdown_connections 2022-07-20T03:14:41.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:41.614+0000 7f28a56c8700 1 -- 172.21.15.61:0/3333517612 wait complete. 2022-07-20T03:14:41.681 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:14:41.681 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:41.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:41.745 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:41.778 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:41.802 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:41.824 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:14:41.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:41.825 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:41.825 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:41.833 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:14:41.833 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:41.834 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:14:41.846 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:41.847 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:41.847 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:41.867 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:41.867 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:14:41.868 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:14:41.868 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:41.869 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:41.869 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:14:41.911 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:41.912 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:14:41.912 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:14:41.913 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:14:42.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 Processor -- start 2022-07-20T03:14:42.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- start start 2022-07-20T03:14:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac05c3f0 0x7f69ac14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f69ac0acef0 0x7f69ac146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f69ac05d480 con 0x7f69ac0b1e00 2022-07-20T03:14:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f69ac0b21e0 con 0x7f69ac0acef0 2022-07-20T03:14:42.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac05cf40 con 0x7f69ac05c3f0 2022-07-20T03:14:42.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f69ac0acef0 0x7f69ac146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:42.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f69ac0acef0 0x7f69ac146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47234/0 (socket says 172.21.15.61:47234) 2022-07-20T03:14:42.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 -- 172.21.15.61:0/2382382025 learned_addr learned my addr 172.21.15.61:0/2382382025 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:42.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b1b7f700 1 --2- 172.21.15.61:0/2382382025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac05c3f0 0x7f69ac14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:42.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b137e700 1 -- 172.21.15.61:0/2382382025 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3982287638 0 0) 0x7f69ac05d480 con 0x7f69ac0b1e00 2022-07-20T03:14:42.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 -- 172.21.15.61:0/2382382025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac05c3f0 msgr2=0x7f69ac14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:42.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 --2- 172.21.15.61:0/2382382025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac05c3f0 0x7f69ac14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 -- 172.21.15.61:0/2382382025 >> v1:172.21.15.117:6789/0 conn(0x7f69ac0b1e00 legacy=0x7f69ac0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:42.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 -- 172.21.15.61:0/2382382025 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f69ac05d1e0 con 0x7f69ac0acef0 2022-07-20T03:14:42.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b2b81700 1 --2- 172.21.15.61:0/2382382025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0acef0 0x7f69ac146410 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f699c002f70 tx=0x7f699c006eb0).ready entity=mon.0 client_cookie=61f0282db3fc925b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:42.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b137e700 1 -- 172.21.15.61:0/2382382025 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c00d570 con 0x7f69ac0acef0 2022-07-20T03:14:42.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b137e700 1 -- 172.21.15.61:0/2382382025 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c000e90 con 0x7f69ac0acef0 2022-07-20T03:14:42.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b137e700 1 -- 172.21.15.61:0/2382382025 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c00d8c0 con 0x7f69ac0acef0 2022-07-20T03:14:42.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- 172.21.15.61:0/2382382025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0acef0 msgr2=0x7f69ac146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:42.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 --2- 172.21.15.61:0/2382382025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0acef0 0x7f69ac146410 secure :-1 s=READY pgs=789 cs=0 l=1 rev1=1 rx=0x7f699c002f70 tx=0x7f699c006eb0).stop 2022-07-20T03:14:42.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- 172.21.15.61:0/2382382025 shutdown_connections 2022-07-20T03:14:42.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 --2- 172.21.15.61:0/2382382025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0acef0 0x7f69ac146410 unknown :-1 s=CLOSED pgs=789 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 --2- 172.21.15.61:0/2382382025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac05c3f0 0x7f69ac14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.025+0000 7f69b45e4700 1 -- 172.21.15.61:0/2382382025 >> 172.21.15.61:0/2382382025 conn(0x7f69ac0611c0 msgr2=0x7f69ac062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:42.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.029+0000 7f69b45e4700 1 -- 172.21.15.61:0/2382382025 shutdown_connections 2022-07-20T03:14:42.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.029+0000 7f69b45e4700 1 -- 172.21.15.61:0/2382382025 wait complete. 2022-07-20T03:14:42.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.029+0000 7f69b45e4700 1 Processor -- start 2022-07-20T03:14:42.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.029+0000 7f69b45e4700 1 -- start start 2022-07-20T03:14:42.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69ac0acef0 0x7f69ac149ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:42.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac0b1e00 0x7f69ac14bf70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:42.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 0x7f69ac0a0b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:42.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac149cd0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac09e950 con 0x7f69ac0acef0 2022-07-20T03:14:42.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f69ac09e5f0 con 0x7f69ac0b1e00 2022-07-20T03:14:42.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 0x7f69ac0a0b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:42.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 0x7f69ac0a0b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47240/0 (socket says 172.21.15.61:47240) 2022-07-20T03:14:42.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 -- 172.21.15.61:0/1834219461 learned_addr learned my addr 172.21.15.61:0/1834219461 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:42.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2380700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69ac0acef0 0x7f69ac149ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:42.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b1b7f700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac0b1e00 0x7f69ac14bf70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:42.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 -- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac0b1e00 msgr2=0x7f69ac14bf70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:42.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac0b1e00 0x7f69ac14bf70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 -- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69ac0acef0 msgr2=0x7f69ac149ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:42.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69ac0acef0 0x7f69ac149ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 -- 172.21.15.61:0/1834219461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f699c00b1e0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b2b81700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 0x7f69ac0a0b20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f699c002f70 tx=0x7f699c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:42.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c011790 con 0x7f69ac0a06a0 2022-07-20T03:14:42.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f699c0118f0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f69ac09eb80 con 0x7f69ac0a06a0 2022-07-20T03:14:42.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f699c011c50 con 0x7f69ac0a06a0 2022-07-20T03:14:42.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.033+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69ac14cc10 con 0x7f69ac0a06a0 2022-07-20T03:14:42.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.037+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f69ac0b04b0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.037+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f699c025030 con 0x7f69ac0a06a0 2022-07-20T03:14:42.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.037+0000 7f69ab7fe700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f698c053ad0 0x7f698c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:42.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.037+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f699c06ced0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.037+0000 7f69b2380700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f698c053ad0 0x7f698c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:42.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.037+0000 7f69b2380700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f698c053ad0 0x7f698c055f90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f69a400d000 tx=0x7f69a4007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:42.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.041+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f699c043fd0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.169+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f69ac0574e0 con 0x7f69ac0a06a0 2022-07-20T03:14:42.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69ab7fe700 1 -- 172.21.15.61:0/1834219461 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v75) v1 ==== 81+0+28947 (secure 0 0 0) 0x7f699c023020 con 0x7f69ac0a06a0 2022-07-20T03:14:42.191 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:42.191 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:14:42.192 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 75, 2022-07-20T03:14:42.192 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:14:42.192 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:14:42.192 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:14:39.263462+0000", 2022-07-20T03:14:42.193 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:14:42.193 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:14:42.193 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:14:42.193 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:14:42.195 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:14:42.195 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:14:42.196 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:14:42.196 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:14:42.196 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:14:42.196 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:42.197 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:14:42.197 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:14:42.197 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:14:42.197 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:14:42.198 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:14:42.198 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 14, 2022-07-20T03:14:42.199 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:14:42.199 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:14:42.199 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:14:42.199 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:14:42.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:14:42.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:14:42.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:14:42.201 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:14:42.201 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:42.201 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:42.201 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:42.201 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:42.202 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:42.211 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:42.211 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:42.211 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:42.211 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:14:42.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:14:42.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:14:42.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:14:42.212 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:14:42.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:42.213 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:42.213 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:42.213 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:42.213 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:42.214 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:42.215 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:42.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.215 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:14:42.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:42.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:42.216 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:42.216 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:42.216 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:42.217 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:42.217 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:42.217 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:42.217 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:42.218 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:42.218 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:42.218 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:42.218 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:42.219 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:42.219 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:42.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:42.219 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:42.220 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:42.220 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:42.220 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:42.220 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:42.221 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:42.221 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:42.221 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:42.221 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:42.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:42.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.230 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:42.230 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:42.231 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:42.233 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:42.233 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:42.233 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:42.234 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:42.234 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:42.235 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:42.235 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:42.242 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:42.242 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:14:42.243 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:14:42.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.243 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:42.244 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:14:42.244 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.245 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:14:42.246 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:14:42.248 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:14:42.267 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:42.268 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:42.268 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:42.268 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:42.269 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:42.269 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:42.277 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:42.278 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:42.278 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:42.278 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:42.279 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:42.279 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:42.279 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:42.292 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:42.293 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:42.293 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:42.293 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:42.293 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:42.294 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:42.294 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:42.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:14:42.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:42.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:42.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:42.295 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:42.296 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:42.297 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:42.297 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:42.298 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:42.299 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:42.299 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:42.299 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:42.300 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:42.300 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:42.300 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:42.301 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:42.301 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:42.302 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:42.302 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:42.302 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:42.303 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:42.303 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:42.303 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:42.304 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:42.304 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:42.305 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:42.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:42.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.305 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:42.306 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:42.307 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:42.307 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:42.307 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:42.307 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:42.308 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:42.308 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:42.308 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:42.309 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:42.309 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:42.309 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:14:42.310 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:42.310 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:14:42.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.311 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:42.311 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:14:42.311 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.312 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:14:42.312 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:14:42.312 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.313 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.313 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.313 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.313 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.314 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.314 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.314 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:42.314 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.315 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.315 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.315 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.316 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.316 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:14:42.317 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.317 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.317 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.318 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.318 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:14:42.318 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.319 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.319 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.323 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.324 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.333 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.333 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:14:42.334 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.334 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.335 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.335 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:14:42.335 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.335 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.336 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.336 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.336 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.336 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.337 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:14:42.337 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.339 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.339 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.339 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.340 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:14:42.340 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.341 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.341 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.342 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.343 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.343 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.343 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:14:42.344 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.344 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.344 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:14:42.345 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:42.345 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.346 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.359 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:14:42.359 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:14:42.359 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:14:42.360 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:14:42.360 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.361 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.361 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.368 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.369 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.370 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:14:42.370 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:14:42.397 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.397 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.400 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.400 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.402 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.402 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.402 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.403 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 59, 2022-07-20T03:14:42.403 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.403 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.404 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.405 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.405 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:14:42.406 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.406 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.407 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.407 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.408 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:14:42.408 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.409 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.421 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.422 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.423 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.445 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.446 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:14:42.446 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.446 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.447 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.447 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:14:42.448 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.448 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.449 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.449 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.449 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.450 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.450 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.451 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:14:42.452 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.453 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.453 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:14:42.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.455 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.456 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.456 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.457 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.458 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.458 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:14:42.458 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.459 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.459 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.460 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.460 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:14:42.461 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:42.462 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.462 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.463 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.463 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:14:42.463 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:14:42.464 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:14:42.464 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:14:42.465 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.466 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.466 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.467 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.467 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.468 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.468 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:14:42.468 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:14:42.469 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.469 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.469 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.470 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.470 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.471 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.471 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.472 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:42.472 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.472 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.473 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.473 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.474 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.475 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:14:42.484 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.484 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.485 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.485 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.486 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:14:42.488 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.489 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.489 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.490 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.491 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.510 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.511 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.511 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.516 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:14:42.518 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.518 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.519 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.520 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.520 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:14:42.521 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.530 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.531 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.531 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.531 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.532 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.532 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.533 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:14:42.533 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.534 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.544 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.545 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.545 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:14:42.546 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.546 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.546 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.547 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.548 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.551 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.551 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:14:42.552 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.552 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.552 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.552 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.552 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:14:42.553 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:14:42.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.553 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.553 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.554 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:14:42.554 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:14:42.554 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:14:42.555 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:14:42.555 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.555 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.555 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.556 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.556 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.556 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.556 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:14:42.557 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:14:42.557 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.557 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.557 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.558 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.559 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.559 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.559 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.559 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:42.560 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.560 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.560 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.560 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.560 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.561 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.561 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:14:42.561 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.561 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.562 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.563 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.563 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:14:42.563 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.564 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.564 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.564 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.564 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.565 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.565 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.565 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.565 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:14:42.565 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.566 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.566 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.566 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.567 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:14:42.567 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.568 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.568 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.568 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.573 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.573 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.574 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.574 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.575 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:14:42.575 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.576 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.576 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.577 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.578 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:14:42.579 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.580 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.580 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.581 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.581 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.582 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.583 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.583 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.584 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:14:42.584 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.595 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.596 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.597 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:14:42.597 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:14:42.613 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.613 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.614 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.614 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:14:42.615 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:14:42.616 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:14:42.616 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:14:42.617 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.617 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.618 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.618 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.619 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.619 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.620 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:14:42.620 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:14:42.621 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.621 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.632 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.632 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.633 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.633 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.634 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.634 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-20T03:14:42.635 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.636 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.636 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.637 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.637 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.637 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.638 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:14:42.639 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.639 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.640 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.640 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.640 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:14:42.641 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.642 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.642 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.643 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.643 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.643 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.644 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.644 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.645 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:14:42.645 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.646 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.647 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.647 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.648 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:14:42.648 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.649 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.649 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.650 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.651 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.651 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.651 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.652 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.652 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:14:42.653 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.660 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.662 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.662 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.663 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:14:42.663 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.664 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.664 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.665 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.665 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.666 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.667 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.667 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.668 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:14:42.669 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.669 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.670 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.670 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.671 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:14:42.671 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:14:42.672 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.672 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.673 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.673 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:14:42.674 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:14:42.674 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:14:42.675 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:14:42.675 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.676 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.676 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.677 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.677 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.678 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.678 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:14:42.679 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:14:42.680 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.680 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.681 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.681 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.682 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.683 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.683 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.684 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:42.684 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.685 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.685 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.686 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.686 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.687 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.687 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:14:42.688 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.698 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.699 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.699 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.700 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:14:42.700 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.701 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.701 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.702 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.703 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.703 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.704 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.704 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.705 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:14:42.705 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.705 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.708 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.708 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:14:42.709 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.709 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.710 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.710 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.710 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.711 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.711 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.712 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.712 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:14:42.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.713 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.713 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.714 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.714 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:14:42.714 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.715 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.715 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.716 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.716 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.717 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.718 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.718 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.719 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:14:42.719 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.721 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.721 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.721 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:14:42.722 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:14:42.723 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.723 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.723 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.723 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:14:42.724 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:14:42.724 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:14:42.724 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:14:42.724 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.725 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.725 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.725 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.725 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.726 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.726 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:14:42.726 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:14:42.727 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.728 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.728 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.728 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.728 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.729 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.729 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.729 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 69, 2022-07-20T03:14:42.729 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.730 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.730 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.730 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.730 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.731 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.731 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:14:42.731 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.731 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.732 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.732 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.732 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:14:42.733 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.733 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.733 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.733 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.734 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.734 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.734 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.734 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.734 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:14:42.735 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.735 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.735 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.736 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.736 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:14:42.737 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.737 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.741 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.741 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.741 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.742 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.742 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.743 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.743 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:14:42.743 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.744 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.744 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.744 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.745 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:14:42.745 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.747 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.747 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.748 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.748 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.749 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.750 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.750 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.751 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:14:42.751 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.752 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.753 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.753 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:14:42.761 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:14:42.761 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.763 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.770 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.773 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:14:42.773 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:14:42.773 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:14:42.774 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:14:42.774 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.775 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.775 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.775 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.776 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.776 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.776 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:14:42.777 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:14:42.777 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:42.778 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:42.778 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:42.779 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:42.779 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:42.780 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:42.780 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:42.781 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-20T03:14:42.781 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:42.782 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:42.782 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:42.782 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.784 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.784 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.785 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:14:42.785 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.786 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.787 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.787 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:14:42.788 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.788 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.789 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.789 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.790 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:42.801 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.801 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.801 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.802 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:14:42.802 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.802 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.802 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.803 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.803 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:14:42.803 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.803 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.804 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.804 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.805 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:42.805 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.805 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.806 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.806 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:14:42.806 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.806 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.807 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.807 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.808 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:14:42.808 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.808 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.808 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.809 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.809 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:42.809 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:42.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.810 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:42.810 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:14:42.810 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.811 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.811 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.811 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:42.811 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:14:42.811 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:14:42.812 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.812 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.812 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.812 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:14:42.813 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:14:42.813 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:14:42.813 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:14:42.813 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:42.814 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:42.814 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:42.814 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:42.814 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.821 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:42.821 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:14:42.821 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.822 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:14:42.822 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.822 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.823 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.823 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.823 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.824 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:14:42.824 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.824 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.825 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.825 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:14:42.825 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.825 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.825 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.827 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.827 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.828 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:14:42.828 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.828 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.829 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.829 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:14:42.829 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.830 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.830 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.830 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.831 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.831 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:14:42.831 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.832 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.832 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.832 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:14:42.832 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.832 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.833 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.833 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.833 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.833 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:14:42.834 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.834 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.834 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.834 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:14:42.835 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.835 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.835 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.835 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:14:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.836 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.837 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.838 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:14:42.838 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.838 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.839 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.839 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.839 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.839 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:14:42.840 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.840 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.840 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.840 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:14:42.841 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.841 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.841 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.841 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.842 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.842 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:14:42.842 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.842 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.843 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:42.843 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:14:42.843 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:14:42.843 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:14:42.845 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:14:42.845 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:14:42.845 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:14:42.846 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:14:42.846 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:14:42.846 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.846 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:42.847 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:14:42.847 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:14:42.848 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:14:42.848 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:14:42.848 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:14:42.848 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/460461689": "2022-07-21T03:14:35.145105+0000", 2022-07-20T03:14:42.849 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/460461689": "2022-07-21T03:14:35.145105+0000", 2022-07-20T03:14:42.849 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.124:0/2780278373": "2022-07-20T04:14:22.364604+0000" 2022-07-20T03:14:42.850 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.850 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:14:42.858 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:14:42.864 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:14:42.865 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:14:42.865 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:14:42.865 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:14:42.865 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:14:42.866 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:14:42.866 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:42.866 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:42.866 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:14:42.867 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:14:42.867 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:14:42.867 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:14:42.867 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:14:42.868 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:14:42.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f698c053ad0 msgr2=0x7f698c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:42.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f698c053ad0 0x7f698c055f90 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f69a400d000 tx=0x7f69a4007f10).stop 2022-07-20T03:14:42.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 msgr2=0x7f69ac0a0b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:42.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 0x7f69ac0a0b20 secure :-1 s=READY pgs=790 cs=0 l=1 rev1=1 rx=0x7f699c002f70 tx=0x7f699c015000).stop 2022-07-20T03:14:42.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 shutdown_connections 2022-07-20T03:14:42.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f698c053ad0 0x7f698c055f90 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f69ac0a06a0 0x7f69ac0a0b20 unknown :-1 s=CLOSED pgs=790 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f69ac0b1e00 0x7f69ac14bf70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 --2- 172.21.15.61:0/1834219461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f69ac0acef0 0x7f69ac149ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:42.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 >> 172.21.15.61:0/1834219461 conn(0x7f69ac05c3f0 msgr2=0x7f69ac069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:42.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 shutdown_connections 2022-07-20T03:14:42.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.173+0000 7f69b45e4700 1 -- 172.21.15.61:0/1834219461 wait complete. 2022-07-20T03:14:42.873 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:43.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.981+0000 7f9fbcee2700 1 Processor -- start 2022-07-20T03:14:43.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.981+0000 7f9fbcee2700 1 -- start start 2022-07-20T03:14:43.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fbcee2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9fb80b2530 0x7f9fb80abef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fbcee2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 0x7f9fb8141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fbcee2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fb80b2910 con 0x7f9fb80620e0 2022-07-20T03:14:43.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fbcee2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fb805fe70 con 0x7f9fb80b2530 2022-07-20T03:14:43.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fbcee2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb8060110 con 0x7f9fb8146ed0 2022-07-20T03:14:43.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 0x7f9fb8141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb659c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9fb80b2530 0x7f9fb80abef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 0x7f9fb8141fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:33920/0 (socket says 172.21.15.61:33920) 2022-07-20T03:14:43.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1213093851 learned_addr learned my addr 172.21.15.61:0/1213093851 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:43.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1213093851 >> v2:172.21.15.117:3300/0 conn(0x7f9fb80b2530 msgr2=0x7f9fb80abef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:43.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 --2- 172.21.15.61:0/1213093851 >> v2:172.21.15.117:3300/0 conn(0x7f9fb80b2530 0x7f9fb80abef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1213093851 >> v1:172.21.15.117:6789/0 conn(0x7f9fb80620e0 legacy=0x7f9fb81467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1213093851 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb8146320 con 0x7f9fb8146ed0 2022-07-20T03:14:43.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb659c700 1 --2- 172.21.15.61:0/1213093851 >> v2:172.21.15.117:3300/0 conn(0x7f9fb80b2530 0x7f9fb80abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:43.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb6d9d700 1 --2- 172.21.15.61:0/1213093851 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 0x7f9fb8141fc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9fb00032f0 tx=0x7f9fb0009290).ready entity=mon.2 client_cookie=2fcfd7a52d6e7909 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb5d9b700 1 -- 172.21.15.61:0/1213093851 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb000b8a0 con 0x7f9fb8146ed0 2022-07-20T03:14:43.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb5d9b700 1 -- 172.21.15.61:0/1213093851 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb000ba00 con 0x7f9fb8146ed0 2022-07-20T03:14:43.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.985+0000 7f9fb5d9b700 1 -- 172.21.15.61:0/1213093851 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb000bb70 con 0x7f9fb8146ed0 2022-07-20T03:14:43.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1213093851 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 msgr2=0x7f9fb8141fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1213093851 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 0x7f9fb8141fc0 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f9fb00032f0 tx=0x7f9fb0009290).stop 2022-07-20T03:14:43.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1213093851 shutdown_connections 2022-07-20T03:14:43.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1213093851 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb8146ed0 0x7f9fb8141fc0 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1213093851 >> v2:172.21.15.117:3300/0 conn(0x7f9fb80b2530 0x7f9fb80abef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1213093851 >> 172.21.15.61:0/1213093851 conn(0x7f9fb80b3690 msgr2=0x7f9fb80b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:43.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1213093851 shutdown_connections 2022-07-20T03:14:43.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1213093851 wait complete. 2022-07-20T03:14:43.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 Processor -- start 2022-07-20T03:14:43.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.989+0000 7f9fbcee2700 1 -- start start 2022-07-20T03:14:43.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fb80620e0 0x7f9fb81429b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb80b2530 0x7f9fb8138280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 0x7f9fb8138770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb809b8f0 con 0x7f9fb8146ed0 2022-07-20T03:14:43.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb809bb90 con 0x7f9fb80620e0 2022-07-20T03:14:43.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fb809c030 con 0x7f9fb80b2530 2022-07-20T03:14:43.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 0x7f9fb8138770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 0x7f9fb8138770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47254/0 (socket says 172.21.15.61:47254) 2022-07-20T03:14:43.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1297158932 learned_addr learned my addr 172.21.15.61:0/1297158932 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:43.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9faffff700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb80b2530 0x7f9fb8138280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb659c700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fb80620e0 0x7f9fb81429b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb80b2530 msgr2=0x7f9fb8138280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb80b2530 0x7f9fb8138280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fb80620e0 msgr2=0x7f9fb81429b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fb80620e0 0x7f9fb81429b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 -- 172.21.15.61:0/1297158932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fa800b180 con 0x7f9fb8146ed0 2022-07-20T03:14:43.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fb6d9d700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 0x7f9fb8138770 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f9fb00032f0 tx=0x7f9fb0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb000b8a0 con 0x7f9fb8146ed0 2022-07-20T03:14:43.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb0021440 con 0x7f9fb8146ed0 2022-07-20T03:14:43.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fb0009530 con 0x7f9fb8146ed0 2022-07-20T03:14:43.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb00217a0 con 0x7f9fb8146ed0 2022-07-20T03:14:43.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fb809ad70 con 0x7f9fb8146ed0 2022-07-20T03:14:43.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.993+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fb805faf0 con 0x7f9fb8146ed0 2022-07-20T03:14:43.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.997+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9fb000f070 con 0x7f9fb8146ed0 2022-07-20T03:14:43.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.997+0000 7f9faf7fe700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053a40 0x7f9f9c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.997+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7f9fb006cf70 con 0x7f9fb8146ed0 2022-07-20T03:14:43.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.997+0000 7f9fb659c700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053a40 0x7f9f9c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:42.997+0000 7f9fb659c700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053a40 0x7f9f9c055f00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f9fa800b000 tx=0x7f9fa8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.001+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9fb0044200 con 0x7f9fb8146ed0 2022-07-20T03:14:43.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9fb809af90 con 0x7f9fb8146ed0 2022-07-20T03:14:43.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9faf7fe700 1 -- 172.21.15.61:0/1297158932 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f9fb0015070 con 0x7f9fb8146ed0 2022-07-20T03:14:43.168 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:43.169 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38},{"gid":6585,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2463303201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2463303201},{"type":"v1","addr":"172.21.15.61:6833","nonce":2463303201}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":42}],"filesystems":[]} 2022-07-20T03:14:43.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053a40 msgr2=0x7f9f9c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053a40 0x7f9f9c055f00 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f9fa800b000 tx=0x7f9fa8006df0).stop 2022-07-20T03:14:43.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 msgr2=0x7f9fb8138770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 0x7f9fb8138770 secure :-1 s=READY pgs=791 cs=0 l=1 rev1=1 rx=0x7f9fb00032f0 tx=0x7f9fb0016000).stop 2022-07-20T03:14:43.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 shutdown_connections 2022-07-20T03:14:43.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053a40 0x7f9f9c055f00 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fb8146ed0 0x7f9fb8138770 unknown :-1 s=CLOSED pgs=791 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fb80b2530 0x7f9fb8138280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 --2- 172.21.15.61:0/1297158932 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fb80620e0 0x7f9fb81429b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 >> 172.21.15.61:0/1297158932 conn(0x7f9fb80b3690 msgr2=0x7f9fb81454f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:43.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 shutdown_connections 2022-07-20T03:14:43.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.165+0000 7f9fbcee2700 1 -- 172.21.15.61:0/1297158932 wait complete. 2022-07-20T03:14:43.177 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 42 2022-07-20T03:14:43.185 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:43.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.289+0000 7fef18c68700 1 Processor -- start 2022-07-20T03:14:43.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.289+0000 7fef18c68700 1 -- start start 2022-07-20T03:14:43.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fef1413c790 0x7fef140b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef140b9a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fef14142ba0 con 0x7fef140b9f00 2022-07-20T03:14:43.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fef141429b0 con 0x7fef1413c790 2022-07-20T03:14:43.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef14146330 con 0x7fef140b3ca0 2022-07-20T03:14:43.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef13fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fef140b9f00 0x7fef14143000 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43860/0 (socket says 172.21.15.61:43860) 2022-07-20T03:14:43.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef13fff700 1 -- 172.21.15.61:0/2630029422 learned_addr learned my addr 172.21.15.61:0/2630029422 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:43.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef137fe700 1 --2- 172.21.15.61:0/2630029422 >> v2:172.21.15.61:3300/0 conn(0x7fef1413c790 0x7fef140b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef0affd700 1 --2- 172.21.15.61:0/2630029422 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef140b9a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef12ffd700 1 -- 172.21.15.61:0/2630029422 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 787092892 0 0) 0x7fef14142ba0 con 0x7fef140b9f00 2022-07-20T03:14:43.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef12ffd700 1 -- 172.21.15.61:0/2630029422 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feef4003260 con 0x7fef140b9f00 2022-07-20T03:14:43.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef12ffd700 1 -- 172.21.15.61:0/2630029422 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 36988750 0 0) 0x7feef4003260 con 0x7fef140b9f00 2022-07-20T03:14:43.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef137fe700 1 -- 172.21.15.61:0/2630029422 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 msgr2=0x7fef140b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:43.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef137fe700 1 --2- 172.21.15.61:0/2630029422 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef140b9a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef137fe700 1 -- 172.21.15.61:0/2630029422 >> v1:172.21.15.61:6789/0 conn(0x7fef140b9f00 legacy=0x7fef14143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef137fe700 1 -- 172.21.15.61:0/2630029422 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef140b4c00 con 0x7fef1413c790 2022-07-20T03:14:43.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef137fe700 1 --2- 172.21.15.61:0/2630029422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef1413c790 0x7fef140b4710 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fef0c001b80 tx=0x7fef04006df0).ready entity=mon.0 client_cookie=e76aedc29c97013c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef12ffd700 1 -- 172.21.15.61:0/2630029422 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef0c0019e0 con 0x7fef1413c790 2022-07-20T03:14:43.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef12ffd700 1 -- 172.21.15.61:0/2630029422 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fef04000e90 con 0x7fef1413c790 2022-07-20T03:14:43.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef12ffd700 1 -- 172.21.15.61:0/2630029422 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fef04012460 con 0x7fef1413c790 2022-07-20T03:14:43.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- 172.21.15.61:0/2630029422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef1413c790 msgr2=0x7fef140b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 --2- 172.21.15.61:0/2630029422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef1413c790 0x7fef140b4710 secure :-1 s=READY pgs=793 cs=0 l=1 rev1=1 rx=0x7fef0c001b80 tx=0x7fef04006df0).stop 2022-07-20T03:14:43.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- 172.21.15.61:0/2630029422 shutdown_connections 2022-07-20T03:14:43.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 --2- 172.21.15.61:0/2630029422 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef140b9a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 --2- 172.21.15.61:0/2630029422 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef1413c790 0x7fef140b4710 unknown :-1 s=CLOSED pgs=793 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- 172.21.15.61:0/2630029422 >> 172.21.15.61:0/2630029422 conn(0x7fef140678b0 msgr2=0x7fef140b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:43.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- 172.21.15.61:0/2630029422 shutdown_connections 2022-07-20T03:14:43.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.293+0000 7fef18c68700 1 -- 172.21.15.61:0/2630029422 wait complete. 2022-07-20T03:14:43.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 Processor -- start 2022-07-20T03:14:43.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 -- start start 2022-07-20T03:14:43.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef14152890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 0x7fef140b1e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef1413c790 0x7fef140af400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef140b10d0 con 0x7fef140b9f00 2022-07-20T03:14:43.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef140ae4b0 con 0x7fef1413c790 2022-07-20T03:14:43.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fef141530d0 con 0x7fef140b3ca0 2022-07-20T03:14:43.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 0x7fef140b1e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 0x7fef140b1e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47266/0 (socket says 172.21.15.61:47266) 2022-07-20T03:14:43.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef137fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef14152890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 -- 172.21.15.61:0/1129385375 learned_addr learned my addr 172.21.15.61:0/1129385375 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef13fff700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef1413c790 0x7fef140af400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 -- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 msgr2=0x7fef14152890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef14152890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 -- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef1413c790 msgr2=0x7fef140af400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef1413c790 0x7fef140af400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 -- 172.21.15.61:0/1129385375 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef0400b180 con 0x7fef140b9f00 2022-07-20T03:14:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef0affd700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 0x7fef140b1e70 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fef14153210 tx=0x7feefc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc010780 con 0x7fef140b9f00 2022-07-20T03:14:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feefc0108e0 con 0x7fef140b9f00 2022-07-20T03:14:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef14152d80 con 0x7fef140b9f00 2022-07-20T03:14:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feefc010c40 con 0x7fef140b9f00 2022-07-20T03:14:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.297+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef1413faa0 con 0x7fef140b9f00 2022-07-20T03:14:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.301+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feefc01c460 con 0x7fef140b9f00 2022-07-20T03:14:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.301+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef140b2d10 con 0x7fef140b9f00 2022-07-20T03:14:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.301+0000 7fef117fa700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feef8053ad0 0x7feef8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.301+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(75..75 src has 1..75) v4 ==== 5382+0+0 (secure 0 0 0) 0x7feefc06c0a0 con 0x7fef140b9f00 2022-07-20T03:14:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.301+0000 7fef137fe700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feef8053ad0 0x7feef8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.301+0000 7fef137fe700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feef8053ad0 0x7feef8055f90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fef0401a000 tx=0x7fef04014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.305+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feefc043140 con 0x7fef140b9f00 2022-07-20T03:14:43.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fef14146090 con 0x7fef140b9f00 2022-07-20T03:14:43.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef117fa700 1 -- 172.21.15.61:0/1129385375 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 42 v42) v1 ==== 94+0+1209 (secure 0 0 0) 0x7feefc046760 con 0x7fef140b9f00 2022-07-20T03:14:43.478 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:43.478 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":42,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5660,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/896475791","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":896475791},{"type":"v1","addr":"172.21.15.117:6835","nonce":896475791}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":38},{"gid":6585,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/2463303201","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2463303201},{"type":"v1","addr":"172.21.15.61:6833","nonce":2463303201}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":42}],"filesystems":[]} 2022-07-20T03:14:43.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feef8053ad0 msgr2=0x7feef8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feef8053ad0 0x7feef8055f90 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7fef0401a000 tx=0x7fef04014000).stop 2022-07-20T03:14:43.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 msgr2=0x7fef140b1e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 0x7fef140b1e70 secure :-1 s=READY pgs=794 cs=0 l=1 rev1=1 rx=0x7fef14153210 tx=0x7feefc006df0).stop 2022-07-20T03:14:43.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 shutdown_connections 2022-07-20T03:14:43.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feef8053ad0 0x7feef8055f90 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fef1413c790 0x7fef140af400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fef140b9f00 0x7fef140b1e70 unknown :-1 s=CLOSED pgs=794 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 --2- 172.21.15.61:0/1129385375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fef140b3ca0 0x7fef14152890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 >> 172.21.15.61:0/1129385375 conn(0x7fef140678b0 msgr2=0x7fef14142f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:43.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 shutdown_connections 2022-07-20T03:14:43.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.469+0000 7fef18c68700 1 -- 172.21.15.61:0/1129385375 wait complete. 2022-07-20T03:14:43.505 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 42 2022-07-20T03:14:43.509 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:14:43.512 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:14:43.513 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:14:43.513 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:14:43.514 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:14:43.556 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:14:43.557 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:14:43.557 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:14:43.559 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:14:43.559 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:14:43.565 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:14:43.565 DEBUG:teuthology.parallel:result is None 2022-07-20T03:14:43.582 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:14:43.583 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:14:43.600 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:14:43.601 DEBUG:teuthology.parallel:result is None 2022-07-20T03:14:43.601 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:14:43.635 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:14:43.676 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:14:43.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.701+0000 7fbdd4fdc700 1 Processor -- start 2022-07-20T03:14:43.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- start start 2022-07-20T03:14:43.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00b26c0 0x7fbdd013be40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00b0f90 con 0x7fbdd00ad220 2022-07-20T03:14:43.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbdd00b2aa0 con 0x7fbdd00aeb50 2022-07-20T03:14:43.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd00b0620 con 0x7fbdd00b26c0 2022-07-20T03:14:43.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcdd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbdd00ad220 0x7fbdd013c330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43884/0 (socket says 172.21.15.61:43884) 2022-07-20T03:14:43.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcdd9b700 1 -- 172.21.15.61:0/2464910281 learned_addr learned my addr 172.21.15.61:0/2464910281 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:43.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdce59c700 1 --2- 172.21.15.61:0/2464910281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00b26c0 0x7fbdd013be40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcd59a700 1 -- 172.21.15.61:0/2464910281 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3375400490 0 0) 0x7fbdd00b0f90 con 0x7fbdd00ad220 2022-07-20T03:14:43.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcd59a700 1 -- 172.21.15.61:0/2464910281 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdb0003260 con 0x7fbdd00ad220 2022-07-20T03:14:43.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcd59a700 1 -- 172.21.15.61:0/2464910281 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2427449949 0 0) 0x7fbdd00b2aa0 con 0x7fbdd00aeb50 2022-07-20T03:14:43.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdce59c700 1 -- 172.21.15.61:0/2464910281 >> v1:172.21.15.117:6789/0 conn(0x7fbdd00aeb50 legacy=0x7fbdd00aefc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdce59c700 1 -- 172.21.15.61:0/2464910281 >> v1:172.21.15.61:6789/0 conn(0x7fbdd00ad220 legacy=0x7fbdd013c330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdce59c700 1 -- 172.21.15.61:0/2464910281 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd00b08c0 con 0x7fbdd00b26c0 2022-07-20T03:14:43.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdce59c700 1 --2- 172.21.15.61:0/2464910281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00b26c0 0x7fbdd013be40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbdc00032f0 tx=0x7fbdc000a370).ready entity=mon.2 client_cookie=5bd0def96c7b8566 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcd59a700 1 -- 172.21.15.61:0/2464910281 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc000cba0 con 0x7fbdd00b26c0 2022-07-20T03:14:43.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcd59a700 1 -- 172.21.15.61:0/2464910281 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc000cd00 con 0x7fbdd00b26c0 2022-07-20T03:14:43.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdcd59a700 1 -- 172.21.15.61:0/2464910281 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc0010600 con 0x7fbdd00b26c0 2022-07-20T03:14:43.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/2464910281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00b26c0 msgr2=0x7fbdd013be40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/2464910281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00b26c0 0x7fbdd013be40 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7fbdc00032f0 tx=0x7fbdc000a370).stop 2022-07-20T03:14:43.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/2464910281 shutdown_connections 2022-07-20T03:14:43.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/2464910281 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00b26c0 0x7fbdd013be40 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/2464910281 >> 172.21.15.61:0/2464910281 conn(0x7fbdd0062f30 msgr2=0x7fbdd0063340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:43.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/2464910281 shutdown_connections 2022-07-20T03:14:43.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.705+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/2464910281 wait complete. 2022-07-20T03:14:43.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 Processor -- start 2022-07-20T03:14:43.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 -- start start 2022-07-20T03:14:43.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00ad220 0x7fbdd00677d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00aeb50 0x7fbdd01510b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 0x7fbdd01515a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd014de60 con 0x7fbdd00b26c0 2022-07-20T03:14:43.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd013bf10 con 0x7fbdd00aeb50 2022-07-20T03:14:43.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdd013c1b0 con 0x7fbdd00ad220 2022-07-20T03:14:43.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 0x7fbdd01515a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 0x7fbdd01515a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47294/0 (socket says 172.21.15.61:47294) 2022-07-20T03:14:43.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 -- 172.21.15.61:0/3327636235 learned_addr learned my addr 172.21.15.61:0/3327636235 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:43.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdcdd9b700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00aeb50 0x7fbdd01510b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdce59c700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00ad220 0x7fbdd00677d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 -- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00ad220 msgr2=0x7fbdd00677d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00ad220 0x7fbdd00677d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 -- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00aeb50 msgr2=0x7fbdd01510b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:43.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00aeb50 0x7fbdd01510b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:43.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 -- 172.21.15.61:0/3327636235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc000a610 con 0x7fbdd00b26c0 2022-07-20T03:14:43.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdced9d700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 0x7fbdd01515a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fbdc8000bc0 tx=0x7fbdc80079e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc800ce00 con 0x7fbdd00b26c0 2022-07-20T03:14:43.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdc8012400 con 0x7fbdd00b26c0 2022-07-20T03:14:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdd013de00 con 0x7fbdd00b26c0 2022-07-20T03:14:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdc8012760 con 0x7fbdd00b26c0 2022-07-20T03:14:43.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.709+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdd013e870 con 0x7fbdd00b26c0 2022-07-20T03:14:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.713+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdd014dcb0 con 0x7fbdd00b26c0 2022-07-20T03:14:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.713+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbdc8019030 con 0x7fbdd00b26c0 2022-07-20T03:14:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.713+0000 7fbdc77fe700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053a60 0x7fbdb4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:43.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.713+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(76..76 src has 1..76) v4 ==== 5382+0+0 (secure 0 0 0) 0x7fbdc806ae50 con 0x7fbdd00b26c0 2022-07-20T03:14:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.713+0000 7fbdce59c700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053a60 0x7fbdb4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.713+0000 7fbdce59c700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053a60 0x7fbdb4055f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fbdc0019000 tx=0x7fbdc000a370).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:43.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.717+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbdc8042060 con 0x7fbdd00b26c0 2022-07-20T03:14:43.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:43.857+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fbdd013c370 con 0x7fbdd00b26c0 2022-07-20T03:14:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.697+0000 7fbdc77fe700 1 -- 172.21.15.61:0/3327636235 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v77) v1 ==== 97+0+0 (secure 0 0 0) 0x7fbdc8045680 con 0x7fbdd00b26c0 2022-07-20T03:14:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053a60 msgr2=0x7fbdb4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053a60 0x7fbdb4055f20 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fbdc0019000 tx=0x7fbdc000a370).stop 2022-07-20T03:14:44.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 msgr2=0x7fbdd01515a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 0x7fbdd01515a0 secure :-1 s=READY pgs=797 cs=0 l=1 rev1=1 rx=0x7fbdc8000bc0 tx=0x7fbdc80079e0).stop 2022-07-20T03:14:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 shutdown_connections 2022-07-20T03:14:44.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053a60 0x7fbdb4055f20 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdd00aeb50 0x7fbdd01510b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdd00b26c0 0x7fbdd01515a0 unknown :-1 s=CLOSED pgs=797 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 --2- 172.21.15.61:0/3327636235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdd00ad220 0x7fbdd00677d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 >> 172.21.15.61:0/3327636235 conn(0x7fbdd0062f30 msgr2=0x7fbdd00b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:44.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 shutdown_connections 2022-07-20T03:14:44.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.701+0000 7fbdd4fdc700 1 -- 172.21.15.61:0/3327636235 wait complete. 2022-07-20T03:14:44.718 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:14:44.720 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:14:44.723 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:14:44.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 Processor -- start 2022-07-20T03:14:44.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 -- start start 2022-07-20T03:14:44.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f800b1ef0 0x7f8f80062040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:44.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8f80061760 0x7f8f800ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:44.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f800b22d0 con 0x7f8f80098ae0 2022-07-20T03:14:44.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800abe50 con 0x7f8f800b1ef0 2022-07-20T03:14:44.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f800ac0f0 con 0x7f8f80061760 2022-07-20T03:14:44.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f800b1ef0 0x7f8f80062040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:44.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8f800b1ef0 0x7f8f80062040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47298/0 (socket says 172.21.15.61:47298) 2022-07-20T03:14:44.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 -- 172.21.15.61:0/464522377 learned_addr learned my addr 172.21.15.61:0/464522377 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:44.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f849f4700 1 --2- 172.21.15.61:0/464522377 >> v2:172.21.15.117:3300/0 conn(0x7f8f80061760 0x7f8f800ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:44.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 -- 172.21.15.61:0/464522377 >> v2:172.21.15.117:3300/0 conn(0x7f8f80061760 msgr2=0x7f8f800ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:44.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 --2- 172.21.15.61:0/464522377 >> v2:172.21.15.117:3300/0 conn(0x7f8f80061760 0x7f8f800ac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 -- 172.21.15.61:0/464522377 >> v1:172.21.15.117:6789/0 conn(0x7f8f80098ae0 legacy=0x7f8f80061050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:44.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 -- 172.21.15.61:0/464522377 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f800ac390 con 0x7f8f800b1ef0 2022-07-20T03:14:44.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7ffff700 1 --2- 172.21.15.61:0/464522377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f800b1ef0 0x7f8f80062040 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f8f70002f70 tx=0x7f8f70006eb0).ready entity=mon.0 client_cookie=804eac1558ce8aa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:44.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7effd700 1 -- 172.21.15.61:0/464522377 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f7000d610 con 0x7f8f800b1ef0 2022-07-20T03:14:44.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7effd700 1 -- 172.21.15.61:0/464522377 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f70000e90 con 0x7f8f800b1ef0 2022-07-20T03:14:44.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f7effd700 1 -- 172.21.15.61:0/464522377 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f7000d9a0 con 0x7f8f800b1ef0 2022-07-20T03:14:44.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 -- 172.21.15.61:0/464522377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f800b1ef0 msgr2=0x7f8f80062040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:44.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.869+0000 7f8f851f5700 1 --2- 172.21.15.61:0/464522377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f800b1ef0 0x7f8f80062040 secure :-1 s=READY pgs=798 cs=0 l=1 rev1=1 rx=0x7f8f70002f70 tx=0x7f8f70006eb0).stop 2022-07-20T03:14:44.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- 172.21.15.61:0/464522377 shutdown_connections 2022-07-20T03:14:44.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 --2- 172.21.15.61:0/464522377 >> v2:172.21.15.117:3300/0 conn(0x7f8f80061760 0x7f8f800ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 --2- 172.21.15.61:0/464522377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f800b1ef0 0x7f8f80062040 unknown :-1 s=CLOSED pgs=798 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- 172.21.15.61:0/464522377 >> 172.21.15.61:0/464522377 conn(0x7f8f800b2ca0 msgr2=0x7f8f800b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:44.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- 172.21.15.61:0/464522377 shutdown_connections 2022-07-20T03:14:44.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- 172.21.15.61:0/464522377 wait complete. 2022-07-20T03:14:44.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 Processor -- start 2022-07-20T03:14:44.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- start start 2022-07-20T03:14:44.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 0x7f8f80159e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:44.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f80098ae0 0x7f8f80143f50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:44.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f800b1ef0 0x7f8f80066f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:44.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f800aec10 con 0x7f8f80061760 2022-07-20T03:14:44.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f80149e50 con 0x7f8f800b1ef0 2022-07-20T03:14:44.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f851f5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f8014a080 con 0x7f8f80098ae0 2022-07-20T03:14:44.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 0x7f8f80159e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:44.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 0x7f8f80159e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47302/0 (socket says 172.21.15.61:47302) 2022-07-20T03:14:44.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 -- 172.21.15.61:0/1731421060 learned_addr learned my addr 172.21.15.61:0/1731421060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:44.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f849f4700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f800b1ef0 0x7f8f80066f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:44.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 -- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f80098ae0 msgr2=0x7f8f80143f50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:44.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7f7fe700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f80098ae0 0x7f8f80143f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:44.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f80098ae0 0x7f8f80143f50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 -- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f800b1ef0 msgr2=0x7f8f80066f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:44.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f800b1ef0 0x7f8f80066f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:44.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 -- 172.21.15.61:0/1731421060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f7000b1e0 con 0x7f8f80061760 2022-07-20T03:14:44.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f849f4700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f800b1ef0 0x7f8f80066f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:44.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7f7fe700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f80098ae0 0x7f8f80143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:44.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.873+0000 7f8f7ffff700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 0x7f8f80159e40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f8f70002f70 tx=0x7f8f70015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:44.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f70011790 con 0x7f8f80061760 2022-07-20T03:14:44.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f700118f0 con 0x7f8f80061760 2022-07-20T03:14:44.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f70011c50 con 0x7f8f80061760 2022-07-20T03:14:44.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8014a320 con 0x7f8f80061760 2022-07-20T03:14:44.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f800b03a0 con 0x7f8f80061760 2022-07-20T03:14:44.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f800ad6b0 con 0x7f8f80061760 2022-07-20T03:14:44.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f7001a040 con 0x7f8f80061760 2022-07-20T03:14:44.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7d7fa700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f580539b0 0x7f8f58055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:44.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8f7006c980 con 0x7f8f80061760 2022-07-20T03:14:44.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7f7fe700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f580539b0 0x7f8f58055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:44.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.877+0000 7f8f7f7fe700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f580539b0 0x7f8f58055e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f8f68008000 tx=0x7f8f68009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:44.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:44.885+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f800ad6b0 con 0x7f8f80061760 2022-07-20T03:14:45.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f8f800ad6b0 con 0x7f8f80061760 2022-07-20T03:14:45.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f7d7fa700 1 -- 172.21.15.61:0/1731421060 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v27) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f8f800ad6b0 con 0x7f8f80061760 2022-07-20T03:14:45.056 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:45.056 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:14:45.056 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:14:45.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.057 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:14:45.057 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:14:45.058 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.058 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:14:45.059 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.059 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.059 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.061 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:14:45.062 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:14:45.062 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.063 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:14:45.063 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.064 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.068 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.068 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.069 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.078 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:14:45.078 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:14:45.080 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.081 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.082 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.082 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.083 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.087 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:14:45.088 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:14:45.088 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.088 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.088 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.089 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.089 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.091 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:14:45.091 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:14:45.091 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.092 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.092 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.092 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.092 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.093 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:14:45.096 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:14:45.096 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.096 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.097 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.097 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.097 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.097 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.098 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.098 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:14:45.098 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:14:45.098 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.099 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.099 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.099 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.099 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.100 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.100 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:14:45.100 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:14:45.101 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.101 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.101 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.101 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.102 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.102 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.103 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.103 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:14:45.103 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:14:45.103 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.104 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.104 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.104 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.104 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.107 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.108 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:14:45.108 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:14:45.108 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.109 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.114 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.114 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.114 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.115 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.115 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.115 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:14:45.115 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:14:45.116 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.116 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:14:45.116 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:14:45.116 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:14:45.116 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.117 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:14:45.117 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:14:45.118 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.118 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:14:45.118 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:14:45.118 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:14:45.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.119 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:14:45.119 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:14:45.120 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.120 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:14:45.120 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:14:45.120 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:14:45.121 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.130 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.131 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:14:45.131 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:14:45.132 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.132 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:14:45.132 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:14:45.133 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:14:45.133 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.142 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.142 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:14:45.143 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:14:45.143 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.144 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:14:45.144 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:14:45.145 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.145 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:45.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:45.146 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:14:45.147 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:14:45.147 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:14:45.148 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:14:45.148 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:14:45.149 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:14:45.149 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.150 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:45.150 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:45.151 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:14:45.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f580539b0 msgr2=0x7f8f58055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f580539b0 0x7f8f58055e70 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f8f68008000 tx=0x7f8f68009040).stop 2022-07-20T03:14:45.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 msgr2=0x7f8f80159e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 0x7f8f80159e40 secure :-1 s=READY pgs=799 cs=0 l=1 rev1=1 rx=0x7f8f70002f70 tx=0x7f8f70015000).stop 2022-07-20T03:14:45.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 shutdown_connections 2022-07-20T03:14:45.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f580539b0 0x7f8f58055e70 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f800b1ef0 0x7f8f80066f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f80098ae0 0x7f8f80143f50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 --2- 172.21.15.61:0/1731421060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f80061760 0x7f8f80159e40 unknown :-1 s=CLOSED pgs=799 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 >> 172.21.15.61:0/1731421060 conn(0x7f8f800b2ca0 msgr2=0x7f8f801454b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:45.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 shutdown_connections 2022-07-20T03:14:45.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.049+0000 7f8f851f5700 1 -- 172.21.15.61:0/1731421060 wait complete. 2022-07-20T03:14:45.161 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:45.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 Processor -- start 2022-07-20T03:14:45.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 -- start start 2022-07-20T03:14:45.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6624061d90 0x7f6624099fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6624098ae0 0x7f662409a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6624062170 con 0x7f66240ac850 2022-07-20T03:14:45.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f662405d530 con 0x7f6624098ae0 2022-07-20T03:14:45.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f662aebe700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f66240abe70 con 0x7f6624061d90 2022-07-20T03:14:45.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6624098ae0 0x7f662409a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6624098ae0 0x7f662409a4d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47314/0 (socket says 172.21.15.61:47314) 2022-07-20T03:14:45.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 -- 172.21.15.61:0/3278285236 learned_addr learned my addr 172.21.15.61:0/3278285236 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:45.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f6629ebc700 1 --2- 172.21.15.61:0/3278285236 >> v2:172.21.15.117:3300/0 conn(0x7f6624061d90 0x7f6624099fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 -- 172.21.15.61:0/3278285236 >> v2:172.21.15.117:3300/0 conn(0x7f6624061d90 msgr2=0x7f6624099fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:45.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 --2- 172.21.15.61:0/3278285236 >> v2:172.21.15.117:3300/0 conn(0x7f6624061d90 0x7f6624099fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 -- 172.21.15.61:0/3278285236 >> v1:172.21.15.117:6789/0 conn(0x7f66240ac850 legacy=0x7f6624144000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f6628eba700 1 -- 172.21.15.61:0/3278285236 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2650608499 0 0) 0x7f6624062170 con 0x7f66240ac850 2022-07-20T03:14:45.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 -- 172.21.15.61:0/3278285236 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66240ac110 con 0x7f6624098ae0 2022-07-20T03:14:45.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f66296bb700 1 --2- 172.21.15.61:0/3278285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624098ae0 0x7f662409a4d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f6614002f70 tx=0x7f6614006eb0).ready entity=mon.0 client_cookie=2d79e98407bcbe1a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.265+0000 7f6628eba700 1 -- 172.21.15.61:0/3278285236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f661400d5c0 con 0x7f6624098ae0 2022-07-20T03:14:45.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f6628eba700 1 -- 172.21.15.61:0/3278285236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6614000e90 con 0x7f6624098ae0 2022-07-20T03:14:45.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f6628eba700 1 -- 172.21.15.61:0/3278285236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f661400d910 con 0x7f6624098ae0 2022-07-20T03:14:45.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- 172.21.15.61:0/3278285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624098ae0 msgr2=0x7f662409a4d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 --2- 172.21.15.61:0/3278285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624098ae0 0x7f662409a4d0 secure :-1 s=READY pgs=800 cs=0 l=1 rev1=1 rx=0x7f6614002f70 tx=0x7f6614006eb0).stop 2022-07-20T03:14:45.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- 172.21.15.61:0/3278285236 shutdown_connections 2022-07-20T03:14:45.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 --2- 172.21.15.61:0/3278285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624098ae0 0x7f662409a4d0 unknown :-1 s=CLOSED pgs=800 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 --2- 172.21.15.61:0/3278285236 >> v2:172.21.15.117:3300/0 conn(0x7f6624061d90 0x7f6624099fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- 172.21.15.61:0/3278285236 >> 172.21.15.61:0/3278285236 conn(0x7f66240b2ca0 msgr2=0x7f66240b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:45.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- 172.21.15.61:0/3278285236 shutdown_connections 2022-07-20T03:14:45.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- 172.21.15.61:0/3278285236 wait complete. 2022-07-20T03:14:45.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 Processor -- start 2022-07-20T03:14:45.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- start start 2022-07-20T03:14:45.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 0x7f662413f5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6624098ae0 0x7f662413ef50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66240ac850 0x7f662413a010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6624098430 con 0x7f6624061d90 2022-07-20T03:14:45.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6624060a80 con 0x7f66240ac850 2022-07-20T03:14:45.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f662aebe700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6624060cb0 con 0x7f6624098ae0 2022-07-20T03:14:45.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.269+0000 7f6629ebc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 0x7f662413f5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 0x7f662413f5e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47320/0 (socket says 172.21.15.61:47320) 2022-07-20T03:14:45.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 -- 172.21.15.61:0/3711248119 learned_addr learned my addr 172.21.15.61:0/3711248119 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:45.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f66296bb700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6624098ae0 0x7f662413ef50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f662a6bd700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66240ac850 0x7f662413a010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 -- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6624098ae0 msgr2=0x7f662413ef50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6624098ae0 0x7f662413ef50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 -- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66240ac850 msgr2=0x7f662413a010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66240ac850 0x7f662413a010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 -- 172.21.15.61:0/3711248119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f661400b1e0 con 0x7f6624061d90 2022-07-20T03:14:45.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f66296bb700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6624098ae0 0x7f662413ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:45.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f662a6bd700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66240ac850 0x7f662413a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:45.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6629ebc700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 0x7f662413f5e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f66241402d0 tx=0x7f6620006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6620010590 con 0x7f6624061d90 2022-07-20T03:14:45.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f66200106f0 con 0x7f6624061d90 2022-07-20T03:14:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66240ae6e0 con 0x7f6624061d90 2022-07-20T03:14:45.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6620010a50 con 0x7f6624061d90 2022-07-20T03:14:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66240ae3f0 con 0x7f6624061d90 2022-07-20T03:14:45.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.273+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66240ae870 con 0x7f6624061d90 2022-07-20T03:14:45.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.277+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6620010bb0 con 0x7f6624061d90 2022-07-20T03:14:45.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.277+0000 7f6612ffd700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66040539d0 0x7f6604055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.277+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f662006b740 con 0x7f6624061d90 2022-07-20T03:14:45.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.277+0000 7f66296bb700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66040539d0 0x7f6604055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.277+0000 7f66296bb700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66040539d0 0x7f6604055e90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f661401a000 tx=0x7f6614015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.281+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6620042a20 con 0x7f6624061d90 2022-07-20T03:14:45.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.409+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f66240ac3b0 con 0x7f6624061d90 2022-07-20T03:14:45.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.409+0000 7f6612ffd700 1 -- 172.21.15.61:0/3711248119 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v77) v1 ==== 74+0+13230 (secure 0 0 0) 0x7f6620046040 con 0x7f6624061d90 2022-07-20T03:14:45.426 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:45.427 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:44.691935+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":14,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":59,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":69,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:45.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66040539d0 msgr2=0x7f6604055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66040539d0 0x7f6604055e90 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f661401a000 tx=0x7f6614015000).stop 2022-07-20T03:14:45.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 msgr2=0x7f662413f5e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 0x7f662413f5e0 secure :-1 s=READY pgs=801 cs=0 l=1 rev1=1 rx=0x7f66241402d0 tx=0x7f6620006d90).stop 2022-07-20T03:14:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 shutdown_connections 2022-07-20T03:14:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f66040539d0 0x7f6604055e90 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f66240ac850 0x7f662413a010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6624098ae0 0x7f662413ef50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 --2- 172.21.15.61:0/3711248119 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6624061d90 0x7f662413f5e0 unknown :-1 s=CLOSED pgs=801 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 >> 172.21.15.61:0/3711248119 conn(0x7f66240b2ca0 msgr2=0x7f662409b4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:45.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 shutdown_connections 2022-07-20T03:14:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.413+0000 7f662aebe700 1 -- 172.21.15.61:0/3711248119 wait complete. 2022-07-20T03:14:45.437 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:14:45.438 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:14:45.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.549+0000 7fc53e0ab700 1 Processor -- start 2022-07-20T03:14:45.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.549+0000 7fc53e0ab700 1 -- start start 2022-07-20T03:14:45.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc53809df40 0x7fc538145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5380607e0 con 0x7fc538060ee0 2022-07-20T03:14:45.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc538060a80 con 0x7fc5380b4d60 2022-07-20T03:14:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc538060d20 con 0x7fc53809df40 2022-07-20T03:14:45.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc537fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc538060ee0 0x7fc53809cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43926/0 (socket says 172.21.15.61:43926) 2022-07-20T03:14:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc537fff700 1 -- 172.21.15.61:0/327802035 learned_addr learned my addr 172.21.15.61:0/327802035 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:45.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc5377fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc53809df40 0x7fc538145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 406354879 0 0) 0x7fc5380607e0 con 0x7fc538060ee0 2022-07-20T03:14:45.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc51c003260 con 0x7fc538060ee0 2022-07-20T03:14:45.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3931879683 0 0) 0x7fc538060a80 con 0x7fc5380b4d60 2022-07-20T03:14:45.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc5380607e0 con 0x7fc5380b4d60 2022-07-20T03:14:45.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc5377fe700 1 -- 172.21.15.61:0/327802035 >> v1:172.21.15.117:6789/0 conn(0x7fc5380b4d60 legacy=0x7fc538145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3871008436 0 0) 0x7fc51c003260 con 0x7fc538060ee0 2022-07-20T03:14:45.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc5377fe700 1 -- 172.21.15.61:0/327802035 >> v1:172.21.15.61:6789/0 conn(0x7fc538060ee0 legacy=0x7fc53809cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc5377fe700 1 -- 172.21.15.61:0/327802035 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc53809e320 con 0x7fc53809df40 2022-07-20T03:14:45.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc5377fe700 1 --2- 172.21.15.61:0/327802035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538145680 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fc530001490 tx=0x7fc528006df0).ready entity=mon.0 client_cookie=38188d57c76da047 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5300019e0 con 0x7fc53809df40 2022-07-20T03:14:45.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc528000e90 con 0x7fc53809df40 2022-07-20T03:14:45.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc536ffd700 1 -- 172.21.15.61:0/327802035 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc52800beb0 con 0x7fc53809df40 2022-07-20T03:14:45.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- 172.21.15.61:0/327802035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 msgr2=0x7fc538145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/327802035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538145680 secure :-1 s=READY pgs=803 cs=0 l=1 rev1=1 rx=0x7fc530001490 tx=0x7fc528006df0).stop 2022-07-20T03:14:45.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- 172.21.15.61:0/327802035 shutdown_connections 2022-07-20T03:14:45.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/327802035 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538145680 unknown :-1 s=CLOSED pgs=803 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- 172.21.15.61:0/327802035 >> 172.21.15.61:0/327802035 conn(0x7fc5380b5f30 msgr2=0x7fc5380b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:45.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- 172.21.15.61:0/327802035 shutdown_connections 2022-07-20T03:14:45.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.553+0000 7fc53e0ab700 1 -- 172.21.15.61:0/327802035 wait complete. 2022-07-20T03:14:45.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 Processor -- start 2022-07-20T03:14:45.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 -- start start 2022-07-20T03:14:45.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc538060ee0 0x7fc538144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538155b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5380b4d60 0x7fc538156010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc53809bc20 con 0x7fc53809df40 2022-07-20T03:14:45.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc538156500 con 0x7fc5380b4d60 2022-07-20T03:14:45.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.557+0000 7fc53e0ab700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5380b2b20 con 0x7fc538060ee0 2022-07-20T03:14:45.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538155b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538155b20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47332/0 (socket says 172.21.15.61:47332) 2022-07-20T03:14:45.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 -- 172.21.15.61:0/2743120099 learned_addr learned my addr 172.21.15.61:0/2743120099 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:45.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc537fff700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5380b4d60 0x7fc538156010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc5377fe700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc538060ee0 0x7fc538144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 -- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc538060ee0 msgr2=0x7fc538144ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc538060ee0 0x7fc538144ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 -- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5380b4d60 msgr2=0x7fc538156010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5380b4d60 0x7fc538156010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 -- 172.21.15.61:0/2743120099 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc52800b120 con 0x7fc53809df40 2022-07-20T03:14:45.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc52effd700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538155b20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fc520001110 tx=0x7fc520007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc52000cde0 con 0x7fc53809df40 2022-07-20T03:14:45.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc520012400 con 0x7fc53809df40 2022-07-20T03:14:45.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5380b2dc0 con 0x7fc53809df40 2022-07-20T03:14:45.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc520012760 con 0x7fc53809df40 2022-07-20T03:14:45.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5380b09e0 con 0x7fc53809df40 2022-07-20T03:14:45.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5380574e0 con 0x7fc53809df40 2022-07-20T03:14:45.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.561+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc520019030 con 0x7fc53809df40 2022-07-20T03:14:45.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.565+0000 7fc5357fa700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc510053920 0x7fc510055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.565+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fc52006ae00 con 0x7fc53809df40 2022-07-20T03:14:45.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.565+0000 7fc5377fe700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc510053920 0x7fc510055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.565+0000 7fc5377fe700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc510053920 0x7fc510055de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fc52801a000 tx=0x7fc528014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.569+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc520042090 con 0x7fc53809df40 2022-07-20T03:14:45.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.697+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fc538061810 con 0x7fc53809df40 2022-07-20T03:14:45.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.697+0000 7fc5357fa700 1 -- 172.21.15.61:0/2743120099 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v77) v1 ==== 110+0+10 (secure 0 0 0) 0x7fc5200456b0 con 0x7fc53809df40 2022-07-20T03:14:45.703 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:14:45.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.697+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc510053920 msgr2=0x7fc510055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.697+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc510053920 0x7fc510055de0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7fc52801a000 tx=0x7fc528014000).stop 2022-07-20T03:14:45.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 msgr2=0x7fc538155b20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538155b20 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7fc520001110 tx=0x7fc520007ab0).stop 2022-07-20T03:14:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 shutdown_connections 2022-07-20T03:14:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc510053920 0x7fc510055de0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5380b4d60 0x7fc538156010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc53809df40 0x7fc538155b20 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 --2- 172.21.15.61:0/2743120099 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc538060ee0 0x7fc538144ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 >> 172.21.15.61:0/2743120099 conn(0x7fc5380b5f30 msgr2=0x7fc538146710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:45.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 shutdown_connections 2022-07-20T03:14:45.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.701+0000 7fc53e0ab700 1 -- 172.21.15.61:0/2743120099 wait complete. 2022-07-20T03:14:45.730 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:14:45.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.845+0000 7f6598cd7700 1 Processor -- start 2022-07-20T03:14:45.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.845+0000 7f6598cd7700 1 -- start start 2022-07-20T03:14:45.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65941462a0 0x7f6594146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f6594060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65940afdb0 con 0x7f65940988b0 2022-07-20T03:14:45.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65940b0050 con 0x7f65941462a0 2022-07-20T03:14:45.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f659405fcd0 con 0x7f6594146b70 2022-07-20T03:14:45.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f659259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65941462a0 0x7f6594146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f659259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65941462a0 0x7f6594146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47340/0 (socket says 172.21.15.61:47340) 2022-07-20T03:14:45.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6592d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f6594060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f659259c700 1 -- 172.21.15.61:0/692297953 learned_addr learned my addr 172.21.15.61:0/692297953 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:45.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6592d9d700 1 -- 172.21.15.61:0/692297953 >> v2:172.21.15.61:3300/0 conn(0x7f65941462a0 msgr2=0x7f6594146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:45.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6592d9d700 1 --2- 172.21.15.61:0/692297953 >> v2:172.21.15.61:3300/0 conn(0x7f65941462a0 0x7f6594146680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6592d9d700 1 -- 172.21.15.61:0/692297953 >> v1:172.21.15.117:6789/0 conn(0x7f65940988b0 legacy=0x7f65941422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6592d9d700 1 -- 172.21.15.61:0/692297953 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65940afae0 con 0x7f6594146b70 2022-07-20T03:14:45.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6592d9d700 1 --2- 172.21.15.61:0/692297953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f6594060170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f658c0032f0 tx=0x7f658c0091d0).ready entity=mon.2 client_cookie=cd918d6ee7761270 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f659159a700 1 -- 172.21.15.61:0/692297953 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f658c00b6c0 con 0x7f6594146b70 2022-07-20T03:14:45.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f659159a700 1 -- 172.21.15.61:0/692297953 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c00b820 con 0x7f6594146b70 2022-07-20T03:14:45.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f659159a700 1 -- 172.21.15.61:0/692297953 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f658c00b990 con 0x7f6594146b70 2022-07-20T03:14:45.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- 172.21.15.61:0/692297953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 msgr2=0x7f6594060170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 --2- 172.21.15.61:0/692297953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f6594060170 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f658c0032f0 tx=0x7f658c0091d0).stop 2022-07-20T03:14:45.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- 172.21.15.61:0/692297953 shutdown_connections 2022-07-20T03:14:45.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 --2- 172.21.15.61:0/692297953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f6594060170 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 --2- 172.21.15.61:0/692297953 >> v2:172.21.15.61:3300/0 conn(0x7f65941462a0 0x7f6594146680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- 172.21.15.61:0/692297953 >> 172.21.15.61:0/692297953 conn(0x7f65940b5f00 msgr2=0x7f65940b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:45.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- 172.21.15.61:0/692297953 shutdown_connections 2022-07-20T03:14:45.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.849+0000 7f6598cd7700 1 -- 172.21.15.61:0/692297953 wait complete. 2022-07-20T03:14:45.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 Processor -- start 2022-07-20T03:14:45.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- start start 2022-07-20T03:14:45.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65940988b0 0x7f659409dd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65941462a0 0x7f659405c440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f65941524e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f659409d140 con 0x7f65941462a0 2022-07-20T03:14:45.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b2f90 con 0x7f65940988b0 2022-07-20T03:14:45.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65940b31f0 con 0x7f6594146b70 2022-07-20T03:14:45.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6591d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65941462a0 0x7f659405c440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6591d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65941462a0 0x7f659405c440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47346/0 (socket says 172.21.15.61:47346) 2022-07-20T03:14:45.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6591d9b700 1 -- 172.21.15.61:0/2624383042 learned_addr learned my addr 172.21.15.61:0/2624383042 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:45.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f65941524e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f659259c700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65940988b0 0x7f659409dd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 -- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65940988b0 msgr2=0x7f659409dd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65940988b0 0x7f659409dd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 -- 172.21.15.61:0/2624383042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65941462a0 msgr2=0x7f659405c440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:45.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65941462a0 0x7f659405c440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:45.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 -- 172.21.15.61:0/2624383042 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f658c009470 con 0x7f6594146b70 2022-07-20T03:14:45.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6592d9d700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f65941524e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f658c0032f0 tx=0x7f658c016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f658c011e20 con 0x7f6594146b70 2022-07-20T03:14:45.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f658c020400 con 0x7f6594146b70 2022-07-20T03:14:45.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f658c020760 con 0x7f6594146b70 2022-07-20T03:14:45.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65940b3490 con 0x7f6594146b70 2022-07-20T03:14:45.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6594151fa0 con 0x7f6594146b70 2022-07-20T03:14:45.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.853+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65940574e0 con 0x7f6594146b70 2022-07-20T03:14:45.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.857+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f658c01e070 con 0x7f6594146b70 2022-07-20T03:14:45.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.857+0000 7f658b7fe700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6578053920 0x7f6578055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:45.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.857+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f658c06d070 con 0x7f6594146b70 2022-07-20T03:14:45.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.857+0000 7f659259c700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6578053920 0x7f6578055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:45.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.857+0000 7f659259c700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6578053920 0x7f6578055de0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f6584008000 tx=0x7f6584008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:45.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.865+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f658c044300 con 0x7f6594146b70 2022-07-20T03:14:46.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.993+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f6594067ea0 con 0x7f6594146b70 2022-07-20T03:14:46.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.993+0000 7f658b7fe700 1 -- 172.21.15.61:0/2624383042 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v77) v1 ==== 92+0+10 (secure 0 0 0) 0x7f658c015070 con 0x7f6594146b70 2022-07-20T03:14:46.008 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:46.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6578053920 msgr2=0x7f6578055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6578053920 0x7f6578055de0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f6584008000 tx=0x7f6584008040).stop 2022-07-20T03:14:46.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 msgr2=0x7f65941524e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f65941524e0 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f658c0032f0 tx=0x7f658c016000).stop 2022-07-20T03:14:46.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 shutdown_connections 2022-07-20T03:14:46.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6578053920 0x7f6578055de0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6594146b70 0x7f65941524e0 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65941462a0 0x7f659405c440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 --2- 172.21.15.61:0/2624383042 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65940988b0 0x7f659409dd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 >> 172.21.15.61:0/2624383042 conn(0x7f65940b5f00 msgr2=0x7f6594141f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:46.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 shutdown_connections 2022-07-20T03:14:46.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:45.997+0000 7f6598cd7700 1 -- 172.21.15.61:0/2624383042 wait complete. 2022-07-20T03:14:46.020 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:14:46.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 Processor -- start 2022-07-20T03:14:46.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- start start 2022-07-20T03:14:46.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b5c0b5650 0x7f2b5c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0adff0 0x7f2b5c147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b5c147010 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c1472b0 con 0x7f2b5c0b5650 2022-07-20T03:14:46.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b5c146d40 con 0x7f2b5c0adff0 2022-07-20T03:14:46.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b5c0b5650 0x7f2b5c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b5c0b5650 0x7f2b5c0b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47354/0 (socket says 172.21.15.61:47354) 2022-07-20T03:14:46.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 -- 172.21.15.61:0/497206895 learned_addr learned my addr 172.21.15.61:0/497206895 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:46.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5a59c700 1 --2- 172.21.15.61:0/497206895 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0adff0 0x7f2b5c147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 -- 172.21.15.61:0/497206895 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0adff0 msgr2=0x7f2b5c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:46.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 --2- 172.21.15.61:0/497206895 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0adff0 0x7f2b5c147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 -- 172.21.15.61:0/497206895 >> v1:172.21.15.117:6789/0 conn(0x7f2b5c0af8e0 legacy=0x7f2b5c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:46.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 -- 172.21.15.61:0/497206895 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b5c147c40 con 0x7f2b5c0b5650 2022-07-20T03:14:46.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b5ad9d700 1 --2- 172.21.15.61:0/497206895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c0b5a30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f2b5c060b70 tx=0x7f2b44006eb0).ready entity=mon.0 client_cookie=3a73c371a27a9436 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:46.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b59d9b700 1 -- 172.21.15.61:0/497206895 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b4400d5c0 con 0x7f2b5c0b5650 2022-07-20T03:14:46.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b59d9b700 1 -- 172.21.15.61:0/497206895 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b44000e90 con 0x7f2b5c0b5650 2022-07-20T03:14:46.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b59d9b700 1 -- 172.21.15.61:0/497206895 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b4400d910 con 0x7f2b5c0b5650 2022-07-20T03:14:46.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- 172.21.15.61:0/497206895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0b5650 msgr2=0x7f2b5c0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/497206895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c0b5a30 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f2b5c060b70 tx=0x7f2b44006eb0).stop 2022-07-20T03:14:46.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- 172.21.15.61:0/497206895 shutdown_connections 2022-07-20T03:14:46.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/497206895 >> v2:172.21.15.117:3300/0 conn(0x7f2b5c0adff0 0x7f2b5c147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/497206895 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c0b5a30 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- 172.21.15.61:0/497206895 >> 172.21.15.61:0/497206895 conn(0x7f2b5c0634e0 msgr2=0x7f2b5c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:46.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- 172.21.15.61:0/497206895 shutdown_connections 2022-07-20T03:14:46.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.153+0000 7f2b60a1a700 1 -- 172.21.15.61:0/497206895 wait complete. 2022-07-20T03:14:46.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 Processor -- start 2022-07-20T03:14:46.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 -- start start 2022-07-20T03:14:46.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0adff0 0x7f2b5c10e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 0x7f2b5c13d770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c13dc60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c111770 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c066fd0 con 0x7f2b5c0b5650 2022-07-20T03:14:46.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b5c111030 con 0x7f2b5c0adff0 2022-07-20T03:14:46.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 0x7f2b5c13d770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 0x7f2b5c13d770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47358/0 (socket says 172.21.15.61:47358) 2022-07-20T03:14:46.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 -- 172.21.15.61:0/3302859566 learned_addr learned my addr 172.21.15.61:0/3302859566 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:46.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5b59e700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c13dc60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5ad9d700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0adff0 0x7f2b5c10e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 -- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0adff0 msgr2=0x7f2b5c10e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0adff0 0x7f2b5c10e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 -- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c0b5650 msgr2=0x7f2b5c13dc60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c13dc60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 -- 172.21.15.61:0/3302859566 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b4400b1e0 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b5a59c700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 0x7f2b5c13d770 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f2b5400a000 tx=0x7f2b54008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:46.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b54005a70 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b54005bd0 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b5c1112d0 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b54006030 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.157+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b5c13e150 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.161+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b5c05c4a0 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.161+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b54006190 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.161+0000 7f2b53fff700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b400539b0 0x7f2b40055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.161+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f2b5406b9f0 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.161+0000 7f2b5ad9d700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b400539b0 0x7f2b40055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.161+0000 7f2b5ad9d700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b400539b0 0x7f2b40055e70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f2b4401c000 tx=0x7f2b44015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:46.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.165+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b5406de60 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.297+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f2b5c0574e0 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.297+0000 7f2b53fff700 1 -- 172.21.15.61:0/3302859566 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v77) v1 ==== 84+0+143 (secure 0 0 0) 0x7f2b5400c030 con 0x7f2b5c0af8e0 2022-07-20T03:14:46.308 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:46.309 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:14:46.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:46.324 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:14:46.324 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:14:46.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:46.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:46.325 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:14:46.325 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:14:46.326 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:46.326 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:14:46.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b400539b0 msgr2=0x7f2b40055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b400539b0 0x7f2b40055e70 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f2b4401c000 tx=0x7f2b44015000).stop 2022-07-20T03:14:46.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 msgr2=0x7f2b5c13d770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 0x7f2b5c13d770 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7f2b5400a000 tx=0x7f2b54008040).stop 2022-07-20T03:14:46.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 shutdown_connections 2022-07-20T03:14:46.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b400539b0 0x7f2b40055e70 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b5c0b5650 0x7f2b5c13dc60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b5c0af8e0 0x7f2b5c13d770 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 --2- 172.21.15.61:0/3302859566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b5c0adff0 0x7f2b5c10e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 >> 172.21.15.61:0/3302859566 conn(0x7f2b5c0634e0 msgr2=0x7f2b5c0ae500 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:46.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 shutdown_connections 2022-07-20T03:14:46.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.301+0000 7f2b60a1a700 1 -- 172.21.15.61:0/3302859566 wait complete. 2022-07-20T03:14:46.343 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:14:46.344 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:14:46.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 Processor -- start 2022-07-20T03:14:46.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 -- start start 2022-07-20T03:14:46.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f50240b3cf0 0x7f5024142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5024142f40 0x7f50241419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f50240ba180 con 0x7f50240b9da0 2022-07-20T03:14:46.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f50240ad5b0 con 0x7f50240b3cf0 2022-07-20T03:14:46.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.457+0000 7f5028d0e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50241461b0 con 0x7f5024142f40 2022-07-20T03:14:46.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5022ffd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f50240b3cf0 0x7f5024142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5023fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5024142f40 0x7f50241419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f50237fe700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f50240b9da0 0x7f50241468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:52468/0 (socket says 172.21.15.61:52468) 2022-07-20T03:14:46.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f50237fe700 1 -- 172.21.15.61:0/3258991033 learned_addr learned my addr 172.21.15.61:0/3258991033 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:46.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5023fff700 1 -- 172.21.15.61:0/3258991033 >> v2:172.21.15.117:3300/0 conn(0x7f50240b3cf0 msgr2=0x7f5024142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:46.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5023fff700 1 --2- 172.21.15.61:0/3258991033 >> v2:172.21.15.117:3300/0 conn(0x7f50240b3cf0 0x7f5024142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5023fff700 1 -- 172.21.15.61:0/3258991033 >> v1:172.21.15.117:6789/0 conn(0x7f50240b9da0 legacy=0x7f50241468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5023fff700 1 -- 172.21.15.61:0/3258991033 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5024146450 con 0x7f5024142f40 2022-07-20T03:14:46.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5022ffd700 1 --2- 172.21.15.61:0/3258991033 >> v2:172.21.15.117:3300/0 conn(0x7f50240b3cf0 0x7f5024142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:46.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5023fff700 1 --2- 172.21.15.61:0/3258991033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5024142f40 0x7f50241419a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f501c0032f0 tx=0x7f501c0091d0).ready entity=mon.2 client_cookie=ec19bb88ea6b14eb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:46.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f50227fc700 1 -- 172.21.15.61:0/3258991033 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501c00b680 con 0x7f5024142f40 2022-07-20T03:14:46.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f50227fc700 1 -- 172.21.15.61:0/3258991033 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c00b7e0 con 0x7f5024142f40 2022-07-20T03:14:46.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f50227fc700 1 -- 172.21.15.61:0/3258991033 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501c00b990 con 0x7f5024142f40 2022-07-20T03:14:46.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 -- 172.21.15.61:0/3258991033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5024142f40 msgr2=0x7f50241419a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 --2- 172.21.15.61:0/3258991033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5024142f40 0x7f50241419a0 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f501c0032f0 tx=0x7f501c0091d0).stop 2022-07-20T03:14:46.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 -- 172.21.15.61:0/3258991033 shutdown_connections 2022-07-20T03:14:46.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 --2- 172.21.15.61:0/3258991033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5024142f40 0x7f50241419a0 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 --2- 172.21.15.61:0/3258991033 >> v2:172.21.15.117:3300/0 conn(0x7f50240b3cf0 0x7f5024142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 -- 172.21.15.61:0/3258991033 >> 172.21.15.61:0/3258991033 conn(0x7f50240678c0 msgr2=0x7f50240b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:46.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 -- 172.21.15.61:0/3258991033 shutdown_connections 2022-07-20T03:14:46.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.461+0000 7f5028d0e700 1 -- 172.21.15.61:0/3258991033 wait complete. 2022-07-20T03:14:46.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 Processor -- start 2022-07-20T03:14:46.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 -- start start 2022-07-20T03:14:46.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f50240b3cf0 0x7f50240b7f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50240b9da0 0x7f5024143f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 0x7f5024152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50240b1e80 con 0x7f5024142f40 2022-07-20T03:14:46.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5024151c90 con 0x7f50240b9da0 2022-07-20T03:14:46.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f50240b2e60 con 0x7f50240b3cf0 2022-07-20T03:14:46.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 0x7f5024152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 0x7f5024152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47376/0 (socket says 172.21.15.61:47376) 2022-07-20T03:14:46.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 -- 172.21.15.61:0/925574063 learned_addr learned my addr 172.21.15.61:0/925574063 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:46.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5022ffd700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50240b9da0 0x7f5024143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f50237fe700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f50240b3cf0 0x7f50240b7f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 -- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f50240b3cf0 msgr2=0x7f50240b7f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f50240b3cf0 0x7f50240b7f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 -- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50240b9da0 msgr2=0x7f5024143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:46.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50240b9da0 0x7f5024143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:46.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 -- 172.21.15.61:0/925574063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f501400b180 con 0x7f5024142f40 2022-07-20T03:14:46.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5023fff700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 0x7f5024152130 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f501c0032f0 tx=0x7f501c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:46.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501c0205e0 con 0x7f5024142f40 2022-07-20T03:14:46.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f501c020740 con 0x7f5024142f40 2022-07-20T03:14:46.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f501c009470 con 0x7f5024142f40 2022-07-20T03:14:46.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f501c020aa0 con 0x7f5024142f40 2022-07-20T03:14:46.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.465+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5024145470 con 0x7f5024142f40 2022-07-20T03:14:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.469+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50240b2040 con 0x7f5024142f40 2022-07-20T03:14:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.469+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f501c01e070 con 0x7f5024142f40 2022-07-20T03:14:46.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.469+0000 7f5020ff9700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5008053af0 0x7f5008055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:46.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.469+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(77..77 src has 1..77) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f501c06cdd0 con 0x7f5024142f40 2022-07-20T03:14:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.469+0000 7f50237fe700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5008053af0 0x7f5008055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:46.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.469+0000 7f50237fe700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5008053af0 0x7f5008055fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f500c00d000 tx=0x7f500c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:46.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.477+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f501c043fe0 con 0x7f5024142f40 2022-07-20T03:14:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:46.601+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f50240667f0 con 0x7f5024142f40 2022-07-20T03:14:47.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.329+0000 7f5020ff9700 1 -- 172.21.15.61:0/925574063 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v78) v1 ==== 133+0+0 (secure 0 0 0) 0x7f501c015020 con 0x7f5024142f40 2022-07-20T03:14:47.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5008053af0 msgr2=0x7f5008055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5008053af0 0x7f5008055fb0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f500c00d000 tx=0x7f500c009040).stop 2022-07-20T03:14:47.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 msgr2=0x7f5024152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 0x7f5024152130 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f501c0032f0 tx=0x7f501c016000).stop 2022-07-20T03:14:47.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 shutdown_connections 2022-07-20T03:14:47.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5008053af0 0x7f5008055fb0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f50240b9da0 0x7f5024143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5024142f40 0x7f5024152130 unknown :-1 s=CLOSED pgs=807 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 --2- 172.21.15.61:0/925574063 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f50240b3cf0 0x7f50240b7f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 >> 172.21.15.61:0/925574063 conn(0x7f50240678c0 msgr2=0x7f50240ae4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:47.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 shutdown_connections 2022-07-20T03:14:47.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.337+0000 7f5028d0e700 1 -- 172.21.15.61:0/925574063 wait complete. 2022-07-20T03:14:47.355 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:14:47.357 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:14:47.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 Processor -- start 2022-07-20T03:14:47.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 -- start start 2022-07-20T03:14:47.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc9e005bf30 0x7fc9e005d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e0143ea0 0x7fc9e0144350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc9e00b4230 con 0x7fc9e00adfa0 2022-07-20T03:14:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc9e00677b0 con 0x7fc9e005bf30 2022-07-20T03:14:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e730d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e00b2950 con 0x7fc9e0143ea0 2022-07-20T03:14:47.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5b0a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc9e00adfa0 0x7fc9e013a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43980/0 (socket says 172.21.15.61:43980) 2022-07-20T03:14:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5b0a700 1 -- 172.21.15.61:0/2987750931 learned_addr learned my addr 172.21.15.61:0/2987750931 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:47.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e6b0c700 1 --2- 172.21.15.61:0/2987750931 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e0143ea0 0x7fc9e0144350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e630b700 1 --2- 172.21.15.61:0/2987750931 >> v2:172.21.15.117:3300/0 conn(0x7fc9e005bf30 0x7fc9e005d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5309700 1 -- 172.21.15.61:0/2987750931 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4016872847 0 0) 0x7fc9e00b4230 con 0x7fc9e00adfa0 2022-07-20T03:14:47.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5309700 1 -- 172.21.15.61:0/2987750931 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc9c8003260 con 0x7fc9e00adfa0 2022-07-20T03:14:47.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e6b0c700 1 -- 172.21.15.61:0/2987750931 >> v2:172.21.15.117:3300/0 conn(0x7fc9e005bf30 msgr2=0x7fc9e005d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:47.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e6b0c700 1 --2- 172.21.15.61:0/2987750931 >> v2:172.21.15.117:3300/0 conn(0x7fc9e005bf30 0x7fc9e005d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e6b0c700 1 -- 172.21.15.61:0/2987750931 >> v1:172.21.15.61:6789/0 conn(0x7fc9e00adfa0 legacy=0x7fc9e013a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e6b0c700 1 -- 172.21.15.61:0/2987750931 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9e00b3850 con 0x7fc9e0143ea0 2022-07-20T03:14:47.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e6b0c700 1 --2- 172.21.15.61:0/2987750931 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e0143ea0 0x7fc9e0144350 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc9d80032f0 tx=0x7fc9d80091d0).ready entity=mon.2 client_cookie=e2384860b1520793 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:47.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5309700 1 -- 172.21.15.61:0/2987750931 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d800b600 con 0x7fc9e0143ea0 2022-07-20T03:14:47.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5309700 1 -- 172.21.15.61:0/2987750931 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d800b760 con 0x7fc9e0143ea0 2022-07-20T03:14:47.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.465+0000 7fc9e5309700 1 -- 172.21.15.61:0/2987750931 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d800b8d0 con 0x7fc9e0143ea0 2022-07-20T03:14:47.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- 172.21.15.61:0/2987750931 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e0143ea0 msgr2=0x7fc9e0144350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 --2- 172.21.15.61:0/2987750931 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e0143ea0 0x7fc9e0144350 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc9d80032f0 tx=0x7fc9d80091d0).stop 2022-07-20T03:14:47.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- 172.21.15.61:0/2987750931 shutdown_connections 2022-07-20T03:14:47.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 --2- 172.21.15.61:0/2987750931 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e0143ea0 0x7fc9e0144350 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 --2- 172.21.15.61:0/2987750931 >> v2:172.21.15.117:3300/0 conn(0x7fc9e005bf30 0x7fc9e005d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- 172.21.15.61:0/2987750931 >> 172.21.15.61:0/2987750931 conn(0x7fc9e0064b90 msgr2=0x7fc9e0064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:47.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- 172.21.15.61:0/2987750931 shutdown_connections 2022-07-20T03:14:47.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- 172.21.15.61:0/2987750931 wait complete. 2022-07-20T03:14:47.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 Processor -- start 2022-07-20T03:14:47.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- start start 2022-07-20T03:14:47.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e005bf30 0x7fc9e0147b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 0x7fc9e013c940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9e0143ea0 0x7fc9e013ce30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e013f890 con 0x7fc9e00adfa0 2022-07-20T03:14:47.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e013e730 con 0x7fc9e0143ea0 2022-07-20T03:14:47.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e730d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc9e013f350 con 0x7fc9e005bf30 2022-07-20T03:14:47.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 0x7fc9e013c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e630b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e005bf30 0x7fc9e0147b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 0x7fc9e013c940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47390/0 (socket says 172.21.15.61:47390) 2022-07-20T03:14:47.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e630b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e005bf30 0x7fc9e0147b40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:34058/0 (socket says 172.21.15.61:34058) 2022-07-20T03:14:47.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 -- 172.21.15.61:0/1170608251 learned_addr learned my addr 172.21.15.61:0/1170608251 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:47.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e6b0c700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9e0143ea0 0x7fc9e013ce30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 -- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e005bf30 msgr2=0x7fc9e0147b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e005bf30 0x7fc9e0147b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 -- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9e0143ea0 msgr2=0x7fc9e013ce30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9e0143ea0 0x7fc9e013ce30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 -- 172.21.15.61:0/1170608251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc9d8009470 con 0x7fc9e00adfa0 2022-07-20T03:14:47.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.469+0000 7fc9e5b0a700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 0x7fc9e013c940 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7fc9d0001110 tx=0x7fc9d0007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:47.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d000ce80 con 0x7fc9e00adfa0 2022-07-20T03:14:47.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc9e013f5f0 con 0x7fc9e00adfa0 2022-07-20T03:14:47.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc9d00070d0 con 0x7fc9e00adfa0 2022-07-20T03:14:47.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc9d0007430 con 0x7fc9e00adfa0 2022-07-20T03:14:47.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc9e01102b0 con 0x7fc9e00adfa0 2022-07-20T03:14:47.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc9e00b3af0 con 0x7fc9e00adfa0 2022-07-20T03:14:47.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc9d0010070 con 0x7fc9e00adfa0 2022-07-20T03:14:47.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9cf7fe700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9c0053ad0 0x7fc9c0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.473+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(78..78 src has 1..78) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fc9d006b2e0 con 0x7fc9e00adfa0 2022-07-20T03:14:47.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.477+0000 7fc9e630b700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9c0053ad0 0x7fc9c0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.477+0000 7fc9e630b700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9c0053ad0 0x7fc9c0055f90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fc9dc00b000 tx=0x7fc9dc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:47.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.481+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc9d0042310 con 0x7fc9e00adfa0 2022-07-20T03:14:47.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.609+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fc9e00574e0 con 0x7fc9e00adfa0 2022-07-20T03:14:47.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.685+0000 7fc9cf7fe700 1 -- 172.21.15.61:0/1170608251 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v79) v1 ==== 125+0+0 (secure 0 0 0) 0x7fc9d0018070 con 0x7fc9e00adfa0 2022-07-20T03:14:47.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9c0053ad0 msgr2=0x7fc9c0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9c0053ad0 0x7fc9c0055f90 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fc9dc00b000 tx=0x7fc9dc006df0).stop 2022-07-20T03:14:47.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 msgr2=0x7fc9e013c940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 0x7fc9e013c940 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7fc9d0001110 tx=0x7fc9d0007aa0).stop 2022-07-20T03:14:47.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 shutdown_connections 2022-07-20T03:14:47.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9c0053ad0 0x7fc9c0055f90 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc9e0143ea0 0x7fc9e013ce30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc9e00adfa0 0x7fc9e013c940 unknown :-1 s=CLOSED pgs=809 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 --2- 172.21.15.61:0/1170608251 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc9e005bf30 0x7fc9e0147b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 >> 172.21.15.61:0/1170608251 conn(0x7fc9e0064b90 msgr2=0x7fc9e00acae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:47.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 shutdown_connections 2022-07-20T03:14:47.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.689+0000 7fc9e730d700 1 -- 172.21.15.61:0/1170608251 wait complete. 2022-07-20T03:14:47.757 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:14:47.758 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:14:47.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c255c700 1 Processor -- start 2022-07-20T03:14:47.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c255c700 1 -- start start 2022-07-20T03:14:47.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c255c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc05cf40 0x7fc8bc062f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c255c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0b3000 con 0x7fc8bc0b2be0 2022-07-20T03:14:47.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c255c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8bc0b1e00 con 0x7fc8bc0b3880 2022-07-20T03:14:47.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c255c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc144f40 con 0x7fc8bc05cf40 2022-07-20T03:14:47.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c0af9700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc8bc0b2be0 0x7fc8bc13a060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:43996/0 (socket says 172.21.15.61:43996) 2022-07-20T03:14:47.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.885+0000 7fc8c0af9700 1 -- 172.21.15.61:0/1604864411 learned_addr learned my addr 172.21.15.61:0/1604864411 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:47.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8bbfff700 1 --2- 172.21.15.61:0/1604864411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc05cf40 0x7fc8bc062f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:47.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156522237 0 0) 0x7fc8bc0b3000 con 0x7fc8bc0b2be0 2022-07-20T03:14:47.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8a8003260 con 0x7fc8bc0b2be0 2022-07-20T03:14:47.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3441067595 0 0) 0x7fc8bc0b1e00 con 0x7fc8bc0b3880 2022-07-20T03:14:47.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc8bc0b3000 con 0x7fc8bc0b3880 2022-07-20T03:14:47.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8bbfff700 1 -- 172.21.15.61:0/1604864411 >> v1:172.21.15.117:6789/0 conn(0x7fc8bc0b3880 legacy=0x7fc8bc0b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8bbfff700 1 -- 172.21.15.61:0/1604864411 >> v1:172.21.15.61:6789/0 conn(0x7fc8bc0b2be0 legacy=0x7fc8bc13a060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1166938922 0 0) 0x7fc8a8003260 con 0x7fc8bc0b2be0 2022-07-20T03:14:47.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8bbfff700 1 -- 172.21.15.61:0/1604864411 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8bc0add50 con 0x7fc8bc05cf40 2022-07-20T03:14:47.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8bbfff700 1 --2- 172.21.15.61:0/1604864411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc05cf40 0x7fc8bc062f30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc8b4001490 tx=0x7fc8b0006df0).ready entity=mon.2 client_cookie=59a3abbc2a327c89 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:47.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8b40019e0 con 0x7fc8bc05cf40 2022-07-20T03:14:47.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b0000e90 con 0x7fc8bc05cf40 2022-07-20T03:14:47.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8baffd700 1 -- 172.21.15.61:0/1604864411 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8b0012460 con 0x7fc8bc05cf40 2022-07-20T03:14:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 -- 172.21.15.61:0/1604864411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc05cf40 msgr2=0x7fc8bc062f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:47.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 --2- 172.21.15.61:0/1604864411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc05cf40 0x7fc8bc062f30 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fc8b4001490 tx=0x7fc8b0006df0).stop 2022-07-20T03:14:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 -- 172.21.15.61:0/1604864411 shutdown_connections 2022-07-20T03:14:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 --2- 172.21.15.61:0/1604864411 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc05cf40 0x7fc8bc062f30 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:47.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 -- 172.21.15.61:0/1604864411 >> 172.21.15.61:0/1604864411 conn(0x7fc8bc0b7d00 msgr2=0x7fc8bc0b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 -- 172.21.15.61:0/1604864411 shutdown_connections 2022-07-20T03:14:47.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 -- 172.21.15.61:0/1604864411 wait complete. 2022-07-20T03:14:47.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 Processor -- start 2022-07-20T03:14:47.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.889+0000 7fc8c255c700 1 -- start start 2022-07-20T03:14:47.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 0x7fc8bc0678e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc0b2be0 0x7fc8bc0b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc0b3880 0x7fc8bc155a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:47.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1434b0 con 0x7fc8bc05cf40 2022-07-20T03:14:47.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc143210 con 0x7fc8bc0b2be0 2022-07-20T03:14:47.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc8bc1555c0 con 0x7fc8bc0b3880 2022-07-20T03:14:47.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 0x7fc8bc0678e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bb7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc0b2be0 0x7fc8bc0b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 0x7fc8bc0678e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47402/0 (socket says 172.21.15.61:47402) 2022-07-20T03:14:48.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 -- 172.21.15.61:0/185072850 learned_addr learned my addr 172.21.15.61:0/185072850 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:48.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c0af9700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc0b3880 0x7fc8bc155a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 -- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc0b3880 msgr2=0x7fc8bc155a60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc0b3880 0x7fc8bc155a60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 -- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc0b2be0 msgr2=0x7fc8bc0b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc0b2be0 0x7fc8bc0b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 -- 172.21.15.61:0/185072850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8b000b180 con 0x7fc8bc05cf40 2022-07-20T03:14:48.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bb7fe700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc0b2be0 0x7fc8bc0b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:48.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8bbfff700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 0x7fc8bc0678e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7fc8b4001490 tx=0x7fc8b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8b0016030 con 0x7fc8bc05cf40 2022-07-20T03:14:48.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc8b000db50 con 0x7fc8bc05cf40 2022-07-20T03:14:48.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc8bc142f30 con 0x7fc8bc05cf40 2022-07-20T03:14:48.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc8b0012460 con 0x7fc8bc05cf40 2022-07-20T03:14:48.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.893+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8bc0673c0 con 0x7fc8bc05cf40 2022-07-20T03:14:48.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.897+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8bc0adff0 con 0x7fc8bc05cf40 2022-07-20T03:14:48.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.897+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc8b0024020 con 0x7fc8bc05cf40 2022-07-20T03:14:48.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.897+0000 7fc8b97fa700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc89c053a40 0x7fc89c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.897+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(79..79 src has 1..79) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fc8b0017070 con 0x7fc8bc05cf40 2022-07-20T03:14:48.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.897+0000 7fc8bb7fe700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc89c053a40 0x7fc89c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.897+0000 7fc8bb7fe700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc89c053a40 0x7fc89c055f00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fc8ac00d000 tx=0x7fc8ac007a10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:47.905+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc8b0043ca0 con 0x7fc8bc05cf40 2022-07-20T03:14:48.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.069+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fc8bc0574e0 con 0x7fc8bc05cf40 2022-07-20T03:14:48.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.081+0000 7fc8b97fa700 1 -- 172.21.15.61:0/185072850 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 15 and data pool 16 v44) v1 ==== 189+0+0 (secure 0 0 0) 0x7fc8b00472c0 con 0x7fc8bc05cf40 2022-07-20T03:14:48.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc89c053a40 msgr2=0x7fc89c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc89c053a40 0x7fc89c055f00 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fc8ac00d000 tx=0x7fc8ac007a10).stop 2022-07-20T03:14:48.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 msgr2=0x7fc8bc0678e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 0x7fc8bc0678e0 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7fc8b4001490 tx=0x7fc8b0014000).stop 2022-07-20T03:14:48.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 shutdown_connections 2022-07-20T03:14:48.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc89c053a40 0x7fc89c055f00 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8bc0b2be0 0x7fc8bc0b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8bc0b3880 0x7fc8bc155a60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 --2- 172.21.15.61:0/185072850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8bc05cf40 0x7fc8bc0678e0 unknown :-1 s=CLOSED pgs=811 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 >> 172.21.15.61:0/185072850 conn(0x7fc8bc0b7d00 msgr2=0x7fc8bc0b6ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:48.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 shutdown_connections 2022-07-20T03:14:48.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.085+0000 7fc8c255c700 1 -- 172.21.15.61:0/185072850 wait complete. 2022-07-20T03:14:48.106 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 15 and data pool 16 2022-07-20T03:14:48.107 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:48.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda97e1f700 1 Processor -- start 2022-07-20T03:14:48.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda97e1f700 1 -- start start 2022-07-20T03:14:48.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda97e1f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fda900ac260 0x7fda900ac640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda97e1f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda90096fa0 con 0x7fda90141200 2022-07-20T03:14:48.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda97e1f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fda900b22d0 con 0x7fda900b3620 2022-07-20T03:14:48.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda97e1f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fda900b26d0 con 0x7fda900ac260 2022-07-20T03:14:48.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda95bbb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fda900ac260 0x7fda900ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda963bc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fda90141200 0x7fda901400c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44012/0 (socket says 172.21.15.61:44012) 2022-07-20T03:14:48.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda963bc700 1 -- 172.21.15.61:0/1678124265 learned_addr learned my addr 172.21.15.61:0/1678124265 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:48.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda94bb9700 1 -- 172.21.15.61:0/1678124265 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3629984542 0 0) 0x7fda90096fa0 con 0x7fda90141200 2022-07-20T03:14:48.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.205+0000 7fda94bb9700 1 -- 172.21.15.61:0/1678124265 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fda78003260 con 0x7fda90141200 2022-07-20T03:14:48.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda95bbb700 1 -- 172.21.15.61:0/1678124265 >> v1:172.21.15.117:6789/0 conn(0x7fda900b3620 legacy=0x7fda90096890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda95bbb700 1 -- 172.21.15.61:0/1678124265 >> v1:172.21.15.61:6789/0 conn(0x7fda90141200 legacy=0x7fda901400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda95bbb700 1 -- 172.21.15.61:0/1678124265 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda90140ac0 con 0x7fda900ac260 2022-07-20T03:14:48.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda95bbb700 1 --2- 172.21.15.61:0/1678124265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda900ac640 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fda80002f70 tx=0x7fda80006df0).ready entity=mon.0 client_cookie=ba93356cc2e70514 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda94bb9700 1 -- 172.21.15.61:0/1678124265 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda8000d610 con 0x7fda900ac260 2022-07-20T03:14:48.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda94bb9700 1 -- 172.21.15.61:0/1678124265 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda80000e90 con 0x7fda900ac260 2022-07-20T03:14:48.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda94bb9700 1 -- 172.21.15.61:0/1678124265 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda8000d960 con 0x7fda900ac260 2022-07-20T03:14:48.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 -- 172.21.15.61:0/1678124265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 msgr2=0x7fda900ac640 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 --2- 172.21.15.61:0/1678124265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda900ac640 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7fda80002f70 tx=0x7fda80006df0).stop 2022-07-20T03:14:48.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 -- 172.21.15.61:0/1678124265 shutdown_connections 2022-07-20T03:14:48.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 --2- 172.21.15.61:0/1678124265 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda900ac640 unknown :-1 s=CLOSED pgs=813 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 -- 172.21.15.61:0/1678124265 >> 172.21.15.61:0/1678124265 conn(0x7fda900b47d0 msgr2=0x7fda9005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:48.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 -- 172.21.15.61:0/1678124265 shutdown_connections 2022-07-20T03:14:48.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.209+0000 7fda97e1f700 1 -- 172.21.15.61:0/1678124265 wait complete. 2022-07-20T03:14:48.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 Processor -- start 2022-07-20T03:14:48.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 -- start start 2022-07-20T03:14:48.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda90098670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda900b3620 0x7fda90143700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda90141200 0x7fda90143bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda900b0e40 con 0x7fda900ac260 2022-07-20T03:14:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda900b0540 con 0x7fda90141200 2022-07-20T03:14:48.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fda900b0010 con 0x7fda900b3620 2022-07-20T03:14:48.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda90098670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda90098670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47416/0 (socket says 172.21.15.61:47416) 2022-07-20T03:14:48.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 -- 172.21.15.61:0/821573779 learned_addr learned my addr 172.21.15.61:0/821573779 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:48.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda963bc700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda90141200 0x7fda90143bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda953ba700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda900b3620 0x7fda90143700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 -- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda900b3620 msgr2=0x7fda90143700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda900b3620 0x7fda90143700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 -- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda90141200 msgr2=0x7fda90143bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda90141200 0x7fda90143bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 -- 172.21.15.61:0/821573779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda8000b180 con 0x7fda900ac260 2022-07-20T03:14:48.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda95bbb700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda90098670 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fda80002f70 tx=0x7fda80014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda80016070 con 0x7fda900ac260 2022-07-20T03:14:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fda8000de90 con 0x7fda900ac260 2022-07-20T03:14:48.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fda8001d3f0 con 0x7fda900ac260 2022-07-20T03:14:48.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda90144ab0 con 0x7fda900ac260 2022-07-20T03:14:48.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.213+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda900b0970 con 0x7fda900ac260 2022-07-20T03:14:48.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.217+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fda80024030 con 0x7fda900ac260 2022-07-20T03:14:48.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.217+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda900b06b0 con 0x7fda900ac260 2022-07-20T03:14:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.217+0000 7fda7effd700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda70053a40 0x7fda70055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.217+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fda8006cdc0 con 0x7fda900ac260 2022-07-20T03:14:48.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.217+0000 7fda953ba700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda70053a40 0x7fda70055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.217+0000 7fda953ba700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda70053a40 0x7fda70055f00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fda8c00b000 tx=0x7fda8c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.221+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fda80044020 con 0x7fda900ac260 2022-07-20T03:14:48.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.349+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fda90140d60 con 0x7fda900ac260 2022-07-20T03:14:48.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.349+0000 7fda7effd700 1 -- 172.21.15.61:0/821573779 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16115 (secure 0 0 0) 0x7fda80022070 con 0x7fda900ac260 2022-07-20T03:14:48.354 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:48.354 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:48.072013+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:14:46.661180+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T03:14:47.668660+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:48.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda70053a40 msgr2=0x7fda70055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda70053a40 0x7fda70055f00 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fda8c00b000 tx=0x7fda8c009040).stop 2022-07-20T03:14:48.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 msgr2=0x7fda90098670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda90098670 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fda80002f70 tx=0x7fda80014000).stop 2022-07-20T03:14:48.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 shutdown_connections 2022-07-20T03:14:48.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fda70053a40 0x7fda70055f00 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fda90141200 0x7fda90143bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fda900b3620 0x7fda90143700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 --2- 172.21.15.61:0/821573779 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fda900ac260 0x7fda90098670 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 >> 172.21.15.61:0/821573779 conn(0x7fda900b47d0 msgr2=0x7fda9013fd20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:48.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 shutdown_connections 2022-07-20T03:14:48.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.353+0000 7fda97e1f700 1 -- 172.21.15.61:0/821573779 wait complete. 2022-07-20T03:14:48.367 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:48.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 Processor -- start 2022-07-20T03:14:48.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 -- start start 2022-07-20T03:14:48.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fccb00b0570 0x7fccb00b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb01424e0 0x7fccb00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fccb01433d0 con 0x7fccb00aea10 2022-07-20T03:14:48.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fccb0143b10 con 0x7fccb00b0570 2022-07-20T03:14:48.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0143670 con 0x7fccb01424e0 2022-07-20T03:14:48.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fccb00b0570 0x7fccb00b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fccb00b0570 0x7fccb00b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47424/0 (socket says 172.21.15.61:47424) 2022-07-20T03:14:48.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 -- 172.21.15.61:0/4052306469 learned_addr learned my addr 172.21.15.61:0/4052306469 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:48.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb4912700 1 --2- 172.21.15.61:0/4052306469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb01424e0 0x7fccb00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 -- 172.21.15.61:0/4052306469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb01424e0 msgr2=0x7fccb00ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:48.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 --2- 172.21.15.61:0/4052306469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb01424e0 0x7fccb00ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 -- 172.21.15.61:0/4052306469 >> v1:172.21.15.117:6789/0 conn(0x7fccb00aea10 legacy=0x7fccb0141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 -- 172.21.15.61:0/4052306469 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb00b1010 con 0x7fccb00b0570 2022-07-20T03:14:48.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaffff700 1 --2- 172.21.15.61:0/4052306469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb00b0570 0x7fccb00b0950 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fcca0002f70 tx=0x7fcca0006eb0).ready entity=mon.0 client_cookie=2d8ac62cad86c060 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaeffd700 1 -- 172.21.15.61:0/4052306469 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcca000d5c0 con 0x7fccb00b0570 2022-07-20T03:14:48.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaeffd700 1 -- 172.21.15.61:0/4052306469 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca0000e90 con 0x7fccb00b0570 2022-07-20T03:14:48.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccaeffd700 1 -- 172.21.15.61:0/4052306469 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcca000d950 con 0x7fccb00b0570 2022-07-20T03:14:48.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 -- 172.21.15.61:0/4052306469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb00b0570 msgr2=0x7fccb00b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.489+0000 7fccb5113700 1 --2- 172.21.15.61:0/4052306469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb00b0570 0x7fccb00b0950 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7fcca0002f70 tx=0x7fcca0006eb0).stop 2022-07-20T03:14:48.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- 172.21.15.61:0/4052306469 shutdown_connections 2022-07-20T03:14:48.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 --2- 172.21.15.61:0/4052306469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb01424e0 0x7fccb00ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 --2- 172.21.15.61:0/4052306469 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb00b0570 0x7fccb00b0950 unknown :-1 s=CLOSED pgs=815 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- 172.21.15.61:0/4052306469 >> 172.21.15.61:0/4052306469 conn(0x7fccb00b9360 msgr2=0x7fccb00b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:48.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- 172.21.15.61:0/4052306469 shutdown_connections 2022-07-20T03:14:48.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- 172.21.15.61:0/4052306469 wait complete. 2022-07-20T03:14:48.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 Processor -- start 2022-07-20T03:14:48.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- start start 2022-07-20T03:14:48.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb00aea10 0x7fccb0112ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccb00b0570 0x7fccb013e050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 0x7fccb013e540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0113b60 con 0x7fccb01424e0 2022-07-20T03:14:48.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb01150f0 con 0x7fccb00b0570 2022-07-20T03:14:48.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb5113700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fccb0115390 con 0x7fccb00aea10 2022-07-20T03:14:48.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 0x7fccb013e540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 0x7fccb013e540 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47434/0 (socket says 172.21.15.61:47434) 2022-07-20T03:14:48.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 -- 172.21.15.61:0/1666774894 learned_addr learned my addr 172.21.15.61:0/1666774894 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:48.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccaf7fe700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccb00b0570 0x7fccb013e050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccaffff700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb00aea10 0x7fccb0112ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 -- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb00aea10 msgr2=0x7fccb0112ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb00aea10 0x7fccb0112ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 -- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccb00b0570 msgr2=0x7fccb013e050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccb00b0570 0x7fccb013e050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 -- 172.21.15.61:0/1666774894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca000b1e0 con 0x7fccb01424e0 2022-07-20T03:14:48.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.493+0000 7fccb4912700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 0x7fccb013e540 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fcca800a000 tx=0x7fcca8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcca800f710 con 0x7fccb01424e0 2022-07-20T03:14:48.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcca800f870 con 0x7fccb01424e0 2022-07-20T03:14:48.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb00b3de0 con 0x7fccb01424e0 2022-07-20T03:14:48.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcca800fbd0 con 0x7fccb01424e0 2022-07-20T03:14:48.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb00b4280 con 0x7fccb01424e0 2022-07-20T03:14:48.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb00b3fa0 con 0x7fccb01424e0 2022-07-20T03:14:48.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcca801a030 con 0x7fccb01424e0 2022-07-20T03:14:48.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccad7fa700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcc94053af0 0x7fcc94055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.497+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcca806bc10 con 0x7fccb01424e0 2022-07-20T03:14:48.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.501+0000 7fccaffff700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcc94053af0 0x7fcc94055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.501+0000 7fccaffff700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcc94053af0 0x7fcc94055fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fcca001c000 tx=0x7fcca0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.505+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcca8042b10 con 0x7fccb01424e0 2022-07-20T03:14:48.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.633+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fccb00ace30 con 0x7fccb01424e0 2022-07-20T03:14:48.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.633+0000 7fccad7fa700 1 -- 172.21.15.61:0/1666774894 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v80) v1 ==== 74+0+16115 (secure 0 0 0) 0x7fcca8046130 con 0x7fccb01424e0 2022-07-20T03:14:48.642 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:48.643 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:48.072013+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:14:46.661180+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T03:14:47.668660+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"80","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:48.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcc94053af0 msgr2=0x7fcc94055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcc94053af0 0x7fcc94055fb0 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fcca001c000 tx=0x7fcca0015000).stop 2022-07-20T03:14:48.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 msgr2=0x7fccb013e540 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 0x7fccb013e540 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fcca800a000 tx=0x7fcca8008040).stop 2022-07-20T03:14:48.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 shutdown_connections 2022-07-20T03:14:48.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcc94053af0 0x7fcc94055fb0 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fccb00b0570 0x7fccb013e050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fccb01424e0 0x7fccb013e540 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 --2- 172.21.15.61:0/1666774894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fccb00aea10 0x7fccb0112ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 >> 172.21.15.61:0/1666774894 conn(0x7fccb00b9360 msgr2=0x7fccb00b78e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:48.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 shutdown_connections 2022-07-20T03:14:48.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.637+0000 7fccb5113700 1 -- 172.21.15.61:0/1666774894 wait complete. 2022-07-20T03:14:48.669 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:14:48.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 Processor -- start 2022-07-20T03:14:48.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- start start 2022-07-20T03:14:48.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fde680b5b50 0x7fde680accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680ac5e0 con 0x7fde6805c120 2022-07-20T03:14:48.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde680b5f30 con 0x7fde680b0480 2022-07-20T03:14:48.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde6805cf70 con 0x7fde680b5b50 2022-07-20T03:14:48.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fde680b5b50 0x7fde680accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fde680b5b50 0x7fde680accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47438/0 (socket says 172.21.15.61:47438) 2022-07-20T03:14:48.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 -- 172.21.15.61:0/3647893793 learned_addr learned my addr 172.21.15.61:0/3647893793 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:48.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6d758700 1 --1- 172.21.15.61:0/3647893793 >> v1:172.21.15.61:6789/0 conn(0x7fde6805c120 0x7fde68144320 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44040/0 (socket says 172.21.15.61:44040) 2022-07-20T03:14:48.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 -- 172.21.15.61:0/3647893793 >> v1:172.21.15.117:6789/0 conn(0x7fde680b0480 legacy=0x7fde6814a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 -- 172.21.15.61:0/3647893793 >> v1:172.21.15.61:6789/0 conn(0x7fde6805c120 legacy=0x7fde68144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 -- 172.21.15.61:0/3647893793 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde680ac810 con 0x7fde680b5b50 2022-07-20T03:14:48.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde677fe700 1 -- 172.21.15.61:0/3647893793 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1285419760 0 0) 0x7fde680ac5e0 con 0x7fde6805c120 2022-07-20T03:14:48.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6cf57700 1 --2- 172.21.15.61:0/3647893793 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b5b50 0x7fde680accb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fde58002f70 tx=0x7fde58006eb0).ready entity=mon.0 client_cookie=4ab3def6a317e9fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde677fe700 1 -- 172.21.15.61:0/3647893793 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde5800d660 con 0x7fde680b5b50 2022-07-20T03:14:48.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde677fe700 1 -- 172.21.15.61:0/3647893793 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58000e90 con 0x7fde680b5b50 2022-07-20T03:14:48.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde677fe700 1 -- 172.21.15.61:0/3647893793 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde5800d9b0 con 0x7fde680b5b50 2022-07-20T03:14:48.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- 172.21.15.61:0/3647893793 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b5b50 msgr2=0x7fde680accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/3647893793 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b5b50 0x7fde680accb0 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7fde58002f70 tx=0x7fde58006eb0).stop 2022-07-20T03:14:48.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- 172.21.15.61:0/3647893793 shutdown_connections 2022-07-20T03:14:48.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/3647893793 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde680b5b50 0x7fde680accb0 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- 172.21.15.61:0/3647893793 >> 172.21.15.61:0/3647893793 conn(0x7fde680635e0 msgr2=0x7fde680b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:48.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.801+0000 7fde6f1bb700 1 -- 172.21.15.61:0/3647893793 shutdown_connections 2022-07-20T03:14:48.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- 172.21.15.61:0/3647893793 wait complete. 2022-07-20T03:14:48.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 Processor -- start 2022-07-20T03:14:48.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- start start 2022-07-20T03:14:48.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 0x7fde680ae3b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680b0480 0x7fde68064a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680b5b50 0x7fde68064ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68148db0 con 0x7fde6805c120 2022-07-20T03:14:48.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde68149050 con 0x7fde680b0480 2022-07-20T03:14:48.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde681492f0 con 0x7fde680b5b50 2022-07-20T03:14:48.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 0x7fde680ae3b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 0x7fde680ae3b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47444/0 (socket says 172.21.15.61:47444) 2022-07-20T03:14:48.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 -- 172.21.15.61:0/1308182338 learned_addr learned my addr 172.21.15.61:0/1308182338 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:48.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6d758700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680b5b50 0x7fde68064ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 -- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680b5b50 msgr2=0x7fde68064ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde67fff700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680b0480 0x7fde68064a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680b5b50 0x7fde68064ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 -- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680b0480 msgr2=0x7fde68064a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:48.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680b0480 0x7fde68064a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:48.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 -- 172.21.15.61:0/1308182338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde5800b1e0 con 0x7fde6805c120 2022-07-20T03:14:48.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6d758700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680b5b50 0x7fde68064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:48.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde67fff700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680b0480 0x7fde68064a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:14:48.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6cf57700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 0x7fde680ae3b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fde58002f70 tx=0x7fde58015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde58011a00 con 0x7fde6805c120 2022-07-20T03:14:48.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde58011b60 con 0x7fde6805c120 2022-07-20T03:14:48.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde68141fd0 con 0x7fde6805c120 2022-07-20T03:14:48.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde5801e460 con 0x7fde6805c120 2022-07-20T03:14:48.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.805+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde68149f00 con 0x7fde6805c120 2022-07-20T03:14:48.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.809+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fde58025030 con 0x7fde6805c120 2022-07-20T03:14:48.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.809+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde6814ad80 con 0x7fde6805c120 2022-07-20T03:14:48.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.809+0000 7fde65ffb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:48.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.809+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(80..80 src has 1..80) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fde5806d3b0 con 0x7fde6805c120 2022-07-20T03:14:48.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.809+0000 7fde67fff700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:48.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.809+0000 7fde67fff700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fde5c008000 tx=0x7fde5c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:48.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.817+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde58044310 con 0x7fde6805c120 2022-07-20T03:14:48.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:48.985+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fde680574e0 con 0x7fde6805c120 2022-07-20T03:14:49.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.089+0000 7fde65ffb700 1 -- 172.21.15.61:0/1308182338 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v45) v1 ==== 118+0+0 (secure 0 0 0) 0x7fde580230b0 con 0x7fde6805c120 2022-07-20T03:14:49.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 msgr2=0x7fde48055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fde5c008000 tx=0x7fde5c009040).stop 2022-07-20T03:14:49.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 msgr2=0x7fde680ae3b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 0x7fde680ae3b0 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fde58002f70 tx=0x7fde58015000).stop 2022-07-20T03:14:49.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 shutdown_connections 2022-07-20T03:14:49.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde48053a40 0x7fde48055f00 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde680b0480 0x7fde68064a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde680b5b50 0x7fde68064ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 --2- 172.21.15.61:0/1308182338 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde6805c120 0x7fde680ae3b0 unknown :-1 s=CLOSED pgs=819 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 >> 172.21.15.61:0/1308182338 conn(0x7fde680635e0 msgr2=0x7fde68145440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:49.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 shutdown_connections 2022-07-20T03:14:49.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.093+0000 7fde6f1bb700 1 -- 172.21.15.61:0/1308182338 wait complete. 2022-07-20T03:14:49.131 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:49.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.265+0000 7f1c18d36700 1 Processor -- start 2022-07-20T03:14:49.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- start start 2022-07-20T03:14:49.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1c140b3110 0x7f1c14061d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1c140b34f0 0x7f1c141414b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1c140acaf0 con 0x7f1c141419a0 2022-07-20T03:14:49.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140acd90 con 0x7f1c140b3110 2022-07-20T03:14:49.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1c140b2190 con 0x7f1c140b34f0 2022-07-20T03:14:49.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1c140b3110 0x7f1c14061d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c12d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1c141419a0 0x7f1c14144060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44054/0 (socket says 172.21.15.61:44054) 2022-07-20T03:14:49.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c12d9d700 1 -- 172.21.15.61:0/277676985 learned_addr learned my addr 172.21.15.61:0/277676985 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:49.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c11d9b700 1 --2- 172.21.15.61:0/277676985 >> v2:172.21.15.117:3300/0 conn(0x7f1c140b34f0 0x7f1c141414b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1159a700 1 -- 172.21.15.61:0/277676985 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2497430974 0 0) 0x7f1c140acaf0 con 0x7f1c141419a0 2022-07-20T03:14:49.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1159a700 1 -- 172.21.15.61:0/277676985 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1bf8003260 con 0x7f1c141419a0 2022-07-20T03:14:49.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1259c700 1 -- 172.21.15.61:0/277676985 >> v2:172.21.15.117:3300/0 conn(0x7f1c140b34f0 msgr2=0x7f1c141414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:49.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1259c700 1 --2- 172.21.15.61:0/277676985 >> v2:172.21.15.117:3300/0 conn(0x7f1c140b34f0 0x7f1c141414b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1259c700 1 -- 172.21.15.61:0/277676985 >> v1:172.21.15.61:6789/0 conn(0x7f1c141419a0 legacy=0x7f1c14144060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1259c700 1 -- 172.21.15.61:0/277676985 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c140b3940 con 0x7f1c140b3110 2022-07-20T03:14:49.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1259c700 1 --2- 172.21.15.61:0/277676985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b3110 0x7f1c14061d80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f1c140ad7d0 tx=0x7f1c08006df0).ready entity=mon.0 client_cookie=9302a5dfd9a7f517 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:49.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1159a700 1 -- 172.21.15.61:0/277676985 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0800d4f0 con 0x7f1c140b3110 2022-07-20T03:14:49.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1159a700 1 -- 172.21.15.61:0/277676985 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c08000e90 con 0x7f1c140b3110 2022-07-20T03:14:49.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c1159a700 1 -- 172.21.15.61:0/277676985 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0800d840 con 0x7f1c140b3110 2022-07-20T03:14:49.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- 172.21.15.61:0/277676985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b3110 msgr2=0x7f1c14061d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 --2- 172.21.15.61:0/277676985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b3110 0x7f1c14061d80 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7f1c140ad7d0 tx=0x7f1c08006df0).stop 2022-07-20T03:14:49.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- 172.21.15.61:0/277676985 shutdown_connections 2022-07-20T03:14:49.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 --2- 172.21.15.61:0/277676985 >> v2:172.21.15.117:3300/0 conn(0x7f1c140b34f0 0x7f1c141414b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 --2- 172.21.15.61:0/277676985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b3110 0x7f1c14061d80 unknown :-1 s=CLOSED pgs=821 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- 172.21.15.61:0/277676985 >> 172.21.15.61:0/277676985 conn(0x7f1c140b47e0 msgr2=0x7f1c1405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:49.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- 172.21.15.61:0/277676985 shutdown_connections 2022-07-20T03:14:49.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.269+0000 7f1c18d36700 1 -- 172.21.15.61:0/277676985 wait complete. 2022-07-20T03:14:49.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 Processor -- start 2022-07-20T03:14:49.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 -- start start 2022-07-20T03:14:49.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c140b3110 0x7f1c14142100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 0x7f1c141425f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c141419a0 0x7f1c141532a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14150870 con 0x7f1c140b34f0 2022-07-20T03:14:49.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c14142ae0 con 0x7f1c140b3110 2022-07-20T03:14:49.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1c140b0a20 con 0x7f1c141419a0 2022-07-20T03:14:49.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 0x7f1c141425f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 0x7f1c141425f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47460/0 (socket says 172.21.15.61:47460) 2022-07-20T03:14:49.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 -- 172.21.15.61:0/2541414784 learned_addr learned my addr 172.21.15.61:0/2541414784 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:49.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c1259c700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c140b3110 0x7f1c14142100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c12d9d700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c141419a0 0x7f1c141532a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 -- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c141419a0 msgr2=0x7f1c141532a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c141419a0 0x7f1c141532a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 -- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c140b3110 msgr2=0x7f1c14142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c140b3110 0x7f1c14142100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 -- 172.21.15.61:0/2541414784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c0800b180 con 0x7f1c140b34f0 2022-07-20T03:14:49.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c11d9b700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 0x7f1c141425f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f1c0400c000 tx=0x7f1c040091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:49.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0400c850 con 0x7f1c140b34f0 2022-07-20T03:14:49.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c0400c9b0 con 0x7f1c140b34f0 2022-07-20T03:14:49.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c14150aa0 con 0x7f1c140b34f0 2022-07-20T03:14:49.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1c0400cd10 con 0x7f1c140b34f0 2022-07-20T03:14:49.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.273+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c1413f390 con 0x7f1c140b34f0 2022-07-20T03:14:49.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.277+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1c04022070 con 0x7f1c140b34f0 2022-07-20T03:14:49.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.277+0000 7f1c037fe700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bf00539b0 0x7f1bf0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.277+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1c0406c560 con 0x7f1c140b34f0 2022-07-20T03:14:49.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.277+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c14150c30 con 0x7f1c140b34f0 2022-07-20T03:14:49.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.277+0000 7f1c1259c700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bf00539b0 0x7f1bf0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.277+0000 7f1c1259c700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bf00539b0 0x7f1bf0055e70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f1c0801a000 tx=0x7f1c08014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:49.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.285+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1c040434c0 con 0x7f1c140b34f0 2022-07-20T03:14:49.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.449+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f1c140b2630 con 0x7f1c140b34f0 2022-07-20T03:14:49.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.449+0000 7f1c037fe700 1 -- 172.21.15.61:0/2541414784 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f1c04046ae0 con 0x7f1c140b34f0 2022-07-20T03:14:49.463 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:49.464 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:14:49.076244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6643":{"gid":6643,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/843120801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":843120801},{"type":"v1","addr":"172.21.15.117:6835","nonce":843120801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:14:49.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bf00539b0 msgr2=0x7f1bf0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bf00539b0 0x7f1bf0055e70 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f1c0801a000 tx=0x7f1c08014000).stop 2022-07-20T03:14:49.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 msgr2=0x7f1c141425f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 0x7f1c141425f0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f1c0400c000 tx=0x7f1c040091d0).stop 2022-07-20T03:14:49.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 shutdown_connections 2022-07-20T03:14:49.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1bf00539b0 0x7f1bf0055e70 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1c141419a0 0x7f1c141532a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1c140b34f0 0x7f1c141425f0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 --2- 172.21.15.61:0/2541414784 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1c140b3110 0x7f1c14142100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 >> 172.21.15.61:0/2541414784 conn(0x7f1c140b47e0 msgr2=0x7f1c140b5d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:49.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 shutdown_connections 2022-07-20T03:14:49.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.453+0000 7f1c18d36700 1 -- 172.21.15.61:0/2541414784 wait complete. 2022-07-20T03:14:49.559 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-20T03:14:49.560 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:49.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 Processor -- start 2022-07-20T03:14:49.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 -- start start 2022-07-20T03:14:49.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc5d40b1e00 0x7fc5d414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d4146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc5d405d480 con 0x7fc5d405c3f0 2022-07-20T03:14:49.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc5d40b21e0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d9e02700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d405cf40 con 0x7fc5d40acef0 2022-07-20T03:14:49.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d37fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc5d40b1e00 0x7fc5d414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d4146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d2ffd700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fc5d405c3f0 0x7fc5d40ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:52568/0 (socket says 172.21.15.61:52568) 2022-07-20T03:14:49.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d2ffd700 1 -- 172.21.15.61:0/181787524 learned_addr learned my addr 172.21.15.61:0/181787524 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d37fe700 1 -- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 msgr2=0x7fc5d4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d37fe700 1 --2- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d4146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d37fe700 1 -- 172.21.15.61:0/181787524 >> v1:172.21.15.117:6789/0 conn(0x7fc5d405c3f0 legacy=0x7fc5d40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d37fe700 1 -- 172.21.15.61:0/181787524 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5d405d1e0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d37fe700 1 --2- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40b1e00 0x7fc5d414a6d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fc5c4001490 tx=0x7fc5bc006df0).ready entity=mon.1 client_cookie=b5c8043265f9d5c0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:49.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d27fc700 1 -- 172.21.15.61:0/181787524 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5bc00d4a0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d27fc700 1 -- 172.21.15.61:0/181787524 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5bc000e90 con 0x7fc5d40b1e00 2022-07-20T03:14:49.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.681+0000 7fc5d27fc700 1 -- 172.21.15.61:0/181787524 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5bc00d7f0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40b1e00 msgr2=0x7fc5d414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40b1e00 0x7fc5d414a6d0 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fc5c4001490 tx=0x7fc5bc006df0).stop 2022-07-20T03:14:49.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- 172.21.15.61:0/181787524 shutdown_connections 2022-07-20T03:14:49.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d4146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/181787524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40b1e00 0x7fc5d414a6d0 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- 172.21.15.61:0/181787524 >> 172.21.15.61:0/181787524 conn(0x7fc5d40611c0 msgr2=0x7fc5d4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:49.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- 172.21.15.61:0/181787524 shutdown_connections 2022-07-20T03:14:49.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- 172.21.15.61:0/181787524 wait complete. 2022-07-20T03:14:49.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 Processor -- start 2022-07-20T03:14:49.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- start start 2022-07-20T03:14:49.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d409e4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 0x7fc5d40b4410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40a1b30 0x7fc5d409fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d409f280 con 0x7fc5d40b1e00 2022-07-20T03:14:49.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d409f520 con 0x7fc5d40a1b30 2022-07-20T03:14:49.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d9e02700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc5d409f7c0 con 0x7fc5d40acef0 2022-07-20T03:14:49.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 0x7fc5d40b4410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d37fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d409e4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 0x7fc5d40b4410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47474/0 (socket says 172.21.15.61:47474) 2022-07-20T03:14:49.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d37fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d409e4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:34144/0 (socket says 172.21.15.61:34144) 2022-07-20T03:14:49.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d2ffd700 1 -- 172.21.15.61:0/2074404461 learned_addr learned my addr 172.21.15.61:0/2074404461 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:49.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.685+0000 7fc5d3fff700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40a1b30 0x7fc5d409fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d2ffd700 1 -- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 msgr2=0x7fc5d409e4a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d2ffd700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d409e4a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d2ffd700 1 -- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40a1b30 msgr2=0x7fc5d409fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d2ffd700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40a1b30 0x7fc5d409fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d2ffd700 1 -- 172.21.15.61:0/2074404461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc5bc00b180 con 0x7fc5d40b1e00 2022-07-20T03:14:49.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d2ffd700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 0x7fc5d40b4410 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fc5c4001380 tx=0x7fc5c400caf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:49.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c4005b60 con 0x7fc5d40b1e00 2022-07-20T03:14:49.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc5c4005cc0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc5d40a02b0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc5c4007300 con 0x7fc5d40b1e00 2022-07-20T03:14:49.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc5d4159d10 con 0x7fc5d40b1e00 2022-07-20T03:14:49.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc5d40b04b0 con 0x7fc5d40b1e00 2022-07-20T03:14:49.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc5c4010030 con 0x7fc5d40b1e00 2022-07-20T03:14:49.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d0ff9700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053ad0 0x7fc5b8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:49.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc5c406bf60 con 0x7fc5d40b1e00 2022-07-20T03:14:49.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.689+0000 7fc5d37fe700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053ad0 0x7fc5b8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:49.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.693+0000 7fc5d37fe700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053ad0 0x7fc5b8055f90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc5bc01a000 tx=0x7fc5bc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:49.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.697+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc5c4042e20 con 0x7fc5d40b1e00 2022-07-20T03:14:49.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.825+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc5d4069a60 con 0x7fc5d40b1e00 2022-07-20T03:14:49.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.825+0000 7fc5d0ff9700 1 -- 172.21.15.61:0/2074404461 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16089 (secure 0 0 0) 0x7fc5c4046440 con 0x7fc5d40b1e00 2022-07-20T03:14:49.945 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:49.950 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":81,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:49.075215+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:14:46.661180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T03:14:47.668660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:49.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053ad0 msgr2=0x7fc5b8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053ad0 0x7fc5b8055f90 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7fc5bc01a000 tx=0x7fc5bc014000).stop 2022-07-20T03:14:49.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 msgr2=0x7fc5d40b4410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:49.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 0x7fc5d40b4410 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fc5c4001380 tx=0x7fc5c400caf0).stop 2022-07-20T03:14:49.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 shutdown_connections 2022-07-20T03:14:49.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc5b8053ad0 0x7fc5b8055f90 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc5d40a1b30 0x7fc5d409fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc5d40b1e00 0x7fc5d40b4410 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 --2- 172.21.15.61:0/2074404461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc5d40acef0 0x7fc5d409e4a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:49.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 >> 172.21.15.61:0/2074404461 conn(0x7fc5d405c3f0 msgr2=0x7fc5d4069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:50.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 shutdown_connections 2022-07-20T03:14:50.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:49.829+0000 7fc5d9e02700 1 -- 172.21.15.61:0/2074404461 wait complete. 2022-07-20T03:14:50.006 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:14:50.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.129+0000 7fd98de31700 1 Processor -- start 2022-07-20T03:14:50.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.129+0000 7fd98de31700 1 -- start start 2022-07-20T03:14:50.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.129+0000 7fd98de31700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd988144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.129+0000 7fd98de31700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9880649d0 con 0x7fd988066cb0 2022-07-20T03:14:50.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.129+0000 7fd98de31700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd9881442c0 con 0x7fd9880b1c30 2022-07-20T03:14:50.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.129+0000 7fd98de31700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd988144560 con 0x7fd98805cf40 2022-07-20T03:14:50.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd987fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd988066cb0 0x7fd988141100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44082/0 (socket says 172.21.15.61:44082) 2022-07-20T03:14:50.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd987fff700 1 -- 172.21.15.61:0/3269994273 learned_addr learned my addr 172.21.15.61:0/3269994273 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9877fe700 1 --2- 172.21.15.61:0/3269994273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd988144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9877fe700 1 -- 172.21.15.61:0/3269994273 >> v1:172.21.15.117:6789/0 conn(0x7fd9880b1c30 legacy=0x7fd988061da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9877fe700 1 -- 172.21.15.61:0/3269994273 >> v1:172.21.15.61:6789/0 conn(0x7fd988066cb0 legacy=0x7fd988141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9867fc700 1 -- 172.21.15.61:0/3269994273 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1886967904 0 0) 0x7fd9880649d0 con 0x7fd988066cb0 2022-07-20T03:14:50.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9877fe700 1 -- 172.21.15.61:0/3269994273 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd988144800 con 0x7fd98805cf40 2022-07-20T03:14:50.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9877fe700 1 --2- 172.21.15.61:0/3269994273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd988144ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd970001160 tx=0x7fd978006df0).ready entity=mon.2 client_cookie=ea6b8733c94831fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:50.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9867fc700 1 -- 172.21.15.61:0/3269994273 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd97800d570 con 0x7fd98805cf40 2022-07-20T03:14:50.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9867fc700 1 -- 172.21.15.61:0/3269994273 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd978000e90 con 0x7fd98805cf40 2022-07-20T03:14:50.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd9867fc700 1 -- 172.21.15.61:0/3269994273 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd97800d8c0 con 0x7fd98805cf40 2022-07-20T03:14:50.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 -- 172.21.15.61:0/3269994273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 msgr2=0x7fd988144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 --2- 172.21.15.61:0/3269994273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd988144ca0 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fd970001160 tx=0x7fd978006df0).stop 2022-07-20T03:14:50.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 -- 172.21.15.61:0/3269994273 shutdown_connections 2022-07-20T03:14:50.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 --2- 172.21.15.61:0/3269994273 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd988144ca0 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 -- 172.21.15.61:0/3269994273 >> 172.21.15.61:0/3269994273 conn(0x7fd9880b7d60 msgr2=0x7fd9880b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:50.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 -- 172.21.15.61:0/3269994273 shutdown_connections 2022-07-20T03:14:50.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.133+0000 7fd98de31700 1 -- 172.21.15.61:0/3269994273 wait complete. 2022-07-20T03:14:50.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 Processor -- start 2022-07-20T03:14:50.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 -- start start 2022-07-20T03:14:50.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd98809a930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd988066cb0 0x7fd9880af5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 0x7fd98809c0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd988150b70 con 0x7fd9880b1c30 2022-07-20T03:14:50.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880afdb0 con 0x7fd988066cb0 2022-07-20T03:14:50.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd98de31700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd9880afac0 con 0x7fd98805cf40 2022-07-20T03:14:50.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 0x7fd98809c0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd9877fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd98809a930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 0x7fd98809c0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47490/0 (socket says 172.21.15.61:47490) 2022-07-20T03:14:50.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 -- 172.21.15.61:0/3634501216 learned_addr learned my addr 172.21.15.61:0/3634501216 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:50.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd986ffd700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd988066cb0 0x7fd9880af5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 -- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 msgr2=0x7fd98809a930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd98809a930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 -- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd988066cb0 msgr2=0x7fd9880af5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd988066cb0 0x7fd9880af5d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 -- 172.21.15.61:0/3634501216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd97800b180 con 0x7fd9880b1c30 2022-07-20T03:14:50.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd9877fe700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd98809a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:50.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.137+0000 7fd987fff700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 0x7fd98809c0b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7fd980000f40 tx=0x7fd980007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:50.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd980005ba0 con 0x7fd9880b1c30 2022-07-20T03:14:50.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd980005d00 con 0x7fd9880b1c30 2022-07-20T03:14:50.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9880ada10 con 0x7fd9880b1c30 2022-07-20T03:14:50.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd9800072c0 con 0x7fd9880b1c30 2022-07-20T03:14:50.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9881508e0 con 0x7fd9880b1c30 2022-07-20T03:14:50.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9880aff70 con 0x7fd9880b1c30 2022-07-20T03:14:50.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd98001a060 con 0x7fd9880b1c30 2022-07-20T03:14:50.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.141+0000 7fd984ff9700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd96c053940 0x7fd96c055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.145+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd98006b820 con 0x7fd9880b1c30 2022-07-20T03:14:50.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.145+0000 7fd9877fe700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd96c053940 0x7fd96c055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.145+0000 7fd9877fe700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd96c053940 0x7fd96c055e00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fd97801a000 tx=0x7fd978014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:50.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.149+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd980042780 con 0x7fd9880b1c30 2022-07-20T03:14:50.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.313+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fd988069a60 con 0x7fd9880b1c30 2022-07-20T03:14:50.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.321+0000 7fd984ff9700 1 -- 172.21.15.61:0/3634501216 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v28) v1 ==== 184+0+0 (secure 0 0 0) 0x7fd980018020 con 0x7fd9880b1c30 2022-07-20T03:14:50.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd96c053940 msgr2=0x7fd96c055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd96c053940 0x7fd96c055e00 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fd97801a000 tx=0x7fd978014000).stop 2022-07-20T03:14:50.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 msgr2=0x7fd98809c0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 0x7fd98809c0b0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7fd980000f40 tx=0x7fd980007d80).stop 2022-07-20T03:14:50.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 shutdown_connections 2022-07-20T03:14:50.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd96c053940 0x7fd96c055e00 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd988066cb0 0x7fd9880af5d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd9880b1c30 0x7fd98809c0b0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 --2- 172.21.15.61:0/3634501216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd98805cf40 0x7fd98809a930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 >> 172.21.15.61:0/3634501216 conn(0x7fd9880b7d60 msgr2=0x7fd9881409c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:50.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 shutdown_connections 2022-07-20T03:14:50.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.325+0000 7fd98de31700 1 -- 172.21.15.61:0/3634501216 wait complete. 2022-07-20T03:14:50.369 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:14:50.369 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:14:50.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 Processor -- start 2022-07-20T03:14:50.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 -- start start 2022-07-20T03:14:50.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcdc00acc10 0x7fcdc00acff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcdc00966b0 0x7fcdc00b0c20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcdc00abde0 con 0x7fcdc0095b10 2022-07-20T03:14:50.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc0096c80 con 0x7fcdc00acc10 2022-07-20T03:14:50.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc6356700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcdc0098910 con 0x7fcdc00966b0 2022-07-20T03:14:50.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcdc00acc10 0x7fcdc00acff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcdc00acc10 0x7fcdc00acff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47494/0 (socket says 172.21.15.61:47494) 2022-07-20T03:14:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 -- 172.21.15.61:0/3245241768 learned_addr learned my addr 172.21.15.61:0/3245241768 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5b55700 1 --2- 172.21.15.61:0/3245241768 >> v2:172.21.15.117:3300/0 conn(0x7fcdc00966b0 0x7fcdc00b0c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 -- 172.21.15.61:0/3245241768 >> v2:172.21.15.117:3300/0 conn(0x7fcdc00966b0 msgr2=0x7fcdc00b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 --2- 172.21.15.61:0/3245241768 >> v2:172.21.15.117:3300/0 conn(0x7fcdc00966b0 0x7fcdc00b0c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 -- 172.21.15.61:0/3245241768 >> v1:172.21.15.117:6789/0 conn(0x7fcdc0095b10 legacy=0x7fcdc0095fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 -- 172.21.15.61:0/3245241768 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdc0098b20 con 0x7fcdc00acc10 2022-07-20T03:14:50.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdc5354700 1 --2- 172.21.15.61:0/3245241768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc00acc10 0x7fcdc00acff0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7fcdbc002f70 tx=0x7fcdbc006eb0).ready entity=mon.0 client_cookie=b05219fe7b0b811e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdb7fff700 1 -- 172.21.15.61:0/3245241768 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdbc00d610 con 0x7fcdc00acc10 2022-07-20T03:14:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdb7fff700 1 -- 172.21.15.61:0/3245241768 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc000e90 con 0x7fcdc00acc10 2022-07-20T03:14:50.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.541+0000 7fcdb7fff700 1 -- 172.21.15.61:0/3245241768 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdbc00d9a0 con 0x7fcdc00acc10 2022-07-20T03:14:50.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- 172.21.15.61:0/3245241768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc00acc10 msgr2=0x7fcdc00acff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 --2- 172.21.15.61:0/3245241768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc00acc10 0x7fcdc00acff0 secure :-1 s=READY pgs=826 cs=0 l=1 rev1=1 rx=0x7fcdbc002f70 tx=0x7fcdbc006eb0).stop 2022-07-20T03:14:50.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- 172.21.15.61:0/3245241768 shutdown_connections 2022-07-20T03:14:50.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 --2- 172.21.15.61:0/3245241768 >> v2:172.21.15.117:3300/0 conn(0x7fcdc00966b0 0x7fcdc00b0c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 --2- 172.21.15.61:0/3245241768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc00acc10 0x7fcdc00acff0 unknown :-1 s=CLOSED pgs=826 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- 172.21.15.61:0/3245241768 >> 172.21.15.61:0/3245241768 conn(0x7fcdc00b6210 msgr2=0x7fcdc00b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:50.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- 172.21.15.61:0/3245241768 shutdown_connections 2022-07-20T03:14:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- 172.21.15.61:0/3245241768 wait complete. 2022-07-20T03:14:50.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 Processor -- start 2022-07-20T03:14:50.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- start start 2022-07-20T03:14:50.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 0x7fcdc00996b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdc00966b0 0x7fcdc0141540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdc00acc10 0x7fcdc0141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc00af5e0 con 0x7fcdc0095b10 2022-07-20T03:14:50.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc00af880 con 0x7fcdc00acc10 2022-07-20T03:14:50.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc6356700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcdc00ad960 con 0x7fcdc00966b0 2022-07-20T03:14:50.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 0x7fcdc00996b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 0x7fcdc00996b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47500/0 (socket says 172.21.15.61:47500) 2022-07-20T03:14:50.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 -- 172.21.15.61:0/4023615726 learned_addr learned my addr 172.21.15.61:0/4023615726 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:50.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc4b53700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdc00966b0 0x7fcdc0141540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5b55700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdc00acc10 0x7fcdc0141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 -- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdc00966b0 msgr2=0x7fcdc0141540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdc00966b0 0x7fcdc0141540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 -- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdc00acc10 msgr2=0x7fcdc0141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdc00acc10 0x7fcdc0141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.545+0000 7fcdc5354700 1 -- 172.21.15.61:0/4023615726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdbc00b1e0 con 0x7fcdc0095b10 2022-07-20T03:14:50.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdc5354700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 0x7fcdc00996b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fcdbc002f70 tx=0x7fcdbc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:50.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdbc011860 con 0x7fcdc0095b10 2022-07-20T03:14:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdbc0119c0 con 0x7fcdc0095b10 2022-07-20T03:14:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdc00ae3b0 con 0x7fcdc0095b10 2022-07-20T03:14:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdbc011d20 con 0x7fcdc0095b10 2022-07-20T03:14:50.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdc0140980 con 0x7fcdc0095b10 2022-07-20T03:14:50.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcdbc025030 con 0x7fcdc0095b10 2022-07-20T03:14:50.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdc00af420 con 0x7fcdc0095b10 2022-07-20T03:14:50.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.549+0000 7fcdb67fc700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcda8053a40 0x7fcda8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:50.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.553+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcdbc06d1e0 con 0x7fcdc0095b10 2022-07-20T03:14:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.553+0000 7fcdc4b53700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcda8053a40 0x7fcda8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.553+0000 7fcdc4b53700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcda8053a40 0x7fcda8055f00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fcdb000d000 tx=0x7fcdb0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:50.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.557+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcdbc044030 con 0x7fcdc0095b10 2022-07-20T03:14:50.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.717+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcdc00ae6c0 con 0x7fcdc0095b10 2022-07-20T03:14:50.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.725+0000 7fcdb67fc700 1 -- 172.21.15.61:0/4023615726 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v29) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcdbc023020 con 0x7fcdc0095b10 2022-07-20T03:14:50.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcda8053a40 msgr2=0x7fcda8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcda8053a40 0x7fcda8055f00 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fcdb000d000 tx=0x7fcdb0007b90).stop 2022-07-20T03:14:50.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 msgr2=0x7fcdc00996b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:50.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 0x7fcdc00996b0 secure :-1 s=READY pgs=827 cs=0 l=1 rev1=1 rx=0x7fcdbc002f70 tx=0x7fcdbc015000).stop 2022-07-20T03:14:50.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 shutdown_connections 2022-07-20T03:14:50.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcda8053a40 0x7fcda8055f00 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcdc00acc10 0x7fcdc0141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcdc00966b0 0x7fcdc0141540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 --2- 172.21.15.61:0/4023615726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcdc0095b10 0x7fcdc00996b0 unknown :-1 s=CLOSED pgs=827 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:50.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 >> 172.21.15.61:0/4023615726 conn(0x7fcdc00b6210 msgr2=0x7fcdc00b0bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:50.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 shutdown_connections 2022-07-20T03:14:50.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.729+0000 7fcdc6356700 1 -- 172.21.15.61:0/4023615726 wait complete. 2022-07-20T03:14:50.837 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:14:50.838 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:14:51.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 Processor -- start 2022-07-20T03:14:51.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 -- start start 2022-07-20T03:14:51.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0b6800 0x7f9fbc0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9fbc0ad0e0 0x7f9fbc1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9fbc0ae280 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9fbc0b6be0 con 0x7f9fbc0ad0e0 2022-07-20T03:14:51.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc2244700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc0ae4b0 con 0x7f9fbc0b6800 2022-07-20T03:14:51.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1a43700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9fbc0ad0e0 0x7f9fbc1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc0a41700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9fbc0b3cb0 0x7f9fbc145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44110/0 (socket says 172.21.15.61:44110) 2022-07-20T03:14:51.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc0a41700 1 -- 172.21.15.61:0/2779895072 learned_addr learned my addr 172.21.15.61:0/2779895072 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:51.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1242700 1 --2- 172.21.15.61:0/2779895072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0b6800 0x7f9fbc0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1a43700 1 -- 172.21.15.61:0/2779895072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0b6800 msgr2=0x7f9fbc0aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:51.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1a43700 1 --2- 172.21.15.61:0/2779895072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0b6800 0x7f9fbc0aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1a43700 1 -- 172.21.15.61:0/2779895072 >> v1:172.21.15.61:6789/0 conn(0x7f9fbc0b3cb0 legacy=0x7f9fbc145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1a43700 1 -- 172.21.15.61:0/2779895072 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fbc0ae750 con 0x7f9fbc0ad0e0 2022-07-20T03:14:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.977+0000 7f9fc1a43700 1 --2- 172.21.15.61:0/2779895072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0ad0e0 0x7f9fbc1418f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f9fb40032f0 tx=0x7f9fb40091d0).ready entity=mon.0 client_cookie=acb1b3945a45a4c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:51.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fbbfff700 1 -- 172.21.15.61:0/2779895072 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb400b6c0 con 0x7f9fbc0ad0e0 2022-07-20T03:14:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fbbfff700 1 -- 172.21.15.61:0/2779895072 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb400b820 con 0x7f9fbc0ad0e0 2022-07-20T03:14:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fbbfff700 1 -- 172.21.15.61:0/2779895072 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb400b990 con 0x7f9fbc0ad0e0 2022-07-20T03:14:51.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 -- 172.21.15.61:0/2779895072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0ad0e0 msgr2=0x7f9fbc1418f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 --2- 172.21.15.61:0/2779895072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0ad0e0 0x7f9fbc1418f0 secure :-1 s=READY pgs=829 cs=0 l=1 rev1=1 rx=0x7f9fb40032f0 tx=0x7f9fb40091d0).stop 2022-07-20T03:14:51.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 -- 172.21.15.61:0/2779895072 shutdown_connections 2022-07-20T03:14:51.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 --2- 172.21.15.61:0/2779895072 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0ad0e0 0x7f9fbc1418f0 unknown :-1 s=CLOSED pgs=829 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 --2- 172.21.15.61:0/2779895072 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0b6800 0x7f9fbc0aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 -- 172.21.15.61:0/2779895072 >> 172.21.15.61:0/2779895072 conn(0x7f9fbc0b9dd0 msgr2=0x7f9fbc0b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:51.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 -- 172.21.15.61:0/2779895072 shutdown_connections 2022-07-20T03:14:51.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 -- 172.21.15.61:0/2779895072 wait complete. 2022-07-20T03:14:51.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 Processor -- start 2022-07-20T03:14:51.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.981+0000 7f9fc2244700 1 -- start start 2022-07-20T03:14:51.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0ad0e0 0x7f9fbc155450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 0x7f9fbc0af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fbc0b6800 0x7f9fbc1514b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc13f8f0 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc1151d0 con 0x7f9fbc0b6800 2022-07-20T03:14:51.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9fbc152040 con 0x7f9fbc0ad0e0 2022-07-20T03:14:51.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 0x7f9fbc0af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 0x7f9fbc0af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47516/0 (socket says 172.21.15.61:47516) 2022-07-20T03:14:51.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 -- 172.21.15.61:0/3129441896 learned_addr learned my addr 172.21.15.61:0/3129441896 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:51.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc1242700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0ad0e0 0x7f9fbc155450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc1a43700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fbc0b6800 0x7f9fbc1514b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 -- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0ad0e0 msgr2=0x7f9fbc155450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0ad0e0 0x7f9fbc155450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 -- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fbc0b6800 msgr2=0x7f9fbc1514b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fbc0b6800 0x7f9fbc1514b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 -- 172.21.15.61:0/3129441896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9fb4009470 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc0a41700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 0x7f9fbc0af9c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f9fbc066600 tx=0x7f9fb0007c40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:51.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb00070d0 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9fb0007230 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9fbc0afeb0 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9fb0005610 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.985+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9fbc0b23e0 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.989+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9fbc115390 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.989+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9fb0010070 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.989+0000 7f9fba7fc700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053ad0 0x7f9f9c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.989+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9fb006b3e0 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.989+0000 7f9fc1242700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053ad0 0x7f9f9c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.993+0000 7f9fc1242700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053ad0 0x7f9f9c055f90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f9fac00b000 tx=0x7f9fac006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:50.993+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9fb0042340 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.157+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f9fbc152200 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.165+0000 7f9fba7fc700 1 -- 172.21.15.61:0/3129441896 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v30) v1 ==== 184+0+0 (secure 0 0 0) 0x7f9fb0045960 con 0x7f9fbc0b3cb0 2022-07-20T03:14:51.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053ad0 msgr2=0x7f9f9c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053ad0 0x7f9f9c055f90 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f9fac00b000 tx=0x7f9fac006df0).stop 2022-07-20T03:14:51.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 msgr2=0x7f9fbc0af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 0x7f9fbc0af9c0 secure :-1 s=READY pgs=830 cs=0 l=1 rev1=1 rx=0x7f9fbc066600 tx=0x7f9fb0007c40).stop 2022-07-20T03:14:51.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 shutdown_connections 2022-07-20T03:14:51.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f9c053ad0 0x7f9f9c055f90 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9fbc0b6800 0x7f9fbc1514b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9fbc0b3cb0 0x7f9fbc0af9c0 unknown :-1 s=CLOSED pgs=830 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 --2- 172.21.15.61:0/3129441896 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9fbc0ad0e0 0x7f9fbc155450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 >> 172.21.15.61:0/3129441896 conn(0x7f9fbc0b9dd0 msgr2=0x7f9fbc140300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:51.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 shutdown_connections 2022-07-20T03:14:51.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.169+0000 7f9fc2244700 1 -- 172.21.15.61:0/3129441896 wait complete. 2022-07-20T03:14:51.207 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:14:51.207 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:51.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 Processor -- start 2022-07-20T03:14:51.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 -- start start 2022-07-20T03:14:51.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f328c0adff0 0x7f328c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0af8e0 0x7f328c05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f328c147010 con 0x7f328c0b5650 2022-07-20T03:14:51.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f328c1472b0 con 0x7f328c0adff0 2022-07-20T03:14:51.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c146d40 con 0x7f328c0af8e0 2022-07-20T03:14:51.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f328c0adff0 0x7f328c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3291c74700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0af8e0 0x7f328c05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f328c0adff0 0x7f328c0b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:57202/0 (socket says 172.21.15.61:57202) 2022-07-20T03:14:51.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 -- 172.21.15.61:0/2072294761 learned_addr learned my addr 172.21.15.61:0/2072294761 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:51.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 -- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0af8e0 msgr2=0x7f328c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:51.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 --2- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0af8e0 0x7f328c05bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 -- 172.21.15.61:0/2072294761 >> v1:172.21.15.117:6789/0 conn(0x7f328c0b5650 legacy=0x7f328c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:51.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 -- 172.21.15.61:0/2072294761 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f328c0acae0 con 0x7f328c0adff0 2022-07-20T03:14:51.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3290c72700 1 --2- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b5a30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f32880032f0 tx=0x7f3288009250).ready entity=mon.1 client_cookie=d5f4459c583ea0b1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:51.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3283fff700 1 -- 172.21.15.61:0/2072294761 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3288009ea0 con 0x7f328c0adff0 2022-07-20T03:14:51.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3283fff700 1 -- 172.21.15.61:0/2072294761 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f328800b8a0 con 0x7f328c0adff0 2022-07-20T03:14:51.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3283fff700 1 -- 172.21.15.61:0/2072294761 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f328800ba10 con 0x7f328c0adff0 2022-07-20T03:14:51.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 -- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 msgr2=0x7f328c0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.313+0000 7f3292475700 1 --2- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b5a30 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f32880032f0 tx=0x7f3288009250).stop 2022-07-20T03:14:51.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 -- 172.21.15.61:0/2072294761 shutdown_connections 2022-07-20T03:14:51.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 --2- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0af8e0 0x7f328c05bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 --2- 172.21.15.61:0/2072294761 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b5a30 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 -- 172.21.15.61:0/2072294761 >> 172.21.15.61:0/2072294761 conn(0x7f328c0634e0 msgr2=0x7f328c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:51.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 -- 172.21.15.61:0/2072294761 shutdown_connections 2022-07-20T03:14:51.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 -- 172.21.15.61:0/2072294761 wait complete. 2022-07-20T03:14:51.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 Processor -- start 2022-07-20T03:14:51.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.317+0000 7f3292475700 1 -- start start 2022-07-20T03:14:51.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b0900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 0x7f328c066250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0b5650 0x7f328c066740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c111dd0 con 0x7f328c0af8e0 2022-07-20T03:14:51.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f328c063b10 con 0x7f328c0adff0 2022-07-20T03:14:51.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f328c111090 con 0x7f328c0b5650 2022-07-20T03:14:51.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 0x7f328c066250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 0x7f328c066250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47530/0 (socket says 172.21.15.61:47530) 2022-07-20T03:14:51.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3291c74700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0b5650 0x7f328c066740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 -- 172.21.15.61:0/599678629 learned_addr learned my addr 172.21.15.61:0/599678629 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:51.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3291473700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b0900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 -- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0b5650 msgr2=0x7f328c066740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0b5650 0x7f328c066740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 -- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 msgr2=0x7f328c0b0900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b0900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 -- 172.21.15.61:0/599678629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32880094d0 con 0x7f328c0af8e0 2022-07-20T03:14:51.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3291c74700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0b5650 0x7f328c066740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:51.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3290c72700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 0x7f328c066250 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f32880032f0 tx=0x7f3288016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:51.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3288006440 con 0x7f328c0af8e0 2022-07-20T03:14:51.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32880065a0 con 0x7f328c0af8e0 2022-07-20T03:14:51.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f328c110510 con 0x7f328c0af8e0 2022-07-20T03:14:51.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3288006900 con 0x7f328c0af8e0 2022-07-20T03:14:51.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f328c063ec0 con 0x7f328c0af8e0 2022-07-20T03:14:51.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f328c111f40 con 0x7f328c0af8e0 2022-07-20T03:14:51.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.321+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3288027020 con 0x7f328c0af8e0 2022-07-20T03:14:51.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.325+0000 7f32827fc700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32740539b0 0x7f3274055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:51.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.325+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f328806d210 con 0x7f328c0af8e0 2022-07-20T03:14:51.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.325+0000 7f3291473700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32740539b0 0x7f3274055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:51.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.325+0000 7f3291473700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32740539b0 0x7f3274055e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f327c00d000 tx=0x7f327c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:51.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.329+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3288044170 con 0x7f328c0af8e0 2022-07-20T03:14:51.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f328c0574e0 con 0x7f328c0af8e0 2022-07-20T03:14:51.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f32827fc700 1 -- 172.21.15.61:0/599678629 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f3288015020 con 0x7f328c0af8e0 2022-07-20T03:14:51.503 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:51.504 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:14:49.076244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6643":{"gid":6643,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/843120801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":843120801},{"type":"v1","addr":"172.21.15.117:6835","nonce":843120801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:14:51.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32740539b0 msgr2=0x7f3274055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32740539b0 0x7f3274055e70 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f327c00d000 tx=0x7f327c009040).stop 2022-07-20T03:14:51.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 msgr2=0x7f328c066250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:51.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 0x7f328c066250 secure :-1 s=READY pgs=831 cs=0 l=1 rev1=1 rx=0x7f32880032f0 tx=0x7f3288016000).stop 2022-07-20T03:14:51.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 shutdown_connections 2022-07-20T03:14:51.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32740539b0 0x7f3274055e70 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f328c0b5650 0x7f328c066740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f328c0af8e0 0x7f328c066250 unknown :-1 s=CLOSED pgs=831 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 --2- 172.21.15.61:0/599678629 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f328c0adff0 0x7f328c0b0900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:51.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.493+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 >> 172.21.15.61:0/599678629 conn(0x7f328c0634e0 msgr2=0x7f328c0ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:51.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.497+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 shutdown_connections 2022-07-20T03:14:51.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.497+0000 7f3292475700 1 -- 172.21.15.61:0/599678629 wait complete. 2022-07-20T03:14:51.529 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-20T03:14:51.734 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:14:48.072382+0000', 'modified': '2022-07-20T03:14:49.076244+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6643}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6643': {'gid': 6643, 'name': 'b', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/843120801', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 843120801}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 843120801}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:14:51.735 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:14:51.736 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:14:51.874 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6643,"want_state":"up:active","state":"up:active","rank_uptime":3.7825521339999999,"mdsmap_epoch":45,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":8.1815933449999996} 2022-07-20T03:14:51.875 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6643, "mdsmap_epoch": 45, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 3.782552134, "state": "up:active", "uptime": 8.181593345, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:14:51.878 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:52.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.997+0000 7fa91ac26700 1 Processor -- start 2022-07-20T03:14:52.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:51.997+0000 7fa91ac26700 1 -- start start 2022-07-20T03:14:52.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa9140b1200 0x7fa9140b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa91405bd90 0x7fa914142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9140aea30 con 0x7fa9140b5650 2022-07-20T03:14:52.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa914142970 con 0x7fa91405bd90 2022-07-20T03:14:52.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa9140aec60 con 0x7fa9140b1200 2022-07-20T03:14:52.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa91405bd90 0x7fa914142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa91405bd90 0x7fa914142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47540/0 (socket says 172.21.15.61:47540) 2022-07-20T03:14:52.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 -- 172.21.15.61:0/1378478033 learned_addr learned my addr 172.21.15.61:0/1378478033 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:52.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa919423700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa9140b1200 0x7fa9140b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 -- 172.21.15.61:0/1378478033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa9140b1200 msgr2=0x7fa9140b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:52.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa9140b1200 0x7fa9140b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 -- 172.21.15.61:0/1378478033 >> v1:172.21.15.117:6789/0 conn(0x7fa9140b5650 legacy=0x7fa91405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 -- 172.21.15.61:0/1378478033 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9140af180 con 0x7fa91405bd90 2022-07-20T03:14:52.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa919423700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa9140b1200 0x7fa9140b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:52.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91a425700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa91405bd90 0x7fa914142d60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7fa90c0032f0 tx=0x7fa90c009290).ready entity=mon.0 client_cookie=5473dc1c332192f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa918c22700 1 -- 172.21.15.61:0/1378478033 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa90c00b8a0 con 0x7fa91405bd90 2022-07-20T03:14:52.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa918c22700 1 -- 172.21.15.61:0/1378478033 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c00ba00 con 0x7fa91405bd90 2022-07-20T03:14:52.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa918c22700 1 -- 172.21.15.61:0/1378478033 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa90c00bb70 con 0x7fa91405bd90 2022-07-20T03:14:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- 172.21.15.61:0/1378478033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa91405bd90 msgr2=0x7fa914142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa91405bd90 0x7fa914142d60 secure :-1 s=READY pgs=832 cs=0 l=1 rev1=1 rx=0x7fa90c0032f0 tx=0x7fa90c009290).stop 2022-07-20T03:14:52.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- 172.21.15.61:0/1378478033 shutdown_connections 2022-07-20T03:14:52.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa91405bd90 0x7fa914142d60 unknown :-1 s=CLOSED pgs=832 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 --2- 172.21.15.61:0/1378478033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa9140b1200 0x7fa9140b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- 172.21.15.61:0/1378478033 >> 172.21.15.61:0/1378478033 conn(0x7fa9140634e0 msgr2=0x7fa9140b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:52.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- 172.21.15.61:0/1378478033 shutdown_connections 2022-07-20T03:14:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.001+0000 7fa91ac26700 1 -- 172.21.15.61:0/1378478033 wait complete. 2022-07-20T03:14:52.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 Processor -- start 2022-07-20T03:14:52.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- start start 2022-07-20T03:14:52.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa91405bd90 0x7fa914152e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9140b1200 0x7fa91413f6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 0x7fa91413d230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa914146000 con 0x7fa9140b5650 2022-07-20T03:14:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa9141451d0 con 0x7fa9140b1200 2022-07-20T03:14:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa914144f30 con 0x7fa91405bd90 2022-07-20T03:14:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 0x7fa91413d230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 0x7fa91413d230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47546/0 (socket says 172.21.15.61:47546) 2022-07-20T03:14:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 -- 172.21.15.61:0/2092730109 learned_addr learned my addr 172.21.15.61:0/2092730109 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa919c24700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa91405bd90 0x7fa914152e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa919423700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9140b1200 0x7fa91413f6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 -- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa91405bd90 msgr2=0x7fa914152e00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa91405bd90 0x7fa914152e00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 -- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9140b1200 msgr2=0x7fa91413f6c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9140b1200 0x7fa91413f6c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 -- 172.21.15.61:0/2092730109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa90c009530 con 0x7fa9140b5650 2022-07-20T03:14:52.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa919423700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9140b1200 0x7fa91413f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:52.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91a425700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 0x7fa91413d230 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fa90c0032f0 tx=0x7fa90c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa90c00bb70 con 0x7fa9140b5650 2022-07-20T03:14:52.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa90c020470 con 0x7fa9140b5650 2022-07-20T03:14:52.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa914145d60 con 0x7fa9140b5650 2022-07-20T03:14:52.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa90c0207d0 con 0x7fa9140b5650 2022-07-20T03:14:52.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa914144b30 con 0x7fa9140b5650 2022-07-20T03:14:52.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.005+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa9140574e0 con 0x7fa9140b5650 2022-07-20T03:14:52.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.009+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa90c00f070 con 0x7fa9140b5650 2022-07-20T03:14:52.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.009+0000 7fa902ffd700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8f4053af0 0x7fa8f4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.009+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa90c06cc30 con 0x7fa9140b5650 2022-07-20T03:14:52.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.009+0000 7fa919c24700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8f4053af0 0x7fa8f4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.009+0000 7fa919c24700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8f4053af0 0x7fa8f4055fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fa90400d000 tx=0x7fa904000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.013+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa90c073080 con 0x7fa9140b5650 2022-07-20T03:14:52.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.141+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa914067ea0 con 0x7fa9140b5650 2022-07-20T03:14:52.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa902ffd700 1 -- 172.21.15.61:0/2092730109 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16089 (secure 0 0 0) 0x7fa90c015070 con 0x7fa9140b5650 2022-07-20T03:14:52.214 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:52.215 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":81,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:49.075215+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:14:46.661180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T03:14:47.668660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:52.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8f4053af0 msgr2=0x7fa8f4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8f4053af0 0x7fa8f4055fb0 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7fa90400d000 tx=0x7fa904000bc0).stop 2022-07-20T03:14:52.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 msgr2=0x7fa91413d230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 0x7fa91413d230 secure :-1 s=READY pgs=833 cs=0 l=1 rev1=1 rx=0x7fa90c0032f0 tx=0x7fa90c016000).stop 2022-07-20T03:14:52.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 shutdown_connections 2022-07-20T03:14:52.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa8f4053af0 0x7fa8f4055fb0 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9140b1200 0x7fa91413f6c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9140b5650 0x7fa91413d230 unknown :-1 s=CLOSED pgs=833 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 --2- 172.21.15.61:0/2092730109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa91405bd90 0x7fa914152e00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 >> 172.21.15.61:0/2092730109 conn(0x7fa9140634e0 msgr2=0x7fa9140b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:52.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 shutdown_connections 2022-07-20T03:14:52.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.145+0000 7fa91ac26700 1 -- 172.21.15.61:0/2092730109 wait complete. 2022-07-20T03:14:52.293 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:14:46.661180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:14:47.668660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:14:52.301 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:14:52.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.441+0000 7fbebe775700 1 Processor -- start 2022-07-20T03:14:52.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.441+0000 7fbebe775700 1 -- start start 2022-07-20T03:14:52.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebe775700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 0x7fbeb81407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebe775700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbeb80aef30 0x7fbeb8129340 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebe775700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbeb80b5dd0 con 0x7fbeb8140cd0 2022-07-20T03:14:52.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebe775700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbeb80b1750 con 0x7fbeb80aef30 2022-07-20T03:14:52.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebe775700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb80652a0 con 0x7fbeb80b59f0 2022-07-20T03:14:52.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebcf72700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbeb8140cd0 0x7fbeb81445a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44148/0 (socket says 172.21.15.61:44148) 2022-07-20T03:14:52.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebcf72700 1 -- 172.21.15.61:0/1505639737 learned_addr learned my addr 172.21.15.61:0/1505639737 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:52.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebdf74700 1 --2- 172.21.15.61:0/1505639737 >> v2:172.21.15.117:3300/0 conn(0x7fbeb80aef30 0x7fbeb8129340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 0x7fbeb81407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2504916846 0 0) 0x7fbeb80b5dd0 con 0x7fbeb8140cd0 2022-07-20T03:14:52.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea0003260 con 0x7fbeb8140cd0 2022-07-20T03:14:52.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2270855882 0 0) 0x7fbea0003260 con 0x7fbeb8140cd0 2022-07-20T03:14:52.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 msgr2=0x7fbeb81407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:52.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 0x7fbeb81407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 >> v2:172.21.15.117:3300/0 conn(0x7fbeb80aef30 msgr2=0x7fbeb8129340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:52.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 --2- 172.21.15.61:0/1505639737 >> v2:172.21.15.117:3300/0 conn(0x7fbeb80aef30 0x7fbeb8129340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb8143e60 con 0x7fbeb8140cd0 2022-07-20T03:14:52.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fbea8001660 con 0x7fbeb8140cd0 2022-07-20T03:14:52.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebdf74700 1 --2- 172.21.15.61:0/1505639737 >> v2:172.21.15.117:3300/0 conn(0x7fbeb80aef30 0x7fbeb8129340 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:52.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 >> v1:172.21.15.61:6789/0 conn(0x7fbeb8140cd0 legacy=0x7fbeb81445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea0003750 0x7fbea0003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0004500 0x7fbea0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 0x7fbea0004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb80b1750 con 0x7fbea0003750 2022-07-20T03:14:52.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb80652a0 con 0x7fbea0004500 2022-07-20T03:14:52.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea000b750 con 0x7fbeb80b59f0 2022-07-20T03:14:52.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea0003750 0x7fbea0003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebcf72700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0004500 0x7fbea0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebdf74700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 0x7fbea0004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 -- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 msgr2=0x7fbea0004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:52.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb80b59f0 0x7fbea0004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 -- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0004500 msgr2=0x7fbea0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:52.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0004500 0x7fbea0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 -- 172.21.15.61:0/1505639737 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0003260 con 0x7fbea0003750 2022-07-20T03:14:52.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebcf72700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0004500 0x7fbea0004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:52.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbebd773700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea0003750 0x7fbea0003b70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fbeb400c000 tx=0x7fbeb4006df0).ready entity=mon.0 client_cookie=97f7e557c806badd server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb400e830 con 0x7fbea0003750 2022-07-20T03:14:52.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb400e990 con 0x7fbea0003750 2022-07-20T03:14:52.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.445+0000 7fbea7fff700 1 -- 172.21.15.61:0/1505639737 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb400ecf0 con 0x7fbea0003750 2022-07-20T03:14:52.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- 172.21.15.61:0/1505639737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea0003750 msgr2=0x7fbea0003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 --2- 172.21.15.61:0/1505639737 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea0003750 0x7fbea0003b70 secure :-1 s=READY pgs=835 cs=0 l=1 rev1=1 rx=0x7fbeb400c000 tx=0x7fbeb4006df0).stop 2022-07-20T03:14:52.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 -- 172.21.15.61:0/1505639737 reap_dead start 2022-07-20T03:14:52.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- 172.21.15.61:0/1505639737 shutdown_connections 2022-07-20T03:14:52.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- 172.21.15.61:0/1505639737 >> 172.21.15.61:0/1505639737 conn(0x7fbeb8063480 msgr2=0x7fbeb80b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:52.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- 172.21.15.61:0/1505639737 shutdown_connections 2022-07-20T03:14:52.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- 172.21.15.61:0/1505639737 wait complete. 2022-07-20T03:14:52.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 Processor -- start 2022-07-20T03:14:52.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- start start 2022-07-20T03:14:52.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0003750 0x7fbeb80b4c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb8131540 0x7fbeb8134630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 0x7fbeb8130180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8134b20 con 0x7fbeb8155900 2022-07-20T03:14:52.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb81320a0 con 0x7fbea0003750 2022-07-20T03:14:52.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebe775700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbeb8132490 con 0x7fbeb8131540 2022-07-20T03:14:52.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 0x7fbeb8130180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 0x7fbeb8130180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47566/0 (socket says 172.21.15.61:47566) 2022-07-20T03:14:52.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 -- 172.21.15.61:0/3199974768 learned_addr learned my addr 172.21.15.61:0/3199974768 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebcf72700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb8131540 0x7fbeb8134630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebd773700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0003750 0x7fbeb80b4c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 -- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb8131540 msgr2=0x7fbeb8134630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb8131540 0x7fbeb8134630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 -- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0003750 msgr2=0x7fbeb80b4c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0003750 0x7fbeb80b4c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.449+0000 7fbebdf74700 1 -- 172.21.15.61:0/3199974768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbeb8132730 con 0x7fbeb8155900 2022-07-20T03:14:52.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbebdf74700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 0x7fbeb8130180 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fbeb000b000 tx=0x7fbeb00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb001a070 con 0x7fbeb8155900 2022-07-20T03:14:52.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbeb000f9e0 con 0x7fbeb8155900 2022-07-20T03:14:52.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbeb8131c00 con 0x7fbeb8155900 2022-07-20T03:14:52.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbeb000fd40 con 0x7fbeb8155900 2022-07-20T03:14:52.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbeb8130670 con 0x7fbeb8155900 2022-07-20T03:14:52.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbeb0014440 con 0x7fbeb8155900 2022-07-20T03:14:52.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbeb81322f0 con 0x7fbeb8155900 2022-07-20T03:14:52.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbea67fc700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe98053a40 0x7fbe98055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbeb006c690 con 0x7fbeb8155900 2022-07-20T03:14:52.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.453+0000 7fbebd773700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe98053a40 0x7fbe98055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.457+0000 7fbebd773700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe98053a40 0x7fbe98055f00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fbeb401e000 tx=0x7fbeb4019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.461+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbeb81322f0 con 0x7fbeb8155900 2022-07-20T03:14:52.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.589+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fbeb81322f0 con 0x7fbeb8155900 2022-07-20T03:14:52.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.589+0000 7fbea67fc700 1 -- 172.21.15.61:0/3199974768 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7fbeb81322f0 con 0x7fbeb8155900 2022-07-20T03:14:52.679 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:14:52.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.589+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe98053a40 msgr2=0x7fbe98055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.589+0000 7fbebe775700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe98053a40 0x7fbe98055f00 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fbeb401e000 tx=0x7fbeb4019000).stop 2022-07-20T03:14:52.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 msgr2=0x7fbeb8130180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 0x7fbeb8130180 secure :-1 s=READY pgs=836 cs=0 l=1 rev1=1 rx=0x7fbeb000b000 tx=0x7fbeb00091d0).stop 2022-07-20T03:14:52.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 shutdown_connections 2022-07-20T03:14:52.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe98053a40 0x7fbe98055f00 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbeb8155900 0x7fbeb8130180 unknown :-1 s=CLOSED pgs=836 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbeb8131540 0x7fbeb8134630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 --2- 172.21.15.61:0/3199974768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea0003750 0x7fbeb80b4c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 >> 172.21.15.61:0/3199974768 conn(0x7fbeb8063480 msgr2=0x7fbeb805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:52.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 shutdown_connections 2022-07-20T03:14:52.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.593+0000 7fbebe775700 1 -- 172.21.15.61:0/3199974768 wait complete. 2022-07-20T03:14:52.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:14:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.857+0000 7f64c314b700 1 Processor -- start 2022-07-20T03:14:52.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.857+0000 7f64c314b700 1 -- start start 2022-07-20T03:14:52.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f64bc0b59d0 0x7f64bc05d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f64bc128d70 0x7f64bc13fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64bc0b5db0 con 0x7f64bc0af180 2022-07-20T03:14:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc1291b0 con 0x7f64bc0b59d0 2022-07-20T03:14:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64bc0ad800 con 0x7f64bc128d70 2022-07-20T03:14:52.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c2149700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f64bc0b59d0 0x7f64bc05d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1948700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f64bc0af180 0x7f64bc128660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44170/0 (socket says 172.21.15.61:44170) 2022-07-20T03:14:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1948700 1 -- 172.21.15.61:0/3581610805 learned_addr learned my addr 172.21.15.61:0/3581610805 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c294a700 1 --2- 172.21.15.61:0/3581610805 >> v2:172.21.15.117:3300/0 conn(0x7f64bc128d70 0x7f64bc13fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1147700 1 -- 172.21.15.61:0/3581610805 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2688843628 0 0) 0x7f64bc0b5db0 con 0x7f64bc0af180 2022-07-20T03:14:52.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1147700 1 -- 172.21.15.61:0/3581610805 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64a4003260 con 0x7f64bc0af180 2022-07-20T03:14:52.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c2149700 1 -- 172.21.15.61:0/3581610805 >> v2:172.21.15.117:3300/0 conn(0x7f64bc128d70 msgr2=0x7f64bc13fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:52.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c2149700 1 --2- 172.21.15.61:0/3581610805 >> v2:172.21.15.117:3300/0 conn(0x7f64bc128d70 0x7f64bc13fc00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c2149700 1 -- 172.21.15.61:0/3581610805 >> v1:172.21.15.61:6789/0 conn(0x7f64bc0af180 legacy=0x7f64bc128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c2149700 1 -- 172.21.15.61:0/3581610805 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc0adaa0 con 0x7f64bc0b59d0 2022-07-20T03:14:52.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c2149700 1 --2- 172.21.15.61:0/3581610805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc0b59d0 0x7f64bc05d1c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f64b0002f70 tx=0x7f64b0006df0).ready entity=mon.0 client_cookie=90fab4e8330d6351 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1147700 1 -- 172.21.15.61:0/3581610805 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64b000d4f0 con 0x7f64bc0b59d0 2022-07-20T03:14:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1147700 1 -- 172.21.15.61:0/3581610805 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b0000e90 con 0x7f64bc0b59d0 2022-07-20T03:14:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c1147700 1 -- 172.21.15.61:0/3581610805 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64b000d840 con 0x7f64bc0b59d0 2022-07-20T03:14:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- 172.21.15.61:0/3581610805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc0b59d0 msgr2=0x7f64bc05d1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 --2- 172.21.15.61:0/3581610805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc0b59d0 0x7f64bc05d1c0 secure :-1 s=READY pgs=838 cs=0 l=1 rev1=1 rx=0x7f64b0002f70 tx=0x7f64b0006df0).stop 2022-07-20T03:14:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- 172.21.15.61:0/3581610805 shutdown_connections 2022-07-20T03:14:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 --2- 172.21.15.61:0/3581610805 >> v2:172.21.15.117:3300/0 conn(0x7f64bc128d70 0x7f64bc13fc00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 --2- 172.21.15.61:0/3581610805 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc0b59d0 0x7f64bc05d1c0 unknown :-1 s=CLOSED pgs=838 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- 172.21.15.61:0/3581610805 >> 172.21.15.61:0/3581610805 conn(0x7f64bc0634c0 msgr2=0x7f64bc0b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:52.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- 172.21.15.61:0/3581610805 shutdown_connections 2022-07-20T03:14:52.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.861+0000 7f64c314b700 1 -- 172.21.15.61:0/3581610805 wait complete. 2022-07-20T03:14:52.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 Processor -- start 2022-07-20T03:14:52.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 -- start start 2022-07-20T03:14:52.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64bc0af180 0x7f64bc126f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64bc0b59d0 0x7f64bc05c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 0x7f64bc125f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc123730 con 0x7f64bc128d70 2022-07-20T03:14:52.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc122a10 con 0x7f64bc0af180 2022-07-20T03:14:52.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c314b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64bc144a40 con 0x7f64bc0b59d0 2022-07-20T03:14:52.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 0x7f64bc125f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 0x7f64bc125f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47578/0 (socket says 172.21.15.61:47578) 2022-07-20T03:14:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 -- 172.21.15.61:0/804285846 learned_addr learned my addr 172.21.15.61:0/804285846 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c1948700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64bc0b59d0 0x7f64bc05c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c2149700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64bc0af180 0x7f64bc126f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 -- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64bc0b59d0 msgr2=0x7f64bc05c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64bc0b59d0 0x7f64bc05c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 -- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64bc0af180 msgr2=0x7f64bc126f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64bc0af180 0x7f64bc126f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:52.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c294a700 1 -- 172.21.15.61:0/804285846 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64b000b180 con 0x7f64bc128d70 2022-07-20T03:14:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.865+0000 7f64c2149700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64bc0af180 0x7f64bc126f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64c294a700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 0x7f64bc125f10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f64b4006e00 tx=0x7f64b40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64b400c670 con 0x7f64bc128d70 2022-07-20T03:14:52.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64b400c7d0 con 0x7f64bc128d70 2022-07-20T03:14:52.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64bc122770 con 0x7f64bc128d70 2022-07-20T03:14:52.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64b400cb30 con 0x7f64bc128d70 2022-07-20T03:14:52.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64bc124460 con 0x7f64bc128d70 2022-07-20T03:14:52.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f64b400cc90 con 0x7f64bc128d70 2022-07-20T03:14:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64bc123420 con 0x7f64bc128d70 2022-07-20T03:14:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64af7fe700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f649c0539b0 0x7f649c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f64b406c230 con 0x7f64bc128d70 2022-07-20T03:14:52.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.869+0000 7f64c2149700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f649c0539b0 0x7f649c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:52.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.873+0000 7f64c2149700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f649c0539b0 0x7f649c055e70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f64b0018000 tx=0x7f64b0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:52.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:52.877+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64b4043020 con 0x7f64bc128d70 2022-07-20T03:14:53.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.005+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f64bc0ad4f0 con 0x7f64bc128d70 2022-07-20T03:14:53.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.005+0000 7f64af7fe700 1 -- 172.21.15.61:0/804285846 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f64b4046640 con 0x7f64bc128d70 2022-07-20T03:14:53.012 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:53.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f649c0539b0 msgr2=0x7f649c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f649c0539b0 0x7f649c055e70 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f64b0018000 tx=0x7f64b0014000).stop 2022-07-20T03:14:53.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 msgr2=0x7f64bc125f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 0x7f64bc125f10 secure :-1 s=READY pgs=839 cs=0 l=1 rev1=1 rx=0x7f64b4006e00 tx=0x7f64b40091d0).stop 2022-07-20T03:14:53.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 shutdown_connections 2022-07-20T03:14:53.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f649c0539b0 0x7f649c055e70 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64bc128d70 0x7f64bc125f10 unknown :-1 s=CLOSED pgs=839 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64bc0b59d0 0x7f64bc05c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 --2- 172.21.15.61:0/804285846 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64bc0af180 0x7f64bc126f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 >> 172.21.15.61:0/804285846 conn(0x7f64bc0634c0 msgr2=0x7f64bc0b00f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 shutdown_connections 2022-07-20T03:14:53.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.009+0000 7f64c314b700 1 -- 172.21.15.61:0/804285846 wait complete. 2022-07-20T03:14:53.037 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:14:53.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.153+0000 7fc39a6de700 1 Processor -- start 2022-07-20T03:14:53.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.153+0000 7fc39a6de700 1 -- start start 2022-07-20T03:14:53.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.153+0000 7fc39a6de700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc3940b4d60 0x7fc394145d90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.153+0000 7fc39a6de700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3940607e0 con 0x7fc394060ee0 2022-07-20T03:14:53.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.153+0000 7fc39a6de700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc394060a80 con 0x7fc39409df40 2022-07-20T03:14:53.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc394060d20 con 0x7fc3940b4d60 2022-07-20T03:14:53.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc398c7b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc394060ee0 0x7fc39409cba0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44184/0 (socket says 172.21.15.61:44184) 2022-07-20T03:14:53.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc398c7b700 1 -- 172.21.15.61:0/610715045 learned_addr learned my addr 172.21.15.61:0/610715045 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:53.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc3937fe700 1 --2- 172.21.15.61:0/610715045 >> v2:172.21.15.117:3300/0 conn(0x7fc3940b4d60 0x7fc394145d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc392ffd700 1 -- 172.21.15.61:0/610715045 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1889261637 0 0) 0x7fc3940607e0 con 0x7fc394060ee0 2022-07-20T03:14:53.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc392ffd700 1 -- 172.21.15.61:0/610715045 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc380003260 con 0x7fc394060ee0 2022-07-20T03:14:53.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc3937fe700 1 -- 172.21.15.61:0/610715045 >> v1:172.21.15.117:6789/0 conn(0x7fc39409df40 legacy=0x7fc394145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc392ffd700 1 -- 172.21.15.61:0/610715045 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956953292 0 0) 0x7fc394060a80 con 0x7fc39409df40 2022-07-20T03:14:53.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc3937fe700 1 -- 172.21.15.61:0/610715045 >> v1:172.21.15.61:6789/0 conn(0x7fc394060ee0 legacy=0x7fc39409cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc3937fe700 1 -- 172.21.15.61:0/610715045 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc39409e320 con 0x7fc3940b4d60 2022-07-20T03:14:53.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc3937fe700 1 --2- 172.21.15.61:0/610715045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc394145d90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc388002f70 tx=0x7fc388006df0).ready entity=mon.1 client_cookie=62a3f60bea4eaf3f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc392ffd700 1 -- 172.21.15.61:0/610715045 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc38800d610 con 0x7fc3940b4d60 2022-07-20T03:14:53.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc392ffd700 1 -- 172.21.15.61:0/610715045 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc388000e90 con 0x7fc3940b4d60 2022-07-20T03:14:53.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc392ffd700 1 -- 172.21.15.61:0/610715045 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc38800d960 con 0x7fc3940b4d60 2022-07-20T03:14:53.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 -- 172.21.15.61:0/610715045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 msgr2=0x7fc394145d90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 --2- 172.21.15.61:0/610715045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc394145d90 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7fc388002f70 tx=0x7fc388006df0).stop 2022-07-20T03:14:53.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 -- 172.21.15.61:0/610715045 shutdown_connections 2022-07-20T03:14:53.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 --2- 172.21.15.61:0/610715045 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc394145d90 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 -- 172.21.15.61:0/610715045 >> 172.21.15.61:0/610715045 conn(0x7fc3940b5f30 msgr2=0x7fc3940b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 -- 172.21.15.61:0/610715045 shutdown_connections 2022-07-20T03:14:53.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.157+0000 7fc39a6de700 1 -- 172.21.15.61:0/610715045 wait complete. 2022-07-20T03:14:53.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 Processor -- start 2022-07-20T03:14:53.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 -- start start 2022-07-20T03:14:53.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc394060ee0 0x7fc3941412e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 0x7fc3940b23d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc3940b28c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc394152d40 con 0x7fc39409df40 2022-07-20T03:14:53.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3940b2f20 con 0x7fc3940b4d60 2022-07-20T03:14:53.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc3940ae970 con 0x7fc394060ee0 2022-07-20T03:14:53.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 0x7fc3940b23d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 0x7fc3940b23d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47590/0 (socket says 172.21.15.61:47590) 2022-07-20T03:14:53.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 -- 172.21.15.61:0/2802818638 learned_addr learned my addr 172.21.15.61:0/2802818638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:53.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc393fff700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc394060ee0 0x7fc3941412e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc398c7b700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc3940b28c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 -- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc394060ee0 msgr2=0x7fc3941412e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc394060ee0 0x7fc3941412e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 -- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 msgr2=0x7fc3940b28c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc3940b28c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 -- 172.21.15.61:0/2802818638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc38800b180 con 0x7fc39409df40 2022-07-20T03:14:53.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc393fff700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc394060ee0 0x7fc3941412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:53.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3937fe700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 0x7fc3940b23d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fc388002f70 tx=0x7fc388014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc388016070 con 0x7fc39409df40 2022-07-20T03:14:53.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc38800de90 con 0x7fc39409df40 2022-07-20T03:14:53.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3941417d0 con 0x7fc39409df40 2022-07-20T03:14:53.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc388013e80 con 0x7fc39409df40 2022-07-20T03:14:53.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.161+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc394152a20 con 0x7fc39409df40 2022-07-20T03:14:53.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.165+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3940af0b0 con 0x7fc39409df40 2022-07-20T03:14:53.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.165+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc388025030 con 0x7fc39409df40 2022-07-20T03:14:53.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.165+0000 7fc3917fa700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3740539d0 0x7fc374055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.165+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc38806cd90 con 0x7fc39409df40 2022-07-20T03:14:53.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.165+0000 7fc393fff700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3740539d0 0x7fc374055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.165+0000 7fc393fff700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3740539d0 0x7fc374055e90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc38400b000 tx=0x7fc384009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.173+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc3940af0b0 con 0x7fc39409df40 2022-07-20T03:14:53.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.297+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fc3940af0b0 con 0x7fc39409df40 2022-07-20T03:14:53.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.297+0000 7fc3917fa700 1 -- 172.21.15.61:0/2802818638 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7fc3940af0b0 con 0x7fc39409df40 2022-07-20T03:14:53.304 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:53.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3740539d0 msgr2=0x7fc374055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3740539d0 0x7fc374055e90 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fc38400b000 tx=0x7fc384009040).stop 2022-07-20T03:14:53.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 msgr2=0x7fc3940b23d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 0x7fc3940b23d0 secure :-1 s=READY pgs=841 cs=0 l=1 rev1=1 rx=0x7fc388002f70 tx=0x7fc388014000).stop 2022-07-20T03:14:53.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 shutdown_connections 2022-07-20T03:14:53.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3740539d0 0x7fc374055e90 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc3940b4d60 0x7fc3940b28c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc39409df40 0x7fc3940b23d0 unknown :-1 s=CLOSED pgs=841 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 --2- 172.21.15.61:0/2802818638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc394060ee0 0x7fc3941412e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 >> 172.21.15.61:0/2802818638 conn(0x7fc3940b5f30 msgr2=0x7fc39405cd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 shutdown_connections 2022-07-20T03:14:53.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.301+0000 7fc39a6de700 1 -- 172.21.15.61:0/2802818638 wait complete. 2022-07-20T03:14:53.323 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:14:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.433+0000 7f0552278700 1 Processor -- start 2022-07-20T03:14:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.433+0000 7f0552278700 1 -- start start 2022-07-20T03:14:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f054c0adfa0 0x7f054c05d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f054c143ea0 0x7f054c144330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c0b4230 con 0x7f054c05bf30 2022-07-20T03:14:53.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0677b0 con 0x7f054c143ea0 2022-07-20T03:14:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f054c0b2950 con 0x7f054c0adfa0 2022-07-20T03:14:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f054c143ea0 0x7f054c144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f054c143ea0 0x7f054c144330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47600/0 (socket says 172.21.15.61:47600) 2022-07-20T03:14:53.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 -- 172.21.15.61:0/3767850315 learned_addr learned my addr 172.21.15.61:0/3767850315 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:53.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0550a75700 1 --2- 172.21.15.61:0/3767850315 >> v2:172.21.15.117:3300/0 conn(0x7f054c0adfa0 0x7f054c05d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 -- 172.21.15.61:0/3767850315 >> v2:172.21.15.117:3300/0 conn(0x7f054c0adfa0 msgr2=0x7f054c05d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:53.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 --2- 172.21.15.61:0/3767850315 >> v2:172.21.15.117:3300/0 conn(0x7f054c0adfa0 0x7f054c05d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 -- 172.21.15.61:0/3767850315 >> v1:172.21.15.117:6789/0 conn(0x7f054c05bf30 legacy=0x7f054c13a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 -- 172.21.15.61:0/3767850315 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c0b3850 con 0x7f054c143ea0 2022-07-20T03:14:53.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0550a75700 1 --2- 172.21.15.61:0/3767850315 >> v2:172.21.15.117:3300/0 conn(0x7f054c0adfa0 0x7f054c05d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:53.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0551a77700 1 --2- 172.21.15.61:0/3767850315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c143ea0 0x7f054c144330 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f05440032f0 tx=0x7f0544009290).ready entity=mon.0 client_cookie=f49631ea8ef93a4e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f053ffff700 1 -- 172.21.15.61:0/3767850315 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f054400b870 con 0x7f054c143ea0 2022-07-20T03:14:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f053ffff700 1 -- 172.21.15.61:0/3767850315 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f054400b9d0 con 0x7f054c143ea0 2022-07-20T03:14:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f053ffff700 1 -- 172.21.15.61:0/3767850315 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f054400bb40 con 0x7f054c143ea0 2022-07-20T03:14:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- 172.21.15.61:0/3767850315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c143ea0 msgr2=0x7f054c144330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 --2- 172.21.15.61:0/3767850315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c143ea0 0x7f054c144330 secure :-1 s=READY pgs=842 cs=0 l=1 rev1=1 rx=0x7f05440032f0 tx=0x7f0544009290).stop 2022-07-20T03:14:53.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- 172.21.15.61:0/3767850315 shutdown_connections 2022-07-20T03:14:53.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 --2- 172.21.15.61:0/3767850315 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c143ea0 0x7f054c144330 unknown :-1 s=CLOSED pgs=842 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 --2- 172.21.15.61:0/3767850315 >> v2:172.21.15.117:3300/0 conn(0x7f054c0adfa0 0x7f054c05d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- 172.21.15.61:0/3767850315 >> 172.21.15.61:0/3767850315 conn(0x7f054c064b90 msgr2=0x7f054c064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- 172.21.15.61:0/3767850315 shutdown_connections 2022-07-20T03:14:53.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- 172.21.15.61:0/3767850315 wait complete. 2022-07-20T03:14:53.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 Processor -- start 2022-07-20T03:14:53.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.437+0000 7f0552278700 1 -- start start 2022-07-20T03:14:53.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 0x7f054c14fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0adfa0 0x7f054c13c780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c143ea0 0x7f054c13b790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c146ad0 con 0x7f054c05bf30 2022-07-20T03:14:53.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c146fa0 con 0x7f054c143ea0 2022-07-20T03:14:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c146d00 con 0x7f054c0adfa0 2022-07-20T03:14:53.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 0x7f054c14fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551a77700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c143ea0 0x7f054c13b790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 0x7f054c14fb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47602/0 (socket says 172.21.15.61:47602) 2022-07-20T03:14:53.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 -- 172.21.15.61:0/388627058 learned_addr learned my addr 172.21.15.61:0/388627058 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:53.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0550a75700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0adfa0 0x7f054c13c780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 -- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0adfa0 msgr2=0x7f054c13c780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0adfa0 0x7f054c13c780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 -- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c143ea0 msgr2=0x7f054c13b790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c143ea0 0x7f054c13b790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 -- 172.21.15.61:0/388627058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0544009530 con 0x7f054c05bf30 2022-07-20T03:14:53.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551a77700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c143ea0 0x7f054c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:53.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0551276700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 0x7f054c14fb90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f054c0ae3f0 tx=0x7f0540007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0540007030 con 0x7f054c05bf30 2022-07-20T03:14:53.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0540007190 con 0x7f054c05bf30 2022-07-20T03:14:53.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0540006510 con 0x7f054c05bf30 2022-07-20T03:14:53.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c1450f0 con 0x7f054c05bf30 2022-07-20T03:14:53.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.441+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c140510 con 0x7f054c05bf30 2022-07-20T03:14:53.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.445+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c13f590 con 0x7f054c05bf30 2022-07-20T03:14:53.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.445+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0540010030 con 0x7f054c05bf30 2022-07-20T03:14:53.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.445+0000 7f053e7fc700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0534053ad0 0x7f0534055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.445+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f054006bb60 con 0x7f054c05bf30 2022-07-20T03:14:53.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.445+0000 7f0550a75700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0534053ad0 0x7f0534055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.445+0000 7f0550a75700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0534053ad0 0x7f0534055f90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f054800c000 tx=0x7f0548008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.449+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f054c13f590 con 0x7f054c05bf30 2022-07-20T03:14:53.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.577+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f054c13f590 con 0x7f054c05bf30 2022-07-20T03:14:53.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.577+0000 7f053e7fc700 1 -- 172.21.15.61:0/388627058 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f054c13f590 con 0x7f054c05bf30 2022-07-20T03:14:53.623 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:53.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0534053ad0 msgr2=0x7f0534055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0534053ad0 0x7f0534055f90 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f054800c000 tx=0x7f0548008040).stop 2022-07-20T03:14:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 msgr2=0x7f054c14fb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 0x7f054c14fb90 secure :-1 s=READY pgs=843 cs=0 l=1 rev1=1 rx=0x7f054c0ae3f0 tx=0x7f0540007c10).stop 2022-07-20T03:14:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 shutdown_connections 2022-07-20T03:14:53.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0534053ad0 0x7f0534055f90 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c143ea0 0x7f054c13b790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0adfa0 0x7f054c13c780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 --2- 172.21.15.61:0/388627058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c05bf30 0x7f054c14fb90 unknown :-1 s=CLOSED pgs=843 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 >> 172.21.15.61:0/388627058 conn(0x7f054c064b90 msgr2=0x7f054c05c310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 shutdown_connections 2022-07-20T03:14:53.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.581+0000 7f0552278700 1 -- 172.21.15.61:0/388627058 wait complete. 2022-07-20T03:14:53.653 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:14:53.654 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:53.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bbfff700 1 Processor -- start 2022-07-20T03:14:53.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bbfff700 1 -- start start 2022-07-20T03:14:53.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bbfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb8bc0af8e0 0x7fb8bc0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bbfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc147010 con 0x7fb8bc0b5650 2022-07-20T03:14:53.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bbfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8bc1472b0 con 0x7fb8bc0adff0 2022-07-20T03:14:53.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bbfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8bc146d40 con 0x7fb8bc0af8e0 2022-07-20T03:14:53.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8baffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb8bc0b5650 0x7fb8bc147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44208/0 (socket says 172.21.15.61:44208) 2022-07-20T03:14:53.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8baffd700 1 -- 172.21.15.61:0/2860553538 learned_addr learned my addr 172.21.15.61:0/2860553538 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:53.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8bb7fe700 1 --2- 172.21.15.61:0/2860553538 >> v2:172.21.15.117:3300/0 conn(0x7fb8bc0af8e0 0x7fb8bc0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 884683177 0 0) 0x7fb8bc147010 con 0x7fb8bc0b5650 2022-07-20T03:14:53.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb89c003260 con 0x7fb8bc0b5650 2022-07-20T03:14:53.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 965945913 0 0) 0x7fb89c003260 con 0x7fb8bc0b5650 2022-07-20T03:14:53.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 >> v2:172.21.15.117:3300/0 conn(0x7fb8bc0af8e0 msgr2=0x7fb8bc0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:53.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 --2- 172.21.15.61:0/2860553538 >> v2:172.21.15.117:3300/0 conn(0x7fb8bc0af8e0 0x7fb8bc0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 >> v1:172.21.15.117:6789/0 conn(0x7fb8bc0adff0 legacy=0x7fb8bc05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.757+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc05c4a0 con 0x7fb8bc0b5650 2022-07-20T03:14:53.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fb8ac0019e0 con 0x7fb8bc0b5650 2022-07-20T03:14:53.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 >> v1:172.21.15.61:6789/0 conn(0x7fb8bc0b5650 legacy=0x7fb8bc147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c003750 0x7fb89c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb89c004500 0x7fb89c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb89c004e70 0x7fb89c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc146d40 con 0x7fb89c003750 2022-07-20T03:14:53.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c00bdd0 con 0x7fb89c004500 2022-07-20T03:14:53.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb89c00bf60 con 0x7fb89c004e70 2022-07-20T03:14:53.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bb7fe700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb89c004e70 0x7fb89c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb89c004500 0x7fb89c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8baffd700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c003750 0x7fb89c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 -- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb89c004e70 msgr2=0x7fb89c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:53.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb89c004e70 0x7fb89c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 -- 172.21.15.61:0/2860553538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c003750 msgr2=0x7fb89c003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:53.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c003750 0x7fb89c003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 -- 172.21.15.61:0/2860553538 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb89c003260 con 0x7fb89c004500 2022-07-20T03:14:53.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bb7fe700 1 -- 172.21.15.61:0/2860553538 reap_dead start 2022-07-20T03:14:53.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8baffd700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c003750 0x7fb89c003b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:14:53.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb89c004500 0x7fb89c004980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fb8bc0622d0 tx=0x7fb8a4007a50).ready entity=mon.1 client_cookie=59ddedee0ae8f393 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8a4007030 con 0x7fb89c004500 2022-07-20T03:14:53.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8a4007190 con 0x7fb89c004500 2022-07-20T03:14:53.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8b9ffb700 1 -- 172.21.15.61:0/2860553538 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8a4006560 con 0x7fb89c004500 2022-07-20T03:14:53.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 -- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb89c004500 msgr2=0x7fb89c004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb89c004500 0x7fb89c004980 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7fb8bc0622d0 tx=0x7fb8a4007a50).stop 2022-07-20T03:14:53.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 -- 172.21.15.61:0/2860553538 shutdown_connections 2022-07-20T03:14:53.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/2860553538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb89c004500 0x7fb89c004980 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 -- 172.21.15.61:0/2860553538 >> 172.21.15.61:0/2860553538 conn(0x7fb8bc0634e0 msgr2=0x7fb8bc0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 -- 172.21.15.61:0/2860553538 shutdown_connections 2022-07-20T03:14:53.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.761+0000 7fb8bbfff700 1 -- 172.21.15.61:0/2860553538 wait complete. 2022-07-20T03:14:53.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 Processor -- start 2022-07-20T03:14:53.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 -- start start 2022-07-20T03:14:53.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 0x7fb8bc067440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8bc111110 0x7fb8bc10e470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8bc065fd0 0x7fb8bc066480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc0654e0 con 0x7fb89c004500 2022-07-20T03:14:53.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc066fc0 con 0x7fb8bc065fd0 2022-07-20T03:14:53.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8bc064a60 con 0x7fb8bc111110 2022-07-20T03:14:53.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 0x7fb8bc067440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 0x7fb8bc067440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47624/0 (socket says 172.21.15.61:47624) 2022-07-20T03:14:53.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 -- 172.21.15.61:0/1008922743 learned_addr learned my addr 172.21.15.61:0/1008922743 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:53.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8bc111110 0x7fb8bc10e470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bb7fe700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8bc065fd0 0x7fb8bc066480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 -- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8bc111110 msgr2=0x7fb8bc10e470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8bc111110 0x7fb8bc10e470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 -- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8bc065fd0 msgr2=0x7fb8bc066480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8bc065fd0 0x7fb8bc066480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 -- 172.21.15.61:0/1008922743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8bc064770 con 0x7fb89c004500 2022-07-20T03:14:53.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8baffd700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 0x7fb8bc067440 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fb8bc0650a0 tx=0x7fb8ac002210).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8ac00c7c0 con 0x7fb89c004500 2022-07-20T03:14:53.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8ac00c920 con 0x7fb89c004500 2022-07-20T03:14:53.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8bc066970 con 0x7fb89c004500 2022-07-20T03:14:53.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8ac00cc80 con 0x7fb89c004500 2022-07-20T03:14:53.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.765+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8bc111870 con 0x7fb89c004500 2022-07-20T03:14:53.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.769+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8bc067150 con 0x7fb89c004500 2022-07-20T03:14:53.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.769+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb8ac014430 con 0x7fb89c004500 2022-07-20T03:14:53.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.769+0000 7fb8b3fff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8a00539b0 0x7fb8a0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:53.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.769+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8ac06bf30 con 0x7fb89c004500 2022-07-20T03:14:53.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.769+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8a00539b0 0x7fb8a0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:53.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.773+0000 7fb8ba7fc700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8a00539b0 0x7fb8a0055e70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb8a4019000 tx=0x7fb8a4013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:53.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.777+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8ac042e30 con 0x7fb89c004500 2022-07-20T03:14:53.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb8bc0574e0 con 0x7fb89c004500 2022-07-20T03:14:53.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8b3fff700 1 -- 172.21.15.61:0/1008922743 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fb8ac046450 con 0x7fb89c004500 2022-07-20T03:14:53.948 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:53.949 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:14:49.076244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6643":{"gid":6643,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/843120801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":843120801},{"type":"v1","addr":"172.21.15.117:6835","nonce":843120801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:14:53.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8a00539b0 msgr2=0x7fb8a0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8a00539b0 0x7fb8a0055e70 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7fb8a4019000 tx=0x7fb8a4013000).stop 2022-07-20T03:14:53.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 msgr2=0x7fb8bc067440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:53.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 0x7fb8bc067440 secure :-1 s=READY pgs=845 cs=0 l=1 rev1=1 rx=0x7fb8bc0650a0 tx=0x7fb8ac002210).stop 2022-07-20T03:14:53.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 shutdown_connections 2022-07-20T03:14:53.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8a00539b0 0x7fb8a0055e70 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8bc065fd0 0x7fb8bc066480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8bc111110 0x7fb8bc10e470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 --2- 172.21.15.61:0/1008922743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb89c004500 0x7fb8bc067440 unknown :-1 s=CLOSED pgs=845 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:53.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.941+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 >> 172.21.15.61:0/1008922743 conn(0x7fb8bc0634e0 msgr2=0x7fb8bc0b46d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:53.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.945+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 shutdown_connections 2022-07-20T03:14:53.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:53.945+0000 7fb8bbfff700 1 -- 172.21.15.61:0/1008922743 wait complete. 2022-07-20T03:14:53.968 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-20T03:14:54.195 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:14:48.072382+0000', 'modified': '2022-07-20T03:14:49.076244+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6643}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6643': {'gid': 6643, 'name': 'b', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/843120801', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 843120801}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 843120801}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:14:54.196 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:14:54.197 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:14:54.314 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6643,"want_state":"up:active","state":"up:active","rank_uptime":6.2269689320000001,"mdsmap_epoch":45,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":10.626008657} 2022-07-20T03:14:54.315 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6643, "mdsmap_epoch": 45, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 6.226968932, "state": "up:active", "uptime": 10.626008657, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:14:54.315 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:14:54.316 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:14:54.316 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:14:54.316 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:54.317 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:54.339 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:14:54.415 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:14:54.416 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:14:54.483 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:14:54.483 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:14:54.485 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:54.485 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:14:54.549 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:14:54.551 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:54.606 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:14:54.636 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:14:54.637 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:14:54.681 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23073]: starting ceph client 2022-07-20T03:14:54.682 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:14:54.678+0000 7f4eace72080 -1 init, newargv = 0x5652f77af490 newargc=9 2022-07-20T03:14:54.691 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23073]: starting fuse 2022-07-20T03:14:54.692 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:14:54.694 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:14:54.695 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:54.696 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:14:54.729 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:14:54.730 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:14:54.732 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:54.734 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:14:54.735 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:14:54.736 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:14:54.737 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:14:54.738 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:14:54.740 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:54.742 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:14:54.743 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:14:54.744 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:14:54.745 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:54.769 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:14:54.770 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:14:54.770 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:14:54.771 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:14:54.771 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:54.771 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:14:54.772 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:14:54.772 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:14:54.773 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:14:54.773 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:14:54.773 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:14:54.775 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:14:54.776 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:14:54.853 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23073.asok 2022-07-20T03:14:54.856 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23073.asok 2022-07-20T03:14:54.857 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:54.858 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23073.asok status 2022-07-20T03:14:54.967 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:14:54.967 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:14:54.967 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:14:54.968 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:14:54.968 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:14:54.968 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:14:54.969 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:14:54.969 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23073", 2022-07-20T03:14:54.969 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:14:54.970 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:54.970 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:14:54.970 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:14:54.970 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 6814, 2022-07-20T03:14:54.971 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:14:54.971 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:14:54.971 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:14:54.971 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 6814 2022-07-20T03:14:54.972 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:54.980 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:14:54.981 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:14:54.981 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:14:54.982 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 4155348693 2022-07-20T03:14:54.982 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:14:54.983 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:54.983 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:14:54.984 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:14:54.984 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:14:54.985 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 4155348693 2022-07-20T03:14:54.985 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:14:54.994 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.6814 172.21.15.124:0/4155348693", 2022-07-20T03:14:54.995 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/4155348693", 2022-07-20T03:14:54.995 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:14:54.995 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 45, 2022-07-20T03:14:54.996 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 81, 2022-07-20T03:14:54.996 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:14:54.997 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:14:54.997 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:14:55.009 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:14:55.064 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:55.066 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:55.067 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:55.089 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:14:55.090 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:14:55.091 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:55.092 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:55.150 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:14:55.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 Processor -- start 2022-07-20T03:14:55.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 -- start start 2022-07-20T03:14:55.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1ba00b1ef0 0x7f1ba0062040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0098ae0 0x7f1ba00ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1ba0061050 0x7f1ba00614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba00b22d0 con 0x7f1ba0061050 2022-07-20T03:14:55.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ba00abe50 con 0x7f1ba00b1ef0 2022-07-20T03:14:55.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba6d9f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00ac0f0 con 0x7f1ba0098ae0 2022-07-20T03:14:55.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba659e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1ba0061050 0x7f1ba00614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba559c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0098ae0 0x7f1ba00ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba659e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1ba0061050 0x7f1ba00614a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47634/0 (socket says 172.21.15.61:47634) 2022-07-20T03:14:55.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba5d9d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1ba00b1ef0 0x7f1ba0062040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba659e700 1 -- 172.21.15.61:0/2302161059 learned_addr learned my addr 172.21.15.61:0/2302161059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba5d9d700 1 -- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0098ae0 msgr2=0x7f1ba00ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:55.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba5d9d700 1 --2- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0098ae0 0x7f1ba00ac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba5d9d700 1 -- 172.21.15.61:0/2302161059 >> v2:172.21.15.61:3300/0 conn(0x7f1ba0061050 msgr2=0x7f1ba00614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:55.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba5d9d700 1 --2- 172.21.15.61:0/2302161059 >> v2:172.21.15.61:3300/0 conn(0x7f1ba0061050 0x7f1ba00614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.261+0000 7f1ba5d9d700 1 -- 172.21.15.61:0/2302161059 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ba00ac390 con 0x7f1ba00b1ef0 2022-07-20T03:14:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba5d9d700 1 --2- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0062040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f1b9c0032f0 tx=0x7f1b9c0091a0).ready entity=mon.1 client_cookie=5fbab5bdb6fa2bb7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba4d9b700 1 -- 172.21.15.61:0/2302161059 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b9c00b600 con 0x7f1ba00b1ef0 2022-07-20T03:14:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba4d9b700 1 -- 172.21.15.61:0/2302161059 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b9c00b760 con 0x7f1ba00b1ef0 2022-07-20T03:14:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba4d9b700 1 -- 172.21.15.61:0/2302161059 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b9c00b8d0 con 0x7f1ba00b1ef0 2022-07-20T03:14:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 msgr2=0x7f1ba0062040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0062040 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f1b9c0032f0 tx=0x7f1b9c0091a0).stop 2022-07-20T03:14:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/2302161059 shutdown_connections 2022-07-20T03:14:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/2302161059 >> v2:172.21.15.61:3300/0 conn(0x7f1ba0061050 0x7f1ba00614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0098ae0 0x7f1ba00ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/2302161059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0062040 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/2302161059 >> 172.21.15.61:0/2302161059 conn(0x7f1ba00b2ca0 msgr2=0x7f1ba00b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:55.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/2302161059 shutdown_connections 2022-07-20T03:14:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/2302161059 wait complete. 2022-07-20T03:14:55.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 Processor -- start 2022-07-20T03:14:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.265+0000 7f1ba6d9f700 1 -- start start 2022-07-20T03:14:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 0x7f1ba00b4ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ba0098ae0 0x7f1ba0159940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0159e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba009b240 con 0x7f1ba0098ae0 2022-07-20T03:14:55.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba00ae0c0 con 0x7f1ba00b1ef0 2022-07-20T03:14:55.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ba009b470 con 0x7f1ba0061050 2022-07-20T03:14:55.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba5d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 0x7f1ba00b4ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ba0098ae0 0x7f1ba0159940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba5d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 0x7f1ba00b4ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:34308/0 (socket says 172.21.15.61:34308) 2022-07-20T03:14:55.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba5d9d700 1 -- 172.21.15.61:0/422886711 learned_addr learned my addr 172.21.15.61:0/422886711 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:55.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba659e700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0159e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 -- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 msgr2=0x7f1ba00b4ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 0x7f1ba00b4ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 -- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 msgr2=0x7f1ba0159e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0159e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 -- 172.21.15.61:0/422886711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b9c009420 con 0x7f1ba0098ae0 2022-07-20T03:14:55.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba5d9d700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 0x7f1ba00b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:55.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba659e700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:55.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba559c700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ba0098ae0 0x7f1ba0159940 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f1b9400c000 tx=0x7f1b94006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b94010780 con 0x7f1ba0098ae0 2022-07-20T03:14:55.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b940108e0 con 0x7f1ba0098ae0 2022-07-20T03:14:55.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ba009b710 con 0x7f1ba0098ae0 2022-07-20T03:14:55.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b94010c40 con 0x7f1ba0098ae0 2022-07-20T03:14:55.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.269+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ba009a970 con 0x7f1ba0098ae0 2022-07-20T03:14:55.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.273+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ba00574e0 con 0x7f1ba0098ae0 2022-07-20T03:14:55.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.273+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1b9401c460 con 0x7f1ba0098ae0 2022-07-20T03:14:55.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.273+0000 7f1b92ffd700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b80053920 0x7f1b80055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.273+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1b9406c3c0 con 0x7f1ba0098ae0 2022-07-20T03:14:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.273+0000 7f1ba5d9d700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b80053920 0x7f1b80055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.277+0000 7f1ba5d9d700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b80053920 0x7f1b80055de0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f1b9c01b000 tx=0x7f1b9c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.281+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b94043130 con 0x7f1ba0098ae0 2022-07-20T03:14:55.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.409+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1ba00447e0 con 0x7f1ba0098ae0 2022-07-20T03:14:55.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.409+0000 7f1b92ffd700 1 -- 172.21.15.61:0/422886711 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v81) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f1b94046750 con 0x7f1ba0098ae0 2022-07-20T03:14:55.421 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:55.422 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":81,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:14:49.075215+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":15,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:14:46.661180+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":16,"pool_name":"cephfs_data","create_time":"2022-07-20T03:14:47.668660+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"81","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:14:55.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b80053920 msgr2=0x7f1b80055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b80053920 0x7f1b80055de0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f1b9c01b000 tx=0x7f1b9c016000).stop 2022-07-20T03:14:55.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ba0098ae0 msgr2=0x7f1ba0159940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ba0098ae0 0x7f1ba0159940 secure :-1 s=READY pgs=846 cs=0 l=1 rev1=1 rx=0x7f1b9400c000 tx=0x7f1b94006df0).stop 2022-07-20T03:14:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 shutdown_connections 2022-07-20T03:14:55.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b80053920 0x7f1b80055de0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ba00b1ef0 0x7f1ba0159e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ba0098ae0 0x7f1ba0159940 unknown :-1 s=CLOSED pgs=846 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 --2- 172.21.15.61:0/422886711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ba0061050 0x7f1ba00b4ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 >> 172.21.15.61:0/422886711 conn(0x7f1ba00b2ca0 msgr2=0x7f1ba01433c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:55.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 shutdown_connections 2022-07-20T03:14:55.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.413+0000 7f1ba6d9f700 1 -- 172.21.15.61:0/422886711 wait complete. 2022-07-20T03:14:55.431 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 15, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:14:46.661180+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 16, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:14:47.668660+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '81', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:14:55.433 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:14:55.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.537+0000 7f268e475700 1 Processor -- start 2022-07-20T03:14:55.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.537+0000 7f268e475700 1 -- start start 2022-07-20T03:14:55.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268e475700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f268814b2e0 0x7f26880601b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268e475700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f26880b36b0 con 0x7f2688098920 2022-07-20T03:14:55.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268e475700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2688098540 con 0x7f2688098160 2022-07-20T03:14:55.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268e475700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f26880b5d80 con 0x7f268814b2e0 2022-07-20T03:14:55.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268ca12700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2688098920 0x7f2688146330 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44246/0 (socket says 172.21.15.61:44246) 2022-07-20T03:14:55.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268ca12700 1 -- 172.21.15.61:0/2306749504 learned_addr learned my addr 172.21.15.61:0/2306749504 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:55.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f26877fe700 1 --2- 172.21.15.61:0/2306749504 >> v2:172.21.15.117:3300/0 conn(0x7f268814b2e0 0x7f26880601b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f2686ffd700 1 -- 172.21.15.61:0/2306749504 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3712265855 0 0) 0x7f26880b36b0 con 0x7f2688098920 2022-07-20T03:14:55.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f2686ffd700 1 -- 172.21.15.61:0/2306749504 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2668003260 con 0x7f2688098920 2022-07-20T03:14:55.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f26877fe700 1 -- 172.21.15.61:0/2306749504 >> v1:172.21.15.117:6789/0 conn(0x7f2688098160 legacy=0x7f268814abd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f26877fe700 1 -- 172.21.15.61:0/2306749504 >> v1:172.21.15.61:6789/0 conn(0x7f2688098920 legacy=0x7f2688146330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f26877fe700 1 -- 172.21.15.61:0/2306749504 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f26880b38e0 con 0x7f268814b2e0 2022-07-20T03:14:55.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f26877fe700 1 --2- 172.21.15.61:0/2306749504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f26880601b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f2670002f70 tx=0x7f2670006df0).ready entity=mon.1 client_cookie=f4994d3626d6c0a4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f2686ffd700 1 -- 172.21.15.61:0/2306749504 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f267000d610 con 0x7f268814b2e0 2022-07-20T03:14:55.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f2686ffd700 1 -- 172.21.15.61:0/2306749504 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2670000e90 con 0x7f268814b2e0 2022-07-20T03:14:55.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f2686ffd700 1 -- 172.21.15.61:0/2306749504 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f267000d960 con 0x7f268814b2e0 2022-07-20T03:14:55.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268e475700 1 -- 172.21.15.61:0/2306749504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 msgr2=0x7f26880601b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.541+0000 7f268e475700 1 --2- 172.21.15.61:0/2306749504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f26880601b0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f2670002f70 tx=0x7f2670006df0).stop 2022-07-20T03:14:55.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- 172.21.15.61:0/2306749504 shutdown_connections 2022-07-20T03:14:55.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 --2- 172.21.15.61:0/2306749504 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f26880601b0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- 172.21.15.61:0/2306749504 >> 172.21.15.61:0/2306749504 conn(0x7f26880b5f30 msgr2=0x7f26880b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:55.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- 172.21.15.61:0/2306749504 shutdown_connections 2022-07-20T03:14:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- 172.21.15.61:0/2306749504 wait complete. 2022-07-20T03:14:55.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 Processor -- start 2022-07-20T03:14:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- start start 2022-07-20T03:14:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 0x7f268814c980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2688098920 0x7f2688145f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f2688146420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f268809ad50 con 0x7f2688098160 2022-07-20T03:14:55.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2688099730 con 0x7f268814b2e0 2022-07-20T03:14:55.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268e475700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f268814ce70 con 0x7f2688098920 2022-07-20T03:14:55.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f2687fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 0x7f268814c980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f2687fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 0x7f268814c980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47650/0 (socket says 172.21.15.61:47650) 2022-07-20T03:14:55.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f2687fff700 1 -- 172.21.15.61:0/4040478768 learned_addr learned my addr 172.21.15.61:0/4040478768 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:55.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f26877fe700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2688098920 0x7f2688145f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.545+0000 7f268ca12700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f2688146420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f2687fff700 1 -- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2688098920 msgr2=0x7f2688145f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f2687fff700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2688098920 0x7f2688145f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f2687fff700 1 -- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 msgr2=0x7f2688146420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f2687fff700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f2688146420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f2687fff700 1 -- 172.21.15.61:0/4040478768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f267000b180 con 0x7f2688098160 2022-07-20T03:14:55.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f26877fe700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2688098920 0x7f2688145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:55.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f2687fff700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 0x7f268814c980 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f268805f440 tx=0x7f26780013f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2678006af0 con 0x7f2688098160 2022-07-20T03:14:55.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2678006c50 con 0x7f2688098160 2022-07-20T03:14:55.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f268809bd00 con 0x7f2688098160 2022-07-20T03:14:55.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2678007030 con 0x7f2688098160 2022-07-20T03:14:55.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f268809a9d0 con 0x7f2688098160 2022-07-20T03:14:55.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f26880574e0 con 0x7f2688098160 2022-07-20T03:14:55.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.549+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2678010030 con 0x7f2688098160 2022-07-20T03:14:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.553+0000 7f26857fa700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f266c0539d0 0x7f266c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.553+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f267806bc70 con 0x7f2688098160 2022-07-20T03:14:55.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.553+0000 7f26877fe700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f266c0539d0 0x7f266c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.553+0000 7f26877fe700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f266c0539d0 0x7f266c055e90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f267001a000 tx=0x7f2670014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.553+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2678042bd0 con 0x7f2688098160 2022-07-20T03:14:55.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.677+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f268805db10 con 0x7f2688098160 2022-07-20T03:14:55.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.677+0000 7f26857fa700 1 -- 172.21.15.61:0/4040478768 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v81) v1 ==== 110+0+10 (secure 0 0 0) 0x7f26780461f0 con 0x7f2688098160 2022-07-20T03:14:55.712 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:14:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f266c0539d0 msgr2=0x7f266c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f266c0539d0 0x7f266c055e90 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f267001a000 tx=0x7f2670014000).stop 2022-07-20T03:14:55.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 msgr2=0x7f268814c980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 0x7f268814c980 secure :-1 s=READY pgs=848 cs=0 l=1 rev1=1 rx=0x7f268805f440 tx=0x7f26780013f0).stop 2022-07-20T03:14:55.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 shutdown_connections 2022-07-20T03:14:55.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f266c0539d0 0x7f266c055e90 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f268814b2e0 0x7f2688146420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2688098920 0x7f2688145f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 --2- 172.21.15.61:0/4040478768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2688098160 0x7f268814c980 unknown :-1 s=CLOSED pgs=848 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 >> 172.21.15.61:0/4040478768 conn(0x7f26880b5f30 msgr2=0x7f26880b5090 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:55.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 shutdown_connections 2022-07-20T03:14:55.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.681+0000 7f268e475700 1 -- 172.21.15.61:0/4040478768 wait complete. 2022-07-20T03:14:55.733 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:14:55.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 Processor -- start 2022-07-20T03:14:55.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 -- start start 2022-07-20T03:14:55.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9f600af180 0x7f9f6005d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013fc00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600b5db0 con 0x7f9f600b59d0 2022-07-20T03:14:55.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600ad800 con 0x7f9f600af180 2022-07-20T03:14:55.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f667af700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f600adaa0 con 0x7f9f60128d70 2022-07-20T03:14:55.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f657ad700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9f600b59d0 0x7f9f60128660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44260/0 (socket says 172.21.15.61:44260) 2022-07-20T03:14:55.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f657ad700 1 -- 172.21.15.61:0/1393784385 learned_addr learned my addr 172.21.15.61:0/1393784385 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:55.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f65fae700 1 --2- 172.21.15.61:0/1393784385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013fc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f64fac700 1 --2- 172.21.15.61:0/1393784385 >> v2:172.21.15.117:3300/0 conn(0x7f9f600af180 0x7f9f6005d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f4bfff700 1 -- 172.21.15.61:0/1393784385 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 295775134 0 0) 0x7f9f600b5db0 con 0x7f9f600b59d0 2022-07-20T03:14:55.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f4bfff700 1 -- 172.21.15.61:0/1393784385 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f44003260 con 0x7f9f600b59d0 2022-07-20T03:14:55.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f65fae700 1 -- 172.21.15.61:0/1393784385 >> v2:172.21.15.117:3300/0 conn(0x7f9f600af180 msgr2=0x7f9f6005d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:55.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f65fae700 1 --2- 172.21.15.61:0/1393784385 >> v2:172.21.15.117:3300/0 conn(0x7f9f600af180 0x7f9f6005d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f65fae700 1 -- 172.21.15.61:0/1393784385 >> v1:172.21.15.61:6789/0 conn(0x7f9f600b59d0 legacy=0x7f9f60128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f65fae700 1 -- 172.21.15.61:0/1393784385 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600ad4f0 con 0x7f9f60128d70 2022-07-20T03:14:55.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.857+0000 7f9f64fac700 1 --2- 172.21.15.61:0/1393784385 >> v2:172.21.15.117:3300/0 conn(0x7f9f600af180 0x7f9f6005d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:14:55.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f65fae700 1 --2- 172.21.15.61:0/1393784385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013fc00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9f580032f0 tx=0x7f9f580091d0).ready entity=mon.2 client_cookie=4a52562e1de7c621 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:55.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f4bfff700 1 -- 172.21.15.61:0/1393784385 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5800b600 con 0x7f9f60128d70 2022-07-20T03:14:55.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f4bfff700 1 -- 172.21.15.61:0/1393784385 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5800b760 con 0x7f9f60128d70 2022-07-20T03:14:55.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f4bfff700 1 -- 172.21.15.61:0/1393784385 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5800b8d0 con 0x7f9f60128d70 2022-07-20T03:14:55.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 -- 172.21.15.61:0/1393784385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 msgr2=0x7f9f6013fc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 --2- 172.21.15.61:0/1393784385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013fc00 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f9f580032f0 tx=0x7f9f580091d0).stop 2022-07-20T03:14:55.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 -- 172.21.15.61:0/1393784385 shutdown_connections 2022-07-20T03:14:55.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 --2- 172.21.15.61:0/1393784385 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013fc00 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 --2- 172.21.15.61:0/1393784385 >> v2:172.21.15.117:3300/0 conn(0x7f9f600af180 0x7f9f6005d1c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 -- 172.21.15.61:0/1393784385 >> 172.21.15.61:0/1393784385 conn(0x7f9f600634c0 msgr2=0x7f9f600b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 -- 172.21.15.61:0/1393784385 shutdown_connections 2022-07-20T03:14:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.861+0000 7f9f667af700 1 -- 172.21.15.61:0/1393784385 wait complete. 2022-07-20T03:14:55.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 Processor -- start 2022-07-20T03:14:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 -- start start 2022-07-20T03:14:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 0x7f9f6013a1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f600b59d0 0x7f9f6013ca60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013cf50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:55.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f60067700 con 0x7f9f600af180 2022-07-20T03:14:55.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f601393a0 con 0x7f9f600b59d0 2022-07-20T03:14:55.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f60139600 con 0x7f9f60128d70 2022-07-20T03:14:55.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 0x7f9f6013a1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 0x7f9f6013a1a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47664/0 (socket says 172.21.15.61:47664) 2022-07-20T03:14:55.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 -- 172.21.15.61:0/3216679986 learned_addr learned my addr 172.21.15.61:0/3216679986 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:55.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f64fac700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f600b59d0 0x7f9f6013ca60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f65fae700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013cf50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:55.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 -- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 msgr2=0x7f9f6013cf50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:55.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013cf50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:55.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 -- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f600b59d0 msgr2=0x7f9f6013ca60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f600b59d0 0x7f9f6013ca60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 -- 172.21.15.61:0/3216679986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f5400b180 con 0x7f9f600af180 2022-07-20T03:14:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f657ad700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 0x7f9f6013a1a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f9f5c001110 tx=0x7f9f5c007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5c0070d0 con 0x7f9f600af180 2022-07-20T03:14:56.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5c007230 con 0x7f9f600af180 2022-07-20T03:14:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f58009470 con 0x7f9f600af180 2022-07-20T03:14:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5c005610 con 0x7f9f600af180 2022-07-20T03:14:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.865+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f6013d5e0 con 0x7f9f600af180 2022-07-20T03:14:56.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.869+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6013b2d0 con 0x7f9f600af180 2022-07-20T03:14:56.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.869+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f5c010070 con 0x7f9f600af180 2022-07-20T03:14:56.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.869+0000 7f9f52ffd700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053af0 0x7f9f3c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.869+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f5c06b320 con 0x7f9f600af180 2022-07-20T03:14:56.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.869+0000 7f9f64fac700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053af0 0x7f9f3c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.869+0000 7f9f64fac700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053af0 0x7f9f3c055fb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9f5400b000 tx=0x7f9f54008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:55.877+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f5c068e90 con 0x7f9f600af180 2022-07-20T03:14:56.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9f6013d5e0 con 0x7f9f600af180 2022-07-20T03:14:56.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f52ffd700 1 -- 172.21.15.61:0/3216679986 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v81) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9f6013d5e0 con 0x7f9f600af180 2022-07-20T03:14:56.055 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:56.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053af0 msgr2=0x7f9f3c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053af0 0x7f9f3c055fb0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f9f5400b000 tx=0x7f9f54008040).stop 2022-07-20T03:14:56.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 msgr2=0x7f9f6013a1a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 0x7f9f6013a1a0 secure :-1 s=READY pgs=850 cs=0 l=1 rev1=1 rx=0x7f9f5c001110 tx=0x7f9f5c007aa0).stop 2022-07-20T03:14:56.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 shutdown_connections 2022-07-20T03:14:56.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053af0 0x7f9f3c055fb0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f600b59d0 0x7f9f6013ca60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f60128d70 0x7f9f6013cf50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 --2- 172.21.15.61:0/3216679986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f600af180 0x7f9f6013a1a0 unknown :-1 s=CLOSED pgs=850 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 >> 172.21.15.61:0/3216679986 conn(0x7f9f600634c0 msgr2=0x7f9f600afb90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 shutdown_connections 2022-07-20T03:14:56.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.005+0000 7f9f667af700 1 -- 172.21.15.61:0/3216679986 wait complete. 2022-07-20T03:14:56.065 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:14:56.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 Processor -- start 2022-07-20T03:14:56.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- start start 2022-07-20T03:14:56.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86240b1a00 0x7f8624061bb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86240b2030 con 0x7f862405fc20 2022-07-20T03:14:56.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f862414a670 con 0x7f862405f840 2022-07-20T03:14:56.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86240606f0 con 0x7f86240b1a00 2022-07-20T03:14:56.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86240b1a00 0x7f8624061bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f86240b1a00 0x7f8624061bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47676/0 (socket says 172.21.15.61:47676) 2022-07-20T03:14:56.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 -- 172.21.15.61:0/4157134601 learned_addr learned my addr 172.21.15.61:0/4157134601 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:56.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 -- 172.21.15.61:0/4157134601 >> v1:172.21.15.117:6789/0 conn(0x7f862405f840 legacy=0x7f862414ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 -- 172.21.15.61:0/4157134601 >> v1:172.21.15.61:6789/0 conn(0x7f862405fc20 legacy=0x7f86241466b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 -- 172.21.15.61:0/4157134601 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f862414b220 con 0x7f86240b1a00 2022-07-20T03:14:56.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f86295e3700 1 --2- 172.21.15.61:0/4157134601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f8624061bb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f861c0032f0 tx=0x7f861c009290).ready entity=mon.0 client_cookie=e0d55bbd2ba1f464 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f861b7fe700 1 -- 172.21.15.61:0/4157134601 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f861c00b8a0 con 0x7f86240b1a00 2022-07-20T03:14:56.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f861b7fe700 1 -- 172.21.15.61:0/4157134601 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c00ba00 con 0x7f86240b1a00 2022-07-20T03:14:56.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f861b7fe700 1 -- 172.21.15.61:0/4157134601 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f861c00bb70 con 0x7f86240b1a00 2022-07-20T03:14:56.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- 172.21.15.61:0/4157134601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 msgr2=0x7f8624061bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 --2- 172.21.15.61:0/4157134601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f8624061bb0 secure :-1 s=READY pgs=852 cs=0 l=1 rev1=1 rx=0x7f861c0032f0 tx=0x7f861c009290).stop 2022-07-20T03:14:56.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- 172.21.15.61:0/4157134601 shutdown_connections 2022-07-20T03:14:56.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 --2- 172.21.15.61:0/4157134601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f8624061bb0 unknown :-1 s=CLOSED pgs=852 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- 172.21.15.61:0/4157134601 >> 172.21.15.61:0/4157134601 conn(0x7f86240b2af0 msgr2=0x7f86240b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- 172.21.15.61:0/4157134601 shutdown_connections 2022-07-20T03:14:56.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.177+0000 7f8629de4700 1 -- 172.21.15.61:0/4157134601 wait complete. 2022-07-20T03:14:56.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 Processor -- start 2022-07-20T03:14:56.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 -- start start 2022-07-20T03:14:56.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f862405f840 0x7f8624099160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f862405fc20 0x7f8624144e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f86240afda0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240af2e0 con 0x7f86240b1a00 2022-07-20T03:14:56.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86240ad970 con 0x7f862405fc20 2022-07-20T03:14:56.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86240b0290 con 0x7f862405f840 2022-07-20T03:14:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f86240afda0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f86240afda0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47682/0 (socket says 172.21.15.61:47682) 2022-07-20T03:14:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 -- 172.21.15.61:0/2755745483 learned_addr learned my addr 172.21.15.61:0/2755745483 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8628de2700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f862405f840 0x7f8624099160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f861bfff700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f862405fc20 0x7f8624144e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 -- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f862405f840 msgr2=0x7f8624099160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f862405f840 0x7f8624099160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 -- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f862405fc20 msgr2=0x7f8624144e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f862405fc20 0x7f8624144e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 -- 172.21.15.61:0/2755745483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f861c009530 con 0x7f86240b1a00 2022-07-20T03:14:56.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f86295e3700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f86240afda0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f861c0032f0 tx=0x7f861c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f861c00bde0 con 0x7f86240b1a00 2022-07-20T03:14:56.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f861c01f3f0 con 0x7f86240b1a00 2022-07-20T03:14:56.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86241454e0 con 0x7f86240b1a00 2022-07-20T03:14:56.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f861c01f750 con 0x7f86240b1a00 2022-07-20T03:14:56.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.181+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86241433d0 con 0x7f86240b1a00 2022-07-20T03:14:56.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.185+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86240574e0 con 0x7f86240b1a00 2022-07-20T03:14:56.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.185+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f861c026030 con 0x7f86240b1a00 2022-07-20T03:14:56.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.185+0000 7f8619ffb700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8604053920 0x7f8604055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.185+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f861c06cc60 con 0x7f86240b1a00 2022-07-20T03:14:56.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.185+0000 7f8628de2700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8604053920 0x7f8604055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.185+0000 7f8628de2700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8604053920 0x7f8604055de0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f862000d000 tx=0x7f8620007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.193+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f861c043a30 con 0x7f86240b1a00 2022-07-20T03:14:56.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.321+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f86241436e0 con 0x7f86240b1a00 2022-07-20T03:14:56.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.321+0000 7f8619ffb700 1 -- 172.21.15.61:0/2755745483 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v81) v1 ==== 104+0+10 (secure 0 0 0) 0x7f861c0240a0 con 0x7f86240b1a00 2022-07-20T03:14:56.331 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:56.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8604053920 msgr2=0x7f8604055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8604053920 0x7f8604055de0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f862000d000 tx=0x7f8620007b90).stop 2022-07-20T03:14:56.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 msgr2=0x7f86240afda0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f86240afda0 secure :-1 s=READY pgs=853 cs=0 l=1 rev1=1 rx=0x7f861c0032f0 tx=0x7f861c016000).stop 2022-07-20T03:14:56.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 shutdown_connections 2022-07-20T03:14:56.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8604053920 0x7f8604055de0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f862405fc20 0x7f8624144e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86240b1a00 0x7f86240afda0 unknown :-1 s=CLOSED pgs=853 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 --2- 172.21.15.61:0/2755745483 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f862405f840 0x7f8624099160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 >> 172.21.15.61:0/2755745483 conn(0x7f86240b2af0 msgr2=0x7f8624061bb0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 shutdown_connections 2022-07-20T03:14:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.325+0000 7f8629de4700 1 -- 172.21.15.61:0/2755745483 wait complete. 2022-07-20T03:14:56.350 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:14:56.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.461+0000 7f2114ec6700 1 Processor -- start 2022-07-20T03:14:56.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.461+0000 7f2114ec6700 1 -- start start 2022-07-20T03:14:56.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f21100b2ec0 0x7f21100b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21101461c0 0x7f21101465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f21100b3f10 con 0x7f211005d160 2022-07-20T03:14:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21100b4ea0 con 0x7f21101461c0 2022-07-20T03:14:56.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f21100b3790 con 0x7f21100b2ec0 2022-07-20T03:14:56.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21101461c0 0x7f21101465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f21101461c0 0x7f21101465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47690/0 (socket says 172.21.15.61:47690) 2022-07-20T03:14:56.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 -- 172.21.15.61:0/1947348939 learned_addr learned my addr 172.21.15.61:0/1947348939 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:56.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210f7fe700 1 --2- 172.21.15.61:0/1947348939 >> v2:172.21.15.117:3300/0 conn(0x7f21100b2ec0 0x7f21100b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 -- 172.21.15.61:0/1947348939 >> v2:172.21.15.117:3300/0 conn(0x7f21100b2ec0 msgr2=0x7f21100b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:56.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 --2- 172.21.15.61:0/1947348939 >> v2:172.21.15.117:3300/0 conn(0x7f21100b2ec0 0x7f21100b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 -- 172.21.15.61:0/1947348939 >> v1:172.21.15.61:6789/0 conn(0x7f211005d160 legacy=0x7f2110142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 -- 172.21.15.61:0/1947348939 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21100b2620 con 0x7f21101461c0 2022-07-20T03:14:56.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210ffff700 1 --2- 172.21.15.61:0/1947348939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f21101465f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f21080032f0 tx=0x7f2108009290).ready entity=mon.0 client_cookie=b2b5187ad3e2c90e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210e7fc700 1 -- 172.21.15.61:0/1947348939 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210800b870 con 0x7f21101461c0 2022-07-20T03:14:56.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210e7fc700 1 -- 172.21.15.61:0/1947348939 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f210800b9d0 con 0x7f21101461c0 2022-07-20T03:14:56.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f210e7fc700 1 -- 172.21.15.61:0/1947348939 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210800bb40 con 0x7f21101461c0 2022-07-20T03:14:56.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- 172.21.15.61:0/1947348939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 msgr2=0x7f21101465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 --2- 172.21.15.61:0/1947348939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f21101465f0 secure :-1 s=READY pgs=855 cs=0 l=1 rev1=1 rx=0x7f21080032f0 tx=0x7f2108009290).stop 2022-07-20T03:14:56.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- 172.21.15.61:0/1947348939 shutdown_connections 2022-07-20T03:14:56.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 --2- 172.21.15.61:0/1947348939 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f21101465f0 unknown :-1 s=CLOSED pgs=855 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 --2- 172.21.15.61:0/1947348939 >> v2:172.21.15.117:3300/0 conn(0x7f21100b2ec0 0x7f21100b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- 172.21.15.61:0/1947348939 >> 172.21.15.61:0/1947348939 conn(0x7f2110066e20 msgr2=0x7f2110067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- 172.21.15.61:0/1947348939 shutdown_connections 2022-07-20T03:14:56.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- 172.21.15.61:0/1947348939 wait complete. 2022-07-20T03:14:56.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 Processor -- start 2022-07-20T03:14:56.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.465+0000 7f2114ec6700 1 -- start start 2022-07-20T03:14:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f211005d160 0x7f2110144460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21100b2ec0 0x7f211013a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f2110152af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2110155950 con 0x7f21101461c0 2022-07-20T03:14:56.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2110155b80 con 0x7f21100b2ec0 2022-07-20T03:14:56.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2110152650 con 0x7f211005d160 2022-07-20T03:14:56.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f2110152af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f211005d160 0x7f2110144460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f2110152af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47696/0 (socket says 172.21.15.61:47696) 2022-07-20T03:14:56.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 -- 172.21.15.61:0/2030701725 learned_addr learned my addr 172.21.15.61:0/2030701725 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:56.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210effd700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21100b2ec0 0x7f211013a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 -- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f211005d160 msgr2=0x7f2110144460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f211005d160 0x7f2110144460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 -- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21100b2ec0 msgr2=0x7f211013a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21100b2ec0 0x7f211013a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 -- 172.21.15.61:0/2030701725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2108009530 con 0x7f21101461c0 2022-07-20T03:14:56.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210ffff700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f2110152af0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21080032f0 tx=0x7f2108016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f210800b870 con 0x7f21101461c0 2022-07-20T03:14:56.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2108021630 con 0x7f21101461c0 2022-07-20T03:14:56.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f21101523e0 con 0x7f21101461c0 2022-07-20T03:14:56.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2108021990 con 0x7f21101461c0 2022-07-20T03:14:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.469+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f211013dbc0 con 0x7f21101461c0 2022-07-20T03:14:56.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.473+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2110152810 con 0x7f21101461c0 2022-07-20T03:14:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.473+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f210800f070 con 0x7f21101461c0 2022-07-20T03:14:56.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.473+0000 7f210cff9700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20f0053a60 0x7f20f0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.473+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f210806d210 con 0x7f21101461c0 2022-07-20T03:14:56.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.473+0000 7f210f7fe700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20f0053a60 0x7f20f0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.477+0000 7f210f7fe700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20f0053a60 0x7f20f0055f20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f210000b000 tx=0x7f2100006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.477+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2108044170 con 0x7f21101461c0 2022-07-20T03:14:56.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.601+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f21100b28c0 con 0x7f21101461c0 2022-07-20T03:14:56.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.601+0000 7f210cff9700 1 -- 172.21.15.61:0/2030701725 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v81) v1 ==== 100+0+10 (secure 0 0 0) 0x7f21080150a0 con 0x7f21101461c0 2022-07-20T03:14:56.606 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:14:56.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20f0053a60 msgr2=0x7f20f0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20f0053a60 0x7f20f0055f20 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7f210000b000 tx=0x7f2100006df0).stop 2022-07-20T03:14:56.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 msgr2=0x7f2110152af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f2110152af0 secure :-1 s=READY pgs=856 cs=0 l=1 rev1=1 rx=0x7f21080032f0 tx=0x7f2108016000).stop 2022-07-20T03:14:56.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 shutdown_connections 2022-07-20T03:14:56.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20f0053a60 0x7f20f0055f20 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f21100b2ec0 0x7f211013a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f21101461c0 0x7f2110152af0 unknown :-1 s=CLOSED pgs=856 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 --2- 172.21.15.61:0/2030701725 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f211005d160 0x7f2110144460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 >> 172.21.15.61:0/2030701725 conn(0x7f2110066e20 msgr2=0x7f2110067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 shutdown_connections 2022-07-20T03:14:56.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.605+0000 7f2114ec6700 1 -- 172.21.15.61:0/2030701725 wait complete. 2022-07-20T03:14:56.624 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:14:56.625 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:56.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.729+0000 7f3783fff700 1 Processor -- start 2022-07-20T03:14:56.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.729+0000 7f3783fff700 1 -- start start 2022-07-20T03:14:56.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.729+0000 7f3783fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37840af8e0 0x7f37840b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.729+0000 7f3783fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3784147010 con 0x7f37840adff0 2022-07-20T03:14:56.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.729+0000 7f3783fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37841472b0 con 0x7f37840b5650 2022-07-20T03:14:56.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.729+0000 7f3783fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3784146d40 con 0x7f37840af8e0 2022-07-20T03:14:56.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37827fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f37840adff0 0x7f378405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44298/0 (socket says 172.21.15.61:44298) 2022-07-20T03:14:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37827fc700 1 -- 172.21.15.61:0/3007131473 learned_addr learned my addr 172.21.15.61:0/3007131473 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:14:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37837fe700 1 --2- 172.21.15.61:0/3007131473 >> v2:172.21.15.61:3300/0 conn(0x7f37840af8e0 0x7f37840b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1139517485 0 0) 0x7f3784147010 con 0x7f37840adff0 2022-07-20T03:14:56.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3768003260 con 0x7f37840adff0 2022-07-20T03:14:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359215798 0 0) 0x7f37841472b0 con 0x7f37840b5650 2022-07-20T03:14:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3784147010 con 0x7f37840b5650 2022-07-20T03:14:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4019416543 0 0) 0x7f3768003260 con 0x7f37840adff0 2022-07-20T03:14:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37837fe700 1 -- 172.21.15.61:0/3007131473 >> v1:172.21.15.117:6789/0 conn(0x7f37840b5650 legacy=0x7f3784147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37837fe700 1 -- 172.21.15.61:0/3007131473 >> v1:172.21.15.61:6789/0 conn(0x7f37840adff0 legacy=0x7f378405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37837fe700 1 -- 172.21.15.61:0/3007131473 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f378405c4a0 con 0x7f37840af8e0 2022-07-20T03:14:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f37837fe700 1 --2- 172.21.15.61:0/3007131473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840af8e0 0x7f37840b5a30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f3778001290 tx=0x7f377c0091d0).ready entity=mon.0 client_cookie=b6a27f28720d43bc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37780017e0 con 0x7f37840af8e0 2022-07-20T03:14:56.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c00f510 con 0x7f37840af8e0 2022-07-20T03:14:56.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3781ffb700 1 -- 172.21.15.61:0/3007131473 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f377c00f680 con 0x7f37840af8e0 2022-07-20T03:14:56.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 -- 172.21.15.61:0/3007131473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840af8e0 msgr2=0x7f37840b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 --2- 172.21.15.61:0/3007131473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840af8e0 0x7f37840b5a30 secure :-1 s=READY pgs=858 cs=0 l=1 rev1=1 rx=0x7f3778001290 tx=0x7f377c0091d0).stop 2022-07-20T03:14:56.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 -- 172.21.15.61:0/3007131473 shutdown_connections 2022-07-20T03:14:56.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 --2- 172.21.15.61:0/3007131473 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840af8e0 0x7f37840b5a30 unknown :-1 s=CLOSED pgs=858 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 -- 172.21.15.61:0/3007131473 >> 172.21.15.61:0/3007131473 conn(0x7f37840634e0 msgr2=0x7f37840b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 -- 172.21.15.61:0/3007131473 shutdown_connections 2022-07-20T03:14:56.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 -- 172.21.15.61:0/3007131473 wait complete. 2022-07-20T03:14:56.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 Processor -- start 2022-07-20T03:14:56.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.733+0000 7f3783fff700 1 -- start start 2022-07-20T03:14:56.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37840adff0 0x7f3784156d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37840af8e0 0x7f3784063990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 0x7f378410f140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3784156350 con 0x7f37840b5650 2022-07-20T03:14:56.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37841565f0 con 0x7f37840adff0 2022-07-20T03:14:56.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3784156890 con 0x7f37840af8e0 2022-07-20T03:14:56.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 0x7f378410f140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 0x7f378410f140 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47708/0 (socket says 172.21.15.61:47708) 2022-07-20T03:14:56.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 -- 172.21.15.61:0/440040941 learned_addr learned my addr 172.21.15.61:0/440040941 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:56.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3782ffd700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37840adff0 0x7f3784156d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37827fc700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37840af8e0 0x7f3784063990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 -- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37840af8e0 msgr2=0x7f3784063990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37840af8e0 0x7f3784063990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 -- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37840adff0 msgr2=0x7f3784156d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37840adff0 0x7f3784156d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 -- 172.21.15.61:0/440040941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f377c009470 con 0x7f37840b5650 2022-07-20T03:14:56.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f37837fe700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 0x7f378410f140 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f3778001290 tx=0x7f377c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f377c0032f0 con 0x7f37840b5650 2022-07-20T03:14:56.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f377c011b80 con 0x7f37840b5650 2022-07-20T03:14:56.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3784157220 con 0x7f37840b5650 2022-07-20T03:14:56.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f377c021480 con 0x7f37840b5650 2022-07-20T03:14:56.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f378410e6d0 con 0x7f37840b5650 2022-07-20T03:14:56.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.737+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3784156a50 con 0x7f37840b5650 2022-07-20T03:14:56.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.741+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f377c00f070 con 0x7f37840b5650 2022-07-20T03:14:56.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.741+0000 7f3773fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3760053ad0 0x7f3760055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:56.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.741+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f377c06d0a0 con 0x7f37840b5650 2022-07-20T03:14:56.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.741+0000 7f3782ffd700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3760053ad0 0x7f3760055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:56.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.741+0000 7f3782ffd700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3760053ad0 0x7f3760055f90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f377400d000 tx=0x7f3774007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:56.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.745+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f377c043f60 con 0x7f37840b5650 2022-07-20T03:14:56.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.905+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f37840574e0 con 0x7f37840b5650 2022-07-20T03:14:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.905+0000 7f3773fff700 1 -- 172.21.15.61:0/440040941 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f377c015090 con 0x7f37840b5650 2022-07-20T03:14:56.912 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:56.912 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:14:49.076244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6643":{"gid":6643,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/843120801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":843120801},{"type":"v1","addr":"172.21.15.117:6835","nonce":843120801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:14:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3760053ad0 msgr2=0x7f3760055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3760053ad0 0x7f3760055f90 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7f377400d000 tx=0x7f3774007a20).stop 2022-07-20T03:14:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 msgr2=0x7f378410f140 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 0x7f378410f140 secure :-1 s=READY pgs=859 cs=0 l=1 rev1=1 rx=0x7f3778001290 tx=0x7f377c016000).stop 2022-07-20T03:14:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 shutdown_connections 2022-07-20T03:14:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3760053ad0 0x7f3760055f90 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37840b5650 0x7f378410f140 unknown :-1 s=CLOSED pgs=859 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37840af8e0 0x7f3784063990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 --2- 172.21.15.61:0/440040941 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37840adff0 0x7f3784156d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 >> 172.21.15.61:0/440040941 conn(0x7f37840634e0 msgr2=0x7f378405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 shutdown_connections 2022-07-20T03:14:56.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:56.909+0000 7f3783fff700 1 -- 172.21.15.61:0/440040941 wait complete. 2022-07-20T03:14:56.917 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-20T03:14:57.150 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 45, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:14:48.072382+0000', 'modified': '2022-07-20T03:14:49.076244+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 6643}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_6643': {'gid': 6643, 'name': 'b', 'rank': 0, 'incarnation': 44, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/843120801', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 843120801}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 843120801}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [16], 'metadata_pool': 15, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:14:57.151 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:14:57.152 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:14:57.301 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":6643,"want_state":"up:active","state":"up:active","rank_uptime":9.1953966470000008,"mdsmap_epoch":45,"osdmap_epoch":81,"osdmap_epoch_barrier":81,"uptime":13.594436965} 2022-07-20T03:14:57.318 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 6643, "mdsmap_epoch": 45, "osdmap_epoch": 81, "osdmap_epoch_barrier": 81, "rank_uptime": 9.195396647, "state": "up:active", "uptime": 13.594436965, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:14:57.341 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:14:57.371 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:14:57.372 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:14:57.372 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:57.373 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:57.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:14:57.455 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:14:57.457 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:14:57.488 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:14:57.512 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:14:57.518 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:57.518 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:14:57.663 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:14:57.664 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:57.700 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:14:57.792 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:14:57.793 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:14:57.811 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:14:57.769+0000 7f58d8d69080 -1 init, newargv = 0x564165b893a0 newargc=9ceph-fuse[22523]: starting ceph client 2022-07-20T03:14:57.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr: 2022-07-20T03:14:57.812 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22523]: starting fuse 2022-07-20T03:14:57.859 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:14:57.860 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:14:57.862 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:57.862 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:14:57.928 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:14:57.929 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:14:57.933 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:57.934 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:14:57.935 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:14:57.936 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:14:57.937 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:14:57.937 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:14:57.938 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:14:57.944 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:14:57.945 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:14:57.963 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:14:57.963 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:14:57.965 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:14:57.967 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:14:57.968 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:14:57.972 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:14:57.973 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:14:57.979 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:14:58.025 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:14:58.028 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:14:58.029 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:14:58.031 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:14:58.032 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:14:58.033 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:14:58.053 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:14:58.126 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22523.asok 2022-07-20T03:14:58.129 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22523.asok 2022-07-20T03:14:58.130 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:58.132 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22523.asok status 2022-07-20T03:14:58.266 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:14:58.267 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:14:58.267 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:14:58.267 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:14:58.268 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:14:58.268 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:14:58.268 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:14:58.268 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22523", 2022-07-20T03:14:58.269 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:14:58.269 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:14:58.269 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:14:58.269 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:14:58.270 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 6873, 2022-07-20T03:14:58.271 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:14:58.271 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:14:58.271 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:14:58.272 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 6873 2022-07-20T03:14:58.272 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:14:58.272 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:14:58.273 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:14:58.273 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:14:58.273 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2332513678 2022-07-20T03:14:58.274 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:14:58.274 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:14:58.274 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:14:58.274 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:14:58.275 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:14:58.275 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2332513678 2022-07-20T03:14:58.275 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:14:58.275 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.6873 172.21.15.163:0/2332513678", 2022-07-20T03:14:58.276 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/2332513678", 2022-07-20T03:14:58.276 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:14:58.276 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 45, 2022-07-20T03:14:58.276 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 81, 2022-07-20T03:14:58.277 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:14:58.277 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:14:58.277 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:14:58.277 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:14:58.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:58.280 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:58.280 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:58.300 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:14:58.301 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:14:58.302 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:58.303 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:58.367 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:14:58.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 Processor -- start 2022-07-20T03:14:58.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 -- start start 2022-07-20T03:14:58.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c1413e690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:58.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c140af150 0x7f9c1413eb80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:58.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c14064140 con 0x7f9c14063d60 2022-07-20T03:14:58.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c14143800 con 0x7f9c140af150 2022-07-20T03:14:58.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c1359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14143aa0 con 0x7f9c140b1440 2022-07-20T03:14:58.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c12d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c140af150 0x7f9c1413eb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:58.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c12d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c140af150 0x7f9c1413eb80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47716/0 (socket says 172.21.15.61:47716) 2022-07-20T03:14:58.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c12d9d700 1 -- 172.21.15.61:0/2321705262 learned_addr learned my addr 172.21.15.61:0/2321705262 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:58.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c0bfff700 1 --2- 172.21.15.61:0/2321705262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c1413e690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:58.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c12d9d700 1 -- 172.21.15.61:0/2321705262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 msgr2=0x7f9c1413e690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:58.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c12d9d700 1 --2- 172.21.15.61:0/2321705262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c1413e690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.481+0000 7f9c12d9d700 1 -- 172.21.15.61:0/2321705262 >> v1:172.21.15.117:6789/0 conn(0x7f9c14063d60 legacy=0x7f9c140b57e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:58.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c12d9d700 1 -- 172.21.15.61:0/2321705262 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c140b1050 con 0x7f9c140af150 2022-07-20T03:14:58.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c12d9d700 1 --2- 172.21.15.61:0/2321705262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c1413eb80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f9c0c0032f0 tx=0x7f9c0c009290).ready entity=mon.0 client_cookie=fd82fe6ed0142f43 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:58.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c11d9b700 1 -- 172.21.15.61:0/2321705262 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c0c00ba90 con 0x7f9c140af150 2022-07-20T03:14:58.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c11d9b700 1 -- 172.21.15.61:0/2321705262 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c0c00bbf0 con 0x7f9c140af150 2022-07-20T03:14:58.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c11d9b700 1 -- 172.21.15.61:0/2321705262 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c0c010470 con 0x7f9c140af150 2022-07-20T03:14:58.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 -- 172.21.15.61:0/2321705262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 msgr2=0x7f9c1413eb80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:58.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 --2- 172.21.15.61:0/2321705262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c1413eb80 secure :-1 s=READY pgs=861 cs=0 l=1 rev1=1 rx=0x7f9c0c0032f0 tx=0x7f9c0c009290).stop 2022-07-20T03:14:58.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 -- 172.21.15.61:0/2321705262 shutdown_connections 2022-07-20T03:14:58.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 --2- 172.21.15.61:0/2321705262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c1413eb80 unknown :-1 s=CLOSED pgs=861 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 --2- 172.21.15.61:0/2321705262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c1413e690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 -- 172.21.15.61:0/2321705262 >> 172.21.15.61:0/2321705262 conn(0x7f9c140634e0 msgr2=0x7f9c140b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:58.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 -- 172.21.15.61:0/2321705262 shutdown_connections 2022-07-20T03:14:58.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 -- 172.21.15.61:0/2321705262 wait complete. 2022-07-20T03:14:58.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 Processor -- start 2022-07-20T03:14:58.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.485+0000 7f9c1359e700 1 -- start start 2022-07-20T03:14:58.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c14063d60 0x7f9c14065560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:58.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c14141150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:58.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c140665a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:58.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14064350 con 0x7f9c140af150 2022-07-20T03:14:58.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14114140 con 0x7f9c14063d60 2022-07-20T03:14:58.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c14064580 con 0x7f9c140b1440 2022-07-20T03:14:58.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c14141150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:58.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c14141150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47720/0 (socket says 172.21.15.61:47720) 2022-07-20T03:14:58.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 -- 172.21.15.61:0/4250644987 learned_addr learned my addr 172.21.15.61:0/4250644987 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:58.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1259c700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c14063d60 0x7f9c14065560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:58.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c12d9d700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c140665a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:58.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 -- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 msgr2=0x7f9c140665a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:58.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c140665a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 -- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c14063d60 msgr2=0x7f9c14065560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:58.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c14063d60 0x7f9c14065560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 -- 172.21.15.61:0/4250644987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c0c009530 con 0x7f9c140af150 2022-07-20T03:14:58.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0bfff700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c14141150 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7f9bfc00c000 tx=0x7f9bfc006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:58.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bfc011750 con 0x7f9c140af150 2022-07-20T03:14:58.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9bfc0118b0 con 0x7f9c140af150 2022-07-20T03:14:58.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c14113e60 con 0x7f9c140af150 2022-07-20T03:14:58.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9bfc011c10 con 0x7f9c140af150 2022-07-20T03:14:58.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.489+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c14066310 con 0x7f9c140af150 2022-07-20T03:14:58.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.493+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9bfc011d70 con 0x7f9c140af150 2022-07-20T03:14:58.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.493+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c1413f070 con 0x7f9c140af150 2022-07-20T03:14:58.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.493+0000 7f9c0b7fe700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9bf8053ad0 0x7f9bf8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:58.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.493+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9bfc06c140 con 0x7f9c140af150 2022-07-20T03:14:58.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.493+0000 7f9c1259c700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9bf8053ad0 0x7f9bf8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:58.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.493+0000 7f9c1259c700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9bf8053ad0 0x7f9bf8055f90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f9c0400d000 tx=0x7f9c04000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:58.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.497+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9bfc042fa0 con 0x7f9c140af150 2022-07-20T03:14:58.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.661+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c140574e0 con 0x7f9c140af150 2022-07-20T03:14:58.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.661+0000 7f9c0b7fe700 1 -- 172.21.15.61:0/4250644987 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2140 (secure 0 0 0) 0x7f9bfc0465c0 con 0x7f9c140af150 2022-07-20T03:14:58.667 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:58.668 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:14:49.076244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6643":{"gid":6643,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/843120801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":843120801},{"type":"v1","addr":"172.21.15.117:6835","nonce":843120801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:14:58.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9bf8053ad0 msgr2=0x7f9bf8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:58.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9bf8053ad0 0x7f9bf8055f90 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7f9c0400d000 tx=0x7f9c04000bc0).stop 2022-07-20T03:14:58.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 msgr2=0x7f9c14141150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:58.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c14141150 secure :-1 s=READY pgs=862 cs=0 l=1 rev1=1 rx=0x7f9bfc00c000 tx=0x7f9bfc006d90).stop 2022-07-20T03:14:58.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 shutdown_connections 2022-07-20T03:14:58.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9bf8053ad0 0x7f9bf8055f90 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c140b1440 0x7f9c140665a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c140af150 0x7f9c14141150 unknown :-1 s=CLOSED pgs=862 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 --2- 172.21.15.61:0/4250644987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c14063d60 0x7f9c14065560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:58.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 >> 172.21.15.61:0/4250644987 conn(0x7f9c140634e0 msgr2=0x7f9c140b0630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:58.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 shutdown_connections 2022-07-20T03:14:58.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:58.665+0000 7f9c1359e700 1 -- 172.21.15.61:0/4250644987 wait complete. 2022-07-20T03:14:58.676 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-20T03:14:58.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json dump inode 1 2022-07-20T03:14:58.822 INFO:teuthology.orchestra.run.smithi117.stdout:{"path":"/","ino":1,"rdev":0,"ctime":"2022-07-20T03:14:58.361958+0000","btime":"2022-07-20T03:14:48.078467+0000","mode":17407,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":4194304,"stripe_count":1,"object_size":4194304,"pool_id":16,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":0,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-20T03:14:48.078467+0000","atime":"0.000000","time_warp_seq":0,"change_attr":2,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-20T03:14:58.361958+0000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"2022-07-20T03:14:48.078467+0000"},"version":4,"file_data_version":0,"xattr_version":1,"backtrace_version":3,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"lock":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1,"authpin":0},"nref":4,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{},"linklock":{},"dirfragtreelock":{},"filelock":{},"xattrlock":{},"snaplock":{},"nestlock":{},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent"],"client_caps":[{"client_id":6814,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2},{"client_id":6873,"pending":"pLsXsFs","issued":"pLsXsFs","wanted":"-","last_sent":2}],"loner":-1,"want_loner":-1,"mds_caps_wanted":[]} 2022-07-20T03:14:58.824 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "2022-07-20T03:14:48.078467+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "0.000000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": {}, "backtrace_version": 3, "btime": "2022-07-20T03:14:48.078467+0000", "change_attr": 2, "client_caps": [ { "client_id": 6814, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" }, { "client_id": 6873, "issued": "pLsXsFs", "last_sent": 2, "pending": "pLsXsFs", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-20T03:14:58.361958+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": {}, "flocklock": {}, "gid": 0, "ino": 1, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 4194304, "pool_id": 16, "pool_ns": "", "stripe_count": 1, "stripe_unit": 4194304 }, "linklock": {}, "loner": -1, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 17407, "mtime": "2022-07-20T03:14:48.078467+0000", "nestlock": {}, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/", "pins": { "authpin": 0, "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "lock": 0, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "2022-07-20T03:14:58.361958+0000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 0, "truncate_size": 18446744073709551615, "uid": 0, "version": 4, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": -1, "xattr_version": 1, "xattrlock": {}, "xattrs": [] } 2022-07-20T03:14:58.825 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:14:58.826 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:58.849 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:58.872 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:58.895 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:58.918 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:14:58.952 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:14:58.953 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:58.953 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:58.954 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:58.963 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:14:58.964 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:14:58.965 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:14:58.966 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:14:58.966 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:59.027 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23073]: fuse finished with error 0 and tester_r 0 2022-07-20T03:14:59.035 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:59.036 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:59.037 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:59.059 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:14:59.060 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:14:59.078 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:59.079 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:14:59.127 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:59.128 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:59.129 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:59.140 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:14:59.142 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:14:59.143 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:14:59.143 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:14:59.144 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:59.175 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22523]: fuse finished with error 0 and tester_r 0 2022-07-20T03:14:59.176 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:59.177 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:59.177 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:59.226 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:14:59.227 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:14:59.364 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:59.365 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:14:59.373 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:14:59.374 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:14:59.375 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:14:59.387 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:14:59.388 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:59.390 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:14:59.391 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:14:59.392 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:14:59.417 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:14:59.438 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:14:59.439 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:14:59.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 Processor -- start 2022-07-20T03:14:59.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- start start 2022-07-20T03:14:59.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f9c0aea20 0x7f9f9c140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9f9c0b1a70 0x7f9f9c0b1f20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f9c0aee00 con 0x7f9f9c0675f0 2022-07-20T03:14:59.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c144470 con 0x7f9f9c0aea20 2022-07-20T03:14:59.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f9c144710 con 0x7f9f9c0b1a70 2022-07-20T03:14:59.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f9c0aea20 0x7f9f9c140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:59.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f9c0aea20 0x7f9f9c140a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47726/0 (socket says 172.21.15.61:47726) 2022-07-20T03:14:59.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 -- 172.21.15.61:0/2748244460 learned_addr learned my addr 172.21.15.61:0/2748244460 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:59.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9affd700 1 --1- 172.21.15.61:0/2748244460 >> v1:172.21.15.61:6789/0 conn(0x7f9f9c0675f0 0x7f9f9c140f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44326/0 (socket says 172.21.15.61:44326) 2022-07-20T03:14:59.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9bfff700 1 --2- 172.21.15.61:0/2748244460 >> v2:172.21.15.117:3300/0 conn(0x7f9f9c0b1a70 0x7f9f9c0b1f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:59.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 -- 172.21.15.61:0/2748244460 >> v2:172.21.15.117:3300/0 conn(0x7f9f9c0b1a70 msgr2=0x7f9f9c0b1f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:14:59.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 --2- 172.21.15.61:0/2748244460 >> v2:172.21.15.117:3300/0 conn(0x7f9f9c0b1a70 0x7f9f9c0b1f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:59.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 -- 172.21.15.61:0/2748244460 >> v1:172.21.15.61:6789/0 conn(0x7f9f9c0675f0 legacy=0x7f9f9c140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:59.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 -- 172.21.15.61:0/2748244460 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f9c1449b0 con 0x7f9f9c0aea20 2022-07-20T03:14:59.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9b7fe700 1 --2- 172.21.15.61:0/2748244460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c0aea20 0x7f9f9c140a90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f9f9c0b1750 tx=0x7f9f8c009290).ready entity=mon.0 client_cookie=31459aeddc6748a3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:59.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9a7fc700 1 -- 172.21.15.61:0/2748244460 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f8c00b860 con 0x7f9f9c0aea20 2022-07-20T03:14:59.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9a7fc700 1 -- 172.21.15.61:0/2748244460 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f8c00b9c0 con 0x7f9f9c0aea20 2022-07-20T03:14:59.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9f9a7fc700 1 -- 172.21.15.61:0/2748244460 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f8c00bb30 con 0x7f9f9c0aea20 2022-07-20T03:14:59.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2748244460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c0aea20 msgr2=0x7f9f9c140a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:59.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2748244460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c0aea20 0x7f9f9c140a90 secure :-1 s=READY pgs=864 cs=0 l=1 rev1=1 rx=0x7f9f9c0b1750 tx=0x7f9f8c009290).stop 2022-07-20T03:14:59.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2748244460 shutdown_connections 2022-07-20T03:14:59.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2748244460 >> v2:172.21.15.117:3300/0 conn(0x7f9f9c0b1a70 0x7f9f9c0b1f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:59.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2748244460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c0aea20 0x7f9f9c140a90 unknown :-1 s=CLOSED pgs=864 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:59.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2748244460 >> 172.21.15.61:0/2748244460 conn(0x7f9f9c0611c0 msgr2=0x7f9f9c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:14:59.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2748244460 shutdown_connections 2022-07-20T03:14:59.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.549+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2748244460 wait complete. 2022-07-20T03:14:59.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 Processor -- start 2022-07-20T03:14:59.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 -- start start 2022-07-20T03:14:59.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f9c0aea20 0x7f9f9c064a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:59.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c0b1a70 0x7f9f9c066140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:59.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 0x7f9f9c1373b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:59.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c066cc0 con 0x7f9f9c153440 2022-07-20T03:14:59.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c152fb0 con 0x7f9f9c0b1a70 2022-07-20T03:14:59.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9fa1f84700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f9c064480 con 0x7f9f9c0aea20 2022-07-20T03:14:59.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 0x7f9f9c1373b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:59.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9b7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f9c0aea20 0x7f9f9c064a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:59.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 0x7f9f9c1373b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47736/0 (socket says 172.21.15.61:47736) 2022-07-20T03:14:59.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 -- 172.21.15.61:0/2914901034 learned_addr learned my addr 172.21.15.61:0/2914901034 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:14:59.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9affd700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c0b1a70 0x7f9f9c066140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:59.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 -- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f9c0aea20 msgr2=0x7f9f9c064a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:59.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f9c0aea20 0x7f9f9c064a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:59.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 -- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c0b1a70 msgr2=0x7f9f9c066140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:14:59.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c0b1a70 0x7f9f9c066140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:14:59.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 -- 172.21.15.61:0/2914901034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c009530 con 0x7f9f9c153440 2022-07-20T03:14:59.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f9bfff700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 0x7f9f9c1373b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f9f9400c000 tx=0x7f9f94006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:59.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.553+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f94011750 con 0x7f9f9c153440 2022-07-20T03:14:59.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f940118b0 con 0x7f9f9c153440 2022-07-20T03:14:59.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f9c064720 con 0x7f9f9c153440 2022-07-20T03:14:59.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f94011c10 con 0x7f9f9c153440 2022-07-20T03:14:59.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f9c0659a0 con 0x7f9f9c153440 2022-07-20T03:14:59.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f9c144e50 con 0x7f9f9c153440 2022-07-20T03:14:59.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f94011d70 con 0x7f9f9c153440 2022-07-20T03:14:59.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9f98ff9700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f7c0539d0 0x7f9f7c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:14:59.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f9406c160 con 0x7f9f9c153440 2022-07-20T03:14:59.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.557+0000 7f9f9b7fe700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f7c0539d0 0x7f9f7c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:14:59.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.561+0000 7f9f9b7fe700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f7c0539d0 0x7f9f7c055e90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f9f8c01b000 tx=0x7f9f8c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:14:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.565+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f940430c0 con 0x7f9f9c153440 2022-07-20T03:14:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.693+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f9f9c0574e0 con 0x7f9f9c153440 2022-07-20T03:14:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.693+0000 7f9f98ff9700 1 -- 172.21.15.61:0/2914901034 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v81) v1 ==== 81+0+33696 (secure 0 0 0) 0x7f9f940466e0 con 0x7f9f9c153440 2022-07-20T03:14:59.704 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:14:59.705 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:14:59.705 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 81, 2022-07-20T03:14:59.710 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:14:59.710 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:14:59.711 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:14:49.075215+0000", 2022-07-20T03:14:59.711 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:14:59.712 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:14:59.712 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:14:59.713 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:14:59.713 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:14:59.714 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:14:59.714 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:14:59.715 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:14:59.715 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:14:59.716 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:59.716 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:14:59.716 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:14:59.717 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:14:59.717 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:14:59.718 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:14:59.728 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 16, 2022-07-20T03:14:59.728 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:14:59.729 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:14:59.729 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:14:59.730 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:14:59.730 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:14:59.731 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.731 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:14:59.731 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:14:59.732 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:14:59.732 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:59.733 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:59.733 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:59.734 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:59.734 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:59.735 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:59.735 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:59.736 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:59.736 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:14:59.737 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:14:59.737 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:14:59.738 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:14:59.738 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:14:59.738 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:59.739 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:59.740 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:59.740 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:59.741 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:59.741 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:59.742 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:59.742 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.742 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:14:59.743 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:59.743 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:59.744 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:59.744 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:59.745 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:59.745 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:59.746 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:59.746 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:59.747 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:59.747 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:59.747 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:59.748 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:59.748 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:59.749 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:59.749 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:59.750 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:59.750 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:59.751 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:59.751 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:59.751 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:59.752 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:59.752 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:59.753 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:59.753 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:59.754 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:59.754 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:59.755 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.755 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:59.755 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:59.756 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:59.756 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:59.757 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:59.757 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:59.758 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:59.758 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:59.759 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:59.759 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:59.760 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:59.760 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:14:59.760 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:14:59.761 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.761 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:59.762 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:14:59.762 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.763 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.763 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.764 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:14:59.764 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:14:59.764 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:14:59.765 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:59.765 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:59.766 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:59.766 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:59.767 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:59.767 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:59.768 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:59.768 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:59.769 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:59.769 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:59.769 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:59.770 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:59.770 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:59.771 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:59.771 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:59.772 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:59.772 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:59.773 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:59.773 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:59.773 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:59.774 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.774 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:14:59.775 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:59.775 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:59.776 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:59.776 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:59.777 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:59.777 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:59.777 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:59.778 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:59.778 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:59.779 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:59.779 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:59.780 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:59.780 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:59.781 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:59.781 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:59.781 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:59.782 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:59.782 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:59.783 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:59.783 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:59.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:59.784 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:59.785 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:59.785 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:59.786 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:59.786 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:59.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.787 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:59.787 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:59.788 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:59.790 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:59.803 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:59.804 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:59.804 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:59.805 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:59.805 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:59.806 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:59.806 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:59.806 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:14:59.807 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:59.807 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:14:59.808 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.808 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.809 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.809 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 15, 2022-07-20T03:14:59.810 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:14:59.810 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:14:46.661180+0000", 2022-07-20T03:14:59.811 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:59.811 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:59.811 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:59.812 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:59.812 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:59.813 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:59.813 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:59.814 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:59.814 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:59.815 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:59.815 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:59.815 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:59.816 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:59.816 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:59.817 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:59.817 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:59.823 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:59.824 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:59.826 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:59.826 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:59.826 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.827 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "81", 2022-07-20T03:14:59.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:59.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:59.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:59.836 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:59.836 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:59.837 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:59.837 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:59.843 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:59.843 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:59.845 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:59.845 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:59.846 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:59.847 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:59.847 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:59.848 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:59.851 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:59.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.852 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:59.852 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:59.854 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:59.854 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:59.855 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:59.855 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:59.856 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:59.856 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:14:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:14:59.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:14:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:14:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:14:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:14:59.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.862 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.862 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.862 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 16, 2022-07-20T03:14:59.863 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:14:59.863 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:14:47.668660+0000", 2022-07-20T03:14:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:14:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:14:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:14:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:14:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:14:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:14:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:14:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:14:59.867 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:14:59.867 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:14:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:14:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:14:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:14:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:14:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:14:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:14:59.870 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:14:59.871 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:14:59.871 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:14:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:14:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "81", 2022-07-20T03:14:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:14:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:14:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:14:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:14:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:14:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:14:59.875 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:14:59.875 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:14:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:14:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:14:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:14:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:14:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:14:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:14:59.878 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:14:59.878 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:14:59.879 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:14:59.879 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:14:59.880 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:14:59.880 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:14:59.880 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:14:59.881 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:14:59.881 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:14:59.887 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:14:59.887 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:14:59.888 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:14:59.889 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.889 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:14:59.891 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:14:59.892 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:14:59.893 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:14:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:14:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:14:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:14:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:14:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:14:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:14:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:14:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:14:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:14:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:14:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:14:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:14:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:14:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:14:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:14:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:14:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:14:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:14:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:59.911 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.911 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.911 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.912 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:14:59.912 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.914 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.914 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:14:59.915 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.915 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.916 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.916 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.918 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.919 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:14:59.920 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.920 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.921 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.921 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.922 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:14:59.922 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.923 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.923 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.925 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.925 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.926 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:14:59.926 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.927 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.927 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.928 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.928 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:14:59.929 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:14:59.929 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.930 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.930 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.931 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:14:59.931 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:14:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:14:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:14:59.933 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:59.933 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:59.934 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:59.934 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:14:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:14:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:59.937 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:59.937 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:59.938 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:14:59.938 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:14:59.939 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:14:59.939 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:14:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:14:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:14:59.941 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:14:59.944 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:14:59.945 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.945 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.959 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.960 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:14:59.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.961 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.971 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:14:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.974 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:14:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:14:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.978 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:14:59.978 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.980 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.981 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:14:59.981 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.982 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.982 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:14:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.984 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.984 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.985 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:14:59.985 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.987 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.987 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:14:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:14:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:14:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:14:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.990 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.990 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:14:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:14:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:14:59.992 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:14:59.992 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.993 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.993 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:14:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:14:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:14:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:14:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:14:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:14:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:14:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:14:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:14:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:14:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:14:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:14:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:14:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:14:59.999 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:14:59.999 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:00.002 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:00.002 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.003 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.003 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:15:00.014 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.015 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.015 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:15:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.039 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:00.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:15:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:15:00.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.043 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:15:00.046 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:15:00.048 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.066 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.068 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:15:00.068 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:15:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.072 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.072 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:15:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:15:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:15:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:15:00.074 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:00.074 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:00.075 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:00.075 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:00.098 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:15:00.098 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:00.099 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:00.099 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:00.100 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:00.100 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:00.100 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:00.101 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:00.101 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:00.113 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:00.113 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:00.114 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:00.114 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.115 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.116 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.117 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:15:00.117 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.118 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.120 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:15:00.120 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.121 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.121 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.122 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.122 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:00.123 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.123 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:15:00.125 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.126 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.126 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:15:00.127 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.128 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.128 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.130 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:00.131 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.132 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:15:00.133 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.133 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.134 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:15:00.135 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.136 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.136 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.137 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:00.138 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:15:00.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.141 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.141 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.142 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:15:00.142 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:00.143 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.143 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.144 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.144 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:15:00.145 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:15:00.145 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:15:00.146 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:15:00.146 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:00.146 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:00.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:00.147 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.149 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:00.149 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:15:00.150 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:00.150 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:00.151 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:00.151 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:00.152 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:00.152 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:00.153 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:00.153 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:00.153 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:00.154 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:00.154 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:00.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.155 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.155 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.156 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:15:00.156 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.157 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.157 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.158 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:15:00.158 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.159 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.159 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.160 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:00.161 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.162 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:15:00.163 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.164 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.164 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.165 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:15:00.165 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.166 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.167 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.167 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.168 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:00.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.169 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:15:00.170 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.172 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:15:00.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.175 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:00.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.178 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:15:00.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.180 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:15:00.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:00.182 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.182 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.183 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:15:00.184 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:15:00.184 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:15:00.185 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:15:00.185 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:00.186 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:00.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:00.187 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.189 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:00.189 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:15:00.190 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:00.190 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:00.190 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:00.191 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:00.191 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:00.192 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:00.192 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:00.193 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:00.193 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:00.194 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:00.194 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:00.195 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.195 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.196 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.196 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:15:00.196 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.197 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.198 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:15:00.199 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.199 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.200 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.200 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.201 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:00.201 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.202 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.202 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.203 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:15:00.203 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.204 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.204 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.205 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:15:00.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.206 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.206 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.207 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:00.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.208 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.209 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:15:00.210 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:15:00.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.212 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.213 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.214 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:00.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:15:00.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.217 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:15:00.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:00.217 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.217 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.218 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:15:00.218 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:15:00.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:15:00.219 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:15:00.219 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:00.219 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:00.219 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:00.220 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.220 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:00.221 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:15:00.221 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:00.221 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:00.221 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:00.222 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:00.222 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:00.222 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:00.223 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:00.223 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:00.223 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:00.223 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:00.223 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:00.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.224 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:15:00.225 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.225 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.225 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.226 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:15:00.226 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.226 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.227 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.227 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.227 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:00.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.227 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.228 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:15:00.228 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.228 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.229 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:15:00.229 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.230 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.230 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.230 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:00.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.231 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:15:00.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.232 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:15:00.233 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.233 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.233 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.233 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:00.234 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.234 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.234 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:15:00.235 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.235 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.236 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:15:00.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:00.236 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.236 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.237 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:15:00.237 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:15:00.237 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:15:00.238 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:15:00.238 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:00.238 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:00.238 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:00.239 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.239 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.239 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.239 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:00.239 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:15:00.240 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:00.240 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:00.240 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:00.240 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:00.241 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:00.241 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:00.241 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:00.241 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-20T03:15:00.242 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:00.242 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:00.242 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:00.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.243 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:15:00.243 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:15:00.245 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.245 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.245 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.245 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:00.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.246 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:15:00.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.247 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:15:00.248 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.248 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.248 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.249 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.249 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:00.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:15:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.250 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:15:00.251 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:00.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:00.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.253 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:00.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:15:00.253 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.254 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:00.255 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:15:00.255 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:00.255 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.255 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.256 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:15:00.256 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:15:00.256 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:15:00.256 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:15:00.257 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:00.257 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:00.257 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:00.258 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:00.258 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.258 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:00.258 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:15:00.259 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.259 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:00.259 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.259 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.259 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.260 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.260 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.260 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:15:00.260 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.261 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.261 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:00.261 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.268 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.269 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.269 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.269 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.269 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:15:00.270 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.270 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.270 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.270 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:00.271 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.271 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.271 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.271 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.272 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:15:00.272 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.272 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.273 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:00.273 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.274 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.274 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.274 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.274 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.275 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:15:00.275 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.275 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.276 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.276 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:00.277 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.277 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.278 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.278 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.278 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.278 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:15:00.279 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.279 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.279 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.279 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:00.280 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.280 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.280 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.280 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.281 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:15:00.282 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.283 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:00.283 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.283 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.283 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.284 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.284 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.284 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:15:00.285 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.285 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.285 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:00.286 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:00.286 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:00.286 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:00.286 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:00.286 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:00.287 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:00.287 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:15:00.310 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:00.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.311 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:00.311 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:15:00.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:15:00.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:15:00.313 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:15:00.314 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:15:00.314 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:15:00.315 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:15:00.315 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:15:00.316 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:15:00.316 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:15:00.346 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:15:00.346 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:15:00.348 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:15:00.348 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:00.349 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:00.349 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:15:00.349 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:15:00.350 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:15:00.350 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:15:00.351 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:15:00.351 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:00.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f7c0539d0 msgr2=0x7f9f7c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f7c0539d0 0x7f9f7c055e90 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7f9f8c01b000 tx=0x7f9f8c016000).stop 2022-07-20T03:15:00.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 msgr2=0x7f9f9c1373b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 0x7f9f9c1373b0 secure :-1 s=READY pgs=865 cs=0 l=1 rev1=1 rx=0x7f9f9400c000 tx=0x7f9f94006df0).stop 2022-07-20T03:15:00.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 shutdown_connections 2022-07-20T03:15:00.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f7c0539d0 0x7f9f7c055e90 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f9c0b1a70 0x7f9f9c066140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f9c153440 0x7f9f9c1373b0 unknown :-1 s=CLOSED pgs=865 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 --2- 172.21.15.61:0/2914901034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f9c0aea20 0x7f9f9c064a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 >> 172.21.15.61:0/2914901034 conn(0x7f9f9c0675f0 msgr2=0x7f9f9c0b1f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:00.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 shutdown_connections 2022-07-20T03:15:00.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:14:59.697+0000 7f9fa1f84700 1 -- 172.21.15.61:0/2914901034 wait complete. 2022-07-20T03:15:00.361 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:00.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 Processor -- start 2022-07-20T03:15:00.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 -- start start 2022-07-20T03:15:00.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aecf0 0x7fcbe4140240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcbe40aff80 0x7fcbe405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbe40b5d70 con 0x7fcbe40b5990 2022-07-20T03:15:00.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbe40651c0 con 0x7fcbe40aff80 2022-07-20T03:15:00.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbea216700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe40b0690 con 0x7fcbe40aecf0 2022-07-20T03:15:00.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcbe40aff80 0x7fcbe405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcbe40aff80 0x7fcbe405bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47744/0 (socket says 172.21.15.61:47744) 2022-07-20T03:15:00.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 -- 172.21.15.61:0/2308080434 learned_addr learned my addr 172.21.15.61:0/2308080434 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:00.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe8a13700 1 --2- 172.21.15.61:0/2308080434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aecf0 0x7fcbe4140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 -- 172.21.15.61:0/2308080434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aecf0 msgr2=0x7fcbe4140240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:00.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 --2- 172.21.15.61:0/2308080434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aecf0 0x7fcbe4140240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 -- 172.21.15.61:0/2308080434 >> v1:172.21.15.61:6789/0 conn(0x7fcbe40b5990 legacy=0x7fcbe413fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 -- 172.21.15.61:0/2308080434 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbe40b0930 con 0x7fcbe40aff80 2022-07-20T03:15:00.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.473+0000 7fcbe9a15700 1 --2- 172.21.15.61:0/2308080434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aff80 0x7fcbe405bd90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fcbdc0032f0 tx=0x7fcbdc009290).ready entity=mon.0 client_cookie=b0b95b83ef6bc29f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:00.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbd3fff700 1 -- 172.21.15.61:0/2308080434 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbdc00b8a0 con 0x7fcbe40aff80 2022-07-20T03:15:00.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbd3fff700 1 -- 172.21.15.61:0/2308080434 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbdc00ba00 con 0x7fcbe40aff80 2022-07-20T03:15:00.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbd3fff700 1 -- 172.21.15.61:0/2308080434 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbdc00bb70 con 0x7fcbe40aff80 2022-07-20T03:15:00.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 -- 172.21.15.61:0/2308080434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aff80 msgr2=0x7fcbe405bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 --2- 172.21.15.61:0/2308080434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aff80 0x7fcbe405bd90 secure :-1 s=READY pgs=867 cs=0 l=1 rev1=1 rx=0x7fcbdc0032f0 tx=0x7fcbdc009290).stop 2022-07-20T03:15:00.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 -- 172.21.15.61:0/2308080434 shutdown_connections 2022-07-20T03:15:00.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 --2- 172.21.15.61:0/2308080434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aff80 0x7fcbe405bd90 unknown :-1 s=CLOSED pgs=867 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 --2- 172.21.15.61:0/2308080434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aecf0 0x7fcbe4140240 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 -- 172.21.15.61:0/2308080434 >> 172.21.15.61:0/2308080434 conn(0x7fcbe40634e0 msgr2=0x7fcbe40b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:00.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 -- 172.21.15.61:0/2308080434 shutdown_connections 2022-07-20T03:15:00.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 -- 172.21.15.61:0/2308080434 wait complete. 2022-07-20T03:15:00.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 Processor -- start 2022-07-20T03:15:00.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.477+0000 7fcbea216700 1 -- start start 2022-07-20T03:15:00.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 0x7fcbe4141c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aff80 0x7fcbe4066690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe40b5990 0x7fcbe41443b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4139120 con 0x7fcbe40aecf0 2022-07-20T03:15:00.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe4138e00 con 0x7fcbe40b5990 2022-07-20T03:15:00.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbe41438c0 con 0x7fcbe40aff80 2022-07-20T03:15:00.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 0x7fcbe4141c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 0x7fcbe4141c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47746/0 (socket says 172.21.15.61:47746) 2022-07-20T03:15:00.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 -- 172.21.15.61:0/3592661943 learned_addr learned my addr 172.21.15.61:0/3592661943 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:00.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe8a13700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aff80 0x7fcbe4066690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9a15700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe40b5990 0x7fcbe41443b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 -- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aff80 msgr2=0x7fcbe4066690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aff80 0x7fcbe4066690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 -- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe40b5990 msgr2=0x7fcbe41443b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe40b5990 0x7fcbe41443b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 -- 172.21.15.61:0/3592661943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbdc009530 con 0x7fcbe40aecf0 2022-07-20T03:15:00.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbe9214700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 0x7fcbe4141c70 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fcbe4067c30 tx=0x7fcbe0007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:00.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbe0012400 con 0x7fcbe40aecf0 2022-07-20T03:15:00.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcbe0012560 con 0x7fcbe40aecf0 2022-07-20T03:15:00.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbe413abc0 con 0x7fcbe40aecf0 2022-07-20T03:15:00.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcbe00128c0 con 0x7fcbe40aecf0 2022-07-20T03:15:00.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbe4143f60 con 0x7fcbe40aecf0 2022-07-20T03:15:00.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.481+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbe4143a80 con 0x7fcbe40aecf0 2022-07-20T03:15:00.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.485+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcbe0019030 con 0x7fcbe40aecf0 2022-07-20T03:15:00.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.485+0000 7fcbd27fc700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcbc40539b0 0x7fcbc4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.485+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcbe006b120 con 0x7fcbe40aecf0 2022-07-20T03:15:00.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.485+0000 7fcbe8a13700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcbc40539b0 0x7fcbc4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.485+0000 7fcbe8a13700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcbc40539b0 0x7fcbc4055e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fcbd4006000 tx=0x7fcbd4006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:00.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.489+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcbe4143a80 con 0x7fcbe40aecf0 2022-07-20T03:15:00.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.645+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcbe4067930 con 0x7fcbe40aecf0 2022-07-20T03:15:00.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbd27fc700 1 -- 172.21.15.61:0/3592661943 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 45 v45) v1 ==== 94+0+2140 (secure 0 0 0) 0x7fcbe0042080 con 0x7fcbe40aecf0 2022-07-20T03:15:00.656 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:00.657 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":45,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":45,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:14:49.076244+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":6643},"failed":[],"damaged":[],"stopped":[],"info":{"gid_6643":{"gid":6643,"name":"b","rank":0,"incarnation":44,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/843120801","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":843120801},{"type":"v1","addr":"172.21.15.117:6835","nonce":843120801}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:15:00.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcbc40539b0 msgr2=0x7fcbc4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcbc40539b0 0x7fcbc4055e70 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fcbd4006000 tx=0x7fcbd4006d90).stop 2022-07-20T03:15:00.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 msgr2=0x7fcbe4141c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 0x7fcbe4141c70 secure :-1 s=READY pgs=868 cs=0 l=1 rev1=1 rx=0x7fcbe4067c30 tx=0x7fcbe0007c50).stop 2022-07-20T03:15:00.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 shutdown_connections 2022-07-20T03:15:00.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcbc40539b0 0x7fcbc4055e70 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbe40b5990 0x7fcbe41443b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbe40aff80 0x7fcbe4066690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 --2- 172.21.15.61:0/3592661943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbe40aecf0 0x7fcbe4141c70 unknown :-1 s=CLOSED pgs=868 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 >> 172.21.15.61:0/3592661943 conn(0x7fcbe40634e0 msgr2=0x7fcbe41411b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:00.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 shutdown_connections 2022-07-20T03:15:00.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.649+0000 7fcbea216700 1 -- 172.21.15.61:0/3592661943 wait complete. 2022-07-20T03:15:00.677 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 45 2022-07-20T03:15:00.684 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:15:00.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 Processor -- start 2022-07-20T03:15:00.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 -- start start 2022-07-20T03:15:00.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b5860 0x7fbfe00acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbfe0146770 0x7fbfe014c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe00b46e0 con 0x7fbfe00b4f80 2022-07-20T03:15:00.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00ad990 con 0x7fbfe0146770 2022-07-20T03:15:00.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdf59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe00b4b10 con 0x7fbfe00b5860 2022-07-20T03:15:00.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfddd9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbfe00b4f80 0x7fbfe0146060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44352/0 (socket says 172.21.15.61:44352) 2022-07-20T03:15:00.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfddd9b700 1 -- 172.21.15.61:0/454752035 learned_addr learned my addr 172.21.15.61:0/454752035 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:00.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfded9d700 1 --2- 172.21.15.61:0/454752035 >> v2:172.21.15.61:3300/0 conn(0x7fbfe0146770 0x7fbfe014c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfde59c700 1 --2- 172.21.15.61:0/454752035 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b5860 0x7fbfe00acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdd59a700 1 -- 172.21.15.61:0/454752035 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2048902398 0 0) 0x7fbfe00b46e0 con 0x7fbfe00b4f80 2022-07-20T03:15:00.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdd59a700 1 -- 172.21.15.61:0/454752035 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc0003260 con 0x7fbfe00b4f80 2022-07-20T03:15:00.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfde59c700 1 -- 172.21.15.61:0/454752035 >> v2:172.21.15.61:3300/0 conn(0x7fbfe0146770 msgr2=0x7fbfe014c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:00.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfde59c700 1 --2- 172.21.15.61:0/454752035 >> v2:172.21.15.61:3300/0 conn(0x7fbfe0146770 0x7fbfe014c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfde59c700 1 -- 172.21.15.61:0/454752035 >> v1:172.21.15.61:6789/0 conn(0x7fbfe00b4f80 legacy=0x7fbfe0146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfde59c700 1 -- 172.21.15.61:0/454752035 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe00ad3d0 con 0x7fbfe00b5860 2022-07-20T03:15:00.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfde59c700 1 --2- 172.21.15.61:0/454752035 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b5860 0x7fbfe00acae0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fbfd0002f70 tx=0x7fbfd0006df0).ready entity=mon.2 client_cookie=5bf25e3c1a4997e7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:00.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdd59a700 1 -- 172.21.15.61:0/454752035 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd000d4a0 con 0x7fbfe00b5860 2022-07-20T03:15:00.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdd59a700 1 -- 172.21.15.61:0/454752035 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0000e90 con 0x7fbfe00b5860 2022-07-20T03:15:00.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.789+0000 7fbfdd59a700 1 -- 172.21.15.61:0/454752035 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd000d7f0 con 0x7fbfe00b5860 2022-07-20T03:15:00.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 -- 172.21.15.61:0/454752035 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b5860 msgr2=0x7fbfe00acae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/454752035 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b5860 0x7fbfe00acae0 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7fbfd0002f70 tx=0x7fbfd0006df0).stop 2022-07-20T03:15:00.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 -- 172.21.15.61:0/454752035 shutdown_connections 2022-07-20T03:15:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/454752035 >> v2:172.21.15.61:3300/0 conn(0x7fbfe0146770 0x7fbfe014c670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/454752035 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b5860 0x7fbfe00acae0 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 -- 172.21.15.61:0/454752035 >> 172.21.15.61:0/454752035 conn(0x7fbfe00b5e10 msgr2=0x7fbfe00b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:00.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 -- 172.21.15.61:0/454752035 shutdown_connections 2022-07-20T03:15:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 -- 172.21.15.61:0/454752035 wait complete. 2022-07-20T03:15:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 Processor -- start 2022-07-20T03:15:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.793+0000 7fbfdf59e700 1 -- start start 2022-07-20T03:15:00.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b4f80 0x7fbfe0115400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe00b5860 0x7fbfe00b3140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 0x7fbfe0115a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0151e40 con 0x7fbfe0146770 2022-07-20T03:15:00.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe01520e0 con 0x7fbfe00b5860 2022-07-20T03:15:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0152380 con 0x7fbfe00b4f80 2022-07-20T03:15:00.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 0x7fbfe0115a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 0x7fbfe0115a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47764/0 (socket says 172.21.15.61:47764) 2022-07-20T03:15:00.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 -- 172.21.15.61:0/840503827 learned_addr learned my addr 172.21.15.61:0/840503827 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:00.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfde59c700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b4f80 0x7fbfe0115400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfddd9b700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe00b5860 0x7fbfe00b3140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 -- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b4f80 msgr2=0x7fbfe0115400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b4f80 0x7fbfe0115400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 -- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe00b5860 msgr2=0x7fbfe00b3140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:00.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe00b5860 0x7fbfe00b3140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:00.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 -- 172.21.15.61:0/840503827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfd000b180 con 0x7fbfe0146770 2022-07-20T03:15:00.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfde59c700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b4f80 0x7fbfe0115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:00.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfded9d700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 0x7fbfe0115a50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fbfd800c000 tx=0x7fbfd8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:00.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd8010780 con 0x7fbfe0146770 2022-07-20T03:15:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd80108e0 con 0x7fbfe0146770 2022-07-20T03:15:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe00b3630 con 0x7fbfe0146770 2022-07-20T03:15:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd8010c40 con 0x7fbfe0146770 2022-07-20T03:15:00.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.797+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe0112560 con 0x7fbfe0146770 2022-07-20T03:15:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.801+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe0152540 con 0x7fbfe0146770 2022-07-20T03:15:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.801+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbfd801c460 con 0x7fbfe0146770 2022-07-20T03:15:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.801+0000 7fbfd77fe700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc4053a40 0x7fbfc4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:00.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.801+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(81..81 src has 1..81) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbfd806c3b0 con 0x7fbfe0146770 2022-07-20T03:15:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.801+0000 7fbfde59c700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc4053a40 0x7fbfc4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:00.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.801+0000 7fbfde59c700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc4053a40 0x7fbfc4055f00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbfd0018000 tx=0x7fbfd0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:00.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.805+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfd8043210 con 0x7fbfe0146770 2022-07-20T03:15:00.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:00.965+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbfe00af0f0 con 0x7fbfe0146770 2022-07-20T03:15:01.111 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:15:01.109+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:15:01.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.121+0000 7fbfd77fe700 1 -- 172.21.15.61:0/840503827 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v46) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbfd8046830 con 0x7fbfe0146770 2022-07-20T03:15:01.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc4053a40 msgr2=0x7fbfc4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc4053a40 0x7fbfc4055f00 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fbfd0018000 tx=0x7fbfd0014000).stop 2022-07-20T03:15:01.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 msgr2=0x7fbfe0115a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 0x7fbfe0115a50 secure :-1 s=READY pgs=870 cs=0 l=1 rev1=1 rx=0x7fbfd800c000 tx=0x7fbfd8006df0).stop 2022-07-20T03:15:01.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 shutdown_connections 2022-07-20T03:15:01.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc4053a40 0x7fbfc4055f00 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe00b5860 0x7fbfe00b3140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe0146770 0x7fbfe0115a50 unknown :-1 s=CLOSED pgs=870 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 --2- 172.21.15.61:0/840503827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe00b4f80 0x7fbfe0115400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 >> 172.21.15.61:0/840503827 conn(0x7fbfe00b5e10 msgr2=0x7fbfe00b8bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:01.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 shutdown_connections 2022-07-20T03:15:01.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.125+0000 7fbfdf59e700 1 -- 172.21.15.61:0/840503827 wait complete. 2022-07-20T03:15:01.133 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:15:01.145 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:01.152 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:15:01.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.253+0000 7f5bcda4a700 1 Processor -- start 2022-07-20T03:15:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.253+0000 7f5bcda4a700 1 -- start start 2022-07-20T03:15:01.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5bc805c3f0 0x7f5bc814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc80acef0 0x7f5bc8146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5bc805d480 con 0x7f5bc80b1e00 2022-07-20T03:15:01.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5bc80b21e0 con 0x7f5bc805c3f0 2022-07-20T03:15:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc805cf40 con 0x7f5bc80acef0 2022-07-20T03:15:01.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5bc805c3f0 0x7f5bc814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5bc805c3f0 0x7f5bc814a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47768/0 (socket says 172.21.15.61:47768) 2022-07-20T03:15:01.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5bc80b1e00 0x7f5bc80ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44368/0 (socket says 172.21.15.61:44368) 2022-07-20T03:15:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 -- 172.21.15.61:0/3159974151 learned_addr learned my addr 172.21.15.61:0/3159974151 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:01.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc77fe700 1 --2- 172.21.15.61:0/3159974151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc80acef0 0x7f5bc8146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 -- 172.21.15.61:0/3159974151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc80acef0 msgr2=0x7f5bc8146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:01.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 --2- 172.21.15.61:0/3159974151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc80acef0 0x7f5bc8146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 -- 172.21.15.61:0/3159974151 >> v1:172.21.15.61:6789/0 conn(0x7f5bc80b1e00 legacy=0x7f5bc80ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc5ffb700 1 -- 172.21.15.61:0/3159974151 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956024357 0 0) 0x7f5bc805d480 con 0x7f5bc80b1e00 2022-07-20T03:15:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 -- 172.21.15.61:0/3159974151 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bc805d1e0 con 0x7f5bc805c3f0 2022-07-20T03:15:01.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc67fc700 1 --2- 172.21.15.61:0/3159974151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc805c3f0 0x7f5bc814a6d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f5bb80032f0 tx=0x7f5bb8009290).ready entity=mon.0 client_cookie=1ff66e09d78b4f54 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc5ffb700 1 -- 172.21.15.61:0/3159974151 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb800b8a0 con 0x7f5bc805c3f0 2022-07-20T03:15:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc5ffb700 1 -- 172.21.15.61:0/3159974151 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bb800ba00 con 0x7f5bc805c3f0 2022-07-20T03:15:01.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bc5ffb700 1 -- 172.21.15.61:0/3159974151 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bb800bb70 con 0x7f5bc805c3f0 2022-07-20T03:15:01.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- 172.21.15.61:0/3159974151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc805c3f0 msgr2=0x7f5bc814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/3159974151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc805c3f0 0x7f5bc814a6d0 secure :-1 s=READY pgs=874 cs=0 l=1 rev1=1 rx=0x7f5bb80032f0 tx=0x7f5bb8009290).stop 2022-07-20T03:15:01.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- 172.21.15.61:0/3159974151 shutdown_connections 2022-07-20T03:15:01.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/3159974151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc80acef0 0x7f5bc8146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/3159974151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc805c3f0 0x7f5bc814a6d0 unknown :-1 s=CLOSED pgs=874 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- 172.21.15.61:0/3159974151 >> 172.21.15.61:0/3159974151 conn(0x7f5bc80611c0 msgr2=0x7f5bc8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:01.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- 172.21.15.61:0/3159974151 shutdown_connections 2022-07-20T03:15:01.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.257+0000 7f5bcda4a700 1 -- 172.21.15.61:0/3159974151 wait complete. 2022-07-20T03:15:01.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 Processor -- start 2022-07-20T03:15:01.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 -- start start 2022-07-20T03:15:01.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 0x7f5bc80a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc80b1e00 0x7f5bc81468f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc81443f0 0x7f5bc81448a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80b5260 con 0x7f5bc80acef0 2022-07-20T03:15:01.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80a13e0 con 0x7f5bc80b1e00 2022-07-20T03:15:01.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5bc80b5490 con 0x7f5bc81443f0 2022-07-20T03:15:01.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 0x7f5bc80a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 0x7f5bc80a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47774/0 (socket says 172.21.15.61:47774) 2022-07-20T03:15:01.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 -- 172.21.15.61:0/868971591 learned_addr learned my addr 172.21.15.61:0/868971591 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:01.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc67fc700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc80b1e00 0x7f5bc81468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc77fe700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc81443f0 0x7f5bc81448a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 -- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc81443f0 msgr2=0x7f5bc81448a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc81443f0 0x7f5bc81448a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 -- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc80b1e00 msgr2=0x7f5bc81468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc80b1e00 0x7f5bc81468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 -- 172.21.15.61:0/868971591 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5bb8009530 con 0x7f5bc80acef0 2022-07-20T03:15:01.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc77fe700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc81443f0 0x7f5bc81448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:01.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bc6ffd700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 0x7f5bc80a1840 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f5bbc000ed0 tx=0x7f5bbc007ab0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:01.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bbc007170 con 0x7f5bc80acef0 2022-07-20T03:15:01.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5bbc0072d0 con 0x7f5bc80acef0 2022-07-20T03:15:01.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5bbc004710 con 0x7f5bc80acef0 2022-07-20T03:15:01.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5bc80b5d40 con 0x7f5bc80acef0 2022-07-20T03:15:01.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.261+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5bc80a1e50 con 0x7f5bc80acef0 2022-07-20T03:15:01.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.265+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5bc80a2100 con 0x7f5bc80acef0 2022-07-20T03:15:01.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.265+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5bbc004870 con 0x7f5bc80acef0 2022-07-20T03:15:01.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.265+0000 7f5bb7fff700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ba4053ad0 0x7f5ba4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.265+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f5bbc06b7f0 con 0x7f5bc80acef0 2022-07-20T03:15:01.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.265+0000 7f5bc67fc700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ba4053ad0 0x7f5ba4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.265+0000 7f5bc67fc700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ba4053ad0 0x7f5ba4055f90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f5bb801b000 tx=0x7f5bb8016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:01.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.273+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5bbc042700 con 0x7f5bc80acef0 2022-07-20T03:15:01.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.441+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5bc80a1550 con 0x7f5bc80acef0 2022-07-20T03:15:01.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.441+0000 7f5bb7fff700 1 -- 172.21.15.61:0/868971591 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 46 v46) v1 ==== 94+0+1779 (secure 0 0 0) 0x7f5bbc045d20 con 0x7f5bc80acef0 2022-07-20T03:15:01.476 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:01.476 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":46,"default_fscid":7,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44}],"filesystems":[{"mdsmap":{"epoch":46,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:14:48.072382+0000","modified":"2022-07-20T03:15:01.110477+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":82,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[16],"metadata_pool":15,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":7}]} 2022-07-20T03:15:01.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ba4053ad0 msgr2=0x7f5ba4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ba4053ad0 0x7f5ba4055f90 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f5bb801b000 tx=0x7f5bb8016000).stop 2022-07-20T03:15:01.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 msgr2=0x7f5bc80a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 0x7f5bc80a1840 secure :-1 s=READY pgs=875 cs=0 l=1 rev1=1 rx=0x7f5bbc000ed0 tx=0x7f5bbc007ab0).stop 2022-07-20T03:15:01.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 shutdown_connections 2022-07-20T03:15:01.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5ba4053ad0 0x7f5ba4055f90 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5bc80b1e00 0x7f5bc81468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5bc81443f0 0x7f5bc81448a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 --2- 172.21.15.61:0/868971591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5bc80acef0 0x7f5bc80a1840 unknown :-1 s=CLOSED pgs=875 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 >> 172.21.15.61:0/868971591 conn(0x7f5bc805c3f0 msgr2=0x7f5bc81454d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:01.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 shutdown_connections 2022-07-20T03:15:01.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.445+0000 7f5bcda4a700 1 -- 172.21.15.61:0/868971591 wait complete. 2022-07-20T03:15:01.481 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 46 2022-07-20T03:15:01.510 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:15:01.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 Processor -- start 2022-07-20T03:15:01.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 -- start start 2022-07-20T03:15:01.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cf00b9eb0 0x7f9cf005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf005c990 0x7f9cf0146430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9cf00b0c20 con 0x7f9cf00b5320 2022-07-20T03:15:01.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9cf00ba290 con 0x7f9cf00b9eb0 2022-07-20T03:15:01.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf6baa700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0142e00 con 0x7f9cf005c990 2022-07-20T03:15:01.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf4946700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cf00b9eb0 0x7f9cf005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9ceffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9cf00b5320 0x7f9cf005c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44380/0 (socket says 172.21.15.61:44380) 2022-07-20T03:15:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9cf4946700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9cf00b9eb0 0x7f9cf005bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47782/0 (socket says 172.21.15.61:47782) 2022-07-20T03:15:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.637+0000 7f9ceffff700 1 -- 172.21.15.61:0/647194236 learned_addr learned my addr 172.21.15.61:0/647194236 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:01.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf5147700 1 --2- 172.21.15.61:0/647194236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf005c990 0x7f9cf0146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf4946700 1 -- 172.21.15.61:0/647194236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf005c990 msgr2=0x7f9cf0146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:01.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf4946700 1 --2- 172.21.15.61:0/647194236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf005c990 0x7f9cf0146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf4946700 1 -- 172.21.15.61:0/647194236 >> v1:172.21.15.61:6789/0 conn(0x7f9cf00b5320 legacy=0x7f9cf005c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cef7fe700 1 -- 172.21.15.61:0/647194236 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2709530598 0 0) 0x7f9cf00b0c20 con 0x7f9cf00b5320 2022-07-20T03:15:01.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf4946700 1 -- 172.21.15.61:0/647194236 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf00afed0 con 0x7f9cf00b9eb0 2022-07-20T03:15:01.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf4946700 1 --2- 172.21.15.61:0/647194236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf00b9eb0 0x7f9cf005bd90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7f9ce0002f70 tx=0x7f9ce0006df0).ready entity=mon.0 client_cookie=abbf93a92df2e736 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:01.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cef7fe700 1 -- 172.21.15.61:0/647194236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ce000d570 con 0x7f9cf00b9eb0 2022-07-20T03:15:01.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cef7fe700 1 -- 172.21.15.61:0/647194236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce0000e90 con 0x7f9cf00b9eb0 2022-07-20T03:15:01.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cef7fe700 1 -- 172.21.15.61:0/647194236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ce000d8c0 con 0x7f9cf00b9eb0 2022-07-20T03:15:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 -- 172.21.15.61:0/647194236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf00b9eb0 msgr2=0x7f9cf005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/647194236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf00b9eb0 0x7f9cf005bd90 secure :-1 s=READY pgs=877 cs=0 l=1 rev1=1 rx=0x7f9ce0002f70 tx=0x7f9ce0006df0).stop 2022-07-20T03:15:01.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 -- 172.21.15.61:0/647194236 shutdown_connections 2022-07-20T03:15:01.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/647194236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf005c990 0x7f9cf0146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/647194236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf00b9eb0 0x7f9cf005bd90 unknown :-1 s=CLOSED pgs=877 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 -- 172.21.15.61:0/647194236 >> 172.21.15.61:0/647194236 conn(0x7f9cf0067980 msgr2=0x7f9cf00b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:01.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 -- 172.21.15.61:0/647194236 shutdown_connections 2022-07-20T03:15:01.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 -- 172.21.15.61:0/647194236 wait complete. 2022-07-20T03:15:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.641+0000 7f9cf6baa700 1 Processor -- start 2022-07-20T03:15:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 -- start start 2022-07-20T03:15:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 0x7f9cf013f260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cf00b5320 0x7f9cf0151e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf0147de0 0x7f9cf0152330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0145c10 con 0x7f9cf005c990 2022-07-20T03:15:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf0145eb0 con 0x7f9cf00b5320 2022-07-20T03:15:01.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9cf01486b0 con 0x7f9cf0147de0 2022-07-20T03:15:01.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 0x7f9cf013f260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 0x7f9cf013f260 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47788/0 (socket says 172.21.15.61:47788) 2022-07-20T03:15:01.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 -- 172.21.15.61:0/843272129 learned_addr learned my addr 172.21.15.61:0/843272129 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:01.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf5147700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf0147de0 0x7f9cf0152330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9ceffff700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cf00b5320 0x7f9cf0151e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 -- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf0147de0 msgr2=0x7f9cf0152330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf0147de0 0x7f9cf0152330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 -- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cf00b5320 msgr2=0x7f9cf0151e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:01.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cf00b5320 0x7f9cf0151e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:01.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 -- 172.21.15.61:0/843272129 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ce000b180 con 0x7f9cf005c990 2022-07-20T03:15:01.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9ceffff700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cf00b5320 0x7f9cf0151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:01.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf4946700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 0x7f9cf013f260 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f9ce0002f70 tx=0x7f9ce0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:01.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ce0016070 con 0x7f9cf005c990 2022-07-20T03:15:01.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cf0152820 con 0x7f9cf005c990 2022-07-20T03:15:01.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9ce0013970 con 0x7f9cf005c990 2022-07-20T03:15:01.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cf0148cf0 con 0x7f9cf005c990 2022-07-20T03:15:01.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.645+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9ce0013cd0 con 0x7f9cf005c990 2022-07-20T03:15:01.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.649+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cf00af970 con 0x7f9cf005c990 2022-07-20T03:15:01.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.649+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9ce000f030 con 0x7f9cf005c990 2022-07-20T03:15:01.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.649+0000 7f9cedffb700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9cd4053a40 0x7f9cd4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:01.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.649+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9ce006cdb0 con 0x7f9cf005c990 2022-07-20T03:15:01.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.649+0000 7f9ceffff700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9cd4053a40 0x7f9cd4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:01.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.649+0000 7f9ceffff700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9cd4053a40 0x7f9cd4055f00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f9cd800b000 tx=0x7f9cd8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:01.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.657+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9cf00af970 con 0x7f9cf005c990 2022-07-20T03:15:01.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:01.821+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f9cf00af970 con 0x7f9cf005c990 2022-07-20T03:15:02.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.125+0000 7f9cedffb700 1 -- 172.21.15.61:0/843272129 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v47) v1 ==== 104+0+0 (secure 0 0 0) 0x7f9cf00af970 con 0x7f9cf005c990 2022-07-20T03:15:02.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9cd4053a40 msgr2=0x7f9cd4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9cd4053a40 0x7f9cd4055f00 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f9cd800b000 tx=0x7f9cd8009040).stop 2022-07-20T03:15:02.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 msgr2=0x7f9cf013f260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 0x7f9cf013f260 secure :-1 s=READY pgs=878 cs=0 l=1 rev1=1 rx=0x7f9ce0002f70 tx=0x7f9ce0014000).stop 2022-07-20T03:15:02.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 shutdown_connections 2022-07-20T03:15:02.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9cd4053a40 0x7f9cd4055f00 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9cf00b5320 0x7f9cf0151e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9cf0147de0 0x7f9cf0152330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 --2- 172.21.15.61:0/843272129 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9cf005c990 0x7f9cf013f260 unknown :-1 s=CLOSED pgs=878 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 >> 172.21.15.61:0/843272129 conn(0x7f9cf0067980 msgr2=0x7f9cf00b2fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:02.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 shutdown_connections 2022-07-20T03:15:02.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.129+0000 7f9cf6baa700 1 -- 172.21.15.61:0/843272129 wait complete. 2022-07-20T03:15:02.150 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:15:02.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.261+0000 7feb26020700 1 Processor -- start 2022-07-20T03:15:02.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.261+0000 7feb26020700 1 -- start start 2022-07-20T03:15:02.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb20067370 0x7feb20067750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:02.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb2013ad00 con 0x7feb200acbc0 2022-07-20T03:15:02.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb200b3c60 con 0x7feb200b2130 2022-07-20T03:15:02.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb200b3f00 con 0x7feb20067370 2022-07-20T03:15:02.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2501e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb20067370 0x7feb20067750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:02.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2481d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feb200acbc0 0x7feb2013a5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44396/0 (socket says 172.21.15.61:44396) 2022-07-20T03:15:02.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2501e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb20067370 0x7feb20067750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47796/0 (socket says 172.21.15.61:47796) 2022-07-20T03:15:02.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2481d700 1 -- 172.21.15.61:0/2868278138 learned_addr learned my addr 172.21.15.61:0/2868278138 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:02.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb0ffff700 1 -- 172.21.15.61:0/2868278138 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 324346524 0 0) 0x7feb200b3c60 con 0x7feb200b2130 2022-07-20T03:15:02.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb0ffff700 1 -- 172.21.15.61:0/2868278138 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feb08003260 con 0x7feb200b2130 2022-07-20T03:15:02.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2501e700 1 -- 172.21.15.61:0/2868278138 >> v1:172.21.15.117:6789/0 conn(0x7feb200b2130 legacy=0x7feb200b2a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2501e700 1 -- 172.21.15.61:0/2868278138 >> v1:172.21.15.61:6789/0 conn(0x7feb200acbc0 legacy=0x7feb2013a5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2501e700 1 -- 172.21.15.61:0/2868278138 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200b41a0 con 0x7feb20067370 2022-07-20T03:15:02.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb2501e700 1 --2- 172.21.15.61:0/2868278138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb20067370 0x7feb20067750 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7feb18001490 tx=0x7feb1c006df0).ready entity=mon.0 client_cookie=5a7e0c60a8c69a76 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:02.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb0ffff700 1 -- 172.21.15.61:0/2868278138 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb1c00d5c0 con 0x7feb20067370 2022-07-20T03:15:02.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb0ffff700 1 -- 172.21.15.61:0/2868278138 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb1c000e90 con 0x7feb20067370 2022-07-20T03:15:02.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb0ffff700 1 -- 172.21.15.61:0/2868278138 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb1c00d910 con 0x7feb20067370 2022-07-20T03:15:02.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- 172.21.15.61:0/2868278138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb20067370 msgr2=0x7feb20067750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 --2- 172.21.15.61:0/2868278138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb20067370 0x7feb20067750 secure :-1 s=READY pgs=880 cs=0 l=1 rev1=1 rx=0x7feb18001490 tx=0x7feb1c006df0).stop 2022-07-20T03:15:02.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- 172.21.15.61:0/2868278138 shutdown_connections 2022-07-20T03:15:02.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 --2- 172.21.15.61:0/2868278138 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb20067370 0x7feb20067750 unknown :-1 s=CLOSED pgs=880 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- 172.21.15.61:0/2868278138 >> 172.21.15.61:0/2868278138 conn(0x7feb20064c00 msgr2=0x7feb20064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:02.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- 172.21.15.61:0/2868278138 shutdown_connections 2022-07-20T03:15:02.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- 172.21.15.61:0/2868278138 wait complete. 2022-07-20T03:15:02.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 Processor -- start 2022-07-20T03:15:02.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.265+0000 7feb26020700 1 -- start start 2022-07-20T03:15:02.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb20067370 0x7feb20142000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:02.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 0x7feb2013e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:02.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb200b2130 0x7feb2013efd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:02.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2013f4c0 con 0x7feb200acbc0 2022-07-20T03:15:02.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2013b020 con 0x7feb20067370 2022-07-20T03:15:02.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb200b1150 con 0x7feb200b2130 2022-07-20T03:15:02.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 0x7feb2013e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 0x7feb2013e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47804/0 (socket says 172.21.15.61:47804) 2022-07-20T03:15:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2501e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb20067370 0x7feb20142000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 -- 172.21.15.61:0/2736710524 learned_addr learned my addr 172.21.15.61:0/2736710524 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:02.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2581f700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb200b2130 0x7feb2013efd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:02.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 -- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb200b2130 msgr2=0x7feb2013efd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb200b2130 0x7feb2013efd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 -- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb20067370 msgr2=0x7feb20142000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:02.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb20067370 0x7feb20142000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:02.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 -- 172.21.15.61:0/2736710524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1c00b180 con 0x7feb200acbc0 2022-07-20T03:15:02.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2581f700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb200b2130 0x7feb2013efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:02.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb2481d700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 0x7feb2013e9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7feb20065b70 tx=0x7feb10007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:02.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb100070d0 con 0x7feb200acbc0 2022-07-20T03:15:02.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb10007230 con 0x7feb200acbc0 2022-07-20T03:15:02.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2014fdf0 con 0x7feb200acbc0 2022-07-20T03:15:02.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb10005610 con 0x7feb200acbc0 2022-07-20T03:15:02.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.269+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb200b0e60 con 0x7feb200acbc0 2022-07-20T03:15:02.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.273+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feb10010070 con 0x7feb200acbc0 2022-07-20T03:15:02.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.273+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb2014ff80 con 0x7feb200acbc0 2022-07-20T03:15:02.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.273+0000 7feb0e7fc700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb00053a40 0x7feb00055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:02.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.273+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(82..82 src has 1..82) v4 ==== 6159+0+0 (secure 0 0 0) 0x7feb1006b510 con 0x7feb200acbc0 2022-07-20T03:15:02.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.273+0000 7feb2501e700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb00053a40 0x7feb00055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:02.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.273+0000 7feb2501e700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb00053a40 0x7feb00055f00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7feb1c01a000 tx=0x7feb1c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:02.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.281+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feb10042310 con 0x7feb200acbc0 2022-07-20T03:15:02.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:02.413+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7feb200b3920 con 0x7feb200acbc0 2022-07-20T03:15:03.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.133+0000 7feb0e7fc700 1 -- 172.21.15.61:0/2736710524 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v83) v1 ==== 185+0+0 (secure 0 0 0) 0x7feb10018070 con 0x7feb200acbc0 2022-07-20T03:15:03.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.137+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb00053a40 msgr2=0x7feb00055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:03.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.137+0000 7feb26020700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb00053a40 0x7feb00055f00 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7feb1c01a000 tx=0x7feb1c014000).stop 2022-07-20T03:15:03.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.137+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 msgr2=0x7feb2013e9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:03.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 0x7feb2013e9c0 secure :-1 s=READY pgs=881 cs=0 l=1 rev1=1 rx=0x7feb20065b70 tx=0x7feb10007c00).stop 2022-07-20T03:15:03.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 shutdown_connections 2022-07-20T03:15:03.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb00053a40 0x7feb00055f00 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb200b2130 0x7feb2013efd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb200acbc0 0x7feb2013e9c0 unknown :-1 s=CLOSED pgs=881 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 --2- 172.21.15.61:0/2736710524 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb20067370 0x7feb20142000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 >> 172.21.15.61:0/2736710524 conn(0x7feb20064c00 msgr2=0x7feb20064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:03.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 shutdown_connections 2022-07-20T03:15:03.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.141+0000 7feb26020700 1 -- 172.21.15.61:0/2736710524 wait complete. 2022-07-20T03:15:03.263 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:15:03.264 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:15:03.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.385+0000 7f5e23054700 1 Processor -- start 2022-07-20T03:15:03.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.385+0000 7f5e23054700 1 -- start start 2022-07-20T03:15:03.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e23054700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e1c0b6060 0x7f5e1c144710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:03.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e23054700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5e1c144c00 0x7f5e1c0b01d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:03.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e23054700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5e1c0b4440 con 0x7f5e1c0b51e0 2022-07-20T03:15:03.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e23054700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c143fd0 con 0x7f5e1c0b6060 2022-07-20T03:15:03.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e23054700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5e1c144270 con 0x7f5e1c144c00 2022-07-20T03:15:03.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e1c0b6060 0x7f5e1c144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:03.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5e1c0b6060 0x7f5e1c144710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47810/0 (socket says 172.21.15.61:47810) 2022-07-20T03:15:03.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 -- 172.21.15.61:0/1002731749 learned_addr learned my addr 172.21.15.61:0/1002731749 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22853700 1 --1- 172.21.15.61:0/1002731749 >> v1:172.21.15.61:6789/0 conn(0x7f5e1c0b51e0 0x7f5e1c0b06c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44410/0 (socket says 172.21.15.61:44410) 2022-07-20T03:15:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e21851700 1 --2- 172.21.15.61:0/1002731749 >> v2:172.21.15.117:3300/0 conn(0x7f5e1c144c00 0x7f5e1c0b01d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e21050700 1 -- 172.21.15.61:0/1002731749 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1665299250 0 0) 0x7f5e1c0b4440 con 0x7f5e1c0b51e0 2022-07-20T03:15:03.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e21050700 1 -- 172.21.15.61:0/1002731749 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5e04003260 con 0x7f5e1c0b51e0 2022-07-20T03:15:03.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 -- 172.21.15.61:0/1002731749 >> v2:172.21.15.117:3300/0 conn(0x7f5e1c144c00 msgr2=0x7f5e1c0b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:03.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 --2- 172.21.15.61:0/1002731749 >> v2:172.21.15.117:3300/0 conn(0x7f5e1c144c00 0x7f5e1c0b01d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 -- 172.21.15.61:0/1002731749 >> v1:172.21.15.61:6789/0 conn(0x7f5e1c0b51e0 legacy=0x7f5e1c0b06c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:03.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 -- 172.21.15.61:0/1002731749 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1c0b40c0 con 0x7f5e1c0b6060 2022-07-20T03:15:03.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e22052700 1 --2- 172.21.15.61:0/1002731749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b6060 0x7f5e1c144710 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f5e18002f70 tx=0x7f5e18006eb0).ready entity=mon.0 client_cookie=7bca08eb2af8ddcc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:03.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e21050700 1 -- 172.21.15.61:0/1002731749 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1800d5c0 con 0x7f5e1c0b6060 2022-07-20T03:15:03.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e21050700 1 -- 172.21.15.61:0/1002731749 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e18000e90 con 0x7f5e1c0b6060 2022-07-20T03:15:03.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.389+0000 7f5e21050700 1 -- 172.21.15.61:0/1002731749 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e1800d950 con 0x7f5e1c0b6060 2022-07-20T03:15:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 -- 172.21.15.61:0/1002731749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b6060 msgr2=0x7f5e1c144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 --2- 172.21.15.61:0/1002731749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b6060 0x7f5e1c144710 secure :-1 s=READY pgs=883 cs=0 l=1 rev1=1 rx=0x7f5e18002f70 tx=0x7f5e18006eb0).stop 2022-07-20T03:15:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 -- 172.21.15.61:0/1002731749 shutdown_connections 2022-07-20T03:15:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 --2- 172.21.15.61:0/1002731749 >> v2:172.21.15.117:3300/0 conn(0x7f5e1c144c00 0x7f5e1c0b01d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 --2- 172.21.15.61:0/1002731749 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b6060 0x7f5e1c144710 unknown :-1 s=CLOSED pgs=883 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 -- 172.21.15.61:0/1002731749 >> 172.21.15.61:0/1002731749 conn(0x7f5e1c063d10 msgr2=0x7f5e1c0b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 -- 172.21.15.61:0/1002731749 shutdown_connections 2022-07-20T03:15:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 -- 172.21.15.61:0/1002731749 wait complete. 2022-07-20T03:15:03.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 Processor -- start 2022-07-20T03:15:03.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.393+0000 7f5e23054700 1 -- start start 2022-07-20T03:15:03.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 0x7f5e1c1469a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:03.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e1c0b6060 0x7f5e1c14fa90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e1c144c00 0x7f5e1c1535b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c153150 con 0x7f5e1c0b51e0 2022-07-20T03:15:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c065a10 con 0x7f5e1c144c00 2022-07-20T03:15:03.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5e1c065770 con 0x7f5e1c0b6060 2022-07-20T03:15:03.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22052700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 0x7f5e1c1469a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:03.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22052700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 0x7f5e1c1469a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47816/0 (socket says 172.21.15.61:47816) 2022-07-20T03:15:03.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e1c144c00 0x7f5e1c1535b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22052700 1 -- 172.21.15.61:0/1253269856 learned_addr learned my addr 172.21.15.61:0/1253269856 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e21851700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e1c0b6060 0x7f5e1c14fa90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 -- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e1c0b6060 msgr2=0x7f5e1c14fa90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:03.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e1c0b6060 0x7f5e1c14fa90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 -- 172.21.15.61:0/1253269856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 msgr2=0x7f5e1c1469a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 0x7f5e1c1469a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 -- 172.21.15.61:0/1253269856 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e1800b1e0 con 0x7f5e1c144c00 2022-07-20T03:15:03.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22052700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 0x7f5e1c1469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:03.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e22853700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e1c144c00 0x7f5e1c1535b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f5e14001490 tx=0x7f5e14007d20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:03.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e14005ae0 con 0x7f5e1c144c00 2022-07-20T03:15:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5e14005c40 con 0x7f5e1c144c00 2022-07-20T03:15:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5e1c14ff80 con 0x7f5e1c144c00 2022-07-20T03:15:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5e140070d0 con 0x7f5e1c144c00 2022-07-20T03:15:03.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.397+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5e1c066840 con 0x7f5e1c144c00 2022-07-20T03:15:03.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.401+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5e1c065590 con 0x7f5e1c144c00 2022-07-20T03:15:03.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.401+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5e14010070 con 0x7f5e1c144c00 2022-07-20T03:15:03.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.401+0000 7f5e0b7fe700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5dfc053a60 0x7f5dfc055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:03.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.401+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(83..83 src has 1..83) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f5e1406b720 con 0x7f5e1c144c00 2022-07-20T03:15:03.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.401+0000 7f5e22052700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5dfc053a60 0x7f5dfc055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:03.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.401+0000 7f5e22052700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5dfc053a60 0x7f5dfc055f20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f5e1801c000 tx=0x7f5e18015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:03.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.409+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5e14042650 con 0x7f5e1c144c00 2022-07-20T03:15:03.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:03.541+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f5e1c069a20 con 0x7f5e1c144c00 2022-07-20T03:15:04.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.149+0000 7f5e0b7fe700 1 -- 172.21.15.61:0/1253269856 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v84) v1 ==== 173+0+0 (secure 0 0 0) 0x7f5e14045c70 con 0x7f5e1c144c00 2022-07-20T03:15:04.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5dfc053a60 msgr2=0x7f5dfc055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5dfc053a60 0x7f5dfc055f20 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f5e1801c000 tx=0x7f5e18015000).stop 2022-07-20T03:15:04.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e1c144c00 msgr2=0x7f5e1c1535b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e1c144c00 0x7f5e1c1535b0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f5e14001490 tx=0x7f5e14007d20).stop 2022-07-20T03:15:04.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 shutdown_connections 2022-07-20T03:15:04.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5dfc053a60 0x7f5dfc055f20 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5e1c144c00 0x7f5e1c1535b0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5e1c0b6060 0x7f5e1c14fa90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 --2- 172.21.15.61:0/1253269856 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5e1c0b51e0 0x7f5e1c1469a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 >> 172.21.15.61:0/1253269856 conn(0x7f5e1c063d10 msgr2=0x7f5e1c0b3420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:04.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 shutdown_connections 2022-07-20T03:15:04.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.153+0000 7f5e23054700 1 -- 172.21.15.61:0/1253269856 wait complete. 2022-07-20T03:15:04.160 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:15:04.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:15:04.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.281+0000 7f52e463f700 1 Processor -- start 2022-07-20T03:15:04.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.281+0000 7f52e463f700 1 -- start start 2022-07-20T03:15:04.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f52dc0b00f0 0x7f52dc0b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc142dc0 con 0x7f52dc0b3df0 2022-07-20T03:15:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f52dc0ba280 con 0x7f52dc0b9ea0 2022-07-20T03:15:04.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f52dc146280 con 0x7f52dc0b00f0 2022-07-20T03:15:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f52dc0b00f0 0x7f52dc0b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f52dc0b00f0 0x7f52dc0b4300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47824/0 (socket says 172.21.15.61:47824) 2022-07-20T03:15:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 -- 172.21.15.61:0/492105025 learned_addr learned my addr 172.21.15.61:0/492105025 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:04.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e2bdc700 1 --1- 172.21.15.61:0/492105025 >> v1:172.21.15.61:6789/0 conn(0x7f52dc0b3df0 0x7f52dc141c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44426/0 (socket says 172.21.15.61:44426) 2022-07-20T03:15:04.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 -- 172.21.15.61:0/492105025 >> v1:172.21.15.117:6789/0 conn(0x7f52dc0b9ea0 legacy=0x7f52dc143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 -- 172.21.15.61:0/492105025 >> v1:172.21.15.61:6789/0 conn(0x7f52dc0b3df0 legacy=0x7f52dc141c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 -- 172.21.15.61:0/492105025 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52dc142ff0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e1bda700 1 --2- 172.21.15.61:0/492105025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0b4300 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f52cc002f70 tx=0x7f52cc006eb0).ready entity=mon.0 client_cookie=9ea9e22acbd9cfb9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:04.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e13d9700 1 -- 172.21.15.61:0/492105025 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52cc00d660 con 0x7f52dc0b00f0 2022-07-20T03:15:04.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e13d9700 1 -- 172.21.15.61:0/492105025 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52cc000e90 con 0x7f52dc0b00f0 2022-07-20T03:15:04.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e13d9700 1 -- 172.21.15.61:0/492105025 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52cc00d9f0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- 172.21.15.61:0/492105025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 msgr2=0x7f52dc0b4300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 --2- 172.21.15.61:0/492105025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0b4300 secure :-1 s=READY pgs=885 cs=0 l=1 rev1=1 rx=0x7f52cc002f70 tx=0x7f52cc006eb0).stop 2022-07-20T03:15:04.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- 172.21.15.61:0/492105025 shutdown_connections 2022-07-20T03:15:04.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 --2- 172.21.15.61:0/492105025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0b4300 unknown :-1 s=CLOSED pgs=885 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- 172.21.15.61:0/492105025 >> 172.21.15.61:0/492105025 conn(0x7f52dc067960 msgr2=0x7f52dc0b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:04.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- 172.21.15.61:0/492105025 shutdown_connections 2022-07-20T03:15:04.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.285+0000 7f52e463f700 1 -- 172.21.15.61:0/492105025 wait complete. 2022-07-20T03:15:04.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 Processor -- start 2022-07-20T03:15:04.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 -- start start 2022-07-20T03:15:04.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0adc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f52dc0b3df0 0x7f52dc0ae120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52dc0b9ea0 0x7f52dc115280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc1140d0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc0af800 con 0x7f52dc0b9ea0 2022-07-20T03:15:04.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f52dc152fd0 con 0x7f52dc0b3df0 2022-07-20T03:15:04.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0adc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0adc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47830/0 (socket says 172.21.15.61:47830) 2022-07-20T03:15:04.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 -- 172.21.15.61:0/838475051 learned_addr learned my addr 172.21.15.61:0/838475051 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:04.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e2bdc700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52dc0b9ea0 0x7f52dc115280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e1bda700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f52dc0b3df0 0x7f52dc0ae120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 -- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f52dc0b3df0 msgr2=0x7f52dc0ae120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f52dc0b3df0 0x7f52dc0ae120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 -- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52dc0b9ea0 msgr2=0x7f52dc115280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52dc0b9ea0 0x7f52dc115280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 -- 172.21.15.61:0/838475051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52cc00b1e0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e1bda700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f52dc0b3df0 0x7f52dc0ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:04.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e2bdc700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52dc0b9ea0 0x7f52dc115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:04.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e23db700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0adc30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f52dc1522e0 tx=0x7f52d8007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:04.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52d8012400 con 0x7f52dc0b00f0 2022-07-20T03:15:04.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f52d8012560 con 0x7f52dc0b00f0 2022-07-20T03:15:04.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52dc0af360 con 0x7f52dc0b00f0 2022-07-20T03:15:04.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f52d80128c0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.289+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52dc0aede0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.293+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52dc153190 con 0x7f52dc0b00f0 2022-07-20T03:15:04.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.293+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f52d8019030 con 0x7f52dc0b00f0 2022-07-20T03:15:04.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.293+0000 7f52cb7fe700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f52bc0539d0 0x7f52bc055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.293+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f52d806af50 con 0x7f52dc0b00f0 2022-07-20T03:15:04.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.293+0000 7f52e1bda700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f52bc0539d0 0x7f52bc055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.297+0000 7f52e1bda700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f52bc0539d0 0x7f52bc055e90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f52cc01c000 tx=0x7f52cc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:04.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.301+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f52d8042080 con 0x7f52dc0b00f0 2022-07-20T03:15:04.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.429+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f52dc0574e0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52cb7fe700 1 -- 172.21.15.61:0/838475051 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v13) v1 ==== 104+0+0 (secure 0 0 0) 0x7f52d80456a0 con 0x7f52dc0b00f0 2022-07-20T03:15:04.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f52bc0539d0 msgr2=0x7f52bc055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f52bc0539d0 0x7f52bc055e90 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f52cc01c000 tx=0x7f52cc015000).stop 2022-07-20T03:15:04.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 msgr2=0x7f52dc0adc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0adc30 secure :-1 s=READY pgs=886 cs=0 l=1 rev1=1 rx=0x7f52dc1522e0 tx=0x7f52d8007c50).stop 2022-07-20T03:15:04.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 shutdown_connections 2022-07-20T03:15:04.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f52bc0539d0 0x7f52bc055e90 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f52dc0b9ea0 0x7f52dc115280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f52dc0b3df0 0x7f52dc0ae120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 --2- 172.21.15.61:0/838475051 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f52dc0b00f0 0x7f52dc0adc30 unknown :-1 s=CLOSED pgs=886 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 >> 172.21.15.61:0/838475051 conn(0x7f52dc067960 msgr2=0x7f52dc0b1d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:04.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 shutdown_connections 2022-07-20T03:15:04.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.441+0000 7f52e463f700 1 -- 172.21.15.61:0/838475051 wait complete. 2022-07-20T03:15:04.459 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode' 2022-07-20T03:15:04.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 Processor -- start 2022-07-20T03:15:04.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 -- start start 2022-07-20T03:15:04.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b413a590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd4b4143f00 0x7fd4b413aa80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4b4065f20 con 0x7fd4b40b48d0 2022-07-20T03:15:04.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4b40b29c0 con 0x7fd4b4143f00 2022-07-20T03:15:04.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b8845700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b40661c0 con 0x7fd4b405cda0 2022-07-20T03:15:04.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b2d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b413a590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4b359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd4b4143f00 0x7fd4b413aa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4abfff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fd4b40b48d0 0x7fd4b40b38d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:52940/0 (socket says 172.21.15.61:52940) 2022-07-20T03:15:04.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.557+0000 7fd4abfff700 1 -- 172.21.15.61:0/4123474823 learned_addr learned my addr 172.21.15.61:0/4123474823 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:04.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b359e700 1 -- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 msgr2=0x7fd4b413a590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:04.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b359e700 1 --2- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b413a590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b359e700 1 -- 172.21.15.61:0/4123474823 >> v1:172.21.15.117:6789/0 conn(0x7fd4b40b48d0 legacy=0x7fd4b40b38d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b259c700 1 -- 172.21.15.61:0/4123474823 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3028301917 0 0) 0x7fd4b4065f20 con 0x7fd4b40b48d0 2022-07-20T03:15:04.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b359e700 1 -- 172.21.15.61:0/4123474823 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4b4066460 con 0x7fd4b4143f00 2022-07-20T03:15:04.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b359e700 1 --2- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b413aa80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fd4ac0032f0 tx=0x7fd4ac0091d0).ready entity=mon.1 client_cookie=e0e135763f788eea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:04.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b259c700 1 -- 172.21.15.61:0/4123474823 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac00b6c0 con 0x7fd4b4143f00 2022-07-20T03:15:04.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b259c700 1 -- 172.21.15.61:0/4123474823 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac00b820 con 0x7fd4b4143f00 2022-07-20T03:15:04.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b259c700 1 -- 172.21.15.61:0/4123474823 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac00b990 con 0x7fd4b4143f00 2022-07-20T03:15:04.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 -- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 msgr2=0x7fd4b413aa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 --2- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b413aa80 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fd4ac0032f0 tx=0x7fd4ac0091d0).stop 2022-07-20T03:15:04.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 -- 172.21.15.61:0/4123474823 shutdown_connections 2022-07-20T03:15:04.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 --2- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b413aa80 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 --2- 172.21.15.61:0/4123474823 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b413a590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 -- 172.21.15.61:0/4123474823 >> 172.21.15.61:0/4123474823 conn(0x7fd4b4064b90 msgr2=0x7fd4b4064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:04.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 -- 172.21.15.61:0/4123474823 shutdown_connections 2022-07-20T03:15:04.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.561+0000 7fd4b8845700 1 -- 172.21.15.61:0/4123474823 wait complete. 2022-07-20T03:15:04.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 Processor -- start 2022-07-20T03:15:04.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 -- start start 2022-07-20T03:15:04.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b40b5760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b40ae990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 0x7fd4b40aee80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b40afb40 con 0x7fd4b40ae540 2022-07-20T03:15:04.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b40b1370 con 0x7fd4b4143f00 2022-07-20T03:15:04.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b4154330 con 0x7fd4b405cda0 2022-07-20T03:15:04.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 0x7fd4b40aee80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 0x7fd4b40aee80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47846/0 (socket says 172.21.15.61:47846) 2022-07-20T03:15:04.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 -- 172.21.15.61:0/2375841599 learned_addr learned my addr 172.21.15.61:0/2375841599 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:04.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4abfff700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b40ae990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b2d9d700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b40b5760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 -- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 msgr2=0x7fd4b40b5760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b40b5760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 -- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 msgr2=0x7fd4b40ae990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:04.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b40ae990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:04.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 -- 172.21.15.61:0/2375841599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac009470 con 0x7fd4b40ae540 2022-07-20T03:15:04.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b359e700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 0x7fd4b40aee80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fd4b40af950 tx=0x7fd4ac016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:04.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac00b990 con 0x7fd4b40ae540 2022-07-20T03:15:04.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4ac021630 con 0x7fd4b40ae540 2022-07-20T03:15:04.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b4140480 con 0x7fd4b40ae540 2022-07-20T03:15:04.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4ac00b6c0 con 0x7fd4b40ae540 2022-07-20T03:15:04.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.565+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b40af370 con 0x7fd4b40ae540 2022-07-20T03:15:04.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.569+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd4ac027020 con 0x7fd4b40ae540 2022-07-20T03:15:04.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.569+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b413f170 con 0x7fd4b40ae540 2022-07-20T03:15:04.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.569+0000 7fd4b0d99700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd498053ad0 0x7fd498055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:04.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.569+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd4ac06cff0 con 0x7fd4b40ae540 2022-07-20T03:15:04.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.569+0000 7fd4b2d9d700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd498053ad0 0x7fd498055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:04.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.569+0000 7fd4b2d9d700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd498053ad0 0x7fd498055f90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd4a400b000 tx=0x7fd4a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:04.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.577+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4ac0441c0 con 0x7fd4b40ae540 2022-07-20T03:15:04.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:04.741+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]} v 0) v1 -- 0x7fd4b4110490 con 0x7fd4b40ae540 2022-07-20T03:15:05.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.149+0000 7fd4b0d99700 1 -- 172.21.15.61:0/2375841599 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode"]}]=0 v355) v1 ==== 126+0+0 (secure 0 0 0) 0x7fd4ac015020 con 0x7fd4b40ae540 2022-07-20T03:15:05.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd498053ad0 msgr2=0x7fd498055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:05.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd498053ad0 0x7fd498055f90 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fd4a400b000 tx=0x7fd4a4006df0).stop 2022-07-20T03:15:05.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 msgr2=0x7fd4b40aee80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:05.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 0x7fd4b40aee80 secure :-1 s=READY pgs=887 cs=0 l=1 rev1=1 rx=0x7fd4b40af950 tx=0x7fd4ac016000).stop 2022-07-20T03:15:05.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 shutdown_connections 2022-07-20T03:15:05.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd498053ad0 0x7fd498055f90 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4b4143f00 0x7fd4b40ae990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4b40ae540 0x7fd4b40aee80 unknown :-1 s=CLOSED pgs=887 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 --2- 172.21.15.61:0/2375841599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4b405cda0 0x7fd4b40b5760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 >> 172.21.15.61:0/2375841599 conn(0x7fd4b4064b90 msgr2=0x7fd4b40b61a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:05.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 shutdown_connections 2022-07-20T03:15:05.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.153+0000 7fd4b8845700 1 -- 172.21.15.61:0/2375841599 wait complete. 2022-07-20T03:15:05.173 INFO:tasks.cephfs_test_runner:test_dump_inode (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:15:05.178 INFO:tasks.cephfs_test_runner:Starting test: test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:15:05.179 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-20T03:15:05.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 Processor -- start 2022-07-20T03:15:05.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 -- start start 2022-07-20T03:15:05.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd65805d0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:05.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd658143ea0 0x7fd658144330 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:05.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd6580b4230 con 0x7fd6580adfa0 2022-07-20T03:15:05.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd6580677b0 con 0x7fd658143ea0 2022-07-20T03:15:05.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ea05700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd6580b2950 con 0x7fd65805bf30 2022-07-20T03:15:05.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65d202700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd6580adfa0 0x7fd65813a550 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44452/0 (socket says 172.21.15.61:44452) 2022-07-20T03:15:05.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65d202700 1 -- 172.21.15.61:0/3526803933 learned_addr learned my addr 172.21.15.61:0/3526803933 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:05.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65e204700 1 --2- 172.21.15.61:0/3526803933 >> v2:172.21.15.61:3300/0 conn(0x7fd658143ea0 0x7fd658144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:05.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ca01700 1 -- 172.21.15.61:0/3526803933 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1098989738 0 0) 0x7fd6580b4230 con 0x7fd6580adfa0 2022-07-20T03:15:05.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65da03700 1 --2- 172.21.15.61:0/3526803933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd65805d0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:05.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ca01700 1 -- 172.21.15.61:0/3526803933 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd640003260 con 0x7fd6580adfa0 2022-07-20T03:15:05.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65ca01700 1 -- 172.21.15.61:0/3526803933 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 798660887 0 0) 0x7fd640003260 con 0x7fd6580adfa0 2022-07-20T03:15:05.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65e204700 1 -- 172.21.15.61:0/3526803933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 msgr2=0x7fd65805d0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:05.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65e204700 1 --2- 172.21.15.61:0/3526803933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd65805d0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65e204700 1 -- 172.21.15.61:0/3526803933 >> v1:172.21.15.61:6789/0 conn(0x7fd6580adfa0 legacy=0x7fd65813a550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:05.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.289+0000 7fd65e204700 1 -- 172.21.15.61:0/3526803933 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6580b3850 con 0x7fd658143ea0 2022-07-20T03:15:05.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 --2- 172.21.15.61:0/3526803933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658144330 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fd648001800 tx=0x7fd6500091d0).ready entity=mon.0 client_cookie=fcf641c922dcf65c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:05.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ca01700 1 -- 172.21.15.61:0/3526803933 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd648001660 con 0x7fd658143ea0 2022-07-20T03:15:05.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ca01700 1 -- 172.21.15.61:0/3526803933 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd65000f510 con 0x7fd658143ea0 2022-07-20T03:15:05.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ca01700 1 -- 172.21.15.61:0/3526803933 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd65000f680 con 0x7fd658143ea0 2022-07-20T03:15:05.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- 172.21.15.61:0/3526803933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 msgr2=0x7fd658144330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:05.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3526803933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658144330 secure :-1 s=READY pgs=889 cs=0 l=1 rev1=1 rx=0x7fd648001800 tx=0x7fd6500091d0).stop 2022-07-20T03:15:05.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- 172.21.15.61:0/3526803933 shutdown_connections 2022-07-20T03:15:05.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3526803933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658144330 unknown :-1 s=CLOSED pgs=889 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3526803933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd65805d0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- 172.21.15.61:0/3526803933 >> 172.21.15.61:0/3526803933 conn(0x7fd658064b90 msgr2=0x7fd658064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:05.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- 172.21.15.61:0/3526803933 shutdown_connections 2022-07-20T03:15:05.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- 172.21.15.61:0/3526803933 wait complete. 2022-07-20T03:15:05.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 Processor -- start 2022-07-20T03:15:05.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- start start 2022-07-20T03:15:05.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd658060a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:05.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6580adfa0 0x7fd658151d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:05.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658138aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:05.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd658139870 con 0x7fd658143ea0 2022-07-20T03:15:05.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6581553e0 con 0x7fd6580adfa0 2022-07-20T03:15:05.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65ea05700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd658061540 con 0x7fd65805bf30 2022-07-20T03:15:05.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658138aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:05.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658138aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47860/0 (socket says 172.21.15.61:47860) 2022-07-20T03:15:05.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 -- 172.21.15.61:0/3916877308 learned_addr learned my addr 172.21.15.61:0/3916877308 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:05.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65d202700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6580adfa0 0x7fd658151d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:05.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65da03700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd658060a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:05.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 -- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 msgr2=0x7fd658060a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:05.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd658060a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 -- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6580adfa0 msgr2=0x7fd658151d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:05.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6580adfa0 0x7fd658151d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:05.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 -- 172.21.15.61:0/3916877308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd650009470 con 0x7fd658143ea0 2022-07-20T03:15:05.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65da03700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd658060a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:05.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.293+0000 7fd65e204700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658138aa0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fd648001800 tx=0x7fd650016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:05.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd6500032d0 con 0x7fd658143ea0 2022-07-20T03:15:05.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd658138f90 con 0x7fd658143ea0 2022-07-20T03:15:05.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd650011c30 con 0x7fd658143ea0 2022-07-20T03:15:05.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd658061080 con 0x7fd658143ea0 2022-07-20T03:15:05.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd650021440 con 0x7fd658143ea0 2022-07-20T03:15:05.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd658155570 con 0x7fd658143ea0 2022-07-20T03:15:05.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd65000f070 con 0x7fd658143ea0 2022-07-20T03:15:05.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd646ffd700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd638053ad0 0x7fd638055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:05.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd65006c810 con 0x7fd658143ea0 2022-07-20T03:15:05.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.297+0000 7fd65da03700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd638053ad0 0x7fd638055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:05.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.301+0000 7fd65da03700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd638053ad0 0x7fd638055f90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd65400b000 tx=0x7fd654008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:05.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.305+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd658155570 con 0x7fd658143ea0 2022-07-20T03:15:05.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:05.477+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7fd658155570 con 0x7fd658143ea0 2022-07-20T03:15:06.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.161+0000 7fd646ffd700 1 -- 172.21.15.61:0/3916877308 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v356) v1 ==== 141+0+0 (secure 0 0 0) 0x7fd658155570 con 0x7fd658143ea0 2022-07-20T03:15:06.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd638053ad0 msgr2=0x7fd638055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd638053ad0 0x7fd638055f90 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fd65400b000 tx=0x7fd654008040).stop 2022-07-20T03:15:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 msgr2=0x7fd658138aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658138aa0 secure :-1 s=READY pgs=890 cs=0 l=1 rev1=1 rx=0x7fd648001800 tx=0x7fd650016000).stop 2022-07-20T03:15:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 shutdown_connections 2022-07-20T03:15:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd638053ad0 0x7fd638055f90 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd6580adfa0 0x7fd658151d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd658143ea0 0x7fd658138aa0 unknown :-1 s=CLOSED pgs=890 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 --2- 172.21.15.61:0/3916877308 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd65805bf30 0x7fd658060a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 >> 172.21.15.61:0/3916877308 conn(0x7fd658064b90 msgr2=0x7fd6580ad560 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:06.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 shutdown_connections 2022-07-20T03:15:06.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.165+0000 7fd65ea05700 1 -- 172.21.15.61:0/3916877308 wait complete. 2022-07-20T03:15:06.183 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:15:06.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb3fff700 1 Processor -- start 2022-07-20T03:15:06.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb3fff700 1 -- start start 2022-07-20T03:15:06.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb3fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fafb40b2ec0 0x7fafb40b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb3fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40b3f10 con 0x7fafb405d160 2022-07-20T03:15:06.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb3fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fafb40b3790 con 0x7fafb41461c0 2022-07-20T03:15:06.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb3fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fafb40b4ea0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb27fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fafb405d160 0x7fafb4142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44466/0 (socket says 172.21.15.61:44466) 2022-07-20T03:15:06.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb27fc700 1 -- 172.21.15.61:0/3952433897 learned_addr learned my addr 172.21.15.61:0/3952433897 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:06.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb2ffd700 1 --2- 172.21.15.61:0/3952433897 >> v2:172.21.15.117:3300/0 conn(0x7fafb40b2ec0 0x7fafb40b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 603331363 0 0) 0x7fafb40b3f10 con 0x7fafb405d160 2022-07-20T03:15:06.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.297+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7faf94003260 con 0x7fafb405d160 2022-07-20T03:15:06.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2426977545 0 0) 0x7fafb40b3790 con 0x7fafb41461c0 2022-07-20T03:15:06.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fafb40b3f10 con 0x7fafb41461c0 2022-07-20T03:15:06.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 453551959 0 0) 0x7faf94003260 con 0x7fafb405d160 2022-07-20T03:15:06.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb2ffd700 1 -- 172.21.15.61:0/3952433897 >> v1:172.21.15.117:6789/0 conn(0x7fafb41461c0 legacy=0x7fafb41465f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb2ffd700 1 -- 172.21.15.61:0/3952433897 >> v1:172.21.15.61:6789/0 conn(0x7fafb405d160 legacy=0x7fafb4142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb2ffd700 1 -- 172.21.15.61:0/3952433897 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb40b2620 con 0x7fafb40b2ec0 2022-07-20T03:15:06.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb2ffd700 1 --2- 172.21.15.61:0/3952433897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb40b2ec0 0x7fafb40b32a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fafa4001110 tx=0x7faf9c006df0).ready entity=mon.1 client_cookie=879310e37229a2df server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:06.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafa4001660 con 0x7fafb40b2ec0 2022-07-20T03:15:06.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faf9c000e90 con 0x7fafb40b2ec0 2022-07-20T03:15:06.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb1ffb700 1 -- 172.21.15.61:0/3952433897 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faf9c012460 con 0x7fafb40b2ec0 2022-07-20T03:15:06.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 -- 172.21.15.61:0/3952433897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb40b2ec0 msgr2=0x7fafb40b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 --2- 172.21.15.61:0/3952433897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb40b2ec0 0x7fafb40b32a0 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fafa4001110 tx=0x7faf9c006df0).stop 2022-07-20T03:15:06.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 -- 172.21.15.61:0/3952433897 shutdown_connections 2022-07-20T03:15:06.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 --2- 172.21.15.61:0/3952433897 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb40b2ec0 0x7fafb40b32a0 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 -- 172.21.15.61:0/3952433897 >> 172.21.15.61:0/3952433897 conn(0x7fafb4066e20 msgr2=0x7fafb4067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:06.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 -- 172.21.15.61:0/3952433897 shutdown_connections 2022-07-20T03:15:06.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 -- 172.21.15.61:0/3952433897 wait complete. 2022-07-20T03:15:06.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 Processor -- start 2022-07-20T03:15:06.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.301+0000 7fafb3fff700 1 -- start start 2022-07-20T03:15:06.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fafb405d160 0x7fafb4155b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 0x7fafb413d3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb41461c0 0x7fafb40ad300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb41124c0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb41126f0 con 0x7fafb41461c0 2022-07-20T03:15:06.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fafb4141580 con 0x7fafb405d160 2022-07-20T03:15:06.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 0x7fafb413d3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 0x7fafb413d3f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47874/0 (socket says 172.21.15.61:47874) 2022-07-20T03:15:06.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 -- 172.21.15.61:0/231006151 learned_addr learned my addr 172.21.15.61:0/231006151 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:06.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb2ffd700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fafb405d160 0x7fafb4155b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb37fe700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb41461c0 0x7fafb40ad300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 -- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fafb405d160 msgr2=0x7fafb4155b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fafb405d160 0x7fafb4155b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 -- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb41461c0 msgr2=0x7fafb40ad300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb41461c0 0x7fafb40ad300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 -- 172.21.15.61:0/231006151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf9c00b180 con 0x7fafb40b2ec0 2022-07-20T03:15:06.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb37fe700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb41461c0 0x7fafb40ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:06.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb27fc700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 0x7fafb413d3f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7fafa4000fe0 tx=0x7fafa40018d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:06.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafa4007170 con 0x7fafb40b2ec0 2022-07-20T03:15:06.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fafa40072d0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb4140a60 con 0x7fafb40b2ec0 2022-07-20T03:15:06.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fafa4004920 con 0x7fafb40b2ec0 2022-07-20T03:15:06.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb4141000 con 0x7fafb40b2ec0 2022-07-20T03:15:06.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.305+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb41128f0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.309+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fafa401b070 con 0x7fafb40b2ec0 2022-07-20T03:15:06.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.309+0000 7fafabfff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faf98053940 0x7faf98055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.309+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fafa406b880 con 0x7fafb40b2ec0 2022-07-20T03:15:06.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.309+0000 7fafb2ffd700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faf98053940 0x7faf98055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.309+0000 7fafb2ffd700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faf98053940 0x7faf98055e00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7faf9c018000 tx=0x7faf9c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:06.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.317+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fafa4042ac0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.441+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fafb40b28c0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.449+0000 7fafabfff700 1 -- 172.21.15.61:0/231006151 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v14)=0 v14) v1 ==== 122+0+0 (secure 0 0 0) 0x7fafa40220c0 con 0x7fafb40b2ec0 2022-07-20T03:15:06.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faf98053940 msgr2=0x7faf98055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faf98053940 0x7faf98055e00 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7faf9c018000 tx=0x7faf9c014000).stop 2022-07-20T03:15:06.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 msgr2=0x7fafb413d3f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 0x7fafb413d3f0 secure :-1 s=READY pgs=892 cs=0 l=1 rev1=1 rx=0x7fafa4000fe0 tx=0x7fafa40018d0).stop 2022-07-20T03:15:06.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 shutdown_connections 2022-07-20T03:15:06.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faf98053940 0x7faf98055e00 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fafb41461c0 0x7fafb40ad300 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fafb40b2ec0 0x7fafb413d3f0 unknown :-1 s=CLOSED pgs=892 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 --2- 172.21.15.61:0/231006151 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fafb405d160 0x7fafb4155b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 >> 172.21.15.61:0/231006151 conn(0x7fafb4066e20 msgr2=0x7fafb40b5180 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:06.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 shutdown_connections 2022-07-20T03:15:06.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.453+0000 7fafb3fff700 1 -- 172.21.15.61:0/231006151 wait complete. 2022-07-20T03:15:06.475 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:15:06.476 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:06.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:06.543 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:06.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:06.591 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:06.605 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:06.611 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:06.618 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:15:06.619 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:06.620 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:06.621 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:06.628 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:06.633 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:06.639 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:06.640 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:15:06.641 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:06.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:06.642 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:06.643 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:06.644 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:06.652 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:06.654 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:15:06.655 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:06.656 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:06.657 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:06.657 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:06.669 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:06.671 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:06.672 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:15:06.673 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:15:06.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.781+0000 7f8bcd6ee700 1 Processor -- start 2022-07-20T03:15:06.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.781+0000 7f8bcd6ee700 1 -- start start 2022-07-20T03:15:06.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc80ab010 0x7f8bc805d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8bc813e1f0 0x7f8bc813e680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8bc813eb70 con 0x7f8bc805cda0 2022-07-20T03:15:06.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8bc8061070 con 0x7f8bc813e1f0 2022-07-20T03:15:06.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80ab6f0 con 0x7f8bc80ab010 2022-07-20T03:15:06.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8bc805cda0 0x7f8bc813f7a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44480/0 (socket says 172.21.15.61:44480) 2022-07-20T03:15:06.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc7fff700 1 -- 172.21.15.61:0/2523424118 learned_addr learned my addr 172.21.15.61:0/2523424118 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:06.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcceed700 1 --2- 172.21.15.61:0/2523424118 >> v2:172.21.15.117:3300/0 conn(0x7f8bc813e1f0 0x7f8bc813e680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc77fe700 1 --2- 172.21.15.61:0/2523424118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc80ab010 0x7f8bc805d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc6ffd700 1 -- 172.21.15.61:0/2523424118 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3197028209 0 0) 0x7f8bc813eb70 con 0x7f8bc805cda0 2022-07-20T03:15:06.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc6ffd700 1 -- 172.21.15.61:0/2523424118 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8bb4003260 con 0x7f8bc805cda0 2022-07-20T03:15:06.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc77fe700 1 -- 172.21.15.61:0/2523424118 >> v2:172.21.15.117:3300/0 conn(0x7f8bc813e1f0 msgr2=0x7f8bc813e680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:06.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc77fe700 1 --2- 172.21.15.61:0/2523424118 >> v2:172.21.15.117:3300/0 conn(0x7f8bc813e1f0 0x7f8bc813e680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc77fe700 1 -- 172.21.15.61:0/2523424118 >> v1:172.21.15.61:6789/0 conn(0x7f8bc805cda0 legacy=0x7f8bc813f7a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc77fe700 1 -- 172.21.15.61:0/2523424118 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc80ab990 con 0x7f8bc80ab010 2022-07-20T03:15:06.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcceed700 1 --2- 172.21.15.61:0/2523424118 >> v2:172.21.15.117:3300/0 conn(0x7f8bc813e1f0 0x7f8bc813e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:06.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc77fe700 1 --2- 172.21.15.61:0/2523424118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc80ab010 0x7f8bc805d180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8bb8002f70 tx=0x7f8bb8006df0).ready entity=mon.2 client_cookie=f5269faf20e4832f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:06.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc6ffd700 1 -- 172.21.15.61:0/2523424118 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bb800d4a0 con 0x7f8bc80ab010 2022-07-20T03:15:06.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc6ffd700 1 -- 172.21.15.61:0/2523424118 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb8000e90 con 0x7f8bc80ab010 2022-07-20T03:15:06.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bc6ffd700 1 -- 172.21.15.61:0/2523424118 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bb800d7f0 con 0x7f8bc80ab010 2022-07-20T03:15:06.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/2523424118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc80ab010 msgr2=0x7f8bc805d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/2523424118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc80ab010 0x7f8bc805d180 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f8bb8002f70 tx=0x7f8bb8006df0).stop 2022-07-20T03:15:06.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/2523424118 shutdown_connections 2022-07-20T03:15:06.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/2523424118 >> v2:172.21.15.117:3300/0 conn(0x7f8bc813e1f0 0x7f8bc813e680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/2523424118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc80ab010 0x7f8bc805d180 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/2523424118 >> 172.21.15.61:0/2523424118 conn(0x7f8bc80b5e60 msgr2=0x7f8bc80b1e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:06.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/2523424118 shutdown_connections 2022-07-20T03:15:06.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.785+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/2523424118 wait complete. 2022-07-20T03:15:06.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 Processor -- start 2022-07-20T03:15:06.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 -- start start 2022-07-20T03:15:06.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc805cda0 0x7f8bc8134d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 0x7f8bc814fa80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc813e1f0 0x7f8bc81535c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc81417d0 con 0x7f8bc80ab010 2022-07-20T03:15:06.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc80aebd0 con 0x7f8bc805cda0 2022-07-20T03:15:06.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8bc8141530 con 0x7f8bc813e1f0 2022-07-20T03:15:06.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 0x7f8bc814fa80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 0x7f8bc814fa80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47886/0 (socket says 172.21.15.61:47886) 2022-07-20T03:15:06.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc805cda0 0x7f8bc8134d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 -- 172.21.15.61:0/868547631 learned_addr learned my addr 172.21.15.61:0/868547631 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:06.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcceed700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc813e1f0 0x7f8bc81535c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 -- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc813e1f0 msgr2=0x7f8bc81535c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc813e1f0 0x7f8bc81535c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 -- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc805cda0 msgr2=0x7f8bc8134d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:06.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc805cda0 0x7f8bc8134d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:06.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 -- 172.21.15.61:0/868547631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8bc0009470 con 0x7f8bc80ab010 2022-07-20T03:15:06.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc77fe700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 0x7f8bc814fa80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f8bb8002f70 tx=0x7f8bb8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:06.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bb80139c0 con 0x7f8bc80ab010 2022-07-20T03:15:06.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8bb8013b20 con 0x7f8bc80ab010 2022-07-20T03:15:06.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb800b180 con 0x7f8bc80ab010 2022-07-20T03:15:06.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8bb8013e80 con 0x7f8bc80ab010 2022-07-20T03:15:06.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.789+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bc814ff70 con 0x7f8bc80ab010 2022-07-20T03:15:06.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.793+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bc80abc30 con 0x7f8bc80ab010 2022-07-20T03:15:06.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.793+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8bb8025030 con 0x7f8bc80ab010 2022-07-20T03:15:06.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.793+0000 7f8bc57fa700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8ba8053920 0x7f8ba8055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:06.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.793+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8bb806ca10 con 0x7f8bc80ab010 2022-07-20T03:15:06.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.793+0000 7f8bc7fff700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8ba8053920 0x7f8ba8055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:06.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.797+0000 7f8bc7fff700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8ba8053920 0x7f8ba8055de0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f8bbc00d000 tx=0x7f8bbc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:06.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.801+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8bb8043b00 con 0x7f8bc80ab010 2022-07-20T03:15:06.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.933+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f8bc80574e0 con 0x7f8bc80ab010 2022-07-20T03:15:06.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.933+0000 7f8bc57fa700 1 -- 172.21.15.61:0/868547631 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v84) v1 ==== 81+0+28876 (secure 0 0 0) 0x7f8bb8023070 con 0x7f8bc80ab010 2022-07-20T03:15:06.940 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:06.940 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:06.940 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 84, 2022-07-20T03:15:06.941 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:15:06.941 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:15:06.941 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:15:04.134091+0000", 2022-07-20T03:15:06.941 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:06.942 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:06.942 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:15:06.942 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:15:06.942 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:15:06.943 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:15:06.943 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:15:06.943 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:15:06.943 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:15:06.943 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:06.944 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:15:06.944 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:15:06.944 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:15:06.944 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:15:06.945 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:15:06.945 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 16, 2022-07-20T03:15:06.945 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:15:06.945 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:15:06.946 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:15:06.946 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:15:06.946 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:15:06.946 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:06.946 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:15:06.947 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:15:06.947 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:15:06.947 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:06.947 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:06.948 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:06.948 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:06.948 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:06.948 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:06.949 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:06.949 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:06.949 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:15:06.949 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:15:06.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:15:06.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:15:06.950 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:15:06.950 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:06.951 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:06.951 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:06.951 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:06.951 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:06.952 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:06.953 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:06.953 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:06.954 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:15:06.956 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:06.957 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:06.957 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:06.957 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:06.957 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:06.958 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:06.958 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:06.958 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:06.958 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:06.959 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:06.959 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:06.959 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:06.960 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:06.960 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:06.960 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:06.960 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:06.961 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:06.961 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:06.961 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:06.962 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:06.962 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:06.962 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:06.962 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:06.963 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:06.963 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:06.963 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:06.963 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:06.964 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:06.964 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:06.964 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:06.964 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:06.965 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:06.966 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:06.967 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:06.968 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:06.968 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:06.969 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:06.969 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:06.972 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:06.973 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:15:06.973 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:06.974 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:06.975 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:15:06.975 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:06.975 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:06.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:06.976 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:15:06.976 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:15:06.976 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:15:06.990 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:06.991 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:06.992 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:06.993 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:06.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:06.994 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:06.994 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:06.994 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:06.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:06.995 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:06.995 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:06.995 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:06.995 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:15:06.996 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:06.996 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:06.996 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:06.996 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:06.997 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:06.997 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:06.997 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:06.997 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:06.998 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:06.998 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:06.998 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:06.998 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:06.999 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:06.999 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:06.999 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:06.999 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:06.999 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:07.000 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:07.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:07.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:07.000 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:07.001 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:07.001 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:07.001 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:07.001 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:07.002 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:07.002 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.002 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:07.002 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:07.003 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:07.003 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:07.003 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:07.003 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:07.003 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:07.004 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:07.004 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:07.005 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:07.005 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:07.005 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:07.006 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:07.006 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:15:07.007 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.007 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.008 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:07.008 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:15:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:15:07.009 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.010 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.010 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.010 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.011 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.011 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.011 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.011 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.011 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.012 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.012 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.012 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.012 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.013 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:15:07.013 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.013 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:15:07.014 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.015 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:15:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.017 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:15:07.018 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.018 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.018 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.018 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.019 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.019 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.019 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.019 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:15:07.020 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.020 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.020 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.021 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.021 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:15:07.021 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.021 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.022 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.022 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.022 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.022 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.023 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.023 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:15:07.024 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.024 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.025 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.025 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:15:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.026 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:15:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:15:07.027 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:15:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:15:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.028 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.029 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:07.030 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:15:07.030 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.030 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.031 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.031 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.031 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.032 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.032 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.032 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.033 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.033 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.033 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.035 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:15:07.035 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:15:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.038 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.038 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.038 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:15:07.039 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.039 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.041 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:15:07.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.041 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:15:07.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.044 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:15:07.046 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.046 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.046 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.047 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.047 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:15:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:15:07.050 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:07.050 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.050 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.051 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:15:07.051 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:15:07.051 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:15:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:15:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.052 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.053 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.053 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.053 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:15:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.054 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.055 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.055 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.056 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.057 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.057 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.058 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:15:07.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:15:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.060 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.062 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:15:07.062 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:15:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.063 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.064 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.065 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:15:07.066 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.066 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:15:07.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.067 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.067 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.068 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.068 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.068 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.069 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:15:07.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.070 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.070 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:15:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.071 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.072 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:15:07.072 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:15:07.072 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:15:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:15:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.073 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.074 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.075 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:07.075 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:15:07.075 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.075 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.076 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.077 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.077 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.077 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.077 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.078 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.078 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.078 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.079 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:15:07.079 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.079 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:15:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.080 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.082 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:15:07.082 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.083 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.083 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:15:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:15:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:15:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:15:07.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:15:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:15:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:15:07.094 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:15:07.094 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:15:07.095 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.095 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.095 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.096 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:15:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:15:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:15:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:15:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.106 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:15:07.106 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.106 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.107 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.107 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.107 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.107 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.108 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.108 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.108 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:15:07.108 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:15:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:15:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:15:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:15:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:15:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:15:07.116 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:15:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:15:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.125 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.125 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:15:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.130 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:15:07.131 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.131 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.132 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.133 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:15:07.135 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.136 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.137 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:15:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.142 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:15:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:15:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.151 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:15:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.155 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:15:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:15:07.158 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:15:07.158 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:15:07.159 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:15:07.159 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.159 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.161 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.161 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.161 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:07.162 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:15:07.162 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.163 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.163 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.163 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.164 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.164 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:07.165 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.167 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:15:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:15:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:15:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:15:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:15:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:15:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:15:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:15:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:15:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:15:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:15:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:15:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.196 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.196 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:15:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:07.198 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:07.198 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:07.199 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:07.199 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:07.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 68, 2022-07-20T03:15:07.202 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:07.202 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:15:07.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.205 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.206 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.206 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:15:07.207 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.207 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.209 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.209 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:15:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:15:07.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.212 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.213 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.214 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:15:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.217 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.217 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.218 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:15:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.220 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.220 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.221 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.221 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:07.222 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:07.222 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:15:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.225 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:07.225 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:15:07.226 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:07.226 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.228 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:15:07.228 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:15:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:15:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:15:07.230 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:07.230 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.232 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:07.232 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:15:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.234 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.234 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:15:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:07.238 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.238 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:15:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.241 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.241 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.243 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.243 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.244 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:15:07.244 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:15:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:15:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:15:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:15:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:15:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:15:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:15:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:15:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:15:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:15:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6835/843120801": "2022-07-21T03:15:00.968523+0000", 2022-07-20T03:15:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6834/843120801": "2022-07-21T03:15:00.968523+0000" 2022-07-20T03:15:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:15:07.290 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:15:07.301 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:15:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:15:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:15:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:15:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:15:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:15:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:15:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:15:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:15:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:15:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:15:07.307 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:07.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8ba8053920 msgr2=0x7f8ba8055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8ba8053920 0x7f8ba8055de0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f8bbc00d000 tx=0x7f8bbc009040).stop 2022-07-20T03:15:07.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 msgr2=0x7f8bc814fa80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 0x7f8bc814fa80 secure :-1 s=READY pgs=894 cs=0 l=1 rev1=1 rx=0x7f8bb8002f70 tx=0x7f8bb8014000).stop 2022-07-20T03:15:07.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 shutdown_connections 2022-07-20T03:15:07.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8ba8053920 0x7f8ba8055de0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8bc813e1f0 0x7f8bc81535c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8bc80ab010 0x7f8bc814fa80 unknown :-1 s=CLOSED pgs=894 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 --2- 172.21.15.61:0/868547631 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8bc805cda0 0x7f8bc8134d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 >> 172.21.15.61:0/868547631 conn(0x7f8bc80b5e60 msgr2=0x7f8bc80b31c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:07.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 shutdown_connections 2022-07-20T03:15:07.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:06.937+0000 7f8bcd6ee700 1 -- 172.21.15.61:0/868547631 wait complete. 2022-07-20T03:15:07.324 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:07.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.433+0000 7fc00e5c4700 1 Processor -- start 2022-07-20T03:15:07.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.433+0000 7fc00e5c4700 1 -- start start 2022-07-20T03:15:07.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fc00813fb30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0080b5d70 con 0x7fc0080aecf0 2022-07-20T03:15:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc0080651c0 con 0x7fc0080aff80 2022-07-20T03:15:07.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080b0690 con 0x7fc0080b5990 2022-07-20T03:15:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc0080aecf0 0x7fc008140020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44490/0 (socket says 172.21.15.61:44490) 2022-07-20T03:15:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 -- 172.21.15.61:0/3440421466 learned_addr learned my addr 172.21.15.61:0/3440421466 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:07.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00d5c2700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fc00813fb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1221727559 0 0) 0x7fc0080b5d70 con 0x7fc0080aecf0 2022-07-20T03:15:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfec003260 con 0x7fc0080aecf0 2022-07-20T03:15:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 281638898 0 0) 0x7fbfec003260 con 0x7fc0080aecf0 2022-07-20T03:15:07.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 msgr2=0x7fc00813fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fc00813fb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 >> v1:172.21.15.117:6789/0 conn(0x7fc0080aff80 legacy=0x7fc00805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0080b0930 con 0x7fc0080aecf0 2022-07-20T03:15:07.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fc0040017e0 con 0x7fc0080aecf0 2022-07-20T03:15:07.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00d5c2700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fc00813fb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:07.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 >> v1:172.21.15.61:6789/0 conn(0x7fc0080aecf0 legacy=0x7fc008140020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfec003780 0x7fbfec003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fbfec004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fbfec004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc0080b0690 con 0x7fbfec004500 2022-07-20T03:15:07.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec00b780 con 0x7fbfec003780 2022-07-20T03:15:07.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfec00b940 con 0x7fc0080b5990 2022-07-20T03:15:07.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fbfec004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00d5c2700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfec003780 0x7fbfec003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00ddc3700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fbfec004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 -- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 msgr2=0x7fbfec004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:07.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc0080b5990 0x7fbfec004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 -- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfec003780 msgr2=0x7fbfec003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:07.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfec003780 0x7fbfec003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 -- 172.21.15.61:0/3440421466 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfec003260 con 0x7fbfec004500 2022-07-20T03:15:07.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fbfec004980 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fc0040014f0 tx=0x7fc0080623f0).ready entity=mon.0 client_cookie=7b6e70d6b1ac6a01 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:07.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc004007110 con 0x7fbfec004500 2022-07-20T03:15:07.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc004007270 con 0x7fbfec004500 2022-07-20T03:15:07.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fbffffff700 1 -- 172.21.15.61:0/3440421466 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc004001d60 con 0x7fbfec004500 2022-07-20T03:15:07.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- 172.21.15.61:0/3440421466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 msgr2=0x7fbfec004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/3440421466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fbfec004980 secure :-1 s=READY pgs=896 cs=0 l=1 rev1=1 rx=0x7fc0040014f0 tx=0x7fc0080623f0).stop 2022-07-20T03:15:07.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00ddc3700 1 -- 172.21.15.61:0/3440421466 reap_dead start 2022-07-20T03:15:07.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- 172.21.15.61:0/3440421466 shutdown_connections 2022-07-20T03:15:07.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- 172.21.15.61:0/3440421466 >> 172.21.15.61:0/3440421466 conn(0x7fc0080634e0 msgr2=0x7fc0080b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:07.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- 172.21.15.61:0/3440421466 shutdown_connections 2022-07-20T03:15:07.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.437+0000 7fc00e5c4700 1 -- 172.21.15.61:0/3440421466 wait complete. 2022-07-20T03:15:07.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 Processor -- start 2022-07-20T03:15:07.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 -- start start 2022-07-20T03:15:07.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fc00814fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 0x7fc008067360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc008067850 0x7fc008153770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc008112800 con 0x7fbfec004500 2022-07-20T03:15:07.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc008112560 con 0x7fc00810e130 2022-07-20T03:15:07.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00e5c4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc0081532d0 con 0x7fc008067850 2022-07-20T03:15:07.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fc00814fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00cdc1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 0x7fc008067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00cdc1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 0x7fc008067360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:57584/0 (socket says 172.21.15.61:57584) 2022-07-20T03:15:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00ddc3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc008067850 0x7fc008153770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00cdc1700 1 -- 172.21.15.61:0/2178410201 learned_addr learned my addr 172.21.15.61:0/2178410201 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 -- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc008067850 msgr2=0x7fc008153770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc008067850 0x7fc008153770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 -- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 msgr2=0x7fc008067360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 0x7fc008067360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 -- 172.21.15.61:0/2178410201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0081441e0 con 0x7fbfec004500 2022-07-20T03:15:07.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 0x7fc008067360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:07.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.441+0000 7fc00d5c2700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fc00814fbb0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fbff800c000 tx=0x7fbff8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:07.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbff80128d0 con 0x7fbfec004500 2022-07-20T03:15:07.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbff8012a30 con 0x7fbfec004500 2022-07-20T03:15:07.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc0081500a0 con 0x7fbfec004500 2022-07-20T03:15:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbff8012d90 con 0x7fbfec004500 2022-07-20T03:15:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc008153c60 con 0x7fbfec004500 2022-07-20T03:15:07.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc0080574e0 con 0x7fbfec004500 2022-07-20T03:15:07.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbff8014470 con 0x7fbfec004500 2022-07-20T03:15:07.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fbffe7fc700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbff00539d0 0x7fbff0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fbff806c0e0 con 0x7fbfec004500 2022-07-20T03:15:07.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.445+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbff00539d0 0x7fbff0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.449+0000 7fc00cdc1700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbff00539d0 0x7fbff0055e90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc00401a000 tx=0x7fc004013000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:07.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.453+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbff8043220 con 0x7fbfec004500 2022-07-20T03:15:07.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.617+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc008067ea0 con 0x7fbfec004500 2022-07-20T03:15:07.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.617+0000 7fbffe7fc700 1 -- 172.21.15.61:0/2178410201 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+1212 (secure 0 0 0) 0x7fbff8046840 con 0x7fbfec004500 2022-07-20T03:15:07.622 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:07.622 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44},{"gid":6104,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2927313230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2927313230},{"type":"v1","addr":"172.21.15.117:6835","nonce":2927313230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":47}],"filesystems":[]} 2022-07-20T03:15:07.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbff00539d0 msgr2=0x7fbff0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbff00539d0 0x7fbff0055e90 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7fc00401a000 tx=0x7fc004013000).stop 2022-07-20T03:15:07.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 msgr2=0x7fc00814fbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fc00814fbb0 secure :-1 s=READY pgs=897 cs=0 l=1 rev1=1 rx=0x7fbff800c000 tx=0x7fbff8008040).stop 2022-07-20T03:15:07.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 shutdown_connections 2022-07-20T03:15:07.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbff00539d0 0x7fbff0055e90 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc00810e130 0x7fc008067360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc008067850 0x7fc008153770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 --2- 172.21.15.61:0/2178410201 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfec004500 0x7fc00814fbb0 unknown :-1 s=CLOSED pgs=897 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 >> 172.21.15.61:0/2178410201 conn(0x7fc0080634e0 msgr2=0x7fc00813cd10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:07.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 shutdown_connections 2022-07-20T03:15:07.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.621+0000 7fc00e5c4700 1 -- 172.21.15.61:0/2178410201 wait complete. 2022-07-20T03:15:07.627 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 47 2022-07-20T03:15:07.644 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:07.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d5afb700 1 Processor -- start 2022-07-20T03:15:07.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d5afb700 1 -- start start 2022-07-20T03:15:07.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d5afb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71d0098ae0 0x7f71d0061760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d5afb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d00b22d0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d5afb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f71d00abe50 con 0x7f71d0062040 2022-07-20T03:15:07.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d5afb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71d00ac0f0 con 0x7f71d0098ae0 2022-07-20T03:15:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71d0098ae0 0x7f71d0061760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f71d0098ae0 0x7f71d0061760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47916/0 (socket says 172.21.15.61:47916) 2022-07-20T03:15:07.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 -- 172.21.15.61:0/3995843245 learned_addr learned my addr 172.21.15.61:0/3995843245 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:07.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 -- 172.21.15.61:0/3995843245 >> v1:172.21.15.117:6789/0 conn(0x7f71d0062040 legacy=0x7f71d0146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 -- 172.21.15.61:0/3995843245 >> v1:172.21.15.61:6789/0 conn(0x7f71d00b1ef0 legacy=0x7f71d0061050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:07.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 -- 172.21.15.61:0/3995843245 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71d00ac390 con 0x7f71d0098ae0 2022-07-20T03:15:07.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71d52fa700 1 --2- 172.21.15.61:0/3995843245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d0098ae0 0x7f71d0061760 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f71d0060ef0 tx=0x7f71c0006eb0).ready entity=mon.0 client_cookie=a4c1228a748391b9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:07.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71cf7fe700 1 -- 172.21.15.61:0/3995843245 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c000d5c0 con 0x7f71d0098ae0 2022-07-20T03:15:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71cf7fe700 1 -- 172.21.15.61:0/3995843245 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c0000e90 con 0x7f71d0098ae0 2022-07-20T03:15:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.749+0000 7f71cf7fe700 1 -- 172.21.15.61:0/3995843245 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c000d950 con 0x7f71d0098ae0 2022-07-20T03:15:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 -- 172.21.15.61:0/3995843245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d0098ae0 msgr2=0x7f71d0061760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 --2- 172.21.15.61:0/3995843245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d0098ae0 0x7f71d0061760 secure :-1 s=READY pgs=899 cs=0 l=1 rev1=1 rx=0x7f71d0060ef0 tx=0x7f71c0006eb0).stop 2022-07-20T03:15:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 -- 172.21.15.61:0/3995843245 shutdown_connections 2022-07-20T03:15:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 --2- 172.21.15.61:0/3995843245 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d0098ae0 0x7f71d0061760 unknown :-1 s=CLOSED pgs=899 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 -- 172.21.15.61:0/3995843245 >> 172.21.15.61:0/3995843245 conn(0x7f71d00b2ca0 msgr2=0x7f71d00b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 -- 172.21.15.61:0/3995843245 shutdown_connections 2022-07-20T03:15:07.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 -- 172.21.15.61:0/3995843245 wait complete. 2022-07-20T03:15:07.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 Processor -- start 2022-07-20T03:15:07.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.753+0000 7f71d5afb700 1 -- start start 2022-07-20T03:15:07.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71d0062040 0x7f71d00b02c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71d0098ae0 0x7f71d00ad640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 0x7f71d00b4d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d00992e0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f71d0099b60 con 0x7f71d0062040 2022-07-20T03:15:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71d0099e00 con 0x7f71d0098ae0 2022-07-20T03:15:07.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 0x7f71d00b4d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 0x7f71d00b4d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47922/0 (socket says 172.21.15.61:47922) 2022-07-20T03:15:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 -- 172.21.15.61:0/622415582 learned_addr learned my addr 172.21.15.61:0/622415582 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:07.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d4af9700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71d0062040 0x7f71d00b02c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71cffff700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71d0098ae0 0x7f71d00ad640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 -- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71d0098ae0 msgr2=0x7f71d00ad640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71d0098ae0 0x7f71d00ad640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 -- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71d0062040 msgr2=0x7f71d00b02c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71d0062040 0x7f71d00b02c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 -- 172.21.15.61:0/622415582 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71c000b1e0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d52fa700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 0x7f71d00b4d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f71d0060ef0 tx=0x7f71c0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c0011790 con 0x7f71d00b1ef0 2022-07-20T03:15:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f71c00118f0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71d00b5270 con 0x7f71d00b1ef0 2022-07-20T03:15:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f71c0011c50 con 0x7f71d00b1ef0 2022-07-20T03:15:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.757+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f71d0099870 con 0x7f71d00b1ef0 2022-07-20T03:15:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.761+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71d0099fc0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.761+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f71c0025030 con 0x7f71d00b1ef0 2022-07-20T03:15:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.761+0000 7f71cdffb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71b0053a40 0x7f71b0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.761+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(84..84 src has 1..84) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f71c006cfb0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.761+0000 7f71d4af9700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71b0053a40 0x7f71b0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:07.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.761+0000 7f71d4af9700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71b0053a40 0x7f71b0055f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f71c800d000 tx=0x7f71c8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:07.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.769+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f71c00440e0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.929+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f71d00574e0 con 0x7f71d00b1ef0 2022-07-20T03:15:07.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.929+0000 7f71cdffb700 1 -- 172.21.15.61:0/622415582 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 47 v47) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f71c0023090 con 0x7f71d00b1ef0 2022-07-20T03:15:07.937 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:07.937 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":47,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":5936,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3725240447","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3725240447},{"type":"v1","addr":"172.21.15.61:6833","nonce":3725240447}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":44},{"gid":6104,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2927313230","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2927313230},{"type":"v1","addr":"172.21.15.117:6835","nonce":2927313230}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":47}],"filesystems":[]} 2022-07-20T03:15:07.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71b0053a40 msgr2=0x7f71b0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71b0053a40 0x7f71b0055f00 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f71c800d000 tx=0x7f71c8009040).stop 2022-07-20T03:15:07.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 msgr2=0x7f71d00b4d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:07.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 0x7f71d00b4d80 secure :-1 s=READY pgs=900 cs=0 l=1 rev1=1 rx=0x7f71d0060ef0 tx=0x7f71c0015000).stop 2022-07-20T03:15:07.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 shutdown_connections 2022-07-20T03:15:07.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71b0053a40 0x7f71b0055f00 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71d00b1ef0 0x7f71d00b4d80 unknown :-1 s=CLOSED pgs=900 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71d0098ae0 0x7f71d00ad640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 --2- 172.21.15.61:0/622415582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71d0062040 0x7f71d00b02c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:07.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 >> 172.21.15.61:0/622415582 conn(0x7f71d00b2ca0 msgr2=0x7f71d01470d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:07.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 shutdown_connections 2022-07-20T03:15:07.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:07.933+0000 7f71d5afb700 1 -- 172.21.15.61:0/622415582 wait complete. 2022-07-20T03:15:07.949 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 47 2022-07-20T03:15:07.960 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:15:07.961 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:15:07.962 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:15:07.962 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:15:07.963 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:15:07.964 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:15:07.964 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:15:07.965 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:15:07.982 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:15:07.998 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:15:08.001 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:15:08.006 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:15:08.014 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:15:08.018 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:15:08.020 DEBUG:teuthology.parallel:result is None 2022-07-20T03:15:08.020 DEBUG:teuthology.parallel:result is None 2022-07-20T03:15:08.027 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:15:08.087 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:15:08.094 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:15:08.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.129+0000 7fd1060e1700 1 Processor -- start 2022-07-20T03:15:08.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.129+0000 7fd1060e1700 1 -- start start 2022-07-20T03:15:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1060e1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd1000b6200 0x7fd1000b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1060e1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000b65e0 con 0x7fd10009b760 2022-07-20T03:15:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1060e1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd1000b0350 con 0x7fd10013dfc0 2022-07-20T03:15:08.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1060e1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd1000aaed0 con 0x7fd1000b6200 2022-07-20T03:15:08.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1048de700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd10009b760 0x7fd1000b0de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44538/0 (socket says 172.21.15.61:44538) 2022-07-20T03:15:08.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1048de700 1 -- 172.21.15.61:0/1120359779 learned_addr learned my addr 172.21.15.61:0/1120359779 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:08.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1050df700 1 --2- 172.21.15.61:0/1120359779 >> v2:172.21.15.117:3300/0 conn(0x7fd1000b6200 0x7fd1000b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:08.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd0effff700 1 -- 172.21.15.61:0/1120359779 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1339228989 0 0) 0x7fd1000b65e0 con 0x7fd10009b760 2022-07-20T03:15:08.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd0effff700 1 -- 172.21.15.61:0/1120359779 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd0e8003260 con 0x7fd10009b760 2022-07-20T03:15:08.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd0effff700 1 -- 172.21.15.61:0/1120359779 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1584899797 0 0) 0x7fd0e8003260 con 0x7fd10009b760 2022-07-20T03:15:08.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1050df700 1 -- 172.21.15.61:0/1120359779 >> v1:172.21.15.117:6789/0 conn(0x7fd10013dfc0 legacy=0x7fd10013e3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:08.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1050df700 1 -- 172.21.15.61:0/1120359779 >> v1:172.21.15.61:6789/0 conn(0x7fd10009b760 legacy=0x7fd1000b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:08.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1050df700 1 -- 172.21.15.61:0/1120359779 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd1000ac9c0 con 0x7fd1000b6200 2022-07-20T03:15:08.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1050df700 1 --2- 172.21.15.61:0/1120359779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b08f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fd0f0001800 tx=0x7fd0fc00a370).ready entity=mon.1 client_cookie=598533fd0a1ce08e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:08.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd0effff700 1 -- 172.21.15.61:0/1120359779 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0fc014070 con 0x7fd1000b6200 2022-07-20T03:15:08.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd0effff700 1 -- 172.21.15.61:0/1120359779 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0fc00c870 con 0x7fd1000b6200 2022-07-20T03:15:08.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd0effff700 1 -- 172.21.15.61:0/1120359779 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0f0001660 con 0x7fd1000b6200 2022-07-20T03:15:08.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1060e1700 1 -- 172.21.15.61:0/1120359779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 msgr2=0x7fd1000b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:08.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.133+0000 7fd1060e1700 1 --2- 172.21.15.61:0/1120359779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b08f0 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7fd0f0001800 tx=0x7fd0fc00a370).stop 2022-07-20T03:15:08.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- 172.21.15.61:0/1120359779 shutdown_connections 2022-07-20T03:15:08.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 --2- 172.21.15.61:0/1120359779 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b08f0 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:08.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- 172.21.15.61:0/1120359779 >> 172.21.15.61:0/1120359779 conn(0x7fd100063d10 msgr2=0x7fd1000b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:08.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- 172.21.15.61:0/1120359779 shutdown_connections 2022-07-20T03:15:08.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- 172.21.15.61:0/1120359779 wait complete. 2022-07-20T03:15:08.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 Processor -- start 2022-07-20T03:15:08.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- start start 2022-07-20T03:15:08.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd10009b760 0x7fd10013b910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:08.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b4020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:08.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 0x7fd100151050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:08.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd100140660 con 0x7fd10013dfc0 2022-07-20T03:15:08.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd10013c690 con 0x7fd1000b6200 2022-07-20T03:15:08.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1060e1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd10013c930 con 0x7fd10009b760 2022-07-20T03:15:08.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 0x7fd100151050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:08.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 0x7fd100151050 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47948/0 (socket says 172.21.15.61:47948) 2022-07-20T03:15:08.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 -- 172.21.15.61:0/64523078 learned_addr learned my addr 172.21.15.61:0/64523078 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:08.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1048de700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b4020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:08.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1050df700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd10009b760 0x7fd10013b910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:08.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 -- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd10009b760 msgr2=0x7fd10013b910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:08.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd10009b760 0x7fd10013b910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:08.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 -- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 msgr2=0x7fd1000b4020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:08.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b4020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:08.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.137+0000 7fd1058e0700 1 -- 172.21.15.61:0/64523078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd0fc00a610 con 0x7fd10013dfc0 2022-07-20T03:15:08.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd1058e0700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 0x7fd100151050 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7fd10006ad70 tx=0x7fd0f8007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:08.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0f8007030 con 0x7fd10013dfc0 2022-07-20T03:15:08.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd0f8007190 con 0x7fd10013dfc0 2022-07-20T03:15:08.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd10013bfc0 con 0x7fd10013dfc0 2022-07-20T03:15:08.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd0f8006510 con 0x7fd10013dfc0 2022-07-20T03:15:08.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd10013c460 con 0x7fd10013dfc0 2022-07-20T03:15:08.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd1000574e0 con 0x7fd10013dfc0 2022-07-20T03:15:08.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd0f8010030 con 0x7fd10013dfc0 2022-07-20T03:15:08.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd0ee7fc700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd0e0053ad0 0x7fd0e0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:08.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(85..85 src has 1..85) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd0f806b6c0 con 0x7fd10013dfc0 2022-07-20T03:15:08.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.141+0000 7fd1050df700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd0e0053ad0 0x7fd0e0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:08.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.145+0000 7fd1050df700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd0e0053ad0 0x7fd0e0055f90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fd0fc01b000 tx=0x7fd0fc016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:08.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.153+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd0f8042900 con 0x7fd10013dfc0 2022-07-20T03:15:08.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:08.289+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fd100069a20 con 0x7fd10013dfc0 2022-07-20T03:15:09.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.185+0000 7fd0ee7fc700 1 -- 172.21.15.61:0/64523078 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v86) v1 ==== 97+0+0 (secure 0 0 0) 0x7fd0f8045f20 con 0x7fd10013dfc0 2022-07-20T03:15:09.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd0e0053ad0 msgr2=0x7fd0e0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd0e0053ad0 0x7fd0e0055f90 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7fd0fc01b000 tx=0x7fd0fc016000).stop 2022-07-20T03:15:09.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 msgr2=0x7fd100151050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 0x7fd100151050 secure :-1 s=READY pgs=905 cs=0 l=1 rev1=1 rx=0x7fd10006ad70 tx=0x7fd0f8007c20).stop 2022-07-20T03:15:09.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 shutdown_connections 2022-07-20T03:15:09.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd0e0053ad0 0x7fd0e0055f90 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd1000b6200 0x7fd1000b4020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd10013dfc0 0x7fd100151050 unknown :-1 s=CLOSED pgs=905 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 --2- 172.21.15.61:0/64523078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd10009b760 0x7fd10013b910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 >> 172.21.15.61:0/64523078 conn(0x7fd100063d10 msgr2=0x7fd10005ff10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:09.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 shutdown_connections 2022-07-20T03:15:09.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.189+0000 7fd1060e1700 1 -- 172.21.15.61:0/64523078 wait complete. 2022-07-20T03:15:09.204 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:15:09.228 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:15:09.229 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:15:09.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.333+0000 7f5fcc236700 1 Processor -- start 2022-07-20T03:15:09.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- start start 2022-07-20T03:15:09.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5fc40b1e00 0x7f5fc414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc405d480 con 0x7f5fc405c3f0 2022-07-20T03:15:09.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5fc40b21e0 con 0x7f5fc40acef0 2022-07-20T03:15:09.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5fc405cf40 con 0x7f5fc40b1e00 2022-07-20T03:15:09.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc97d1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5fc405c3f0 0x7f5fc40ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44552/0 (socket says 172.21.15.61:44552) 2022-07-20T03:15:09.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc97d1700 1 -- 172.21.15.61:0/3664550164 learned_addr learned my addr 172.21.15.61:0/3664550164 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:09.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc9fd2700 1 --2- 172.21.15.61:0/3664550164 >> v2:172.21.15.117:3300/0 conn(0x7f5fc40b1e00 0x7f5fc414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc8fd0700 1 -- 172.21.15.61:0/3664550164 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2940194418 0 0) 0x7f5fc405d480 con 0x7f5fc405c3f0 2022-07-20T03:15:09.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc8fd0700 1 -- 172.21.15.61:0/3664550164 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5fac003260 con 0x7f5fc405c3f0 2022-07-20T03:15:09.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc8fd0700 1 -- 172.21.15.61:0/3664550164 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3270511125 0 0) 0x7f5fac003260 con 0x7f5fc405c3f0 2022-07-20T03:15:09.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/3664550164 >> v1:172.21.15.117:6789/0 conn(0x7f5fc40acef0 legacy=0x7f5fc4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/3664550164 >> v1:172.21.15.61:6789/0 conn(0x7f5fc405c3f0 legacy=0x7f5fc40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/3664550164 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fc405d1e0 con 0x7f5fc40b1e00 2022-07-20T03:15:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc9fd2700 1 --2- 172.21.15.61:0/3664550164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc40b1e00 0x7f5fc414a6d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f5fc0001800 tx=0x7f5fb8006df0).ready entity=mon.1 client_cookie=90f719755976dcb5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:09.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc8fd0700 1 -- 172.21.15.61:0/3664550164 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fc0001660 con 0x7f5fc40b1e00 2022-07-20T03:15:09.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc8fd0700 1 -- 172.21.15.61:0/3664550164 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8000e90 con 0x7f5fc40b1e00 2022-07-20T03:15:09.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fc8fd0700 1 -- 172.21.15.61:0/3664550164 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb800d4f0 con 0x7f5fc40b1e00 2022-07-20T03:15:09.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- 172.21.15.61:0/3664550164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc40b1e00 msgr2=0x7f5fc414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 --2- 172.21.15.61:0/3664550164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc40b1e00 0x7f5fc414a6d0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f5fc0001800 tx=0x7f5fb8006df0).stop 2022-07-20T03:15:09.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- 172.21.15.61:0/3664550164 shutdown_connections 2022-07-20T03:15:09.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 --2- 172.21.15.61:0/3664550164 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc40b1e00 0x7f5fc414a6d0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- 172.21.15.61:0/3664550164 >> 172.21.15.61:0/3664550164 conn(0x7f5fc40611c0 msgr2=0x7f5fc4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- 172.21.15.61:0/3664550164 shutdown_connections 2022-07-20T03:15:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.337+0000 7f5fcc236700 1 -- 172.21.15.61:0/3664550164 wait complete. 2022-07-20T03:15:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 Processor -- start 2022-07-20T03:15:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 -- start start 2022-07-20T03:15:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 0x7f5fc409f8b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5fc40b1e00 0x7f5fc409e8e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc41424c0 0x7f5fc4142970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc4142e60 con 0x7f5fc40acef0 2022-07-20T03:15:09.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40a0ba0 con 0x7f5fc41424c0 2022-07-20T03:15:09.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5fc40a01c0 con 0x7f5fc40b1e00 2022-07-20T03:15:09.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 0x7f5fc409f8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 0x7f5fc409f8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47960/0 (socket says 172.21.15.61:47960) 2022-07-20T03:15:09.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/75166271 learned_addr learned my addr 172.21.15.61:0/75166271 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:09.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc97d1700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5fc40b1e00 0x7f5fc409e8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fca7d3700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc41424c0 0x7f5fc4142970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5fc40b1e00 msgr2=0x7f5fc409e8e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5fc40b1e00 0x7f5fc409e8e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc41424c0 msgr2=0x7f5fc4142970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc41424c0 0x7f5fc4142970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 -- 172.21.15.61:0/75166271 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fb800b180 con 0x7f5fc40acef0 2022-07-20T03:15:09.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fc9fd2700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 0x7f5fc409f8b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f5fc0001800 tx=0x7f5fb8014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:09.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb8002f90 con 0x7f5fc40acef0 2022-07-20T03:15:09.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5fb8012860 con 0x7f5fc40acef0 2022-07-20T03:15:09.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fc40a0460 con 0x7f5fc40acef0 2022-07-20T03:15:09.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5fb8012bc0 con 0x7f5fc40acef0 2022-07-20T03:15:09.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.341+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fc409fe40 con 0x7f5fc40acef0 2022-07-20T03:15:09.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.345+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fc40b04b0 con 0x7f5fc40acef0 2022-07-20T03:15:09.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.345+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5fb800f030 con 0x7f5fc40acef0 2022-07-20T03:15:09.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.345+0000 7f5fb6ffd700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5fa4053a60 0x7f5fa4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.345+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5fb8018030 con 0x7f5fc40acef0 2022-07-20T03:15:09.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.345+0000 7f5fc97d1700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5fa4053a60 0x7f5fa4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.349+0000 7f5fc97d1700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5fa4053a60 0x7f5fa4055f20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5fc000d000 tx=0x7f5fc0000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:09.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.353+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5fb8043940 con 0x7f5fc40acef0 2022-07-20T03:15:09.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.513+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5fc40574e0 con 0x7f5fc40acef0 2022-07-20T03:15:09.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fb6ffd700 1 -- 172.21.15.61:0/75166271 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v31) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f5fb802e0b0 con 0x7f5fc40acef0 2022-07-20T03:15:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:09.526 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:15:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:15:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:15:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.530 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:15:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:15:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:15:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:15:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:15:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:15:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.554 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.554 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.555 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:15:09.555 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:15:09.555 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.557 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.557 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.559 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:15:09.559 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:15:09.560 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.561 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.561 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.562 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.563 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.563 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.564 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:15:09.564 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:15:09.565 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.565 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.566 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.570 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.571 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.571 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.571 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.571 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:15:09.572 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:15:09.572 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.572 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.572 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.573 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.574 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:15:09.574 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:15:09.574 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.574 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.575 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.576 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.576 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.576 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.576 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.577 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:15:09.577 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:15:09.577 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.577 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.578 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.578 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.578 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.578 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.578 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.579 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:15:09.579 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:15:09.579 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.579 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:09.580 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:09.580 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:09.580 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.580 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:15:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:15:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.581 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:09.583 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:09.583 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.584 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:15:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:15:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:09.585 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:09.586 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.586 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.586 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.586 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:15:09.587 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:15:09.587 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.587 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:09.587 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:09.587 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:09.588 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.588 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.588 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.588 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.589 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:15:09.589 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:15:09.589 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.590 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:09.590 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:15:09.590 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.590 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:15:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:15:09.591 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:15:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:09.592 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.593 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:09.593 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:09.593 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:09.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5fa4053a60 msgr2=0x7f5fa4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5fa4053a60 0x7f5fa4055f20 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f5fc000d000 tx=0x7f5fc0000bc0).stop 2022-07-20T03:15:09.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 msgr2=0x7f5fc409f8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 0x7f5fc409f8b0 secure :-1 s=READY pgs=907 cs=0 l=1 rev1=1 rx=0x7f5fc0001800 tx=0x7f5fb8014000).stop 2022-07-20T03:15:09.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 shutdown_connections 2022-07-20T03:15:09.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5fa4053a60 0x7f5fa4055f20 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5fc41424c0 0x7f5fc4142970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5fc40b1e00 0x7f5fc409e8e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 --2- 172.21.15.61:0/75166271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5fc40acef0 0x7f5fc409f8b0 unknown :-1 s=CLOSED pgs=907 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 >> 172.21.15.61:0/75166271 conn(0x7f5fc405c3f0 msgr2=0x7f5fc40b46f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:09.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 shutdown_connections 2022-07-20T03:15:09.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.517+0000 7f5fcc236700 1 -- 172.21.15.61:0/75166271 wait complete. 2022-07-20T03:15:09.600 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:09.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.705+0000 7feabe77d700 1 Processor -- start 2022-07-20T03:15:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.705+0000 7feabe77d700 1 -- start start 2022-07-20T03:15:09.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab80b0570 0x7feab80b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feab80aea10 0x7feab8141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7feab81422c0 0x7feab80ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab8143b10 con 0x7feab80aea10 2022-07-20T03:15:09.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feab8142750 con 0x7feab81422c0 2022-07-20T03:15:09.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab81433d0 con 0x7feab80b0570 2022-07-20T03:15:09.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feab80aea10 0x7feab8141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feab80aea10 0x7feab8141dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47968/0 (socket says 172.21.15.61:47968) 2022-07-20T03:15:09.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 -- 172.21.15.61:0/6391002 learned_addr learned my addr 172.21.15.61:0/6391002 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:09.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabd77b700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab80b0570 0x7feab80b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabdf7c700 1 --2- 172.21.15.61:0/6391002 >> v2:172.21.15.117:3300/0 conn(0x7feab81422c0 0x7feab80ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 -- 172.21.15.61:0/6391002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab80b0570 msgr2=0x7feab80b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:09.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab80b0570 0x7feab80b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 -- 172.21.15.61:0/6391002 >> v2:172.21.15.117:3300/0 conn(0x7feab81422c0 msgr2=0x7feab80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:09.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 --2- 172.21.15.61:0/6391002 >> v2:172.21.15.117:3300/0 conn(0x7feab81422c0 0x7feab80ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 -- 172.21.15.61:0/6391002 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab8143670 con 0x7feab80aea10 2022-07-20T03:15:09.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabd77b700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab80b0570 0x7feab80b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:09.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabdf7c700 1 --2- 172.21.15.61:0/6391002 >> v2:172.21.15.117:3300/0 conn(0x7feab81422c0 0x7feab80ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:09.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabcf7a700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80aea10 0x7feab8141dd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7feaac002f70 tx=0x7feaac006eb0).ready entity=mon.0 client_cookie=966ce2f733a0a7e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:09.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feaabfff700 1 -- 172.21.15.61:0/6391002 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaac00d570 con 0x7feab80aea10 2022-07-20T03:15:09.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feaabfff700 1 -- 172.21.15.61:0/6391002 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac000e90 con 0x7feab80aea10 2022-07-20T03:15:09.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feaabfff700 1 -- 172.21.15.61:0/6391002 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaac00d900 con 0x7feab80aea10 2022-07-20T03:15:09.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- 172.21.15.61:0/6391002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80aea10 msgr2=0x7feab8141dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80aea10 0x7feab8141dd0 secure :-1 s=READY pgs=908 cs=0 l=1 rev1=1 rx=0x7feaac002f70 tx=0x7feaac006eb0).stop 2022-07-20T03:15:09.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- 172.21.15.61:0/6391002 shutdown_connections 2022-07-20T03:15:09.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- 172.21.15.61:0/6391002 >> v2:172.21.15.117:3300/0 conn(0x7feab81422c0 0x7feab80ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80aea10 0x7feab8141dd0 unknown :-1 s=CLOSED pgs=908 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 --2- 172.21.15.61:0/6391002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab80b0570 0x7feab80b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- 172.21.15.61:0/6391002 >> 172.21.15.61:0/6391002 conn(0x7feab80b9360 msgr2=0x7feab80b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:09.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- 172.21.15.61:0/6391002 shutdown_connections 2022-07-20T03:15:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.709+0000 7feabe77d700 1 -- 172.21.15.61:0/6391002 wait complete. 2022-07-20T03:15:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 Processor -- start 2022-07-20T03:15:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 -- start start 2022-07-20T03:15:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab80aea10 0x7feab81440c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 0x7feab813ed00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab81422c0 0x7feab813f1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab81460b0 con 0x7feab80b0570 2022-07-20T03:15:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feab813f6e0 con 0x7feab80aea10 2022-07-20T03:15:09.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 0x7feab813ed00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabe77d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feab8146a50 con 0x7feab81422c0 2022-07-20T03:15:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 0x7feab813ed00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47976/0 (socket says 172.21.15.61:47976) 2022-07-20T03:15:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 -- 172.21.15.61:0/3418208081 learned_addr learned my addr 172.21.15.61:0/3418208081 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:09.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabdf7c700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab81422c0 0x7feab813f1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabd77b700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab80aea10 0x7feab81440c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 -- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab81422c0 msgr2=0x7feab813f1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab81422c0 0x7feab813f1f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 -- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab80aea10 msgr2=0x7feab81440c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab80aea10 0x7feab81440c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabcf7a700 1 -- 172.21.15.61:0/3418208081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feaac00b1e0 con 0x7feab80b0570 2022-07-20T03:15:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.713+0000 7feabd77b700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab80aea10 0x7feab81440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:09.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feabcf7a700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 0x7feab813ed00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7feab8146970 tx=0x7feaac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaac011860 con 0x7feab80b0570 2022-07-20T03:15:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feaac0119c0 con 0x7feab80b0570 2022-07-20T03:15:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feab8155e00 con 0x7feab80b0570 2022-07-20T03:15:09.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feaac011d20 con 0x7feab80b0570 2022-07-20T03:15:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feab80b1780 con 0x7feab80b0570 2022-07-20T03:15:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feab80b0d80 con 0x7feab80b0570 2022-07-20T03:15:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feaac025030 con 0x7feab80b0570 2022-07-20T03:15:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feaaa7fc700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feaa00539b0 0x7feaa0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:09.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7feaac06cef0 con 0x7feab80b0570 2022-07-20T03:15:09.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.717+0000 7feabd77b700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feaa00539b0 0x7feaa0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:09.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.721+0000 7feabd77b700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feaa00539b0 0x7feaa0055e70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7feab400b000 tx=0x7feab4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:09.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.725+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feaac044180 con 0x7feab80b0570 2022-07-20T03:15:09.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.857+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7feab8155f90 con 0x7feab80b0570 2022-07-20T03:15:09.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.857+0000 7feaaa7fc700 1 -- 172.21.15.61:0/3418208081 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v86) v1 ==== 74+0+13230 (secure 0 0 0) 0x7feaac023090 con 0x7feab80b0570 2022-07-20T03:15:09.875 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:09.875 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":86,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:09.173412+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":16,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":68,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:09.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feaa00539b0 msgr2=0x7feaa0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feaa00539b0 0x7feaa0055e70 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7feab400b000 tx=0x7feab4008040).stop 2022-07-20T03:15:09.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 msgr2=0x7feab813ed00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:09.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 0x7feab813ed00 secure :-1 s=READY pgs=909 cs=0 l=1 rev1=1 rx=0x7feab8146970 tx=0x7feaac015000).stop 2022-07-20T03:15:09.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 shutdown_connections 2022-07-20T03:15:09.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feaa00539b0 0x7feaa0055e70 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feab81422c0 0x7feab813f1f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feab80b0570 0x7feab813ed00 unknown :-1 s=CLOSED pgs=909 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 --2- 172.21.15.61:0/3418208081 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feab80aea10 0x7feab81440c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:09.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 >> 172.21.15.61:0/3418208081 conn(0x7feab80b9360 msgr2=0x7feab80b85e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:09.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 shutdown_connections 2022-07-20T03:15:09.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:09.861+0000 7feabe77d700 1 -- 172.21.15.61:0/3418208081 wait complete. 2022-07-20T03:15:09.908 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:15:09.908 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 Processor -- start 2022-07-20T03:15:10.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- start start 2022-07-20T03:15:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f25dc0b9da0 0x7f25dc1468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc0ba180 con 0x7f25dc1419a0 2022-07-20T03:15:10.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25dc1461b0 con 0x7f25dc0b3cf0 2022-07-20T03:15:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25dc0ad5b0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25dbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f25dc0b9da0 0x7f25dc1468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e0a3e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f25dc1419a0 0x7f25dc141e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44584/0 (socket says 172.21.15.61:44584) 2022-07-20T03:15:10.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e0a3e700 1 -- 172.21.15.61:0/3324709075 learned_addr learned my addr 172.21.15.61:0/3324709075 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:10.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25dbfff700 1 -- 172.21.15.61:0/3324709075 >> v1:172.21.15.117:6789/0 conn(0x7f25dc0b3cf0 legacy=0x7f25dc142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25dbfff700 1 -- 172.21.15.61:0/3324709075 >> v1:172.21.15.61:6789/0 conn(0x7f25dc1419a0 legacy=0x7f25dc141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25dbfff700 1 -- 172.21.15.61:0/3324709075 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25dc146450 con 0x7f25dc0b9da0 2022-07-20T03:15:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25dbfff700 1 --2- 172.21.15.61:0/3324709075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc1468f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f25cc002f70 tx=0x7f25cc006df0).ready entity=mon.0 client_cookie=e22a9f3d2fd1f8c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25db7fe700 1 -- 172.21.15.61:0/3324709075 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25cc00d610 con 0x7f25dc0b9da0 2022-07-20T03:15:10.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25db7fe700 1 -- 172.21.15.61:0/3324709075 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25cc000e90 con 0x7f25dc0b9da0 2022-07-20T03:15:10.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25db7fe700 1 -- 172.21.15.61:0/3324709075 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25cc00d960 con 0x7f25dc0b9da0 2022-07-20T03:15:10.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- 172.21.15.61:0/3324709075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 msgr2=0x7f25dc1468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 --2- 172.21.15.61:0/3324709075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc1468f0 secure :-1 s=READY pgs=911 cs=0 l=1 rev1=1 rx=0x7f25cc002f70 tx=0x7f25cc006df0).stop 2022-07-20T03:15:10.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- 172.21.15.61:0/3324709075 shutdown_connections 2022-07-20T03:15:10.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 --2- 172.21.15.61:0/3324709075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc1468f0 unknown :-1 s=CLOSED pgs=911 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- 172.21.15.61:0/3324709075 >> 172.21.15.61:0/3324709075 conn(0x7f25dc0678c0 msgr2=0x7f25dc0b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- 172.21.15.61:0/3324709075 shutdown_connections 2022-07-20T03:15:10.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.029+0000 7f25e123f700 1 -- 172.21.15.61:0/3324709075 wait complete. 2022-07-20T03:15:10.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 Processor -- start 2022-07-20T03:15:10.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 -- start start 2022-07-20T03:15:10.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc0b3cf0 0x7f25dc155ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc0afcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25dc1419a0 0x7f25dc113390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc13f6e0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc1563e0 con 0x7f25dc0b3cf0 2022-07-20T03:15:10.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e123f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25dc0b2be0 con 0x7f25dc1419a0 2022-07-20T03:15:10.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc0afcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc0afcd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:47990/0 (socket says 172.21.15.61:47990) 2022-07-20T03:15:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 -- 172.21.15.61:0/2762724747 learned_addr learned my addr 172.21.15.61:0/2762724747 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:10.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25dbfff700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc0b3cf0 0x7f25dc155ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25e0a3e700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25dc1419a0 0x7f25dc113390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 -- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25dc1419a0 msgr2=0x7f25dc113390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25dc1419a0 0x7f25dc113390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 -- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc0b3cf0 msgr2=0x7f25dc155ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc0b3cf0 0x7f25dc155ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.033+0000 7f25d37fe700 1 -- 172.21.15.61:0/2762724747 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25cc00b180 con 0x7f25dc0b9da0 2022-07-20T03:15:10.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d37fe700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc0afcd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f25dc156520 tx=0x7f25c4007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25c4012400 con 0x7f25dc0b9da0 2022-07-20T03:15:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25c4012560 con 0x7f25dc0b9da0 2022-07-20T03:15:10.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f25dc0b26a0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f25c40128c0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f25dc0b01c0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f25dc0574e0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f25c4019030 con 0x7f25dc0b9da0 2022-07-20T03:15:10.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d9ffb700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25b4053ad0 0x7f25b4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f25c406aea0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.037+0000 7f25dbfff700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25b4053ad0 0x7f25b4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.041+0000 7f25dbfff700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25b4053ad0 0x7f25b4055f90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f25cc01c000 tx=0x7f25cc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.045+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f25c4042030 con 0x7f25dc0b9da0 2022-07-20T03:15:10.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.173+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f25dc151ff0 con 0x7f25dc0b9da0 2022-07-20T03:15:10.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.173+0000 7f25d9ffb700 1 -- 172.21.15.61:0/2762724747 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v86) v1 ==== 110+0+10 (secure 0 0 0) 0x7f25c4045650 con 0x7f25dc0b9da0 2022-07-20T03:15:10.179 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:10.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25b4053ad0 msgr2=0x7f25b4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25b4053ad0 0x7f25b4055f90 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f25cc01c000 tx=0x7f25cc015000).stop 2022-07-20T03:15:10.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 msgr2=0x7f25dc0afcd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc0afcd0 secure :-1 s=READY pgs=912 cs=0 l=1 rev1=1 rx=0x7f25dc156520 tx=0x7f25c4007c20).stop 2022-07-20T03:15:10.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 shutdown_connections 2022-07-20T03:15:10.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25b4053ad0 0x7f25b4055f90 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25dc1419a0 0x7f25dc113390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25dc0b9da0 0x7f25dc0afcd0 unknown :-1 s=CLOSED pgs=912 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 --2- 172.21.15.61:0/2762724747 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25dc0b3cf0 0x7f25dc155ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 >> 172.21.15.61:0/2762724747 conn(0x7f25dc0678c0 msgr2=0x7f25dc140110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 shutdown_connections 2022-07-20T03:15:10.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.177+0000 7f25e123f700 1 -- 172.21.15.61:0/2762724747 wait complete. 2022-07-20T03:15:10.195 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:10.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.313+0000 7fa6550d6700 1 Processor -- start 2022-07-20T03:15:10.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa6550d6700 1 -- start start 2022-07-20T03:15:10.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa6550d6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa65005cf40 0x7fa65009a910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa6550d6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa6500b2810 con 0x7fa650098c90 2022-07-20T03:15:10.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa6550d6700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa65013ff20 con 0x7fa6500b14a0 2022-07-20T03:15:10.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa6550d6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa65009a280 con 0x7fa65005cf40 2022-07-20T03:15:10.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64ed9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa65005cf40 0x7fa65009a910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64659c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa650098c90 0x7fa650142f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44596/0 (socket says 172.21.15.61:44596) 2022-07-20T03:15:10.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64659c700 1 -- 172.21.15.61:0/622892177 learned_addr learned my addr 172.21.15.61:0/622892177 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:10.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64ed9d700 1 -- 172.21.15.61:0/622892177 >> v1:172.21.15.117:6789/0 conn(0x7fa6500b14a0 legacy=0x7fa650143610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64e59c700 1 -- 172.21.15.61:0/622892177 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3556626604 0 0) 0x7fa6500b2810 con 0x7fa650098c90 2022-07-20T03:15:10.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64ed9d700 1 -- 172.21.15.61:0/622892177 >> v1:172.21.15.61:6789/0 conn(0x7fa650098c90 legacy=0x7fa650142f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64ed9d700 1 -- 172.21.15.61:0/622892177 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa6500b0f00 con 0x7fa65005cf40 2022-07-20T03:15:10.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64ed9d700 1 --2- 172.21.15.61:0/622892177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa65009a910 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7fa640002f70 tx=0x7fa640006df0).ready entity=mon.0 client_cookie=a649a0403e4bca42 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64e59c700 1 -- 172.21.15.61:0/622892177 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64000d610 con 0x7fa65005cf40 2022-07-20T03:15:10.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64e59c700 1 -- 172.21.15.61:0/622892177 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa640000e90 con 0x7fa65005cf40 2022-07-20T03:15:10.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.317+0000 7fa64e59c700 1 -- 172.21.15.61:0/622892177 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64000d960 con 0x7fa65005cf40 2022-07-20T03:15:10.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 -- 172.21.15.61:0/622892177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 msgr2=0x7fa65009a910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 --2- 172.21.15.61:0/622892177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa65009a910 secure :-1 s=READY pgs=914 cs=0 l=1 rev1=1 rx=0x7fa640002f70 tx=0x7fa640006df0).stop 2022-07-20T03:15:10.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 -- 172.21.15.61:0/622892177 shutdown_connections 2022-07-20T03:15:10.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 --2- 172.21.15.61:0/622892177 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa65009a910 unknown :-1 s=CLOSED pgs=914 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 -- 172.21.15.61:0/622892177 >> 172.21.15.61:0/622892177 conn(0x7fa6500b5b70 msgr2=0x7fa6500b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 -- 172.21.15.61:0/622892177 shutdown_connections 2022-07-20T03:15:10.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 -- 172.21.15.61:0/622892177 wait complete. 2022-07-20T03:15:10.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 Processor -- start 2022-07-20T03:15:10.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.321+0000 7fa6550d6700 1 -- start start 2022-07-20T03:15:10.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa650155b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa650098c90 0x7fa65005fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6500b14a0 0x7fa650066f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa65008f9a0 con 0x7fa65005cf40 2022-07-20T03:15:10.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa650066a90 con 0x7fa6500b14a0 2022-07-20T03:15:10.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa6500914b0 con 0x7fa650098c90 2022-07-20T03:15:10.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa650155b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa650155b30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48002/0 (socket says 172.21.15.61:48002) 2022-07-20T03:15:10.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 -- 172.21.15.61:0/2731559424 learned_addr learned my addr 172.21.15.61:0/2731559424 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:10.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64659c700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa650098c90 0x7fa65005fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64f59e700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6500b14a0 0x7fa650066f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 -- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa650098c90 msgr2=0x7fa65005fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa650098c90 0x7fa65005fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 -- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6500b14a0 msgr2=0x7fa650066f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6500b14a0 0x7fa650066f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 -- 172.21.15.61:0/2731559424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa64000b180 con 0x7fa65005cf40 2022-07-20T03:15:10.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64f59e700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6500b14a0 0x7fa650066f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:10.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64ed9d700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa650155b30 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7fa640002f70 tx=0x7fa640015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa64000bed0 con 0x7fa65005cf40 2022-07-20T03:15:10.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa64000de90 con 0x7fa65005cf40 2022-07-20T03:15:10.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa650091750 con 0x7fa65005cf40 2022-07-20T03:15:10.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa640011de0 con 0x7fa65005cf40 2022-07-20T03:15:10.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6500653e0 con 0x7fa65005cf40 2022-07-20T03:15:10.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.325+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa6500b1130 con 0x7fa65005cf40 2022-07-20T03:15:10.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.329+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa640025030 con 0x7fa65005cf40 2022-07-20T03:15:10.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.329+0000 7fa64cd99700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa634053a40 0x7fa634055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.329+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa64006cfa0 con 0x7fa65005cf40 2022-07-20T03:15:10.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.329+0000 7fa64659c700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa634053a40 0x7fa634055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.329+0000 7fa64659c700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa634053a40 0x7fa634055f00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fa63800d000 tx=0x7fa6380011a0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.333+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa640044230 con 0x7fa65005cf40 2022-07-20T03:15:10.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.461+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa6500574e0 con 0x7fa65005cf40 2022-07-20T03:15:10.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa64cd99700 1 -- 172.21.15.61:0/2731559424 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v86) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa640023020 con 0x7fa65005cf40 2022-07-20T03:15:10.467 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:10.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa634053a40 msgr2=0x7fa634055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa634053a40 0x7fa634055f00 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7fa63800d000 tx=0x7fa6380011a0).stop 2022-07-20T03:15:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 msgr2=0x7fa650155b30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa650155b30 secure :-1 s=READY pgs=915 cs=0 l=1 rev1=1 rx=0x7fa640002f70 tx=0x7fa640015000).stop 2022-07-20T03:15:10.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 shutdown_connections 2022-07-20T03:15:10.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa634053a40 0x7fa634055f00 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa6500b14a0 0x7fa650066f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa650098c90 0x7fa65005fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 --2- 172.21.15.61:0/2731559424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa65005cf40 0x7fa650155b30 unknown :-1 s=CLOSED pgs=915 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 >> 172.21.15.61:0/2731559424 conn(0x7fa6500b5b70 msgr2=0x7fa65009a830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 shutdown_connections 2022-07-20T03:15:10.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.465+0000 7fa6550d6700 1 -- 172.21.15.61:0/2731559424 wait complete. 2022-07-20T03:15:10.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:15:10.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 Processor -- start 2022-07-20T03:15:10.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- start start 2022-07-20T03:15:10.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b405c3f0 0x7f68b414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b405d480 con 0x7f68b40acef0 2022-07-20T03:15:10.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68b40b21e0 con 0x7f68b40b1e00 2022-07-20T03:15:10.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b405cf40 con 0x7f68b405c3f0 2022-07-20T03:15:10.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b37fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f68b40acef0 0x7f68b4146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44612/0 (socket says 172.21.15.61:44612) 2022-07-20T03:15:10.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b37fe700 1 -- 172.21.15.61:0/1307332827 learned_addr learned my addr 172.21.15.61:0/1307332827 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:10.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b27fc700 1 --2- 172.21.15.61:0/1307332827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b405c3f0 0x7f68b414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b1ffb700 1 -- 172.21.15.61:0/1307332827 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1424249657 0 0) 0x7f68b40b21e0 con 0x7f68b40b1e00 2022-07-20T03:15:10.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b1ffb700 1 -- 172.21.15.61:0/1307332827 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6894003260 con 0x7f68b40b1e00 2022-07-20T03:15:10.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b27fc700 1 -- 172.21.15.61:0/1307332827 >> v1:172.21.15.117:6789/0 conn(0x7f68b40b1e00 legacy=0x7f68b40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b27fc700 1 -- 172.21.15.61:0/1307332827 >> v1:172.21.15.61:6789/0 conn(0x7f68b40acef0 legacy=0x7f68b4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b1ffb700 1 -- 172.21.15.61:0/1307332827 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3942748741 0 0) 0x7f68b405d480 con 0x7f68b40acef0 2022-07-20T03:15:10.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b27fc700 1 -- 172.21.15.61:0/1307332827 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68b405d1e0 con 0x7f68b405c3f0 2022-07-20T03:15:10.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b27fc700 1 --2- 172.21.15.61:0/1307332827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b405c3f0 0x7f68b414a6d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f68ac0032f0 tx=0x7f68ac0091d0).ready entity=mon.2 client_cookie=651a68233cedb311 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b1ffb700 1 -- 172.21.15.61:0/1307332827 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68ac00b6c0 con 0x7f68b405c3f0 2022-07-20T03:15:10.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b1ffb700 1 -- 172.21.15.61:0/1307332827 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68ac00b820 con 0x7f68b405c3f0 2022-07-20T03:15:10.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b1ffb700 1 -- 172.21.15.61:0/1307332827 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68ac00b990 con 0x7f68b405c3f0 2022-07-20T03:15:10.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- 172.21.15.61:0/1307332827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b405c3f0 msgr2=0x7f68b414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 --2- 172.21.15.61:0/1307332827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b405c3f0 0x7f68b414a6d0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f68ac0032f0 tx=0x7f68ac0091d0).stop 2022-07-20T03:15:10.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- 172.21.15.61:0/1307332827 shutdown_connections 2022-07-20T03:15:10.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 --2- 172.21.15.61:0/1307332827 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b405c3f0 0x7f68b414a6d0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- 172.21.15.61:0/1307332827 >> 172.21.15.61:0/1307332827 conn(0x7f68b40611c0 msgr2=0x7f68b4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- 172.21.15.61:0/1307332827 shutdown_connections 2022-07-20T03:15:10.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.609+0000 7f68b9302700 1 -- 172.21.15.61:0/1307332827 wait complete. 2022-07-20T03:15:10.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 Processor -- start 2022-07-20T03:15:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 -- start start 2022-07-20T03:15:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68b40acef0 0x7f68b40b5af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b40b1e00 0x7f68b41405b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 0x7f68b40b4ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b414a400 con 0x7f68b4149f50 2022-07-20T03:15:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4149880 con 0x7f68b40acef0 2022-07-20T03:15:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b9302700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68b4149350 con 0x7f68b40b1e00 2022-07-20T03:15:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 0x7f68b40b4ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 0x7f68b40b4ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48020/0 (socket says 172.21.15.61:48020) 2022-07-20T03:15:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 -- 172.21.15.61:0/3495078581 learned_addr learned my addr 172.21.15.61:0/3495078581 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b27fc700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b40b1e00 0x7f68b41405b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b2ffd700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68b40acef0 0x7f68b40b5af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 -- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b40b1e00 msgr2=0x7f68b41405b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b40b1e00 0x7f68b41405b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 -- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68b40acef0 msgr2=0x7f68b40b5af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68b40acef0 0x7f68b40b5af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 -- 172.21.15.61:0/3495078581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68ac009470 con 0x7f68b4149f50 2022-07-20T03:15:10.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68b37fe700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 0x7f68b40b4ae0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f68b4149e30 tx=0x7f68a40079c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.613+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a400cdf0 con 0x7f68b4149f50 2022-07-20T03:15:10.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68a40070d0 con 0x7f68b4149f50 2022-07-20T03:15:10.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68b4149ab0 con 0x7f68b4149f50 2022-07-20T03:15:10.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68a4007430 con 0x7f68b4149f50 2022-07-20T03:15:10.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68b414a820 con 0x7f68b4149f50 2022-07-20T03:15:10.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68b41491b0 con 0x7f68b4149f50 2022-07-20T03:15:10.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f68a401a060 con 0x7f68b4149f50 2022-07-20T03:15:10.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68abfff700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6898053a60 0x7f6898055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f68a406b1b0 con 0x7f68b4149f50 2022-07-20T03:15:10.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68b2ffd700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6898053a60 0x7f6898055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.617+0000 7f68b2ffd700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6898053a60 0x7f6898055f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f689c00d000 tx=0x7f689c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.625+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68b41491b0 con 0x7f68b4149f50 2022-07-20T03:15:10.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.749+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f68b4149c40 con 0x7f68b4149f50 2022-07-20T03:15:10.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.749+0000 7f68abfff700 1 -- 172.21.15.61:0/3495078581 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v86) v1 ==== 84+0+143 (secure 0 0 0) 0x7f68a4018020 con 0x7f68b4149f50 2022-07-20T03:15:10.756 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:10.757 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:15:10.757 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:10.758 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:15:10.758 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:15:10.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:10.761 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:10.762 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:15:10.763 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:15:10.764 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:10.764 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:15:10.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6898053a60 msgr2=0x7f6898055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6898053a60 0x7f6898055f20 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f689c00d000 tx=0x7f689c007a20).stop 2022-07-20T03:15:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 msgr2=0x7f68b40b4ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 0x7f68b40b4ae0 secure :-1 s=READY pgs=917 cs=0 l=1 rev1=1 rx=0x7f68b4149e30 tx=0x7f68a40079c0).stop 2022-07-20T03:15:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 shutdown_connections 2022-07-20T03:15:10.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6898053a60 0x7f6898055f20 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68b4149f50 0x7f68b40b4ae0 unknown :-1 s=CLOSED pgs=917 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68b40b1e00 0x7f68b41405b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 --2- 172.21.15.61:0/3495078581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68b40acef0 0x7f68b40b5af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 >> 172.21.15.61:0/3495078581 conn(0x7f68b405c3f0 msgr2=0x7f68b40afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 shutdown_connections 2022-07-20T03:15:10.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.753+0000 7f68b9302700 1 -- 172.21.15.61:0/3495078581 wait complete. 2022-07-20T03:15:10.774 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:15:10.774 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:15:10.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2d07a700 1 Processor -- start 2022-07-20T03:15:10.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2d07a700 1 -- start start 2022-07-20T03:15:10.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2d07a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb280b1460 0x7fcb280aee30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2d07a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2813f970 con 0x7fcb280b1080 2022-07-20T03:15:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2d07a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcb2813fbe0 con 0x7fcb2813eb20 2022-07-20T03:15:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2d07a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2813fe80 con 0x7fcb280b1460 2022-07-20T03:15:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb27fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcb280b1080 0x7fcb2813e410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44622/0 (socket says 172.21.15.61:44622) 2022-07-20T03:15:10.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb27fff700 1 -- 172.21.15.61:0/172270542 learned_addr learned my addr 172.21.15.61:0/172270542 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2c879700 1 --2- 172.21.15.61:0/172270542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb280b1460 0x7fcb280aee30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb26ffd700 1 -- 172.21.15.61:0/172270542 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3648484763 0 0) 0x7fcb2813f970 con 0x7fcb280b1080 2022-07-20T03:15:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb26ffd700 1 -- 172.21.15.61:0/172270542 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb0c003260 con 0x7fcb280b1080 2022-07-20T03:15:10.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb26ffd700 1 -- 172.21.15.61:0/172270542 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1676185594 0 0) 0x7fcb2813fbe0 con 0x7fcb2813eb20 2022-07-20T03:15:10.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2c879700 1 -- 172.21.15.61:0/172270542 >> v1:172.21.15.117:6789/0 conn(0x7fcb2813eb20 legacy=0x7fcb280ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2c879700 1 -- 172.21.15.61:0/172270542 >> v1:172.21.15.61:6789/0 conn(0x7fcb280b1080 legacy=0x7fcb2813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2c879700 1 -- 172.21.15.61:0/172270542 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb280af320 con 0x7fcb280b1460 2022-07-20T03:15:10.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb2c879700 1 --2- 172.21.15.61:0/172270542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb280b1460 0x7fcb280aee30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fcb18002f70 tx=0x7fcb18006df0).ready entity=mon.2 client_cookie=6c1aaa2de4cc726e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.889+0000 7fcb26ffd700 1 -- 172.21.15.61:0/172270542 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1800d570 con 0x7fcb280b1460 2022-07-20T03:15:10.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb26ffd700 1 -- 172.21.15.61:0/172270542 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb18000e90 con 0x7fcb280b1460 2022-07-20T03:15:10.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb26ffd700 1 -- 172.21.15.61:0/172270542 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb1800d8c0 con 0x7fcb280b1460 2022-07-20T03:15:10.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- 172.21.15.61:0/172270542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb280b1460 msgr2=0x7fcb280aee30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/172270542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb280b1460 0x7fcb280aee30 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fcb18002f70 tx=0x7fcb18006df0).stop 2022-07-20T03:15:10.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- 172.21.15.61:0/172270542 shutdown_connections 2022-07-20T03:15:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/172270542 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb280b1460 0x7fcb280aee30 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- 172.21.15.61:0/172270542 >> 172.21.15.61:0/172270542 conn(0x7fcb28063480 msgr2=0x7fcb280b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- 172.21.15.61:0/172270542 shutdown_connections 2022-07-20T03:15:10.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- 172.21.15.61:0/172270542 wait complete. 2022-07-20T03:15:10.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 Processor -- start 2022-07-20T03:15:10.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- start start 2022-07-20T03:15:10.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb280b1080 0x7fcb28066040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 0x7fcb28151da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb2813eb20 0x7fcb28152290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb280655b0 con 0x7fcb280b1460 2022-07-20T03:15:10.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2813dae0 con 0x7fcb280b1080 2022-07-20T03:15:10.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2d07a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcb2813dd80 con 0x7fcb2813eb20 2022-07-20T03:15:10.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 0x7fcb28151da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 0x7fcb28151da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48034/0 (socket says 172.21.15.61:48034) 2022-07-20T03:15:10.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb27fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb280b1080 0x7fcb28066040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb2c879700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb2813eb20 0x7fcb28152290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 -- 172.21.15.61:0/1796519937 learned_addr learned my addr 172.21.15.61:0/1796519937 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:10.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 -- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb2813eb20 msgr2=0x7fcb28152290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb2813eb20 0x7fcb28152290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 -- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb280b1080 msgr2=0x7fcb28066040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb280b1080 0x7fcb28066040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb277fe700 1 -- 172.21.15.61:0/1796519937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb1800b180 con 0x7fcb280b1460 2022-07-20T03:15:10.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.893+0000 7fcb27fff700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb280b1080 0x7fcb28066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb277fe700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 0x7fcb28151da0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fcb10000ed0 tx=0x7fcb10007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb10007e90 con 0x7fcb280b1460 2022-07-20T03:15:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcb10012400 con 0x7fcb280b1460 2022-07-20T03:15:10.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb2813cc40 con 0x7fcb280b1460 2022-07-20T03:15:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcb10012760 con 0x7fcb280b1460 2022-07-20T03:15:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb2813d900 con 0x7fcb280b1460 2022-07-20T03:15:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb280574e0 con 0x7fcb280b1460 2022-07-20T03:15:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcb10019030 con 0x7fcb280b1460 2022-07-20T03:15:10.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb257fa700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb00053a60 0x7fcb00055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:10.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(86..86 src has 1..86) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcb1006ad30 con 0x7fcb280b1460 2022-07-20T03:15:10.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb27fff700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb00053a60 0x7fcb00055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:10.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.897+0000 7fcb27fff700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb00053a60 0x7fcb00055f20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fcb2000d000 tx=0x7fcb20009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:10.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:10.905+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcb10041fc0 con 0x7fcb280b1460 2022-07-20T03:15:11.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.025+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fcb2813d900 con 0x7fcb280b1460 2022-07-20T03:15:11.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.201+0000 7fcb257fa700 1 -- 172.21.15.61:0/1796519937 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v87) v1 ==== 133+0+0 (secure 0 0 0) 0x7fcb2813d900 con 0x7fcb280b1460 2022-07-20T03:15:11.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb00053a60 msgr2=0x7fcb00055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb00053a60 0x7fcb00055f20 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7fcb2000d000 tx=0x7fcb20009040).stop 2022-07-20T03:15:11.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 msgr2=0x7fcb28151da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 0x7fcb28151da0 secure :-1 s=READY pgs=919 cs=0 l=1 rev1=1 rx=0x7fcb10000ed0 tx=0x7fcb10007c20).stop 2022-07-20T03:15:11.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 shutdown_connections 2022-07-20T03:15:11.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb00053a60 0x7fcb00055f20 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcb2813eb20 0x7fcb28152290 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcb280b1460 0x7fcb28151da0 unknown :-1 s=CLOSED pgs=919 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 --2- 172.21.15.61:0/1796519937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcb280b1080 0x7fcb28066040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 >> 172.21.15.61:0/1796519937 conn(0x7fcb28063480 msgr2=0x7fcb280ae8c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:11.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 shutdown_connections 2022-07-20T03:15:11.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.205+0000 7fcb2d07a700 1 -- 172.21.15.61:0/1796519937 wait complete. 2022-07-20T03:15:11.213 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:15:11.222 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:15:11.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.337+0000 7f06a6ccd700 1 Processor -- start 2022-07-20T03:15:11.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.337+0000 7f06a6ccd700 1 -- start start 2022-07-20T03:15:11.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f06a0061d90 0x7f06a0143620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:11.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0067f30 con 0x7f06a00b2270 2022-07-20T03:15:11.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f06a0062170 con 0x7f06a00b1df0 2022-07-20T03:15:11.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f06a00abb10 con 0x7f06a0061d90 2022-07-20T03:15:11.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a5ccb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f06a0061d90 0x7f06a0143620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:11.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a64cc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f06a00b2270 0x7f06a00993b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44640/0 (socket says 172.21.15.61:44640) 2022-07-20T03:15:11.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a64cc700 1 -- 172.21.15.61:0/160344461 learned_addr learned my addr 172.21.15.61:0/160344461 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:11.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a4cc9700 1 -- 172.21.15.61:0/160344461 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2506828894 0 0) 0x7f06a0067f30 con 0x7f06a00b2270 2022-07-20T03:15:11.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a4cc9700 1 -- 172.21.15.61:0/160344461 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0688003260 con 0x7f06a00b2270 2022-07-20T03:15:11.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a4cc9700 1 -- 172.21.15.61:0/160344461 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 808013157 0 0) 0x7f06a0062170 con 0x7f06a00b1df0 2022-07-20T03:15:11.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a5ccb700 1 -- 172.21.15.61:0/160344461 >> v1:172.21.15.117:6789/0 conn(0x7f06a00b1df0 legacy=0x7f06a0143b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a5ccb700 1 -- 172.21.15.61:0/160344461 >> v1:172.21.15.61:6789/0 conn(0x7f06a00b2270 legacy=0x7f06a00993b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a5ccb700 1 -- 172.21.15.61:0/160344461 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f06a00abd90 con 0x7f06a0061d90 2022-07-20T03:15:11.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a5ccb700 1 --2- 172.21.15.61:0/160344461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0061d90 0x7f06a0143620 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f0698001b80 tx=0x7f069c006df0).ready entity=mon.0 client_cookie=2c04d22fb0f3c0ae server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:11.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a4cc9700 1 -- 172.21.15.61:0/160344461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06980019e0 con 0x7f06a0061d90 2022-07-20T03:15:11.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a4cc9700 1 -- 172.21.15.61:0/160344461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f069c000e90 con 0x7f06a0061d90 2022-07-20T03:15:11.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a4cc9700 1 -- 172.21.15.61:0/160344461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f069c011400 con 0x7f06a0061d90 2022-07-20T03:15:11.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- 172.21.15.61:0/160344461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0061d90 msgr2=0x7f06a0143620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/160344461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0061d90 0x7f06a0143620 secure :-1 s=READY pgs=921 cs=0 l=1 rev1=1 rx=0x7f0698001b80 tx=0x7f069c006df0).stop 2022-07-20T03:15:11.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- 172.21.15.61:0/160344461 shutdown_connections 2022-07-20T03:15:11.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/160344461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a0061d90 0x7f06a0143620 unknown :-1 s=CLOSED pgs=921 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- 172.21.15.61:0/160344461 >> 172.21.15.61:0/160344461 conn(0x7f06a00b61f0 msgr2=0x7f06a00b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:11.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- 172.21.15.61:0/160344461 shutdown_connections 2022-07-20T03:15:11.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.341+0000 7f06a6ccd700 1 -- 172.21.15.61:0/160344461 wait complete. 2022-07-20T03:15:11.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 Processor -- start 2022-07-20T03:15:11.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 -- start start 2022-07-20T03:15:11.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06a0061d90 0x7f06a00b4360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:11.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 0x7f06a0140f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:11.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f06a01424b0 0x7f06a0141470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:11.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a0098470 con 0x7f06a00b2270 2022-07-20T03:15:11.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a00ae0b0 con 0x7f06a0061d90 2022-07-20T03:15:11.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f06a00b03f0 con 0x7f06a01424b0 2022-07-20T03:15:11.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 0x7f06a0140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 0x7f06a0140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48046/0 (socket says 172.21.15.61:48046) 2022-07-20T03:15:11.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a5ccb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06a0061d90 0x7f06a00b4360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 -- 172.21.15.61:0/3988901751 learned_addr learned my addr 172.21.15.61:0/3988901751 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:11.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a64cc700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f06a01424b0 0x7f06a0141470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:11.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 -- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f06a01424b0 msgr2=0x7f06a0141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f06a01424b0 0x7f06a0141470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 -- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06a0061d90 msgr2=0x7f06a00b4360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:11.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06a0061d90 0x7f06a00b4360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:11.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 -- 172.21.15.61:0/3988901751 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f069c00b180 con 0x7f06a00b2270 2022-07-20T03:15:11.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a54ca700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 0x7f06a0140f80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f0694000ed0 tx=0x7f0694007a90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:11.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0694012400 con 0x7f06a00b2270 2022-07-20T03:15:11.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0694012560 con 0x7f06a00b2270 2022-07-20T03:15:11.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f06940128c0 con 0x7f06a00b2270 2022-07-20T03:15:11.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a01420b0 con 0x7f06a00b2270 2022-07-20T03:15:11.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.345+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a0141b00 con 0x7f06a00b2270 2022-07-20T03:15:11.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.349+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a00574e0 con 0x7f06a00b2270 2022-07-20T03:15:11.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.349+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0694019030 con 0x7f06a00b2270 2022-07-20T03:15:11.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.349+0000 7f0692ffd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0680053ad0 0x7f0680055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:11.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.349+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(87..87 src has 1..87) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f069406aef0 con 0x7f06a00b2270 2022-07-20T03:15:11.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.349+0000 7f06a5ccb700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0680053ad0 0x7f0680055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:11.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.349+0000 7f06a5ccb700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0680053ad0 0x7f0680055f90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f069c019000 tx=0x7f06a00ac030).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:11.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.357+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0694042070 con 0x7f06a00b2270 2022-07-20T03:15:11.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:11.485+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f06a00447e0 con 0x7f06a00b2270 2022-07-20T03:15:12.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.101+0000 7f0692ffd700 1 -- 172.21.15.61:0/3988901751 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v88) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0694045690 con 0x7f06a00b2270 2022-07-20T03:15:12.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.105+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0680053ad0 msgr2=0x7f0680055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.105+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0680053ad0 0x7f0680055f90 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f069c019000 tx=0x7f06a00ac030).stop 2022-07-20T03:15:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.105+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 msgr2=0x7f06a0140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.105+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 0x7f06a0140f80 secure :-1 s=READY pgs=922 cs=0 l=1 rev1=1 rx=0x7f0694000ed0 tx=0x7f0694007a90).stop 2022-07-20T03:15:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 shutdown_connections 2022-07-20T03:15:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0680053ad0 0x7f0680055f90 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f06a01424b0 0x7f06a0141470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f06a00b2270 0x7f06a0140f80 unknown :-1 s=CLOSED pgs=922 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 --2- 172.21.15.61:0/3988901751 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f06a0061d90 0x7f06a00b4360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 >> 172.21.15.61:0/3988901751 conn(0x7f06a00b61f0 msgr2=0x7f06a00b5900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:12.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 shutdown_connections 2022-07-20T03:15:12.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.113+0000 7f06a6ccd700 1 -- 172.21.15.61:0/3988901751 wait complete. 2022-07-20T03:15:12.123 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:15:12.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:15:12.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.269+0000 7fe1e1d3f700 1 Processor -- start 2022-07-20T03:15:12.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.269+0000 7fe1e1d3f700 1 -- start start 2022-07-20T03:15:12.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe1dc060770 0x7fe1dc060b50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc05bfa0 0x7fe1dc1438c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe1dc061980 con 0x7fe1dc0aa9d0 2022-07-20T03:15:12.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b2260 con 0x7fe1dc060770 2022-07-20T03:15:12.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0ab6a0 con 0x7fe1dc05bfa0 2022-07-20T03:15:12.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1d2ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe1dc0aa9d0 0x7fe1dc0aaf90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44650/0 (socket says 172.21.15.61:44650) 2022-07-20T03:15:12.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/665889920 learned_addr learned my addr 172.21.15.61:0/665889920 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:12.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1dbfff700 1 --2- 172.21.15.61:0/665889920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc05bfa0 0x7fe1dc1438c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1db7fe700 1 --2- 172.21.15.61:0/665889920 >> v2:172.21.15.117:3300/0 conn(0x7fe1dc060770 0x7fe1dc060b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1daffd700 1 -- 172.21.15.61:0/665889920 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864527970 0 0) 0x7fe1dc061980 con 0x7fe1dc0aa9d0 2022-07-20T03:15:12.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1daffd700 1 -- 172.21.15.61:0/665889920 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe1bc003260 con 0x7fe1dc0aa9d0 2022-07-20T03:15:12.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1dbfff700 1 -- 172.21.15.61:0/665889920 >> v2:172.21.15.117:3300/0 conn(0x7fe1dc060770 msgr2=0x7fe1dc060b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1dbfff700 1 --2- 172.21.15.61:0/665889920 >> v2:172.21.15.117:3300/0 conn(0x7fe1dc060770 0x7fe1dc060b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1dbfff700 1 -- 172.21.15.61:0/665889920 >> v1:172.21.15.61:6789/0 conn(0x7fe1dc0aa9d0 legacy=0x7fe1dc0aaf90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1dbfff700 1 -- 172.21.15.61:0/665889920 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1dc142d40 con 0x7fe1dc05bfa0 2022-07-20T03:15:12.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1dbfff700 1 --2- 172.21.15.61:0/665889920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc05bfa0 0x7fe1dc1438c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe1d40032f0 tx=0x7fe1d40091d0).ready entity=mon.2 client_cookie=29adc57b7849f640 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1daffd700 1 -- 172.21.15.61:0/665889920 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1d400b600 con 0x7fe1dc05bfa0 2022-07-20T03:15:12.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1daffd700 1 -- 172.21.15.61:0/665889920 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1d400b760 con 0x7fe1dc05bfa0 2022-07-20T03:15:12.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1daffd700 1 -- 172.21.15.61:0/665889920 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1d400b910 con 0x7fe1dc05bfa0 2022-07-20T03:15:12.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/665889920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc05bfa0 msgr2=0x7fe1dc1438c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/665889920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc05bfa0 0x7fe1dc1438c0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fe1d40032f0 tx=0x7fe1d40091d0).stop 2022-07-20T03:15:12.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/665889920 shutdown_connections 2022-07-20T03:15:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/665889920 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc05bfa0 0x7fe1dc1438c0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/665889920 >> v2:172.21.15.117:3300/0 conn(0x7fe1dc060770 0x7fe1dc060b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/665889920 >> 172.21.15.61:0/665889920 conn(0x7fe1dc0b5910 msgr2=0x7fe1dc0b5cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/665889920 shutdown_connections 2022-07-20T03:15:12.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.273+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/665889920 wait complete. 2022-07-20T03:15:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 Processor -- start 2022-07-20T03:15:12.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 -- start start 2022-07-20T03:15:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1dc05bfa0 0x7fe1dc0af0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc060770 0x7fe1dc0b0b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1dc0aa9d0 0x7fe1dc155560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc152e40 con 0x7fe1dc0aa9d0 2022-07-20T03:15:12.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b0450 con 0x7fe1dc05bfa0 2022-07-20T03:15:12.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1e1d3f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe1dc0b06f0 con 0x7fe1dc060770 2022-07-20T03:15:12.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1dc0aa9d0 0x7fe1dc155560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1d2ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc060770 0x7fe1dc0b0b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1dbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1dc0aa9d0 0x7fe1dc155560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48062/0 (socket says 172.21.15.61:48062) 2022-07-20T03:15:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1dbfff700 1 -- 172.21.15.61:0/3803919381 learned_addr learned my addr 172.21.15.61:0/3803919381 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.277+0000 7fe1db7fe700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1dc05bfa0 0x7fe1dc0af0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1dc05bfa0 msgr2=0x7fe1dc0af0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1dc05bfa0 0x7fe1dc0af0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/3803919381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1dc0aa9d0 msgr2=0x7fe1dc155560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1dc0aa9d0 0x7fe1dc155560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d2ffd700 1 -- 172.21.15.61:0/3803919381 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1d4009470 con 0x7fe1dc060770 2022-07-20T03:15:12.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d2ffd700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc060770 0x7fe1dc0b0b90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe1c4001110 tx=0x7fe1c4007a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1c40070d0 con 0x7fe1dc060770 2022-07-20T03:15:12.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe1c4007230 con 0x7fe1dc060770 2022-07-20T03:15:12.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1dc0649a0 con 0x7fe1dc060770 2022-07-20T03:15:12.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe1c4005610 con 0x7fe1dc060770 2022-07-20T03:15:12.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1dc152c00 con 0x7fe1dc060770 2022-07-20T03:15:12.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1dc0b13c0 con 0x7fe1dc060770 2022-07-20T03:15:12.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe1c401a060 con 0x7fe1dc060770 2022-07-20T03:15:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d97fa700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe1c0053a40 0x7fe1c0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.281+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(88..88 src has 1..88) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fe1c406b390 con 0x7fe1dc060770 2022-07-20T03:15:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.285+0000 7fe1db7fe700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe1c0053a40 0x7fe1c0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.285+0000 7fe1db7fe700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe1c0053a40 0x7fe1c0055f00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fe1cc00b000 tx=0x7fe1cc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.293+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe1dc0b13c0 con 0x7fe1dc060770 2022-07-20T03:15:12.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.457+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fe1dc152c00 con 0x7fe1dc060770 2022-07-20T03:15:12.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.473+0000 7fe1d97fa700 1 -- 172.21.15.61:0/3803919381 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 17 and data pool 18 v50) v1 ==== 189+0+0 (secure 0 0 0) 0x7fe1dc152c00 con 0x7fe1dc060770 2022-07-20T03:15:12.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe1c0053a40 msgr2=0x7fe1c0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe1c0053a40 0x7fe1c0055f00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fe1cc00b000 tx=0x7fe1cc006df0).stop 2022-07-20T03:15:12.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc060770 msgr2=0x7fe1dc0b0b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc060770 0x7fe1dc0b0b90 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fe1c4001110 tx=0x7fe1c4007a20).stop 2022-07-20T03:15:12.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 shutdown_connections 2022-07-20T03:15:12.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe1c0053a40 0x7fe1c0055f00 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe1dc0aa9d0 0x7fe1dc155560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe1dc060770 0x7fe1dc0b0b90 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 --2- 172.21.15.61:0/3803919381 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe1dc05bfa0 0x7fe1dc0af0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 >> 172.21.15.61:0/3803919381 conn(0x7fe1dc0b5910 msgr2=0x7fe1dc0b2ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:12.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 shutdown_connections 2022-07-20T03:15:12.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.477+0000 7fe1e1d3f700 1 -- 172.21.15.61:0/3803919381 wait complete. 2022-07-20T03:15:12.484 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 17 and data pool 18 2022-07-20T03:15:12.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:12.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.585+0000 7f6fa1fb2700 1 Processor -- start 2022-07-20T03:15:12.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.585+0000 7f6fa1fb2700 1 -- start start 2022-07-20T03:15:12.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6f9c0b61d0 0x7f6f9c1412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f9c05fb00 0x7f6f9c13feb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f9c0b65b0 con 0x7f6f9c060670 2022-07-20T03:15:12.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9c0b0f10 con 0x7f6f9c05fb00 2022-07-20T03:15:12.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f9c141790 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f9c05fb00 0x7f6f9c13feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6f9c05fb00 0x7f6f9c13feb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48078/0 (socket says 172.21.15.61:48078) 2022-07-20T03:15:12.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 -- 172.21.15.61:0/1499214736 learned_addr learned my addr 172.21.15.61:0/1499214736 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:12.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa0fb0700 1 --2- 172.21.15.61:0/1499214736 >> v2:172.21.15.117:3300/0 conn(0x7f6f9c0b61d0 0x7f6f9c1412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 -- 172.21.15.61:0/1499214736 >> v2:172.21.15.117:3300/0 conn(0x7f6f9c0b61d0 msgr2=0x7f6f9c1412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 --2- 172.21.15.61:0/1499214736 >> v2:172.21.15.117:3300/0 conn(0x7f6f9c0b61d0 0x7f6f9c1412a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 -- 172.21.15.61:0/1499214736 >> v1:172.21.15.117:6789/0 conn(0x7f6f9c060670 legacy=0x7f6f9c143e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 -- 172.21.15.61:0/1499214736 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f9c140b60 con 0x7f6f9c05fb00 2022-07-20T03:15:12.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa17b1700 1 --2- 172.21.15.61:0/1499214736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c13feb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f6f9c05d590 tx=0x7f6f94009290).ready entity=mon.0 client_cookie=99107fceb02e4763 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6f9b7fe700 1 -- 172.21.15.61:0/1499214736 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f9400b830 con 0x7f6f9c05fb00 2022-07-20T03:15:12.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6f9b7fe700 1 -- 172.21.15.61:0/1499214736 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f9400b990 con 0x7f6f9c05fb00 2022-07-20T03:15:12.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6f9b7fe700 1 -- 172.21.15.61:0/1499214736 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f9400bb40 con 0x7f6f9c05fb00 2022-07-20T03:15:12.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/1499214736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c05fb00 msgr2=0x7f6f9c13feb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/1499214736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c13feb0 secure :-1 s=READY pgs=924 cs=0 l=1 rev1=1 rx=0x7f6f9c05d590 tx=0x7f6f94009290).stop 2022-07-20T03:15:12.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/1499214736 shutdown_connections 2022-07-20T03:15:12.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/1499214736 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c13feb0 unknown :-1 s=CLOSED pgs=924 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/1499214736 >> v2:172.21.15.117:3300/0 conn(0x7f6f9c0b61d0 0x7f6f9c1412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/1499214736 >> 172.21.15.61:0/1499214736 conn(0x7f6f9c063cf0 msgr2=0x7f6f9c0b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:12.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/1499214736 shutdown_connections 2022-07-20T03:15:12.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.589+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/1499214736 wait complete. 2022-07-20T03:15:12.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 Processor -- start 2022-07-20T03:15:12.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- start start 2022-07-20T03:15:12.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c114ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f9c060670 0x7f6f9c14fb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 0x7f6f9c110940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c150080 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c111170 con 0x7f6f9c05fb00 2022-07-20T03:15:12.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f9c111410 con 0x7f6f9c060670 2022-07-20T03:15:12.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 0x7f6f9c110940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 0x7f6f9c110940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48084/0 (socket says 172.21.15.61:48084) 2022-07-20T03:15:12.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 -- 172.21.15.61:0/259416310 learned_addr learned my addr 172.21.15.61:0/259416310 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:12.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6f9bfff700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f9c060670 0x7f6f9c14fb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa0fb0700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c114ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 -- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f9c060670 msgr2=0x7f6f9c14fb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f9c060670 0x7f6f9c14fb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 -- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f9c05fb00 msgr2=0x7f6f9c114ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c114ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 -- 172.21.15.61:0/259416310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94009530 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa17b1700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 0x7f6f9c110940 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f6f9c05d590 tx=0x7f6f94016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f940032f0 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f94021440 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f9c153190 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f940217a0 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f9c13e650 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.593+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f9c153320 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.597+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6f94027020 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.597+0000 7f6f99ffb700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f7c0539b0 0x7f6f7c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.597+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6f9406da60 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.597+0000 7f6fa0fb0700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f7c0539b0 0x7f6f7c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.597+0000 7f6fa0fb0700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f7c0539b0 0x7f6f7c055e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6f9000c000 tx=0x7f6f90008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.605+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f940440d0 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.729+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f6f9c140e00 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.733+0000 7f6f99ffb700 1 -- 172.21.15.61:0/259416310 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f6f94015070 con 0x7f6f9c0b61d0 2022-07-20T03:15:12.737 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:12.738 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":89,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:12.462193+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:11.081553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:11.541101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:12.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.733+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f7c0539b0 msgr2=0x7f6f7c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.733+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f7c0539b0 0x7f6f7c055e70 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f6f9000c000 tx=0x7f6f90008040).stop 2022-07-20T03:15:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.733+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 msgr2=0x7f6f9c110940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.733+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 0x7f6f9c110940 secure :-1 s=READY pgs=925 cs=0 l=1 rev1=1 rx=0x7f6f9c05d590 tx=0x7f6f94016000).stop 2022-07-20T03:15:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 shutdown_connections 2022-07-20T03:15:12.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f7c0539b0 0x7f6f7c055e70 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f9c0b61d0 0x7f6f9c110940 unknown :-1 s=CLOSED pgs=925 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f9c060670 0x7f6f9c14fb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 --2- 172.21.15.61:0/259416310 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f9c05fb00 0x7f6f9c114ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 >> 172.21.15.61:0/259416310 conn(0x7f6f9c063cf0 msgr2=0x7f6f9c0b5760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:12.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 shutdown_connections 2022-07-20T03:15:12.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.737+0000 7f6fa1fb2700 1 -- 172.21.15.61:0/259416310 wait complete. 2022-07-20T03:15:12.752 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:12.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.869+0000 7f85f3fff700 1 Processor -- start 2022-07-20T03:15:12.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.869+0000 7f85f3fff700 1 -- start start 2022-07-20T03:15:12.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f85f40adff0 0x7f85f40b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85f405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f85f4147010 con 0x7f85f40b5650 2022-07-20T03:15:12.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85f41472b0 con 0x7f85f40adff0 2022-07-20T03:15:12.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4146d40 con 0x7f85f40af8e0 2022-07-20T03:15:12.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f85f40b5650 0x7f85f4147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44686/0 (socket says 172.21.15.61:44686) 2022-07-20T03:15:12.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 -- 172.21.15.61:0/2642812946 learned_addr learned my addr 172.21.15.61:0/2642812946 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:12.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f27fc700 1 --2- 172.21.15.61:0/2642812946 >> v2:172.21.15.61:3300/0 conn(0x7f85f40adff0 0x7f85f40b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f37fe700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85f405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3377600447 0 0) 0x7f85f4147010 con 0x7f85f40b5650 2022-07-20T03:15:12.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f85d8003260 con 0x7f85f40b5650 2022-07-20T03:15:12.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 168243964 0 0) 0x7f85d8003260 con 0x7f85f40b5650 2022-07-20T03:15:12.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 msgr2=0x7f85f405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85f405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 >> v2:172.21.15.61:3300/0 conn(0x7f85f40adff0 msgr2=0x7f85f40b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 --2- 172.21.15.61:0/2642812946 >> v2:172.21.15.61:3300/0 conn(0x7f85f40adff0 0x7f85f40b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f40acae0 con 0x7f85f40b5650 2022-07-20T03:15:12.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f85ec0019e0 con 0x7f85f40b5650 2022-07-20T03:15:12.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f27fc700 1 --2- 172.21.15.61:0/2642812946 >> v2:172.21.15.61:3300/0 conn(0x7f85f40adff0 0x7f85f40b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:12.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 >> v1:172.21.15.61:6789/0 conn(0x7f85f40b5650 legacy=0x7f85f4147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85d8003750 0x7f85d8003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8004500 0x7f85d8004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85d8004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f41472b0 con 0x7f85d8003750 2022-07-20T03:15:12.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4146d40 con 0x7f85d8004500 2022-07-20T03:15:12.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85d800b750 con 0x7f85f40af8e0 2022-07-20T03:15:12.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f37fe700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85d8004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85d8003750 0x7f85d8003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f27fc700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8004500 0x7f85d8004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 -- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 msgr2=0x7f85d8004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85d8004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 -- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8004500 msgr2=0x7f85d8004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:12.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8004500 0x7f85d8004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 -- 172.21.15.61:0/2642812946 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85d8003260 con 0x7f85d8003750 2022-07-20T03:15:12.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f37fe700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f40af8e0 0x7f85d8004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:12.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f2ffd700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85d8003750 0x7f85d8003b70 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f85ec0016f0 tx=0x7f85ec0021d0).ready entity=mon.0 client_cookie=feb50e8ac6cb2c95 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85ec00cc80 con 0x7f85d8003750 2022-07-20T03:15:12.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85ec00cde0 con 0x7f85d8003750 2022-07-20T03:15:12.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f1ffb700 1 -- 172.21.15.61:0/2642812946 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85ec005b80 con 0x7f85d8003750 2022-07-20T03:15:12.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- 172.21.15.61:0/2642812946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85d8003750 msgr2=0x7f85d8003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2642812946 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85d8003750 0x7f85d8003b70 secure :-1 s=READY pgs=927 cs=0 l=1 rev1=1 rx=0x7f85ec0016f0 tx=0x7f85ec0021d0).stop 2022-07-20T03:15:12.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f37fe700 1 -- 172.21.15.61:0/2642812946 reap_dead start 2022-07-20T03:15:12.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- 172.21.15.61:0/2642812946 shutdown_connections 2022-07-20T03:15:12.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- 172.21.15.61:0/2642812946 >> 172.21.15.61:0/2642812946 conn(0x7f85f40634e0 msgr2=0x7f85f40b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:12.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- 172.21.15.61:0/2642812946 shutdown_connections 2022-07-20T03:15:12.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.873+0000 7f85f3fff700 1 -- 172.21.15.61:0/2642812946 wait complete. 2022-07-20T03:15:12.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 Processor -- start 2022-07-20T03:15:12.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 -- start start 2022-07-20T03:15:12.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8003750 0x7f85f4067570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 0x7f85f4066730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f41599e0 0x7f85f4159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4065840 con 0x7f85f4156fe0 2022-07-20T03:15:12.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4111a60 con 0x7f85d8003750 2022-07-20T03:15:12.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85f4066fc0 con 0x7f85f41599e0 2022-07-20T03:15:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 0x7f85f4066730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 0x7f85f4066730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48102/0 (socket says 172.21.15.61:48102) 2022-07-20T03:15:12.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f2ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8003750 0x7f85f4067570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 -- 172.21.15.61:0/2640234773 learned_addr learned my addr 172.21.15.61:0/2640234773 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:12.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f37fe700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f41599e0 0x7f85f4159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 -- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f41599e0 msgr2=0x7f85f4159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f41599e0 0x7f85f4159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 -- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8003750 msgr2=0x7f85f4067570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:12.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8003750 0x7f85f4067570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 -- 172.21.15.61:0/2640234773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4067260 con 0x7f85f4156fe0 2022-07-20T03:15:12.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f27fc700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 0x7f85f4066730 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f85e8008000 tx=0x7f85e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85e8011e60 con 0x7f85f4156fe0 2022-07-20T03:15:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85e800f940 con 0x7f85f4156fe0 2022-07-20T03:15:12.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85f41112e0 con 0x7f85f4156fe0 2022-07-20T03:15:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.877+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85e800fab0 con 0x7f85f4156fe0 2022-07-20T03:15:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.881+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85f4111670 con 0x7f85f4156fe0 2022-07-20T03:15:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.881+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85f405bd90 con 0x7f85f4156fe0 2022-07-20T03:15:12.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.881+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f85e8024070 con 0x7f85f4156fe0 2022-07-20T03:15:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.881+0000 7f85e3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85d00539b0 0x7f85d0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.881+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f85e806c730 con 0x7f85f4156fe0 2022-07-20T03:15:12.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.881+0000 7f85f2ffd700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85d00539b0 0x7f85d0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.885+0000 7f85f2ffd700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85d00539b0 0x7f85d0055e70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f85ec01b000 tx=0x7f85ec015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:12.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:12.889+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85e8043690 con 0x7f85f4156fe0 2022-07-20T03:15:13.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.013+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f85f40574e0 con 0x7f85f4156fe0 2022-07-20T03:15:13.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85e3fff700 1 -- 172.21.15.61:0/2640234773 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v89) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f85e801b310 con 0x7f85f4156fe0 2022-07-20T03:15:13.021 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:13.021 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":89,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:12.462193+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:11.081553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:11.541101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:13.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85d00539b0 msgr2=0x7f85d0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85d00539b0 0x7f85d0055e70 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f85ec01b000 tx=0x7f85ec015000).stop 2022-07-20T03:15:13.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 msgr2=0x7f85f4066730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 0x7f85f4066730 secure :-1 s=READY pgs=928 cs=0 l=1 rev1=1 rx=0x7f85e8008000 tx=0x7f85e8008040).stop 2022-07-20T03:15:13.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 shutdown_connections 2022-07-20T03:15:13.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85d00539b0 0x7f85d0055e70 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85f41599e0 0x7f85f4159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85f4156fe0 0x7f85f4066730 unknown :-1 s=CLOSED pgs=928 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 --2- 172.21.15.61:0/2640234773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85d8003750 0x7f85f4067570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 >> 172.21.15.61:0/2640234773 conn(0x7f85f40634e0 msgr2=0x7f85f414a250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:13.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 shutdown_connections 2022-07-20T03:15:13.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.017+0000 7f85f3fff700 1 -- 172.21.15.61:0/2640234773 wait complete. 2022-07-20T03:15:13.052 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:15:13.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 Processor -- start 2022-07-20T03:15:13.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 -- start start 2022-07-20T03:15:13.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100b59f0 0x7fca1005c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca100af940 0x7fca1005c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fca1005d2f0 con 0x7fca100addd0 2022-07-20T03:15:13.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fca10146350 con 0x7fca100af940 2022-07-20T03:15:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca100b5dd0 con 0x7fca100b59f0 2022-07-20T03:15:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca100af940 0x7fca1005c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fca100af940 0x7fca1005c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48110/0 (socket says 172.21.15.61:48110) 2022-07-20T03:15:13.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 -- 172.21.15.61:0/1844855541 learned_addr learned my addr 172.21.15.61:0/1844855541 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:13.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fca100addd0 0x7fca10141cf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44710/0 (socket says 172.21.15.61:44710) 2022-07-20T03:15:13.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0f7fe700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100b59f0 0x7fca1005c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 -- 172.21.15.61:0/1844855541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100b59f0 msgr2=0x7fca1005c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100b59f0 0x7fca1005c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 -- 172.21.15.61:0/1844855541 >> v1:172.21.15.61:6789/0 conn(0x7fca100addd0 legacy=0x7fca10141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 -- 172.21.15.61:0/1844855541 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca101460b0 con 0x7fca100af940 2022-07-20T03:15:13.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0f7fe700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100b59f0 0x7fca1005c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:13.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0effd700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100af940 0x7fca1005c5c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7fca080032f0 tx=0x7fca08009290).ready entity=mon.0 client_cookie=418c3fa4371c9b3d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0e7fc700 1 -- 172.21.15.61:0/1844855541 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca0800b8a0 con 0x7fca100af940 2022-07-20T03:15:13.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0e7fc700 1 -- 172.21.15.61:0/1844855541 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca0800ba00 con 0x7fca100af940 2022-07-20T03:15:13.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca0e7fc700 1 -- 172.21.15.61:0/1844855541 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca0800bb70 con 0x7fca100af940 2022-07-20T03:15:13.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 -- 172.21.15.61:0/1844855541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100af940 msgr2=0x7fca1005c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.169+0000 7fca14f3c700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100af940 0x7fca1005c5c0 secure :-1 s=READY pgs=930 cs=0 l=1 rev1=1 rx=0x7fca080032f0 tx=0x7fca08009290).stop 2022-07-20T03:15:13.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- 172.21.15.61:0/1844855541 shutdown_connections 2022-07-20T03:15:13.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100af940 0x7fca1005c5c0 unknown :-1 s=CLOSED pgs=930 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 --2- 172.21.15.61:0/1844855541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100b59f0 0x7fca1005c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- 172.21.15.61:0/1844855541 >> 172.21.15.61:0/1844855541 conn(0x7fca10063480 msgr2=0x7fca100b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:13.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- 172.21.15.61:0/1844855541 shutdown_connections 2022-07-20T03:15:13.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- 172.21.15.61:0/1844855541 wait complete. 2022-07-20T03:15:13.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 Processor -- start 2022-07-20T03:15:13.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- start start 2022-07-20T03:15:13.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca100addd0 0x7fca10067510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100af940 0x7fca10064c10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 0x7fca100662e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca1013e960 con 0x7fca100b59f0 2022-07-20T03:15:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fca10152ce0 con 0x7fca100addd0 2022-07-20T03:15:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca14f3c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fca10066dd0 con 0x7fca100af940 2022-07-20T03:15:13.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.173+0000 7fca0ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 0x7fca100662e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 0x7fca100662e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48118/0 (socket says 172.21.15.61:48118) 2022-07-20T03:15:13.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 -- 172.21.15.61:0/416142324 learned_addr learned my addr 172.21.15.61:0/416142324 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:13.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0f7fe700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca100addd0 0x7fca10067510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0effd700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100af940 0x7fca10064c10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 -- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100af940 msgr2=0x7fca10064c10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100af940 0x7fca10064c10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 -- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca100addd0 msgr2=0x7fca10067510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca100addd0 0x7fca10067510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 -- 172.21.15.61:0/416142324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fca08009530 con 0x7fca100b59f0 2022-07-20T03:15:13.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0ffff700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 0x7fca100662e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fca10062640 tx=0x7fca04007b90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca040070d0 con 0x7fca100b59f0 2022-07-20T03:15:13.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fca04007230 con 0x7fca100b59f0 2022-07-20T03:15:13.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fca10067070 con 0x7fca100b59f0 2022-07-20T03:15:13.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fca04005740 con 0x7fca100b59f0 2022-07-20T03:15:13.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fca10065f60 con 0x7fca100b59f0 2022-07-20T03:15:13.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.177+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fca10067200 con 0x7fca100b59f0 2022-07-20T03:15:13.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.181+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fca0401a070 con 0x7fca100b59f0 2022-07-20T03:15:13.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.181+0000 7fca0cff9700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9f00539b0 0x7fc9f0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.181+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(89..89 src has 1..89) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fca0406afc0 con 0x7fca100b59f0 2022-07-20T03:15:13.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.181+0000 7fca0f7fe700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9f00539b0 0x7fc9f0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.181+0000 7fca0f7fe700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9f00539b0 0x7fc9f0055e70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fca0000c000 tx=0x7fca00008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.185+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fca04041f20 con 0x7fca100b59f0 2022-07-20T03:15:13.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.349+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fca1005cfe0 con 0x7fca100b59f0 2022-07-20T03:15:13.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.481+0000 7fca0cff9700 1 -- 172.21.15.61:0/416142324 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v51) v1 ==== 118+0+0 (secure 0 0 0) 0x7fca04045540 con 0x7fca100b59f0 2022-07-20T03:15:13.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9f00539b0 msgr2=0x7fc9f0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9f00539b0 0x7fc9f0055e70 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7fca0000c000 tx=0x7fca00008040).stop 2022-07-20T03:15:13.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 msgr2=0x7fca100662e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 0x7fca100662e0 secure :-1 s=READY pgs=931 cs=0 l=1 rev1=1 rx=0x7fca10062640 tx=0x7fca04007b90).stop 2022-07-20T03:15:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 shutdown_connections 2022-07-20T03:15:13.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc9f00539b0 0x7fc9f0055e70 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fca100b59f0 0x7fca100662e0 unknown :-1 s=CLOSED pgs=931 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fca100af940 0x7fca10064c10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 --2- 172.21.15.61:0/416142324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fca100addd0 0x7fca10067510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 >> 172.21.15.61:0/416142324 conn(0x7fca10063480 msgr2=0x7fca1005c0d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:13.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 shutdown_connections 2022-07-20T03:15:13.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.485+0000 7fca14f3c700 1 -- 172.21.15.61:0/416142324 wait complete. 2022-07-20T03:15:13.504 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:13.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31aae9700 1 Processor -- start 2022-07-20T03:15:13.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31aae9700 1 -- start start 2022-07-20T03:15:13.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31aae9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 0x7fd3140b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31aae9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd314147010 con 0x7fd3140adff0 2022-07-20T03:15:13.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31aae9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3141472b0 con 0x7fd3140af8e0 2022-07-20T03:15:13.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31aae9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd314146d40 con 0x7fd3140b5650 2022-07-20T03:15:13.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd3192e6700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd3140adff0 0x7fd314147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44722/0 (socket says 172.21.15.61:44722) 2022-07-20T03:15:13.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd3192e6700 1 -- 172.21.15.61:0/1225510660 learned_addr learned my addr 172.21.15.61:0/1225510660 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:13.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 0x7fd3140b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3633000761 0 0) 0x7fd314147010 con 0x7fd3140adff0 2022-07-20T03:15:13.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd300003260 con 0x7fd3140adff0 2022-07-20T03:15:13.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2982857153 0 0) 0x7fd3141472b0 con 0x7fd3140af8e0 2022-07-20T03:15:13.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd314147010 con 0x7fd3140af8e0 2022-07-20T03:15:13.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2902330650 0 0) 0x7fd300003260 con 0x7fd3140adff0 2022-07-20T03:15:13.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 msgr2=0x7fd3140b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:13.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 0x7fd3140b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 >> v1:172.21.15.117:6789/0 conn(0x7fd3140af8e0 legacy=0x7fd31405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd31405c4a0 con 0x7fd3140adff0 2022-07-20T03:15:13.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fd30c001b60 con 0x7fd3140adff0 2022-07-20T03:15:13.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 >> v1:172.21.15.61:6789/0 conn(0x7fd3140adff0 legacy=0x7fd314147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd300003910 0x7fd300003d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 0x7fd300004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3000049f0 0x7fd300004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd314146d40 con 0x7fd3000049f0 2022-07-20T03:15:13.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd30000b780 con 0x7fd300003910 2022-07-20T03:15:13.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd30000b940 con 0x7fd3140b5650 2022-07-20T03:15:13.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd300003910 0x7fd300003d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd3192e6700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 0x7fd300004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31a2e8700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3000049f0 0x7fd300004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 -- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 msgr2=0x7fd300004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:13.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140b5650 0x7fd300004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 -- 172.21.15.61:0/1225510660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3000049f0 msgr2=0x7fd300004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:13.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3000049f0 0x7fd300004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 -- 172.21.15.61:0/1225510660 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd300003260 con 0x7fd300003910 2022-07-20T03:15:13.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd31a2e8700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3000049f0 0x7fd300004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:13.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.613+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd300003910 0x7fd300003d10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fd30400c000 tx=0x7fd304006df0).ready entity=mon.1 client_cookie=b624ee2d8c90e9ad server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30400e830 con 0x7fd300003910 2022-07-20T03:15:13.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd30400e990 con 0x7fd300003910 2022-07-20T03:15:13.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd318ae5700 1 -- 172.21.15.61:0/1225510660 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30400ecf0 con 0x7fd300003910 2022-07-20T03:15:13.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd300003910 msgr2=0x7fd300003d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1225510660 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd300003910 0x7fd300003d10 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fd30400c000 tx=0x7fd304006df0).stop 2022-07-20T03:15:13.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31a2e8700 1 -- 172.21.15.61:0/1225510660 reap_dead start 2022-07-20T03:15:13.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- 172.21.15.61:0/1225510660 shutdown_connections 2022-07-20T03:15:13.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- 172.21.15.61:0/1225510660 >> 172.21.15.61:0/1225510660 conn(0x7fd3140634e0 msgr2=0x7fd3140b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:13.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- 172.21.15.61:0/1225510660 shutdown_connections 2022-07-20T03:15:13.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- 172.21.15.61:0/1225510660 wait complete. 2022-07-20T03:15:13.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 Processor -- start 2022-07-20T03:15:13.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- start start 2022-07-20T03:15:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 0x7fd314063c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140675b0 0x7fd31411aef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd314156280 0x7fd314159e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd314064620 con 0x7fd300003910 2022-07-20T03:15:13.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd314064160 con 0x7fd314156280 2022-07-20T03:15:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31aae9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd314144ed0 con 0x7fd3140675b0 2022-07-20T03:15:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd319ae7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 0x7fd314063c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd319ae7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 0x7fd314063c70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48134/0 (socket says 172.21.15.61:48134) 2022-07-20T03:15:13.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd319ae7700 1 -- 172.21.15.61:0/1125868369 learned_addr learned my addr 172.21.15.61:0/1125868369 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:13.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd31a2e8700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd314156280 0x7fd314159e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.617+0000 7fd3192e6700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140675b0 0x7fd31411aef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd319ae7700 1 -- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140675b0 msgr2=0x7fd31411aef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140675b0 0x7fd31411aef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd319ae7700 1 -- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd314156280 msgr2=0x7fd314159e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd314156280 0x7fd314159e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd319ae7700 1 -- 172.21.15.61:0/1125868369 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd314159a10 con 0x7fd300003910 2022-07-20T03:15:13.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd319ae7700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 0x7fd314063c70 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fd3140642a0 tx=0x7fd304019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30401b820 con 0x7fd300003910 2022-07-20T03:15:13.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd304000e90 con 0x7fd300003910 2022-07-20T03:15:13.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd314064380 con 0x7fd300003910 2022-07-20T03:15:13.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd30401bb70 con 0x7fd300003910 2022-07-20T03:15:13.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd314156700 con 0x7fd300003910 2022-07-20T03:15:13.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd31415a6a0 con 0x7fd300003910 2022-07-20T03:15:13.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd30402d070 con 0x7fd300003910 2022-07-20T03:15:13.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd30affd700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2f4053a40 0x7fd2f4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd30406d730 con 0x7fd300003910 2022-07-20T03:15:13.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.621+0000 7fd3192e6700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2f4053a40 0x7fd2f4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.625+0000 7fd3192e6700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2f4053a40 0x7fd2f4055f00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fd30c009000 tx=0x7fd30c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.629+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd3040480d0 con 0x7fd300003910 2022-07-20T03:15:13.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.789+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd314159ba0 con 0x7fd300003910 2022-07-20T03:15:13.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.789+0000 7fd30affd700 1 -- 172.21.15.61:0/1125868369 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fd304047cc0 con 0x7fd300003910 2022-07-20T03:15:13.796 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:13.797 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:13.467553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7008":{"gid":7008,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1014823584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1014823584},{"type":"v1","addr":"172.21.15.61:6833","nonce":1014823584}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:13.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2f4053a40 msgr2=0x7fd2f4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2f4053a40 0x7fd2f4055f00 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7fd30c009000 tx=0x7fd30c009040).stop 2022-07-20T03:15:13.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 msgr2=0x7fd314063c70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 0x7fd314063c70 secure :-1 s=READY pgs=933 cs=0 l=1 rev1=1 rx=0x7fd3140642a0 tx=0x7fd304019000).stop 2022-07-20T03:15:13.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 shutdown_connections 2022-07-20T03:15:13.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2f4053a40 0x7fd2f4055f00 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd314156280 0x7fd314159e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3140675b0 0x7fd31411aef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 --2- 172.21.15.61:0/1125868369 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd300003910 0x7fd314063c70 unknown :-1 s=CLOSED pgs=933 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 >> 172.21.15.61:0/1125868369 conn(0x7fd3140634e0 msgr2=0x7fd3141458d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:13.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 shutdown_connections 2022-07-20T03:15:13.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.793+0000 7fd31aae9700 1 -- 172.21.15.61:0/1125868369 wait complete. 2022-07-20T03:15:13.807 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-20T03:15:13.810 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:13.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f6a262700 1 Processor -- start 2022-07-20T03:15:13.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f6a262700 1 -- start start 2022-07-20T03:15:13.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f6a262700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4f640b9ef0 0x7f4f640b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f6a262700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f640ba2d0 con 0x7f4f6405d160 2022-07-20T03:15:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f6a262700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4f640b4560 con 0x7f4f640b2960 2022-07-20T03:15:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f6a262700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4f640b4800 con 0x7f4f640b9ef0 2022-07-20T03:15:13.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f637fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4f640b9ef0 0x7f4f640b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f62ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4f6405d160 0x7f4f640b2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44740/0 (socket says 172.21.15.61:44740) 2022-07-20T03:15:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.909+0000 7f4f62ffd700 1 -- 172.21.15.61:0/4166706735 learned_addr learned my addr 172.21.15.61:0/4166706735 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f627fc700 1 -- 172.21.15.61:0/4166706735 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 734415625 0 0) 0x7f4f640ba2d0 con 0x7f4f6405d160 2022-07-20T03:15:13.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f627fc700 1 -- 172.21.15.61:0/4166706735 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4f50003260 con 0x7f4f6405d160 2022-07-20T03:15:13.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f637fe700 1 -- 172.21.15.61:0/4166706735 >> v1:172.21.15.117:6789/0 conn(0x7f4f640b2960 legacy=0x7f4f64142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f637fe700 1 -- 172.21.15.61:0/4166706735 >> v1:172.21.15.61:6789/0 conn(0x7f4f6405d160 legacy=0x7f4f640b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f637fe700 1 -- 172.21.15.61:0/4166706735 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f640b1db0 con 0x7f4f640b9ef0 2022-07-20T03:15:13.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f637fe700 1 --2- 172.21.15.61:0/4166706735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b9ef0 0x7f4f640b4e50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f4f58002f70 tx=0x7f4f58006df0).ready entity=mon.0 client_cookie=405555a0437ce8d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f627fc700 1 -- 172.21.15.61:0/4166706735 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f5800d610 con 0x7f4f640b9ef0 2022-07-20T03:15:13.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f627fc700 1 -- 172.21.15.61:0/4166706735 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f58000e90 con 0x7f4f640b9ef0 2022-07-20T03:15:13.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f627fc700 1 -- 172.21.15.61:0/4166706735 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f5800d960 con 0x7f4f640b9ef0 2022-07-20T03:15:13.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 -- 172.21.15.61:0/4166706735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b9ef0 msgr2=0x7f4f640b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 --2- 172.21.15.61:0/4166706735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b9ef0 0x7f4f640b4e50 secure :-1 s=READY pgs=935 cs=0 l=1 rev1=1 rx=0x7f4f58002f70 tx=0x7f4f58006df0).stop 2022-07-20T03:15:13.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 -- 172.21.15.61:0/4166706735 shutdown_connections 2022-07-20T03:15:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 --2- 172.21.15.61:0/4166706735 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b9ef0 0x7f4f640b4e50 unknown :-1 s=CLOSED pgs=935 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 -- 172.21.15.61:0/4166706735 >> 172.21.15.61:0/4166706735 conn(0x7f4f640679d0 msgr2=0x7f4f640b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 -- 172.21.15.61:0/4166706735 shutdown_connections 2022-07-20T03:15:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 -- 172.21.15.61:0/4166706735 wait complete. 2022-07-20T03:15:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 Processor -- start 2022-07-20T03:15:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.913+0000 7f4f6a262700 1 -- start start 2022-07-20T03:15:13.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f6405d160 0x7f4f64146600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 0x7f4f640af3d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4f640b9ef0 0x7f4f64144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640af200 con 0x7f4f640b2960 2022-07-20T03:15:13.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4f640af8c0 con 0x7f4f6405d160 2022-07-20T03:15:13.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4f64145ec0 con 0x7f4f640b9ef0 2022-07-20T03:15:13.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 0x7f4f640af3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 0x7f4f640af3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48150/0 (socket says 172.21.15.61:48150) 2022-07-20T03:15:13.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 -- 172.21.15.61:0/3560120078 learned_addr learned my addr 172.21.15.61:0/3560120078 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:13.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f637fe700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f6405d160 0x7f4f64146600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f63fff700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4f640b9ef0 0x7f4f64144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 -- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4f640b9ef0 msgr2=0x7f4f64144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4f640b9ef0 0x7f4f64144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 -- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f6405d160 msgr2=0x7f4f64146600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:13.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f6405d160 0x7f4f64146600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:13.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 -- 172.21.15.61:0/3560120078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4f5800b180 con 0x7f4f640b2960 2022-07-20T03:15:13.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f62ffd700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 0x7f4f640af3d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f4f640afa00 tx=0x7f4f54007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f540072e0 con 0x7f4f640b2960 2022-07-20T03:15:13.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4f54007440 con 0x7f4f640b2960 2022-07-20T03:15:13.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4f64146160 con 0x7f4f640b2960 2022-07-20T03:15:13.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4f54005800 con 0x7f4f640b2960 2022-07-20T03:15:13.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4f64144930 con 0x7f4f640b2960 2022-07-20T03:15:13.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.917+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4f640add90 con 0x7f4f640b2960 2022-07-20T03:15:13.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.921+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4f54010070 con 0x7f4f640b2960 2022-07-20T03:15:13.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.921+0000 7f4f60ff9700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4f44053af0 0x7f4f44055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:13.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.921+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4f5406b580 con 0x7f4f640b2960 2022-07-20T03:15:13.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.921+0000 7f4f637fe700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4f44053af0 0x7f4f44055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:13.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.921+0000 7f4f637fe700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4f44053af0 0x7f4f44055fb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f4f58018000 tx=0x7f4f58014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:13.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:13.925+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4f640add90 con 0x7f4f640b2960 2022-07-20T03:15:14.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.053+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f4f640add90 con 0x7f4f640b2960 2022-07-20T03:15:14.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f60ff9700 1 -- 172.21.15.61:0/3560120078 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v90) v1 ==== 74+0+16102 (secure 0 0 0) 0x7f4f640add90 con 0x7f4f640b2960 2022-07-20T03:15:14.064 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:14.064 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":90,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:13.467143+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:11.081553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:11.541101+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:14.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4f44053af0 msgr2=0x7f4f44055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4f44053af0 0x7f4f44055fb0 secure :-1 s=READY pgs=456 cs=0 l=1 rev1=1 rx=0x7f4f58018000 tx=0x7f4f58014000).stop 2022-07-20T03:15:14.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 msgr2=0x7f4f640af3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 0x7f4f640af3d0 secure :-1 s=READY pgs=936 cs=0 l=1 rev1=1 rx=0x7f4f640afa00 tx=0x7f4f54007ad0).stop 2022-07-20T03:15:14.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 shutdown_connections 2022-07-20T03:15:14.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4f44053af0 0x7f4f44055fb0 unknown :-1 s=CLOSED pgs=456 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4f640b9ef0 0x7f4f64144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4f640b2960 0x7f4f640af3d0 unknown :-1 s=CLOSED pgs=936 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 --2- 172.21.15.61:0/3560120078 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4f6405d160 0x7f4f64146600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 >> 172.21.15.61:0/3560120078 conn(0x7f4f640679d0 msgr2=0x7f4f64142d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:14.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 shutdown_connections 2022-07-20T03:15:14.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.057+0000 7f4f6a262700 1 -- 172.21.15.61:0/3560120078 wait complete. 2022-07-20T03:15:14.074 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:15:14.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6c922700 1 Processor -- start 2022-07-20T03:15:14.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6c922700 1 -- start start 2022-07-20T03:15:14.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6c922700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8d680b9ef0 0x7f8d680b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6c922700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d680b3e40 con 0x7f8d680b3650 2022-07-20T03:15:14.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6c922700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d680ba2d0 con 0x7f8d680b2290 2022-07-20T03:15:14.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6c922700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d680b4070 con 0x7f8d680b9ef0 2022-07-20T03:15:14.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d65d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8d680b3650 0x7f8d680b48b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44756/0 (socket says 172.21.15.61:44756) 2022-07-20T03:15:14.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d65d9b700 1 -- 172.21.15.61:0/2568351991 learned_addr learned my addr 172.21.15.61:0/2568351991 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:14.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6659c700 1 --2- 172.21.15.61:0/2568351991 >> v2:172.21.15.117:3300/0 conn(0x7f8d680b9ef0 0x7f8d680b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2403541517 0 0) 0x7f8d680b3e40 con 0x7f8d680b3650 2022-07-20T03:15:14.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d48003260 con 0x7f8d680b3650 2022-07-20T03:15:14.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 766853896 0 0) 0x7f8d680ba2d0 con 0x7f8d680b2290 2022-07-20T03:15:14.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8d680b3e40 con 0x7f8d680b2290 2022-07-20T03:15:14.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 507223010 0 0) 0x7f8d48003260 con 0x7f8d680b3650 2022-07-20T03:15:14.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 >> v2:172.21.15.117:3300/0 conn(0x7f8d680b9ef0 msgr2=0x7f8d680b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:14.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 --2- 172.21.15.61:0/2568351991 >> v2:172.21.15.117:3300/0 conn(0x7f8d680b9ef0 0x7f8d680b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 >> v1:172.21.15.117:6789/0 conn(0x7f8d680b2290 legacy=0x7f8d68141d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d680b1df0 con 0x7f8d680b3650 2022-07-20T03:15:14.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f8d58001660 con 0x7f8d680b3650 2022-07-20T03:15:14.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 >> v1:172.21.15.61:6789/0 conn(0x7f8d680b3650 legacy=0x7f8d680b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d480038e0 0x7f8d48003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48004500 0x7f8d48004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d48004e70 0x7f8d4800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d680b4070 con 0x7f8d480038e0 2022-07-20T03:15:14.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4800bcf0 con 0x7f8d48004500 2022-07-20T03:15:14.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d4800beb0 con 0x7f8d48004e70 2022-07-20T03:15:14.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6659c700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d480038e0 0x7f8d48003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d65d9b700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48004500 0x7f8d48004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d66d9d700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d48004e70 0x7f8d4800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6659c700 1 -- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d48004e70 msgr2=0x7f8d4800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:14.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6659c700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d48004e70 0x7f8d4800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.189+0000 7f8d6659c700 1 -- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48004500 msgr2=0x7f8d48004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:14.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6659c700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d48004500 0x7f8d48004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d66d9d700 1 -- 172.21.15.61:0/2568351991 reap_dead start 2022-07-20T03:15:14.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6659c700 1 -- 172.21.15.61:0/2568351991 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d48003260 con 0x7f8d480038e0 2022-07-20T03:15:14.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6659c700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d480038e0 0x7f8d48003ce0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f8d5000c000 tx=0x7f8d50006df0).ready entity=mon.0 client_cookie=aaf6b91eac0edadf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d5000e800 con 0x7f8d480038e0 2022-07-20T03:15:14.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d5000e960 con 0x7f8d480038e0 2022-07-20T03:15:14.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6559a700 1 -- 172.21.15.61:0/2568351991 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d5000ecc0 con 0x7f8d480038e0 2022-07-20T03:15:14.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- 172.21.15.61:0/2568351991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d480038e0 msgr2=0x7f8d48003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d480038e0 0x7f8d48003ce0 secure :-1 s=READY pgs=938 cs=0 l=1 rev1=1 rx=0x7f8d5000c000 tx=0x7f8d50006df0).stop 2022-07-20T03:15:14.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- 172.21.15.61:0/2568351991 shutdown_connections 2022-07-20T03:15:14.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 --2- 172.21.15.61:0/2568351991 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d480038e0 0x7f8d48003ce0 unknown :-1 s=CLOSED pgs=938 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- 172.21.15.61:0/2568351991 >> 172.21.15.61:0/2568351991 conn(0x7f8d68067980 msgr2=0x7f8d680b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:14.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- 172.21.15.61:0/2568351991 shutdown_connections 2022-07-20T03:15:14.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- 172.21.15.61:0/2568351991 wait complete. 2022-07-20T03:15:14.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 Processor -- start 2022-07-20T03:15:14.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- start start 2022-07-20T03:15:14.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480038e0 0x7f8d680adb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 0x7f8d681522c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d680ad310 0x7f8d681527b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d68140440 con 0x7f8d68156420 2022-07-20T03:15:14.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d680ae210 con 0x7f8d680ad310 2022-07-20T03:15:14.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6c922700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d680ae4b0 con 0x7f8d480038e0 2022-07-20T03:15:14.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d65d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 0x7f8d681522c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d65d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 0x7f8d681522c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48170/0 (socket says 172.21.15.61:48170) 2022-07-20T03:15:14.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d65d9b700 1 -- 172.21.15.61:0/3537740492 learned_addr learned my addr 172.21.15.61:0/3537740492 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:14.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d66d9d700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d680ad310 0x7f8d681527b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.193+0000 7f8d6659c700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480038e0 0x7f8d680adb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d65d9b700 1 -- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480038e0 msgr2=0x7f8d680adb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d65d9b700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480038e0 0x7f8d680adb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d65d9b700 1 -- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d680ad310 msgr2=0x7f8d681527b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d65d9b700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d680ad310 0x7f8d681527b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d65d9b700 1 -- 172.21.15.61:0/3537740492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d6813df20 con 0x7f8d68156420 2022-07-20T03:15:14.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d65d9b700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 0x7f8d681522c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f8d5800d000 tx=0x7f8d68142dd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d58012540 con 0x7f8d68156420 2022-07-20T03:15:14.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d580126a0 con 0x7f8d68156420 2022-07-20T03:15:14.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d681400c0 con 0x7f8d68156420 2022-07-20T03:15:14.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d58012a00 con 0x7f8d68156420 2022-07-20T03:15:14.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d6813e8d0 con 0x7f8d68156420 2022-07-20T03:15:14.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8d5801b030 con 0x7f8d68156420 2022-07-20T03:15:14.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d5f7fe700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d4c053a60 0x7f8d4c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(90..90 src has 1..90) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8d5806b6b0 con 0x7f8d68156420 2022-07-20T03:15:14.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d681402b0 con 0x7f8d68156420 2022-07-20T03:15:14.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.197+0000 7f8d6659c700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d4c053a60 0x7f8d4c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.201+0000 7f8d6659c700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d4c053a60 0x7f8d4c055f20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f8d5001d000 tx=0x7f8d50019000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.201+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8d58019030 con 0x7f8d68156420 2022-07-20T03:15:14.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.373+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f8d680ae040 con 0x7f8d68156420 2022-07-20T03:15:14.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d5f7fe700 1 -- 172.21.15.61:0/3537740492 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v32) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8d58046040 con 0x7f8d68156420 2022-07-20T03:15:14.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d4c053a60 msgr2=0x7f8d4c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d4c053a60 0x7f8d4c055f20 secure :-1 s=READY pgs=457 cs=0 l=1 rev1=1 rx=0x7f8d5001d000 tx=0x7f8d50019000).stop 2022-07-20T03:15:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 msgr2=0x7f8d681522c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 0x7f8d681522c0 secure :-1 s=READY pgs=939 cs=0 l=1 rev1=1 rx=0x7f8d5800d000 tx=0x7f8d68142dd0).stop 2022-07-20T03:15:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 shutdown_connections 2022-07-20T03:15:14.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d4c053a60 0x7f8d4c055f20 unknown :-1 s=CLOSED pgs=457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d680ad310 0x7f8d681527b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d68156420 0x7f8d681522c0 unknown :-1 s=CLOSED pgs=939 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 --2- 172.21.15.61:0/3537740492 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d480038e0 0x7f8d680adb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 >> 172.21.15.61:0/3537740492 conn(0x7f8d68067980 msgr2=0x7f8d681423c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 shutdown_connections 2022-07-20T03:15:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.381+0000 7f8d6c922700 1 -- 172.21.15.61:0/3537740492 wait complete. 2022-07-20T03:15:14.395 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:15:14.400 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:15:14.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.537+0000 7f407e662700 1 Processor -- start 2022-07-20T03:15:14.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.537+0000 7f407e662700 1 -- start start 2022-07-20T03:15:14.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40780ae280 con 0x7f40780b6800 2022-07-20T03:15:14.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40780b6be0 con 0x7f40780b3cb0 2022-07-20T03:15:14.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40780ae4b0 con 0x7f40780ad0e0 2022-07-20T03:15:14.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407d660700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f40780b6800 0x7f4078145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44774/0 (socket says 172.21.15.61:44774) 2022-07-20T03:15:14.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407d660700 1 -- 172.21.15.61:0/438435595 learned_addr learned my addr 172.21.15.61:0/438435595 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:14.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407de61700 1 --2- 172.21.15.61:0/438435595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407de61700 1 -- 172.21.15.61:0/438435595 >> v1:172.21.15.117:6789/0 conn(0x7f40780b3cb0 legacy=0x7f40781418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f406bfff700 1 -- 172.21.15.61:0/438435595 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4125853344 0 0) 0x7f40780ae280 con 0x7f40780b6800 2022-07-20T03:15:14.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407de61700 1 -- 172.21.15.61:0/438435595 >> v1:172.21.15.61:6789/0 conn(0x7f40780b6800 legacy=0x7f4078145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407de61700 1 -- 172.21.15.61:0/438435595 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40780ae750 con 0x7f40780ad0e0 2022-07-20T03:15:14.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407de61700 1 --2- 172.21.15.61:0/438435595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780aebf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4074001110 tx=0x7f40700091d0).ready entity=mon.2 client_cookie=6915f2e69ba5de2a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f406bfff700 1 -- 172.21.15.61:0/438435595 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f407000b630 con 0x7f40780ad0e0 2022-07-20T03:15:14.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f406bfff700 1 -- 172.21.15.61:0/438435595 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f407000b790 con 0x7f40780ad0e0 2022-07-20T03:15:14.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f406bfff700 1 -- 172.21.15.61:0/438435595 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f407000b900 con 0x7f40780ad0e0 2022-07-20T03:15:14.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- 172.21.15.61:0/438435595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 msgr2=0x7f40780aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 --2- 172.21.15.61:0/438435595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780aebf0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f4074001110 tx=0x7f40700091d0).stop 2022-07-20T03:15:14.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- 172.21.15.61:0/438435595 shutdown_connections 2022-07-20T03:15:14.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 --2- 172.21.15.61:0/438435595 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780aebf0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- 172.21.15.61:0/438435595 >> 172.21.15.61:0/438435595 conn(0x7f40780b9dd0 msgr2=0x7f40780b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:14.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- 172.21.15.61:0/438435595 shutdown_connections 2022-07-20T03:15:14.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.541+0000 7f407e662700 1 -- 172.21.15.61:0/438435595 wait complete. 2022-07-20T03:15:14.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 Processor -- start 2022-07-20T03:15:14.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 -- start start 2022-07-20T03:15:14.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780b2890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 0x7f407813eb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40780b6800 0x7f407813f080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40780b32e0 con 0x7f40780b3cb0 2022-07-20T03:15:14.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40780b3080 con 0x7f40780b6800 2022-07-20T03:15:14.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f40780b0a70 con 0x7f40780ad0e0 2022-07-20T03:15:14.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 0x7f407813eb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 0x7f407813eb90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48184/0 (socket says 172.21.15.61:48184) 2022-07-20T03:15:14.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 -- 172.21.15.61:0/951918500 learned_addr learned my addr 172.21.15.61:0/951918500 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:14.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407d660700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780b2890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407de61700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40780b6800 0x7f407813f080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 -- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 msgr2=0x7f40780b2890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780b2890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 -- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40780b6800 msgr2=0x7f407813f080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40780b6800 0x7f407813f080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 -- 172.21.15.61:0/951918500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4070009470 con 0x7f40780b3cb0 2022-07-20T03:15:14.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407ce5f700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 0x7f407813eb90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f40740011d0 tx=0x7f4074007bd0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4074007030 con 0x7f40780b3cb0 2022-07-20T03:15:14.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4074007190 con 0x7f40780b3cb0 2022-07-20T03:15:14.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40780b2d80 con 0x7f40780b3cb0 2022-07-20T03:15:14.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4074006510 con 0x7f40780b3cb0 2022-07-20T03:15:14.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.545+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f407813f570 con 0x7f40780b3cb0 2022-07-20T03:15:14.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.549+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40780b5530 con 0x7f40780b3cb0 2022-07-20T03:15:14.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.549+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4074010030 con 0x7f40780b3cb0 2022-07-20T03:15:14.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.549+0000 7f406a7fc700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4060053a40 0x7f4060055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.549+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f407406bec0 con 0x7f40780b3cb0 2022-07-20T03:15:14.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.549+0000 7f407d660700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4060053a40 0x7f4060055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.549+0000 7f407d660700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4060053a40 0x7f4060055f00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f406c00d000 tx=0x7f406c007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.553+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4074042d10 con 0x7f40780b3cb0 2022-07-20T03:15:14.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.721+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f40780574e0 con 0x7f40780b3cb0 2022-07-20T03:15:14.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f406a7fc700 1 -- 172.21.15.61:0/951918500 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v33) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4074046330 con 0x7f40780b3cb0 2022-07-20T03:15:14.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4060053a40 msgr2=0x7f4060055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4060053a40 0x7f4060055f00 secure :-1 s=READY pgs=458 cs=0 l=1 rev1=1 rx=0x7f406c00d000 tx=0x7f406c007c00).stop 2022-07-20T03:15:14.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 msgr2=0x7f407813eb90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 0x7f407813eb90 secure :-1 s=READY pgs=941 cs=0 l=1 rev1=1 rx=0x7f40740011d0 tx=0x7f4074007bd0).stop 2022-07-20T03:15:14.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 shutdown_connections 2022-07-20T03:15:14.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4060053a40 0x7f4060055f00 unknown :-1 s=CLOSED pgs=458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40780b6800 0x7f407813f080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40780b3cb0 0x7f407813eb90 unknown :-1 s=CLOSED pgs=941 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 --2- 172.21.15.61:0/951918500 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f40780ad0e0 0x7f40780b2890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 >> 172.21.15.61:0/951918500 conn(0x7f40780b9dd0 msgr2=0x7f4078141a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:14.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 shutdown_connections 2022-07-20T03:15:14.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.729+0000 7f407e662700 1 -- 172.21.15.61:0/951918500 wait complete. 2022-07-20T03:15:14.742 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:15:14.747 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:15:14.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 Processor -- start 2022-07-20T03:15:14.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- start start 2022-07-20T03:15:14.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13040b5970 0x7f13040ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304143e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13040b0b80 con 0x7f13040af8c0 2022-07-20T03:15:14.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f130405d4d0 con 0x7f13040b5970 2022-07-20T03:15:14.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13040b5d50 con 0x7f1304143a10 2022-07-20T03:15:14.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13040b5970 0x7f13040ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13040b5970 0x7f13040ae100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48190/0 (socket says 172.21.15.61:48190) 2022-07-20T03:15:14.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 -- 172.21.15.61:0/1987565262 learned_addr learned my addr 172.21.15.61:0/1987565262 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:14.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309735700 1 --2- 172.21.15.61:0/1987565262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304143e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 -- 172.21.15.61:0/1987565262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 msgr2=0x7f1304143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:14.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 --2- 172.21.15.61:0/1987565262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304143e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 -- 172.21.15.61:0/1987565262 >> v1:172.21.15.117:6789/0 conn(0x7f13040af8c0 legacy=0x7f13040ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:14.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 -- 172.21.15.61:0/1987565262 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13040b0db0 con 0x7f13040b5970 2022-07-20T03:15:14.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1308f34700 1 --2- 172.21.15.61:0/1987565262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f13040ae100 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f130405d010 tx=0x7f12f4006eb0).ready entity=mon.0 client_cookie=2f72a219e016dd7c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f13037fe700 1 -- 172.21.15.61:0/1987565262 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f400d570 con 0x7f13040b5970 2022-07-20T03:15:14.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f13037fe700 1 -- 172.21.15.61:0/1987565262 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12f4000e90 con 0x7f13040b5970 2022-07-20T03:15:14.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f13037fe700 1 -- 172.21.15.61:0/1987565262 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12f400d8c0 con 0x7f13040b5970 2022-07-20T03:15:14.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- 172.21.15.61:0/1987565262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 msgr2=0x7f13040ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 --2- 172.21.15.61:0/1987565262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f13040ae100 secure :-1 s=READY pgs=942 cs=0 l=1 rev1=1 rx=0x7f130405d010 tx=0x7f12f4006eb0).stop 2022-07-20T03:15:14.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- 172.21.15.61:0/1987565262 shutdown_connections 2022-07-20T03:15:14.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 --2- 172.21.15.61:0/1987565262 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304143e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 --2- 172.21.15.61:0/1987565262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f13040ae100 unknown :-1 s=CLOSED pgs=942 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- 172.21.15.61:0/1987565262 >> 172.21.15.61:0/1987565262 conn(0x7f1304063490 msgr2=0x7f13040b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:14.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- 172.21.15.61:0/1987565262 shutdown_connections 2022-07-20T03:15:14.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.861+0000 7f1309f36700 1 -- 172.21.15.61:0/1987565262 wait complete. 2022-07-20T03:15:14.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 Processor -- start 2022-07-20T03:15:14.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 -- start start 2022-07-20T03:15:14.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13040af8c0 0x7f130414cf60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f1304156850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304156d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130415a0e0 con 0x7f13040b5970 2022-07-20T03:15:14.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f130410e1b0 con 0x7f13040af8c0 2022-07-20T03:15:14.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309f36700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f130415a310 con 0x7f1304143a10 2022-07-20T03:15:14.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f1304156850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309735700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304156d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1308f34700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13040af8c0 0x7f130414cf60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309735700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304156d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:34870/0 (socket says 172.21.15.61:34870) 2022-07-20T03:15:14.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1309735700 1 -- 172.21.15.61:0/1954961032 learned_addr learned my addr 172.21.15.61:0/1954961032 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:14.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 -- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 msgr2=0x7f1304156d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304156d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 -- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13040af8c0 msgr2=0x7f130414cf60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:14.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13040af8c0 0x7f130414cf60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:14.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 -- 172.21.15.61:0/1954961032 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12f400b1e0 con 0x7f13040b5970 2022-07-20T03:15:14.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.865+0000 7f1303fff700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f1304156850 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f12ec002f30 tx=0x7f12ec006cb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12ec012410 con 0x7f13040b5970 2022-07-20T03:15:14.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f12ec012570 con 0x7f13040b5970 2022-07-20T03:15:14.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f130415a5b0 con 0x7f13040b5970 2022-07-20T03:15:14.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f12ec0128d0 con 0x7f13040b5970 2022-07-20T03:15:14.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1304156180 con 0x7f13040b5970 2022-07-20T03:15:14.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13040acae0 con 0x7f13040b5970 2022-07-20T03:15:14.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f12ec00f030 con 0x7f13040b5970 2022-07-20T03:15:14.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1301ffb700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12e8053ad0 0x7f12e8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:14.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f12ec06b970 con 0x7f13040b5970 2022-07-20T03:15:14.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.869+0000 7f1308f34700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12e8053ad0 0x7f12e8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:14.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.873+0000 7f1308f34700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12e8053ad0 0x7f12e8055f90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f12f401c000 tx=0x7f12f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:14.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:14.877+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f12ec0428d0 con 0x7f13040b5970 2022-07-20T03:15:15.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.045+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f13040574e0 con 0x7f13040b5970 2022-07-20T03:15:15.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1301ffb700 1 -- 172.21.15.61:0/1954961032 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v34) v1 ==== 184+0+0 (secure 0 0 0) 0x7f12ec045ef0 con 0x7f13040b5970 2022-07-20T03:15:15.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12e8053ad0 msgr2=0x7f12e8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12e8053ad0 0x7f12e8055f90 secure :-1 s=READY pgs=459 cs=0 l=1 rev1=1 rx=0x7f12f401c000 tx=0x7f12f4015000).stop 2022-07-20T03:15:15.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 msgr2=0x7f1304156850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f1304156850 secure :-1 s=READY pgs=943 cs=0 l=1 rev1=1 rx=0x7f12ec002f30 tx=0x7f12ec006cb0).stop 2022-07-20T03:15:15.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 shutdown_connections 2022-07-20T03:15:15.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12e8053ad0 0x7f12e8055f90 unknown :-1 s=CLOSED pgs=459 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1304143a10 0x7f1304156d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13040b5970 0x7f1304156850 unknown :-1 s=CLOSED pgs=943 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 --2- 172.21.15.61:0/1954961032 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13040af8c0 0x7f130414cf60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 >> 172.21.15.61:0/1954961032 conn(0x7f1304063490 msgr2=0x7f1304145fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:15.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 shutdown_connections 2022-07-20T03:15:15.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.057+0000 7f1309f36700 1 -- 172.21.15.61:0/1954961032 wait complete. 2022-07-20T03:15:15.070 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:15:15.076 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:15.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 Processor -- start 2022-07-20T03:15:15.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 -- start start 2022-07-20T03:15:15.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c640b5650 0x7f9c640b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c6405bd90 0x7f9c64142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c640aea30 con 0x7f9c640b1200 2022-07-20T03:15:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c64142970 con 0x7f9c640b5650 2022-07-20T03:15:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69c02700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c640aec60 con 0x7f9c6405bd90 2022-07-20T03:15:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c63fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9c640b1200 0x7f9c6405c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44802/0 (socket says 172.21.15.61:44802) 2022-07-20T03:15:15.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c63fff700 1 -- 172.21.15.61:0/2682665599 learned_addr learned my addr 172.21.15.61:0/2682665599 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c68c00700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c640b5650 0x7f9c640b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c69401700 1 --2- 172.21.15.61:0/2682665599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c6405bd90 0x7f9c64142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c637fe700 1 -- 172.21.15.61:0/2682665599 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1325841122 0 0) 0x7f9c640aea30 con 0x7f9c640b1200 2022-07-20T03:15:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c637fe700 1 -- 172.21.15.61:0/2682665599 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c44003260 con 0x7f9c640b1200 2022-07-20T03:15:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c68c00700 1 -- 172.21.15.61:0/2682665599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c6405bd90 msgr2=0x7f9c64142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:15.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c68c00700 1 --2- 172.21.15.61:0/2682665599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c6405bd90 0x7f9c64142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c637fe700 1 -- 172.21.15.61:0/2682665599 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4134425937 0 0) 0x7f9c44003260 con 0x7f9c640b1200 2022-07-20T03:15:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c68c00700 1 -- 172.21.15.61:0/2682665599 >> v1:172.21.15.61:6789/0 conn(0x7f9c640b1200 legacy=0x7f9c6405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.177+0000 7f9c68c00700 1 -- 172.21.15.61:0/2682665599 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c640af180 con 0x7f9c640b5650 2022-07-20T03:15:15.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c68c00700 1 --2- 172.21.15.61:0/2682665599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c640b5a30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f9c4c001980 tx=0x7f9c54006df0).ready entity=mon.0 client_cookie=176bddc277a8fb66 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:15.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c637fe700 1 -- 172.21.15.61:0/2682665599 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c4c0017e0 con 0x7f9c640b5650 2022-07-20T03:15:15.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c637fe700 1 -- 172.21.15.61:0/2682665599 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c54000e90 con 0x7f9c640b5650 2022-07-20T03:15:15.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c637fe700 1 -- 172.21.15.61:0/2682665599 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c54012460 con 0x7f9c640b5650 2022-07-20T03:15:15.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 -- 172.21.15.61:0/2682665599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 msgr2=0x7f9c640b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 --2- 172.21.15.61:0/2682665599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c640b5a30 secure :-1 s=READY pgs=945 cs=0 l=1 rev1=1 rx=0x7f9c4c001980 tx=0x7f9c54006df0).stop 2022-07-20T03:15:15.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 -- 172.21.15.61:0/2682665599 shutdown_connections 2022-07-20T03:15:15.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 --2- 172.21.15.61:0/2682665599 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c6405bd90 0x7f9c64142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 --2- 172.21.15.61:0/2682665599 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c640b5a30 unknown :-1 s=CLOSED pgs=945 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 -- 172.21.15.61:0/2682665599 >> 172.21.15.61:0/2682665599 conn(0x7f9c640634e0 msgr2=0x7f9c640b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 -- 172.21.15.61:0/2682665599 shutdown_connections 2022-07-20T03:15:15.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 -- 172.21.15.61:0/2682665599 wait complete. 2022-07-20T03:15:15.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 Processor -- start 2022-07-20T03:15:15.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.181+0000 7f9c69c02700 1 -- start start 2022-07-20T03:15:15.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c6405bd90 0x7f9c641460f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c640b1200 0x7f9c6413e7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c64155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6413fa50 con 0x7f9c640b5650 2022-07-20T03:15:15.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6413fcf0 con 0x7f9c6405bd90 2022-07-20T03:15:15.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c6413ff90 con 0x7f9c640b1200 2022-07-20T03:15:15.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c64155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c64155a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48214/0 (socket says 172.21.15.61:48214) 2022-07-20T03:15:15.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 -- 172.21.15.61:0/3640956085 learned_addr learned my addr 172.21.15.61:0/3640956085 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:15.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c63fff700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c640b1200 0x7f9c6413e7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c68c00700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c6405bd90 0x7f9c641460f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 -- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c640b1200 msgr2=0x7f9c6413e7b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c640b1200 0x7f9c6413e7b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 -- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c6405bd90 msgr2=0x7f9c641460f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c6405bd90 0x7f9c641460f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 -- 172.21.15.61:0/3640956085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c5400b180 con 0x7f9c640b5650 2022-07-20T03:15:15.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69401700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c64155a00 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f9c5c00c000 tx=0x7f9c5c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:15.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c5c012600 con 0x7f9c640b5650 2022-07-20T03:15:15.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c5c012760 con 0x7f9c640b5650 2022-07-20T03:15:15.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c64146d10 con 0x7f9c640b5650 2022-07-20T03:15:15.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c5c012ac0 con 0x7f9c640b5650 2022-07-20T03:15:15.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c6413eca0 con 0x7f9c640b5650 2022-07-20T03:15:15.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c640574e0 con 0x7f9c640b5650 2022-07-20T03:15:15.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.185+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9c5c022070 con 0x7f9c640b5650 2022-07-20T03:15:15.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.189+0000 7f9c61ffb700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c48053ad0 0x7f9c48055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.189+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9c5c06c380 con 0x7f9c640b5650 2022-07-20T03:15:15.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.189+0000 7f9c68c00700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c48053ad0 0x7f9c48055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.189+0000 7f9c68c00700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c48053ad0 0x7f9c48055f90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f9c5401a000 tx=0x7f9c54014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:15.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.193+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c5c043390 con 0x7f9c640b5650 2022-07-20T03:15:15.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.360+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9c64156230 con 0x7f9c640b5650 2022-07-20T03:15:15.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.360+0000 7f9c61ffb700 1 -- 172.21.15.61:0/3640956085 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f9c5c016090 con 0x7f9c640b5650 2022-07-20T03:15:15.369 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:15.369 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:13.467553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7008":{"gid":7008,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1014823584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1014823584},{"type":"v1","addr":"172.21.15.61:6833","nonce":1014823584}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:15.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c48053ad0 msgr2=0x7f9c48055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c48053ad0 0x7f9c48055f90 secure :-1 s=READY pgs=460 cs=0 l=1 rev1=1 rx=0x7f9c5401a000 tx=0x7f9c54014000).stop 2022-07-20T03:15:15.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 msgr2=0x7f9c64155a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c64155a00 secure :-1 s=READY pgs=946 cs=0 l=1 rev1=1 rx=0x7f9c5c00c000 tx=0x7f9c5c0091d0).stop 2022-07-20T03:15:15.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 shutdown_connections 2022-07-20T03:15:15.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c48053ad0 0x7f9c48055f90 unknown :-1 s=CLOSED pgs=460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c640b5650 0x7f9c64155a00 unknown :-1 s=CLOSED pgs=946 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c640b1200 0x7f9c6413e7b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 --2- 172.21.15.61:0/3640956085 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c6405bd90 0x7f9c641460f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 >> 172.21.15.61:0/3640956085 conn(0x7f9c640634e0 msgr2=0x7f9c6413bf40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:15.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 shutdown_connections 2022-07-20T03:15:15.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.364+0000 7f9c69c02700 1 -- 172.21.15.61:0/3640956085 wait complete. 2022-07-20T03:15:15.378 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-20T03:15:15.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:12.462538+0000', 'modified': '2022-07-20T03:15:13.467553+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7008}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7008': {'gid': 7008, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1014823584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1014823584}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1014823584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:15:15.585 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:15:15.586 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:15:15.705 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7008,"want_state":"up:active","state":"up:active","rank_uptime":3.2263824739999998,"mdsmap_epoch":51,"osdmap_epoch":91,"osdmap_epoch_barrier":89,"uptime":7.59823264} 2022-07-20T03:15:15.706 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7008, "mdsmap_epoch": 51, "osdmap_epoch": 91, "osdmap_epoch_barrier": 89, "rank_uptime": 3.226382474, "state": "up:active", "uptime": 7.59823264, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:15:15.707 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:15.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 Processor -- start 2022-07-20T03:15:15.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 -- start start 2022-07-20T03:15:15.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f60540b5650 0x7f60540b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f605405bd90 0x7f6054142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f60540aea30 con 0x7f60540b1200 2022-07-20T03:15:15.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6054142970 con 0x7f60540b5650 2022-07-20T03:15:15.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.812+0000 7f60595e3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f60540aec60 con 0x7f605405bd90 2022-07-20T03:15:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f60540b5650 0x7f60540b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f60540b5650 0x7f60540b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:57898/0 (socket says 172.21.15.61:57898) 2022-07-20T03:15:15.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 -- 172.21.15.61:0/2796437052 learned_addr learned my addr 172.21.15.61:0/2796437052 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:15.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6058de2700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f605405bd90 0x7f6054142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 -- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f605405bd90 msgr2=0x7f6054142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:15.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f605405bd90 0x7f6054142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 -- 172.21.15.61:0/2796437052 >> v1:172.21.15.117:6789/0 conn(0x7f60540b1200 legacy=0x7f605405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6052ffd700 1 -- 172.21.15.61:0/2796437052 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 445470157 0 0) 0x7f60540aea30 con 0x7f60540b1200 2022-07-20T03:15:15.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 -- 172.21.15.61:0/2796437052 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60540af180 con 0x7f60540b5650 2022-07-20T03:15:15.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6058de2700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f605405bd90 0x7f6054142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:15.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6053fff700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60540b5650 0x7f60540b5a30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f6044002f70 tx=0x7f6044006eb0).ready entity=mon.1 client_cookie=c1825efb264d3c89 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:15.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6052ffd700 1 -- 172.21.15.61:0/2796437052 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604400d570 con 0x7f60540b5650 2022-07-20T03:15:15.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6052ffd700 1 -- 172.21.15.61:0/2796437052 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6044000e90 con 0x7f60540b5650 2022-07-20T03:15:15.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f6052ffd700 1 -- 172.21.15.61:0/2796437052 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f604400d8c0 con 0x7f60540b5650 2022-07-20T03:15:15.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 -- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60540b5650 msgr2=0x7f60540b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60540b5650 0x7f60540b5a30 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f6044002f70 tx=0x7f6044006eb0).stop 2022-07-20T03:15:15.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 -- 172.21.15.61:0/2796437052 shutdown_connections 2022-07-20T03:15:15.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f605405bd90 0x7f6054142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 --2- 172.21.15.61:0/2796437052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f60540b5650 0x7f60540b5a30 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 -- 172.21.15.61:0/2796437052 >> 172.21.15.61:0/2796437052 conn(0x7f60540634e0 msgr2=0x7f60540b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:15.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 -- 172.21.15.61:0/2796437052 shutdown_connections 2022-07-20T03:15:15.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.816+0000 7f60595e3700 1 -- 172.21.15.61:0/2796437052 wait complete. 2022-07-20T03:15:15.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 Processor -- start 2022-07-20T03:15:15.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 -- start start 2022-07-20T03:15:15.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f605405bd90 0x7f605413d930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 0x7f6054141400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60540b5650 0x7f6054152030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605413cf50 con 0x7f60540b1200 2022-07-20T03:15:15.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f605413d1f0 con 0x7f605405bd90 2022-07-20T03:15:15.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f605413d490 con 0x7f60540b5650 2022-07-20T03:15:15.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 0x7f6054141400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 0x7f6054141400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48224/0 (socket says 172.21.15.61:48224) 2022-07-20T03:15:15.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 -- 172.21.15.61:0/3023169332 learned_addr learned my addr 172.21.15.61:0/3023169332 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:15.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f6053fff700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f605405bd90 0x7f605413d930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f6058de2700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60540b5650 0x7f6054152030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 -- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60540b5650 msgr2=0x7f6054152030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60540b5650 0x7f6054152030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 -- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f605405bd90 msgr2=0x7f605413d930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f605405bd90 0x7f605413d930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 -- 172.21.15.61:0/3023169332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f604c009470 con 0x7f60540b1200 2022-07-20T03:15:15.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f6053fff700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f605405bd90 0x7f605413d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:15.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f6058de2700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60540b5650 0x7f6054152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:15.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60537fe700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 0x7f6054141400 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f603c0010d0 tx=0x7f603c007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:15.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603c00f470 con 0x7f60540b1200 2022-07-20T03:15:15.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f603c00f5d0 con 0x7f60540b1200 2022-07-20T03:15:15.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f604400b1e0 con 0x7f60540b1200 2022-07-20T03:15:15.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f603c00f930 con 0x7f60540b1200 2022-07-20T03:15:15.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.820+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f605413fdc0 con 0x7f60540b1200 2022-07-20T03:15:15.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.824+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60540574e0 con 0x7f60540b1200 2022-07-20T03:15:15.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.824+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f603c019030 con 0x7f60540b1200 2022-07-20T03:15:15.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.824+0000 7f60517fa700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f602c053ad0 0x7f602c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:15.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.824+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f603c06b1d0 con 0x7f60540b1200 2022-07-20T03:15:15.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.824+0000 7f6053fff700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f602c053ad0 0x7f602c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:15.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.824+0000 7f6053fff700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f602c053ad0 0x7f602c055f90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f604401a000 tx=0x7f6044015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:15.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.828+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f603c0421e0 con 0x7f60540b1200 2022-07-20T03:15:15.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.952+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f60540b1a10 con 0x7f60540b1200 2022-07-20T03:15:15.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60517fa700 1 -- 172.21.15.61:0/3023169332 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f603c024ef0 con 0x7f60540b1200 2022-07-20T03:15:15.961 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:15.961 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":91,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:14.477402+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:11.081553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:11.541101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:15.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f602c053ad0 msgr2=0x7f602c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f602c053ad0 0x7f602c055f90 secure :-1 s=READY pgs=461 cs=0 l=1 rev1=1 rx=0x7f604401a000 tx=0x7f6044015000).stop 2022-07-20T03:15:15.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 msgr2=0x7f6054141400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:15.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 0x7f6054141400 secure :-1 s=READY pgs=947 cs=0 l=1 rev1=1 rx=0x7f603c0010d0 tx=0x7f603c007ac0).stop 2022-07-20T03:15:15.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 shutdown_connections 2022-07-20T03:15:15.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f602c053ad0 0x7f602c055f90 unknown :-1 s=CLOSED pgs=461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f60540b5650 0x7f6054152030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f60540b1200 0x7f6054141400 unknown :-1 s=CLOSED pgs=947 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 --2- 172.21.15.61:0/3023169332 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f605405bd90 0x7f605413d930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:15.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 >> 172.21.15.61:0/3023169332 conn(0x7f60540634e0 msgr2=0x7f60540b3a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:15.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 shutdown_connections 2022-07-20T03:15:15.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:15.956+0000 7f60595e3700 1 -- 172.21.15.61:0/3023169332 wait complete. 2022-07-20T03:15:15.978 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:15:11.081553+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:15:11.541101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:15:15.980 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:16.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd7116700 1 Processor -- start 2022-07-20T03:15:16.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd7116700 1 -- start start 2022-07-20T03:15:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd7116700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6cd0099300 0x7f6cd0099750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd7116700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd00b3a50 con 0x7f6cd00ac180 2022-07-20T03:15:16.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd7116700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd00ac610 con 0x7f6cd00b3670 2022-07-20T03:15:16.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd7116700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6cd013d3b0 con 0x7f6cd0099300 2022-07-20T03:15:16.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd56b3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6cd0099300 0x7f6cd0099750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd56b3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6cd0099300 0x7f6cd0099750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48236/0 (socket says 172.21.15.61:48236) 2022-07-20T03:15:16.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6cd56b3700 1 -- 172.21.15.61:0/3480954877 learned_addr learned my addr 172.21.15.61:0/3480954877 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.088+0000 7f6ccffff700 1 --1- 172.21.15.61:0/3480954877 >> v1:172.21.15.61:6789/0 conn(0x7f6cd00ac180 0x7f6cd013f920 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44832/0 (socket says 172.21.15.61:44832) 2022-07-20T03:15:16.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 -- 172.21.15.61:0/3480954877 >> v1:172.21.15.117:6789/0 conn(0x7f6cd00b3670 legacy=0x7f6cd01437e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 -- 172.21.15.61:0/3480954877 >> v1:172.21.15.61:6789/0 conn(0x7f6cd00ac180 legacy=0x7f6cd013f920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 -- 172.21.15.61:0/3480954877 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd01430a0 con 0x7f6cd0099300 2022-07-20T03:15:16.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 --2- 172.21.15.61:0/3480954877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd0099300 0x7f6cd0099750 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f6cc80032f0 tx=0x7f6cc800b530).ready entity=mon.0 client_cookie=6c567bb2924dd078 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6ccf7fe700 1 -- 172.21.15.61:0/3480954877 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc8010660 con 0x7f6cd0099300 2022-07-20T03:15:16.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6ccf7fe700 1 -- 172.21.15.61:0/3480954877 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc80107c0 con 0x7f6cd0099300 2022-07-20T03:15:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6ccf7fe700 1 -- 172.21.15.61:0/3480954877 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc8010b20 con 0x7f6cd0099300 2022-07-20T03:15:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- 172.21.15.61:0/3480954877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd0099300 msgr2=0x7f6cd0099750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 --2- 172.21.15.61:0/3480954877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd0099300 0x7f6cd0099750 secure :-1 s=READY pgs=949 cs=0 l=1 rev1=1 rx=0x7f6cc80032f0 tx=0x7f6cc800b530).stop 2022-07-20T03:15:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- 172.21.15.61:0/3480954877 shutdown_connections 2022-07-20T03:15:16.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 --2- 172.21.15.61:0/3480954877 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd0099300 0x7f6cd0099750 unknown :-1 s=CLOSED pgs=949 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- 172.21.15.61:0/3480954877 >> 172.21.15.61:0/3480954877 conn(0x7f6cd00b6c50 msgr2=0x7f6cd00b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- 172.21.15.61:0/3480954877 shutdown_connections 2022-07-20T03:15:16.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- 172.21.15.61:0/3480954877 wait complete. 2022-07-20T03:15:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 Processor -- start 2022-07-20T03:15:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- start start 2022-07-20T03:15:16.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd0099300 0x7f6cd0151760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd00ac180 0x7f6cd0141a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 0x7f6cd0141f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd0151f00 con 0x7f6cd00b3670 2022-07-20T03:15:16.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd01521a0 con 0x7f6cd0099300 2022-07-20T03:15:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd7116700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd0151c50 con 0x7f6cd00ac180 2022-07-20T03:15:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 0x7f6cd0141f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 0x7f6cd0141f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48242/0 (socket says 172.21.15.61:48242) 2022-07-20T03:15:16.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd56b3700 1 -- 172.21.15.61:0/2346598350 learned_addr learned my addr 172.21.15.61:0/2346598350 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6cd4eb2700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd0099300 0x7f6cd0151760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.092+0000 7f6ccffff700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd00ac180 0x7f6cd0141a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd56b3700 1 -- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd00ac180 msgr2=0x7f6cd0141a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd56b3700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd00ac180 0x7f6cd0141a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd56b3700 1 -- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd0099300 msgr2=0x7f6cd0151760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd56b3700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd0099300 0x7f6cd0151760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd56b3700 1 -- 172.21.15.61:0/2346598350 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cc800b7d0 con 0x7f6cd00b3670 2022-07-20T03:15:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd56b3700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 0x7f6cd0141f20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f6cc80032f0 tx=0x7f6cd00aba60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc8018610 con 0x7f6cd00b3670 2022-07-20T03:15:16.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cc8018770 con 0x7f6cd00b3670 2022-07-20T03:15:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cc8018ad0 con 0x7f6cd00b3670 2022-07-20T03:15:16.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd0142410 con 0x7f6cd00b3670 2022-07-20T03:15:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd00add70 con 0x7f6cd00b3670 2022-07-20T03:15:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cd00574e0 con 0x7f6cd00b3670 2022-07-20T03:15:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6cc8021020 con 0x7f6cd00b3670 2022-07-20T03:15:16.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.096+0000 7f6ccdffb700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb00539b0 0x7f6cb0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.100+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6cc806fec0 con 0x7f6cd00b3670 2022-07-20T03:15:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.100+0000 7f6cd4eb2700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb00539b0 0x7f6cb0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.100+0000 7f6cd4eb2700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb00539b0 0x7f6cb0055e70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f6cc000d000 tx=0x7f6cc0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.104+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6cc8046e20 con 0x7f6cd00b3670 2022-07-20T03:15:16.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.228+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f6cd005db10 con 0x7f6cd00b3670 2022-07-20T03:15:16.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6ccdffb700 1 -- 172.21.15.61:0/2346598350 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v91) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6cc804a440 con 0x7f6cd00b3670 2022-07-20T03:15:16.236 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:16.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb00539b0 msgr2=0x7f6cb0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb00539b0 0x7f6cb0055e70 secure :-1 s=READY pgs=462 cs=0 l=1 rev1=1 rx=0x7f6cc000d000 tx=0x7f6cc0007b90).stop 2022-07-20T03:15:16.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 msgr2=0x7f6cd0141f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 0x7f6cd0141f20 secure :-1 s=READY pgs=950 cs=0 l=1 rev1=1 rx=0x7f6cc80032f0 tx=0x7f6cd00aba60).stop 2022-07-20T03:15:16.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 shutdown_connections 2022-07-20T03:15:16.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb00539b0 0x7f6cb0055e70 unknown :-1 s=CLOSED pgs=462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd00b3670 0x7f6cd0141f20 unknown :-1 s=CLOSED pgs=950 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd00ac180 0x7f6cd0141a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 --2- 172.21.15.61:0/2346598350 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd0099300 0x7f6cd0151760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 >> 172.21.15.61:0/2346598350 conn(0x7f6cd00b6c50 msgr2=0x7f6cd013bad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 shutdown_connections 2022-07-20T03:15:16.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.232+0000 7f6cd7116700 1 -- 172.21.15.61:0/2346598350 wait complete. 2022-07-20T03:15:16.249 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 Processor -- start 2022-07-20T03:15:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 -- start start 2022-07-20T03:15:16.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7fd805c0d0 0x7f7fd80b3770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7fd81439c0 0x7f7fd8142ad0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fd805c4b0 con 0x7f7fd805d200 2022-07-20T03:15:16.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd80b3da0 con 0x7f7fd805c0d0 2022-07-20T03:15:16.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fde6d6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fd80b4040 con 0x7f7fd81439c0 2022-07-20T03:15:16.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7fd805c0d0 0x7f7fd80b3770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7fd805c0d0 0x7f7fd80b3770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48246/0 (socket says 172.21.15.61:48246) 2022-07-20T03:15:16.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 -- 172.21.15.61:0/1097910103 learned_addr learned my addr 172.21.15.61:0/1097910103 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdded5700 1 --2- 172.21.15.61:0/1097910103 >> v2:172.21.15.117:3300/0 conn(0x7f7fd81439c0 0x7f7fd8142ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 -- 172.21.15.61:0/1097910103 >> v2:172.21.15.117:3300/0 conn(0x7f7fd81439c0 msgr2=0x7f7fd8142ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:16.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 --2- 172.21.15.61:0/1097910103 >> v2:172.21.15.117:3300/0 conn(0x7f7fd81439c0 0x7f7fd8142ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 -- 172.21.15.61:0/1097910103 >> v1:172.21.15.117:6789/0 conn(0x7f7fd805d200 legacy=0x7f7fd81432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 -- 172.21.15.61:0/1097910103 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd8146440 con 0x7f7fd805c0d0 2022-07-20T03:15:16.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdded5700 1 --2- 172.21.15.61:0/1097910103 >> v2:172.21.15.117:3300/0 conn(0x7f7fd81439c0 0x7f7fd8142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:16.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fdd6d4700 1 --2- 172.21.15.61:0/1097910103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805c0d0 0x7f7fd80b3770 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f7fd4002f70 tx=0x7f7fd4006eb0).ready entity=mon.0 client_cookie=c76ab903418960e8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fcbfff700 1 -- 172.21.15.61:0/1097910103 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd400d5c0 con 0x7f7fd805c0d0 2022-07-20T03:15:16.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fcbfff700 1 -- 172.21.15.61:0/1097910103 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd4000e90 con 0x7f7fd805c0d0 2022-07-20T03:15:16.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.356+0000 7f7fcbfff700 1 -- 172.21.15.61:0/1097910103 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd400d950 con 0x7f7fd805c0d0 2022-07-20T03:15:16.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- 172.21.15.61:0/1097910103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805c0d0 msgr2=0x7f7fd80b3770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/1097910103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805c0d0 0x7f7fd80b3770 secure :-1 s=READY pgs=951 cs=0 l=1 rev1=1 rx=0x7f7fd4002f70 tx=0x7f7fd4006eb0).stop 2022-07-20T03:15:16.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- 172.21.15.61:0/1097910103 shutdown_connections 2022-07-20T03:15:16.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/1097910103 >> v2:172.21.15.117:3300/0 conn(0x7f7fd81439c0 0x7f7fd8142ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/1097910103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805c0d0 0x7f7fd80b3770 unknown :-1 s=CLOSED pgs=951 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- 172.21.15.61:0/1097910103 >> 172.21.15.61:0/1097910103 conn(0x7f7fd8064b90 msgr2=0x7f7fd8064f70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- 172.21.15.61:0/1097910103 shutdown_connections 2022-07-20T03:15:16.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- 172.21.15.61:0/1097910103 wait complete. 2022-07-20T03:15:16.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 Processor -- start 2022-07-20T03:15:16.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- start start 2022-07-20T03:15:16.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fd805c0d0 0x7f7fd805fa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 0x7f7fd80b65a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fd81439c0 0x7f7fd80af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80611b0 con 0x7f7fd805d200 2022-07-20T03:15:16.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80afbe0 con 0x7f7fd805c0d0 2022-07-20T03:15:16.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.360+0000 7f7fde6d6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fd80613e0 con 0x7f7fd81439c0 2022-07-20T03:15:16.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 0x7f7fd80b65a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdd6d4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fd805c0d0 0x7f7fd805fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 0x7f7fd80b65a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48254/0 (socket says 172.21.15.61:48254) 2022-07-20T03:15:16.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 -- 172.21.15.61:0/3451621418 learned_addr learned my addr 172.21.15.61:0/3451621418 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdded5700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fd81439c0 0x7f7fd80af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 -- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fd81439c0 msgr2=0x7f7fd80af6f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fd81439c0 0x7f7fd80af6f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 -- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fd805c0d0 msgr2=0x7f7fd805fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fd805c0d0 0x7f7fd805fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 -- 172.21.15.61:0/3451621418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd0009470 con 0x7f7fd805d200 2022-07-20T03:15:16.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdced3700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 0x7f7fd80b65a0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f7fd8065830 tx=0x7f7fcc007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fcc012400 con 0x7f7fd805d200 2022-07-20T03:15:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fcc012560 con 0x7f7fd805d200 2022-07-20T03:15:16.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fd400b1e0 con 0x7f7fd805d200 2022-07-20T03:15:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fcc0128c0 con 0x7f7fd805d200 2022-07-20T03:15:16.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fd81458f0 con 0x7f7fd805d200 2022-07-20T03:15:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7fcc019030 con 0x7f7fd805d200 2022-07-20T03:15:16.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fd80615a0 con 0x7f7fd805d200 2022-07-20T03:15:16.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fca7fc700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053940 0x7f7fc0055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7fcc06b210 con 0x7f7fd805d200 2022-07-20T03:15:16.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.364+0000 7f7fdd6d4700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053940 0x7f7fc0055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.368+0000 7f7fdd6d4700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053940 0x7f7fc0055e00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f7fd401c000 tx=0x7f7fd4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.372+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7fd80615a0 con 0x7f7fd805d200 2022-07-20T03:15:16.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.500+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7fd80615a0 con 0x7f7fd805d200 2022-07-20T03:15:16.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.500+0000 7f7fca7fc700 1 -- 172.21.15.61:0/3451621418 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7fd80615a0 con 0x7f7fd805d200 2022-07-20T03:15:16.507 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:16.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053940 msgr2=0x7f7fc0055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053940 0x7f7fc0055e00 secure :-1 s=READY pgs=463 cs=0 l=1 rev1=1 rx=0x7f7fd401c000 tx=0x7f7fd4015000).stop 2022-07-20T03:15:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 msgr2=0x7f7fd80b65a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 0x7f7fd80b65a0 secure :-1 s=READY pgs=952 cs=0 l=1 rev1=1 rx=0x7f7fd8065830 tx=0x7f7fcc007c50).stop 2022-07-20T03:15:16.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 shutdown_connections 2022-07-20T03:15:16.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053940 0x7f7fc0055e00 unknown :-1 s=CLOSED pgs=463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fd81439c0 0x7f7fd80af6f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fd805d200 0x7f7fd80b65a0 unknown :-1 s=CLOSED pgs=952 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 --2- 172.21.15.61:0/3451621418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fd805c0d0 0x7f7fd805fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 >> 172.21.15.61:0/3451621418 conn(0x7f7fd8064b90 msgr2=0x7f7fd805c720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 shutdown_connections 2022-07-20T03:15:16.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.504+0000 7f7fde6d6700 1 -- 172.21.15.61:0/3451621418 wait complete. 2022-07-20T03:15:16.520 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:15:16.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 Processor -- start 2022-07-20T03:15:16.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 -- start start 2022-07-20T03:15:16.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96200b1080 0x7f96200b1460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f962013e410 0x7f962013e850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f962013f970 con 0x7f96200ae720 2022-07-20T03:15:16.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f962013fbe0 con 0x7f962013e410 2022-07-20T03:15:16.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.628+0000 7f9624f6e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f962013fe80 con 0x7f96200b1080 2022-07-20T03:15:16.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f962013e410 0x7f962013e850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f962013e410 0x7f962013e850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48260/0 (socket says 172.21.15.61:48260) 2022-07-20T03:15:16.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 -- 172.21.15.61:0/3610240625 learned_addr learned my addr 172.21.15.61:0/3610240625 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f961f7fe700 1 --2- 172.21.15.61:0/3610240625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96200b1080 0x7f96200b1460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 -- 172.21.15.61:0/3610240625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96200b1080 msgr2=0x7f96200b1460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:16.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 --2- 172.21.15.61:0/3610240625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96200b1080 0x7f96200b1460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 -- 172.21.15.61:0/3610240625 >> v1:172.21.15.117:6789/0 conn(0x7f96200ae720 legacy=0x7f96200aeb70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 -- 172.21.15.61:0/3610240625 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f962013ed40 con 0x7f962013e410 2022-07-20T03:15:16.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9616ffd700 1 --2- 172.21.15.61:0/3610240625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962013e410 0x7f962013e850 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f9608002f70 tx=0x7f9608006f40).ready entity=mon.0 client_cookie=5be7cef3afea1367 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f961effd700 1 -- 172.21.15.61:0/3610240625 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f960800dc70 con 0x7f962013e410 2022-07-20T03:15:16.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f961effd700 1 -- 172.21.15.61:0/3610240625 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9608000e90 con 0x7f962013e410 2022-07-20T03:15:16.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f961effd700 1 -- 172.21.15.61:0/3610240625 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f960800be90 con 0x7f962013e410 2022-07-20T03:15:16.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- 172.21.15.61:0/3610240625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962013e410 msgr2=0x7f962013e850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 --2- 172.21.15.61:0/3610240625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962013e410 0x7f962013e850 secure :-1 s=READY pgs=953 cs=0 l=1 rev1=1 rx=0x7f9608002f70 tx=0x7f9608006f40).stop 2022-07-20T03:15:16.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- 172.21.15.61:0/3610240625 shutdown_connections 2022-07-20T03:15:16.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 --2- 172.21.15.61:0/3610240625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f962013e410 0x7f962013e850 unknown :-1 s=CLOSED pgs=953 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 --2- 172.21.15.61:0/3610240625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f96200b1080 0x7f96200b1460 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- 172.21.15.61:0/3610240625 >> 172.21.15.61:0/3610240625 conn(0x7f9620063480 msgr2=0x7f96200b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- 172.21.15.61:0/3610240625 shutdown_connections 2022-07-20T03:15:16.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- 172.21.15.61:0/3610240625 wait complete. 2022-07-20T03:15:16.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 Processor -- start 2022-07-20T03:15:16.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- start start 2022-07-20T03:15:16.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96200ae720 0x7f9620063d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96200b1080 0x7f9620137be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 0x7f96201521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96201335a0 con 0x7f96200ae720 2022-07-20T03:15:16.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9620145fb0 con 0x7f96200b1080 2022-07-20T03:15:16.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f9624f6e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9620138a50 con 0x7f962013e410 2022-07-20T03:15:16.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f961f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96200ae720 0x7f9620063d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.632+0000 7f961ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 0x7f96201521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f9616ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96200b1080 0x7f9620137be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 0x7f96201521a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:34940/0 (socket says 172.21.15.61:34940) 2022-07-20T03:15:16.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961ffff700 1 -- 172.21.15.61:0/1213409818 learned_addr learned my addr 172.21.15.61:0/1213409818 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961f7fe700 1 -- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 msgr2=0x7f96201521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961f7fe700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 0x7f96201521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961f7fe700 1 -- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96200b1080 msgr2=0x7f9620137be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961f7fe700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96200b1080 0x7f9620137be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961f7fe700 1 -- 172.21.15.61:0/1213409818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f960800b460 con 0x7f96200ae720 2022-07-20T03:15:16.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961ffff700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 0x7f96201521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:16.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f9616ffd700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96200b1080 0x7f9620137be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:16.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961f7fe700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96200ae720 0x7f9620063d40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f96201336e0 tx=0x7f9610008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9610011780 con 0x7f96200ae720 2022-07-20T03:15:16.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f96100118e0 con 0x7f96200ae720 2022-07-20T03:15:16.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9610011c40 con 0x7f96200ae720 2022-07-20T03:15:16.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9620144bb0 con 0x7f96200ae720 2022-07-20T03:15:16.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9620145b10 con 0x7f96200ae720 2022-07-20T03:15:16.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.636+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9620145cd0 con 0x7f96200ae720 2022-07-20T03:15:16.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.640+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9610012420 con 0x7f96200ae720 2022-07-20T03:15:16.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.640+0000 7f961d7fa700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9604053ad0 0x7f9604055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.640+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f961006bb00 con 0x7f96200ae720 2022-07-20T03:15:16.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.640+0000 7f9616ffd700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9604053ad0 0x7f9604055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.640+0000 7f9616ffd700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9604053ad0 0x7f9604055f90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f960801b000 tx=0x7f9608006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.644+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9610042a00 con 0x7f96200ae720 2022-07-20T03:15:16.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.768+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f96200af280 con 0x7f96200ae720 2022-07-20T03:15:16.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.768+0000 7f961d7fa700 1 -- 172.21.15.61:0/1213409818 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9610046020 con 0x7f96200ae720 2022-07-20T03:15:16.774 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:16.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9604053ad0 msgr2=0x7f9604055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9604053ad0 0x7f9604055f90 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f960801b000 tx=0x7f9608006f40).stop 2022-07-20T03:15:16.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96200ae720 msgr2=0x7f9620063d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96200ae720 0x7f9620063d40 secure :-1 s=READY pgs=954 cs=0 l=1 rev1=1 rx=0x7f96201336e0 tx=0x7f9610008040).stop 2022-07-20T03:15:16.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 shutdown_connections 2022-07-20T03:15:16.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9604053ad0 0x7f9604055f90 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f96200b1080 0x7f9620137be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f962013e410 0x7f96201521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 --2- 172.21.15.61:0/1213409818 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f96200ae720 0x7f9620063d40 unknown :-1 s=CLOSED pgs=954 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 >> 172.21.15.61:0/1213409818 conn(0x7f9620063480 msgr2=0x7f96200b1a40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 shutdown_connections 2022-07-20T03:15:16.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.772+0000 7f9624f6e700 1 -- 172.21.15.61:0/1213409818 wait complete. 2022-07-20T03:15:16.789 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:15:16.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.892+0000 7f0bdb59e700 1 Processor -- start 2022-07-20T03:15:16.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.892+0000 7f0bdb59e700 1 -- start start 2022-07-20T03:15:16.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc0ac460 0x7f0bdc0ab0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdb59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0bdc0b4d70 0x7f0bdc0b5200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdb59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0bdc0ac840 con 0x7f0bdc142ac0 2022-07-20T03:15:16.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdb59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0bdc143240 con 0x7f0bdc0b4d70 2022-07-20T03:15:16.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc1434e0 con 0x7f0bdc0ac460 2022-07-20T03:15:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0bdc0b4d70 0x7f0bdc0b5200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0bdc0b4d70 0x7f0bdc0b5200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:57958/0 (socket says 172.21.15.61:57958) 2022-07-20T03:15:16.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 -- 172.21.15.61:0/2329795392 learned_addr learned my addr 172.21.15.61:0/2329795392 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bda59c700 1 --2- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc0ac460 0x7f0bdc0ab0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 -- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc0ac460 msgr2=0x7f0bdc0ab0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:16.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 --2- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc0ac460 0x7f0bdc0ab0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 -- 172.21.15.61:0/2329795392 >> v1:172.21.15.117:6789/0 conn(0x7f0bdc142ac0 legacy=0x7f0bdc13c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 -- 172.21.15.61:0/2329795392 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bdc143780 con 0x7f0bdc0b4d70 2022-07-20T03:15:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bdad9d700 1 --2- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc0b5200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f0bd40032f0 tx=0x7f0bd4009290).ready entity=mon.1 client_cookie=2dba3597014074da server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bd959a700 1 -- 172.21.15.61:0/2329795392 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd400b8a0 con 0x7f0bdc0b4d70 2022-07-20T03:15:16.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bd959a700 1 -- 172.21.15.61:0/2329795392 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bd400ba00 con 0x7f0bdc0b4d70 2022-07-20T03:15:16.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.896+0000 7f0bd959a700 1 -- 172.21.15.61:0/2329795392 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bd400bb70 con 0x7f0bdc0b4d70 2022-07-20T03:15:16.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 -- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 msgr2=0x7f0bdc0b5200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc0b5200 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f0bd40032f0 tx=0x7f0bd4009290).stop 2022-07-20T03:15:16.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 -- 172.21.15.61:0/2329795392 shutdown_connections 2022-07-20T03:15:16.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc0b5200 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/2329795392 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc0ac460 0x7f0bdc0ab0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 -- 172.21.15.61:0/2329795392 >> 172.21.15.61:0/2329795392 conn(0x7f0bdc0632b0 msgr2=0x7f0bdc063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:16.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 -- 172.21.15.61:0/2329795392 shutdown_connections 2022-07-20T03:15:16.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 -- 172.21.15.61:0/2329795392 wait complete. 2022-07-20T03:15:16.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 Processor -- start 2022-07-20T03:15:16.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.900+0000 7f0bdb59e700 1 -- start start 2022-07-20T03:15:16.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 0x7f0bdc065440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc142ac0 0x7f0bdc1471d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc149680 con 0x7f0bdc0ac460 2022-07-20T03:15:16.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc156180 con 0x7f0bdc0b4d70 2022-07-20T03:15:16.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0bdc064d00 con 0x7f0bdc142ac0 2022-07-20T03:15:16.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 0x7f0bdc065440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 0x7f0bdc065440 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48282/0 (socket says 172.21.15.61:48282) 2022-07-20T03:15:16.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 -- 172.21.15.61:0/1189379039 learned_addr learned my addr 172.21.15.61:0/1189379039 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:16.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bd9d9b700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdad9d700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc142ac0 0x7f0bdc1471d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 -- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc142ac0 msgr2=0x7f0bdc1471d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc142ac0 0x7f0bdc1471d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 -- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 msgr2=0x7f0bdc155850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:16.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc155850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:16.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 -- 172.21.15.61:0/1189379039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0bd4009530 con 0x7f0bdc0ac460 2022-07-20T03:15:16.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bda59c700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 0x7f0bdc065440 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f0bdc0668f0 tx=0x7f0bcc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bcc010780 con 0x7f0bdc0ac460 2022-07-20T03:15:16.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0bcc0108e0 con 0x7f0bdc0ac460 2022-07-20T03:15:16.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0bdc064fa0 con 0x7f0bdc0ac460 2022-07-20T03:15:16.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0bcc010c40 con 0x7f0bdc0ac460 2022-07-20T03:15:16.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0bdc065fe0 con 0x7f0bdc0ac460 2022-07-20T03:15:16.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.904+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0bdc142f40 con 0x7f0bdc0ac460 2022-07-20T03:15:16.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.908+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0bcc026430 con 0x7f0bdc0ac460 2022-07-20T03:15:16.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.908+0000 7f0bcb7fe700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0bb80539b0 0x7f0bb8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:16.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.908+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0bcc06ced0 con 0x7f0bdc0ac460 2022-07-20T03:15:16.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.908+0000 7f0bd9d9b700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0bb80539b0 0x7f0bb8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:16.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.908+0000 7f0bd9d9b700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0bb80539b0 0x7f0bb8055e70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f0bd000d000 tx=0x7f0bd0007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:16.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:16.916+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0bcc043cc0 con 0x7f0bdc0ac460 2022-07-20T03:15:17.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f0bdc0574e0 con 0x7f0bdc0ac460 2022-07-20T03:15:17.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bcb7fe700 1 -- 172.21.15.61:0/1189379039 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+10 (secure 0 0 0) 0x7f0bcc0472e0 con 0x7f0bdc0ac460 2022-07-20T03:15:17.056 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:17.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0bb80539b0 msgr2=0x7f0bb8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0bb80539b0 0x7f0bb8055e70 secure :-1 s=READY pgs=465 cs=0 l=1 rev1=1 rx=0x7f0bd000d000 tx=0x7f0bd0007b70).stop 2022-07-20T03:15:17.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 msgr2=0x7f0bdc065440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 0x7f0bdc065440 secure :-1 s=READY pgs=955 cs=0 l=1 rev1=1 rx=0x7f0bdc0668f0 tx=0x7f0bcc006df0).stop 2022-07-20T03:15:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 shutdown_connections 2022-07-20T03:15:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0bb80539b0 0x7f0bb8055e70 unknown :-1 s=CLOSED pgs=465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0bdc0b4d70 0x7f0bdc155850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0bdc142ac0 0x7f0bdc1471d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 --2- 172.21.15.61:0/1189379039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0bdc0ac460 0x7f0bdc065440 unknown :-1 s=CLOSED pgs=955 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 >> 172.21.15.61:0/1189379039 conn(0x7f0bdc0632b0 msgr2=0x7f0bdc0b1000 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:17.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 shutdown_connections 2022-07-20T03:15:17.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.044+0000 7f0bdb59e700 1 -- 172.21.15.61:0/1189379039 wait complete. 2022-07-20T03:15:17.076 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:15:17.077 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:17.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 Processor -- start 2022-07-20T03:15:17.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 -- start start 2022-07-20T03:15:17.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdc340b9ef0 0x7fdc340b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:17.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc340b2960 0x7fdc34142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:17.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdc340ba2d0 con 0x7fdc3405d160 2022-07-20T03:15:17.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdc340b4560 con 0x7fdc340b9ef0 2022-07-20T03:15:17.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.176+0000 7fdc392dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc340b4800 con 0x7fdc340b2960 2022-07-20T03:15:17.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc32ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdc340b9ef0 0x7fdc340b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:17.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc327fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdc3405d160 0x7fdc340b2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44886/0 (socket says 172.21.15.61:44886) 2022-07-20T03:15:17.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc327fc700 1 -- 172.21.15.61:0/2754379855 learned_addr learned my addr 172.21.15.61:0/2754379855 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:17.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc337fe700 1 --2- 172.21.15.61:0/2754379855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc340b2960 0x7fdc34142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:17.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc32ffd700 1 -- 172.21.15.61:0/2754379855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc340b2960 msgr2=0x7fdc34142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:17.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc32ffd700 1 --2- 172.21.15.61:0/2754379855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc340b2960 0x7fdc34142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc32ffd700 1 -- 172.21.15.61:0/2754379855 >> v1:172.21.15.61:6789/0 conn(0x7fdc3405d160 legacy=0x7fdc340b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc32ffd700 1 -- 172.21.15.61:0/2754379855 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc340b1db0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc32ffd700 1 --2- 172.21.15.61:0/2754379855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc340b4e50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7fdc28002f70 tx=0x7fdc28006df0).ready entity=mon.0 client_cookie=1ea21604cc1f4d85 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc31ffb700 1 -- 172.21.15.61:0/2754379855 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2800d570 con 0x7fdc340b9ef0 2022-07-20T03:15:17.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc31ffb700 1 -- 172.21.15.61:0/2754379855 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc28000e90 con 0x7fdc340b9ef0 2022-07-20T03:15:17.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc31ffb700 1 -- 172.21.15.61:0/2754379855 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2800d8c0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 -- 172.21.15.61:0/2754379855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 msgr2=0x7fdc340b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2754379855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc340b4e50 secure :-1 s=READY pgs=957 cs=0 l=1 rev1=1 rx=0x7fdc28002f70 tx=0x7fdc28006df0).stop 2022-07-20T03:15:17.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 -- 172.21.15.61:0/2754379855 shutdown_connections 2022-07-20T03:15:17.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2754379855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc340b2960 0x7fdc34142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2754379855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc340b4e50 unknown :-1 s=CLOSED pgs=957 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 -- 172.21.15.61:0/2754379855 >> 172.21.15.61:0/2754379855 conn(0x7fdc340679d0 msgr2=0x7fdc340b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:17.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 -- 172.21.15.61:0/2754379855 shutdown_connections 2022-07-20T03:15:17.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.180+0000 7fdc392dc700 1 -- 172.21.15.61:0/2754379855 wait complete. 2022-07-20T03:15:17.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 Processor -- start 2022-07-20T03:15:17.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 -- start start 2022-07-20T03:15:17.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc3405d160 0x7fdc341479e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:17.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc340b2960 0x7fdc34145000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:17.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc3413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:17.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341446e0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc341457f0 con 0x7fdc340b2960 2022-07-20T03:15:17.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc392dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdc34145a90 con 0x7fdc3405d160 2022-07-20T03:15:17.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc3413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:17.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc3413d940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48298/0 (socket says 172.21.15.61:48298) 2022-07-20T03:15:17.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 -- 172.21.15.61:0/2638679058 learned_addr learned my addr 172.21.15.61:0/2638679058 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:17.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc32ffd700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc3405d160 0x7fdc341479e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:17.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc327fc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc340b2960 0x7fdc34145000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:17.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 -- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc3405d160 msgr2=0x7fdc341479e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc3405d160 0x7fdc341479e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 -- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc340b2960 msgr2=0x7fdc34145000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc340b2960 0x7fdc34145000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.184+0000 7fdc337fe700 1 -- 172.21.15.61:0/2638679058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc2800b180 con 0x7fdc340b9ef0 2022-07-20T03:15:17.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc337fe700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc3413d940 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7fdc2c006e00 tx=0x7fdc2c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:17.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2c00c640 con 0x7fdc340b9ef0 2022-07-20T03:15:17.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdc2c00c7a0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc341454f0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdc2c00cb00 con 0x7fdc340b9ef0 2022-07-20T03:15:17.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc340af0e0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc340b3780 con 0x7fdc340b9ef0 2022-07-20T03:15:17.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdc2c022070 con 0x7fdc340b9ef0 2022-07-20T03:15:17.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc23fff700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc10053a40 0x7fdc10055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:17.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdc2c06c4c0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc32ffd700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc10053a40 0x7fdc10055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:17.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.188+0000 7fdc32ffd700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc10053a40 0x7fdc10055f00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fdc2801a000 tx=0x7fdc28014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:17.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.192+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdc2c043420 con 0x7fdc340b9ef0 2022-07-20T03:15:17.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.356+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fdc340574e0 con 0x7fdc340b9ef0 2022-07-20T03:15:17.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.356+0000 7fdc23fff700 1 -- 172.21.15.61:0/2638679058 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fdc2c046a40 con 0x7fdc340b9ef0 2022-07-20T03:15:17.361 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:17.361 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:13.467553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7008":{"gid":7008,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1014823584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1014823584},{"type":"v1","addr":"172.21.15.61:6833","nonce":1014823584}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:17.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc10053a40 msgr2=0x7fdc10055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc10053a40 0x7fdc10055f00 secure :-1 s=READY pgs=466 cs=0 l=1 rev1=1 rx=0x7fdc2801a000 tx=0x7fdc28014000).stop 2022-07-20T03:15:17.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 msgr2=0x7fdc3413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:17.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc3413d940 secure :-1 s=READY pgs=958 cs=0 l=1 rev1=1 rx=0x7fdc2c006e00 tx=0x7fdc2c0091d0).stop 2022-07-20T03:15:17.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 shutdown_connections 2022-07-20T03:15:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdc10053a40 0x7fdc10055f00 unknown :-1 s=CLOSED pgs=466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdc340b2960 0x7fdc34145000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdc340b9ef0 0x7fdc3413d940 unknown :-1 s=CLOSED pgs=958 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 --2- 172.21.15.61:0/2638679058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdc3405d160 0x7fdc341479e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:17.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 >> 172.21.15.61:0/2638679058 conn(0x7fdc340679d0 msgr2=0x7fdc340b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:17.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 shutdown_connections 2022-07-20T03:15:17.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:17.360+0000 7fdc392dc700 1 -- 172.21.15.61:0/2638679058 wait complete. 2022-07-20T03:15:17.369 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-20T03:15:17.579 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:12.462538+0000', 'modified': '2022-07-20T03:15:13.467553+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7008}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7008': {'gid': 7008, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1014823584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1014823584}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1014823584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:15:17.580 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:15:17.581 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:15:17.703 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7008,"want_state":"up:active","state":"up:active","rank_uptime":5.2253034989999998,"mdsmap_epoch":51,"osdmap_epoch":91,"osdmap_epoch_barrier":89,"uptime":9.5971626810000004} 2022-07-20T03:15:17.704 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7008, "mdsmap_epoch": 51, "osdmap_epoch": 91, "osdmap_epoch_barrier": 89, "rank_uptime": 5.225303499, "state": "up:active", "uptime": 9.597162681, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:15:17.706 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:15:17.706 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:15:17.707 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:15:17.707 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:17.708 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:17.717 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:15:17.737 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:17.738 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:17.792 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:17.795 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:17.797 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:17.798 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:15:17.807 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:15:17.807 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:17.855 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:15:17.890 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:17.891 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:17.914 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:17.916 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:17.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:17.916 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:15:17.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:15:17.921+0000 7ff5c8761080 -1 init, newargv = 0x55b408508080 newargc=9ceph-fuse[23158 2022-07-20T03:15:17.963 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:]: starting ceph client 2022-07-20T03:15:17.964 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23158]: starting fuse 2022-07-20T03:15:17.970 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:15:17.971 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:15:17.972 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:17.973 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:15:17.978 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:15:17.979 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:15:17.980 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:15:17.981 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:15:17.982 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:17.983 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:15:17.984 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:15:17.985 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:15:17.986 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:17.987 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:15:17.988 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:15:17.989 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:15:17.989 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:15:17.990 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:17.991 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:15:17.998 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:15:17.999 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:15:17.999 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:15:18.000 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:15:18.000 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:18.001 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:15:18.001 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:15:18.075 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23158.asok 2022-07-20T03:15:18.076 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23158.asok 2022-07-20T03:15:18.077 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:18.077 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23158.asok status 2022-07-20T03:15:18.170 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:15:18.170 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:15:18.171 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:15:18.171 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:15:18.172 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:15:18.172 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:15:18.173 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:15:18.174 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23158", 2022-07-20T03:15:18.174 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:15:18.175 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:18.175 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:15:18.176 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:15:18.177 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 7182, 2022-07-20T03:15:18.177 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:15:18.179 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:15:18.180 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:15:18.180 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 7182 2022-07-20T03:15:18.181 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:18.181 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:15:18.182 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:15:18.182 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:15:18.183 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 438534984 2022-07-20T03:15:18.183 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:15:18.184 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:18.184 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:15:18.185 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:15:18.185 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:15:18.186 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 438534984 2022-07-20T03:15:18.194 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:18.195 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.7182 172.21.15.124:0/438534984", 2022-07-20T03:15:18.195 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/438534984", 2022-07-20T03:15:18.196 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:15:18.196 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 51, 2022-07-20T03:15:18.197 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 91, 2022-07-20T03:15:18.197 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:15:18.198 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:15:18.198 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:15:18.199 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:15:18.201 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:18.202 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:18.203 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:18.217 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:15:18.218 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:18.219 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:18.220 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:18.280 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:18.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.392+0000 7f7d96d24700 1 Processor -- start 2022-07-20T03:15:18.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.392+0000 7f7d96d24700 1 -- start start 2022-07-20T03:15:18.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7d9005c3f0 0x7f7d9014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d9005d480 con 0x7f7d900acef0 2022-07-20T03:15:18.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7d900b21e0 con 0x7f7d900b1e00 2022-07-20T03:15:18.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7d9005cf40 con 0x7f7d9005c3f0 2022-07-20T03:15:18.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d952c1700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7d900acef0 0x7f7d90146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44904/0 (socket says 172.21.15.61:44904) 2022-07-20T03:15:18.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d952c1700 1 -- 172.21.15.61:0/2772868974 learned_addr learned my addr 172.21.15.61:0/2772868974 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:18.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8ffff700 1 --2- 172.21.15.61:0/2772868974 >> v2:172.21.15.117:3300/0 conn(0x7f7d9005c3f0 0x7f7d9014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8f7fe700 1 -- 172.21.15.61:0/2772868974 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4039162014 0 0) 0x7f7d9005d480 con 0x7f7d900acef0 2022-07-20T03:15:18.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8f7fe700 1 -- 172.21.15.61:0/2772868974 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7d7c003260 con 0x7f7d900acef0 2022-07-20T03:15:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2772868974 >> v1:172.21.15.117:6789/0 conn(0x7f7d900b1e00 legacy=0x7f7d900ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2772868974 >> v1:172.21.15.61:6789/0 conn(0x7f7d900acef0 legacy=0x7f7d90146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8f7fe700 1 -- 172.21.15.61:0/2772868974 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3138281419 0 0) 0x7f7d7c003260 con 0x7f7d900acef0 2022-07-20T03:15:18.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2772868974 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d9005d1e0 con 0x7f7d9005c3f0 2022-07-20T03:15:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8ffff700 1 --2- 172.21.15.61:0/2772868974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d9005c3f0 0x7f7d9014a6d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f7d88001b80 tx=0x7f7d80006df0).ready entity=mon.1 client_cookie=5b9685f60010ffa7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8f7fe700 1 -- 172.21.15.61:0/2772868974 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d880019e0 con 0x7f7d9005c3f0 2022-07-20T03:15:18.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8f7fe700 1 -- 172.21.15.61:0/2772868974 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80000e90 con 0x7f7d9005c3f0 2022-07-20T03:15:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d8f7fe700 1 -- 172.21.15.61:0/2772868974 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d8000d4f0 con 0x7f7d9005c3f0 2022-07-20T03:15:18.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- 172.21.15.61:0/2772868974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d9005c3f0 msgr2=0x7f7d9014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2772868974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d9005c3f0 0x7f7d9014a6d0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f7d88001b80 tx=0x7f7d80006df0).stop 2022-07-20T03:15:18.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- 172.21.15.61:0/2772868974 shutdown_connections 2022-07-20T03:15:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2772868974 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d9005c3f0 0x7f7d9014a6d0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- 172.21.15.61:0/2772868974 >> 172.21.15.61:0/2772868974 conn(0x7f7d900611c0 msgr2=0x7f7d90062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- 172.21.15.61:0/2772868974 shutdown_connections 2022-07-20T03:15:18.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.396+0000 7f7d96d24700 1 -- 172.21.15.61:0/2772868974 wait complete. 2022-07-20T03:15:18.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 Processor -- start 2022-07-20T03:15:18.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 -- start start 2022-07-20T03:15:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d900acef0 0x7f7d9014cbd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 0x7f7d90142320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d900b58b0 0x7f7d90142810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d90067930 con 0x7f7d900b1e00 2022-07-20T03:15:18.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900b4ed0 con 0x7f7d900b58b0 2022-07-20T03:15:18.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7d900b5170 con 0x7f7d900acef0 2022-07-20T03:15:18.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 0x7f7d90142320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 0x7f7d90142320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48310/0 (socket says 172.21.15.61:48310) 2022-07-20T03:15:18.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2512129347 learned_addr learned my addr 172.21.15.61:0/2512129347 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:18.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d94ac0700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d900acef0 0x7f7d9014cbd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d952c1700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d900b58b0 0x7f7d90142810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d900acef0 msgr2=0x7f7d9014cbd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d900acef0 0x7f7d9014cbd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d900b58b0 msgr2=0x7f7d90142810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d900b58b0 0x7f7d90142810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 -- 172.21.15.61:0/2512129347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d8000b180 con 0x7f7d900b1e00 2022-07-20T03:15:18.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d952c1700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d900b58b0 0x7f7d90142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:18.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8ffff700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 0x7f7d90142320 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f7d88001b80 tx=0x7f7d80014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:18.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d8000d4f0 con 0x7f7d900b1e00 2022-07-20T03:15:18.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7d80017070 con 0x7f7d900b1e00 2022-07-20T03:15:18.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7d8001f650 con 0x7f7d900b1e00 2022-07-20T03:15:18.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d900b5410 con 0x7f7d900b1e00 2022-07-20T03:15:18.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.400+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d900b3f10 con 0x7f7d900b1e00 2022-07-20T03:15:18.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.404+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d900b55a0 con 0x7f7d900b1e00 2022-07-20T03:15:18.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.404+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7d8000f030 con 0x7f7d900b1e00 2022-07-20T03:15:18.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.404+0000 7f7d8dffb700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d70053ad0 0x7f7d70055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.404+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7d8006cd40 con 0x7f7d900b1e00 2022-07-20T03:15:18.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.404+0000 7f7d94ac0700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d70053ad0 0x7f7d70055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.404+0000 7f7d94ac0700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d70053ad0 0x7f7d70055f90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f7d8400d000 tx=0x7f7d84007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:18.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.412+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7d80043c00 con 0x7f7d900b1e00 2022-07-20T03:15:18.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.540+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7d900b04b0 con 0x7f7d900b1e00 2022-07-20T03:15:18.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.544+0000 7f7d8dffb700 1 -- 172.21.15.61:0/2512129347 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v91) v1 ==== 74+0+16089 (secure 0 0 0) 0x7f7d80024030 con 0x7f7d900b1e00 2022-07-20T03:15:18.547 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:18.548 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":91,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:14.477402+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":17,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:11.081553+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"89","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":18,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:11.541101+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"91","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:18.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d70053ad0 msgr2=0x7f7d70055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d70053ad0 0x7f7d70055f90 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7f7d8400d000 tx=0x7f7d84007b90).stop 2022-07-20T03:15:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 msgr2=0x7f7d90142320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 0x7f7d90142320 secure :-1 s=READY pgs=962 cs=0 l=1 rev1=1 rx=0x7f7d88001b80 tx=0x7f7d80014000).stop 2022-07-20T03:15:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 shutdown_connections 2022-07-20T03:15:18.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7d70053ad0 0x7f7d70055f90 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7d900b58b0 0x7f7d90142810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7d900b1e00 0x7f7d90142320 unknown :-1 s=CLOSED pgs=962 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 --2- 172.21.15.61:0/2512129347 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7d900acef0 0x7f7d9014cbd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 >> 172.21.15.61:0/2512129347 conn(0x7f7d9005c3f0 msgr2=0x7f7d901461e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:18.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 shutdown_connections 2022-07-20T03:15:18.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.548+0000 7f7d96d24700 1 -- 172.21.15.61:0/2512129347 wait complete. 2022-07-20T03:15:18.568 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 17, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:15:11.081553+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '89', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 18, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:15:11.541101+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '91', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:15:18.570 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:18.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8c4946700 1 Processor -- start 2022-07-20T03:15:18.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8c4946700 1 -- start start 2022-07-20T03:15:18.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8c4946700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c01468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8c4946700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c00ba180 con 0x7fb8c01419a0 2022-07-20T03:15:18.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8c4946700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8c01461b0 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8c4946700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00ad5b0 con 0x7fb8c00b9da0 2022-07-20T03:15:18.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bf59e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb8c01419a0 0x7fb8c0141e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44918/0 (socket says 172.21.15.61:44918) 2022-07-20T03:15:18.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bf59e700 1 -- 172.21.15.61:0/3400220356 learned_addr learned my addr 172.21.15.61:0/3400220356 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:18.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bed9d700 1 --2- 172.21.15.61:0/3400220356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c01468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bdd9b700 1 -- 172.21.15.61:0/3400220356 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 956659214 0 0) 0x7fb8c00ba180 con 0x7fb8c01419a0 2022-07-20T03:15:18.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bdd9b700 1 -- 172.21.15.61:0/3400220356 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8a4003260 con 0x7fb8c01419a0 2022-07-20T03:15:18.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bdd9b700 1 -- 172.21.15.61:0/3400220356 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2854462625 0 0) 0x7fb8c01461b0 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bed9d700 1 -- 172.21.15.61:0/3400220356 >> v1:172.21.15.117:6789/0 conn(0x7fb8c00b3cf0 legacy=0x7fb8c0142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bed9d700 1 -- 172.21.15.61:0/3400220356 >> v1:172.21.15.61:6789/0 conn(0x7fb8c01419a0 legacy=0x7fb8c0141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bed9d700 1 -- 172.21.15.61:0/3400220356 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c0146450 con 0x7fb8c00b9da0 2022-07-20T03:15:18.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.680+0000 7fb8bed9d700 1 --2- 172.21.15.61:0/3400220356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c01468f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb8b0002f70 tx=0x7fb8b0006df0).ready entity=mon.2 client_cookie=27327fd2ac0255da server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:18.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8bdd9b700 1 -- 172.21.15.61:0/3400220356 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b000d570 con 0x7fb8c00b9da0 2022-07-20T03:15:18.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8bdd9b700 1 -- 172.21.15.61:0/3400220356 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b0000e90 con 0x7fb8c00b9da0 2022-07-20T03:15:18.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8bdd9b700 1 -- 172.21.15.61:0/3400220356 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b000d8c0 con 0x7fb8c00b9da0 2022-07-20T03:15:18.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- 172.21.15.61:0/3400220356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 msgr2=0x7fb8c01468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 --2- 172.21.15.61:0/3400220356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c01468f0 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fb8b0002f70 tx=0x7fb8b0006df0).stop 2022-07-20T03:15:18.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- 172.21.15.61:0/3400220356 shutdown_connections 2022-07-20T03:15:18.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 --2- 172.21.15.61:0/3400220356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c01468f0 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- 172.21.15.61:0/3400220356 >> 172.21.15.61:0/3400220356 conn(0x7fb8c00678c0 msgr2=0x7fb8c00b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:18.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- 172.21.15.61:0/3400220356 shutdown_connections 2022-07-20T03:15:18.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- 172.21.15.61:0/3400220356 wait complete. 2022-07-20T03:15:18.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 Processor -- start 2022-07-20T03:15:18.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- start start 2022-07-20T03:15:18.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 0x7fb8c0113f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c0154e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8c01419a0 0x7fb8c0148210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00b0510 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00b0730 con 0x7fb8c01419a0 2022-07-20T03:15:18.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.684+0000 7fb8c4946700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c00b09d0 con 0x7fb8c00b9da0 2022-07-20T03:15:18.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 0x7fb8c0113f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 0x7fb8c0113f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48322/0 (socket says 172.21.15.61:48322) 2022-07-20T03:15:18.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 -- 172.21.15.61:0/1432874077 learned_addr learned my addr 172.21.15.61:0/1432874077 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:18.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bf59e700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8c01419a0 0x7fb8c0148210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8be59c700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c0154e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 -- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 msgr2=0x7fb8c0154e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c0154e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 -- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8c01419a0 msgr2=0x7fb8c0148210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8c01419a0 0x7fb8c0148210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 -- 172.21.15.61:0/1432874077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8b000b180 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8bed9d700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 0x7fb8c0113f40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fb8b0002f70 tx=0x7fb8b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:18.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b0016070 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8b0013850 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c00ae5e0 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8b0013bb0 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c00b2310 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c00ae770 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb8b0025030 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8affff700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb89c0539b0 0x7fb89c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:18.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8b006ccd0 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.688+0000 7fb8be59c700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb89c0539b0 0x7fb89c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:18.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.692+0000 7fb8be59c700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb89c0539b0 0x7fb89c055e70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb8b400d000 tx=0x7fb8b4007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:18.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.696+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8b0043c30 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.820+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb8c00699b0 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8affff700 1 -- 172.21.15.61:0/1432874077 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v91) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb8b0023020 con 0x7fb8c00b3cf0 2022-07-20T03:15:18.831 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:18.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb89c0539b0 msgr2=0x7fb89c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb89c0539b0 0x7fb89c055e70 secure :-1 s=READY pgs=468 cs=0 l=1 rev1=1 rx=0x7fb8b400d000 tx=0x7fb8b4007a00).stop 2022-07-20T03:15:18.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 msgr2=0x7fb8c0113f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:18.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 0x7fb8c0113f40 secure :-1 s=READY pgs=964 cs=0 l=1 rev1=1 rx=0x7fb8b0002f70 tx=0x7fb8b0014000).stop 2022-07-20T03:15:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 shutdown_connections 2022-07-20T03:15:18.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb89c0539b0 0x7fb89c055e70 unknown :-1 s=CLOSED pgs=468 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8c01419a0 0x7fb8c0148210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8c00b9da0 0x7fb8c0154e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 --2- 172.21.15.61:0/1432874077 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8c00b3cf0 0x7fb8c0113f40 unknown :-1 s=CLOSED pgs=964 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 >> 172.21.15.61:0/1432874077 conn(0x7fb8c00678c0 msgr2=0x7fb8c00b8a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:18.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 shutdown_connections 2022-07-20T03:15:18.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.824+0000 7fb8c4946700 1 -- 172.21.15.61:0/1432874077 wait complete. 2022-07-20T03:15:18.856 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:18.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 Processor -- start 2022-07-20T03:15:18.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 -- start start 2022-07-20T03:15:19.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f601422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f601427b0 0x7f9f60060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f600afdb0 con 0x7f9f601462a0 2022-07-20T03:15:19.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f600b0050 con 0x7f9f601427b0 2022-07-20T03:15:19.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f659c2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6005fcd0 con 0x7f9f600988b0 2022-07-20T03:15:19.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f5f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f601427b0 0x7f9f60060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f5f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f601427b0 0x7f9f60060170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48332/0 (socket says 172.21.15.61:48332) 2022-07-20T03:15:19.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/3083547288 learned_addr learned my addr 172.21.15.61:0/3083547288 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.984+0000 7f9f5effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9f601462a0 0x7f9f60146680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44928/0 (socket says 172.21.15.61:44928) 2022-07-20T03:15:19.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5e7fc700 1 --2- 172.21.15.61:0/3083547288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f601422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/3083547288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 msgr2=0x7f9f601422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:19.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5f7fe700 1 --2- 172.21.15.61:0/3083547288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f601422c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/3083547288 >> v1:172.21.15.61:6789/0 conn(0x7f9f601462a0 legacy=0x7f9f60146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5dffb700 1 -- 172.21.15.61:0/3083547288 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2120588550 0 0) 0x7f9f600afdb0 con 0x7f9f601462a0 2022-07-20T03:15:19.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/3083547288 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f600afae0 con 0x7f9f601427b0 2022-07-20T03:15:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5f7fe700 1 --2- 172.21.15.61:0/3083547288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601427b0 0x7f9f60060170 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f9f580032f0 tx=0x7f9f58009290).ready entity=mon.0 client_cookie=a9507a4b1fda758b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5dffb700 1 -- 172.21.15.61:0/3083547288 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5800b8a0 con 0x7f9f601427b0 2022-07-20T03:15:19.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5dffb700 1 -- 172.21.15.61:0/3083547288 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f5800ba00 con 0x7f9f601427b0 2022-07-20T03:15:19.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f5dffb700 1 -- 172.21.15.61:0/3083547288 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5800bb70 con 0x7f9f601427b0 2022-07-20T03:15:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- 172.21.15.61:0/3083547288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601427b0 msgr2=0x7f9f60060170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 --2- 172.21.15.61:0/3083547288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601427b0 0x7f9f60060170 secure :-1 s=READY pgs=966 cs=0 l=1 rev1=1 rx=0x7f9f580032f0 tx=0x7f9f58009290).stop 2022-07-20T03:15:19.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- 172.21.15.61:0/3083547288 shutdown_connections 2022-07-20T03:15:19.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 --2- 172.21.15.61:0/3083547288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601427b0 0x7f9f60060170 unknown :-1 s=CLOSED pgs=966 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 --2- 172.21.15.61:0/3083547288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f601422c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- 172.21.15.61:0/3083547288 >> 172.21.15.61:0/3083547288 conn(0x7f9f600b5f00 msgr2=0x7f9f600b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- 172.21.15.61:0/3083547288 shutdown_connections 2022-07-20T03:15:19.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- 172.21.15.61:0/3083547288 wait complete. 2022-07-20T03:15:19.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 Processor -- start 2022-07-20T03:15:19.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- start start 2022-07-20T03:15:19.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f60148830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f601427b0 0x7f9f60155c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 0x7f9f60151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f6009cc70 con 0x7f9f601462a0 2022-07-20T03:15:19.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f60155530 con 0x7f9f601427b0 2022-07-20T03:15:19.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.988+0000 7f9f659c2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f601557d0 con 0x7f9f600988b0 2022-07-20T03:15:19.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 0x7f9f60151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 0x7f9f60151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48338/0 (socket says 172.21.15.61:48338) 2022-07-20T03:15:19.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/1163185101 learned_addr learned my addr 172.21.15.61:0/1163185101 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5e7fc700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f601427b0 0x7f9f60155c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5effd700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f60148830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 msgr2=0x7f9f60148830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f60148830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f601427b0 msgr2=0x7f9f60155c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f601427b0 0x7f9f60155c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 -- 172.21.15.61:0/1163185101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f58009530 con 0x7f9f601462a0 2022-07-20T03:15:19.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f5f7fe700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 0x7f9f60151f00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f9f580032f0 tx=0x7f9f58016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f5800b8a0 con 0x7f9f601462a0 2022-07-20T03:15:19.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f58021630 con 0x7f9f601462a0 2022-07-20T03:15:19.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f6009c930 con 0x7f9f601462a0 2022-07-20T03:15:19.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f58021990 con 0x7f9f601462a0 2022-07-20T03:15:19.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f600b24b0 con 0x7f9f601462a0 2022-07-20T03:15:19.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f6009cac0 con 0x7f9f601462a0 2022-07-20T03:15:19.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.992+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f58029070 con 0x7f9f601462a0 2022-07-20T03:15:19.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.996+0000 7f9f4ffff700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053a60 0x7f9f3c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.996+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f5806cf00 con 0x7f9f601462a0 2022-07-20T03:15:19.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.996+0000 7f9f5effd700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053a60 0x7f9f3c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:18.996+0000 7f9f5effd700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053a60 0x7f9f3c055f20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f9f5400b000 tx=0x7f9f54007160).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.000+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f58043dc0 con 0x7f9f601462a0 2022-07-20T03:15:19.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.128+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9f60146d90 con 0x7f9f601462a0 2022-07-20T03:15:19.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.128+0000 7f9f4ffff700 1 -- 172.21.15.61:0/1163185101 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v91) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9f58015020 con 0x7f9f601462a0 2022-07-20T03:15:19.141 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:19.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053a60 msgr2=0x7f9f3c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053a60 0x7f9f3c055f20 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f9f5400b000 tx=0x7f9f54007160).stop 2022-07-20T03:15:19.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 msgr2=0x7f9f60151f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 0x7f9f60151f00 secure :-1 s=READY pgs=967 cs=0 l=1 rev1=1 rx=0x7f9f580032f0 tx=0x7f9f58016000).stop 2022-07-20T03:15:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 shutdown_connections 2022-07-20T03:15:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f3c053a60 0x7f9f3c055f20 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f601427b0 0x7f9f60155c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f601462a0 0x7f9f60151f00 unknown :-1 s=CLOSED pgs=967 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 --2- 172.21.15.61:0/1163185101 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f600988b0 0x7f9f60148830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 >> 172.21.15.61:0/1163185101 conn(0x7f9f600b5f00 msgr2=0x7f9f6005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 shutdown_connections 2022-07-20T03:15:19.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.132+0000 7f9f659c2700 1 -- 172.21.15.61:0/1163185101 wait complete. 2022-07-20T03:15:19.152 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:15:19.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 Processor -- start 2022-07-20T03:15:19.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 -- start start 2022-07-20T03:15:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcd740b1280 0x7fcd7413c3e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 0x7fcd7413c8d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd740ac900 con 0x7fcd740b0480 2022-07-20T03:15:19.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd7413cdc0 con 0x7fcd740b1280 2022-07-20T03:15:19.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7baa1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd740b09b0 con 0x7fcd740b1660 2022-07-20T03:15:19.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 0x7fcd7413c8d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7983d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcd740b1280 0x7fcd7413c3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 0x7fcd7413c8d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35020/0 (socket says 172.21.15.61:35020) 2022-07-20T03:15:19.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 -- 172.21.15.61:0/3421322048 learned_addr learned my addr 172.21.15.61:0/3421322048 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 -- 172.21.15.61:0/3421322048 >> v2:172.21.15.117:3300/0 conn(0x7fcd740b1280 msgr2=0x7fcd7413c3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:19.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 --2- 172.21.15.61:0/3421322048 >> v2:172.21.15.117:3300/0 conn(0x7fcd740b1280 0x7fcd7413c3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 -- 172.21.15.61:0/3421322048 >> v1:172.21.15.117:6789/0 conn(0x7fcd740b0480 legacy=0x7fcd7413f0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 -- 172.21.15.61:0/3421322048 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd740aef30 con 0x7fcd740b1660 2022-07-20T03:15:19.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7a03e700 1 --2- 172.21.15.61:0/3421322048 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 0x7fcd7413c8d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fcd6c0032f0 tx=0x7fcd6c00a430).ready entity=mon.2 client_cookie=cb85f1875f5388ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7883b700 1 -- 172.21.15.61:0/3421322048 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd6c00cc70 con 0x7fcd740b1660 2022-07-20T03:15:19.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7883b700 1 -- 172.21.15.61:0/3421322048 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd6c00cdd0 con 0x7fcd740b1660 2022-07-20T03:15:19.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.260+0000 7fcd7883b700 1 -- 172.21.15.61:0/3421322048 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd6c010600 con 0x7fcd740b1660 2022-07-20T03:15:19.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- 172.21.15.61:0/3421322048 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 msgr2=0x7fcd7413c8d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/3421322048 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 0x7fcd7413c8d0 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7fcd6c0032f0 tx=0x7fcd6c00a430).stop 2022-07-20T03:15:19.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- 172.21.15.61:0/3421322048 shutdown_connections 2022-07-20T03:15:19.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/3421322048 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b1660 0x7fcd7413c8d0 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/3421322048 >> v2:172.21.15.117:3300/0 conn(0x7fcd740b1280 0x7fcd7413c3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- 172.21.15.61:0/3421322048 >> 172.21.15.61:0/3421322048 conn(0x7fcd740635e0 msgr2=0x7fcd740b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- 172.21.15.61:0/3421322048 shutdown_connections 2022-07-20T03:15:19.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- 172.21.15.61:0/3421322048 wait complete. 2022-07-20T03:15:19.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 Processor -- start 2022-07-20T03:15:19.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- start start 2022-07-20T03:15:19.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b0480 0x7fcd74065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 0x7fcd74063c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd740b1660 0x7fcd74142fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7409e080 con 0x7fcd740b1280 2022-07-20T03:15:19.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7409e320 con 0x7fcd740b1660 2022-07-20T03:15:19.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7baa1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd7409e5c0 con 0x7fcd740b0480 2022-07-20T03:15:19.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 0x7fcd74063c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 0x7fcd74063c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48352/0 (socket says 172.21.15.61:48352) 2022-07-20T03:15:19.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 -- 172.21.15.61:0/311803824 learned_addr learned my addr 172.21.15.61:0/311803824 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7983d700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b0480 0x7fcd74065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7a03e700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd740b1660 0x7fcd74142fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 -- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b0480 msgr2=0x7fcd74065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b0480 0x7fcd74065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 -- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd740b1660 msgr2=0x7fcd74142fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd740b1660 0x7fcd74142fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.264+0000 7fcd7903c700 1 -- 172.21.15.61:0/311803824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd6c00a6d0 con 0x7fcd740b1280 2022-07-20T03:15:19.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd7903c700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 0x7fcd74063c60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fcd74062710 tx=0x7fcd64007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd64012400 con 0x7fcd740b1280 2022-07-20T03:15:19.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd64012560 con 0x7fcd740b1280 2022-07-20T03:15:19.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd7413f500 con 0x7fcd740b1280 2022-07-20T03:15:19.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd640128c0 con 0x7fcd740b1280 2022-07-20T03:15:19.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd74064ea0 con 0x7fcd740b1280 2022-07-20T03:15:19.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd740af1d0 con 0x7fcd740b1280 2022-07-20T03:15:19.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcd64019030 con 0x7fcd740b1280 2022-07-20T03:15:19.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd62ffd700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd54053a40 0x7fcd54055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.268+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcd6406b1b0 con 0x7fcd740b1280 2022-07-20T03:15:19.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.272+0000 7fcd7983d700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd54053a40 0x7fcd54055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.272+0000 7fcd7983d700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd54053a40 0x7fcd54055f00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7fcd7000b000 tx=0x7fcd70006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.276+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd64042110 con 0x7fcd740b1280 2022-07-20T03:15:19.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.404+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fcd740574e0 con 0x7fcd740b1280 2022-07-20T03:15:19.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.404+0000 7fcd62ffd700 1 -- 172.21.15.61:0/311803824 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v91) v1 ==== 104+0+10 (secure 0 0 0) 0x7fcd64045730 con 0x7fcd740b1280 2022-07-20T03:15:19.408 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:19.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.404+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd54053a40 msgr2=0x7fcd54055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.404+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd54053a40 0x7fcd54055f00 secure :-1 s=READY pgs=470 cs=0 l=1 rev1=1 rx=0x7fcd7000b000 tx=0x7fcd70006df0).stop 2022-07-20T03:15:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.404+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 msgr2=0x7fcd74063c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.404+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 0x7fcd74063c60 secure :-1 s=READY pgs=968 cs=0 l=1 rev1=1 rx=0x7fcd74062710 tx=0x7fcd64007c50).stop 2022-07-20T03:15:19.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 shutdown_connections 2022-07-20T03:15:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd54053a40 0x7fcd54055f00 unknown :-1 s=CLOSED pgs=470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd740b1660 0x7fcd74142fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd740b1280 0x7fcd74063c60 unknown :-1 s=CLOSED pgs=968 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 --2- 172.21.15.61:0/311803824 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd740b0480 0x7fcd74065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 >> 172.21.15.61:0/311803824 conn(0x7fcd740635e0 msgr2=0x7fcd7405cd40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 shutdown_connections 2022-07-20T03:15:19.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.408+0000 7fcd7baa1700 1 -- 172.21.15.61:0/311803824 wait complete. 2022-07-20T03:15:19.428 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:15:19.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 Processor -- start 2022-07-20T03:15:19.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 -- start start 2022-07-20T03:15:19.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350146040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f63500b14f0 0x7f6350146530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f63500afbc0 con 0x7f63500b59d0 2022-07-20T03:15:19.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6350148080 con 0x7f63500b14f0 2022-07-20T03:15:19.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f6354880700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f63500b00e0 con 0x7f635014ada0 2022-07-20T03:15:19.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.532+0000 7f634ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f63500b14f0 0x7f6350146530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f63500b14f0 0x7f6350146530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48360/0 (socket says 172.21.15.61:48360) 2022-07-20T03:15:19.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f63500b59d0 0x7f63500b07e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44956/0 (socket says 172.21.15.61:44956) 2022-07-20T03:15:19.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 -- 172.21.15.61:0/2180568687 learned_addr learned my addr 172.21.15.61:0/2180568687 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634effd700 1 --2- 172.21.15.61:0/2180568687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 -- 172.21.15.61:0/2180568687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 msgr2=0x7f6350146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:19.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 --2- 172.21.15.61:0/2180568687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350146040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 -- 172.21.15.61:0/2180568687 >> v1:172.21.15.61:6789/0 conn(0x7f63500b59d0 legacy=0x7f63500b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 -- 172.21.15.61:0/2180568687 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63500b5db0 con 0x7f63500b14f0 2022-07-20T03:15:19.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634ffff700 1 --2- 172.21.15.61:0/2180568687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350146530 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f6340002f70 tx=0x7f6340006eb0).ready entity=mon.0 client_cookie=9a595e2dd9092d2a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634e7fc700 1 -- 172.21.15.61:0/2180568687 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f634000d5c0 con 0x7f63500b14f0 2022-07-20T03:15:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634e7fc700 1 -- 172.21.15.61:0/2180568687 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6340000e90 con 0x7f63500b14f0 2022-07-20T03:15:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f634e7fc700 1 -- 172.21.15.61:0/2180568687 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f634000d950 con 0x7f63500b14f0 2022-07-20T03:15:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 -- 172.21.15.61:0/2180568687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 msgr2=0x7f6350146530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 --2- 172.21.15.61:0/2180568687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350146530 secure :-1 s=READY pgs=970 cs=0 l=1 rev1=1 rx=0x7f6340002f70 tx=0x7f6340006eb0).stop 2022-07-20T03:15:19.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 -- 172.21.15.61:0/2180568687 shutdown_connections 2022-07-20T03:15:19.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 --2- 172.21.15.61:0/2180568687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350146530 unknown :-1 s=CLOSED pgs=970 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 --2- 172.21.15.61:0/2180568687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350146040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 -- 172.21.15.61:0/2180568687 >> 172.21.15.61:0/2180568687 conn(0x7f63500634c0 msgr2=0x7f63500b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 -- 172.21.15.61:0/2180568687 shutdown_connections 2022-07-20T03:15:19.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 -- 172.21.15.61:0/2180568687 wait complete. 2022-07-20T03:15:19.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 Processor -- start 2022-07-20T03:15:19.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.536+0000 7f6354880700 1 -- start start 2022-07-20T03:15:19.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350067360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63500b59d0 0x7f6350111220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350111710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350153d60 con 0x7f63500b14f0 2022-07-20T03:15:19.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6350111c00 con 0x7f63500b59d0 2022-07-20T03:15:19.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6350156f30 con 0x7f635014ada0 2022-07-20T03:15:19.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350067360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350067360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48364/0 (socket says 172.21.15.61:48364) 2022-07-20T03:15:19.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 -- 172.21.15.61:0/4241185147 learned_addr learned my addr 172.21.15.61:0/4241185147 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634ffff700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350111710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634effd700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63500b59d0 0x7f6350111220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 -- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 msgr2=0x7f6350111710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350111710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 -- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63500b59d0 msgr2=0x7f6350111220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63500b59d0 0x7f6350111220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 -- 172.21.15.61:0/4241185147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f634000b1e0 con 0x7f63500b14f0 2022-07-20T03:15:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634f7fe700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350067360 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f63500639a0 tx=0x7f6348007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6348004ab0 con 0x7f63500b14f0 2022-07-20T03:15:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6348004c10 con 0x7f63500b14f0 2022-07-20T03:15:19.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63501571d0 con 0x7f63500b14f0 2022-07-20T03:15:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6348007170 con 0x7f63500b14f0 2022-07-20T03:15:19.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6350112070 con 0x7f63500b14f0 2022-07-20T03:15:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f634801b070 con 0x7f63500b14f0 2022-07-20T03:15:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.540+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63500b0340 con 0x7f63500b14f0 2022-07-20T03:15:19.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.544+0000 7f634cff9700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6334053920 0x7f6334055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.544+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f634806b730 con 0x7f63500b14f0 2022-07-20T03:15:19.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.544+0000 7f634effd700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6334053920 0x7f6334055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.544+0000 7f634effd700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6334053920 0x7f6334055de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f633800c000 tx=0x7f6338008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.548+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f63480425f0 con 0x7f63500b14f0 2022-07-20T03:15:19.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.676+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f63500574e0 con 0x7f63500b14f0 2022-07-20T03:15:19.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.676+0000 7f634cff9700 1 -- 172.21.15.61:0/4241185147 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v91) v1 ==== 100+0+10 (secure 0 0 0) 0x7f6348045c10 con 0x7f63500b14f0 2022-07-20T03:15:19.683 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:19.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6334053920 msgr2=0x7f6334055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6334053920 0x7f6334055de0 secure :-1 s=READY pgs=471 cs=0 l=1 rev1=1 rx=0x7f633800c000 tx=0x7f6338008040).stop 2022-07-20T03:15:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 msgr2=0x7f6350067360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350067360 secure :-1 s=READY pgs=971 cs=0 l=1 rev1=1 rx=0x7f63500639a0 tx=0x7f6348007f70).stop 2022-07-20T03:15:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 shutdown_connections 2022-07-20T03:15:19.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6334053920 0x7f6334055de0 unknown :-1 s=CLOSED pgs=471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f63500b59d0 0x7f6350111220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f635014ada0 0x7f6350111710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 --2- 172.21.15.61:0/4241185147 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f63500b14f0 0x7f6350067360 unknown :-1 s=CLOSED pgs=971 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 >> 172.21.15.61:0/4241185147 conn(0x7f63500634c0 msgr2=0x7f635014a530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 shutdown_connections 2022-07-20T03:15:19.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.680+0000 7f6354880700 1 -- 172.21.15.61:0/4241185147 wait complete. 2022-07-20T03:15:19.694 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:15:19.695 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:19.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8759e700 1 Processor -- start 2022-07-20T03:15:19.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8759e700 1 -- start start 2022-07-20T03:15:19.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fee88063d60 0x7fee880b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee88064140 con 0x7fee880b1440 2022-07-20T03:15:19.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8759e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fee880ad900 con 0x7fee880af150 2022-07-20T03:15:19.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fee88143800 con 0x7fee88063d60 2022-07-20T03:15:19.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee8659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fee88063d60 0x7fee880b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee7ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fee880b1440 0x7fee880acae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44970/0 (socket says 172.21.15.61:44970) 2022-07-20T03:15:19.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.796+0000 7fee7ffff700 1 -- 172.21.15.61:0/3963955984 learned_addr learned my addr 172.21.15.61:0/3963955984 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:19.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee85d9b700 1 -- 172.21.15.61:0/3963955984 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2183000110 0 0) 0x7fee88064140 con 0x7fee880b1440 2022-07-20T03:15:19.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee85d9b700 1 -- 172.21.15.61:0/3963955984 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fee68003260 con 0x7fee880b1440 2022-07-20T03:15:19.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8659c700 1 -- 172.21.15.61:0/3963955984 >> v1:172.21.15.117:6789/0 conn(0x7fee880af150 legacy=0x7fee880ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8659c700 1 -- 172.21.15.61:0/3963955984 >> v1:172.21.15.61:6789/0 conn(0x7fee880b1440 legacy=0x7fee880acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8659c700 1 -- 172.21.15.61:0/3963955984 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee88143aa0 con 0x7fee88063d60 2022-07-20T03:15:19.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8659c700 1 --2- 172.21.15.61:0/3963955984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88063d60 0x7fee880b57e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7fee78002f70 tx=0x7fee78006df0).ready entity=mon.0 client_cookie=104fdab714013048 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee85d9b700 1 -- 172.21.15.61:0/3963955984 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee7800d610 con 0x7fee88063d60 2022-07-20T03:15:19.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee85d9b700 1 -- 172.21.15.61:0/3963955984 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee78000e90 con 0x7fee88063d60 2022-07-20T03:15:19.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee85d9b700 1 -- 172.21.15.61:0/3963955984 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee7800d960 con 0x7fee88063d60 2022-07-20T03:15:19.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- 172.21.15.61:0/3963955984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88063d60 msgr2=0x7fee880b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 --2- 172.21.15.61:0/3963955984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88063d60 0x7fee880b57e0 secure :-1 s=READY pgs=973 cs=0 l=1 rev1=1 rx=0x7fee78002f70 tx=0x7fee78006df0).stop 2022-07-20T03:15:19.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- 172.21.15.61:0/3963955984 shutdown_connections 2022-07-20T03:15:19.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 --2- 172.21.15.61:0/3963955984 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee88063d60 0x7fee880b57e0 unknown :-1 s=CLOSED pgs=973 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- 172.21.15.61:0/3963955984 >> 172.21.15.61:0/3963955984 conn(0x7fee880634e0 msgr2=0x7fee880b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- 172.21.15.61:0/3963955984 shutdown_connections 2022-07-20T03:15:19.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- 172.21.15.61:0/3963955984 wait complete. 2022-07-20T03:15:19.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 Processor -- start 2022-07-20T03:15:19.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- start start 2022-07-20T03:15:19.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee88063d60 0x7fee88066b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880af150 0x7fee881136b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 0x7fee88156030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee88115ae0 con 0x7fee880b1440 2022-07-20T03:15:19.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee88155930 con 0x7fee880af150 2022-07-20T03:15:19.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee8759e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fee88115d40 con 0x7fee88063d60 2022-07-20T03:15:19.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee86d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 0x7fee88156030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee86d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 0x7fee88156030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48382/0 (socket says 172.21.15.61:48382) 2022-07-20T03:15:19.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.800+0000 7fee86d9d700 1 -- 172.21.15.61:0/2936823104 learned_addr learned my addr 172.21.15.61:0/2936823104 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:19.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee7ffff700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880af150 0x7fee881136b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee8659c700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee88063d60 0x7fee88066b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee86d9d700 1 -- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee88063d60 msgr2=0x7fee88066b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee86d9d700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee88063d60 0x7fee88066b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee86d9d700 1 -- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880af150 msgr2=0x7fee881136b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee86d9d700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880af150 0x7fee881136b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee86d9d700 1 -- 172.21.15.61:0/2936823104 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee7800b180 con 0x7fee880b1440 2022-07-20T03:15:19.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee8659c700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee88063d60 0x7fee88066b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:19.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee86d9d700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 0x7fee88156030 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7fee880622d0 tx=0x7fee80007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8000e4a0 con 0x7fee880b1440 2022-07-20T03:15:19.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fee8000e600 con 0x7fee880b1440 2022-07-20T03:15:19.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee88155b90 con 0x7fee880b1440 2022-07-20T03:15:19.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fee8000e960 con 0x7fee880b1440 2022-07-20T03:15:19.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee88112540 con 0x7fee880b1440 2022-07-20T03:15:19.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee88112820 con 0x7fee880b1440 2022-07-20T03:15:19.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fee80019030 con 0x7fee880b1440 2022-07-20T03:15:19.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.804+0000 7fee7f7fe700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee6c053ad0 0x7fee6c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:19.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.808+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fee8006b5f0 con 0x7fee880b1440 2022-07-20T03:15:19.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.808+0000 7fee8659c700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee6c053ad0 0x7fee6c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:19.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.808+0000 7fee8659c700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee6c053ad0 0x7fee6c055f90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fee78015000 tx=0x7fee78014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:19.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.812+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fee80042550 con 0x7fee880b1440 2022-07-20T03:15:19.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.976+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fee88112d90 con 0x7fee880b1440 2022-07-20T03:15:19.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.976+0000 7fee7f7fe700 1 -- 172.21.15.61:0/2936823104 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fee88112d90 con 0x7fee880b1440 2022-07-20T03:15:19.981 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:19.981 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:13.467553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7008":{"gid":7008,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1014823584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1014823584},{"type":"v1","addr":"172.21.15.61:6833","nonce":1014823584}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:19.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.976+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee6c053ad0 msgr2=0x7fee6c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.976+0000 7fee8759e700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee6c053ad0 0x7fee6c055f90 secure :-1 s=READY pgs=472 cs=0 l=1 rev1=1 rx=0x7fee78015000 tx=0x7fee78014000).stop 2022-07-20T03:15:19.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.976+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 msgr2=0x7fee88156030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:19.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.976+0000 7fee8759e700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 0x7fee88156030 secure :-1 s=READY pgs=974 cs=0 l=1 rev1=1 rx=0x7fee880622d0 tx=0x7fee80007f40).stop 2022-07-20T03:15:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 shutdown_connections 2022-07-20T03:15:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fee6c053ad0 0x7fee6c055f90 unknown :-1 s=CLOSED pgs=472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fee880af150 0x7fee881136b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fee880b1440 0x7fee88156030 unknown :-1 s=CLOSED pgs=974 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 --2- 172.21.15.61:0/2936823104 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fee88063d60 0x7fee88066b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:19.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 >> 172.21.15.61:0/2936823104 conn(0x7fee880634e0 msgr2=0x7fee880b00d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:19.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 shutdown_connections 2022-07-20T03:15:19.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:19.980+0000 7fee8759e700 1 -- 172.21.15.61:0/2936823104 wait complete. 2022-07-20T03:15:19.987 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-20T03:15:20.199 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 51, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:12.462538+0000', 'modified': '2022-07-20T03:15:13.467553+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7008}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7008': {'gid': 7008, 'name': 'a', 'rank': 0, 'incarnation': 50, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1014823584', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1014823584}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1014823584}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [18], 'metadata_pool': 17, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:15:20.200 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:15:20.201 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:15:20.325 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7008,"want_state":"up:active","state":"up:active","rank_uptime":7.8428050919999999,"mdsmap_epoch":51,"osdmap_epoch":91,"osdmap_epoch_barrier":89,"uptime":12.214661919999999} 2022-07-20T03:15:20.326 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7008, "mdsmap_epoch": 51, "osdmap_epoch": 91, "osdmap_epoch_barrier": 89, "rank_uptime": 7.842805092, "state": "up:active", "uptime": 12.21466192, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:15:20.327 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:15:20.327 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:15:20.328 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:15:20.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.330 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:20.342 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:15:20.404 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:20.405 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:20.417 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:20.419 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:20.420 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.421 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:15:20.469 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:15:20.471 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:20.478 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:15:20.504 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:20.504 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:20.556 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:15:20.552+0000 7f81bef39080 -1 init, newargv = 0x55bb134053e0 newargc=9 2022-07-20T03:15:20.558 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22607]: starting ceph client 2022-07-20T03:15:20.560 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:20.561 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:20.562 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.563 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:15:20.571 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22607]: starting fuse 2022-07-20T03:15:20.576 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:15:20.576 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:15:20.577 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.577 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:15:20.578 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:15:20.578 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:15:20.578 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:15:20.579 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:15:20.579 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:20.580 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:15:20.580 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:15:20.581 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:15:20.587 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:20.587 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:15:20.588 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:15:20.588 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:15:20.595 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:15:20.596 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:20.596 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:15:20.597 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:15:20.597 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:15:20.598 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:15:20.598 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:15:20.598 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:20.599 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:15:20.599 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:15:20.664 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22607.asok 2022-07-20T03:15:20.678 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22607.asok 2022-07-20T03:15:20.679 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.686 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22607.asok status 2022-07-20T03:15:20.819 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:15:20.820 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:15:20.820 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:15:20.820 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:15:20.820 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:15:20.821 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:15:20.821 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:15:20.821 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22607", 2022-07-20T03:15:20.822 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:15:20.822 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:20.822 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:15:20.822 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:15:20.823 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 7230, 2022-07-20T03:15:20.823 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:15:20.823 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:15:20.824 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:15:20.824 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 7230 2022-07-20T03:15:20.824 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:20.825 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:15:20.825 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:15:20.825 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:15:20.825 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 312811642 2022-07-20T03:15:20.826 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:15:20.826 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:20.826 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:15:20.826 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:15:20.827 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:15:20.827 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 312811642 2022-07-20T03:15:20.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:20.827 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.7230 172.21.15.163:0/312811642", 2022-07-20T03:15:20.828 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/312811642", 2022-07-20T03:15:20.828 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:15:20.828 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 51, 2022-07-20T03:15:20.828 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 91, 2022-07-20T03:15:20.829 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:15:20.829 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:15:20.829 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:15:20.829 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:15:20.830 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.830 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:20.831 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:20.846 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:15:20.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:20.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:20.849 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:20.877 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T03:15:20.877 DEBUG:teuthology.orchestra.run.smithi124:> cd /home/ubuntu/cephtest/mnt.0 && sudo mkdir -p foo 2022-07-20T03:15:20.909 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:15:20.910 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:15:20.911 DEBUG:teuthology.orchestra.run.smithi124:> import stat 2022-07-20T03:15:20.913 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:20.916 DEBUG:teuthology.orchestra.run.smithi124:> print(os.stat("/home/ubuntu/cephtest/mnt.0/foo").st_ino) 2022-07-20T03:15:20.917 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:15:21.032 INFO:teuthology.orchestra.run.smithi124.stdout:1099511627776 2022-07-20T03:15:21.224 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:21.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 Processor -- start 2022-07-20T03:15:21.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 -- start start 2022-07-20T03:15:21.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40adc90 0x7fb3f405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:21.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ad430 0x7fb3f405c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:21.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3f40ae070 con 0x7fb3f40b21e0 2022-07-20T03:15:21.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3f40ace10 con 0x7fb3f40ad430 2022-07-20T03:15:21.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f9cf4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f40ad0b0 con 0x7fb3f40adc90 2022-07-20T03:15:21.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3f40ad430 0x7fb3f405c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:21.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb3f40b21e0 0x7fb3f41462c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:44988/0 (socket says 172.21.15.61:44988) 2022-07-20T03:15:21.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f3fff700 1 -- 172.21.15.61:0/492108134 learned_addr learned my addr 172.21.15.61:0/492108134 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:21.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.332+0000 7fb3f37fe700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40adc90 0x7fb3f405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:21.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f27fc700 1 -- 172.21.15.61:0/492108134 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1896212133 0 0) 0x7fb3f40ae070 con 0x7fb3f40b21e0 2022-07-20T03:15:21.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f27fc700 1 -- 172.21.15.61:0/492108134 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3e0003260 con 0x7fb3f40b21e0 2022-07-20T03:15:21.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/492108134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40adc90 msgr2=0x7fb3f405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:21.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f2ffd700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40adc90 0x7fb3f405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/492108134 >> v1:172.21.15.61:6789/0 conn(0x7fb3f40b21e0 legacy=0x7fb3f41462c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:21.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/492108134 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3f4143c10 con 0x7fb3f40ad430 2022-07-20T03:15:21.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f37fe700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40adc90 0x7fb3f405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:21.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f2ffd700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40ad430 0x7fb3f405c280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fb3e4002f70 tx=0x7fb3e4006df0).ready entity=mon.0 client_cookie=33fe261794f5c1ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:21.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f27fc700 1 -- 172.21.15.61:0/492108134 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3e400d570 con 0x7fb3f40ad430 2022-07-20T03:15:21.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f27fc700 1 -- 172.21.15.61:0/492108134 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4000e90 con 0x7fb3f40ad430 2022-07-20T03:15:21.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f27fc700 1 -- 172.21.15.61:0/492108134 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3e400d8c0 con 0x7fb3f40ad430 2022-07-20T03:15:21.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/492108134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40ad430 msgr2=0x7fb3f405c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:21.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40ad430 0x7fb3f405c280 secure :-1 s=READY pgs=977 cs=0 l=1 rev1=1 rx=0x7fb3e4002f70 tx=0x7fb3e4006df0).stop 2022-07-20T03:15:21.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/492108134 shutdown_connections 2022-07-20T03:15:21.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40ad430 0x7fb3f405c280 unknown :-1 s=CLOSED pgs=977 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/492108134 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40adc90 0x7fb3f405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/492108134 >> 172.21.15.61:0/492108134 conn(0x7fb3f40b72c0 msgr2=0x7fb3f40b76d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:21.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/492108134 shutdown_connections 2022-07-20T03:15:21.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.336+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/492108134 wait complete. 2022-07-20T03:15:21.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 Processor -- start 2022-07-20T03:15:21.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 -- start start 2022-07-20T03:15:21.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40ad430 0x7fb3f405cd20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:21.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 0x7fb3f413eb60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:21.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3f40b21e0 0x7fb3f4151fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:21.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f41459e0 con 0x7fb3f40adc90 2022-07-20T03:15:21.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4155880 con 0x7fb3f40b21e0 2022-07-20T03:15:21.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f9cf4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3f4155b20 con 0x7fb3f40ad430 2022-07-20T03:15:21.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 0x7fb3f413eb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:21.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 0x7fb3f413eb60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48394/0 (socket says 172.21.15.61:48394) 2022-07-20T03:15:21.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/3365948855 learned_addr learned my addr 172.21.15.61:0/3365948855 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:21.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f3fff700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3f40b21e0 0x7fb3f4151fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:21.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f37fe700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40ad430 0x7fb3f405cd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:21.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40ad430 msgr2=0x7fb3f405cd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:21.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40ad430 0x7fb3f405cd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3f40b21e0 msgr2=0x7fb3f4151fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:21.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3f40b21e0 0x7fb3f4151fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 -- 172.21.15.61:0/3365948855 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e400b180 con 0x7fb3f40adc90 2022-07-20T03:15:21.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.340+0000 7fb3f2ffd700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 0x7fb3f413eb60 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7fb3e4002f70 tx=0x7fb3e4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:21.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3e4016070 con 0x7fb3f40adc90 2022-07-20T03:15:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3e4013a00 con 0x7fb3f40adc90 2022-07-20T03:15:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3f4155dc0 con 0x7fb3f40adc90 2022-07-20T03:15:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3e4013d60 con 0x7fb3f40adc90 2022-07-20T03:15:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3f4156260 con 0x7fb3f40adc90 2022-07-20T03:15:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3f4155f50 con 0x7fb3f40adc90 2022-07-20T03:15:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb3e4025030 con 0x7fb3f40adc90 2022-07-20T03:15:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f0ff9700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3d4053af0 0x7fb3d4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:21.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.344+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb3e406ce00 con 0x7fb3f40adc90 2022-07-20T03:15:21.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.348+0000 7fb3f37fe700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3d4053af0 0x7fb3d4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:21.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.348+0000 7fb3f37fe700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3d4053af0 0x7fb3d4055fb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fb3e800c000 tx=0x7fb3e8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:21.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.352+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3e4043c50 con 0x7fb3f40adc90 2022-07-20T03:15:21.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.520+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fb3f40574e0 con 0x7fb3f40adc90 2022-07-20T03:15:21.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.520+0000 7fb3f0ff9700 1 -- 172.21.15.61:0/3365948855 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fb3e4023070 con 0x7fb3f40adc90 2022-07-20T03:15:21.526 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:21.526 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:13.467553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7008":{"gid":7008,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1014823584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1014823584},{"type":"v1","addr":"172.21.15.61:6833","nonce":1014823584}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:21.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3d4053af0 msgr2=0x7fb3d4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:21.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3d4053af0 0x7fb3d4055fb0 secure :-1 s=READY pgs=473 cs=0 l=1 rev1=1 rx=0x7fb3e800c000 tx=0x7fb3e8008040).stop 2022-07-20T03:15:21.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 msgr2=0x7fb3f413eb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:21.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 0x7fb3f413eb60 secure :-1 s=READY pgs=978 cs=0 l=1 rev1=1 rx=0x7fb3e4002f70 tx=0x7fb3e4014000).stop 2022-07-20T03:15:21.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 shutdown_connections 2022-07-20T03:15:21.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3d4053af0 0x7fb3d4055fb0 unknown :-1 s=CLOSED pgs=473 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3f40b21e0 0x7fb3f4151fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3f40adc90 0x7fb3f413eb60 unknown :-1 s=CLOSED pgs=978 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 --2- 172.21.15.61:0/3365948855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3f40ad430 0x7fb3f405cd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:21.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 >> 172.21.15.61:0/3365948855 conn(0x7fb3f40b72c0 msgr2=0x7fb3f4067400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:21.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 shutdown_connections 2022-07-20T03:15:21.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:21.524+0000 7fb3f9cf4700 1 -- 172.21.15.61:0/3365948855 wait complete. 2022-07-20T03:15:21.537 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-20T03:15:21.542 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json dump inode 0x10000000000 2022-07-20T03:15:21.662 INFO:teuthology.orchestra.run.smithi061.stdout:{"path":"/foo","ino":1099511627776,"rdev":0,"ctime":"2022-07-20T03:15:20.899768+0000","btime":"2022-07-20T03:15:20.899768+0000","mode":16877,"uid":0,"gid":0,"nlink":1,"dir_layout":{"dir_hash":2,"unused1":0,"unused2":0,"unused3":0},"layout":{"stripe_unit":0,"stripe_count":0,"object_size":0,"pool_id":-1,"pool_ns":""},"old_pools":[],"size":0,"truncate_seq":1,"truncate_size":18446744073709551615,"truncate_from":0,"truncate_pending":0,"mtime":"2022-07-20T03:15:20.899768+0000","atime":"2022-07-20T03:15:20.899768+0000","time_warp_seq":0,"change_attr":0,"export_pin":-1,"export_ephemeral_random_pin":0,"export_ephemeral_distributed_pin":false,"client_ranges":[],"dirstat":{"version":0,"mtime":"0.000000","num_files":0,"num_subdirs":0,"change_attr":0},"rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"accounted_rstat":{"version":0,"rbytes":0,"rfiles":0,"rsubdirs":1,"rsnaps":0,"rctime":"0.000000"},"version":2,"file_data_version":0,"xattr_version":1,"backtrace_version":2,"stray_prior_path":"","max_size_ever":0,"quota":{"max_bytes":0,"max_files":0},"last_scrub_stamp":"0.000000","last_scrub_version":0,"symlink":"","xattrs":[],"dirfragtree":{"splits":[]},"old_inodes":[],"oldest_snap":18446744073709551614,"damage_flags":0,"is_auth":true,"auth_state":{"replicas":{}},"replica_state":{"authority":[0,-2],"replica_nonce":0},"auth_pins":0,"is_frozen":false,"is_freezing":false,"pins":{"request":0,"dirfrag":1,"caps":1,"dirtyparent":1,"dirty":1},"nref":4,"versionlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"authlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"linklock":{},"dirfragtreelock":{},"filelock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"xattrlock":{"gather_set":[],"state":"excl","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"snaplock":{},"nestlock":{"gather_set":[],"state":"lock","is_leased":false,"num_rdlocks":0,"num_wrlocks":0,"num_xlocks":0,"xlock_by":{}},"flocklock":{},"policylock":{},"states":["auth","dirty","dirtyparent","dirtypool"],"client_caps":[{"client_id":7182,"pending":"pAsxLsXsxFsx","issued":"pAsxLsXsxFsx","wanted":"-","last_sent":1}],"loner":7182,"want_loner":7182,"mds_caps_wanted":[]} 2022-07-20T03:15:21.663 DEBUG:tasks.cephfs.filesystem:_json_asok output { "accounted_rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "atime": "2022-07-20T03:15:20.899768+0000", "auth_pins": 0, "auth_state": { "replicas": {} }, "authlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "backtrace_version": 2, "btime": "2022-07-20T03:15:20.899768+0000", "change_attr": 0, "client_caps": [ { "client_id": 7182, "issued": "pAsxLsXsxFsx", "last_sent": 1, "pending": "pAsxLsXsxFsx", "wanted": "-" } ], "client_ranges": [], "ctime": "2022-07-20T03:15:20.899768+0000", "damage_flags": 0, "dir_layout": { "dir_hash": 2, "unused1": 0, "unused2": 0, "unused3": 0 }, "dirfragtree": { "splits": [] }, "dirfragtreelock": {}, "dirstat": { "change_attr": 0, "mtime": "0.000000", "num_files": 0, "num_subdirs": 0, "version": 0 }, "export_ephemeral_distributed_pin": false, "export_ephemeral_random_pin": 0, "export_pin": -1, "file_data_version": 0, "filelock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "flocklock": {}, "gid": 0, "ino": 1099511627776, "is_auth": true, "is_freezing": false, "is_frozen": false, "last_scrub_stamp": "0.000000", "last_scrub_version": 0, "layout": { "object_size": 0, "pool_id": -1, "pool_ns": "", "stripe_count": 0, "stripe_unit": 0 }, "linklock": {}, "loner": 7182, "max_size_ever": 0, "mds_caps_wanted": [], "mode": 16877, "mtime": "2022-07-20T03:15:20.899768+0000", "nestlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "nlink": 1, "nref": 4, "old_inodes": [], "old_pools": [], "oldest_snap": 18446744073709551614, "path": "/foo", "pins": { "caps": 1, "dirfrag": 1, "dirty": 1, "dirtyparent": 1, "request": 0 }, "policylock": {}, "quota": { "max_bytes": 0, "max_files": 0 }, "rdev": 0, "replica_state": { "authority": [ 0, -2 ], "replica_nonce": 0 }, "rstat": { "rbytes": 0, "rctime": "0.000000", "rfiles": 0, "rsnaps": 0, "rsubdirs": 1, "version": 0 }, "size": 0, "snaplock": {}, "states": [ "auth", "dirty", "dirtyparent", "dirtypool" ], "stray_prior_path": "", "symlink": "", "time_warp_seq": 0, "truncate_from": 0, "truncate_pending": 0, "truncate_seq": 1, "truncate_size": 18446744073709551615, "uid": 0, "version": 2, "versionlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "lock", "xlock_by": {} }, "want_loner": 7182, "xattr_version": 1, "xattrlock": { "gather_set": [], "is_leased": false, "num_rdlocks": 0, "num_wrlocks": 0, "num_xlocks": 0, "state": "excl", "xlock_by": {} }, "xattrs": [] } 2022-07-20T03:15:21.664 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:15:21.665 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:21.694 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:21.719 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:21.742 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:21.782 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:21.819 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:15:21.820 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:21.821 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:21.822 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:21.839 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:15:21.840 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:21.841 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:15:21.870 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:15:21.871 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:21.912 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23158]: fuse finished with error 0 and tester_r 0 2022-07-20T03:15:21.916 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:21.917 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:21.917 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:21.966 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:15:21.966 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:15:22.082 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:15:22.083 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:22.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:22.090 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:22.091 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:22.102 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:15:22.103 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:22.103 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:15:22.104 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:15:22.117 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:22.160 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22607]: fuse finished with error 0 and tester_r 0 2022-07-20T03:15:22.162 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:22.163 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:22.163 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:22.171 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:15:22.172 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:15:22.290 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:15:22.291 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:22.343 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:22.344 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:22.346 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:22.355 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:22.356 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:22.357 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:15:22.358 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:15:22.358 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:22.406 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:22.407 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:22.408 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:15:22.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 Processor -- start 2022-07-20T03:15:22.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 -- start start 2022-07-20T03:15:22.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6a740b5650 0x7f6a740b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:22.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:22.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6a74147010 con 0x7f6a740af8e0 2022-07-20T03:15:22.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6a741472b0 con 0x7f6a740b5650 2022-07-20T03:15:22.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a7af3b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a74146d40 con 0x7f6a740adff0 2022-07-20T03:15:22.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6a740b5650 0x7f6a740b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:22.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6a740b5650 0x7f6a740b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48400/0 (socket says 172.21.15.61:48400) 2022-07-20T03:15:22.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 -- 172.21.15.61:0/1251070117 learned_addr learned my addr 172.21.15.61:0/1251070117 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:22.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79738700 1 --2- 172.21.15.61:0/1251070117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:22.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 -- 172.21.15.61:0/1251070117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 msgr2=0x7f6a74147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:22.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 --2- 172.21.15.61:0/1251070117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:22.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 -- 172.21.15.61:0/1251070117 >> v1:172.21.15.117:6789/0 conn(0x7f6a740af8e0 legacy=0x7f6a7405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:22.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.516+0000 7f6a79f39700 1 -- 172.21.15.61:0/1251070117 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a74147c40 con 0x7f6a740b5650 2022-07-20T03:15:22.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a79f39700 1 --2- 172.21.15.61:0/1251070117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a740b5a30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f6a6c0032f0 tx=0x7f6a6c009290).ready entity=mon.0 client_cookie=6d459b55349d1de6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:22.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a78f37700 1 -- 172.21.15.61:0/1251070117 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a6c00b8a0 con 0x7f6a740b5650 2022-07-20T03:15:22.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a78f37700 1 -- 172.21.15.61:0/1251070117 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a6c00ba00 con 0x7f6a740b5650 2022-07-20T03:15:22.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a78f37700 1 -- 172.21.15.61:0/1251070117 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a6c00bb70 con 0x7f6a740b5650 2022-07-20T03:15:22.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 -- 172.21.15.61:0/1251070117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 msgr2=0x7f6a740b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:22.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/1251070117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a740b5a30 secure :-1 s=READY pgs=979 cs=0 l=1 rev1=1 rx=0x7f6a6c0032f0 tx=0x7f6a6c009290).stop 2022-07-20T03:15:22.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 -- 172.21.15.61:0/1251070117 shutdown_connections 2022-07-20T03:15:22.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/1251070117 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:22.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/1251070117 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a740b5a30 unknown :-1 s=CLOSED pgs=979 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:22.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 -- 172.21.15.61:0/1251070117 >> 172.21.15.61:0/1251070117 conn(0x7f6a740634e0 msgr2=0x7f6a740b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:22.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 -- 172.21.15.61:0/1251070117 shutdown_connections 2022-07-20T03:15:22.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 -- 172.21.15.61:0/1251070117 wait complete. 2022-07-20T03:15:22.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 Processor -- start 2022-07-20T03:15:22.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.520+0000 7f6a7af3b700 1 -- start start 2022-07-20T03:15:22.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74148e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:22.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a740af8e0 0x7f6a741599d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:22.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a74063990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:22.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a7410fac0 con 0x7f6a740b5650 2022-07-20T03:15:22.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a74159ec0 con 0x7f6a740af8e0 2022-07-20T03:15:22.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6a74065d40 con 0x7f6a740adff0 2022-07-20T03:15:22.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a74063990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:22.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a79738700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a740af8e0 0x7f6a741599d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:22.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a79f39700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74148e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:22.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a79738700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a740af8e0 0x7f6a741599d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58088/0 (socket says 172.21.15.61:58088) 2022-07-20T03:15:22.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a79738700 1 -- 172.21.15.61:0/3387756283 learned_addr learned my addr 172.21.15.61:0/3387756283 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:22.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a74063990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48410/0 (socket says 172.21.15.61:48410) 2022-07-20T03:15:22.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 -- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 msgr2=0x7f6a74148e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:22.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74148e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:22.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 -- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a740af8e0 msgr2=0x7f6a741599d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:22.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a740af8e0 0x7f6a741599d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:22.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 -- 172.21.15.61:0/3387756283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a6c009530 con 0x7f6a740b5650 2022-07-20T03:15:22.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7a73a700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a74063990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f6a74060b70 tx=0x7f6a70007af0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:22.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a7000cd30 con 0x7f6a740b5650 2022-07-20T03:15:22.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6a7000ce90 con 0x7f6a740b5650 2022-07-20T03:15:22.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a7410f380 con 0x7f6a740b5650 2022-07-20T03:15:22.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6a700125f0 con 0x7f6a740b5650 2022-07-20T03:15:22.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a74149af0 con 0x7f6a740b5650 2022-07-20T03:15:22.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.524+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a7405c4a0 con 0x7f6a740b5650 2022-07-20T03:15:22.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.528+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6a70019030 con 0x7f6a740b5650 2022-07-20T03:15:22.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.528+0000 7f6a62ffd700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a54053ad0 0x7f6a54055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:22.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.528+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6a7006afe0 con 0x7f6a740b5650 2022-07-20T03:15:22.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.528+0000 7f6a79f39700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a54053ad0 0x7f6a54055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:22.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.528+0000 7f6a79f39700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a54053ad0 0x7f6a54055f90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f6a6c01b000 tx=0x7f6a6c016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:22.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.532+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6a70041f40 con 0x7f6a740b5650 2022-07-20T03:15:22.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.664+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f6a740574e0 con 0x7f6a740b5650 2022-07-20T03:15:22.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.664+0000 7f6a62ffd700 1 -- 172.21.15.61:0/3387756283 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v91) v1 ==== 81+0+33696 (secure 0 0 0) 0x7f6a70017070 con 0x7f6a740b5650 2022-07-20T03:15:22.671 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:22.671 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:22.672 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 91, 2022-07-20T03:15:22.672 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:15:22.672 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:15:22.673 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:15:14.477402+0000", 2022-07-20T03:15:22.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:22.673 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:22.673 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:15:22.673 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:15:22.674 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:15:22.674 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:15:22.674 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:15:22.675 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:15:22.675 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:15:22.675 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:22.676 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:15:22.676 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:15:22.676 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:15:22.676 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:15:22.677 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:15:22.677 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 18, 2022-07-20T03:15:22.677 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:15:22.677 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:15:22.677 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:15:22.678 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:15:22.678 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:15:22.678 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.678 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:15:22.679 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:15:22.679 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:15:22.679 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:22.681 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:22.681 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:22.682 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:22.682 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:22.682 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:22.682 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:22.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:22.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:15:22.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:15:22.683 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:15:22.684 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:15:22.684 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:15:22.684 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:22.684 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:22.685 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:22.685 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:22.685 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:22.685 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:22.686 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:22.686 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.686 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:15:22.686 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:22.687 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:22.687 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:22.687 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:22.687 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:22.688 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:22.688 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:22.688 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:22.688 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:22.689 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:22.689 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:22.689 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:22.689 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:22.690 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:22.690 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:22.690 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:22.690 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:22.691 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:22.698 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:22.698 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:22.699 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:22.700 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:22.700 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:22.700 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:22.700 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:22.701 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:22.701 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.701 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:22.701 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:22.702 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:22.703 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:22.703 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:22.703 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:22.703 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:22.704 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:22.704 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:22.704 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:22.704 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:22.706 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:22.706 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:15:22.706 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.707 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:22.707 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:15:22.707 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.707 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.708 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:15:22.708 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:15:22.709 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:15:22.709 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:22.709 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:22.709 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:22.710 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:22.710 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:22.711 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:22.711 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:22.711 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:22.712 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:22.712 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:22.712 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:22.712 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:22.713 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:22.713 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:22.714 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:22.715 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:22.716 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:22.716 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:22.717 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:22.717 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:22.718 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.718 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:15:22.719 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:22.719 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:22.720 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:22.720 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:22.720 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:22.721 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:22.721 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:22.721 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:22.722 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:22.722 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:22.722 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:22.722 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:22.723 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:22.723 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:22.724 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:22.724 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:22.724 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:22.726 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:22.728 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:22.728 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:22.732 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:22.732 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:22.732 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:22.732 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:22.733 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:22.734 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:22.734 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.735 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:22.735 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:22.735 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:22.736 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:22.736 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:22.736 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:22.736 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:22.737 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:22.737 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:22.737 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:22.737 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:22.738 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:22.738 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:22.738 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:15:22.740 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.741 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.741 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.741 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 17, 2022-07-20T03:15:22.741 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:15:22.742 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:15:11.081553+0000", 2022-07-20T03:15:22.742 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:22.742 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:22.742 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:22.743 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:22.743 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:22.743 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:22.743 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:22.744 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:22.744 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:22.744 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:22.744 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:22.745 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:22.745 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:22.745 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:22.745 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:22.746 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:22.746 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:22.746 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:22.746 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:22.747 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:22.747 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.747 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "89", 2022-07-20T03:15:22.747 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:22.748 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:22.748 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:22.748 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:22.748 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:22.749 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:22.749 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:22.749 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:22.749 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:22.750 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:22.750 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:22.750 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:22.750 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:22.751 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:22.751 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:22.751 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:22.751 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:22.752 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:22.752 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:22.752 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:22.752 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:22.753 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:22.753 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:22.753 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:22.753 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:22.754 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:22.754 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.754 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:22.755 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:22.755 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:22.755 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:22.755 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:22.756 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:22.756 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:22.756 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:22.756 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:22.757 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:22.757 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:22.757 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:22.757 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:15:22.758 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:15:22.758 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:15:22.758 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.759 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:22.759 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:15:22.759 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:15:22.759 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.760 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.760 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.760 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 18, 2022-07-20T03:15:22.761 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:15:22.761 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:15:11.541101+0000", 2022-07-20T03:15:22.761 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:22.761 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:22.762 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:22.762 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:22.762 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:22.762 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:22.763 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:22.763 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:22.763 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:22.763 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:22.764 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:22.764 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:22.764 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:22.764 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:22.764 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:22.765 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:22.765 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:22.765 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:22.765 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:22.766 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:22.766 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.766 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "91", 2022-07-20T03:15:22.766 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:22.767 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:22.767 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:22.767 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:22.768 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:22.768 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:22.768 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:22.768 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:22.769 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:22.769 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:22.769 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:22.769 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:22.770 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:22.770 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:22.770 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:22.770 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:22.771 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:22.771 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:22.771 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:22.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:22.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:22.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:22.772 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:22.773 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:22.773 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:22.773 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:22.773 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.774 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:22.774 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:22.774 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:22.774 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:22.775 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:22.775 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:22.776 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:22.776 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:22.776 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:22.776 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:22.777 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:22.777 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:22.777 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:22.777 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:15:22.778 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:15:22.778 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.778 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.778 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.779 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:22.779 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:15:22.779 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.779 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:22.780 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:15:22.780 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.780 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.780 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.781 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.781 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.781 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.781 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.782 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:22.782 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.782 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.782 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.783 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.783 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.783 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.784 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:15:22.784 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.784 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.784 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.785 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.785 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:15:22.785 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.785 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.785 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.786 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:22.786 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.787 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.787 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.787 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:15:22.787 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.788 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.792 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.793 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.793 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:15:22.793 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.793 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.794 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.794 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.794 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:22.795 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.795 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.795 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.795 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:15:22.796 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.796 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.796 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.796 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.797 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:15:22.797 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.797 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.797 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.798 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.798 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:22.798 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.799 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.799 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.799 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:15:22.800 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.800 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.807 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.808 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.808 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:15:22.809 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:22.809 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.810 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.810 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.811 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:15:22.811 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:15:22.812 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:15:22.812 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:15:22.812 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:22.813 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:22.813 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:22.814 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.814 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.816 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.816 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:22.817 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:15:22.817 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.818 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.818 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.819 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.819 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.819 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.820 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.820 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:22.821 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.821 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.822 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.822 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.823 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.823 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.824 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:15:22.824 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.825 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.825 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.828 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.828 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:15:22.829 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.829 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.829 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.829 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.830 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:22.830 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.830 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.830 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.831 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:15:22.831 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.831 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.832 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.832 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.832 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:15:22.832 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.833 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.833 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.833 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.834 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:22.834 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.834 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.834 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.835 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:15:22.835 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.835 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.836 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.836 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.836 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:15:22.837 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.837 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.837 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.837 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.838 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:22.838 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.838 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.838 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.839 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:15:22.839 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.839 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.839 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.840 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.840 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:15:22.840 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:22.841 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.841 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.842 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.842 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:15:22.842 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:15:22.843 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:15:22.843 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:15:22.843 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:22.843 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:22.844 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:22.844 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.845 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.845 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.845 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:22.845 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:15:22.846 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.846 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.846 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.847 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.847 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.847 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.847 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.848 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:22.848 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.848 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.848 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.849 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.849 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.849 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:15:22.851 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.851 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.851 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.852 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.852 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:15:22.853 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.853 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.853 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.853 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.854 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:22.854 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.854 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.855 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.855 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:15:22.855 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.855 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.856 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.856 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.856 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:15:22.856 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.857 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.857 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.857 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.857 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:22.858 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.858 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.858 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.859 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:15:22.859 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.859 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.859 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.860 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.860 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:15:22.860 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.861 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.861 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.861 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.861 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:22.862 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.862 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.862 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.862 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:15:22.863 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.863 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.863 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.864 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.864 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:15:22.864 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:22.864 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.865 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.865 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.865 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:15:22.865 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:15:22.866 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:15:22.866 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:15:22.866 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:22.867 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:22.867 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:22.867 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.867 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.868 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.868 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:22.868 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:15:22.868 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.869 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.869 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.869 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.869 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.870 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.870 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.870 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 89, 2022-07-20T03:15:22.871 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.871 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.871 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.871 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.872 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.872 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.872 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:15:22.872 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.873 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.873 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.873 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:15:22.874 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.874 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.874 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.875 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.875 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:22.876 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.877 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.878 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:15:22.879 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.880 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.880 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.881 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:15:22.881 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.882 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.882 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.882 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.883 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:22.889 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.890 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.890 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.891 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:15:22.891 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.892 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.892 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.893 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.894 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:15:22.894 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.895 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.895 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.896 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:22.897 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.897 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.898 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.908 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:15:22.908 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.909 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.909 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:15:22.910 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:22.910 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.911 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.911 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.911 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:15:22.911 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:15:22.912 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:15:22.912 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:15:22.912 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:22.912 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:22.912 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:22.913 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.913 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.913 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.913 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:22.914 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:15:22.914 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.914 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.914 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.915 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.915 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.915 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.915 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.916 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-20T03:15:22.916 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.916 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.916 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.916 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.917 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.917 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.917 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:15:22.919 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.919 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.920 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.920 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:15:22.921 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.921 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.930 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.931 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.931 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:22.931 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.932 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.932 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.932 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:15:22.932 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.933 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.933 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.933 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.933 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:15:22.935 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.935 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.935 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.935 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.936 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:22.936 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.936 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.936 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.937 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:15:22.937 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.937 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.938 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.938 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.938 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:15:22.939 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.939 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.939 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.939 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.940 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:22.940 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.940 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.940 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.941 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:15:22.941 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.941 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.941 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.943 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.943 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:15:22.944 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:22.944 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.944 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.944 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.945 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:15:22.946 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:15:22.946 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:15:22.946 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:15:22.946 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:22.947 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:22.947 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:22.947 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.947 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.948 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.948 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:22.948 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:15:22.948 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.949 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.949 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.949 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.949 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.950 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.950 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.950 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:22.950 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.951 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.951 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.951 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.951 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.952 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.952 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:15:22.952 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.952 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.953 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.953 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.953 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:15:22.953 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.954 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.955 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.957 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.957 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:22.957 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.958 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.958 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:15:22.958 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.959 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.959 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.959 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.959 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:15:22.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.960 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.970 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.971 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.971 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:22.971 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.972 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:15:22.972 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.972 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.973 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.973 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.973 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:15:22.973 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.974 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.974 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.974 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.974 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:22.975 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.975 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.976 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:15:22.976 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.976 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.979 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.979 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.980 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:15:22.980 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:22.981 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:22.981 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.981 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.982 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:15:22.982 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:15:22.983 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:15:22.983 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:15:22.984 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:22.984 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:22.985 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:22.985 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:22.985 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.986 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.986 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:22.987 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:15:22.988 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:22.988 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:22.989 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:22.989 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:22.990 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:22.990 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:22.991 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:22.992 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-20T03:15:22.992 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:22.993 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:22.993 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:22.994 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:22.994 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.995 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:22.996 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:15:22.996 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:22.996 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:22.997 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:22.998 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:22.999 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:15:22.999 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.000 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.000 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.001 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:23.002 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.002 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.003 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:15:23.004 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.005 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.005 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.006 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:15:23.006 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.006 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.007 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.008 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:23.009 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.010 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.010 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.010 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:15:23.011 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.011 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.012 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.012 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.012 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:15:23.013 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.013 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.013 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.013 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.014 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:23.014 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.014 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.015 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.015 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:15:23.016 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.016 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.017 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.017 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:15:23.018 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:23.018 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.019 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.020 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.021 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:15:23.021 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:15:23.022 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:15:23.022 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:15:23.023 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:23.023 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:23.024 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:23.024 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.025 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.025 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.026 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:23.026 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:15:23.026 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:23.027 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:23.027 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:23.028 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:23.028 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:23.028 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:23.029 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:23.029 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:23.030 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:23.031 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:23.031 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:23.032 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.033 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.033 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:15:23.034 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.034 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.035 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:15:23.036 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.037 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.038 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.038 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.039 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:23.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.041 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:15:23.042 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.043 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.044 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:15:23.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.046 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.047 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:23.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.048 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:15:23.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.049 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.052 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.053 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:15:23.053 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.054 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.055 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.055 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.056 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:23.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:23.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:23.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:15:23.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.058 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.059 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:23.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:15:23.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:23.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.060 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.061 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:15:23.061 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:15:23.062 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:15:23.063 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:15:23.063 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:23.063 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:23.064 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:23.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:23.064 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.064 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:23.065 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:15:23.065 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.066 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:23.066 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.067 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.067 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.068 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.068 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.069 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:15:23.069 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.070 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.070 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.071 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:23.071 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.072 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.072 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.073 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.073 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:15:23.074 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.075 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.075 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.076 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:23.076 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.077 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.077 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.078 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.079 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.080 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:15:23.080 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.082 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:23.082 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.083 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.083 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.084 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.084 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.085 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:15:23.085 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.085 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.086 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.086 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:23.087 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.087 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.088 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.088 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.089 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.090 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:15:23.090 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.091 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.092 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:23.092 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.093 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.094 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.094 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.094 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.095 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:15:23.095 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.097 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:23.098 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.098 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.099 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.102 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.103 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.103 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:15:23.104 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:23.106 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:23.106 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:23.106 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:23.107 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:23.107 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:23.107 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:23.108 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:15:23.109 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:23.109 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.110 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:23.111 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:15:23.111 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:15:23.112 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:15:23.113 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:15:23.113 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:15:23.114 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:15:23.114 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:15:23.115 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:15:23.115 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:15:23.116 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:15:23.117 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:15:23.117 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:15:23.117 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:15:23.118 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:23.118 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:23.119 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:15:23.119 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:15:23.120 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:15:23.120 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:15:23.121 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:15:23.121 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:23.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a54053ad0 msgr2=0x7f6a54055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a54053ad0 0x7f6a54055f90 secure :-1 s=READY pgs=474 cs=0 l=1 rev1=1 rx=0x7f6a6c01b000 tx=0x7f6a6c016000).stop 2022-07-20T03:15:23.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 msgr2=0x7f6a74063990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a74063990 secure :-1 s=READY pgs=980 cs=0 l=1 rev1=1 rx=0x7f6a74060b70 tx=0x7f6a70007af0).stop 2022-07-20T03:15:23.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 shutdown_connections 2022-07-20T03:15:23.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6a54053ad0 0x7f6a54055f90 unknown :-1 s=CLOSED pgs=474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6a740af8e0 0x7f6a741599d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6a740b5650 0x7f6a74063990 unknown :-1 s=CLOSED pgs=980 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 --2- 172.21.15.61:0/3387756283 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6a740adff0 0x7f6a74148e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 >> 172.21.15.61:0/3387756283 conn(0x7f6a740634e0 msgr2=0x7f6a740b3db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:23.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 shutdown_connections 2022-07-20T03:15:23.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:22.668+0000 7f6a7af3b700 1 -- 172.21.15.61:0/3387756283 wait complete. 2022-07-20T03:15:23.132 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:23.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.240+0000 7f72c259d700 1 Processor -- start 2022-07-20T03:15:23.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.240+0000 7f72c259d700 1 -- start start 2022-07-20T03:15:23.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c259d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f72bc0b1ef0 0x7f72bc142f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c259d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72bc0ac3a0 0x7f72bc143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c259d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc143960 0x7f72bc0ab180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c259d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72bc060980 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c259d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72bc0b22d0 con 0x7f72bc0b1ef0 2022-07-20T03:15:23.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c259d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc098810 con 0x7f72bc143960 2022-07-20T03:15:23.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72bc0ac3a0 0x7f72bc143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c159b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f72bc0b1ef0 0x7f72bc142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72bc0ac3a0 0x7f72bc143470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48416/0 (socket says 172.21.15.61:48416) 2022-07-20T03:15:23.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 -- 172.21.15.61:0/1014942120 learned_addr learned my addr 172.21.15.61:0/1014942120 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:23.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c1d9c700 1 --2- 172.21.15.61:0/1014942120 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc143960 0x7f72bc0ab180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 -- 172.21.15.61:0/1014942120 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc143960 msgr2=0x7f72bc0ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:23.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 --2- 172.21.15.61:0/1014942120 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc143960 0x7f72bc0ab180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 -- 172.21.15.61:0/1014942120 >> v2:172.21.15.117:3300/0 conn(0x7f72bc0b1ef0 msgr2=0x7f72bc142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:23.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 --2- 172.21.15.61:0/1014942120 >> v2:172.21.15.117:3300/0 conn(0x7f72bc0b1ef0 0x7f72bc142f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 -- 172.21.15.61:0/1014942120 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72bc060bb0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c159b700 1 --2- 172.21.15.61:0/1014942120 >> v2:172.21.15.117:3300/0 conn(0x7f72bc0b1ef0 0x7f72bc142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:23.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72c0d9a700 1 --2- 172.21.15.61:0/1014942120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc143470 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f72ac002f70 tx=0x7f72ac006eb0).ready entity=mon.0 client_cookie=4a6f96a59e598f4a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:23.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72abfff700 1 -- 172.21.15.61:0/1014942120 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ac00d570 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72abfff700 1 -- 172.21.15.61:0/1014942120 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac000e90 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.244+0000 7f72abfff700 1 -- 172.21.15.61:0/1014942120 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ac00d900 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 -- 172.21.15.61:0/1014942120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 msgr2=0x7f72bc143470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 --2- 172.21.15.61:0/1014942120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc143470 secure :-1 s=READY pgs=981 cs=0 l=1 rev1=1 rx=0x7f72ac002f70 tx=0x7f72ac006eb0).stop 2022-07-20T03:15:23.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 -- 172.21.15.61:0/1014942120 shutdown_connections 2022-07-20T03:15:23.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 --2- 172.21.15.61:0/1014942120 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc143960 0x7f72bc0ab180 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 --2- 172.21.15.61:0/1014942120 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc143470 unknown :-1 s=CLOSED pgs=981 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 --2- 172.21.15.61:0/1014942120 >> v2:172.21.15.117:3300/0 conn(0x7f72bc0b1ef0 0x7f72bc142f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 -- 172.21.15.61:0/1014942120 >> 172.21.15.61:0/1014942120 conn(0x7f72bc0b2ca0 msgr2=0x7f72bc0b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:23.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 -- 172.21.15.61:0/1014942120 shutdown_connections 2022-07-20T03:15:23.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 -- 172.21.15.61:0/1014942120 wait complete. 2022-07-20T03:15:23.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 Processor -- start 2022-07-20T03:15:23.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.248+0000 7f72c259d700 1 -- start start 2022-07-20T03:15:23.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c259d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc05c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c259d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc0b1ef0 0x7f72bc155540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c259d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72bc143960 0x7f72bc155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c259d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc0adb60 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c259d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc1427e0 con 0x7f72bc143960 2022-07-20T03:15:23.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c259d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72bc142a50 con 0x7f72bc0b1ef0 2022-07-20T03:15:23.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc05c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc05c960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48420/0 (socket says 172.21.15.61:48420) 2022-07-20T03:15:23.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 -- 172.21.15.61:0/28852673 learned_addr learned my addr 172.21.15.61:0/28852673 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:23.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c1d9c700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72bc143960 0x7f72bc155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c0d9a700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc0b1ef0 0x7f72bc155540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 -- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc0b1ef0 msgr2=0x7f72bc155540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc0b1ef0 0x7f72bc155540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 -- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72bc143960 msgr2=0x7f72bc155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72bc143960 0x7f72bc155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 -- 172.21.15.61:0/28852673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b800b120 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.252+0000 7f72c159b700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc05c960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f72bc1430c0 tx=0x7f72b8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:23.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72b8012600 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72b8012760 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72b8012ac0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72ac00b1e0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72bc0ade00 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72bc0ad8b0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f72b8012c20 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72aa7fc700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f729c053920 0x7f729c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f72b806c3e0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72c0d9a700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f729c053920 0x7f729c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.256+0000 7f72c0d9a700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f729c053920 0x7f729c055de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f72ac01c000 tx=0x7f72ac015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:23.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.264+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72bc0ad8b0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.428+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f72bc0ad8b0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72aa7fc700 1 -- 172.21.15.61:0/28852673 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 51 v51) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f72bc0ad8b0 con 0x7f72bc0ac3a0 2022-07-20T03:15:23.436 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:23.437 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":51,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":51,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:13.467553+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7008},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7008":{"gid":7008,"name":"a","rank":0,"incarnation":50,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1014823584","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1014823584},{"type":"v1","addr":"172.21.15.61:6833","nonce":1014823584}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:23.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f729c053920 msgr2=0x7f729c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f729c053920 0x7f729c055de0 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7f72ac01c000 tx=0x7f72ac015000).stop 2022-07-20T03:15:23.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 msgr2=0x7f72bc05c960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc05c960 secure :-1 s=READY pgs=982 cs=0 l=1 rev1=1 rx=0x7f72bc1430c0 tx=0x7f72b8008040).stop 2022-07-20T03:15:23.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 shutdown_connections 2022-07-20T03:15:23.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f729c053920 0x7f729c055de0 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72bc143960 0x7f72bc155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72bc0b1ef0 0x7f72bc155540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 --2- 172.21.15.61:0/28852673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72bc0ac3a0 0x7f72bc05c960 unknown :-1 s=CLOSED pgs=982 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 >> 172.21.15.61:0/28852673 conn(0x7f72bc0b2ca0 msgr2=0x7f72bc136450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:23.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 shutdown_connections 2022-07-20T03:15:23.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.432+0000 7f72c259d700 1 -- 172.21.15.61:0/28852673 wait complete. 2022-07-20T03:15:23.443 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 51 2022-07-20T03:15:23.449 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:15:23.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.572+0000 7f994d9f0700 1 Processor -- start 2022-07-20T03:15:23.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.572+0000 7f994d9f0700 1 -- start start 2022-07-20T03:15:23.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480b5990 0x7f99480ae050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9948143600 0x7f9948146960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9948142a90 con 0x7f99480af8e0 2022-07-20T03:15:23.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9948146e50 con 0x7f9948143600 2022-07-20T03:15:23.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480b5d70 con 0x7f99480b5990 2022-07-20T03:15:23.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9948143600 0x7f9948146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994c9ee700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480b5990 0x7f99480ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9948143600 0x7f9948146960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58112/0 (socket says 172.21.15.61:58112) 2022-07-20T03:15:23.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f9947fff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f99480af8e0 0x7f9948142ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:53530/0 (socket says 172.21.15.61:53530) 2022-07-20T03:15:23.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 -- 172.21.15.61:0/3581597204 learned_addr learned my addr 172.21.15.61:0/3581597204 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:23.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 -- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480b5990 msgr2=0x7f99480ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:23.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 --2- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480b5990 0x7f99480ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 -- 172.21.15.61:0/3581597204 >> v1:172.21.15.117:6789/0 conn(0x7f99480af8e0 legacy=0x7f9948142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 -- 172.21.15.61:0/3581597204 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99480afdf0 con 0x7f9948143600 2022-07-20T03:15:23.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f99477fe700 1 -- 172.21.15.61:0/3581597204 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3366303646 0 0) 0x7f9948142a90 con 0x7f99480af8e0 2022-07-20T03:15:23.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994c9ee700 1 --2- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480b5990 0x7f99480ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:23.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d1ef700 1 --2- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f9948146960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f99400032f0 tx=0x7f9940009290).ready entity=mon.1 client_cookie=3ca5a577cb8efe32 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:23.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f99477fe700 1 -- 172.21.15.61:0/3581597204 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f994000b8a0 con 0x7f9948143600 2022-07-20T03:15:23.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f99477fe700 1 -- 172.21.15.61:0/3581597204 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f994000ba00 con 0x7f9948143600 2022-07-20T03:15:23.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f99477fe700 1 -- 172.21.15.61:0/3581597204 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f994000bb70 con 0x7f9948143600 2022-07-20T03:15:23.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 msgr2=0x7f9948146960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 --2- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f9948146960 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f99400032f0 tx=0x7f9940009290).stop 2022-07-20T03:15:23.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- 172.21.15.61:0/3581597204 shutdown_connections 2022-07-20T03:15:23.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 --2- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f9948146960 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 --2- 172.21.15.61:0/3581597204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480b5990 0x7f99480ae050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- 172.21.15.61:0/3581597204 >> 172.21.15.61:0/3581597204 conn(0x7f9948063480 msgr2=0x7f99480b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:23.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- 172.21.15.61:0/3581597204 shutdown_connections 2022-07-20T03:15:23.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.576+0000 7f994d9f0700 1 -- 172.21.15.61:0/3581597204 wait complete. 2022-07-20T03:15:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 Processor -- start 2022-07-20T03:15:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 -- start start 2022-07-20T03:15:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480af8e0 0x7f9948112ae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 0x7f9948113410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f994813ebd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948115bf0 con 0x7f99480b5990 2022-07-20T03:15:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9948115e50 con 0x7f9948143600 2022-07-20T03:15:23.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99480671c0 con 0x7f99480af8e0 2022-07-20T03:15:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 0x7f9948113410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 0x7f9948113410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48436/0 (socket says 172.21.15.61:48436) 2022-07-20T03:15:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 -- 172.21.15.61:0/2125131795 learned_addr learned my addr 172.21.15.61:0/2125131795 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:23.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994c9ee700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480af8e0 0x7f9948112ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d1ef700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f994813ebd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 -- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480af8e0 msgr2=0x7f9948112ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480af8e0 0x7f9948112ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 -- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 msgr2=0x7f994813ebd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:23.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f994813ebd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 -- 172.21.15.61:0/2125131795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f993800b120 con 0x7f99480b5990 2022-07-20T03:15:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9947fff700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 0x7f9948113410 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f9930000ed0 tx=0x7f9930007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:23.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9930007170 con 0x7f99480b5990 2022-07-20T03:15:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f99300072d0 con 0x7f99480b5990 2022-07-20T03:15:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9940009530 con 0x7f99480b5990 2022-07-20T03:15:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9930004710 con 0x7f99480b5990 2022-07-20T03:15:23.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.580+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9948113c90 con 0x7f99480b5990 2022-07-20T03:15:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.584+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9930004870 con 0x7f99480b5990 2022-07-20T03:15:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.584+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9948116050 con 0x7f99480b5990 2022-07-20T03:15:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.584+0000 7f9945ffb700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f992c0539b0 0x7f992c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:23.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.584+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(91..91 src has 1..91) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f993006b760 con 0x7f99480b5990 2022-07-20T03:15:23.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.584+0000 7f994c9ee700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f992c0539b0 0x7f992c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:23.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.584+0000 7f994c9ee700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f992c0539b0 0x7f992c055e70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f9938008000 tx=0x7f9938008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:23.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.592+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9948116050 con 0x7f99480b5990 2022-07-20T03:15:23.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:23.752+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f9948067000 con 0x7f99480b5990 2022-07-20T03:15:24.268 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:15:24.256+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:15:24.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.272+0000 7f9945ffb700 1 -- 172.21.15.61:0/2125131795 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v52) v1 ==== 161+0+0 (secure 0 0 0) 0x7f9930042620 con 0x7f99480b5990 2022-07-20T03:15:24.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f992c0539b0 msgr2=0x7f992c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f992c0539b0 0x7f992c055e70 secure :-1 s=READY pgs=476 cs=0 l=1 rev1=1 rx=0x7f9938008000 tx=0x7f9938008040).stop 2022-07-20T03:15:24.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 msgr2=0x7f9948113410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 0x7f9948113410 secure :-1 s=READY pgs=983 cs=0 l=1 rev1=1 rx=0x7f9930000ed0 tx=0x7f9930007a80).stop 2022-07-20T03:15:24.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 shutdown_connections 2022-07-20T03:15:24.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f992c0539b0 0x7f992c055e70 unknown :-1 s=CLOSED pgs=476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9948143600 0x7f994813ebd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99480b5990 0x7f9948113410 unknown :-1 s=CLOSED pgs=983 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 --2- 172.21.15.61:0/2125131795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99480af8e0 0x7f9948112ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 >> 172.21.15.61:0/2125131795 conn(0x7f9948063480 msgr2=0x7f99480b0a70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:24.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 shutdown_connections 2022-07-20T03:15:24.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.276+0000 7f994d9f0700 1 -- 172.21.15.61:0/2125131795 wait complete. 2022-07-20T03:15:24.293 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:15:24.298 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:24.315 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:15:24.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 Processor -- start 2022-07-20T03:15:24.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 -- start start 2022-07-20T03:15:24.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2b01424e0 0x7ff2b00b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff2b00aea10 0x7ff2b00ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff2b01433d0 con 0x7ff2b00b0570 2022-07-20T03:15:24.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b0143b10 con 0x7ff2b01424e0 2022-07-20T03:15:24.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2b0143670 con 0x7ff2b00aea10 2022-07-20T03:15:24.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2b01424e0 0x7ff2b00b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2b01424e0 0x7ff2b00b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48456/0 (socket says 172.21.15.61:48456) 2022-07-20T03:15:24.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 -- 172.21.15.61:0/1742280631 learned_addr learned my addr 172.21.15.61:0/1742280631 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:24.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b4e21700 1 --1- 172.21.15.61:0/1742280631 >> v1:172.21.15.61:6789/0 conn(0x7ff2b00b0570 0x7ff2b0141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45052/0 (socket says 172.21.15.61:45052) 2022-07-20T03:15:24.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5622700 1 --2- 172.21.15.61:0/1742280631 >> v2:172.21.15.117:3300/0 conn(0x7ff2b00aea10 0x7ff2b00ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 -- 172.21.15.61:0/1742280631 >> v2:172.21.15.117:3300/0 conn(0x7ff2b00aea10 msgr2=0x7ff2b00ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:24.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 --2- 172.21.15.61:0/1742280631 >> v2:172.21.15.117:3300/0 conn(0x7ff2b00aea10 0x7ff2b00ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 -- 172.21.15.61:0/1742280631 >> v1:172.21.15.61:6789/0 conn(0x7ff2b00b0570 legacy=0x7ff2b0141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 -- 172.21.15.61:0/1742280631 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2b00b1010 con 0x7ff2b01424e0 2022-07-20T03:15:24.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2affff700 1 --2- 172.21.15.61:0/1742280631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b00b0950 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7ff2a0002f70 tx=0x7ff2a0006eb0).ready entity=mon.0 client_cookie=2d6675808dbf6a83 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:24.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2af7fe700 1 -- 172.21.15.61:0/1742280631 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2a000d610 con 0x7ff2b01424e0 2022-07-20T03:15:24.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2af7fe700 1 -- 172.21.15.61:0/1742280631 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a0000e90 con 0x7ff2b01424e0 2022-07-20T03:15:24.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2af7fe700 1 -- 172.21.15.61:0/1742280631 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2a000d9a0 con 0x7ff2b01424e0 2022-07-20T03:15:24.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 -- 172.21.15.61:0/1742280631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 msgr2=0x7ff2b00b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.400+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/1742280631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b00b0950 secure :-1 s=READY pgs=988 cs=0 l=1 rev1=1 rx=0x7ff2a0002f70 tx=0x7ff2a0006eb0).stop 2022-07-20T03:15:24.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 -- 172.21.15.61:0/1742280631 shutdown_connections 2022-07-20T03:15:24.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/1742280631 >> v2:172.21.15.117:3300/0 conn(0x7ff2b00aea10 0x7ff2b00ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/1742280631 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b00b0950 unknown :-1 s=CLOSED pgs=988 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 -- 172.21.15.61:0/1742280631 >> 172.21.15.61:0/1742280631 conn(0x7ff2b00b9360 msgr2=0x7ff2b00b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 -- 172.21.15.61:0/1742280631 shutdown_connections 2022-07-20T03:15:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 -- 172.21.15.61:0/1742280631 wait complete. 2022-07-20T03:15:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 Processor -- start 2022-07-20T03:15:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 -- start start 2022-07-20T03:15:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.404+0000 7ff2b5e23700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2b00aea10 0x7ff2b01126f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2b00b0570 0x7ff2b0155a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b0155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b0156410 con 0x7ff2b01424e0 2022-07-20T03:15:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b00b3810 con 0x7ff2b00aea10 2022-07-20T03:15:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2b00b3ab0 con 0x7ff2b00b0570 2022-07-20T03:15:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b0155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b0155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48464/0 (socket says 172.21.15.61:48464) 2022-07-20T03:15:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 -- 172.21.15.61:0/4262470661 learned_addr learned my addr 172.21.15.61:0/4262470661 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2affff700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2b00b0570 0x7ff2b0155a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b4e21700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2b00aea10 0x7ff2b01126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 -- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2b00b0570 msgr2=0x7ff2b0155a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2b00b0570 0x7ff2b0155a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 -- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2b00aea10 msgr2=0x7ff2b01126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2b00aea10 0x7ff2b01126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 -- 172.21.15.61:0/4262470661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2a000b1e0 con 0x7ff2b01424e0 2022-07-20T03:15:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b4e21700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2b00aea10 0x7ff2b01126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5622700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b0155f20 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7ff2a800a000 tx=0x7ff2a8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2a8013620 con 0x7ff2b01424e0 2022-07-20T03:15:24.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff2a8013780 con 0x7ff2b01424e0 2022-07-20T03:15:24.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2b00b3d50 con 0x7ff2b01424e0 2022-07-20T03:15:24.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff2a8013ae0 con 0x7ff2b01424e0 2022-07-20T03:15:24.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2b0112be0 con 0x7ff2b01424e0 2022-07-20T03:15:24.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2b00ace30 con 0x7ff2b01424e0 2022-07-20T03:15:24.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff2a8013c40 con 0x7ff2b01424e0 2022-07-20T03:15:24.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2adffb700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff2940539b0 0x7ff294055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.408+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6159+0+0 (secure 0 0 0) 0x7ff2a806c370 con 0x7ff2b01424e0 2022-07-20T03:15:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.412+0000 7ff2b4e21700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff2940539b0 0x7ff294055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.412+0000 7ff2b4e21700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff2940539b0 0x7ff294055e70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7ff298008000 tx=0x7ff298009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.412+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff2a8043220 con 0x7ff2b01424e0 2022-07-20T03:15:24.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.576+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff2b00574e0 con 0x7ff2b01424e0 2022-07-20T03:15:24.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.576+0000 7ff2adffb700 1 -- 172.21.15.61:0/4262470661 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 52 v52) v1 ==== 94+0+1782 (secure 0 0 0) 0x7ff2a8046840 con 0x7ff2b01424e0 2022-07-20T03:15:24.585 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:24.585 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":52,"default_fscid":8,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49}],"filesystems":[{"mdsmap":{"epoch":52,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:12.462538+0000","modified":"2022-07-20T03:15:24.261777+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":92,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[18],"metadata_pool":17,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":8}]} 2022-07-20T03:15:24.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff2940539b0 msgr2=0x7ff294055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff2940539b0 0x7ff294055e70 secure :-1 s=READY pgs=477 cs=0 l=1 rev1=1 rx=0x7ff298008000 tx=0x7ff298009040).stop 2022-07-20T03:15:24.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 msgr2=0x7ff2b0155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b0155f20 secure :-1 s=READY pgs=989 cs=0 l=1 rev1=1 rx=0x7ff2a800a000 tx=0x7ff2a8008040).stop 2022-07-20T03:15:24.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 shutdown_connections 2022-07-20T03:15:24.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff2940539b0 0x7ff294055e70 unknown :-1 s=CLOSED pgs=477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2b01424e0 0x7ff2b0155f20 unknown :-1 s=CLOSED pgs=989 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2b00b0570 0x7ff2b0155a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 --2- 172.21.15.61:0/4262470661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff2b00aea10 0x7ff2b01126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 >> 172.21.15.61:0/4262470661 conn(0x7ff2b00b9360 msgr2=0x7ff2b0141200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:24.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 shutdown_connections 2022-07-20T03:15:24.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.580+0000 7ff2b5e23700 1 -- 172.21.15.61:0/4262470661 wait complete. 2022-07-20T03:15:24.592 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 52 2022-07-20T03:15:24.595 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:15:24.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 Processor -- start 2022-07-20T03:15:24.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 -- start start 2022-07-20T03:15:24.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d7805c7c0 0x7f3d780ac360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3d780b23c0 0x7f3d780ac850 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d78098cf0 con 0x7f3d780ab250 2022-07-20T03:15:24.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d78096440 con 0x7f3d7805c7c0 2022-07-20T03:15:24.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d77fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d780ac0f0 con 0x7f3d780b23c0 2022-07-20T03:15:24.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d7805c7c0 0x7f3d780ac360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d767fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3d780b23c0 0x7f3d780ac850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3d7805c7c0 0x7f3d780ac360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48468/0 (socket says 172.21.15.61:48468) 2022-07-20T03:15:24.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 -- 172.21.15.61:0/992816294 learned_addr learned my addr 172.21.15.61:0/992816294 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:24.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 -- 172.21.15.61:0/992816294 >> v2:172.21.15.117:3300/0 conn(0x7f3d780b23c0 msgr2=0x7f3d780ac850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:24.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 --2- 172.21.15.61:0/992816294 >> v2:172.21.15.117:3300/0 conn(0x7f3d780b23c0 0x7f3d780ac850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 -- 172.21.15.61:0/992816294 >> v1:172.21.15.117:6789/0 conn(0x7f3d780ab250 legacy=0x7f3d780b0bb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d76ffd700 1 -- 172.21.15.61:0/992816294 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d78095bc0 con 0x7f3d7805c7c0 2022-07-20T03:15:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.696+0000 7f3d767fc700 1 --2- 172.21.15.61:0/992816294 >> v2:172.21.15.117:3300/0 conn(0x7f3d780b23c0 0x7f3d780ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:24.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d76ffd700 1 --2- 172.21.15.61:0/992816294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d780ac360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f3d70002f70 tx=0x7f3d70006eb0).ready entity=mon.0 client_cookie=2dc0e791f043586f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d75ffb700 1 -- 172.21.15.61:0/992816294 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d7000d5c0 con 0x7f3d7805c7c0 2022-07-20T03:15:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d75ffb700 1 -- 172.21.15.61:0/992816294 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d70000e90 con 0x7f3d7805c7c0 2022-07-20T03:15:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d75ffb700 1 -- 172.21.15.61:0/992816294 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d7000d950 con 0x7f3d7805c7c0 2022-07-20T03:15:24.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- 172.21.15.61:0/992816294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 msgr2=0x7f3d780ac360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 --2- 172.21.15.61:0/992816294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d780ac360 secure :-1 s=READY pgs=990 cs=0 l=1 rev1=1 rx=0x7f3d70002f70 tx=0x7f3d70006eb0).stop 2022-07-20T03:15:24.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- 172.21.15.61:0/992816294 shutdown_connections 2022-07-20T03:15:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 --2- 172.21.15.61:0/992816294 >> v2:172.21.15.117:3300/0 conn(0x7f3d780b23c0 0x7f3d780ac850 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 --2- 172.21.15.61:0/992816294 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d780ac360 unknown :-1 s=CLOSED pgs=990 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- 172.21.15.61:0/992816294 >> 172.21.15.61:0/992816294 conn(0x7f3d780b6210 msgr2=0x7f3d780b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:24.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- 172.21.15.61:0/992816294 shutdown_connections 2022-07-20T03:15:24.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- 172.21.15.61:0/992816294 wait complete. 2022-07-20T03:15:24.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 Processor -- start 2022-07-20T03:15:24.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- start start 2022-07-20T03:15:24.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d78061590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d780ab250 0x7f3d7814a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d780b23c0 0x7f3d78066e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d780622f0 con 0x7f3d7805c7c0 2022-07-20T03:15:24.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d78061130 con 0x7f3d780ab250 2022-07-20T03:15:24.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d77fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d78062520 con 0x7f3d780b23c0 2022-07-20T03:15:24.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d78061590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d76ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d78061590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48472/0 (socket says 172.21.15.61:48472) 2022-07-20T03:15:24.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d76ffd700 1 -- 172.21.15.61:0/4049036712 learned_addr learned my addr 172.21.15.61:0/4049036712 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:24.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d767fc700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d780ab250 0x7f3d7814a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d777fe700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d780b23c0 0x7f3d78066e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d767fc700 1 -- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d780b23c0 msgr2=0x7f3d78066e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d767fc700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d780b23c0 0x7f3d78066e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d767fc700 1 -- 172.21.15.61:0/4049036712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 msgr2=0x7f3d78061590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:24.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d767fc700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d78061590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:24.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.700+0000 7f3d767fc700 1 -- 172.21.15.61:0/4049036712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d6000b180 con 0x7f3d780ab250 2022-07-20T03:15:24.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d767fc700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d780ab250 0x7f3d7814a9d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f3d6000c000 tx=0x7f3d60006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d60010780 con 0x7f3d780ab250 2022-07-20T03:15:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d600108e0 con 0x7f3d780ab250 2022-07-20T03:15:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d7000b1e0 con 0x7f3d780ab250 2022-07-20T03:15:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d60010c40 con 0x7f3d780ab250 2022-07-20T03:15:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d7809b9f0 con 0x7f3d780ab250 2022-07-20T03:15:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d780abc50 con 0x7f3d780ab250 2022-07-20T03:15:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3d6001c460 con 0x7f3d780ab250 2022-07-20T03:15:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d6ffff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d500539d0 0x7f3d50055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3d6006c300 con 0x7f3d780ab250 2022-07-20T03:15:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.704+0000 7f3d76ffd700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d500539d0 0x7f3d50055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.708+0000 7f3d76ffd700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d500539d0 0x7f3d50055e90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f3d7001c000 tx=0x7f3d70015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.712+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d60043110 con 0x7f3d780ab250 2022-07-20T03:15:24.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:24.876+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f3d78061270 con 0x7f3d780ab250 2022-07-20T03:15:25.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.276+0000 7f3d6ffff700 1 -- 172.21.15.61:0/4049036712 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v53) v1 ==== 104+0+0 (secure 0 0 0) 0x7f3d60046730 con 0x7f3d780ab250 2022-07-20T03:15:25.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d500539d0 msgr2=0x7f3d50055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:25.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d500539d0 0x7f3d50055e90 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f3d7001c000 tx=0x7f3d70015000).stop 2022-07-20T03:15:25.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d780ab250 msgr2=0x7f3d7814a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:25.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d780ab250 0x7f3d7814a9d0 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f3d6000c000 tx=0x7f3d60006df0).stop 2022-07-20T03:15:25.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 shutdown_connections 2022-07-20T03:15:25.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d500539d0 0x7f3d50055e90 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d780ab250 0x7f3d7814a9d0 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d780b23c0 0x7f3d78066e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 --2- 172.21.15.61:0/4049036712 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d7805c7c0 0x7f3d78061590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 >> 172.21.15.61:0/4049036712 conn(0x7f3d780b6210 msgr2=0x7f3d780b2a00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:25.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 shutdown_connections 2022-07-20T03:15:25.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.280+0000 7f3d77fff700 1 -- 172.21.15.61:0/4049036712 wait complete. 2022-07-20T03:15:25.316 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:15:25.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 Processor -- start 2022-07-20T03:15:25.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 -- start start 2022-07-20T03:15:25.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f71300b5990 0x7f7130146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:25.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713005d210 0x7f713014a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:25.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7130146c70 con 0x7f71300af3e0 2022-07-20T03:15:25.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f71300ad050 con 0x7f71300b5990 2022-07-20T03:15:25.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.432+0000 7f7137555700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f71300b17e0 con 0x7f713005d210 2022-07-20T03:15:25.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f71300b5990 0x7f7130146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:25.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f71300b5990 0x7f7130146070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58162/0 (socket says 172.21.15.61:58162) 2022-07-20T03:15:25.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 -- 172.21.15.61:0/705635886 learned_addr learned my addr 172.21.15.61:0/705635886 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:25.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7135d52700 1 --1- 172.21.15.61:0/705635886 >> v1:172.21.15.117:6789/0 conn(0x7f71300af3e0 0x7f7130146560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:53586/0 (socket says 172.21.15.61:53586) 2022-07-20T03:15:25.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136d54700 1 --2- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713005d210 0x7f713014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:25.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 -- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713005d210 msgr2=0x7f713014a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:25.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 --2- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713005d210 0x7f713014a360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 -- 172.21.15.61:0/705635886 >> v1:172.21.15.117:6789/0 conn(0x7f71300af3e0 legacy=0x7f7130146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:25.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7135551700 1 -- 172.21.15.61:0/705635886 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4167754919 0 0) 0x7f7130146c70 con 0x7f71300af3e0 2022-07-20T03:15:25.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 -- 172.21.15.61:0/705635886 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f71300b5d70 con 0x7f71300b5990 2022-07-20T03:15:25.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7136553700 1 --2- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71300b5990 0x7f7130146070 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f7120002f70 tx=0x7f7120006e50).ready entity=mon.1 client_cookie=e412717bdd6abc47 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:25.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7135551700 1 -- 172.21.15.61:0/705635886 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f712000d5c0 con 0x7f71300b5990 2022-07-20T03:15:25.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7135551700 1 -- 172.21.15.61:0/705635886 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7120000e90 con 0x7f71300b5990 2022-07-20T03:15:25.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7135551700 1 -- 172.21.15.61:0/705635886 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f712000d910 con 0x7f71300b5990 2022-07-20T03:15:25.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 -- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71300b5990 msgr2=0x7f7130146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:25.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 --2- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71300b5990 0x7f7130146070 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7f7120002f70 tx=0x7f7120006e50).stop 2022-07-20T03:15:25.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 -- 172.21.15.61:0/705635886 shutdown_connections 2022-07-20T03:15:25.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 --2- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f713005d210 0x7f713014a360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 --2- 172.21.15.61:0/705635886 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f71300b5990 0x7f7130146070 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 -- 172.21.15.61:0/705635886 >> 172.21.15.61:0/705635886 conn(0x7f7130063480 msgr2=0x7f71300b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:25.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 -- 172.21.15.61:0/705635886 shutdown_connections 2022-07-20T03:15:25.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.436+0000 7f7137555700 1 -- 172.21.15.61:0/705635886 wait complete. 2022-07-20T03:15:25.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 Processor -- start 2022-07-20T03:15:25.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 -- start start 2022-07-20T03:15:25.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713005d210 0x7f713014a4d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:25.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71300af3e0 0x7f7130159e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:25.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 0x7f713015a350 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f713015a840 con 0x7f71300b5990 2022-07-20T03:15:25.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7130111dc0 con 0x7f713005d210 2022-07-20T03:15:25.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7130111b20 con 0x7f71300af3e0 2022-07-20T03:15:25.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 0x7f713015a350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:25.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 0x7f713015a350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48492/0 (socket says 172.21.15.61:48492) 2022-07-20T03:15:25.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 -- 172.21.15.61:0/891062319 learned_addr learned my addr 172.21.15.61:0/891062319 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7135d52700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71300af3e0 0x7f7130159e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:25.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136553700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713005d210 0x7f713014a4d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:25.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 -- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71300af3e0 msgr2=0x7f7130159e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:25.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71300af3e0 0x7f7130159e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 -- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713005d210 msgr2=0x7f713014a4d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:25.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713005d210 0x7f713014a4d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:25.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 -- 172.21.15.61:0/891062319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f712000b1e0 con 0x7f71300b5990 2022-07-20T03:15:25.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7136d54700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 0x7f713015a350 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f7128006e00 tx=0x7f71280091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:25.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f712800c640 con 0x7f71300b5990 2022-07-20T03:15:25.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f712800c7a0 con 0x7f71300b5990 2022-07-20T03:15:25.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f71301599c0 con 0x7f71300b5990 2022-07-20T03:15:25.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f712800cb00 con 0x7f71300b5990 2022-07-20T03:15:25.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.440+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7130144140 con 0x7f71300b5990 2022-07-20T03:15:25.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.444+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f71300afa60 con 0x7f71300b5990 2022-07-20T03:15:25.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.444+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7128022070 con 0x7f71300b5990 2022-07-20T03:15:25.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.444+0000 7f712f7fe700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71100539b0 0x7f7110055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:25.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.444+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(92..92 src has 1..92) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f712806cc00 con 0x7f71300b5990 2022-07-20T03:15:25.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.444+0000 7f7136553700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71100539b0 0x7f7110055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:25.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.444+0000 7f7136553700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71100539b0 0x7f7110055e70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f712001a000 tx=0x7f7120014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:25.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.448+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7128043a00 con 0x7f71300b5990 2022-07-20T03:15:25.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:25.576+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f71300574e0 con 0x7f71300b5990 2022-07-20T03:15:26.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.276+0000 7f712f7fe700 1 -- 172.21.15.61:0/891062319 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v93) v1 ==== 185+0+0 (secure 0 0 0) 0x7f7128029020 con 0x7f71300b5990 2022-07-20T03:15:26.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71100539b0 msgr2=0x7f7110055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71100539b0 0x7f7110055e70 secure :-1 s=READY pgs=480 cs=0 l=1 rev1=1 rx=0x7f712001a000 tx=0x7f7120014000).stop 2022-07-20T03:15:26.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 msgr2=0x7f713015a350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 0x7f713015a350 secure :-1 s=READY pgs=991 cs=0 l=1 rev1=1 rx=0x7f7128006e00 tx=0x7f71280091d0).stop 2022-07-20T03:15:26.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 shutdown_connections 2022-07-20T03:15:26.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f71100539b0 0x7f7110055e70 unknown :-1 s=CLOSED pgs=480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f71300b5990 0x7f713015a350 unknown :-1 s=CLOSED pgs=991 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f71300af3e0 0x7f7130159e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 --2- 172.21.15.61:0/891062319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f713005d210 0x7f713014a4d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 >> 172.21.15.61:0/891062319 conn(0x7f7130063480 msgr2=0x7f71300b4720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:26.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.280+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 shutdown_connections 2022-07-20T03:15:26.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.284+0000 7f7137555700 1 -- 172.21.15.61:0/891062319 wait complete. 2022-07-20T03:15:26.294 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:15:26.302 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:15:26.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.416+0000 7f7b50d5d700 1 Processor -- start 2022-07-20T03:15:26.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.416+0000 7f7b50d5d700 1 -- start start 2022-07-20T03:15:26.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7b4c061ba0 0x7f7b4c142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:26.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c05f840 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7b4c061ba0 0x7f7b4c142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:26.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7b4c061ba0 0x7f7b4c142720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48500/0 (socket says 172.21.15.61:48500) 2022-07-20T03:15:26.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/3125842702 learned_addr learned my addr 172.21.15.61:0/3125842702 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:26.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7b4c0b34d0 0x7f7b4c142010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45094/0 (socket says 172.21.15.61:45094) 2022-07-20T03:15:26.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b4c05fa70 con 0x7f7b4c0ab180 2022-07-20T03:15:26.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b4c05fd10 con 0x7f7b4c061ba0 2022-07-20T03:15:26.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/3125842702 >> v1:172.21.15.117:6789/0 conn(0x7f7b4c0ab180 legacy=0x7f7b4c142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/3125842702 >> v1:172.21.15.61:6789/0 conn(0x7f7b4c0b34d0 legacy=0x7f7b4c142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/3125842702 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b4c05ffb0 con 0x7f7b4c061ba0 2022-07-20T03:15:26.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4ad9d700 1 --2- 172.21.15.61:0/3125842702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c061ba0 0x7f7b4c142720 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f7b440032f0 tx=0x7f7b44009290).ready entity=mon.0 client_cookie=af3648f3ed08b372 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:26.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4959a700 1 -- 172.21.15.61:0/3125842702 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b4400b8a0 con 0x7f7b4c061ba0 2022-07-20T03:15:26.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4959a700 1 -- 172.21.15.61:0/3125842702 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4400ba00 con 0x7f7b4c061ba0 2022-07-20T03:15:26.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b4959a700 1 -- 172.21.15.61:0/3125842702 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b4400bb70 con 0x7f7b4c061ba0 2022-07-20T03:15:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c061ba0 msgr2=0x7f7b4c142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/3125842702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c061ba0 0x7f7b4c142720 secure :-1 s=READY pgs=993 cs=0 l=1 rev1=1 rx=0x7f7b440032f0 tx=0x7f7b44009290).stop 2022-07-20T03:15:26.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 shutdown_connections 2022-07-20T03:15:26.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/3125842702 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c061ba0 0x7f7b4c142720 unknown :-1 s=CLOSED pgs=993 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.420+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 >> 172.21.15.61:0/3125842702 conn(0x7f7b4c0b6c50 msgr2=0x7f7b4c0b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 shutdown_connections 2022-07-20T03:15:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 -- 172.21.15.61:0/3125842702 wait complete. 2022-07-20T03:15:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 Processor -- start 2022-07-20T03:15:26.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 -- start start 2022-07-20T03:15:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b4c061ba0 0x7f7b4c155a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b4c0ab180 0x7f7b4c099230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 0x7f7b4c0ae7b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:26.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0b0390 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c1562c0 con 0x7f7b4c0ab180 2022-07-20T03:15:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.424+0000 7f7b50d5d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b4c0ae070 con 0x7f7b4c061ba0 2022-07-20T03:15:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 0x7f7b4c0ae7b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 0x7f7b4c0ae7b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48506/0 (socket says 172.21.15.61:48506) 2022-07-20T03:15:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/4050192985 learned_addr learned my addr 172.21.15.61:0/4050192985 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4a59c700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b4c061ba0 0x7f7b4c155a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b49d9b700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b4c0ab180 0x7f7b4c099230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b4c061ba0 msgr2=0x7f7b4c155a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b4c061ba0 0x7f7b4c155a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b4c0ab180 msgr2=0x7f7b4c099230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b4c0ab180 0x7f7b4c099230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 -- 172.21.15.61:0/4050192985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b44009530 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b4ad9d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 0x7f7b4c0ae7b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f7b440032f0 tx=0x7f7b44016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:26.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b4400bde0 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b4401f3f0 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b4c0ae310 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b4c156090 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b4401f750 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7b44026030 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.428+0000 7f7b3b7fe700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b280539b0 0x7f7b28055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.432+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b4c0ae4a0 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.432+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(93..93 src has 1..93) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f7b4406c950 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.432+0000 7f7b4a59c700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b280539b0 0x7f7b28055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:26.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.432+0000 7f7b4a59c700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b280539b0 0x7f7b28055e70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f7b3c00d000 tx=0x7f7b3c007b80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:26.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.436+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7b44024030 con 0x7f7b4c0b34d0 2022-07-20T03:15:26.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:26.560+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7b4c0b38b0 con 0x7f7b4c0b34d0 2022-07-20T03:15:27.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.292+0000 7f7b3b7fe700 1 -- 172.21.15.61:0/4050192985 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v94) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7b44015070 con 0x7f7b4c0b34d0 2022-07-20T03:15:27.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b280539b0 msgr2=0x7f7b28055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b280539b0 0x7f7b28055e70 secure :-1 s=READY pgs=481 cs=0 l=1 rev1=1 rx=0x7f7b3c00d000 tx=0x7f7b3c007b80).stop 2022-07-20T03:15:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 msgr2=0x7f7b4c0ae7b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 0x7f7b4c0ae7b0 secure :-1 s=READY pgs=994 cs=0 l=1 rev1=1 rx=0x7f7b440032f0 tx=0x7f7b44016000).stop 2022-07-20T03:15:27.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 shutdown_connections 2022-07-20T03:15:27.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b280539b0 0x7f7b28055e70 unknown :-1 s=CLOSED pgs=481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b4c0ab180 0x7f7b4c099230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b4c0b34d0 0x7f7b4c0ae7b0 unknown :-1 s=CLOSED pgs=994 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 --2- 172.21.15.61:0/4050192985 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b4c061ba0 0x7f7b4c155a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 >> 172.21.15.61:0/4050192985 conn(0x7f7b4c0b6c50 msgr2=0x7f7b4c0b0bd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:27.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 shutdown_connections 2022-07-20T03:15:27.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.296+0000 7f7b50d5d700 1 -- 172.21.15.61:0/4050192985 wait complete. 2022-07-20T03:15:27.304 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:15:27.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:15:27.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 Processor -- start 2022-07-20T03:15:27.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 -- start start 2022-07-20T03:15:27.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72b40b6200 0x7f72b4146320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f72b40b4fd0 0x7f72b414a520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f72b40b48d0 con 0x7f72b40b13f0 2022-07-20T03:15:27.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b41481a0 con 0x7f72b40b6200 2022-07-20T03:15:27.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba9bc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f72b40b65e0 con 0x7f72b40b4fd0 2022-07-20T03:15:27.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72b40b6200 0x7f72b4146320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f72b40b6200 0x7f72b4146320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48510/0 (socket says 172.21.15.61:48510) 2022-07-20T03:15:27.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 -- 172.21.15.61:0/776459567 learned_addr learned my addr 172.21.15.61:0/776459567 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:27.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72ba1bb700 1 --2- 172.21.15.61:0/776459567 >> v2:172.21.15.117:3300/0 conn(0x7f72b40b4fd0 0x7f72b414a520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 -- 172.21.15.61:0/776459567 >> v2:172.21.15.117:3300/0 conn(0x7f72b40b4fd0 msgr2=0x7f72b414a520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:27.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 --2- 172.21.15.61:0/776459567 >> v2:172.21.15.117:3300/0 conn(0x7f72b40b4fd0 0x7f72b414a520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 -- 172.21.15.61:0/776459567 >> v1:172.21.15.117:6789/0 conn(0x7f72b40b13f0 legacy=0x7f72b4146810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.420+0000 7f72b99ba700 1 -- 172.21.15.61:0/776459567 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72b40b4d00 con 0x7f72b40b6200 2022-07-20T03:15:27.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72b99ba700 1 --2- 172.21.15.61:0/776459567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4146320 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f72a8002f70 tx=0x7f72a8006eb0).ready entity=mon.0 client_cookie=b627f1cd984ad5f0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:27.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72b89b8700 1 -- 172.21.15.61:0/776459567 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72a800d5c0 con 0x7f72b40b6200 2022-07-20T03:15:27.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72b89b8700 1 -- 172.21.15.61:0/776459567 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72a8000e90 con 0x7f72b40b6200 2022-07-20T03:15:27.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72b89b8700 1 -- 172.21.15.61:0/776459567 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72a800d950 con 0x7f72b40b6200 2022-07-20T03:15:27.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 -- 172.21.15.61:0/776459567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 msgr2=0x7f72b4146320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/776459567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4146320 secure :-1 s=READY pgs=995 cs=0 l=1 rev1=1 rx=0x7f72a8002f70 tx=0x7f72a8006eb0).stop 2022-07-20T03:15:27.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 -- 172.21.15.61:0/776459567 shutdown_connections 2022-07-20T03:15:27.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/776459567 >> v2:172.21.15.117:3300/0 conn(0x7f72b40b4fd0 0x7f72b414a520 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/776459567 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4146320 unknown :-1 s=CLOSED pgs=995 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 -- 172.21.15.61:0/776459567 >> 172.21.15.61:0/776459567 conn(0x7f72b4063d10 msgr2=0x7f72b40b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:27.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 -- 172.21.15.61:0/776459567 shutdown_connections 2022-07-20T03:15:27.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 -- 172.21.15.61:0/776459567 wait complete. 2022-07-20T03:15:27.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 Processor -- start 2022-07-20T03:15:27.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.424+0000 7f72ba9bc700 1 -- start start 2022-07-20T03:15:27.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b40b13f0 0x7f72b414afd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72b40b4fd0 0x7f72b4144330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4148630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b415a680 con 0x7f72b40b6200 2022-07-20T03:15:27.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72b4066ba0 con 0x7f72b40b13f0 2022-07-20T03:15:27.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f72b415a8f0 con 0x7f72b40b4fd0 2022-07-20T03:15:27.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4148630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4148630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48520/0 (socket says 172.21.15.61:48520) 2022-07-20T03:15:27.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 -- 172.21.15.61:0/2018788992 learned_addr learned my addr 172.21.15.61:0/2018788992 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:27.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72b99ba700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b40b13f0 0x7f72b414afd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72b91b9700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72b40b4fd0 0x7f72b4144330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 -- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72b40b4fd0 msgr2=0x7f72b4144330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72b40b4fd0 0x7f72b4144330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 -- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b40b13f0 msgr2=0x7f72b414afd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b40b13f0 0x7f72b414afd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 -- 172.21.15.61:0/2018788992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72a800b1e0 con 0x7f72b40b6200 2022-07-20T03:15:27.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba1bb700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4148630 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f72ac00c000 tx=0x7f72ac006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:27.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ac011750 con 0x7f72b40b6200 2022-07-20T03:15:27.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f72ac0118b0 con 0x7f72b40b6200 2022-07-20T03:15:27.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72b415ab90 con 0x7f72b40b6200 2022-07-20T03:15:27.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f72ac011c10 con 0x7f72b40b6200 2022-07-20T03:15:27.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.428+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72b4065ed0 con 0x7f72b40b6200 2022-07-20T03:15:27.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.432+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72b40b4360 con 0x7f72b40b6200 2022-07-20T03:15:27.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.432+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f72ac011d70 con 0x7f72b40b6200 2022-07-20T03:15:27.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.432+0000 7f72a6ffd700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7294053a40 0x7f7294055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.432+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f72ac06be60 con 0x7f72b40b6200 2022-07-20T03:15:27.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.432+0000 7f72b99ba700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7294053a40 0x7f7294055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.432+0000 7f72b99ba700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7294053a40 0x7f7294055f00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f72a801c000 tx=0x7f72a8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:27.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.436+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f72ac0430a0 con 0x7f72b40b6200 2022-07-20T03:15:27.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.560+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f72b40574e0 con 0x7f72b40b6200 2022-07-20T03:15:27.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.568+0000 7f72a6ffd700 1 -- 172.21.15.61:0/2018788992 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v15) v1 ==== 104+0+0 (secure 0 0 0) 0x7f72ac0466c0 con 0x7f72b40b6200 2022-07-20T03:15:27.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7294053a40 msgr2=0x7f7294055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7294053a40 0x7f7294055f00 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f72a801c000 tx=0x7f72a8015000).stop 2022-07-20T03:15:27.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 msgr2=0x7f72b4148630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4148630 secure :-1 s=READY pgs=996 cs=0 l=1 rev1=1 rx=0x7f72ac00c000 tx=0x7f72ac006df0).stop 2022-07-20T03:15:27.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 shutdown_connections 2022-07-20T03:15:27.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7294053a40 0x7f7294055f00 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f72b40b6200 0x7f72b4148630 unknown :-1 s=CLOSED pgs=996 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f72b40b4fd0 0x7f72b4144330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 --2- 172.21.15.61:0/2018788992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f72b40b13f0 0x7f72b414afd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 >> 172.21.15.61:0/2018788992 conn(0x7f72b4063d10 msgr2=0x7f72b40b2220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:27.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 shutdown_connections 2022-07-20T03:15:27.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.572+0000 7f72ba9bc700 1 -- 172.21.15.61:0/2018788992 wait complete. 2022-07-20T03:15:27.590 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical' 2022-07-20T03:15:27.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 Processor -- start 2022-07-20T03:15:27.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 -- start start 2022-07-20T03:15:27.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7faed40b59d0 0x7faed40b0040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed40ae3d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faed40adf70 con 0x7faed405d0c0 2022-07-20T03:15:27.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faed40b5db0 con 0x7faed40b59d0 2022-07-20T03:15:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.692+0000 7faedb4ad700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed4146300 con 0x7faed40af920 2022-07-20T03:15:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed40ae3d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faeda4ab700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7faed40b59d0 0x7faed40b0040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed40ae3d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35196/0 (socket says 172.21.15.61:35196) 2022-07-20T03:15:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 -- 172.21.15.61:0/2268241616 learned_addr learned my addr 172.21.15.61:0/2268241616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:27.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 -- 172.21.15.61:0/2268241616 >> v2:172.21.15.117:3300/0 conn(0x7faed40b59d0 msgr2=0x7faed40b0040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:27.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 --2- 172.21.15.61:0/2268241616 >> v2:172.21.15.117:3300/0 conn(0x7faed40b59d0 0x7faed40b0040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 -- 172.21.15.61:0/2268241616 >> v1:172.21.15.117:6789/0 conn(0x7faed405d0c0 legacy=0x7faed40ae8c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 -- 172.21.15.61:0/2268241616 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faed4146060 con 0x7faed40af920 2022-07-20T03:15:27.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faeda4ab700 1 --2- 172.21.15.61:0/2268241616 >> v2:172.21.15.117:3300/0 conn(0x7faed40b59d0 0x7faed40b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:27.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed9caa700 1 --2- 172.21.15.61:0/2268241616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed40ae3d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7faed40add70 tx=0x7faed0006eb0).ready entity=mon.2 client_cookie=527ac20728c70639 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:27.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed94a9700 1 -- 172.21.15.61:0/2268241616 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faed000d5c0 con 0x7faed40af920 2022-07-20T03:15:27.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed94a9700 1 -- 172.21.15.61:0/2268241616 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faed0000e90 con 0x7faed40af920 2022-07-20T03:15:27.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faed94a9700 1 -- 172.21.15.61:0/2268241616 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faed000d950 con 0x7faed40af920 2022-07-20T03:15:27.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 -- 172.21.15.61:0/2268241616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 msgr2=0x7faed40ae3d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 --2- 172.21.15.61:0/2268241616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed40ae3d0 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7faed40add70 tx=0x7faed0006eb0).stop 2022-07-20T03:15:27.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 -- 172.21.15.61:0/2268241616 shutdown_connections 2022-07-20T03:15:27.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 --2- 172.21.15.61:0/2268241616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed40ae3d0 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 --2- 172.21.15.61:0/2268241616 >> v2:172.21.15.117:3300/0 conn(0x7faed40b59d0 0x7faed40b0040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 -- 172.21.15.61:0/2268241616 >> 172.21.15.61:0/2268241616 conn(0x7faed40634c0 msgr2=0x7faed40b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:27.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 -- 172.21.15.61:0/2268241616 shutdown_connections 2022-07-20T03:15:27.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 -- 172.21.15.61:0/2268241616 wait complete. 2022-07-20T03:15:27.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 Processor -- start 2022-07-20T03:15:27.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.696+0000 7faedb4ad700 1 -- start start 2022-07-20T03:15:27.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 0x7faed4064970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed4155810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faed40b59d0 0x7faed4155d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed413fa10 con 0x7faed405d0c0 2022-07-20T03:15:27.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faed4115040 con 0x7faed40b59d0 2022-07-20T03:15:27.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faed41553b0 con 0x7faed40af920 2022-07-20T03:15:27.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 0x7faed4064970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 0x7faed4064970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48530/0 (socket says 172.21.15.61:48530) 2022-07-20T03:15:27.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 -- 172.21.15.61:0/1733119953 learned_addr learned my addr 172.21.15.61:0/1733119953 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:27.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faed9caa700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed4155810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedacac700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faed40b59d0 0x7faed4155d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 -- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 msgr2=0x7faed4155810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed4155810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 -- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faed40b59d0 msgr2=0x7faed4155d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faed40b59d0 0x7faed4155d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 -- 172.21.15.61:0/1733119953 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faec400b180 con 0x7faed405d0c0 2022-07-20T03:15:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faeda4ab700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 0x7faed4064970 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7faec400c000 tx=0x7faec4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec4010780 con 0x7faed405d0c0 2022-07-20T03:15:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faec40108e0 con 0x7faed405d0c0 2022-07-20T03:15:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faed000b1e0 con 0x7faed405d0c0 2022-07-20T03:15:27.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faec4010c40 con 0x7faed405d0c0 2022-07-20T03:15:27.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.700+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faed4156270 con 0x7faed405d0c0 2022-07-20T03:15:27.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.704+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faec401c460 con 0x7faed405d0c0 2022-07-20T03:15:27.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.704+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faed40af2d0 con 0x7faed405d0c0 2022-07-20T03:15:27.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.704+0000 7faecb7fe700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faeb4053a40 0x7faeb4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:27.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.704+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7faec406c020 con 0x7faed405d0c0 2022-07-20T03:15:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.704+0000 7faed9caa700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faeb4053a40 0x7faeb4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.704+0000 7faed9caa700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faeb4053a40 0x7faeb4055f00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7faed001c000 tx=0x7faed0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:27.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.708+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faec4043200 con 0x7faed405d0c0 2022-07-20T03:15:27.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:27.876+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]} v 0) v1 -- 0x7faed40574e0 con 0x7faed405d0c0 2022-07-20T03:15:28.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.292+0000 7faecb7fe700 1 -- 172.21.15.61:0/1733119953 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_dump_inode_hexademical"]}]=0 v377) v1 ==== 138+0+0 (secure 0 0 0) 0x7faec4046820 con 0x7faed405d0c0 2022-07-20T03:15:28.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faeb4053a40 msgr2=0x7faeb4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:28.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faeb4053a40 0x7faeb4055f00 secure :-1 s=READY pgs=483 cs=0 l=1 rev1=1 rx=0x7faed001c000 tx=0x7faed0015000).stop 2022-07-20T03:15:28.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 msgr2=0x7faed4064970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:28.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 0x7faed4064970 secure :-1 s=READY pgs=997 cs=0 l=1 rev1=1 rx=0x7faec400c000 tx=0x7faec4006df0).stop 2022-07-20T03:15:28.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 shutdown_connections 2022-07-20T03:15:28.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faeb4053a40 0x7faeb4055f00 unknown :-1 s=CLOSED pgs=483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faed40b59d0 0x7faed4155d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faed40af920 0x7faed4155810 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 --2- 172.21.15.61:0/1733119953 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faed405d0c0 0x7faed4064970 unknown :-1 s=CLOSED pgs=997 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 >> 172.21.15.61:0/1733119953 conn(0x7faed40634c0 msgr2=0x7faed4142cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:28.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 shutdown_connections 2022-07-20T03:15:28.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.296+0000 7faedb4ad700 1 -- 172.21.15.61:0/1733119953 wait complete. 2022-07-20T03:15:28.320 INFO:tasks.cephfs_test_runner:test_dump_inode_hexademical (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:15:28.321 INFO:tasks.cephfs_test_runner:Starting test: test_filtered_df (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:15:28.322 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-20T03:15:28.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.424+0000 7f32ec905700 1 Processor -- start 2022-07-20T03:15:28.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.424+0000 7f32ec905700 1 -- start start 2022-07-20T03:15:28.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f32e813c790 0x7f32e80b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:28.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e8143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:28.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f32e8142ba0 con 0x7f32e80b3ca0 2022-07-20T03:15:28.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f32e81429b0 con 0x7f32e813c790 2022-07-20T03:15:28.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e8146330 con 0x7f32e813cb70 2022-07-20T03:15:28.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f32e813c790 0x7f32e80b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:28.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f32e813c790 0x7f32e80b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48540/0 (socket says 172.21.15.61:48540) 2022-07-20T03:15:28.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 -- 172.21.15.61:0/711864160 learned_addr learned my addr 172.21.15.61:0/711864160 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:28.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e7fff700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e8143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:28.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e67fc700 1 -- 172.21.15.61:0/711864160 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2053795693 0 0) 0x7f32e8142ba0 con 0x7f32e80b3ca0 2022-07-20T03:15:28.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 -- 172.21.15.61:0/711864160 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 msgr2=0x7f32e8143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:28.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e8143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 -- 172.21.15.61:0/711864160 >> v1:172.21.15.117:6789/0 conn(0x7f32e80b3ca0 legacy=0x7f32e80b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:28.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 -- 172.21.15.61:0/711864160 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e80b4c00 con 0x7f32e813c790 2022-07-20T03:15:28.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e7fff700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e8143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:28.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e77fe700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80b4710 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f32dc002f70 tx=0x7f32dc006eb0).ready entity=mon.0 client_cookie=410e46a31c471d72 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:28.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e67fc700 1 -- 172.21.15.61:0/711864160 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32dc00d570 con 0x7f32e813c790 2022-07-20T03:15:28.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e67fc700 1 -- 172.21.15.61:0/711864160 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32dc000e90 con 0x7f32e813c790 2022-07-20T03:15:28.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32e67fc700 1 -- 172.21.15.61:0/711864160 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32dc00d8c0 con 0x7f32e813c790 2022-07-20T03:15:28.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- 172.21.15.61:0/711864160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 msgr2=0x7f32e80b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:28.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80b4710 secure :-1 s=READY pgs=998 cs=0 l=1 rev1=1 rx=0x7f32dc002f70 tx=0x7f32dc006eb0).stop 2022-07-20T03:15:28.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- 172.21.15.61:0/711864160 shutdown_connections 2022-07-20T03:15:28.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e8143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 --2- 172.21.15.61:0/711864160 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80b4710 unknown :-1 s=CLOSED pgs=998 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- 172.21.15.61:0/711864160 >> 172.21.15.61:0/711864160 conn(0x7f32e80678b0 msgr2=0x7f32e80b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:28.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- 172.21.15.61:0/711864160 shutdown_connections 2022-07-20T03:15:28.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.428+0000 7f32ec905700 1 -- 172.21.15.61:0/711864160 wait complete. 2022-07-20T03:15:28.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 Processor -- start 2022-07-20T03:15:28.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 -- start start 2022-07-20T03:15:28.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e80b3ca0 0x7f32e813ea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80afde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e81557d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:28.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80b02d0 con 0x7f32e813c790 2022-07-20T03:15:28.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80b04f0 con 0x7f32e80b3ca0 2022-07-20T03:15:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32ec905700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f32e80b0790 con 0x7f32e813cb70 2022-07-20T03:15:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80afde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:28.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80afde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48546/0 (socket says 172.21.15.61:48546) 2022-07-20T03:15:28.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 -- 172.21.15.61:0/3622073552 learned_addr learned my addr 172.21.15.61:0/3622073552 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e77fe700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e80b3ca0 0x7f32e813ea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e7fff700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e81557d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:28.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 -- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 msgr2=0x7f32e81557d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:28.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e81557d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 -- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e80b3ca0 msgr2=0x7f32e813ea70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:28.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e80b3ca0 0x7f32e813ea70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:28.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 -- 172.21.15.61:0/3622073552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32e0009470 con 0x7f32e813c790 2022-07-20T03:15:28.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.432+0000 7f32e6ffd700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80afde0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f32d8000ed0 tx=0x7f32d8007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:28.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32d800cea0 con 0x7f32e813c790 2022-07-20T03:15:28.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f32d8012400 con 0x7f32e813c790 2022-07-20T03:15:28.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32dc00b1e0 con 0x7f32e813c790 2022-07-20T03:15:28.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f32d8012760 con 0x7f32e813c790 2022-07-20T03:15:28.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32e80ae3c0 con 0x7f32e813c790 2022-07-20T03:15:28.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32e8155510 con 0x7f32e813c790 2022-07-20T03:15:28.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f32d8019030 con 0x7f32e813c790 2022-07-20T03:15:28.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e4ff9700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32c80539b0 0x7f32c8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:28.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f32d806b090 con 0x7f32e813c790 2022-07-20T03:15:28.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.436+0000 7f32e77fe700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32c80539b0 0x7f32c8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:28.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.440+0000 7f32e77fe700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32c80539b0 0x7f32c8055e70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f32dc01c000 tx=0x7f32dc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:28.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.444+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f32e8155510 con 0x7f32e813c790 2022-07-20T03:15:28.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:28.620+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7f32e8155510 con 0x7f32e813c790 2022-07-20T03:15:29.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32e4ff9700 1 -- 172.21.15.61:0/3622073552 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v378) v1 ==== 130+0+0 (secure 0 0 0) 0x7f32e8155510 con 0x7f32e813c790 2022-07-20T03:15:29.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32c80539b0 msgr2=0x7f32c8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32c80539b0 0x7f32c8055e70 secure :-1 s=READY pgs=484 cs=0 l=1 rev1=1 rx=0x7f32dc01c000 tx=0x7f32dc015000).stop 2022-07-20T03:15:29.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 msgr2=0x7f32e80afde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80afde0 secure :-1 s=READY pgs=999 cs=0 l=1 rev1=1 rx=0x7f32d8000ed0 tx=0x7f32d8007c20).stop 2022-07-20T03:15:29.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 shutdown_connections 2022-07-20T03:15:29.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f32c80539b0 0x7f32c8055e70 unknown :-1 s=CLOSED pgs=484 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f32e813cb70 0x7f32e81557d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f32e813c790 0x7f32e80afde0 unknown :-1 s=CLOSED pgs=999 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 --2- 172.21.15.61:0/3622073552 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f32e80b3ca0 0x7f32e813ea70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.304+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 >> 172.21.15.61:0/3622073552 conn(0x7f32e80678b0 msgr2=0x7f32e80b7a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:29.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.308+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 shutdown_connections 2022-07-20T03:15:29.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.308+0000 7f32ec905700 1 -- 172.21.15.61:0/3622073552 wait complete. 2022-07-20T03:15:29.332 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:15:29.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.448+0000 7f65ed25a700 1 Processor -- start 2022-07-20T03:15:29.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.448+0000 7f65ed25a700 1 -- start start 2022-07-20T03:15:29.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80b0b90 0x7f65e80aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:29.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65e80ac9f0 0x7f65e8140520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:29.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f65e80aced0 con 0x7f65e80ac610 2022-07-20T03:15:29.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f65e8143c80 con 0x7f65e80ac9f0 2022-07-20T03:15:29.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8143f20 con 0x7f65e80b0b90 2022-07-20T03:15:29.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65e80ac9f0 0x7f65e8140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:29.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f65e80ac9f0 0x7f65e8140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48556/0 (socket says 172.21.15.61:48556) 2022-07-20T03:15:29.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 -- 172.21.15.61:0/1020226673 learned_addr learned my addr 172.21.15.61:0/1020226673 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:29.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65df7fe700 1 --2- 172.21.15.61:0/1020226673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80b0b90 0x7f65e80aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:29.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 -- 172.21.15.61:0/1020226673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80b0b90 msgr2=0x7f65e80aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:29.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 --2- 172.21.15.61:0/1020226673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80b0b90 0x7f65e80aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 -- 172.21.15.61:0/1020226673 >> v1:172.21.15.117:6789/0 conn(0x7f65e80ac610 legacy=0x7f65e805fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 -- 172.21.15.61:0/1020226673 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e8098b20 con 0x7f65e80ac9f0 2022-07-20T03:15:29.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65eca59700 1 --2- 172.21.15.61:0/1020226673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80ac9f0 0x7f65e8140520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f65e00032f0 tx=0x7f65e0009290).ready entity=mon.0 client_cookie=869236aeb8416f85 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:29.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65e77fe700 1 -- 172.21.15.61:0/1020226673 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e000b8a0 con 0x7f65e80ac9f0 2022-07-20T03:15:29.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65e77fe700 1 -- 172.21.15.61:0/1020226673 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65e000ba00 con 0x7f65e80ac9f0 2022-07-20T03:15:29.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65e77fe700 1 -- 172.21.15.61:0/1020226673 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65e000bb70 con 0x7f65e80ac9f0 2022-07-20T03:15:29.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- 172.21.15.61:0/1020226673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80ac9f0 msgr2=0x7f65e8140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1020226673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80ac9f0 0x7f65e8140520 secure :-1 s=READY pgs=1000 cs=0 l=1 rev1=1 rx=0x7f65e00032f0 tx=0x7f65e0009290).stop 2022-07-20T03:15:29.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- 172.21.15.61:0/1020226673 shutdown_connections 2022-07-20T03:15:29.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1020226673 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80ac9f0 0x7f65e8140520 unknown :-1 s=CLOSED pgs=1000 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1020226673 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80b0b90 0x7f65e80aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- 172.21.15.61:0/1020226673 >> 172.21.15.61:0/1020226673 conn(0x7f65e80b61f0 msgr2=0x7f65e80b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- 172.21.15.61:0/1020226673 shutdown_connections 2022-07-20T03:15:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.452+0000 7f65ed25a700 1 -- 172.21.15.61:0/1020226673 wait complete. 2022-07-20T03:15:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 Processor -- start 2022-07-20T03:15:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 -- start start 2022-07-20T03:15:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e80ac610 0x7f65e8099280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80ac9f0 0x7f65e80adb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80b0b90 0x7f65e814dff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e814bac0 con 0x7f65e80b0b90 2022-07-20T03:15:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65e80666a0 con 0x7f65e80ac610 2022-07-20T03:15:29.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65ed25a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f65e8151660 con 0x7f65e80ac9f0 2022-07-20T03:15:29.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65eca59700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80b0b90 0x7f65e814dff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:29.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65e7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e80ac610 0x7f65e8099280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:29.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65df7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80ac9f0 0x7f65e80adb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:29.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65eca59700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80b0b90 0x7f65e814dff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48562/0 (socket says 172.21.15.61:48562) 2022-07-20T03:15:29.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65eca59700 1 -- 172.21.15.61:0/1676639616 learned_addr learned my addr 172.21.15.61:0/1676639616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:29.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.456+0000 7f65e7fff700 1 -- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80ac9f0 msgr2=0x7f65e80adb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e7fff700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80ac9f0 0x7f65e80adb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e7fff700 1 -- 172.21.15.61:0/1676639616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80b0b90 msgr2=0x7f65e814dff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e7fff700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80b0b90 0x7f65e814dff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e7fff700 1 -- 172.21.15.61:0/1676639616 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65e0009530 con 0x7f65e80ac610 2022-07-20T03:15:29.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e7fff700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e80ac610 0x7f65e8099280 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f65e814bc00 tx=0x7f65d8007b90).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:29.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d8012400 con 0x7f65e80ac610 2022-07-20T03:15:29.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f65d8012560 con 0x7f65e80ac610 2022-07-20T03:15:29.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f65d80128c0 con 0x7f65e80ac610 2022-07-20T03:15:29.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f65e8151900 con 0x7f65e80ac610 2022-07-20T03:15:29.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f65e80998e0 con 0x7f65e80ac610 2022-07-20T03:15:29.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f65e8151a90 con 0x7f65e80ac610 2022-07-20T03:15:29.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f65d8019030 con 0x7f65e80ac610 2022-07-20T03:15:29.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e5ffb700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f65cc053a40 0x7f65cc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:29.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f65d806af80 con 0x7f65e80ac610 2022-07-20T03:15:29.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.460+0000 7f65df7fe700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f65cc053a40 0x7f65cc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:29.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.464+0000 7f65df7fe700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f65cc053a40 0x7f65cc055f00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f65d000a000 tx=0x7f65d0006e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:29.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.472+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f65d8042030 con 0x7f65e80ac610 2022-07-20T03:15:29.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.600+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f65e80574e0 con 0x7f65e80ac610 2022-07-20T03:15:29.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65e5ffb700 1 -- 172.21.15.61:0/1676639616 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v16)=0 v16) v1 ==== 122+0+0 (secure 0 0 0) 0x7f65d8045650 con 0x7f65e80ac610 2022-07-20T03:15:29.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f65cc053a40 msgr2=0x7f65cc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f65cc053a40 0x7f65cc055f00 secure :-1 s=READY pgs=485 cs=0 l=1 rev1=1 rx=0x7f65d000a000 tx=0x7f65d0006e80).stop 2022-07-20T03:15:29.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e80ac610 msgr2=0x7f65e8099280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:29.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e80ac610 0x7f65e8099280 secure :-1 s=READY pgs=455 cs=0 l=1 rev1=1 rx=0x7f65e814bc00 tx=0x7f65d8007b90).stop 2022-07-20T03:15:29.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 shutdown_connections 2022-07-20T03:15:29.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f65cc053a40 0x7f65cc055f00 unknown :-1 s=CLOSED pgs=485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f65e80b0b90 0x7f65e814dff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f65e80ac9f0 0x7f65e80adb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 --2- 172.21.15.61:0/1676639616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f65e80ac610 0x7f65e8099280 unknown :-1 s=CLOSED pgs=455 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:29.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 >> 172.21.15.61:0/1676639616 conn(0x7f65e80b61f0 msgr2=0x7f65e813f950 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:29.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 shutdown_connections 2022-07-20T03:15:29.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:29.612+0000 7f65ed25a700 1 -- 172.21.15.61:0/1676639616 wait complete. 2022-07-20T03:15:29.639 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:15:29.640 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:29.701 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:29.731 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:29.754 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:29.778 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:29.802 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:15:29.803 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:29.805 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:29.805 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:29.840 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:29.842 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:15:29.842 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:29.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:29.843 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:29.844 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:29.895 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:29.896 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:15:29.897 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:29.897 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:29.898 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:29.899 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:29.915 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:29.916 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:29.920 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:15:29.921 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:15:30.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 Processor -- start 2022-07-20T03:15:30.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 -- start start 2022-07-20T03:15:30.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe6e80b1ef0 0x7fe6e8142090 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6e8142580 0x7fe6e805c620 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6e80b22d0 con 0x7fe6e805cb10 2022-07-20T03:15:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e81472a0 con 0x7fe6e8142580 2022-07-20T03:15:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6e80ac0c0 con 0x7fe6e80b1ef0 2022-07-20T03:15:30.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6e8142580 0x7fe6e805c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6e8142580 0x7fe6e805c620 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48570/0 (socket says 172.21.15.61:48570) 2022-07-20T03:15:30.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 -- 172.21.15.61:0/2830893941 learned_addr learned my addr 172.21.15.61:0/2830893941 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:30.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6e77fe700 1 --2- 172.21.15.61:0/2830893941 >> v2:172.21.15.117:3300/0 conn(0x7fe6e80b1ef0 0x7fe6e8142090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 -- 172.21.15.61:0/2830893941 >> v2:172.21.15.117:3300/0 conn(0x7fe6e80b1ef0 msgr2=0x7fe6e8142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:30.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 --2- 172.21.15.61:0/2830893941 >> v2:172.21.15.117:3300/0 conn(0x7fe6e80b1ef0 0x7fe6e8142090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 -- 172.21.15.61:0/2830893941 >> v1:172.21.15.117:6789/0 conn(0x7fe6e805cb10 legacy=0x7fe6e8061130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 -- 172.21.15.61:0/2830893941 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6e80ac360 con 0x7fe6e8142580 2022-07-20T03:15:30.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6deffd700 1 --2- 172.21.15.61:0/2830893941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e8142580 0x7fe6e805c620 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7fe6d8002f70 tx=0x7fe6d8006eb0).ready entity=mon.0 client_cookie=eb5c8b71641f26de server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6e6ffd700 1 -- 172.21.15.61:0/2830893941 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6d800d5c0 con 0x7fe6e8142580 2022-07-20T03:15:30.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6e6ffd700 1 -- 172.21.15.61:0/2830893941 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d8000e90 con 0x7fe6e8142580 2022-07-20T03:15:30.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6e6ffd700 1 -- 172.21.15.61:0/2830893941 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6d800d950 con 0x7fe6e8142580 2022-07-20T03:15:30.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2830893941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e8142580 msgr2=0x7fe6e805c620 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.032+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2830893941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e8142580 0x7fe6e805c620 secure :-1 s=READY pgs=1001 cs=0 l=1 rev1=1 rx=0x7fe6d8002f70 tx=0x7fe6d8006eb0).stop 2022-07-20T03:15:30.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2830893941 shutdown_connections 2022-07-20T03:15:30.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2830893941 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e8142580 0x7fe6e805c620 unknown :-1 s=CLOSED pgs=1001 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2830893941 >> v2:172.21.15.117:3300/0 conn(0x7fe6e80b1ef0 0x7fe6e8142090 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2830893941 >> 172.21.15.61:0/2830893941 conn(0x7fe6e80b2ca0 msgr2=0x7fe6e80b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:30.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2830893941 shutdown_connections 2022-07-20T03:15:30.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2830893941 wait complete. 2022-07-20T03:15:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 Processor -- start 2022-07-20T03:15:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- start start 2022-07-20T03:15:30.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 0x7fe6e8061390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6e80b1ef0 0x7fe6e8099b20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6e8142580 0x7fe6e8155620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e809c240 con 0x7fe6e805cb10 2022-07-20T03:15:30.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e809b860 con 0x7fe6e8142580 2022-07-20T03:15:30.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6e809bb00 con 0x7fe6e80b1ef0 2022-07-20T03:15:30.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 0x7fe6e8061390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 0x7fe6e8061390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48572/0 (socket says 172.21.15.61:48572) 2022-07-20T03:15:30.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 -- 172.21.15.61:0/2887141038 learned_addr learned my addr 172.21.15.61:0/2887141038 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:30.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e7fff700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6e8142580 0x7fe6e8155620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6deffd700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6e80b1ef0 0x7fe6e8099b20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 -- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6e80b1ef0 msgr2=0x7fe6e8099b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6e80b1ef0 0x7fe6e8099b20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 -- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6e8142580 msgr2=0x7fe6e8155620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6e8142580 0x7fe6e8155620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 -- 172.21.15.61:0/2887141038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6d800b1e0 con 0x7fe6e805cb10 2022-07-20T03:15:30.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e77fe700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 0x7fe6e8061390 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7fe6d000c000 tx=0x7fe6d0006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6d00104c0 con 0x7fe6e805cb10 2022-07-20T03:15:30.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6d0010620 con 0x7fe6e805cb10 2022-07-20T03:15:30.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6e809bda0 con 0x7fe6e805cb10 2022-07-20T03:15:30.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6d0010980 con 0x7fe6e805cb10 2022-07-20T03:15:30.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.036+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6e8145a00 con 0x7fe6e805cb10 2022-07-20T03:15:30.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.040+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6e809a230 con 0x7fe6e805cb10 2022-07-20T03:15:30.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.040+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe6d001a030 con 0x7fe6e805cb10 2022-07-20T03:15:30.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.040+0000 7fe6e57fa700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6c00539d0 0x7fe6c0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.040+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe6d006b6d0 con 0x7fe6e805cb10 2022-07-20T03:15:30.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.040+0000 7fe6deffd700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6c00539d0 0x7fe6c0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.040+0000 7fe6deffd700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6c00539d0 0x7fe6c0055e90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fe6d801c000 tx=0x7fe6d8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.048+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6d00428c0 con 0x7fe6e805cb10 2022-07-20T03:15:30.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.176+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fe6e809bf30 con 0x7fe6e805cb10 2022-07-20T03:15:30.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.180+0000 7fe6e57fa700 1 -- 172.21.15.61:0/2887141038 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v94) v1 ==== 81+0+28876 (secure 0 0 0) 0x7fe6d0045ee0 con 0x7fe6e805cb10 2022-07-20T03:15:30.185 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:30.185 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:30.186 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 94, 2022-07-20T03:15:30.186 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:15:30.186 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:15:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:15:27.279940+0000", 2022-07-20T03:15:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:15:30.188 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:15:30.188 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:15:30.188 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:15:30.188 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:15:30.189 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:15:30.189 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:15:30.189 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:30.189 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:15:30.190 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:15:30.190 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:15:30.191 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:15:30.191 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:15:30.191 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 18, 2022-07-20T03:15:30.191 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:15:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:15:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:15:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:15:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:15:30.193 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.193 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:15:30.200 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:15:30.201 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:15:30.202 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:30.202 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:30.203 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:30.203 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:30.205 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:30.205 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:30.210 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:15:30.210 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:15:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:15:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:15:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:15:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:30.212 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:30.212 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:30.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:30.212 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:15:30.214 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:30.214 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:30.214 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:30.214 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:30.216 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:30.216 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:30.216 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:30.218 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:30.218 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:30.218 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:30.218 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:30.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:30.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:30.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:30.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:30.220 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:30.220 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:30.220 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:30.220 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:30.222 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:30.222 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:30.222 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:30.224 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:30.224 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:15:30.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:15:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:15:30.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:15:30.228 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:15:30.228 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:30.229 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:30.229 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:30.229 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:30.229 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:30.230 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:30.230 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:30.230 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:30.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:30.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:30.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:15:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:30.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:30.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:30.235 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:30.236 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:30.242 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:30.242 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:30.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:30.242 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.243 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:30.243 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:30.244 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:30.245 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:30.245 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:30.245 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:30.245 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:30.246 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:30.246 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:30.246 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:15:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:15:30.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.249 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:30.249 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:15:30.249 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.251 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.251 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:30.252 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.252 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.252 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.252 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.253 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.253 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:15:30.253 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.254 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.255 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:15:30.255 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.255 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.255 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.256 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.256 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.256 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.257 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.257 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:15:30.257 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.258 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:15:30.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.260 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.260 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.261 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.261 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.264 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:15:30.265 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.265 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.265 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:15:30.266 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.266 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.267 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.268 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.268 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.268 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.269 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.269 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:15:30.269 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.270 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.270 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.271 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.271 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:15:30.272 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:30.272 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.273 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.273 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.274 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:15:30.274 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:15:30.275 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:15:30.275 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:15:30.275 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.276 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.276 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.277 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.278 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:30.278 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:15:30.279 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.279 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.280 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.280 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.281 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.282 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:30.282 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.283 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.283 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.284 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:15:30.285 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.293 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.294 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.294 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:15:30.295 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.296 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.296 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.297 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.298 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.299 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:15:30.300 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.300 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.301 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:15:30.302 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.303 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.303 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.303 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.304 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.304 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.305 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:15:30.306 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.307 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.308 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:15:30.309 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.309 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.310 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.310 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.311 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.312 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:15:30.313 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.313 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.314 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.314 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.315 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:15:30.315 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:30.315 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.316 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.316 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:15:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:15:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:15:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:15:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.318 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.318 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.318 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.319 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.319 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:30.319 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:15:30.320 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.320 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.321 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.321 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.321 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.322 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.322 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.323 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:30.323 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.323 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.325 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:15:30.325 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.326 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.326 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.326 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:15:30.326 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.327 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.327 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.327 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.328 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.328 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:15:30.329 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.330 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.330 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.330 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:15:30.331 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.331 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.331 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.332 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.332 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.332 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.333 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:15:30.333 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.333 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.334 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.334 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:15:30.334 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.335 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.335 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.335 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.335 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.336 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.336 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.337 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:15:30.338 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.338 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.339 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.339 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:15:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:15:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:15:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:15:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:15:30.342 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.342 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.342 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:15:30.344 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.344 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.344 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.344 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.345 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.345 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.345 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.345 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 89, 2022-07-20T03:15:30.346 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.346 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.346 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.346 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.347 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.347 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.347 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:15:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.349 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:15:30.349 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.349 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.349 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.350 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.350 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.350 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.351 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.351 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:15:30.351 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.351 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.352 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.352 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.352 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:15:30.352 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.354 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.354 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:15:30.354 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.355 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.355 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.355 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:15:30.356 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.356 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.356 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.356 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:15:30.358 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.358 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:15:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.360 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:15:30.360 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:15:30.360 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:15:30.361 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:15:30.361 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.361 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:15:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.364 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.364 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.364 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.364 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-20T03:15:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:15:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:15:30.368 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.368 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.368 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:15:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:15:30.372 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.372 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.372 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:15:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:15:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:15:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:15:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:15:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:15:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:15:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:15:30.381 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.381 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.381 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.381 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.382 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.382 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.382 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:30.382 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:15:30.383 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.383 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.383 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.383 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.384 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.384 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.385 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.385 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 79, 2022-07-20T03:15:30.386 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.386 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.386 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:15:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.388 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.388 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.388 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:15:30.389 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.389 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.389 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.390 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.390 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.390 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:15:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.392 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.392 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:15:30.392 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.392 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.393 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.393 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.394 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.394 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:15:30.394 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:15:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:15:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:15:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:15:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:15:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:15:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:15:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:15:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 87, 2022-07-20T03:15:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:15:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:15:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:15:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:15:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.412 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:15:30.414 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.414 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.415 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.415 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.415 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:15:30.415 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.416 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.416 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.417 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.417 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.418 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.418 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:15:30.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.419 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.419 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:15:30.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:30.420 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.420 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.421 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:15:30.421 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:15:30.421 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:15:30.422 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:15:30.422 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.422 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.422 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.423 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:15:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:30.424 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:30.425 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:30.425 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:30.425 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:30.425 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:30.426 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 88, 2022-07-20T03:15:30.426 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:30.426 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:30.426 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:30.426 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.427 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.427 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:15:30.428 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.428 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.428 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:15:30.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.429 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.430 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.430 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:30.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.431 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.431 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:15:30.431 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.432 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.432 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:15:30.432 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.433 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.433 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.433 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:30.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.434 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.434 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:15:30.435 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.435 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.435 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:15:30.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.436 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.436 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.437 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:30.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:30.437 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.437 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:30.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:15:30.438 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.438 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.438 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.439 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:30.439 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:15:30.439 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:30.439 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.440 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.440 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.440 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:15:30.440 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:15:30.441 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:15:30.441 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:15:30.442 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:30.442 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:30.446 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:30.446 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:30.446 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.447 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:30.447 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:15:30.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.448 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:30.448 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.449 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.449 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.450 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.450 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.451 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:15:30.451 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.453 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:30.453 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.454 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.454 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.455 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.455 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.455 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:15:30.456 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.457 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.457 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:30.458 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.458 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.458 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.459 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.459 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.459 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:15:30.459 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.460 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.460 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.461 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:30.461 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.462 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.463 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.463 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.464 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.464 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:15:30.465 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.465 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.465 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.465 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:30.466 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.466 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.467 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.467 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.467 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.468 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:15:30.468 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.469 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.469 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.469 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:30.469 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.470 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.470 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.470 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.470 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.471 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:15:30.471 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.471 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.471 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.472 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:30.472 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.472 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.473 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.473 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.474 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.475 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:15:30.475 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.475 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.476 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:30.476 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:30.476 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:30.476 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:30.477 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:30.477 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:30.477 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:30.477 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:15:30.478 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:30.478 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.478 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:30.479 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:15:30.479 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:15:30.479 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:15:30.479 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:15:30.480 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:15:30.480 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/1014823584": "2022-07-21T03:15:23.756559+0000", 2022-07-20T03:15:30.480 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/1014823584": "2022-07-21T03:15:23.756559+0000" 2022-07-20T03:15:30.480 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.481 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:15:30.481 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:15:30.481 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:15:30.481 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:15:30.481 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:15:30.482 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:15:30.482 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:15:30.483 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:15:30.483 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:30.484 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:30.484 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:15:30.485 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:15:30.486 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:15:30.491 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:15:30.491 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:15:30.492 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:30.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6c00539d0 msgr2=0x7fe6c0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6c00539d0 0x7fe6c0055e90 secure :-1 s=READY pgs=486 cs=0 l=1 rev1=1 rx=0x7fe6d801c000 tx=0x7fe6d8015000).stop 2022-07-20T03:15:30.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 msgr2=0x7fe6e8061390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 0x7fe6e8061390 secure :-1 s=READY pgs=1002 cs=0 l=1 rev1=1 rx=0x7fe6d000c000 tx=0x7fe6d0006d90).stop 2022-07-20T03:15:30.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 shutdown_connections 2022-07-20T03:15:30.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6c00539d0 0x7fe6c0055e90 unknown :-1 s=CLOSED pgs=486 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6e8142580 0x7fe6e8155620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6e80b1ef0 0x7fe6e8099b20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 --2- 172.21.15.61:0/2887141038 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6e805cb10 0x7fe6e8061390 unknown :-1 s=CLOSED pgs=1002 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 >> 172.21.15.61:0/2887141038 conn(0x7fe6e80b2ca0 msgr2=0x7fe6e8146420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:30.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 shutdown_connections 2022-07-20T03:15:30.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.184+0000 7fe6eccf2700 1 -- 172.21.15.61:0/2887141038 wait complete. 2022-07-20T03:15:30.501 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f05508f7700 1 Processor -- start 2022-07-20T03:15:30.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f05508f7700 1 -- start start 2022-07-20T03:15:30.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f05508f7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f054c061ff0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f05508f7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c05f970 con 0x7f054c0b21f0 2022-07-20T03:15:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f05508f7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f054c05fc10 con 0x7f054c0ab8b0 2022-07-20T03:15:30.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f05508f7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c060b90 con 0x7f054c0b25d0 2022-07-20T03:15:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f054a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f054c0b21f0 0x7f054c146790 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45178/0 (socket says 172.21.15.61:45178) 2022-07-20T03:15:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f054a59c700 1 -- 172.21.15.61:0/521510522 learned_addr learned my addr 172.21.15.61:0/521510522 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2062813475 0 0) 0x7f054c05f970 con 0x7f054c0b21f0 2022-07-20T03:15:30.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f054ad9d700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f054c061ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f052c003260 con 0x7f054c0b21f0 2022-07-20T03:15:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 95057108 0 0) 0x7f052c003260 con 0x7f054c0b21f0 2022-07-20T03:15:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 msgr2=0x7f054c061ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:30.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f054c061ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 >> v1:172.21.15.117:6789/0 conn(0x7f054c0ab8b0 legacy=0x7f054c0618e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c05feb0 con 0x7f054c0b21f0 2022-07-20T03:15:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f053c001a00 con 0x7f054c0b21f0 2022-07-20T03:15:30.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 >> v1:172.21.15.61:6789/0 conn(0x7f054c0b21f0 legacy=0x7f054c146790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f052c003780 0x7f052c003b60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f052c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f052c0049f0 0x7f052c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c060b90 con 0x7f052c003780 2022-07-20T03:15:30.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f052c00b750 con 0x7f052c0049f0 2022-07-20T03:15:30.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f052c00b8c0 con 0x7f054c0b25d0 2022-07-20T03:15:30.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f054a59c700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f052c003780 0x7f052c003b60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0541d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f052c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f054ad9d700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f052c0049f0 0x7f052c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0541d9b700 1 -- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f052c0049f0 msgr2=0x7f052c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0541d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f052c0049f0 0x7f052c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0541d9b700 1 -- 172.21.15.61:0/521510522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f052c003780 msgr2=0x7f052c003b60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0541d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f052c003780 0x7f052c003b60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f0541d9b700 1 -- 172.21.15.61:0/521510522 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f052c003260 con 0x7f054c0b25d0 2022-07-20T03:15:30.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.612+0000 7f054a59c700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f052c003780 0x7f052c003b60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:30.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f0541d9b700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f052c004500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0534001110 tx=0x7f0534007a00).ready entity=mon.2 client_cookie=e7b630fbb5be9b03 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0534007d10 con 0x7f054c0b25d0 2022-07-20T03:15:30.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0534007030 con 0x7f054c0b25d0 2022-07-20T03:15:30.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f0549d9b700 1 -- 172.21.15.61:0/521510522 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0534007390 con 0x7f054c0b25d0 2022-07-20T03:15:30.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f05508f7700 1 -- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 msgr2=0x7f052c004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f05508f7700 1 --2- 172.21.15.61:0/521510522 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c0b25d0 0x7f052c004500 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f0534001110 tx=0x7f0534007a00).stop 2022-07-20T03:15:30.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f054ad9d700 1 -- 172.21.15.61:0/521510522 reap_dead start 2022-07-20T03:15:30.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f05508f7700 1 -- 172.21.15.61:0/521510522 shutdown_connections 2022-07-20T03:15:30.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f05508f7700 1 -- 172.21.15.61:0/521510522 >> 172.21.15.61:0/521510522 conn(0x7f054c0b3690 msgr2=0x7f054c0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:30.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f05508f7700 1 -- 172.21.15.61:0/521510522 shutdown_connections 2022-07-20T03:15:30.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.616+0000 7f05508f7700 1 -- 172.21.15.61:0/521510522 wait complete. 2022-07-20T03:15:30.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 Processor -- start 2022-07-20T03:15:30.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 -- start start 2022-07-20T03:15:30.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 0x7f054c095d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c096250 0x7f054c156450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c156940 0x7f054c156dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0af690 con 0x7f054c0b25d0 2022-07-20T03:15:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f054c1573a0 con 0x7f054c156940 2022-07-20T03:15:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f054c0ad610 con 0x7f054c096250 2022-07-20T03:15:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 0x7f054c095d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 0x7f054c095d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48592/0 (socket says 172.21.15.61:48592) 2022-07-20T03:15:30.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 -- 172.21.15.61:0/3682231616 learned_addr learned my addr 172.21.15.61:0/3682231616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:30.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f0541d9b700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c096250 0x7f054c156450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054ad9d700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c156940 0x7f054c156dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 -- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c096250 msgr2=0x7f054c156450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c096250 0x7f054c156450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 -- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c156940 msgr2=0x7f054c156dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c156940 0x7f054c156dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 -- 172.21.15.61:0/3682231616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f054c09bf60 con 0x7f054c0b25d0 2022-07-20T03:15:30.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f054a59c700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 0x7f054c095d60 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f053c00d000 tx=0x7f054c0b3c90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f053c00f880 con 0x7f054c0b25d0 2022-07-20T03:15:30.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f053c00f9e0 con 0x7f054c0b25d0 2022-07-20T03:15:30.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f054c0966b0 con 0x7f054c0b25d0 2022-07-20T03:15:30.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f053c00fd40 con 0x7f054c0b25d0 2022-07-20T03:15:30.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f054c159bf0 con 0x7f054c0b25d0 2022-07-20T03:15:30.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.620+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f053c01b030 con 0x7f054c0b25d0 2022-07-20T03:15:30.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.624+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f054c060150 con 0x7f054c0b25d0 2022-07-20T03:15:30.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.624+0000 7f0543fff700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05300539d0 0x7f0530055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.624+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f053c06b9f0 con 0x7f054c0b25d0 2022-07-20T03:15:30.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.624+0000 7f0541d9b700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05300539d0 0x7f0530055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.624+0000 7f0541d9b700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05300539d0 0x7f0530055e90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f053401b000 tx=0x7f0534015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.628+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f053c06de60 con 0x7f054c0b25d0 2022-07-20T03:15:30.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.792+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f054c0574e0 con 0x7f054c0b25d0 2022-07-20T03:15:30.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.792+0000 7f0543fff700 1 -- 172.21.15.61:0/3682231616 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+1206 (secure 0 0 0) 0x7f053c019030 con 0x7f054c0b25d0 2022-07-20T03:15:30.799 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:30.800 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49},{"gid":7263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/23998111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":23998111},{"type":"v1","addr":"172.21.15.61:6833","nonce":23998111}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[]} 2022-07-20T03:15:30.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.792+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05300539d0 msgr2=0x7f0530055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.792+0000 7f05508f7700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05300539d0 0x7f0530055e90 secure :-1 s=READY pgs=487 cs=0 l=1 rev1=1 rx=0x7f053401b000 tx=0x7f0534015000).stop 2022-07-20T03:15:30.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 msgr2=0x7f054c095d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 0x7f054c095d60 secure :-1 s=READY pgs=1004 cs=0 l=1 rev1=1 rx=0x7f053c00d000 tx=0x7f054c0b3c90).stop 2022-07-20T03:15:30.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 shutdown_connections 2022-07-20T03:15:30.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05300539d0 0x7f0530055e90 unknown :-1 s=CLOSED pgs=487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f054c156940 0x7f054c156dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f054c096250 0x7f054c156450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 --2- 172.21.15.61:0/3682231616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f054c0b25d0 0x7f054c095d60 unknown :-1 s=CLOSED pgs=1004 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 >> 172.21.15.61:0/3682231616 conn(0x7f054c0b3690 msgr2=0x7f054c149630 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:30.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 shutdown_connections 2022-07-20T03:15:30.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.796+0000 7f05508f7700 1 -- 172.21.15.61:0/3682231616 wait complete. 2022-07-20T03:15:30.809 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 53 2022-07-20T03:15:30.814 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:30.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 Processor -- start 2022-07-20T03:15:30.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- start start 2022-07-20T03:15:30.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd7c80b0570 0x7fd7c80b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7c80aea10 0x7fd7c8141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd7c81433d0 con 0x7fd7c81422c0 2022-07-20T03:15:30.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c8143b10 con 0x7fd7c80aea10 2022-07-20T03:15:30.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd7c8143670 con 0x7fd7c80b0570 2022-07-20T03:15:30.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7c80aea10 0x7fd7c8141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd7c80aea10 0x7fd7c8141dd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48602/0 (socket says 172.21.15.61:48602) 2022-07-20T03:15:30.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 -- 172.21.15.61:0/1186115943 learned_addr learned my addr 172.21.15.61:0/1186115943 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:30.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cdc8d700 1 --2- 172.21.15.61:0/1186115943 >> v2:172.21.15.117:3300/0 conn(0x7fd7c80b0570 0x7fd7c80b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 -- 172.21.15.61:0/1186115943 >> v2:172.21.15.117:3300/0 conn(0x7fd7c80b0570 msgr2=0x7fd7c80b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:30.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 --2- 172.21.15.61:0/1186115943 >> v2:172.21.15.117:3300/0 conn(0x7fd7c80b0570 0x7fd7c80b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 -- 172.21.15.61:0/1186115943 >> v1:172.21.15.117:6789/0 conn(0x7fd7c81422c0 legacy=0x7fd7c80ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 -- 172.21.15.61:0/1186115943 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7c8142710 con 0x7fd7c80aea10 2022-07-20T03:15:30.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cdc8d700 1 --2- 172.21.15.61:0/1186115943 >> v2:172.21.15.117:3300/0 conn(0x7fd7c80b0570 0x7fd7c80b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:30.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cd48c700 1 --2- 172.21.15.61:0/1186115943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c80aea10 0x7fd7c8141dd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7fd7b8002f70 tx=0x7fd7b8006eb0).ready entity=mon.0 client_cookie=56fe418d410bfcf server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7ccc8b700 1 -- 172.21.15.61:0/1186115943 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7b800d610 con 0x7fd7c80aea10 2022-07-20T03:15:30.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7ccc8b700 1 -- 172.21.15.61:0/1186115943 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7b8000e90 con 0x7fd7c80aea10 2022-07-20T03:15:30.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7ccc8b700 1 -- 172.21.15.61:0/1186115943 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7b800d9a0 con 0x7fd7c80aea10 2022-07-20T03:15:30.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- 172.21.15.61:0/1186115943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c80aea10 msgr2=0x7fd7c8141dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/1186115943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c80aea10 0x7fd7c8141dd0 secure :-1 s=READY pgs=1005 cs=0 l=1 rev1=1 rx=0x7fd7b8002f70 tx=0x7fd7b8006eb0).stop 2022-07-20T03:15:30.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- 172.21.15.61:0/1186115943 shutdown_connections 2022-07-20T03:15:30.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/1186115943 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c80aea10 0x7fd7c8141dd0 unknown :-1 s=CLOSED pgs=1005 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/1186115943 >> v2:172.21.15.117:3300/0 conn(0x7fd7c80b0570 0x7fd7c80b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- 172.21.15.61:0/1186115943 >> 172.21.15.61:0/1186115943 conn(0x7fd7c80b9360 msgr2=0x7fd7c80b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:30.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- 172.21.15.61:0/1186115943 shutdown_connections 2022-07-20T03:15:30.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.916+0000 7fd7cec8f700 1 -- 172.21.15.61:0/1186115943 wait complete. 2022-07-20T03:15:30.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 Processor -- start 2022-07-20T03:15:30.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 -- start start 2022-07-20T03:15:30.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7c80aea10 0x7fd7c8155180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7c80b0570 0x7fd7c81495c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 0x7fd7c8152130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8151d00 con 0x7fd7c81422c0 2022-07-20T03:15:30.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c81466a0 con 0x7fd7c80aea10 2022-07-20T03:15:30.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd7c8146940 con 0x7fd7c80b0570 2022-07-20T03:15:30.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 0x7fd7c8152130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 0x7fd7c8152130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48610/0 (socket says 172.21.15.61:48610) 2022-07-20T03:15:30.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cd48c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7c80b0570 0x7fd7c81495c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 -- 172.21.15.61:0/962741580 learned_addr learned my addr 172.21.15.61:0/962741580 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:30.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cdc8d700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7c80aea10 0x7fd7c8155180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 -- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7c80b0570 msgr2=0x7fd7c81495c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7c80b0570 0x7fd7c81495c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 -- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7c80aea10 msgr2=0x7fd7c8155180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:30.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7c80aea10 0x7fd7c8155180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:30.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 -- 172.21.15.61:0/962741580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7b800b1e0 con 0x7fd7c81422c0 2022-07-20T03:15:30.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cdc8d700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7c80aea10 0x7fd7c8155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:30.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7ce48e700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 0x7fd7c8152130 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7fd7c8066710 tx=0x7fd7c00013a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c0006aa0 con 0x7fd7c81422c0 2022-07-20T03:15:30.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd7c0006c00 con 0x7fd7c81422c0 2022-07-20T03:15:30.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd7c0007030 con 0x7fd7c81422c0 2022-07-20T03:15:30.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7c80b1720 con 0x7fd7c81422c0 2022-07-20T03:15:30.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.920+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7c80b4e80 con 0x7fd7c81422c0 2022-07-20T03:15:30.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.924+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7c8151e70 con 0x7fd7c81422c0 2022-07-20T03:15:30.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.924+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd7c0010030 con 0x7fd7c81422c0 2022-07-20T03:15:30.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.924+0000 7fd7b6ffd700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7a8053ad0 0x7fd7a8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:30.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.924+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(94..94 src has 1..94) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd7c006ba20 con 0x7fd7c81422c0 2022-07-20T03:15:30.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.924+0000 7fd7cdc8d700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7a8053ad0 0x7fd7a8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:30.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.924+0000 7fd7cdc8d700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7a8053ad0 0x7fd7a8055f90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd7c400c000 tx=0x7fd7c4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:30.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:30.932+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd7c8151e70 con 0x7fd7c81422c0 2022-07-20T03:15:31.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.100+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd7c8151e40 con 0x7fd7c81422c0 2022-07-20T03:15:31.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7b6ffd700 1 -- 172.21.15.61:0/962741580 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 53 v53) v1 ==== 94+0+1206 (secure 0 0 0) 0x7fd7c0042bc0 con 0x7fd7c81422c0 2022-07-20T03:15:31.107 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:31.108 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":53,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6949,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.117:6835/2042601675","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2042601675},{"type":"v1","addr":"172.21.15.117:6835","nonce":2042601675}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":49},{"gid":7263,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/23998111","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":23998111},{"type":"v1","addr":"172.21.15.61:6833","nonce":23998111}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":53}],"filesystems":[]} 2022-07-20T03:15:31.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7a8053ad0 msgr2=0x7fd7a8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7a8053ad0 0x7fd7a8055f90 secure :-1 s=READY pgs=488 cs=0 l=1 rev1=1 rx=0x7fd7c400c000 tx=0x7fd7c4008040).stop 2022-07-20T03:15:31.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 msgr2=0x7fd7c8152130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 0x7fd7c8152130 secure :-1 s=READY pgs=1006 cs=0 l=1 rev1=1 rx=0x7fd7c8066710 tx=0x7fd7c00013a0).stop 2022-07-20T03:15:31.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 shutdown_connections 2022-07-20T03:15:31.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd7a8053ad0 0x7fd7a8055f90 unknown :-1 s=CLOSED pgs=488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd7c81422c0 0x7fd7c8152130 unknown :-1 s=CLOSED pgs=1006 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd7c80b0570 0x7fd7c81495c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 --2- 172.21.15.61:0/962741580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd7c80aea10 0x7fd7c8155180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 >> 172.21.15.61:0/962741580 conn(0x7fd7c80b9360 msgr2=0x7fd7c8141ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:31.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 shutdown_connections 2022-07-20T03:15:31.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.104+0000 7fd7cec8f700 1 -- 172.21.15.61:0/962741580 wait complete. 2022-07-20T03:15:31.117 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 53 2022-07-20T03:15:31.121 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:15:31.121 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:15:31.122 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:15:31.123 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:15:31.123 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:15:31.124 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:15:31.124 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:15:31.125 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:15:31.140 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:15:31.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:15:31.142 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:15:31.143 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:15:31.145 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:15:31.145 DEBUG:teuthology.parallel:result is None 2022-07-20T03:15:31.199 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:15:31.199 DEBUG:teuthology.parallel:result is None 2022-07-20T03:15:31.201 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:15:31.216 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:15:31.277 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:15:31.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.292+0000 7fdb67a13700 1 Processor -- start 2022-07-20T03:15:31.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.292+0000 7fdb67a13700 1 -- start start 2022-07-20T03:15:31.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdb601427f0 0x7fdb600ae440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:31.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb600b65d0 con 0x7fdb600b61f0 2022-07-20T03:15:31.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdb60146250 con 0x7fdb600b4030 2022-07-20T03:15:31.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdb600ad6f0 con 0x7fdb601427f0 2022-07-20T03:15:31.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb657af700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdb600b61f0 0x7fdb600b4c80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45230/0 (socket says 172.21.15.61:45230) 2022-07-20T03:15:31.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb657af700 1 -- 172.21.15.61:0/3467985542 learned_addr learned my addr 172.21.15.61:0/3467985542 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:31.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb65fb0700 1 --2- 172.21.15.61:0/3467985542 >> v2:172.21.15.61:3300/0 conn(0x7fdb601427f0 0x7fdb600ae440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:31.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb5ffff700 1 -- 172.21.15.61:0/3467985542 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2542504441 0 0) 0x7fdb600b65d0 con 0x7fdb600b61f0 2022-07-20T03:15:31.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb5ffff700 1 -- 172.21.15.61:0/3467985542 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdb4c003260 con 0x7fdb600b61f0 2022-07-20T03:15:31.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb65fb0700 1 -- 172.21.15.61:0/3467985542 >> v1:172.21.15.117:6789/0 conn(0x7fdb600b4030 legacy=0x7fdb601420e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb65fb0700 1 -- 172.21.15.61:0/3467985542 >> v1:172.21.15.61:6789/0 conn(0x7fdb600b61f0 legacy=0x7fdb600b4c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb65fb0700 1 -- 172.21.15.61:0/3467985542 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb601464f0 con 0x7fdb601427f0 2022-07-20T03:15:31.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb65fb0700 1 --2- 172.21.15.61:0/3467985542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb600ae440 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7fdb600ae310 tx=0x7fdb50006df0).ready entity=mon.0 client_cookie=680cda6e277f20c5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:31.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb5ffff700 1 -- 172.21.15.61:0/3467985542 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5000d610 con 0x7fdb601427f0 2022-07-20T03:15:31.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb5ffff700 1 -- 172.21.15.61:0/3467985542 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb50000e90 con 0x7fdb601427f0 2022-07-20T03:15:31.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb5ffff700 1 -- 172.21.15.61:0/3467985542 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5000d960 con 0x7fdb601427f0 2022-07-20T03:15:31.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 -- 172.21.15.61:0/3467985542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 msgr2=0x7fdb600ae440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 --2- 172.21.15.61:0/3467985542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb600ae440 secure :-1 s=READY pgs=1012 cs=0 l=1 rev1=1 rx=0x7fdb600ae310 tx=0x7fdb50006df0).stop 2022-07-20T03:15:31.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 -- 172.21.15.61:0/3467985542 shutdown_connections 2022-07-20T03:15:31.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 --2- 172.21.15.61:0/3467985542 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb600ae440 unknown :-1 s=CLOSED pgs=1012 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.296+0000 7fdb67a13700 1 -- 172.21.15.61:0/3467985542 >> 172.21.15.61:0/3467985542 conn(0x7fdb60066e90 msgr2=0x7fdb60067270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:31.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.300+0000 7fdb67a13700 1 -- 172.21.15.61:0/3467985542 shutdown_connections 2022-07-20T03:15:31.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.300+0000 7fdb67a13700 1 -- 172.21.15.61:0/3467985542 wait complete. 2022-07-20T03:15:31.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.300+0000 7fdb67a13700 1 Processor -- start 2022-07-20T03:15:31.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.300+0000 7fdb67a13700 1 -- start start 2022-07-20T03:15:31.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb600b4030 0x7fdb6014a400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:31.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb600b61f0 0x7fdb6013f1a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:31.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb6013f690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:31.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6012ff20 con 0x7fdb601427f0 2022-07-20T03:15:31.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb60112670 con 0x7fdb600b61f0 2022-07-20T03:15:31.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdb6013fb80 con 0x7fdb600b4030 2022-07-20T03:15:31.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb6013f690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:31.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb6013f690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48642/0 (socket says 172.21.15.61:48642) 2022-07-20T03:15:31.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 -- 172.21.15.61:0/2584448261 learned_addr learned my addr 172.21.15.61:0/2584448261 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:31.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb64fae700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb600b61f0 0x7fdb6013f1a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb657af700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb600b4030 0x7fdb6014a400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:31.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 -- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb600b4030 msgr2=0x7fdb6014a400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb600b4030 0x7fdb6014a400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 -- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb600b61f0 msgr2=0x7fdb6013f1a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb600b61f0 0x7fdb6013f1a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:31.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 -- 172.21.15.61:0/2584448261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb5000b180 con 0x7fdb601427f0 2022-07-20T03:15:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb65fb0700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb6013f690 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fdb600ae310 tx=0x7fdb50014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5000d290 con 0x7fdb601427f0 2022-07-20T03:15:31.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdb50002f70 con 0x7fdb601427f0 2022-07-20T03:15:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb601527d0 con 0x7fdb601427f0 2022-07-20T03:15:31.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdb5001e5f0 con 0x7fdb601427f0 2022-07-20T03:15:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.304+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb60153390 con 0x7fdb601427f0 2022-07-20T03:15:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.308+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb60152610 con 0x7fdb601427f0 2022-07-20T03:15:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.308+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdb50025030 con 0x7fdb601427f0 2022-07-20T03:15:31.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.308+0000 7fdb5e7fc700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdb40053920 0x7fdb40055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.308+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(96..96 src has 1..96) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fdb5006c8c0 con 0x7fdb601427f0 2022-07-20T03:15:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.308+0000 7fdb657af700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdb40053920 0x7fdb40055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:31.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.308+0000 7fdb657af700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdb40053920 0x7fdb40055de0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fdb5800b000 tx=0x7fdb58009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:31.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.312+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdb50043b00 con 0x7fdb601427f0 2022-07-20T03:15:31.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:31.448+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7fdb60146790 con 0x7fdb601427f0 2022-07-20T03:15:32.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.332+0000 7fdb5e7fc700 1 -- 172.21.15.61:0/2584448261 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v97) v1 ==== 97+0+0 (secure 0 0 0) 0x7fdb50023070 con 0x7fdb601427f0 2022-07-20T03:15:32.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdb40053920 msgr2=0x7fdb40055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdb40053920 0x7fdb40055de0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7fdb5800b000 tx=0x7fdb58009040).stop 2022-07-20T03:15:32.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 msgr2=0x7fdb6013f690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb6013f690 secure :-1 s=READY pgs=1013 cs=0 l=1 rev1=1 rx=0x7fdb600ae310 tx=0x7fdb50014000).stop 2022-07-20T03:15:32.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 shutdown_connections 2022-07-20T03:15:32.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdb40053920 0x7fdb40055de0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdb600b61f0 0x7fdb6013f1a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdb601427f0 0x7fdb6013f690 unknown :-1 s=CLOSED pgs=1013 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 --2- 172.21.15.61:0/2584448261 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdb600b4030 0x7fdb6014a400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 >> 172.21.15.61:0/2584448261 conn(0x7fdb60066e90 msgr2=0x7fdb60142c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:32.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 shutdown_connections 2022-07-20T03:15:32.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.336+0000 7fdb67a13700 1 -- 172.21.15.61:0/2584448261 wait complete. 2022-07-20T03:15:32.353 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:15:32.355 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:15:32.356 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:15:32.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 Processor -- start 2022-07-20T03:15:32.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 -- start start 2022-07-20T03:15:32.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5ccc13ffa0 0x7f5ccc143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ccc0ab430 con 0x7f5ccc0ac860 2022-07-20T03:15:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ccc0ab6d0 con 0x7f5ccc13ffa0 2022-07-20T03:15:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cd1846700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0ab970 con 0x7f5ccc0b21f0 2022-07-20T03:15:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cca7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5ccc0ac860 0x7f5ccc142f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45246/0 (socket says 172.21.15.61:45246) 2022-07-20T03:15:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cca7fc700 1 -- 172.21.15.61:0/3880310932 learned_addr learned my addr 172.21.15.61:0/3880310932 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:32.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccaffd700 1 --2- 172.21.15.61:0/3880310932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccb7fe700 1 --2- 172.21.15.61:0/3880310932 >> v2:172.21.15.117:3300/0 conn(0x7f5ccc13ffa0 0x7f5ccc143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cc9ffb700 1 -- 172.21.15.61:0/3880310932 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1547436708 0 0) 0x7f5ccc0ab430 con 0x7f5ccc0ac860 2022-07-20T03:15:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cc9ffb700 1 -- 172.21.15.61:0/3880310932 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5cac003260 con 0x7f5ccc0ac860 2022-07-20T03:15:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3880310932 >> v2:172.21.15.117:3300/0 conn(0x7f5ccc13ffa0 msgr2=0x7f5ccc143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccaffd700 1 --2- 172.21.15.61:0/3880310932 >> v2:172.21.15.117:3300/0 conn(0x7f5ccc13ffa0 0x7f5ccc143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3880310932 >> v1:172.21.15.61:6789/0 conn(0x7f5ccc0ac860 legacy=0x7f5ccc142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3880310932 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ccc0ab0b0 con 0x7f5ccc0b21f0 2022-07-20T03:15:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccb7fe700 1 --2- 172.21.15.61:0/3880310932 >> v2:172.21.15.117:3300/0 conn(0x7f5ccc13ffa0 0x7f5ccc143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5ccaffd700 1 --2- 172.21.15.61:0/3880310932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b25d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5cb4002f70 tx=0x7f5cb4006df0).ready entity=mon.2 client_cookie=22f5bdb55b1d1c7a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cc9ffb700 1 -- 172.21.15.61:0/3880310932 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cb400d4a0 con 0x7f5ccc0b21f0 2022-07-20T03:15:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cc9ffb700 1 -- 172.21.15.61:0/3880310932 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb4000e90 con 0x7f5ccc0b21f0 2022-07-20T03:15:32.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.512+0000 7f5cc9ffb700 1 -- 172.21.15.61:0/3880310932 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cb400d7f0 con 0x7f5ccc0b21f0 2022-07-20T03:15:32.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 -- 172.21.15.61:0/3880310932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 msgr2=0x7f5ccc0b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3880310932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b25d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7f5cb4002f70 tx=0x7f5cb4006df0).stop 2022-07-20T03:15:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 -- 172.21.15.61:0/3880310932 shutdown_connections 2022-07-20T03:15:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3880310932 >> v2:172.21.15.117:3300/0 conn(0x7f5ccc13ffa0 0x7f5ccc143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3880310932 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b25d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 -- 172.21.15.61:0/3880310932 >> 172.21.15.61:0/3880310932 conn(0x7f5ccc0b3690 msgr2=0x7f5ccc0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 -- 172.21.15.61:0/3880310932 shutdown_connections 2022-07-20T03:15:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 -- 172.21.15.61:0/3880310932 wait complete. 2022-07-20T03:15:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 Processor -- start 2022-07-20T03:15:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.516+0000 7f5cd1846700 1 -- start start 2022-07-20T03:15:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 0x7f5ccc0602e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b3f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc13ffa0 0x7f5ccc09a460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc09b620 con 0x7f5ccc0ac860 2022-07-20T03:15:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0ae620 con 0x7f5ccc13ffa0 2022-07-20T03:15:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ccc0992e0 con 0x7f5ccc0b21f0 2022-07-20T03:15:32.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 0x7f5ccc0602e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 0x7f5ccc0602e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48654/0 (socket says 172.21.15.61:48654) 2022-07-20T03:15:32.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3702162396 learned_addr learned my addr 172.21.15.61:0/3702162396 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:32.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cca7fc700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b3f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccb7fe700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc13ffa0 0x7f5ccc09a460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 msgr2=0x7f5ccc0b3f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b3f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc13ffa0 msgr2=0x7f5ccc09a460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc13ffa0 0x7f5ccc09a460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 -- 172.21.15.61:0/3702162396 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5cbc00b180 con 0x7f5ccc0ac860 2022-07-20T03:15:32.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5ccaffd700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 0x7f5ccc0602e0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f5cb4002f70 tx=0x7f5cb4014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:32.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cb4016070 con 0x7f5ccc0ac860 2022-07-20T03:15:32.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5cb40137d0 con 0x7f5ccc0ac860 2022-07-20T03:15:32.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5cb400b180 con 0x7f5ccc0ac860 2022-07-20T03:15:32.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5cb4013b30 con 0x7f5ccc0ac860 2022-07-20T03:15:32.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.520+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ccc0b5a50 con 0x7f5ccc0ac860 2022-07-20T03:15:32.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.524+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ccc0574e0 con 0x7f5ccc0ac860 2022-07-20T03:15:32.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.524+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5cb4025030 con 0x7f5ccc0ac860 2022-07-20T03:15:32.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.524+0000 7f5cc3fff700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5cb0053a40 0x7f5cb0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.524+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5cb406c9f0 con 0x7f5ccc0ac860 2022-07-20T03:15:32.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.524+0000 7f5cca7fc700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5cb0053a40 0x7f5cb0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.524+0000 7f5cca7fc700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5cb0053a40 0x7f5cb0055f00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f5cc400d000 tx=0x7f5cc4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:32.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.528+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5cb406ee60 con 0x7f5ccc0ac860 2022-07-20T03:15:32.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.692+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f5ccc09ad50 con 0x7f5ccc0ac860 2022-07-20T03:15:32.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.692+0000 7f5cc3fff700 1 -- 172.21.15.61:0/3702162396 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v34) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f5cb4023070 con 0x7f5ccc0ac860 2022-07-20T03:15:32.699 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:32.699 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:32.699 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:15:32.700 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.700 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:15:32.701 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:15:32.701 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.701 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:32.701 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.702 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.702 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.702 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.702 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:15:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:15:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.705 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.705 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.705 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:15:32.705 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:15:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.707 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.707 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.707 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:15:32.707 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:15:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:15:32.710 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:15:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.715 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:15:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:15:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:15:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:15:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:15:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:15:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.725 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.725 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:15:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:15:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:15:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:15:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:15:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:15:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:15:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:15:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.746 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.746 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:15:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:15:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.750 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:15:32.750 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:15:32.751 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.751 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:32.751 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:32.752 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:32.752 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.754 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:15:32.754 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:15:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:32.756 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:15:32.756 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:32.758 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:15:32.758 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:15:32.759 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:32.759 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:15:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:32.762 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:32.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5cb0053a40 msgr2=0x7f5cb0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5cb0053a40 0x7f5cb0055f00 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f5cc400d000 tx=0x7f5cc4009040).stop 2022-07-20T03:15:32.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 msgr2=0x7f5ccc0602e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 0x7f5ccc0602e0 secure :-1 s=READY pgs=1015 cs=0 l=1 rev1=1 rx=0x7f5cb4002f70 tx=0x7f5cb4014000).stop 2022-07-20T03:15:32.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 shutdown_connections 2022-07-20T03:15:32.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5cb0053a40 0x7f5cb0055f00 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ccc13ffa0 0x7f5ccc09a460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ccc0b21f0 0x7f5ccc0b3f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 --2- 172.21.15.61:0/3702162396 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ccc0ac860 0x7f5ccc0602e0 unknown :-1 s=CLOSED pgs=1015 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 >> 172.21.15.61:0/3702162396 conn(0x7f5ccc0b3690 msgr2=0x7f5ccc145100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:32.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 shutdown_connections 2022-07-20T03:15:32.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.696+0000 7f5cd1846700 1 -- 172.21.15.61:0/3702162396 wait complete. 2022-07-20T03:15:32.771 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:32.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 Processor -- start 2022-07-20T03:15:32.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- start start 2022-07-20T03:15:32.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7f2405d160 0x7f7f240b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 0x7f7f24142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7f240ba2d0 con 0x7f7f240b9ef0 2022-07-20T03:15:32.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f240b4560 con 0x7f7f240b2960 2022-07-20T03:15:32.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7f240b4800 con 0x7f7f2405d160 2022-07-20T03:15:32.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f23fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 0x7f7f24142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f23fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 0x7f7f24142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48666/0 (socket says 172.21.15.61:48666) 2022-07-20T03:15:32.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f23fff700 1 -- 172.21.15.61:0/1379510538 learned_addr learned my addr 172.21.15.61:0/1379510538 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:32.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f22ffd700 1 --2- 172.21.15.61:0/1379510538 >> v2:172.21.15.117:3300/0 conn(0x7f7f2405d160 0x7f7f240b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f22ffd700 1 -- 172.21.15.61:0/1379510538 >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 msgr2=0x7f7f24142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:32.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f22ffd700 1 --2- 172.21.15.61:0/1379510538 >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 0x7f7f24142210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f22ffd700 1 -- 172.21.15.61:0/1379510538 >> v1:172.21.15.117:6789/0 conn(0x7f7f240b9ef0 legacy=0x7f7f240b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f22ffd700 1 -- 172.21.15.61:0/1379510538 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f240b1db0 con 0x7f7f2405d160 2022-07-20T03:15:32.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f23fff700 1 --2- 172.21.15.61:0/1379510538 >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 0x7f7f24142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:32.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f22ffd700 1 --2- 172.21.15.61:0/1379510538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f2405d160 0x7f7f240b4e50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f7f14002f70 tx=0x7f7f14006df0).ready entity=mon.1 client_cookie=5fd8754e789a7d1a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:32.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f227fc700 1 -- 172.21.15.61:0/1379510538 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f1400d570 con 0x7f7f2405d160 2022-07-20T03:15:32.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f227fc700 1 -- 172.21.15.61:0/1379510538 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f14000e90 con 0x7f7f2405d160 2022-07-20T03:15:32.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f227fc700 1 -- 172.21.15.61:0/1379510538 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f1400d8c0 con 0x7f7f2405d160 2022-07-20T03:15:32.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- 172.21.15.61:0/1379510538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f2405d160 msgr2=0x7f7f240b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/1379510538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f2405d160 0x7f7f240b4e50 secure :-1 s=READY pgs=464 cs=0 l=1 rev1=1 rx=0x7f7f14002f70 tx=0x7f7f14006df0).stop 2022-07-20T03:15:32.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- 172.21.15.61:0/1379510538 shutdown_connections 2022-07-20T03:15:32.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/1379510538 >> v2:172.21.15.61:3300/0 conn(0x7f7f240b2960 0x7f7f24142210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/1379510538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f2405d160 0x7f7f240b4e50 unknown :-1 s=CLOSED pgs=464 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- 172.21.15.61:0/1379510538 >> 172.21.15.61:0/1379510538 conn(0x7f7f240679d0 msgr2=0x7f7f240b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:32.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- 172.21.15.61:0/1379510538 shutdown_connections 2022-07-20T03:15:32.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.888+0000 7f7f29f0b700 1 -- 172.21.15.61:0/1379510538 wait complete. 2022-07-20T03:15:32.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 Processor -- start 2022-07-20T03:15:32.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 -- start start 2022-07-20T03:15:32.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 0x7f7f24151ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f240b2960 0x7f7f241474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f240b9ef0 0x7f7f2413d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24153070 con 0x7f7f2405d160 2022-07-20T03:15:32.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24152690 con 0x7f7f240b2960 2022-07-20T03:15:32.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7f24152930 con 0x7f7f240b9ef0 2022-07-20T03:15:32.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 0x7f7f24151ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f22ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f240b2960 0x7f7f241474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f237fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 0x7f7f24151ee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48668/0 (socket says 172.21.15.61:48668) 2022-07-20T03:15:32.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f237fe700 1 -- 172.21.15.61:0/7090345 learned_addr learned my addr 172.21.15.61:0/7090345 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:32.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f240b9ef0 0x7f7f2413d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 -- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f240b2960 msgr2=0x7f7f241474b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f240b2960 0x7f7f241474b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 -- 172.21.15.61:0/7090345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 msgr2=0x7f7f24151ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:32.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 0x7f7f24151ee0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:32.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 -- 172.21.15.61:0/7090345 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f1c009530 con 0x7f7f240b9ef0 2022-07-20T03:15:32.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f22ffd700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f240b2960 0x7f7f241474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:32.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f237fe700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 0x7f7f24151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:32.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f23fff700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f240b9ef0 0x7f7f2413d940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7f1c00c000 tx=0x7f7f1c009290).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:32.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f1c00c640 con 0x7f7f240b9ef0 2022-07-20T03:15:32.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7f1c00c7a0 con 0x7f7f240b9ef0 2022-07-20T03:15:32.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f1400b180 con 0x7f7f240b9ef0 2022-07-20T03:15:32.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7f1c00cb00 con 0x7f7f240b9ef0 2022-07-20T03:15:32.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.892+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f241523d0 con 0x7f7f240b9ef0 2022-07-20T03:15:32.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.896+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f240b3780 con 0x7f7f240b9ef0 2022-07-20T03:15:32.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.896+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7f1c022070 con 0x7f7f240b9ef0 2022-07-20T03:15:32.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.896+0000 7f7f20ff9700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f08053af0 0x7f7f08055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:32.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.896+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7f1c06c790 con 0x7f7f240b9ef0 2022-07-20T03:15:32.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.896+0000 7f7f237fe700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f08053af0 0x7f7f08055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:32.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.896+0000 7f7f237fe700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f08053af0 0x7f7f08055fb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f7f0c00d000 tx=0x7f7f0c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:32.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:32.904+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7f1c043a20 con 0x7f7f240b9ef0 2022-07-20T03:15:33.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.028+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7f240574e0 con 0x7f7f240b9ef0 2022-07-20T03:15:33.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.028+0000 7f7f20ff9700 1 -- 172.21.15.61:0/7090345 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v97) v1 ==== 74+0+13230 (secure 0 0 0) 0x7f7f1c029090 con 0x7f7f240b9ef0 2022-07-20T03:15:33.033 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:33.034 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":97,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:32.323153+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":18,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":89,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":79,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":87,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":88,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:33.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f08053af0 msgr2=0x7f7f08055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f08053af0 0x7f7f08055fb0 secure :-1 s=READY pgs=493 cs=0 l=1 rev1=1 rx=0x7f7f0c00d000 tx=0x7f7f0c009040).stop 2022-07-20T03:15:33.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f240b9ef0 msgr2=0x7f7f2413d940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f240b9ef0 0x7f7f2413d940 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f7f1c00c000 tx=0x7f7f1c009290).stop 2022-07-20T03:15:33.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 shutdown_connections 2022-07-20T03:15:33.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7f08053af0 0x7f7f08055fb0 unknown :-1 s=CLOSED pgs=493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7f240b2960 0x7f7f241474b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7f240b9ef0 0x7f7f2413d940 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 --2- 172.21.15.61:0/7090345 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7f2405d160 0x7f7f24151ee0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 >> 172.21.15.61:0/7090345 conn(0x7f7f240679d0 msgr2=0x7f7f24141fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 shutdown_connections 2022-07-20T03:15:33.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.032+0000 7f7f29f0b700 1 -- 172.21.15.61:0/7090345 wait complete. 2022-07-20T03:15:33.047 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:15:33.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:33.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 Processor -- start 2022-07-20T03:15:33.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 -- start start 2022-07-20T03:15:33.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 0x7efdf4062f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf4140920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efdf405d4c0 con 0x7efdf4140e10 2022-07-20T03:15:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efdf40b3aa0 con 0x7efdf405d0e0 2022-07-20T03:15:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfbdc6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf41449f0 con 0x7efdf4099a10 2022-07-20T03:15:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9b62700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 0x7efdf4062f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf4140920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf4140920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35348/0 (socket says 172.21.15.61:35348) 2022-07-20T03:15:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 -- 172.21.15.61:0/3567193908 learned_addr learned my addr 172.21.15.61:0/3567193908 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdfa363700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7efdf4140e10 0x7efdf4061c60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:53778/0 (socket says 172.21.15.61:53778) 2022-07-20T03:15:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9b62700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 0x7efdf4062f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58356/0 (socket says 172.21.15.61:58356) 2022-07-20T03:15:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 -- 172.21.15.61:0/3567193908 >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 msgr2=0x7efdf4062f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:33.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 --2- 172.21.15.61:0/3567193908 >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 0x7efdf4062f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 -- 172.21.15.61:0/3567193908 >> v1:172.21.15.117:6789/0 conn(0x7efdf4140e10 legacy=0x7efdf4061c60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 -- 172.21.15.61:0/3567193908 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf4144c90 con 0x7efdf4099a10 2022-07-20T03:15:33.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9b62700 1 --2- 172.21.15.61:0/3567193908 >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 0x7efdf4062f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf9361700 1 --2- 172.21.15.61:0/3567193908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf4140920 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7efde4002f70 tx=0x7efde4006eb0).ready entity=mon.2 client_cookie=498d20497a5243ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf8b60700 1 -- 172.21.15.61:0/3567193908 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde400d570 con 0x7efdf4099a10 2022-07-20T03:15:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf8b60700 1 -- 172.21.15.61:0/3567193908 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efde4000e90 con 0x7efdf4099a10 2022-07-20T03:15:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.160+0000 7efdf8b60700 1 -- 172.21.15.61:0/3567193908 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efde400d900 con 0x7efdf4099a10 2022-07-20T03:15:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- 172.21.15.61:0/3567193908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 msgr2=0x7efdf4140920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/3567193908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf4140920 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7efde4002f70 tx=0x7efde4006eb0).stop 2022-07-20T03:15:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- 172.21.15.61:0/3567193908 shutdown_connections 2022-07-20T03:15:33.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/3567193908 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf4140920 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/3567193908 >> v2:172.21.15.117:3300/0 conn(0x7efdf405d0e0 0x7efdf4062f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- 172.21.15.61:0/3567193908 >> 172.21.15.61:0/3567193908 conn(0x7efdf40b7cd0 msgr2=0x7efdf40b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- 172.21.15.61:0/3567193908 shutdown_connections 2022-07-20T03:15:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- 172.21.15.61:0/3567193908 wait complete. 2022-07-20T03:15:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 Processor -- start 2022-07-20T03:15:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- start start 2022-07-20T03:15:33.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf405d0e0 0x7efdf40af630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf41540b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 0x7efdf4150130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf41545a0 con 0x7efdf4140e10 2022-07-20T03:15:33.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40afc80 con 0x7efdf405d0e0 2022-07-20T03:15:33.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfbdc6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efdf40aff20 con 0x7efdf4099a10 2022-07-20T03:15:33.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 0x7efdf4150130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 0x7efdf4150130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48682/0 (socket says 172.21.15.61:48682) 2022-07-20T03:15:33.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 -- 172.21.15.61:0/2499839388 learned_addr learned my addr 172.21.15.61:0/2499839388 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:33.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdf9b62700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf405d0e0 0x7efdf40af630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdf9361700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf41540b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 -- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 msgr2=0x7efdf41540b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf41540b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 -- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf405d0e0 msgr2=0x7efdf40af630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf405d0e0 0x7efdf40af630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 -- 172.21.15.61:0/2499839388 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efdf000b180 con 0x7efdf4140e10 2022-07-20T03:15:33.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efdfa363700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 0x7efdf4150130 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7efdf40639a0 tx=0x7efdec007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.164+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec005ae0 con 0x7efdf4140e10 2022-07-20T03:15:33.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efdec005c40 con 0x7efdf4140e10 2022-07-20T03:15:33.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efde400b1e0 con 0x7efdf4140e10 2022-07-20T03:15:33.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efdec0070d0 con 0x7efdf4140e10 2022-07-20T03:15:33.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efdf409bfe0 con 0x7efdf4140e10 2022-07-20T03:15:33.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efdf409ac30 con 0x7efdf4140e10 2022-07-20T03:15:33.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efdec01a060 con 0x7efdf4140e10 2022-07-20T03:15:33.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efde2ffd700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053920 0x7efdd4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efdec06b730 con 0x7efdf4140e10 2022-07-20T03:15:33.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.168+0000 7efdf9b62700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053920 0x7efdd4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.172+0000 7efdf9b62700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053920 0x7efdd4055de0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7efdf000b000 tx=0x7efdf0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.172+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efdf409ac30 con 0x7efdf4140e10 2022-07-20T03:15:33.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.296+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7efdf409ac30 con 0x7efdf4140e10 2022-07-20T03:15:33.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efde2ffd700 1 -- 172.21.15.61:0/2499839388 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v97) v1 ==== 110+0+10 (secure 0 0 0) 0x7efdf409ac30 con 0x7efdf4140e10 2022-07-20T03:15:33.309 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:33.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053920 msgr2=0x7efdd4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053920 0x7efdd4055de0 secure :-1 s=READY pgs=494 cs=0 l=1 rev1=1 rx=0x7efdf000b000 tx=0x7efdf0006df0).stop 2022-07-20T03:15:33.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 msgr2=0x7efdf4150130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 0x7efdf4150130 secure :-1 s=READY pgs=1016 cs=0 l=1 rev1=1 rx=0x7efdf40639a0 tx=0x7efdec007f40).stop 2022-07-20T03:15:33.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 shutdown_connections 2022-07-20T03:15:33.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efdd4053920 0x7efdd4055de0 unknown :-1 s=CLOSED pgs=494 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efdf4140e10 0x7efdf4150130 unknown :-1 s=CLOSED pgs=1016 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efdf4099a10 0x7efdf41540b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 --2- 172.21.15.61:0/2499839388 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efdf405d0e0 0x7efdf40af630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 >> 172.21.15.61:0/2499839388 conn(0x7efdf40b7cd0 msgr2=0x7efdf40ad220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 shutdown_connections 2022-07-20T03:15:33.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.300+0000 7efdfbdc6700 1 -- 172.21.15.61:0/2499839388 wait complete. 2022-07-20T03:15:33.323 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.428+0000 7f7a348b2700 1 Processor -- start 2022-07-20T03:15:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.428+0000 7f7a348b2700 1 -- start start 2022-07-20T03:15:33.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b59d0 0x7f7a300b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7a3014ada0 0x7f7a300b0cd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a300b14f0 0x7f7a3005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a30148080 con 0x7f7a300b14f0 2022-07-20T03:15:33.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a300afbc0 con 0x7f7a3014ada0 2022-07-20T03:15:33.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a300b00e0 con 0x7f7a300b59d0 2022-07-20T03:15:33.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a300b14f0 0x7f7a3005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7a300b14f0 0x7f7a3005bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48694/0 (socket says 172.21.15.61:48694) 2022-07-20T03:15:33.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 -- 172.21.15.61:0/3389914704 learned_addr learned my addr 172.21.15.61:0/3389914704 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:33.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2ed9d700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b59d0 0x7f7a300b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2e59c700 1 --2- 172.21.15.61:0/3389914704 >> v2:172.21.15.117:3300/0 conn(0x7f7a3014ada0 0x7f7a300b0cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 -- 172.21.15.61:0/3389914704 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b59d0 msgr2=0x7f7a300b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b59d0 0x7f7a300b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 -- 172.21.15.61:0/3389914704 >> v2:172.21.15.117:3300/0 conn(0x7f7a3014ada0 msgr2=0x7f7a300b0cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:33.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 --2- 172.21.15.61:0/3389914704 >> v2:172.21.15.117:3300/0 conn(0x7f7a3014ada0 0x7f7a300b0cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 -- 172.21.15.61:0/3389914704 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a300b5db0 con 0x7f7a300b14f0 2022-07-20T03:15:33.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2ed9d700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b59d0 0x7f7a300b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:33.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2e59c700 1 --2- 172.21.15.61:0/3389914704 >> v2:172.21.15.117:3300/0 conn(0x7f7a3014ada0 0x7f7a300b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:33.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2f59e700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b14f0 0x7f7a3005bd90 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f7a280032f0 tx=0x7f7a28009290).ready entity=mon.0 client_cookie=105645c4dea44c5b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2dd9b700 1 -- 172.21.15.61:0/3389914704 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a2800b870 con 0x7f7a300b14f0 2022-07-20T03:15:33.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2dd9b700 1 -- 172.21.15.61:0/3389914704 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a2800b9d0 con 0x7f7a300b14f0 2022-07-20T03:15:33.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a2dd9b700 1 -- 172.21.15.61:0/3389914704 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a2800bb40 con 0x7f7a300b14f0 2022-07-20T03:15:33.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- 172.21.15.61:0/3389914704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b14f0 msgr2=0x7f7a3005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b14f0 0x7f7a3005bd90 secure :-1 s=READY pgs=1017 cs=0 l=1 rev1=1 rx=0x7f7a280032f0 tx=0x7f7a28009290).stop 2022-07-20T03:15:33.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- 172.21.15.61:0/3389914704 shutdown_connections 2022-07-20T03:15:33.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b14f0 0x7f7a3005bd90 unknown :-1 s=CLOSED pgs=1017 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- 172.21.15.61:0/3389914704 >> v2:172.21.15.117:3300/0 conn(0x7f7a3014ada0 0x7f7a300b0cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 --2- 172.21.15.61:0/3389914704 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b59d0 0x7f7a300b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- 172.21.15.61:0/3389914704 >> 172.21.15.61:0/3389914704 conn(0x7f7a300634c0 msgr2=0x7f7a300b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- 172.21.15.61:0/3389914704 shutdown_connections 2022-07-20T03:15:33.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.432+0000 7f7a348b2700 1 -- 172.21.15.61:0/3389914704 wait complete. 2022-07-20T03:15:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 Processor -- start 2022-07-20T03:15:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 -- start start 2022-07-20T03:15:33.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b14f0 0x7f7a30148be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 0x7f7a3010e170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3014ada0 0x7f7a3010e660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a301562c0 con 0x7f7a300b59d0 2022-07-20T03:15:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a30064f20 con 0x7f7a3014ada0 2022-07-20T03:15:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a348b2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3010f320 con 0x7f7a300b14f0 2022-07-20T03:15:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 0x7f7a3010e170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3014ada0 0x7f7a3010e660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 0x7f7a3010e170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48700/0 (socket says 172.21.15.61:48700) 2022-07-20T03:15:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2ed9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b14f0 0x7f7a30148be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 -- 172.21.15.61:0/1183534468 learned_addr learned my addr 172.21.15.61:0/1183534468 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:33.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 -- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b14f0 msgr2=0x7f7a30148be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b14f0 0x7f7a30148be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 -- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3014ada0 msgr2=0x7f7a3010e660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3014ada0 0x7f7a3010e660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2e59c700 1 -- 172.21.15.61:0/1183534468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a28009530 con 0x7f7a300b59d0 2022-07-20T03:15:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.436+0000 7f7a2f59e700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3014ada0 0x7f7a3010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a2e59c700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 0x7f7a3010e170 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f7a300650b0 tx=0x7f7a18008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a18011780 con 0x7f7a300b59d0 2022-07-20T03:15:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a180118e0 con 0x7f7a300b59d0 2022-07-20T03:15:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3010f5c0 con 0x7f7a300b59d0 2022-07-20T03:15:33.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a18011c40 con 0x7f7a300b59d0 2022-07-20T03:15:33.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a300677f0 con 0x7f7a300b59d0 2022-07-20T03:15:33.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a30064d50 con 0x7f7a300b59d0 2022-07-20T03:15:33.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7a1801c460 con 0x7f7a300b59d0 2022-07-20T03:15:33.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a27fff700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a140539d0 0x7f7a14055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f7a1806bca0 con 0x7f7a300b59d0 2022-07-20T03:15:33.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.440+0000 7f7a2ed9d700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a140539d0 0x7f7a14055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.444+0000 7f7a2ed9d700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a140539d0 0x7f7a14055e90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f7a2000c000 tx=0x7f7a20008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.448+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7a18042dc0 con 0x7f7a300b59d0 2022-07-20T03:15:33.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.572+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f7a3010f750 con 0x7f7a300b59d0 2022-07-20T03:15:33.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a27fff700 1 -- 172.21.15.61:0/1183534468 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v97) v1 ==== 92+0+10 (secure 0 0 0) 0x7f7a180463e0 con 0x7f7a300b59d0 2022-07-20T03:15:33.580 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:33.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a140539d0 msgr2=0x7f7a14055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a140539d0 0x7f7a14055e90 secure :-1 s=READY pgs=495 cs=0 l=1 rev1=1 rx=0x7f7a2000c000 tx=0x7f7a20008040).stop 2022-07-20T03:15:33.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 msgr2=0x7f7a3010e170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 0x7f7a3010e170 secure :-1 s=READY pgs=1018 cs=0 l=1 rev1=1 rx=0x7f7a300650b0 tx=0x7f7a18008040).stop 2022-07-20T03:15:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 shutdown_connections 2022-07-20T03:15:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a140539d0 0x7f7a14055e90 unknown :-1 s=CLOSED pgs=495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a3014ada0 0x7f7a3010e660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a300b59d0 0x7f7a3010e170 unknown :-1 s=CLOSED pgs=1018 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 --2- 172.21.15.61:0/1183534468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a300b14f0 0x7f7a30148be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 >> 172.21.15.61:0/1183534468 conn(0x7f7a300634c0 msgr2=0x7f7a300adc30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 shutdown_connections 2022-07-20T03:15:33.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.576+0000 7f7a348b2700 1 -- 172.21.15.61:0/1183534468 wait complete. 2022-07-20T03:15:33.592 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:15:33.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 Processor -- start 2022-07-20T03:15:33.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- start start 2022-07-20T03:15:33.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb41805c3f0 0x7fb41814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180acef0 0x7fb418146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb41805d480 con 0x7fb4180b1e00 2022-07-20T03:15:33.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4180b21e0 con 0x7fb41805c3f0 2022-07-20T03:15:33.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb41805cf40 con 0x7fb4180acef0 2022-07-20T03:15:33.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb41805c3f0 0x7fb41814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41e556700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180acef0 0x7fb418146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb41805c3f0 0x7fb41814a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58384/0 (socket says 172.21.15.61:58384) 2022-07-20T03:15:33.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 -- 172.21.15.61:0/340485430 learned_addr learned my addr 172.21.15.61:0/340485430 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:33.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 -- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180acef0 msgr2=0x7fb418146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:33.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 --2- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180acef0 0x7fb418146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 -- 172.21.15.61:0/340485430 >> v1:172.21.15.117:6789/0 conn(0x7fb4180b1e00 legacy=0x7fb4180ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 -- 172.21.15.61:0/340485430 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41805d1e0 con 0x7fb41805c3f0 2022-07-20T03:15:33.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41d554700 1 --2- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb41805c3f0 0x7fb41814a6d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fb414002f70 tx=0x7fb414006eb0).ready entity=mon.1 client_cookie=95584ead5bcbeb63 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41cd53700 1 -- 172.21.15.61:0/340485430 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41400d570 con 0x7fb41805c3f0 2022-07-20T03:15:33.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41cd53700 1 -- 172.21.15.61:0/340485430 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb414000e90 con 0x7fb41805c3f0 2022-07-20T03:15:33.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41cd53700 1 -- 172.21.15.61:0/340485430 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb41400d900 con 0x7fb41805c3f0 2022-07-20T03:15:33.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb41805c3f0 msgr2=0x7fb41814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb41805c3f0 0x7fb41814a6d0 secure :-1 s=READY pgs=467 cs=0 l=1 rev1=1 rx=0x7fb414002f70 tx=0x7fb414006eb0).stop 2022-07-20T03:15:33.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- 172.21.15.61:0/340485430 shutdown_connections 2022-07-20T03:15:33.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180acef0 0x7fb418146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/340485430 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb41805c3f0 0x7fb41814a6d0 unknown :-1 s=CLOSED pgs=467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- 172.21.15.61:0/340485430 >> 172.21.15.61:0/340485430 conn(0x7fb4180611c0 msgr2=0x7fb418062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- 172.21.15.61:0/340485430 shutdown_connections 2022-07-20T03:15:33.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.700+0000 7fb41ffb9700 1 -- 172.21.15.61:0/340485430 wait complete. 2022-07-20T03:15:33.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 Processor -- start 2022-07-20T03:15:33.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 -- start start 2022-07-20T03:15:33.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180acef0 0x7fb4180b4ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 0x7fb4181571e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180a1d60 0x7fb418159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418065530 con 0x7fb4180b1e00 2022-07-20T03:15:33.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb418156aa0 con 0x7fb4180acef0 2022-07-20T03:15:33.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb418156d40 con 0x7fb4180a1d60 2022-07-20T03:15:33.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 0x7fb4181571e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 0x7fb4181571e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48712/0 (socket says 172.21.15.61:48712) 2022-07-20T03:15:33.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 -- 172.21.15.61:0/1034681696 learned_addr learned my addr 172.21.15.61:0/1034681696 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:33.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41dd55700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180acef0 0x7fb4180b4ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41e556700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180a1d60 0x7fb418159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 -- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180a1d60 msgr2=0x7fb418159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180a1d60 0x7fb418159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 -- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180acef0 msgr2=0x7fb4180b4ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180acef0 0x7fb4180b4ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 -- 172.21.15.61:0/1034681696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb41400b1e0 con 0x7fb4180b1e00 2022-07-20T03:15:33.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41d554700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 0x7fb4181571e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7fb414002f70 tx=0x7fb414015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb414011610 con 0x7fb4180b1e00 2022-07-20T03:15:33.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb414011770 con 0x7fb4180b1e00 2022-07-20T03:15:33.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4180651b0 con 0x7fb4180b1e00 2022-07-20T03:15:33.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb414011ad0 con 0x7fb4180b1e00 2022-07-20T03:15:33.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.704+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb41815a430 con 0x7fb4180b1e00 2022-07-20T03:15:33.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.708+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb418156f00 con 0x7fb4180b1e00 2022-07-20T03:15:33.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.708+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb414025030 con 0x7fb4180b1e00 2022-07-20T03:15:33.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.708+0000 7fb406ffd700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053ad0 0x7fb3f8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.708+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb41406caa0 con 0x7fb4180b1e00 2022-07-20T03:15:33.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.708+0000 7fb41dd55700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053ad0 0x7fb3f8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.708+0000 7fb41dd55700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053ad0 0x7fb3f8055f90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fb40800d000 tx=0x7fb408007b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.712+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb414043c20 con 0x7fb4180b1e00 2022-07-20T03:15:33.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.840+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fb4180b04b0 con 0x7fb4180b1e00 2022-07-20T03:15:33.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.840+0000 7fb406ffd700 1 -- 172.21.15.61:0/1034681696 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v97) v1 ==== 84+0+143 (secure 0 0 0) 0x7fb414023070 con 0x7fb4180b1e00 2022-07-20T03:15:33.846 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:33.847 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:15:33.847 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:33.847 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:15:33.848 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:15:33.848 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:33.848 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:33.849 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:15:33.849 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:15:33.849 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:33.850 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:15:33.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053ad0 msgr2=0x7fb3f8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053ad0 0x7fb3f8055f90 secure :-1 s=READY pgs=496 cs=0 l=1 rev1=1 rx=0x7fb40800d000 tx=0x7fb408007b50).stop 2022-07-20T03:15:33.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 msgr2=0x7fb4181571e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 0x7fb4181571e0 secure :-1 s=READY pgs=1019 cs=0 l=1 rev1=1 rx=0x7fb414002f70 tx=0x7fb414015000).stop 2022-07-20T03:15:33.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 shutdown_connections 2022-07-20T03:15:33.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3f8053ad0 0x7fb3f8055f90 unknown :-1 s=CLOSED pgs=496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4180a1d60 0x7fb418159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4180b1e00 0x7fb4181571e0 unknown :-1 s=CLOSED pgs=1019 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 --2- 172.21.15.61:0/1034681696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4180acef0 0x7fb4180b4ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 >> 172.21.15.61:0/1034681696 conn(0x7fb41805c3f0 msgr2=0x7fb418144850 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 shutdown_connections 2022-07-20T03:15:33.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.844+0000 7fb41ffb9700 1 -- 172.21.15.61:0/1034681696 wait complete. 2022-07-20T03:15:33.864 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:15:33.865 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:15:33.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 Processor -- start 2022-07-20T03:15:33.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- start start 2022-07-20T03:15:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5ae80b3770 0x7f5ae81408d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae805c4c0 con 0x7f5ae80b7bb0 2022-07-20T03:15:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ae80b7f90 con 0x7f5ae80addd0 2022-07-20T03:15:33.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ae805bd90 con 0x7f5ae80b3770 2022-07-20T03:15:33.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5ae80b7bb0 0x7f5ae805ca10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45316/0 (socket says 172.21.15.61:45316) 2022-07-20T03:15:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae6ffd700 1 -- 172.21.15.61:0/1207130538 learned_addr learned my addr 172.21.15.61:0/1207130538 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ade7fc700 1 --2- 172.21.15.61:0/1207130538 >> v2:172.21.15.117:3300/0 conn(0x7f5ae80b3770 0x7f5ae81408d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae67fc700 1 -- 172.21.15.61:0/1207130538 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3102904242 0 0) 0x7f5ae805c4c0 con 0x7f5ae80b7bb0 2022-07-20T03:15:33.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae67fc700 1 -- 172.21.15.61:0/1207130538 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5ac8003260 con 0x7f5ae80b7bb0 2022-07-20T03:15:33.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ade7fc700 1 -- 172.21.15.61:0/1207130538 >> v1:172.21.15.117:6789/0 conn(0x7f5ae80addd0 legacy=0x7f5ae813d620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ade7fc700 1 -- 172.21.15.61:0/1207130538 >> v1:172.21.15.61:6789/0 conn(0x7f5ae80b7bb0 legacy=0x7f5ae805ca10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ade7fc700 1 -- 172.21.15.61:0/1207130538 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ae805bf80 con 0x7f5ae80b3770 2022-07-20T03:15:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ade7fc700 1 --2- 172.21.15.61:0/1207130538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae81408d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f5ad0002f70 tx=0x7f5ad0006df0).ready entity=mon.1 client_cookie=132276bdaab9a08e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:33.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae67fc700 1 -- 172.21.15.61:0/1207130538 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad000d610 con 0x7f5ae80b3770 2022-07-20T03:15:33.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae67fc700 1 -- 172.21.15.61:0/1207130538 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ad0000e90 con 0x7f5ae80b3770 2022-07-20T03:15:33.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae67fc700 1 -- 172.21.15.61:0/1207130538 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ad000d960 con 0x7f5ae80b3770 2022-07-20T03:15:33.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- 172.21.15.61:0/1207130538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 msgr2=0x7f5ae81408d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:33.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/1207130538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae81408d0 secure :-1 s=READY pgs=469 cs=0 l=1 rev1=1 rx=0x7f5ad0002f70 tx=0x7f5ad0006df0).stop 2022-07-20T03:15:33.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- 172.21.15.61:0/1207130538 shutdown_connections 2022-07-20T03:15:33.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/1207130538 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae81408d0 unknown :-1 s=CLOSED pgs=469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:33.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- 172.21.15.61:0/1207130538 >> 172.21.15.61:0/1207130538 conn(0x7f5ae8065720 msgr2=0x7f5ae80b7160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:33.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- 172.21.15.61:0/1207130538 shutdown_connections 2022-07-20T03:15:33.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.976+0000 7f5ae7fff700 1 -- 172.21.15.61:0/1207130538 wait complete. 2022-07-20T03:15:34.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 Processor -- start 2022-07-20T03:15:34.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 -- start start 2022-07-20T03:15:34.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80addd0 0x7f5ae8146060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae813dc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 0x7f5ae813bab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae813ec10 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae813e970 con 0x7f5ae80b3770 2022-07-20T03:15:34.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ae813eef0 con 0x7f5ae80addd0 2022-07-20T03:15:34.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 0x7f5ae813bab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ade7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae813dc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae6ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80addd0 0x7f5ae8146060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 0x7f5ae813bab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48728/0 (socket says 172.21.15.61:48728) 2022-07-20T03:15:34.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 -- 172.21.15.61:0/733040278 learned_addr learned my addr 172.21.15.61:0/733040278 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:34.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 -- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80addd0 msgr2=0x7f5ae8146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80addd0 0x7f5ae8146060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 -- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 msgr2=0x7f5ae813dc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae813dc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 -- 172.21.15.61:0/733040278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ad000b180 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ade7fc700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae813dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:34.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.980+0000 7f5ae77fe700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 0x7f5ae813bab0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f5ae805c8f0 tx=0x7f5ae000e470).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:34.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ae0013070 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5ae000c640 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ae81421e0 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5ae000c9a0 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ae813e380 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ae805c220 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5ae000cb00 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae4ff9700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053a40 0x7f5acc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(97..97 src has 1..97) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5ae0070010 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae6ffd700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053a40 0x7f5acc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.984+0000 7f5ae6ffd700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053a40 0x7f5acc055f00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f5ad800d000 tx=0x7f5ad8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:34.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:33.992+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5ae00472a0 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.112+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f5ae80574e0 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.356+0000 7f5ae4ff9700 1 -- 172.21.15.61:0/733040278 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v98) v1 ==== 133+0+0 (secure 0 0 0) 0x7f5ae004a8c0 con 0x7f5ae80b7bb0 2022-07-20T03:15:34.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053a40 msgr2=0x7f5acc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053a40 0x7f5acc055f00 secure :-1 s=READY pgs=497 cs=0 l=1 rev1=1 rx=0x7f5ad800d000 tx=0x7f5ad8009040).stop 2022-07-20T03:15:34.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 msgr2=0x7f5ae813bab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 0x7f5ae813bab0 secure :-1 s=READY pgs=1021 cs=0 l=1 rev1=1 rx=0x7f5ae805c8f0 tx=0x7f5ae000e470).stop 2022-07-20T03:15:34.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 shutdown_connections 2022-07-20T03:15:34.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5acc053a40 0x7f5acc055f00 unknown :-1 s=CLOSED pgs=497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ae80b3770 0x7f5ae813dc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ae80b7bb0 0x7f5ae813bab0 unknown :-1 s=CLOSED pgs=1021 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 --2- 172.21.15.61:0/733040278 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ae80addd0 0x7f5ae8146060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 >> 172.21.15.61:0/733040278 conn(0x7f5ae8065720 msgr2=0x7f5ae80b6c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:34.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 shutdown_connections 2022-07-20T03:15:34.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.360+0000 7f5ae7fff700 1 -- 172.21.15.61:0/733040278 wait complete. 2022-07-20T03:15:34.373 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:15:34.380 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:15:34.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455ffff700 1 Processor -- start 2022-07-20T03:15:34.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455ffff700 1 -- start start 2022-07-20T03:15:34.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4560142ac0 0x7f45600b5480 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f45600ac840 con 0x7f45600ab0c0 2022-07-20T03:15:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455ffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4560143240 con 0x7f45600ac460 2022-07-20T03:15:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f45601434e0 con 0x7f4560142ac0 2022-07-20T03:15:34.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f4557fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f45600ab0c0 0x7f45600b4d70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45332/0 (socket says 172.21.15.61:45332) 2022-07-20T03:15:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f4557fff700 1 -- 172.21.15.61:0/3234428904 learned_addr learned my addr 172.21.15.61:0/3234428904 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455f7fe700 1 --2- 172.21.15.61:0/3234428904 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4560142ac0 0x7f45600b5480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2037472886 0 0) 0x7f45600ac840 con 0x7f45600ab0c0 2022-07-20T03:15:34.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4540003260 con 0x7f45600ab0c0 2022-07-20T03:15:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732983463 0 0) 0x7f4560143240 con 0x7f45600ac460 2022-07-20T03:15:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.492+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f45600ac840 con 0x7f45600ac460 2022-07-20T03:15:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 -- 172.21.15.61:0/3234428904 >> v1:172.21.15.117:6789/0 conn(0x7f45600ac460 legacy=0x7f456013c830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 -- 172.21.15.61:0/3234428904 >> v1:172.21.15.61:6789/0 conn(0x7f45600ab0c0 legacy=0x7f45600b4d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 -- 172.21.15.61:0/3234428904 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4560143780 con 0x7f4560142ac0 2022-07-20T03:15:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 --2- 172.21.15.61:0/3234428904 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4560142ac0 0x7f45600b5480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f45580032f0 tx=0x7f45580091d0).ready entity=mon.2 client_cookie=885f5d183836885d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:34.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f455800b640 con 0x7f4560142ac0 2022-07-20T03:15:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f455800b7a0 con 0x7f4560142ac0 2022-07-20T03:15:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455e7fc700 1 -- 172.21.15.61:0/3234428904 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f455800b910 con 0x7f4560142ac0 2022-07-20T03:15:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- 172.21.15.61:0/3234428904 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4560142ac0 msgr2=0x7f45600b5480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 --2- 172.21.15.61:0/3234428904 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4560142ac0 0x7f45600b5480 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f45580032f0 tx=0x7f45580091d0).stop 2022-07-20T03:15:34.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- 172.21.15.61:0/3234428904 shutdown_connections 2022-07-20T03:15:34.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 --2- 172.21.15.61:0/3234428904 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4560142ac0 0x7f45600b5480 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- 172.21.15.61:0/3234428904 >> 172.21.15.61:0/3234428904 conn(0x7f45600632b0 msgr2=0x7f4560063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:34.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- 172.21.15.61:0/3234428904 shutdown_connections 2022-07-20T03:15:34.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- 172.21.15.61:0/3234428904 wait complete. 2022-07-20T03:15:34.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 Processor -- start 2022-07-20T03:15:34.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- start start 2022-07-20T03:15:34.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45600ab0c0 0x7f456013faa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45600ac460 0x7f4560145540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 0x7f456013ec40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560145a30 con 0x7f4560142ac0 2022-07-20T03:15:34.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560146b70 con 0x7f45600ac460 2022-07-20T03:15:34.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4560146e10 con 0x7f45600ab0c0 2022-07-20T03:15:34.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 0x7f456013ec40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455effd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45600ab0c0 0x7f456013faa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f4557fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45600ac460 0x7f4560145540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 0x7f456013ec40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48742/0 (socket says 172.21.15.61:48742) 2022-07-20T03:15:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.496+0000 7f455f7fe700 1 -- 172.21.15.61:0/993163759 learned_addr learned my addr 172.21.15.61:0/993163759 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455effd700 1 -- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45600ac460 msgr2=0x7f4560145540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455effd700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45600ac460 0x7f4560145540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455effd700 1 -- 172.21.15.61:0/993163759 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 msgr2=0x7f456013ec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455effd700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 0x7f456013ec40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455effd700 1 -- 172.21.15.61:0/993163759 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4558009470 con 0x7f45600ab0c0 2022-07-20T03:15:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f4557fff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45600ac460 0x7f4560145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455f7fe700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 0x7f456013ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455effd700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45600ab0c0 0x7f456013faa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4550001110 tx=0x7f4550007a20).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4550012400 con 0x7f45600ab0c0 2022-07-20T03:15:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4550012560 con 0x7f45600ab0c0 2022-07-20T03:15:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f456013e410 con 0x7f45600ab0c0 2022-07-20T03:15:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45500128c0 con 0x7f45600ab0c0 2022-07-20T03:15:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4560146550 con 0x7f45600ab0c0 2022-07-20T03:15:34.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560142f50 con 0x7f45600ab0c0 2022-07-20T03:15:34.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4550019030 con 0x7f45600ab0c0 2022-07-20T03:15:34.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455cff9700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4544053a60 0x7f4544055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:34.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.500+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(98..98 src has 1..98) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f455006b070 con 0x7f45600ab0c0 2022-07-20T03:15:34.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.504+0000 7f4557fff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4544053a60 0x7f4544055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:34.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.504+0000 7f4557fff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4544053a60 0x7f4544055f20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f454800d000 tx=0x7f4548009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:34.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.508+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4550042100 con 0x7f45600ab0c0 2022-07-20T03:15:34.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:34.640+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f45600574e0 con 0x7f45600ab0c0 2022-07-20T03:15:35.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.360+0000 7f455cff9700 1 -- 172.21.15.61:0/993163759 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v99) v1 ==== 125+0+0 (secure 0 0 0) 0x7f4550045720 con 0x7f45600ab0c0 2022-07-20T03:15:35.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4544053a60 msgr2=0x7f4544055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4544053a60 0x7f4544055f20 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f454800d000 tx=0x7f4548009040).stop 2022-07-20T03:15:35.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45600ab0c0 msgr2=0x7f456013faa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45600ab0c0 0x7f456013faa0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f4550001110 tx=0x7f4550007a20).stop 2022-07-20T03:15:35.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 shutdown_connections 2022-07-20T03:15:35.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4544053a60 0x7f4544055f20 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f45600ac460 0x7f4560145540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4560142ac0 0x7f456013ec40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 --2- 172.21.15.61:0/993163759 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45600ab0c0 0x7f456013faa0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 >> 172.21.15.61:0/993163759 conn(0x7f45600632b0 msgr2=0x7f45600b4550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:35.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 shutdown_connections 2022-07-20T03:15:35.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.372+0000 7f455ffff700 1 -- 172.21.15.61:0/993163759 wait complete. 2022-07-20T03:15:35.384 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:15:35.389 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:15:35.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.496+0000 7f7ff9855700 1 Processor -- start 2022-07-20T03:15:35.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.496+0000 7f7ff9855700 1 -- start start 2022-07-20T03:15:35.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ff40b6200 0x7f7ff40b08f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff409b760 0x7f7ff40b0de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7ff40b12d0 0x7f7ff405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff40aaed0 con 0x7f7ff40b6200 2022-07-20T03:15:35.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ff40b65e0 con 0x7f7ff40b12d0 2022-07-20T03:15:35.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff40b0350 con 0x7f7ff409b760 2022-07-20T03:15:35.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ff40b6200 0x7f7ff40b08f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ff40b6200 0x7f7ff40b08f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48746/0 (socket says 172.21.15.61:48746) 2022-07-20T03:15:35.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 -- 172.21.15.61:0/886788676 learned_addr learned my addr 172.21.15.61:0/886788676 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:35.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff3fff700 1 --2- 172.21.15.61:0/886788676 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff409b760 0x7f7ff40b0de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9054700 1 --2- 172.21.15.61:0/886788676 >> v2:172.21.15.117:3300/0 conn(0x7f7ff40b12d0 0x7f7ff405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 -- 172.21.15.61:0/886788676 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff409b760 msgr2=0x7f7ff40b0de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:35.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 --2- 172.21.15.61:0/886788676 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff409b760 0x7f7ff40b0de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 -- 172.21.15.61:0/886788676 >> v2:172.21.15.117:3300/0 conn(0x7f7ff40b12d0 msgr2=0x7f7ff405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 --2- 172.21.15.61:0/886788676 >> v2:172.21.15.117:3300/0 conn(0x7f7ff40b12d0 0x7f7ff405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 -- 172.21.15.61:0/886788676 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ff40ac9c0 con 0x7f7ff40b6200 2022-07-20T03:15:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff8853700 1 --2- 172.21.15.61:0/886788676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff40b6200 0x7f7ff40b08f0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f7fe8002f70 tx=0x7f7fe8006eb0).ready entity=mon.0 client_cookie=e0f538f2030a04d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:35.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff37fe700 1 -- 172.21.15.61:0/886788676 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fe800d570 con 0x7f7ff40b6200 2022-07-20T03:15:35.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff37fe700 1 -- 172.21.15.61:0/886788676 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8000e90 con 0x7f7ff40b6200 2022-07-20T03:15:35.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff37fe700 1 -- 172.21.15.61:0/886788676 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fe800d900 con 0x7f7ff40b6200 2022-07-20T03:15:35.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 -- 172.21.15.61:0/886788676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff40b6200 msgr2=0x7f7ff40b08f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.500+0000 7f7ff9855700 1 --2- 172.21.15.61:0/886788676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff40b6200 0x7f7ff40b08f0 secure :-1 s=READY pgs=1023 cs=0 l=1 rev1=1 rx=0x7f7fe8002f70 tx=0x7f7fe8006eb0).stop 2022-07-20T03:15:35.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- 172.21.15.61:0/886788676 shutdown_connections 2022-07-20T03:15:35.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 --2- 172.21.15.61:0/886788676 >> v2:172.21.15.117:3300/0 conn(0x7f7ff40b12d0 0x7f7ff405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 --2- 172.21.15.61:0/886788676 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff409b760 0x7f7ff40b0de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 --2- 172.21.15.61:0/886788676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff40b6200 0x7f7ff40b08f0 unknown :-1 s=CLOSED pgs=1023 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- 172.21.15.61:0/886788676 >> 172.21.15.61:0/886788676 conn(0x7f7ff4063d10 msgr2=0x7f7ff40b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:35.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- 172.21.15.61:0/886788676 shutdown_connections 2022-07-20T03:15:35.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- 172.21.15.61:0/886788676 wait complete. 2022-07-20T03:15:35.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 Processor -- start 2022-07-20T03:15:35.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- start start 2022-07-20T03:15:35.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 0x7f7ff413b6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 0x7f7ff413c7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff40b6200 0x7f7ff4151e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff413d210 con 0x7f7ff409b760 2022-07-20T03:15:35.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff413a780 con 0x7f7ff40b12d0 2022-07-20T03:15:35.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9855700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ff413aa20 con 0x7f7ff40b6200 2022-07-20T03:15:35.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff8853700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 0x7f7ff413b6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 0x7f7ff413c7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9054700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff40b6200 0x7f7ff4151e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff8853700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 0x7f7ff413b6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48752/0 (socket says 172.21.15.61:48752) 2022-07-20T03:15:35.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 0x7f7ff413c7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58434/0 (socket says 172.21.15.61:58434) 2022-07-20T03:15:35.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff9054700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff40b6200 0x7f7ff4151e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35428/0 (socket says 172.21.15.61:35428) 2022-07-20T03:15:35.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.504+0000 7f7ff8853700 1 -- 172.21.15.61:0/4046528755 learned_addr learned my addr 172.21.15.61:0/4046528755 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:35.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff8853700 1 -- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff40b6200 msgr2=0x7f7ff4151e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff8853700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff40b6200 0x7f7ff4151e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff8853700 1 -- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 msgr2=0x7f7ff413c7c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff8853700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 0x7f7ff413c7c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff8853700 1 -- 172.21.15.61:0/4046528755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe800b1e0 con 0x7f7ff409b760 2022-07-20T03:15:35.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff3fff700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 0x7f7ff413c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:35.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff8853700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 0x7f7ff413b6d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f7fe8002f70 tx=0x7f7fe8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:35.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fe80117e0 con 0x7f7ff409b760 2022-07-20T03:15:35.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fe8011940 con 0x7f7ff409b760 2022-07-20T03:15:35.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fe8011ca0 con 0x7f7ff409b760 2022-07-20T03:15:35.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ff409b200 con 0x7f7ff409b760 2022-07-20T03:15:35.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ff408fed0 con 0x7f7ff409b760 2022-07-20T03:15:35.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.508+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ff40574e0 con 0x7f7ff409b760 2022-07-20T03:15:35.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.512+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7fe801a040 con 0x7f7ff409b760 2022-07-20T03:15:35.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.512+0000 7f7ff1ffb700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fd4053920 0x7f7fd4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.512+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(99..99 src has 1..99) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f7fe806cff0 con 0x7f7ff409b760 2022-07-20T03:15:35.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.512+0000 7f7ff3fff700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fd4053920 0x7f7fd4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.512+0000 7f7ff3fff700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fd4053920 0x7f7fd4055de0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f7fe000b000 tx=0x7f7fe0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:35.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.516+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7fe8043ed0 con 0x7f7ff409b760 2022-07-20T03:15:35.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.680+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f7ff4069a20 con 0x7f7ff409b760 2022-07-20T03:15:35.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.692+0000 7f7ff1ffb700 1 -- 172.21.15.61:0/4046528755 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 19 and data pool 20 v55) v1 ==== 189+0+0 (secure 0 0 0) 0x7f7fe8023070 con 0x7f7ff409b760 2022-07-20T03:15:35.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fd4053920 msgr2=0x7f7fd4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fd4053920 0x7f7fd4055de0 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f7fe000b000 tx=0x7f7fe0006df0).stop 2022-07-20T03:15:35.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 msgr2=0x7f7ff413b6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 0x7f7ff413b6d0 secure :-1 s=READY pgs=1024 cs=0 l=1 rev1=1 rx=0x7f7fe8002f70 tx=0x7f7fe8015000).stop 2022-07-20T03:15:35.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 shutdown_connections 2022-07-20T03:15:35.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fd4053920 0x7f7fd4055de0 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ff40b12d0 0x7f7ff413c7c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ff40b6200 0x7f7ff4151e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 --2- 172.21.15.61:0/4046528755 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ff409b760 0x7f7ff413b6d0 unknown :-1 s=CLOSED pgs=1024 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 >> 172.21.15.61:0/4046528755 conn(0x7f7ff4063d10 msgr2=0x7f7ff413e840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:35.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 shutdown_connections 2022-07-20T03:15:35.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.696+0000 7f7ff9855700 1 -- 172.21.15.61:0/4046528755 wait complete. 2022-07-20T03:15:35.711 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 19 and data pool 20 2022-07-20T03:15:35.716 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:35.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 Processor -- start 2022-07-20T03:15:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 -- start start 2022-07-20T03:15:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b01a0 0x7fa1b00b18e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1b00b1dd0 0x7fa1b014a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1b00b1580 con 0x7fa1b00b61d0 2022-07-20T03:15:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1b00b65b0 con 0x7fa1b00b1dd0 2022-07-20T03:15:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1affff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0060a10 con 0x7fa1b00b01a0 2022-07-20T03:15:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1b00b1dd0 0x7fa1b014a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1b00b1dd0 0x7fa1b014a880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48776/0 (socket says 172.21.15.61:48776) 2022-07-20T03:15:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 -- 172.21.15.61:0/3081729668 learned_addr learned my addr 172.21.15.61:0/3081729668 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1ae7fc700 1 --2- 172.21.15.61:0/3081729668 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b01a0 0x7fa1b00b18e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 -- 172.21.15.61:0/3081729668 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b01a0 msgr2=0x7fa1b00b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 --2- 172.21.15.61:0/3081729668 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b01a0 0x7fa1b00b18e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 -- 172.21.15.61:0/3081729668 >> v1:172.21.15.117:6789/0 conn(0x7fa1b00b61d0 legacy=0x7fa1b00ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.808+0000 7fa1af7fe700 1 -- 172.21.15.61:0/3081729668 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1b0060cb0 con 0x7fa1b00b1dd0 2022-07-20T03:15:35.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1af7fe700 1 --2- 172.21.15.61:0/3081729668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b014a880 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7fa1a80032f0 tx=0x7fa1a8009290).ready entity=mon.0 client_cookie=d7f1ad6f979f9eb9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:35.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1adffb700 1 -- 172.21.15.61:0/3081729668 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1a800b8a0 con 0x7fa1b00b1dd0 2022-07-20T03:15:35.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1adffb700 1 -- 172.21.15.61:0/3081729668 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a800ba00 con 0x7fa1b00b1dd0 2022-07-20T03:15:35.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1adffb700 1 -- 172.21.15.61:0/3081729668 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1a800bb70 con 0x7fa1b00b1dd0 2022-07-20T03:15:35.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- 172.21.15.61:0/3081729668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 msgr2=0x7fa1b014a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 --2- 172.21.15.61:0/3081729668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b014a880 secure :-1 s=READY pgs=1025 cs=0 l=1 rev1=1 rx=0x7fa1a80032f0 tx=0x7fa1a8009290).stop 2022-07-20T03:15:35.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- 172.21.15.61:0/3081729668 shutdown_connections 2022-07-20T03:15:35.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 --2- 172.21.15.61:0/3081729668 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b014a880 unknown :-1 s=CLOSED pgs=1025 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 --2- 172.21.15.61:0/3081729668 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b01a0 0x7fa1b00b18e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- 172.21.15.61:0/3081729668 >> 172.21.15.61:0/3081729668 conn(0x7fa1b0063cf0 msgr2=0x7fa1b00b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:35.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- 172.21.15.61:0/3081729668 shutdown_connections 2022-07-20T03:15:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- 172.21.15.61:0/3081729668 wait complete. 2022-07-20T03:15:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 Processor -- start 2022-07-20T03:15:35.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- start start 2022-07-20T03:15:35.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1b00b01a0 0x7fa1b00b3500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b00655c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b61d0 0x7fa1b01477e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b010fa10 con 0x7fa1b00b1dd0 2022-07-20T03:15:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b01470a0 con 0x7fa1b00b01a0 2022-07-20T03:15:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1affff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1b0147340 con 0x7fa1b00b61d0 2022-07-20T03:15:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1ae7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b00655c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1af7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b61d0 0x7fa1b01477e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1aeffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1b00b01a0 0x7fa1b00b3500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1ae7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b00655c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48780/0 (socket says 172.21.15.61:48780) 2022-07-20T03:15:35.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.812+0000 7fa1ae7fc700 1 -- 172.21.15.61:0/2996758100 learned_addr learned my addr 172.21.15.61:0/2996758100 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:35.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1af7fe700 1 -- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1b00b01a0 msgr2=0x7fa1b00b3500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1af7fe700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1b00b01a0 0x7fa1b00b3500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1af7fe700 1 -- 172.21.15.61:0/2996758100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 msgr2=0x7fa1b00655c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1af7fe700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b00655c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1af7fe700 1 -- 172.21.15.61:0/2996758100 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a8009530 con 0x7fa1b00b61d0 2022-07-20T03:15:35.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1ae7fc700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b00655c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:35.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1af7fe700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b61d0 0x7fa1b01477e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa1a80032f0 tx=0x7fa1a8016000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:35.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1a800b8a0 con 0x7fa1b00b61d0 2022-07-20T03:15:35.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1a8021440 con 0x7fa1b00b61d0 2022-07-20T03:15:35.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1a80217a0 con 0x7fa1b00b61d0 2022-07-20T03:15:35.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1b0065ab0 con 0x7fa1b00b61d0 2022-07-20T03:15:35.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1b0064b60 con 0x7fa1b00b61d0 2022-07-20T03:15:35.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1b010f840 con 0x7fa1b00b61d0 2022-07-20T03:15:35.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa1a8029070 con 0x7fa1b00b61d0 2022-07-20T03:15:35.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.816+0000 7fa19ffff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa18c0539b0 0x7fa18c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:35.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.820+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa1a806cea0 con 0x7fa1b00b61d0 2022-07-20T03:15:35.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.820+0000 7fa1aeffd700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa18c0539b0 0x7fa18c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:35.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.820+0000 7fa1aeffd700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa18c0539b0 0x7fa18c055e70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fa1a000d000 tx=0x7fa1a0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:35.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.824+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1a8043e40 con 0x7fa1b00b61d0 2022-07-20T03:15:35.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.952+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fa1b00afa70 con 0x7fa1b00b61d0 2022-07-20T03:15:35.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa19ffff700 1 -- 172.21.15.61:0/2996758100 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16105 (secure 0 0 0) 0x7fa1a8047420 con 0x7fa1b00b61d0 2022-07-20T03:15:35.961 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:35.962 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":100,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:35.686111+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:34.173179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:34.697342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa18c0539b0 msgr2=0x7fa18c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa18c0539b0 0x7fa18c055e70 secure :-1 s=READY pgs=500 cs=0 l=1 rev1=1 rx=0x7fa1a000d000 tx=0x7fa1a0007b90).stop 2022-07-20T03:15:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b61d0 msgr2=0x7fa1b01477e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b61d0 0x7fa1b01477e0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fa1a80032f0 tx=0x7fa1a8016000).stop 2022-07-20T03:15:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 shutdown_connections 2022-07-20T03:15:35.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa18c0539b0 0x7fa18c055e70 unknown :-1 s=CLOSED pgs=500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1b00b61d0 0x7fa1b01477e0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1b00b1dd0 0x7fa1b00655c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 --2- 172.21.15.61:0/2996758100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1b00b01a0 0x7fa1b00b3500 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 >> 172.21.15.61:0/2996758100 conn(0x7fa1b0063cf0 msgr2=0x7fa1b00b44c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:35.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 shutdown_connections 2022-07-20T03:15:35.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:35.956+0000 7fa1affff700 1 -- 172.21.15.61:0/2996758100 wait complete. 2022-07-20T03:15:35.977 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:36.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 Processor -- start 2022-07-20T03:15:36.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 -- start start 2022-07-20T03:15:36.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea140adff0 0x7fea140b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140af8e0 0x7fea1405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fea14147010 con 0x7fea140b5650 2022-07-20T03:15:36.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fea141472b0 con 0x7fea140adff0 2022-07-20T03:15:36.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea14146d40 con 0x7fea140af8e0 2022-07-20T03:15:36.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea140adff0 0x7fea140b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fea140adff0 0x7fea140b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48790/0 (socket says 172.21.15.61:48790) 2022-07-20T03:15:36.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 -- 172.21.15.61:0/2761124816 learned_addr learned my addr 172.21.15.61:0/2761124816 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:36.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea12d9d700 1 --2- 172.21.15.61:0/2761124816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140af8e0 0x7fea1405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 -- 172.21.15.61:0/2761124816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140af8e0 msgr2=0x7fea1405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:36.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 --2- 172.21.15.61:0/2761124816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140af8e0 0x7fea1405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 -- 172.21.15.61:0/2761124816 >> v1:172.21.15.117:6789/0 conn(0x7fea140b5650 legacy=0x7fea14147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 -- 172.21.15.61:0/2761124816 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea140acae0 con 0x7fea140adff0 2022-07-20T03:15:36.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea11d9b700 1 --2- 172.21.15.61:0/2761124816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea140b5a30 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7fea08002f70 tx=0x7fea08006e50).ready entity=mon.0 client_cookie=42139b78223aa310 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1159a700 1 -- 172.21.15.61:0/2761124816 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea0800d5c0 con 0x7fea140adff0 2022-07-20T03:15:36.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1159a700 1 -- 172.21.15.61:0/2761124816 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea08000e90 con 0x7fea140adff0 2022-07-20T03:15:36.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.080+0000 7fea1159a700 1 -- 172.21.15.61:0/2761124816 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea0800d950 con 0x7fea140adff0 2022-07-20T03:15:36.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- 172.21.15.61:0/2761124816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 msgr2=0x7fea140b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 --2- 172.21.15.61:0/2761124816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea140b5a30 secure :-1 s=READY pgs=1026 cs=0 l=1 rev1=1 rx=0x7fea08002f70 tx=0x7fea08006e50).stop 2022-07-20T03:15:36.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- 172.21.15.61:0/2761124816 shutdown_connections 2022-07-20T03:15:36.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 --2- 172.21.15.61:0/2761124816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140af8e0 0x7fea1405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 --2- 172.21.15.61:0/2761124816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea140b5a30 unknown :-1 s=CLOSED pgs=1026 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- 172.21.15.61:0/2761124816 >> 172.21.15.61:0/2761124816 conn(0x7fea140634e0 msgr2=0x7fea140b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:36.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- 172.21.15.61:0/2761124816 shutdown_connections 2022-07-20T03:15:36.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- 172.21.15.61:0/2761124816 wait complete. 2022-07-20T03:15:36.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 Processor -- start 2022-07-20T03:15:36.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- start start 2022-07-20T03:15:36.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea141565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea140af8e0 0x7fea1410feb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140b5650 0x7fea14140fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea14111a90 con 0x7fea140adff0 2022-07-20T03:15:36.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea1410f4d0 con 0x7fea140af8e0 2022-07-20T03:15:36.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fea14111160 con 0x7fea140b5650 2022-07-20T03:15:36.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea141565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea141565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48792/0 (socket says 172.21.15.61:48792) 2022-07-20T03:15:36.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 -- 172.21.15.61:0/1938583517 learned_addr learned my addr 172.21.15.61:0/1938583517 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:36.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea11d9b700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea140af8e0 0x7fea1410feb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea12d9d700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140b5650 0x7fea14140fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 -- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140b5650 msgr2=0x7fea14140fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140b5650 0x7fea14140fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 -- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea140af8e0 msgr2=0x7fea1410feb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea140af8e0 0x7fea1410feb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 -- 172.21.15.61:0/1938583517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea0800b1e0 con 0x7fea140adff0 2022-07-20T03:15:36.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.084+0000 7fea1259c700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea141565a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fea1410eb30 tx=0x7fea0c001400).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea0c00e4a0 con 0x7fea140adff0 2022-07-20T03:15:36.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fea0c00e600 con 0x7fea140adff0 2022-07-20T03:15:36.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea141103a0 con 0x7fea140adff0 2022-07-20T03:15:36.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fea0c00e960 con 0x7fea140adff0 2022-07-20T03:15:36.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea1410ed70 con 0x7fea140adff0 2022-07-20T03:15:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea1410f2f0 con 0x7fea140adff0 2022-07-20T03:15:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fea0c019030 con 0x7fea140adff0 2022-07-20T03:15:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea037fe700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9f0053920 0x7fe9f0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fea0c06b530 con 0x7fea140adff0 2022-07-20T03:15:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea11d9b700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9f0053920 0x7fe9f0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.088+0000 7fea11d9b700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9f0053920 0x7fe9f0055de0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fea0801a000 tx=0x7fea08014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.092+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fea0c042490 con 0x7fea140adff0 2022-07-20T03:15:36.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.220+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fea14111330 con 0x7fea140adff0 2022-07-20T03:15:36.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.220+0000 7fea037fe700 1 -- 172.21.15.61:0/1938583517 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v100) v1 ==== 74+0+16105 (secure 0 0 0) 0x7fea0c045ab0 con 0x7fea140adff0 2022-07-20T03:15:36.225 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:36.225 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":100,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:35.686111+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:34.173179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:34.697342+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:36.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9f0053920 msgr2=0x7fe9f0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9f0053920 0x7fe9f0055de0 secure :-1 s=READY pgs=501 cs=0 l=1 rev1=1 rx=0x7fea0801a000 tx=0x7fea08014000).stop 2022-07-20T03:15:36.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 msgr2=0x7fea141565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea141565a0 secure :-1 s=READY pgs=1027 cs=0 l=1 rev1=1 rx=0x7fea1410eb30 tx=0x7fea0c001400).stop 2022-07-20T03:15:36.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 shutdown_connections 2022-07-20T03:15:36.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9f0053920 0x7fe9f0055de0 unknown :-1 s=CLOSED pgs=501 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fea140af8e0 0x7fea1410feb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fea140b5650 0x7fea14140fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 --2- 172.21.15.61:0/1938583517 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fea140adff0 0x7fea141565a0 unknown :-1 s=CLOSED pgs=1027 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 >> 172.21.15.61:0/1938583517 conn(0x7fea140634e0 msgr2=0x7fea140b3d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:36.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 shutdown_connections 2022-07-20T03:15:36.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.224+0000 7fea1359e700 1 -- 172.21.15.61:0/1938583517 wait complete. 2022-07-20T03:15:36.242 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:15:36.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 Processor -- start 2022-07-20T03:15:36.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 -- start start 2022-07-20T03:15:36.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8b440b62a0 0x7f8b440b43e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8b440b01e0 0x7f8b4413fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b440b6680 con 0x7f8b440b3ed0 2022-07-20T03:15:36.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b441465c0 con 0x7f8b440b01e0 2022-07-20T03:15:36.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b493d7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b44142df0 con 0x7f8b440b62a0 2022-07-20T03:15:36.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8b440b01e0 0x7f8b4413fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8b440b01e0 0x7f8b4413fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48804/0 (socket says 172.21.15.61:48804) 2022-07-20T03:15:36.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 -- 172.21.15.61:0/1458782032 learned_addr learned my addr 172.21.15.61:0/1458782032 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:36.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b42ffd700 1 --2- 172.21.15.61:0/1458782032 >> v2:172.21.15.117:3300/0 conn(0x7f8b440b62a0 0x7f8b440b43e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 -- 172.21.15.61:0/1458782032 >> v2:172.21.15.117:3300/0 conn(0x7f8b440b62a0 msgr2=0x7f8b440b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 --2- 172.21.15.61:0/1458782032 >> v2:172.21.15.117:3300/0 conn(0x7f8b440b62a0 0x7f8b440b43e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 -- 172.21.15.61:0/1458782032 >> v1:172.21.15.117:6789/0 conn(0x7f8b440b3ed0 legacy=0x7f8b44143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.348+0000 7f8b437fe700 1 -- 172.21.15.61:0/1458782032 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b44143090 con 0x7f8b440b01e0 2022-07-20T03:15:36.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b437fe700 1 --2- 172.21.15.61:0/1458782032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b01e0 0x7f8b4413fcc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f8b3c0032f0 tx=0x7f8b3c009290).ready entity=mon.0 client_cookie=973431ca6a936aad server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b41ffb700 1 -- 172.21.15.61:0/1458782032 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c00b870 con 0x7f8b440b01e0 2022-07-20T03:15:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b41ffb700 1 -- 172.21.15.61:0/1458782032 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b3c00b9d0 con 0x7f8b440b01e0 2022-07-20T03:15:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b41ffb700 1 -- 172.21.15.61:0/1458782032 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3c00bb40 con 0x7f8b440b01e0 2022-07-20T03:15:36.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- 172.21.15.61:0/1458782032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b01e0 msgr2=0x7f8b4413fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 --2- 172.21.15.61:0/1458782032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b01e0 0x7f8b4413fcc0 secure :-1 s=READY pgs=1028 cs=0 l=1 rev1=1 rx=0x7f8b3c0032f0 tx=0x7f8b3c009290).stop 2022-07-20T03:15:36.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- 172.21.15.61:0/1458782032 shutdown_connections 2022-07-20T03:15:36.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 --2- 172.21.15.61:0/1458782032 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b01e0 0x7f8b4413fcc0 unknown :-1 s=CLOSED pgs=1028 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 --2- 172.21.15.61:0/1458782032 >> v2:172.21.15.117:3300/0 conn(0x7f8b440b62a0 0x7f8b440b43e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- 172.21.15.61:0/1458782032 >> 172.21.15.61:0/1458782032 conn(0x7f8b44066f00 msgr2=0x7f8b440672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- 172.21.15.61:0/1458782032 shutdown_connections 2022-07-20T03:15:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- 172.21.15.61:0/1458782032 wait complete. 2022-07-20T03:15:36.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 Processor -- start 2022-07-20T03:15:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- start start 2022-07-20T03:15:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b440b01e0 0x7f8b44140d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 0x7f8b44149af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b440b62a0 0x7f8b44149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b441558e0 con 0x7f8b440b3ed0 2022-07-20T03:15:36.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44155b80 con 0x7f8b440b62a0 2022-07-20T03:15:36.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.352+0000 7f8b493d7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b44155e20 con 0x7f8b440b01e0 2022-07-20T03:15:36.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 0x7f8b44149af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 0x7f8b44149af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48808/0 (socket says 172.21.15.61:48808) 2022-07-20T03:15:36.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 -- 172.21.15.61:0/3748716284 learned_addr learned my addr 172.21.15.61:0/3748716284 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:36.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b42ffd700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b440b01e0 0x7f8b44140d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b437fe700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b440b62a0 0x7f8b44149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 -- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b440b01e0 msgr2=0x7f8b44140d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b440b01e0 0x7f8b44140d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 -- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b440b62a0 msgr2=0x7f8b44149fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b440b62a0 0x7f8b44149fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 -- 172.21.15.61:0/3748716284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3c009530 con 0x7f8b440b3ed0 2022-07-20T03:15:36.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b427fc700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 0x7f8b44149af0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f8b440b3b70 tx=0x7f8b34007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b34007030 con 0x7f8b440b3ed0 2022-07-20T03:15:36.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b34007190 con 0x7f8b440b3ed0 2022-07-20T03:15:36.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b34006510 con 0x7f8b440b3ed0 2022-07-20T03:15:36.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b441562c0 con 0x7f8b440b3ed0 2022-07-20T03:15:36.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4414a4d0 con 0x7f8b440b3ed0 2022-07-20T03:15:36.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b44155fe0 con 0x7f8b440b3ed0 2022-07-20T03:15:36.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8b34010030 con 0x7f8b440b3ed0 2022-07-20T03:15:36.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.356+0000 7f8b33fff700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b200539b0 0x7f8b20055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.360+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(100..100 src has 1..100) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8b3406b9d0 con 0x7f8b440b3ed0 2022-07-20T03:15:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.360+0000 7f8b42ffd700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b200539b0 0x7f8b20055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.360+0000 7f8b42ffd700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b200539b0 0x7f8b20055e70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f8b3800b000 tx=0x7f8b38006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.364+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b34042930 con 0x7f8b440b3ed0 2022-07-20T03:15:36.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.532+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f8b440574e0 con 0x7f8b440b3ed0 2022-07-20T03:15:36.689 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:36.692 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:36.696 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:36.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.692+0000 7f8b33fff700 1 -- 172.21.15.61:0/3748716284 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v56) v1 ==== 118+0+0 (secure 0 0 0) 0x7f8b34045f50 con 0x7f8b440b3ed0 2022-07-20T03:15:36.699 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:36.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b200539b0 msgr2=0x7f8b20055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b200539b0 0x7f8b20055e70 secure :-1 s=READY pgs=502 cs=0 l=1 rev1=1 rx=0x7f8b3800b000 tx=0x7f8b38006df0).stop 2022-07-20T03:15:36.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 msgr2=0x7f8b44149af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 0x7f8b44149af0 secure :-1 s=READY pgs=1029 cs=0 l=1 rev1=1 rx=0x7f8b440b3b70 tx=0x7f8b34007c10).stop 2022-07-20T03:15:36.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 shutdown_connections 2022-07-20T03:15:36.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b200539b0 0x7f8b20055e70 unknown :-1 s=CLOSED pgs=502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b440b62a0 0x7f8b44149fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b440b3ed0 0x7f8b44149af0 unknown :-1 s=CLOSED pgs=1029 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 --2- 172.21.15.61:0/3748716284 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b440b01e0 0x7f8b44140d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 >> 172.21.15.61:0/3748716284 conn(0x7f8b44066f00 msgr2=0x7f8b4413eb60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:36.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 shutdown_connections 2022-07-20T03:15:36.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.696+0000 7f8b493d7700 1 -- 172.21.15.61:0/3748716284 wait complete. 2022-07-20T03:15:36.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:15:36.713 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:36.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 Processor -- start 2022-07-20T03:15:36.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- start start 2022-07-20T03:15:36.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c40ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40aff40 con 0x7ff0c40ae5c0 2022-07-20T03:15:36.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff0c40b0bc0 con 0x7ff0c40b1e00 2022-07-20T03:15:36.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c40b21e0 con 0x7ff0c40afa30 2022-07-20T03:15:36.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff0c40ae5c0 0x7ff0c4143200 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45416/0 (socket says 172.21.15.61:45416) 2022-07-20T03:15:36.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c3fff700 1 -- 172.21.15.61:0/1679133274 learned_addr learned my addr 172.21.15.61:0/1679133274 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:36.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0baffd700 1 --2- 172.21.15.61:0/1679133274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c40ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 589526531 0 0) 0x7ff0c40aff40 con 0x7ff0c40ae5c0 2022-07-20T03:15:36.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0a4003260 con 0x7ff0c40ae5c0 2022-07-20T03:15:36.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3989666836 0 0) 0x7ff0c40b0bc0 con 0x7ff0c40b1e00 2022-07-20T03:15:36.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff0c40aff40 con 0x7ff0c40b1e00 2022-07-20T03:15:36.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0baffd700 1 -- 172.21.15.61:0/1679133274 >> v1:172.21.15.117:6789/0 conn(0x7ff0c40b1e00 legacy=0x7ff0c40ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0baffd700 1 -- 172.21.15.61:0/1679133274 >> v1:172.21.15.61:6789/0 conn(0x7ff0c40ae5c0 legacy=0x7ff0c4143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0baffd700 1 -- 172.21.15.61:0/1679133274 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0c40af510 con 0x7ff0c40afa30 2022-07-20T03:15:36.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0baffd700 1 --2- 172.21.15.61:0/1679133274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c40ae0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff0ac002f70 tx=0x7ff0ac006df0).ready entity=mon.2 client_cookie=6c7891e70d851948 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0ac00d5c0 con 0x7ff0c40afa30 2022-07-20T03:15:36.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac000e90 con 0x7ff0c40afa30 2022-07-20T03:15:36.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c2ffd700 1 -- 172.21.15.61:0/1679133274 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0ac00d910 con 0x7ff0c40afa30 2022-07-20T03:15:36.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- 172.21.15.61:0/1679133274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 msgr2=0x7ff0c40ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/1679133274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c40ae0d0 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7ff0ac002f70 tx=0x7ff0ac006df0).stop 2022-07-20T03:15:36.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- 172.21.15.61:0/1679133274 shutdown_connections 2022-07-20T03:15:36.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/1679133274 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c40ae0d0 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- 172.21.15.61:0/1679133274 >> 172.21.15.61:0/1679133274 conn(0x7ff0c40611c0 msgr2=0x7ff0c4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:36.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- 172.21.15.61:0/1679133274 shutdown_connections 2022-07-20T03:15:36.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.812+0000 7ff0c9e91700 1 -- 172.21.15.61:0/1679133274 wait complete. 2022-07-20T03:15:36.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 Processor -- start 2022-07-20T03:15:36.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 -- start start 2022-07-20T03:15:36.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c405c340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0c40b1e00 0x7ff0c4156310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 0x7ff0c4152ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4156800 con 0x7ff0c4152640 2022-07-20T03:15:36.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4066980 con 0x7ff0c40b1e00 2022-07-20T03:15:36.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c9e91700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff0c4066c20 con 0x7ff0c40afa30 2022-07-20T03:15:36.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 0x7ff0c4152ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0c40b1e00 0x7ff0c4156310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c37fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c405c340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 0x7ff0c4152ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48824/0 (socket says 172.21.15.61:48824) 2022-07-20T03:15:36.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c3fff700 1 -- 172.21.15.61:0/424154791 learned_addr learned my addr 172.21.15.61:0/424154791 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:36.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 -- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 msgr2=0x7ff0c405c340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c405c340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 -- 172.21.15.61:0/424154791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 msgr2=0x7ff0c4152ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:36.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 0x7ff0c4152ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:36.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 -- 172.21.15.61:0/424154791 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff0ac00b180 con 0x7ff0c40b1e00 2022-07-20T03:15:36.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c37fe700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c405c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:36.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0c3fff700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 0x7ff0c4152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:36.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.816+0000 7ff0baffd700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0c40b1e00 0x7ff0c4156310 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7ff0ac002f70 tx=0x7ff0ac014000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0ac016070 con 0x7ff0c40b1e00 2022-07-20T03:15:36.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff0ac0137d0 con 0x7ff0c40b1e00 2022-07-20T03:15:36.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff0c4066ec0 con 0x7ff0c40b1e00 2022-07-20T03:15:36.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff0ac013b30 con 0x7ff0c40b1e00 2022-07-20T03:15:36.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff0c40ab510 con 0x7ff0c40b1e00 2022-07-20T03:15:36.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff0c41532f0 con 0x7ff0c40b1e00 2022-07-20T03:15:36.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff0ac00f030 con 0x7ff0c40b1e00 2022-07-20T03:15:36.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c17fa700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0a8053a40 0x7ff0a8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:36.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff0ac06c920 con 0x7ff0c40b1e00 2022-07-20T03:15:36.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.820+0000 7ff0c37fe700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0a8053a40 0x7ff0a8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:36.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.824+0000 7ff0c37fe700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0a8053a40 0x7ff0a8055f00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7ff0b400d000 tx=0x7ff0b4007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:36.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.832+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff0ac0437a0 con 0x7ff0c40b1e00 2022-07-20T03:15:37.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:36.996+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff0c40b0170 con 0x7ff0c40b1e00 2022-07-20T03:15:37.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c17fa700 1 -- 172.21.15.61:0/424154791 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7ff0ac023090 con 0x7ff0c40b1e00 2022-07-20T03:15:37.003 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:37.004 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:35.686626+0000","modified":"2022-07-20T03:15:36.689415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7335":{"gid":7335,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2834789489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2834789489},{"type":"v1","addr":"172.21.15.61:6833","nonce":2834789489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T03:15:37.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0a8053a40 msgr2=0x7ff0a8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0a8053a40 0x7ff0a8055f00 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7ff0b400d000 tx=0x7ff0b4007a00).stop 2022-07-20T03:15:37.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0c40b1e00 msgr2=0x7ff0c4156310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0c40b1e00 0x7ff0c4156310 secure :-1 s=READY pgs=475 cs=0 l=1 rev1=1 rx=0x7ff0ac002f70 tx=0x7ff0ac014000).stop 2022-07-20T03:15:37.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 shutdown_connections 2022-07-20T03:15:37.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff0a8053a40 0x7ff0a8055f00 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff0c40b1e00 0x7ff0c4156310 unknown :-1 s=CLOSED pgs=475 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff0c4152640 0x7ff0c4152ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 --2- 172.21.15.61:0/424154791 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff0c40afa30 0x7ff0c405c340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 >> 172.21.15.61:0/424154791 conn(0x7ff0c40ae5c0 msgr2=0x7ff0c40b30f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 shutdown_connections 2022-07-20T03:15:37.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.000+0000 7ff0c9e91700 1 -- 172.21.15.61:0/424154791 wait complete. 2022-07-20T03:15:37.015 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-20T03:15:37.017 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:37.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 Processor -- start 2022-07-20T03:15:37.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 -- start start 2022-07-20T03:15:37.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f84900b1e00 0x7f849014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f849005c3f0 0x7f84900ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f849005d480 con 0x7f849005bd90 2022-07-20T03:15:37.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84900b21e0 con 0x7f849005c3f0 2022-07-20T03:15:37.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f8496ce2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f849005cf40 con 0x7f84900b1e00 2022-07-20T03:15:37.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f848ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f849005c3f0 0x7f84900ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f848ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f849005c3f0 0x7f84900ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48828/0 (socket says 172.21.15.61:48828) 2022-07-20T03:15:37.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f849527f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f849005bd90 0x7f8490146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45430/0 (socket says 172.21.15.61:45430) 2022-07-20T03:15:37.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.112+0000 7f848ffff700 1 -- 172.21.15.61:0/2337185496 learned_addr learned my addr 172.21.15.61:0/2337185496 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:37.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8494a7e700 1 --2- 172.21.15.61:0/2337185496 >> v2:172.21.15.117:3300/0 conn(0x7f84900b1e00 0x7f849014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848ffff700 1 -- 172.21.15.61:0/2337185496 >> v2:172.21.15.117:3300/0 conn(0x7f84900b1e00 msgr2=0x7f849014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848f7fe700 1 -- 172.21.15.61:0/2337185496 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 282040942 0 0) 0x7f849005d480 con 0x7f849005bd90 2022-07-20T03:15:37.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848ffff700 1 --2- 172.21.15.61:0/2337185496 >> v2:172.21.15.117:3300/0 conn(0x7f84900b1e00 0x7f849014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848ffff700 1 -- 172.21.15.61:0/2337185496 >> v1:172.21.15.61:6789/0 conn(0x7f849005bd90 legacy=0x7f8490146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848ffff700 1 -- 172.21.15.61:0/2337185496 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849005d1e0 con 0x7f849005c3f0 2022-07-20T03:15:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848ffff700 1 --2- 172.21.15.61:0/2337185496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849005c3f0 0x7f84900ac7e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f8480002f70 tx=0x7f8480006eb0).ready entity=mon.0 client_cookie=228205b70282f298 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848f7fe700 1 -- 172.21.15.61:0/2337185496 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848000d610 con 0x7f849005c3f0 2022-07-20T03:15:37.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848f7fe700 1 -- 172.21.15.61:0/2337185496 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8480000e90 con 0x7f849005c3f0 2022-07-20T03:15:37.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f848f7fe700 1 -- 172.21.15.61:0/2337185496 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848000d960 con 0x7f849005c3f0 2022-07-20T03:15:37.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 -- 172.21.15.61:0/2337185496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849005c3f0 msgr2=0x7f84900ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2337185496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849005c3f0 0x7f84900ac7e0 secure :-1 s=READY pgs=1032 cs=0 l=1 rev1=1 rx=0x7f8480002f70 tx=0x7f8480006eb0).stop 2022-07-20T03:15:37.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 -- 172.21.15.61:0/2337185496 shutdown_connections 2022-07-20T03:15:37.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2337185496 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849005c3f0 0x7f84900ac7e0 unknown :-1 s=CLOSED pgs=1032 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2337185496 >> v2:172.21.15.117:3300/0 conn(0x7f84900b1e00 0x7f849014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 -- 172.21.15.61:0/2337185496 >> 172.21.15.61:0/2337185496 conn(0x7f84900611c0 msgr2=0x7f8490062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 -- 172.21.15.61:0/2337185496 shutdown_connections 2022-07-20T03:15:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.116+0000 7f8496ce2700 1 -- 172.21.15.61:0/2337185496 wait complete. 2022-07-20T03:15:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 Processor -- start 2022-07-20T03:15:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 -- start start 2022-07-20T03:15:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849005c3f0 0x7f849009e210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 0x7f84900a1840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8490063a80 0x7f8490159d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8490157340 con 0x7f84900b1e00 2022-07-20T03:15:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84901575e0 con 0x7f849005c3f0 2022-07-20T03:15:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 0x7f84900a1840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f8496ce2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84900a1d30 con 0x7f8490063a80 2022-07-20T03:15:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 0x7f84900a1840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48836/0 (socket says 172.21.15.61:48836) 2022-07-20T03:15:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 -- 172.21.15.61:0/2579142064 learned_addr learned my addr 172.21.15.61:0/2579142064 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:37.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 -- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8490063a80 msgr2=0x7f8490159d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8490063a80 0x7f8490159d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:15:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 -- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849005c3f0 msgr2=0x7f849009e210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849005c3f0 0x7f849009e210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:15:37.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 -- 172.21.15.61:0/2579142064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f848000b1e0 con 0x7f84900b1e00 2022-07-20T03:15:37.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.120+0000 7f848ffff700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 0x7f84900a1840 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f8480002f70 tx=0x7f8480015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848000d290 con 0x7f84900b1e00 2022-07-20T03:15:37.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f848000de90 con 0x7f84900b1e00 2022-07-20T03:15:37.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f848001e460 con 0x7f84900b1e00 2022-07-20T03:15:37.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84900b5e50 con 0x7f84900b1e00 2022-07-20T03:15:37.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84900678b0 con 0x7f84900b1e00 2022-07-20T03:15:37.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8490063e80 con 0x7f84900b1e00 2022-07-20T03:15:37.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f848001a040 con 0x7f84900b1e00 2022-07-20T03:15:37.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f848dffb700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8470053ad0 0x7f8470055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f848006d810 con 0x7f84900b1e00 2022-07-20T03:15:37.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.124+0000 7f8494a7e700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8470053ad0 0x7f8470055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.128+0000 7f8494a7e700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8470053ad0 0x7f8470055f90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f848400a000 tx=0x7f8484008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.132+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8480043d80 con 0x7f84900b1e00 2022-07-20T03:15:37.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.256+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f84900574e0 con 0x7f84900b1e00 2022-07-20T03:15:37.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.256+0000 7f848dffb700 1 -- 172.21.15.61:0/2579142064 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16092 (secure 0 0 0) 0x7f84800472e0 con 0x7f84900b1e00 2022-07-20T03:15:37.265 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:37.266 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":101,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:36.690047+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:34.173179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:34.697342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:37.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8470053ad0 msgr2=0x7f8470055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8470053ad0 0x7f8470055f90 secure :-1 s=READY pgs=504 cs=0 l=1 rev1=1 rx=0x7f848400a000 tx=0x7f8484008040).stop 2022-07-20T03:15:37.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 msgr2=0x7f84900a1840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 0x7f84900a1840 secure :-1 s=READY pgs=1033 cs=0 l=1 rev1=1 rx=0x7f8480002f70 tx=0x7f8480015000).stop 2022-07-20T03:15:37.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 shutdown_connections 2022-07-20T03:15:37.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8470053ad0 0x7f8470055f90 unknown :-1 s=CLOSED pgs=504 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8490063a80 0x7f8490159d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:15:37.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84900b1e00 0x7f84900a1840 unknown :-1 s=CLOSED pgs=1033 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 --2- 172.21.15.61:0/2579142064 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f849005c3f0 0x7f849009e210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:15:37.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 >> 172.21.15.61:0/2579142064 conn(0x7f849005bd90 msgr2=0x7f84900ae710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 shutdown_connections 2022-07-20T03:15:37.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.260+0000 7f8496ce2700 1 -- 172.21.15.61:0/2579142064 wait complete. 2022-07-20T03:15:37.277 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:15:37.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe436daa700 1 Processor -- start 2022-07-20T03:15:37.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe436daa700 1 -- start start 2022-07-20T03:15:37.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe436daa700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe430066cb0 0x7fe43014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe436daa700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe4301480f0 con 0x7fe4300624b0 2022-07-20T03:15:37.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe436daa700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe43005d4c0 con 0x7fe43005d0e0 2022-07-20T03:15:37.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe436daa700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe43014ad70 con 0x7fe430066cb0 2022-07-20T03:15:37.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe427fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe430066cb0 0x7fe43014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe435347700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe4300624b0 0x7fe430150d00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45444/0 (socket says 172.21.15.61:45444) 2022-07-20T03:15:37.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe435347700 1 -- 172.21.15.61:0/3148285183 learned_addr learned my addr 172.21.15.61:0/3148285183 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:37.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe427fff700 1 -- 172.21.15.61:0/3148285183 >> v1:172.21.15.117:6789/0 conn(0x7fe43005d0e0 legacy=0x7fe430061da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe4277fe700 1 -- 172.21.15.61:0/3148285183 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3616773259 0 0) 0x7fe43005d4c0 con 0x7fe43005d0e0 2022-07-20T03:15:37.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe427fff700 1 -- 172.21.15.61:0/3148285183 >> v1:172.21.15.61:6789/0 conn(0x7fe4300624b0 legacy=0x7fe430150d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe427fff700 1 -- 172.21.15.61:0/3148285183 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe43014b010 con 0x7fe430066cb0 2022-07-20T03:15:37.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe427fff700 1 --2- 172.21.15.61:0/3148285183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe430066cb0 0x7fe43014a6d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fe420002f70 tx=0x7fe420006d90).ready entity=mon.0 client_cookie=37a18fdfc153f31d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe4277fe700 1 -- 172.21.15.61:0/3148285183 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe42000f6d0 con 0x7fe430066cb0 2022-07-20T03:15:37.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe4277fe700 1 -- 172.21.15.61:0/3148285183 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe420000e90 con 0x7fe430066cb0 2022-07-20T03:15:37.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.388+0000 7fe4277fe700 1 -- 172.21.15.61:0/3148285183 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe42000fa20 con 0x7fe430066cb0 2022-07-20T03:15:37.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- 172.21.15.61:0/3148285183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe430066cb0 msgr2=0x7fe43014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 --2- 172.21.15.61:0/3148285183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe430066cb0 0x7fe43014a6d0 secure :-1 s=READY pgs=1035 cs=0 l=1 rev1=1 rx=0x7fe420002f70 tx=0x7fe420006d90).stop 2022-07-20T03:15:37.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- 172.21.15.61:0/3148285183 shutdown_connections 2022-07-20T03:15:37.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 --2- 172.21.15.61:0/3148285183 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe430066cb0 0x7fe43014a6d0 unknown :-1 s=CLOSED pgs=1035 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- 172.21.15.61:0/3148285183 >> 172.21.15.61:0/3148285183 conn(0x7fe4300b7d60 msgr2=0x7fe4300b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- 172.21.15.61:0/3148285183 shutdown_connections 2022-07-20T03:15:37.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- 172.21.15.61:0/3148285183 wait complete. 2022-07-20T03:15:37.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 Processor -- start 2022-07-20T03:15:37.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- start start 2022-07-20T03:15:37.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 0x7fe4300b33b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4300624b0 0x7fe430143f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe430066cb0 0x7fe430144430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4300ae7a0 con 0x7fe43005d0e0 2022-07-20T03:15:37.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe43009ada0 con 0x7fe430066cb0 2022-07-20T03:15:37.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe436daa700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe43009b040 con 0x7fe4300624b0 2022-07-20T03:15:37.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe434b46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 0x7fe4300b33b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe434b46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 0x7fe4300b33b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48850/0 (socket says 172.21.15.61:48850) 2022-07-20T03:15:37.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe434b46700 1 -- 172.21.15.61:0/3808190508 learned_addr learned my addr 172.21.15.61:0/3808190508 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:37.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe427fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4300624b0 0x7fe430143f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.392+0000 7fe435347700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe430066cb0 0x7fe430144430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe434b46700 1 -- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4300624b0 msgr2=0x7fe430143f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe434b46700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4300624b0 0x7fe430143f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe434b46700 1 -- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe430066cb0 msgr2=0x7fe430144430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe434b46700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe430066cb0 0x7fe430144430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe434b46700 1 -- 172.21.15.61:0/3808190508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe42000b120 con 0x7fe43005d0e0 2022-07-20T03:15:37.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe434b46700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 0x7fe4300b33b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7fe42c000ed0 tx=0x7fe42c007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe42c012400 con 0x7fe43005d0e0 2022-07-20T03:15:37.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe42c012560 con 0x7fe43005d0e0 2022-07-20T03:15:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe42c0128c0 con 0x7fe43005d0e0 2022-07-20T03:15:37.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe430149880 con 0x7fe43005d0e0 2022-07-20T03:15:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe43009a890 con 0x7fe43005d0e0 2022-07-20T03:15:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe43009b200 con 0x7fe43005d0e0 2022-07-20T03:15:37.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe42c019030 con 0x7fe43005d0e0 2022-07-20T03:15:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.396+0000 7fe425ffb700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe410053920 0x7fe410055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.400+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe42c06b120 con 0x7fe43005d0e0 2022-07-20T03:15:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.400+0000 7fe427fff700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe410053920 0x7fe410055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.400+0000 7fe427fff700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe410053920 0x7fe410055de0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fe42001c000 tx=0x7fe420015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.400+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe42c0420d0 con 0x7fe43005d0e0 2022-07-20T03:15:37.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.568+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fe43014b2b0 con 0x7fe43005d0e0 2022-07-20T03:15:37.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe425ffb700 1 -- 172.21.15.61:0/3808190508 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v35) v1 ==== 184+0+0 (secure 0 0 0) 0x7fe42c0456f0 con 0x7fe43005d0e0 2022-07-20T03:15:37.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe410053920 msgr2=0x7fe410055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe410053920 0x7fe410055de0 secure :-1 s=READY pgs=505 cs=0 l=1 rev1=1 rx=0x7fe42001c000 tx=0x7fe420015000).stop 2022-07-20T03:15:37.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 msgr2=0x7fe4300b33b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 0x7fe4300b33b0 secure :-1 s=READY pgs=1036 cs=0 l=1 rev1=1 rx=0x7fe42c000ed0 tx=0x7fe42c007ac0).stop 2022-07-20T03:15:37.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 shutdown_connections 2022-07-20T03:15:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe410053920 0x7fe410055de0 unknown :-1 s=CLOSED pgs=505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe430066cb0 0x7fe430144430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4300624b0 0x7fe430143f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 --2- 172.21.15.61:0/3808190508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe43005d0e0 0x7fe4300b33b0 unknown :-1 s=CLOSED pgs=1036 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 >> 172.21.15.61:0/3808190508 conn(0x7fe4300b7d60 msgr2=0x7fe4301453f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 shutdown_connections 2022-07-20T03:15:37.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.580+0000 7fe436daa700 1 -- 172.21.15.61:0/3808190508 wait complete. 2022-07-20T03:15:37.590 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:15:37.599 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:15:37.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.720+0000 7f461324e700 1 Processor -- start 2022-07-20T03:15:37.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.720+0000 7f461324e700 1 -- start start 2022-07-20T03:15:37.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f460c0b3d40 0x7f460c0b41f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f460c0b5020 con 0x7f460c0b2550 2022-07-20T03:15:37.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f460c0b3640 con 0x7f460c0b3d40 2022-07-20T03:15:37.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c0add00 con 0x7f460c0b2170 2022-07-20T03:15:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461224c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f460c0b3d40 0x7f460c0b41f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f460c0b3d40 0x7f460c0b41f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58538/0 (socket says 172.21.15.61:58538) 2022-07-20T03:15:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 -- 172.21.15.61:0/2556747140 learned_addr learned my addr 172.21.15.61:0/2556747140 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461224c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0b51e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35528/0 (socket says 172.21.15.61:35528) 2022-07-20T03:15:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 -- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 msgr2=0x7f460c0b51e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:37.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 --2- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0b51e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 -- 172.21.15.61:0/2556747140 >> v1:172.21.15.117:6789/0 conn(0x7f460c0b2550 legacy=0x7f460c0aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 -- 172.21.15.61:0/2556747140 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f460c0adad0 con 0x7f460c0b3d40 2022-07-20T03:15:37.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461224c700 1 --2- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:37.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f4612a4d700 1 --2- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c0b41f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f4604002f70 tx=0x7f4604006f40).ready entity=mon.1 client_cookie=47d50513c52dc26a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461124a700 1 -- 172.21.15.61:0/2556747140 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460400c880 con 0x7f460c0b3d40 2022-07-20T03:15:37.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461124a700 1 -- 172.21.15.61:0/2556747140 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4604000e90 con 0x7f460c0b3d40 2022-07-20T03:15:37.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461124a700 1 -- 172.21.15.61:0/2556747140 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f460400cc10 con 0x7f460c0b3d40 2022-07-20T03:15:37.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 msgr2=0x7f460c0b41f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 --2- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c0b41f0 secure :-1 s=READY pgs=478 cs=0 l=1 rev1=1 rx=0x7f4604002f70 tx=0x7f4604006f40).stop 2022-07-20T03:15:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- 172.21.15.61:0/2556747140 shutdown_connections 2022-07-20T03:15:37.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 --2- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c0b41f0 unknown :-1 s=CLOSED pgs=478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 --2- 172.21.15.61:0/2556747140 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0b51e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- 172.21.15.61:0/2556747140 >> 172.21.15.61:0/2556747140 conn(0x7f460c063150 msgr2=0x7f460c063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- 172.21.15.61:0/2556747140 shutdown_connections 2022-07-20T03:15:37.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.724+0000 7f461324e700 1 -- 172.21.15.61:0/2556747140 wait complete. 2022-07-20T03:15:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 Processor -- start 2022-07-20T03:15:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 -- start start 2022-07-20T03:15:37.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0a8cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 0x7f460c13dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c1524c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c152080 con 0x7f460c0b2550 2022-07-20T03:15:37.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f460c13ebc0 con 0x7f460c0b3d40 2022-07-20T03:15:37.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f460c156150 con 0x7f460c0b2170 2022-07-20T03:15:37.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 0x7f460c13dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461224c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0a8cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4612a4d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c1524c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 0x7f460c13dde0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48866/0 (socket says 172.21.15.61:48866) 2022-07-20T03:15:37.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4612a4d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c1524c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58544/0 (socket says 172.21.15.61:58544) 2022-07-20T03:15:37.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 -- 172.21.15.61:0/2532465581 learned_addr learned my addr 172.21.15.61:0/2532465581 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:37.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461224c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0a8cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35534/0 (socket says 172.21.15.61:35534) 2022-07-20T03:15:37.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 -- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 msgr2=0x7f460c0a8cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0a8cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 -- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 msgr2=0x7f460c1524c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c1524c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 -- 172.21.15.61:0/2532465581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45fc009470 con 0x7f460c0b2550 2022-07-20T03:15:37.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f4611a4b700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 0x7f460c13dde0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f460c0b49a0 tx=0x7f45f4007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45f4012400 con 0x7f460c0b2550 2022-07-20T03:15:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45f4012560 con 0x7f460c0b2550 2022-07-20T03:15:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f460400a360 con 0x7f460c0b2550 2022-07-20T03:15:37.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f45f40128c0 con 0x7f460c0b2550 2022-07-20T03:15:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.728+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f460c064540 con 0x7f460c0b2550 2022-07-20T03:15:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.732+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f460c1565f0 con 0x7f460c0b2550 2022-07-20T03:15:37.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.732+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f45f4019030 con 0x7f460c0b2550 2022-07-20T03:15:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.732+0000 7f460b7fe700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f45f00539b0 0x7f45f0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.732+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f45f406b150 con 0x7f460c0b2550 2022-07-20T03:15:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.732+0000 7f461224c700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f45f00539b0 0x7f45f0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:37.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.732+0000 7f461224c700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f45f00539b0 0x7f45f0055e70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f45fc00b000 tx=0x7f45fc0091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:37.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.736+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45f4042050 con 0x7f460c0b2550 2022-07-20T03:15:37.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.908+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f460c0b38a0 con 0x7f460c0b2550 2022-07-20T03:15:37.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.916+0000 7f460b7fe700 1 -- 172.21.15.61:0/2532465581 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v36) v1 ==== 184+0+0 (secure 0 0 0) 0x7f45f4045670 con 0x7f460c0b2550 2022-07-20T03:15:37.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f45f00539b0 msgr2=0x7f45f0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f45f00539b0 0x7f45f0055e70 secure :-1 s=READY pgs=506 cs=0 l=1 rev1=1 rx=0x7f45fc00b000 tx=0x7f45fc0091d0).stop 2022-07-20T03:15:37.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 msgr2=0x7f460c13dde0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:37.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 0x7f460c13dde0 secure :-1 s=READY pgs=1037 cs=0 l=1 rev1=1 rx=0x7f460c0b49a0 tx=0x7f45f4007c20).stop 2022-07-20T03:15:37.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 shutdown_connections 2022-07-20T03:15:37.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f45f00539b0 0x7f45f0055e70 unknown :-1 s=CLOSED pgs=506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f460c0b3d40 0x7f460c1524c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f460c0b2550 0x7f460c13dde0 unknown :-1 s=CLOSED pgs=1037 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 --2- 172.21.15.61:0/2532465581 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f460c0b2170 0x7f460c0a8cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:37.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 >> 172.21.15.61:0/2532465581 conn(0x7f460c063150 msgr2=0x7f460c0b1910 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:37.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 shutdown_connections 2022-07-20T03:15:37.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:37.920+0000 7f461324e700 1 -- 172.21.15.61:0/2532465581 wait complete. 2022-07-20T03:15:37.928 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:15:37.938 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:15:38.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 Processor -- start 2022-07-20T03:15:38.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 -- start start 2022-07-20T03:15:38.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f246c05cf40 0x7f246c144ca0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f246c066cb0 0x7f246c141100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f246c0649d0 con 0x7f246c0b1c30 2022-07-20T03:15:38.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c1442c0 con 0x7f246c05cf40 2022-07-20T03:15:38.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f24743db700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f246c144560 con 0x7f246c066cb0 2022-07-20T03:15:38.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f246c05cf40 0x7f246c144ca0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f246c05cf40 0x7f246c144ca0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48870/0 (socket says 172.21.15.61:48870) 2022-07-20T03:15:38.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 -- 172.21.15.61:0/2881289956 learned_addr learned my addr 172.21.15.61:0/2881289956 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:38.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472978700 1 --2- 172.21.15.61:0/2881289956 >> v2:172.21.15.117:3300/0 conn(0x7f246c066cb0 0x7f246c141100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 -- 172.21.15.61:0/2881289956 >> v2:172.21.15.117:3300/0 conn(0x7f246c066cb0 msgr2=0x7f246c141100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:38.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 --2- 172.21.15.61:0/2881289956 >> v2:172.21.15.117:3300/0 conn(0x7f246c066cb0 0x7f246c141100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 -- 172.21.15.61:0/2881289956 >> v1:172.21.15.117:6789/0 conn(0x7f246c0b1c30 legacy=0x7f246c061da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:38.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 -- 172.21.15.61:0/2881289956 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f246c144800 con 0x7f246c05cf40 2022-07-20T03:15:38.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2472177700 1 --2- 172.21.15.61:0/2881289956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c05cf40 0x7f246c144ca0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7f246c064670 tx=0x7f2468009290).ready entity=mon.0 client_cookie=fdb31a5bfc4688f6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:38.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2471175700 1 -- 172.21.15.61:0/2881289956 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f246800b830 con 0x7f246c05cf40 2022-07-20T03:15:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2471175700 1 -- 172.21.15.61:0/2881289956 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f246800b990 con 0x7f246c05cf40 2022-07-20T03:15:38.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.052+0000 7f2471175700 1 -- 172.21.15.61:0/2881289956 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f246800bb40 con 0x7f246c05cf40 2022-07-20T03:15:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- 172.21.15.61:0/2881289956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c05cf40 msgr2=0x7f246c144ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 --2- 172.21.15.61:0/2881289956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c05cf40 0x7f246c144ca0 secure :-1 s=READY pgs=1038 cs=0 l=1 rev1=1 rx=0x7f246c064670 tx=0x7f2468009290).stop 2022-07-20T03:15:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- 172.21.15.61:0/2881289956 shutdown_connections 2022-07-20T03:15:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 --2- 172.21.15.61:0/2881289956 >> v2:172.21.15.117:3300/0 conn(0x7f246c066cb0 0x7f246c141100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 --2- 172.21.15.61:0/2881289956 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c05cf40 0x7f246c144ca0 unknown :-1 s=CLOSED pgs=1038 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- 172.21.15.61:0/2881289956 >> 172.21.15.61:0/2881289956 conn(0x7f246c0b7d60 msgr2=0x7f246c0b3d30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:38.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- 172.21.15.61:0/2881289956 shutdown_connections 2022-07-20T03:15:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- 172.21.15.61:0/2881289956 wait complete. 2022-07-20T03:15:38.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 Processor -- start 2022-07-20T03:15:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- start start 2022-07-20T03:15:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f246c05cf40 0x7f246c155740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 0x7f246c09ad40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f246c0b1c30 0x7f246c0b5400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c0953c0 con 0x7f246c066cb0 2022-07-20T03:15:38.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f246c095660 con 0x7f246c0b1c30 2022-07-20T03:15:38.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.056+0000 7f24743db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f246c0b5bf0 con 0x7f246c05cf40 2022-07-20T03:15:38.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 0x7f246c09ad40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 0x7f246c09ad40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48878/0 (socket says 172.21.15.61:48878) 2022-07-20T03:15:38.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 -- 172.21.15.61:0/1064830922 learned_addr learned my addr 172.21.15.61:0/1064830922 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:38.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2472177700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f246c05cf40 0x7f246c155740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2472978700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f246c0b1c30 0x7f246c0b5400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 -- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f246c05cf40 msgr2=0x7f246c155740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f246c05cf40 0x7f246c155740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 -- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f246c0b1c30 msgr2=0x7f246c0b5400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f246c0b1c30 0x7f246c0b5400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 -- 172.21.15.61:0/1064830922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2468009530 con 0x7f246c066cb0 2022-07-20T03:15:38.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f2471976700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 0x7f246c09ad40 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f2460002f80 tx=0x7f2460006d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:38.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2460006030 con 0x7f246c066cb0 2022-07-20T03:15:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2460006190 con 0x7f246c066cb0 2022-07-20T03:15:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f246c094310 con 0x7f246c066cb0 2022-07-20T03:15:38.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24600064f0 con 0x7f246c066cb0 2022-07-20T03:15:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f246c09c910 con 0x7f246c066cb0 2022-07-20T03:15:38.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f246c1415f0 con 0x7f246c066cb0 2022-07-20T03:15:38.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.060+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f246000f070 con 0x7f246c066cb0 2022-07-20T03:15:38.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.064+0000 7f245f7fe700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f244c053a40 0x7f244c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.064+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f246006b450 con 0x7f246c066cb0 2022-07-20T03:15:38.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.064+0000 7f2472177700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f244c053a40 0x7f244c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.064+0000 7f2472177700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f244c053a40 0x7f244c055f00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f246801b000 tx=0x7f2468016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:38.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.068+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24600423b0 con 0x7f246c066cb0 2022-07-20T03:15:38.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.232+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f246c140200 con 0x7f246c066cb0 2022-07-20T03:15:38.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.240+0000 7f245f7fe700 1 -- 172.21.15.61:0/1064830922 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v37) v1 ==== 184+0+0 (secure 0 0 0) 0x7f24600459d0 con 0x7f246c066cb0 2022-07-20T03:15:38.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f244c053a40 msgr2=0x7f244c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f244c053a40 0x7f244c055f00 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f246801b000 tx=0x7f2468016000).stop 2022-07-20T03:15:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 msgr2=0x7f246c09ad40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 0x7f246c09ad40 secure :-1 s=READY pgs=1039 cs=0 l=1 rev1=1 rx=0x7f2460002f80 tx=0x7f2460006d60).stop 2022-07-20T03:15:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 shutdown_connections 2022-07-20T03:15:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f244c053a40 0x7f244c055f00 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f246c0b1c30 0x7f246c0b5400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f246c066cb0 0x7f246c09ad40 unknown :-1 s=CLOSED pgs=1039 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 --2- 172.21.15.61:0/1064830922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f246c05cf40 0x7f246c155740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 >> 172.21.15.61:0/1064830922 conn(0x7f246c0b7d60 msgr2=0x7f246c0b2a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:38.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 shutdown_connections 2022-07-20T03:15:38.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.244+0000 7f24743db700 1 -- 172.21.15.61:0/1064830922 wait complete. 2022-07-20T03:15:38.252 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:15:38.262 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:38.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 Processor -- start 2022-07-20T03:15:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- start start 2022-07-20T03:15:38.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce840af8e0 0x7fce840b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce84147010 con 0x7fce840b5650 2022-07-20T03:15:38.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce841472b0 con 0x7fce840adff0 2022-07-20T03:15:38.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce84146d40 con 0x7fce840af8e0 2022-07-20T03:15:38.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce88aea700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce840af8e0 0x7fce840b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce83fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fce840b5650 0x7fce84147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45482/0 (socket says 172.21.15.61:45482) 2022-07-20T03:15:38.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce83fff700 1 -- 172.21.15.61:0/92165512 learned_addr learned my addr 172.21.15.61:0/92165512 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce88aea700 1 -- 172.21.15.61:0/92165512 >> v1:172.21.15.117:6789/0 conn(0x7fce840adff0 legacy=0x7fce8405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce88aea700 1 -- 172.21.15.61:0/92165512 >> v1:172.21.15.61:6789/0 conn(0x7fce840b5650 legacy=0x7fce84147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce88aea700 1 -- 172.21.15.61:0/92165512 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce8405c4a0 con 0x7fce840af8e0 2022-07-20T03:15:38.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce82ffd700 1 -- 172.21.15.61:0/92165512 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 822262576 0 0) 0x7fce84147010 con 0x7fce840b5650 2022-07-20T03:15:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce88aea700 1 --2- 172.21.15.61:0/92165512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce840b5a30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fce74002f70 tx=0x7fce74006df0).ready entity=mon.0 client_cookie=245de6959023d051 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce82ffd700 1 -- 172.21.15.61:0/92165512 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce7400d610 con 0x7fce840af8e0 2022-07-20T03:15:38.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce82ffd700 1 -- 172.21.15.61:0/92165512 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce74000e90 con 0x7fce840af8e0 2022-07-20T03:15:38.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce82ffd700 1 -- 172.21.15.61:0/92165512 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce7400d960 con 0x7fce840af8e0 2022-07-20T03:15:38.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- 172.21.15.61:0/92165512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 msgr2=0x7fce840b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 --2- 172.21.15.61:0/92165512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce840b5a30 secure :-1 s=READY pgs=1041 cs=0 l=1 rev1=1 rx=0x7fce74002f70 tx=0x7fce74006df0).stop 2022-07-20T03:15:38.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- 172.21.15.61:0/92165512 shutdown_connections 2022-07-20T03:15:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 --2- 172.21.15.61:0/92165512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce840b5a30 unknown :-1 s=CLOSED pgs=1041 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- 172.21.15.61:0/92165512 >> 172.21.15.61:0/92165512 conn(0x7fce840634e0 msgr2=0x7fce840b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:38.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- 172.21.15.61:0/92165512 shutdown_connections 2022-07-20T03:15:38.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.360+0000 7fce892eb700 1 -- 172.21.15.61:0/92165512 wait complete. 2022-07-20T03:15:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 Processor -- start 2022-07-20T03:15:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 -- start start 2022-07-20T03:15:38.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce840adff0 0x7fce84066c80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce84067170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce840b5650 0x7fce841563f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce84065d80 con 0x7fce840af8e0 2022-07-20T03:15:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce841571f0 con 0x7fce840b5650 2022-07-20T03:15:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce84111910 con 0x7fce840adff0 2022-07-20T03:15:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce84067170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce84067170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48892/0 (socket says 172.21.15.61:48892) 2022-07-20T03:15:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 -- 172.21.15.61:0/1389871003 learned_addr learned my addr 172.21.15.61:0/1389871003 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce83fff700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce840adff0 0x7fce84066c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce88aea700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce840b5650 0x7fce841563f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 -- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce840adff0 msgr2=0x7fce84066c80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce840adff0 0x7fce84066c80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 -- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce840b5650 msgr2=0x7fce841563f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce840b5650 0x7fce841563f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 -- 172.21.15.61:0/1389871003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce7400b180 con 0x7fce840af8e0 2022-07-20T03:15:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce837fe700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce84067170 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fce840622d0 tx=0x7fce7c007f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:38.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce7c0068e0 con 0x7fce840af8e0 2022-07-20T03:15:38.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce7c006a40 con 0x7fce840af8e0 2022-07-20T03:15:38.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce84067660 con 0x7fce840af8e0 2022-07-20T03:15:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce7c006bb0 con 0x7fce840af8e0 2022-07-20T03:15:38.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.364+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce84110c40 con 0x7fce840af8e0 2022-07-20T03:15:38.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.368+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce841573b0 con 0x7fce840af8e0 2022-07-20T03:15:38.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.368+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fce7c018020 con 0x7fce840af8e0 2022-07-20T03:15:38.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.368+0000 7fce817fa700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce5c053a60 0x7fce5c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.368+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fce7c06b300 con 0x7fce840af8e0 2022-07-20T03:15:38.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.368+0000 7fce83fff700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce5c053a60 0x7fce5c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.368+0000 7fce83fff700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce5c053a60 0x7fce5c055f20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fce6c00b000 tx=0x7fce6c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:38.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.376+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fce7c042200 con 0x7fce840af8e0 2022-07-20T03:15:38.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.536+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fce84111ad0 con 0x7fce840af8e0 2022-07-20T03:15:38.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.536+0000 7fce817fa700 1 -- 172.21.15.61:0/1389871003 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7fce7c016020 con 0x7fce840af8e0 2022-07-20T03:15:38.540 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:38.541 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:35.686626+0000","modified":"2022-07-20T03:15:36.689415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7335":{"gid":7335,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2834789489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2834789489},{"type":"v1","addr":"172.21.15.61:6833","nonce":2834789489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T03:15:38.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce5c053a60 msgr2=0x7fce5c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce5c053a60 0x7fce5c055f20 secure :-1 s=READY pgs=508 cs=0 l=1 rev1=1 rx=0x7fce6c00b000 tx=0x7fce6c009040).stop 2022-07-20T03:15:38.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 msgr2=0x7fce84067170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:38.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce84067170 secure :-1 s=READY pgs=1042 cs=0 l=1 rev1=1 rx=0x7fce840622d0 tx=0x7fce7c007f90).stop 2022-07-20T03:15:38.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 shutdown_connections 2022-07-20T03:15:38.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce5c053a60 0x7fce5c055f20 unknown :-1 s=CLOSED pgs=508 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce840b5650 0x7fce841563f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce840af8e0 0x7fce84067170 unknown :-1 s=CLOSED pgs=1042 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 --2- 172.21.15.61:0/1389871003 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce840adff0 0x7fce84066c80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:38.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 >> 172.21.15.61:0/1389871003 conn(0x7fce840634e0 msgr2=0x7fce841460f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:38.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 shutdown_connections 2022-07-20T03:15:38.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.540+0000 7fce892eb700 1 -- 172.21.15.61:0/1389871003 wait complete. 2022-07-20T03:15:38.550 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-20T03:15:38.762 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:35.686626+0000', 'modified': '2022-07-20T03:15:36.689415+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7335}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7335': {'gid': 7335, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2834789489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2834789489}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2834789489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:15:38.763 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:15:38.764 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:15:38.885 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7335,"want_state":"up:active","state":"up:active","rank_uptime":3.1855653300000002,"mdsmap_epoch":56,"osdmap_epoch":101,"osdmap_epoch_barrier":100,"uptime":7.6567401650000004} 2022-07-20T03:15:38.886 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7335, "mdsmap_epoch": 56, "osdmap_epoch": 101, "osdmap_epoch_barrier": 100, "rank_uptime": 3.18556533, "state": "up:active", "uptime": 7.656740165, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:15:38.887 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:38.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.988+0000 7f1b5d0e9700 1 Processor -- start 2022-07-20T03:15:38.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.988+0000 7f1b5d0e9700 1 -- start start 2022-07-20T03:15:38.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580b2530 0x7f1b58141fc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1b580acdd0 0x7f1b581424b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:38.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b580b2910 con 0x7f1b580614b0 2022-07-20T03:15:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b580abb60 con 0x7f1b580acdd0 2022-07-20T03:15:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b580abe00 con 0x7f1b580b2530 2022-07-20T03:15:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1b580acdd0 0x7f1b581424b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b56d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580b2530 0x7f1b58141fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1b580acdd0 0x7f1b581424b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58578/0 (socket says 172.21.15.61:58578) 2022-07-20T03:15:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 -- 172.21.15.61:0/2621800002 learned_addr learned my addr 172.21.15.61:0/2621800002 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5759e700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f1b580614b0 0x7f1b58146570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:54004/0 (socket says 172.21.15.61:54004) 2022-07-20T03:15:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 -- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580b2530 msgr2=0x7f1b58141fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 --2- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580b2530 0x7f1b58141fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 -- 172.21.15.61:0/2621800002 >> v1:172.21.15.117:6789/0 conn(0x7f1b580614b0 legacy=0x7f1b58146570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 -- 172.21.15.61:0/2621800002 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b581429a0 con 0x7f1b580acdd0 2022-07-20T03:15:39.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b4ffff700 1 --2- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b581424b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f1b40002f70 tx=0x7f1b40006eb0).ready entity=mon.1 client_cookie=bb533f3fbf70d5d7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5659c700 1 -- 172.21.15.61:0/2621800002 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b4000d570 con 0x7f1b580acdd0 2022-07-20T03:15:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5659c700 1 -- 172.21.15.61:0/2621800002 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b40000e90 con 0x7f1b580acdd0 2022-07-20T03:15:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5659c700 1 -- 172.21.15.61:0/2621800002 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b4000d900 con 0x7f1b580acdd0 2022-07-20T03:15:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 msgr2=0x7f1b581424b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b581424b0 secure :-1 s=READY pgs=482 cs=0 l=1 rev1=1 rx=0x7f1b40002f70 tx=0x7f1b40006eb0).stop 2022-07-20T03:15:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/2621800002 shutdown_connections 2022-07-20T03:15:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b581424b0 unknown :-1 s=CLOSED pgs=482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/2621800002 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580b2530 0x7f1b58141fc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/2621800002 >> 172.21.15.61:0/2621800002 conn(0x7f1b580b36b0 msgr2=0x7f1b580b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/2621800002 shutdown_connections 2022-07-20T03:15:39.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.992+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/2621800002 wait complete. 2022-07-20T03:15:39.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 Processor -- start 2022-07-20T03:15:39.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 -- start start 2022-07-20T03:15:39.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580614b0 0x7f1b5809ab20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b580af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 0x7f1b58098420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58097a40 con 0x7f1b580b2530 2022-07-20T03:15:39.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58097ce0 con 0x7f1b580acdd0 2022-07-20T03:15:39.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b58097f80 con 0x7f1b580614b0 2022-07-20T03:15:39.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 0x7f1b58098420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b56d9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580614b0 0x7f1b5809ab20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 0x7f1b58098420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48908/0 (socket says 172.21.15.61:48908) 2022-07-20T03:15:39.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 -- 172.21.15.61:0/4241994885 learned_addr learned my addr 172.21.15.61:0/4241994885 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:39.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b4ffff700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b580af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 -- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580614b0 msgr2=0x7f1b5809ab20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580614b0 0x7f1b5809ab20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 -- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 msgr2=0x7f1b580af9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b580af9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 -- 172.21.15.61:0/4241994885 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b4000b1e0 con 0x7f1b580b2530 2022-07-20T03:15:39.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5759e700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 0x7f1b58098420 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f1b580b6080 tx=0x7f1b50007f70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b50005ae0 con 0x7f1b580b2530 2022-07-20T03:15:39.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b50005c40 con 0x7f1b580b2530 2022-07-20T03:15:39.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b58098910 con 0x7f1b580b2530 2022-07-20T03:15:39.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b500070d0 con 0x7f1b580b2530 2022-07-20T03:15:39.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:38.996+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b58155600 con 0x7f1b580b2530 2022-07-20T03:15:39.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.000+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1b58097890 con 0x7f1b580b2530 2022-07-20T03:15:39.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.000+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1b50010070 con 0x7f1b580b2530 2022-07-20T03:15:39.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.000+0000 7f1b54d99700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b3c053a40 0x7f1b3c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.000+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1b5006b890 con 0x7f1b580b2530 2022-07-20T03:15:39.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.000+0000 7f1b56d9d700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b3c053a40 0x7f1b3c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.004+0000 7f1b56d9d700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b3c053a40 0x7f1b3c055f00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f1b4800b000 tx=0x7f1b48006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.008+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1b50042660 con 0x7f1b580b2530 2022-07-20T03:15:39.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.136+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1b58098140 con 0x7f1b580b2530 2022-07-20T03:15:39.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.136+0000 7f1b54d99700 1 -- 172.21.15.61:0/4241994885 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16092 (secure 0 0 0) 0x7f1b50045c80 con 0x7f1b580b2530 2022-07-20T03:15:39.146 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:39.146 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":101,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:36.690047+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:34.173179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:34.697342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b3c053a40 msgr2=0x7f1b3c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b3c053a40 0x7f1b3c055f00 secure :-1 s=READY pgs=509 cs=0 l=1 rev1=1 rx=0x7f1b4800b000 tx=0x7f1b48006df0).stop 2022-07-20T03:15:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 msgr2=0x7f1b58098420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 0x7f1b58098420 secure :-1 s=READY pgs=1043 cs=0 l=1 rev1=1 rx=0x7f1b580b6080 tx=0x7f1b50007f70).stop 2022-07-20T03:15:39.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 shutdown_connections 2022-07-20T03:15:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b3c053a40 0x7f1b3c055f00 unknown :-1 s=CLOSED pgs=509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b580acdd0 0x7f1b580af9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b580b2530 0x7f1b58098420 unknown :-1 s=CLOSED pgs=1043 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 --2- 172.21.15.61:0/4241994885 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b580614b0 0x7f1b5809ab20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 >> 172.21.15.61:0/4241994885 conn(0x7f1b580b36b0 msgr2=0x7f1b580b55a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 shutdown_connections 2022-07-20T03:15:39.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.140+0000 7f1b5d0e9700 1 -- 172.21.15.61:0/4241994885 wait complete. 2022-07-20T03:15:39.158 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:15:34.173179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:15:34.697342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:15:39.159 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:39.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.272+0000 7f61a3e4e700 1 Processor -- start 2022-07-20T03:15:39.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.272+0000 7f61a3e4e700 1 -- start start 2022-07-20T03:15:39.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f619c0b5b50 0x7f619c0accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f619c05c120 0x7f619c144320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f619c0ac5e0 con 0x7f619c0b0480 2022-07-20T03:15:39.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c0b5f30 con 0x7f619c05c120 2022-07-20T03:15:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f619c05cf70 con 0x7f619c0b5b50 2022-07-20T03:15:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f619c05c120 0x7f619c144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f619c05c120 0x7f619c144320 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48916/0 (socket says 172.21.15.61:48916) 2022-07-20T03:15:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 -- 172.21.15.61:0/1176817185 learned_addr learned my addr 172.21.15.61:0/1176817185 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:39.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a1bea700 1 --2- 172.21.15.61:0/1176817185 >> v2:172.21.15.117:3300/0 conn(0x7f619c0b5b50 0x7f619c0accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 -- 172.21.15.61:0/1176817185 >> v2:172.21.15.117:3300/0 conn(0x7f619c0b5b50 msgr2=0x7f619c0accb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 --2- 172.21.15.61:0/1176817185 >> v2:172.21.15.117:3300/0 conn(0x7f619c0b5b50 0x7f619c0accb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 -- 172.21.15.61:0/1176817185 >> v1:172.21.15.117:6789/0 conn(0x7f619c0b0480 legacy=0x7f619c14a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 -- 172.21.15.61:0/1176817185 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619c0ac810 con 0x7f619c05c120 2022-07-20T03:15:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a23eb700 1 --2- 172.21.15.61:0/1176817185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c144320 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f6198002f70 tx=0x7f6198006eb0).ready entity=mon.0 client_cookie=ec19df837cdeaad4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a0be8700 1 -- 172.21.15.61:0/1176817185 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f619800d5c0 con 0x7f619c05c120 2022-07-20T03:15:39.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a0be8700 1 -- 172.21.15.61:0/1176817185 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6198000e90 con 0x7f619c05c120 2022-07-20T03:15:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a0be8700 1 -- 172.21.15.61:0/1176817185 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f619800d950 con 0x7f619c05c120 2022-07-20T03:15:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- 172.21.15.61:0/1176817185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 msgr2=0x7f619c144320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/1176817185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c144320 secure :-1 s=READY pgs=1044 cs=0 l=1 rev1=1 rx=0x7f6198002f70 tx=0x7f6198006eb0).stop 2022-07-20T03:15:39.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- 172.21.15.61:0/1176817185 shutdown_connections 2022-07-20T03:15:39.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/1176817185 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c144320 unknown :-1 s=CLOSED pgs=1044 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/1176817185 >> v2:172.21.15.117:3300/0 conn(0x7f619c0b5b50 0x7f619c0accb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- 172.21.15.61:0/1176817185 >> 172.21.15.61:0/1176817185 conn(0x7f619c0635e0 msgr2=0x7f619c0b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- 172.21.15.61:0/1176817185 shutdown_connections 2022-07-20T03:15:39.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.276+0000 7f61a3e4e700 1 -- 172.21.15.61:0/1176817185 wait complete. 2022-07-20T03:15:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 Processor -- start 2022-07-20T03:15:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 -- start start 2022-07-20T03:15:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c156590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f619c0b0480 0x7f619c064320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f619c0b5b50 0x7f619c0acd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c156a80 con 0x7f619c05c120 2022-07-20T03:15:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f619c159c30 con 0x7f619c0b5b50 2022-07-20T03:15:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a3e4e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f619c064810 con 0x7f619c0b0480 2022-07-20T03:15:39.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c156590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c156590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48918/0 (socket says 172.21.15.61:48918) 2022-07-20T03:15:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 -- 172.21.15.61:0/4133730967 learned_addr learned my addr 172.21.15.61:0/4133730967 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a13e9700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f619c0b0480 0x7f619c064320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a23eb700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f619c0b5b50 0x7f619c0acd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 -- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f619c0b0480 msgr2=0x7f619c064320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f619c0b0480 0x7f619c064320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 -- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f619c0b5b50 msgr2=0x7f619c0acd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f619c0b5b50 0x7f619c0acd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.280+0000 7f61a1bea700 1 -- 172.21.15.61:0/4133730967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f619800b1e0 con 0x7f619c05c120 2022-07-20T03:15:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f61a1bea700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c156590 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f619000c000 tx=0x7f6190006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6190011750 con 0x7f619c05c120 2022-07-20T03:15:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f61900118b0 con 0x7f619c05c120 2022-07-20T03:15:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f619c064aa0 con 0x7f619c05c120 2022-07-20T03:15:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6190011c10 con 0x7f619c05c120 2022-07-20T03:15:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f619c0ad2e0 con 0x7f619c05c120 2022-07-20T03:15:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f619c159da0 con 0x7f619c05c120 2022-07-20T03:15:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6190011d70 con 0x7f619c05c120 2022-07-20T03:15:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f618effd700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f617c053ad0 0x7f617c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f619006c240 con 0x7f619c05c120 2022-07-20T03:15:39.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.284+0000 7f61a13e9700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f617c053ad0 0x7f617c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.288+0000 7f61a13e9700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f617c053ad0 0x7f617c055f90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f619400d000 tx=0x7f6194007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.292+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6190043140 con 0x7f619c05c120 2022-07-20T03:15:39.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f619c14ad80 con 0x7f619c05c120 2022-07-20T03:15:39.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f618effd700 1 -- 172.21.15.61:0/4133730967 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v101) v1 ==== 110+0+10 (secure 0 0 0) 0x7f6190046760 con 0x7f619c05c120 2022-07-20T03:15:39.425 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:39.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f617c053ad0 msgr2=0x7f617c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f617c053ad0 0x7f617c055f90 secure :-1 s=READY pgs=510 cs=0 l=1 rev1=1 rx=0x7f619400d000 tx=0x7f6194007f10).stop 2022-07-20T03:15:39.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 msgr2=0x7f619c156590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c156590 secure :-1 s=READY pgs=1045 cs=0 l=1 rev1=1 rx=0x7f619000c000 tx=0x7f6190006df0).stop 2022-07-20T03:15:39.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.420+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 shutdown_connections 2022-07-20T03:15:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f617c053ad0 0x7f617c055f90 unknown :-1 s=CLOSED pgs=510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f619c0b5b50 0x7f619c0acd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f619c0b0480 0x7f619c064320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 --2- 172.21.15.61:0/4133730967 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f619c05c120 0x7f619c156590 unknown :-1 s=CLOSED pgs=1045 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 >> 172.21.15.61:0/4133730967 conn(0x7f619c0635e0 msgr2=0x7f619c144830 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 shutdown_connections 2022-07-20T03:15:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.424+0000 7f61a3e4e700 1 -- 172.21.15.61:0/4133730967 wait complete. 2022-07-20T03:15:39.438 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:39.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 Processor -- start 2022-07-20T03:15:39.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 -- start start 2022-07-20T03:15:39.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f129c05c960 0x7f129c1465e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f129c061fd0 0x7f129c142080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f129c05cd40 con 0x7f129c0b2230 2022-07-20T03:15:39.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c060400 con 0x7f129c05c960 2022-07-20T03:15:39.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.548+0000 7f129bfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f129c061440 con 0x7f129c061fd0 2022-07-20T03:15:39.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f129c05c960 0x7f129c1465e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f129c05c960 0x7f129c1465e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48926/0 (socket says 172.21.15.61:48926) 2022-07-20T03:15:39.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 -- 172.21.15.61:0/3988724931 learned_addr learned my addr 172.21.15.61:0/3988724931 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:39.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129b7fe700 1 --2- 172.21.15.61:0/3988724931 >> v2:172.21.15.117:3300/0 conn(0x7f129c061fd0 0x7f129c142080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 -- 172.21.15.61:0/3988724931 >> v2:172.21.15.117:3300/0 conn(0x7f129c061fd0 msgr2=0x7f129c142080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:39.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 --2- 172.21.15.61:0/3988724931 >> v2:172.21.15.117:3300/0 conn(0x7f129c061fd0 0x7f129c142080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 -- 172.21.15.61:0/3988724931 >> v1:172.21.15.61:6789/0 conn(0x7f129c0b2230 legacy=0x7f129c0618c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 -- 172.21.15.61:0/3988724931 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f129c060680 con 0x7f129c05c960 2022-07-20T03:15:39.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129affd700 1 --2- 172.21.15.61:0/3988724931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c05c960 0x7f129c1465e0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f128c002f70 tx=0x7f128c006eb0).ready entity=mon.0 client_cookie=7cf2ba24d36c861b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129a7fc700 1 -- 172.21.15.61:0/3988724931 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f128c00d610 con 0x7f129c05c960 2022-07-20T03:15:39.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129a7fc700 1 -- 172.21.15.61:0/3988724931 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f128c000e90 con 0x7f129c05c960 2022-07-20T03:15:39.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129a7fc700 1 -- 172.21.15.61:0/3988724931 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f128c00d9a0 con 0x7f129c05c960 2022-07-20T03:15:39.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 -- 172.21.15.61:0/3988724931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c05c960 msgr2=0x7f129c1465e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 --2- 172.21.15.61:0/3988724931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c05c960 0x7f129c1465e0 secure :-1 s=READY pgs=1047 cs=0 l=1 rev1=1 rx=0x7f128c002f70 tx=0x7f128c006eb0).stop 2022-07-20T03:15:39.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 -- 172.21.15.61:0/3988724931 shutdown_connections 2022-07-20T03:15:39.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 --2- 172.21.15.61:0/3988724931 >> v2:172.21.15.117:3300/0 conn(0x7f129c061fd0 0x7f129c142080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 --2- 172.21.15.61:0/3988724931 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c05c960 0x7f129c1465e0 unknown :-1 s=CLOSED pgs=1047 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 -- 172.21.15.61:0/3988724931 >> 172.21.15.61:0/3988724931 conn(0x7f129c0b61f0 msgr2=0x7f129c0b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 -- 172.21.15.61:0/3988724931 shutdown_connections 2022-07-20T03:15:39.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.552+0000 7f129bfff700 1 -- 172.21.15.61:0/3988724931 wait complete. 2022-07-20T03:15:39.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 Processor -- start 2022-07-20T03:15:39.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 -- start start 2022-07-20T03:15:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f129c05c960 0x7f129c066210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f129c061fd0 0x7f129c064b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 0x7f129c065000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c067310 con 0x7f129c0b2230 2022-07-20T03:15:39.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f129c1462a0 con 0x7f129c05c960 2022-07-20T03:15:39.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f129c067070 con 0x7f129c061fd0 2022-07-20T03:15:39.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 0x7f129c065000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f129c05c960 0x7f129c066210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 0x7f129c065000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48936/0 (socket says 172.21.15.61:48936) 2022-07-20T03:15:39.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 -- 172.21.15.61:0/1268677733 learned_addr learned my addr 172.21.15.61:0/1268677733 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:39.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f12927fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f129c061fd0 0x7f129c064b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 -- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f129c061fd0 msgr2=0x7f129c064b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f129c061fd0 0x7f129c064b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 -- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f129c05c960 msgr2=0x7f129c066210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f129c05c960 0x7f129c066210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 -- 172.21.15.61:0/1268677733 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f128c00b1e0 con 0x7f129c0b2230 2022-07-20T03:15:39.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129affd700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f129c05c960 0x7f129c066210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:39.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129b7fe700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 0x7f129c065000 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f129400a000 tx=0x7f1294008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1294013620 con 0x7f129c0b2230 2022-07-20T03:15:39.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1294013780 con 0x7f129c0b2230 2022-07-20T03:15:39.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f129c066700 con 0x7f129c0b2230 2022-07-20T03:15:39.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1294013ae0 con 0x7f129c0b2230 2022-07-20T03:15:39.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.556+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f129c146700 con 0x7f129c0b2230 2022-07-20T03:15:39.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.560+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f129c0b2840 con 0x7f129c0b2230 2022-07-20T03:15:39.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.560+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1294013c40 con 0x7f129c0b2230 2022-07-20T03:15:39.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.560+0000 7f1298ff9700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12800539b0 0x7f1280055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.560+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f129406bf20 con 0x7f129c0b2230 2022-07-20T03:15:39.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.560+0000 7f129affd700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12800539b0 0x7f1280055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.560+0000 7f129affd700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12800539b0 0x7f1280055e70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f128c01c000 tx=0x7f128c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.564+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1294042d10 con 0x7f129c0b2230 2022-07-20T03:15:39.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.692+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f129c0574e0 con 0x7f129c0b2230 2022-07-20T03:15:39.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.692+0000 7f1298ff9700 1 -- 172.21.15.61:0/1268677733 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v101) v1 ==== 92+0+10 (secure 0 0 0) 0x7f1294036020 con 0x7f129c0b2230 2022-07-20T03:15:39.697 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:39.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.692+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12800539b0 msgr2=0x7f1280055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.692+0000 7f129bfff700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12800539b0 0x7f1280055e70 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7f128c01c000 tx=0x7f128c015000).stop 2022-07-20T03:15:39.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.692+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 msgr2=0x7f129c065000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.692+0000 7f129bfff700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 0x7f129c065000 secure :-1 s=READY pgs=1048 cs=0 l=1 rev1=1 rx=0x7f129400a000 tx=0x7f1294008040).stop 2022-07-20T03:15:39.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 shutdown_connections 2022-07-20T03:15:39.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f12800539b0 0x7f1280055e70 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f129c0b2230 0x7f129c065000 unknown :-1 s=CLOSED pgs=1048 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f129c061fd0 0x7f129c064b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 --2- 172.21.15.61:0/1268677733 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f129c05c960 0x7f129c066210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 >> 172.21.15.61:0/1268677733 conn(0x7f129c0b61f0 msgr2=0x7f129c0ab920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 shutdown_connections 2022-07-20T03:15:39.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.696+0000 7f129bfff700 1 -- 172.21.15.61:0/1268677733 wait complete. 2022-07-20T03:15:39.711 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:15:39.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 Processor -- start 2022-07-20T03:15:39.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 -- start start 2022-07-20T03:15:39.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8fc809df40 0x7f8fc8145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc809cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8fc80607e0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8fc8060d20 con 0x7f8fc809df40 2022-07-20T03:15:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fce0dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8060a80 con 0x7f8fc8060ee0 2022-07-20T03:15:39.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8fc809df40 0x7f8fc8145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f8fc80b4d60 0x7f8fc8145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45538/0 (socket says 172.21.15.61:45538) 2022-07-20T03:15:39.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc6ffd700 1 -- 172.21.15.61:0/2308675591 learned_addr learned my addr 172.21.15.61:0/2308675591 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:39.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc7fff700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc809cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc67fc700 1 -- 172.21.15.61:0/2308675591 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1508110217 0 0) 0x7f8fc80607e0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc67fc700 1 -- 172.21.15.61:0/2308675591 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8fa8003260 con 0x7f8fc80b4d60 2022-07-20T03:15:39.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc77fe700 1 -- 172.21.15.61:0/2308675591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 msgr2=0x7f8fc809cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:39.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc77fe700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc809cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc77fe700 1 -- 172.21.15.61:0/2308675591 >> v1:172.21.15.61:6789/0 conn(0x7f8fc80b4d60 legacy=0x7f8fc8145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc77fe700 1 -- 172.21.15.61:0/2308675591 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fc809e320 con 0x7f8fc809df40 2022-07-20T03:15:39.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.820+0000 7f8fc7fff700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc809cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fc77fe700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc809df40 0x7f8fc8145680 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f8fb0002f70 tx=0x7f8fb0006d90).ready entity=mon.0 client_cookie=b2c1d9e85e8dbb42 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fc67fc700 1 -- 172.21.15.61:0/2308675591 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fb000d4a0 con 0x7f8fc809df40 2022-07-20T03:15:39.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fc67fc700 1 -- 172.21.15.61:0/2308675591 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fb0000e90 con 0x7f8fc809df40 2022-07-20T03:15:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fc67fc700 1 -- 172.21.15.61:0/2308675591 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fb000d7f0 con 0x7f8fc809df40 2022-07-20T03:15:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2308675591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc809df40 msgr2=0x7f8fc8145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc809df40 0x7f8fc8145680 secure :-1 s=READY pgs=1050 cs=0 l=1 rev1=1 rx=0x7f8fb0002f70 tx=0x7f8fb0006d90).stop 2022-07-20T03:15:39.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2308675591 shutdown_connections 2022-07-20T03:15:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc809cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2308675591 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc809df40 0x7f8fc8145680 unknown :-1 s=CLOSED pgs=1050 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2308675591 >> 172.21.15.61:0/2308675591 conn(0x7f8fc80b5f30 msgr2=0x7f8fc80b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2308675591 shutdown_connections 2022-07-20T03:15:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2308675591 wait complete. 2022-07-20T03:15:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.824+0000 7f8fce0dc700 1 Processor -- start 2022-07-20T03:15:39.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 -- start start 2022-07-20T03:15:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc80b8a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc809df40 0x7f8fc8155f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 0x7f8fc8152190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8090b80 con 0x7f8fc80b4d60 2022-07-20T03:15:39.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8155820 con 0x7f8fc809df40 2022-07-20T03:15:39.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8fc8155ac0 con 0x7f8fc8060ee0 2022-07-20T03:15:39.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 0x7f8fc8152190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc77fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc80b8a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 0x7f8fc8152190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48950/0 (socket says 172.21.15.61:48950) 2022-07-20T03:15:39.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 -- 172.21.15.61:0/2605102 learned_addr learned my addr 172.21.15.61:0/2605102 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:39.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc6ffd700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc809df40 0x7f8fc8155f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 -- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 msgr2=0x7f8fc80b8a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc80b8a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 -- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc809df40 msgr2=0x7f8fc8155f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc809df40 0x7f8fc8155f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 -- 172.21.15.61:0/2605102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8fb000b120 con 0x7f8fc80b4d60 2022-07-20T03:15:39.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc77fe700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc80b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:39.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc7fff700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 0x7f8fc8152190 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7f8fc0006cc0 tx=0x7f8fc0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc0013620 con 0x7f8fc80b4d60 2022-07-20T03:15:39.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8fc0013780 con 0x7f8fc80b4d60 2022-07-20T03:15:39.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8fc80b01c0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.828+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8fc0013ae0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.832+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8fc8156670 con 0x7f8fc80b4d60 2022-07-20T03:15:39.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.832+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8fc80574e0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.832+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8fc0013c40 con 0x7f8fc80b4d60 2022-07-20T03:15:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.832+0000 7f8fc4ff9700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8fac053ad0 0x7f8fac055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.832+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8fc006bc40 con 0x7f8fc80b4d60 2022-07-20T03:15:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.832+0000 7f8fc77fe700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8fac053ad0 0x7f8fac055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:39.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.836+0000 7f8fc77fe700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8fac053ad0 0x7f8fac055f90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f8fb001a000 tx=0x7f8fb0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:39.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.840+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8fc0042ba0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f8fc809cbe0 con 0x7f8fc80b4d60 2022-07-20T03:15:39.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fc4ff9700 1 -- 172.21.15.61:0/2605102 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+10 (secure 0 0 0) 0x7f8fc000f040 con 0x7f8fc80b4d60 2022-07-20T03:15:39.973 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8fac053ad0 msgr2=0x7f8fac055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8fac053ad0 0x7f8fac055f90 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7f8fb001a000 tx=0x7f8fb0014000).stop 2022-07-20T03:15:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 msgr2=0x7f8fc8152190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 0x7f8fc8152190 secure :-1 s=READY pgs=1051 cs=0 l=1 rev1=1 rx=0x7f8fc0006cc0 tx=0x7f8fc0008040).stop 2022-07-20T03:15:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 shutdown_connections 2022-07-20T03:15:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8fac053ad0 0x7f8fac055f90 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8fc809df40 0x7f8fc8155f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8fc80b4d60 0x7f8fc8152190 unknown :-1 s=CLOSED pgs=1051 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 --2- 172.21.15.61:0/2605102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8fc8060ee0 0x7f8fc80b8a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:39.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 >> 172.21.15.61:0/2605102 conn(0x7f8fc80b5f30 msgr2=0x7f8fc8145450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:39.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 shutdown_connections 2022-07-20T03:15:39.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:39.968+0000 7f8fce0dc700 1 -- 172.21.15.61:0/2605102 wait complete. 2022-07-20T03:15:39.990 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:15:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.096+0000 7fb86f59e700 1 Processor -- start 2022-07-20T03:15:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.096+0000 7fb86f59e700 1 -- start start 2022-07-20T03:15:40.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 0x7fb87005d1c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700b5db0 con 0x7fb8700b59d0 2022-07-20T03:15:40.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86f59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb8700ad800 con 0x7fb870128d70 2022-07-20T03:15:40.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700adaa0 con 0x7fb8700af180 2022-07-20T03:15:40.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86e59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb8700b59d0 0x7fb870128660 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45552/0 (socket says 172.21.15.61:45552) 2022-07-20T03:15:40.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86e59c700 1 -- 172.21.15.61:0/689495128 learned_addr learned my addr 172.21.15.61:0/689495128 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:40.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 0x7fb87005d1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2307910525 0 0) 0x7fb8700b5db0 con 0x7fb8700b59d0 2022-07-20T03:15:40.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb854003260 con 0x7fb8700b59d0 2022-07-20T03:15:40.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869204184 0 0) 0x7fb8700ad800 con 0x7fb870128d70 2022-07-20T03:15:40.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb8700b5db0 con 0x7fb870128d70 2022-07-20T03:15:40.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2855144869 0 0) 0x7fb854003260 con 0x7fb8700b59d0 2022-07-20T03:15:40.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 msgr2=0x7fb87005d1c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:40.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 0x7fb87005d1c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 >> v1:172.21.15.117:6789/0 conn(0x7fb870128d70 legacy=0x7fb87013fc00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8700ad4f0 con 0x7fb8700b59d0 2022-07-20T03:15:40.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fb8680019e0 con 0x7fb8700b59d0 2022-07-20T03:15:40.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 >> v1:172.21.15.61:6789/0 conn(0x7fb8700b59d0 legacy=0x7fb870128660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 0x7fb854003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb854003e20 0x7fb854004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8540049f0 0x7fb854004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8700adaa0 con 0x7fb854003e20 2022-07-20T03:15:40.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb85400da10 con 0x7fb8540049f0 2022-07-20T03:15:40.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb85400dbd0 con 0x7fb8700af180 2022-07-20T03:15:40.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb854003e20 0x7fb854004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86e59c700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 0x7fb854003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86ed9d700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8540049f0 0x7fb854004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 -- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 msgr2=0x7fb854003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:40.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8700af180 0x7fb854003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 -- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8540049f0 msgr2=0x7fb854004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:40.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8540049f0 0x7fb854004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 -- 172.21.15.61:0/689495128 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854003260 con 0x7fb854003e20 2022-07-20T03:15:40.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86ed9d700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8540049f0 0x7fb854004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:40.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb854003e20 0x7fb854004500 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fb86400c000 tx=0x7fb864006df0).ready entity=mon.0 client_cookie=57184ecf9d5c566a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:40.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86400e830 con 0x7fb854003e20 2022-07-20T03:15:40.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86400e990 con 0x7fb854003e20 2022-07-20T03:15:40.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.100+0000 7fb86d59a700 1 -- 172.21.15.61:0/689495128 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86400ecf0 con 0x7fb854003e20 2022-07-20T03:15:40.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/689495128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb854003e20 msgr2=0x7fb854004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 --2- 172.21.15.61:0/689495128 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb854003e20 0x7fb854004500 secure :-1 s=READY pgs=1053 cs=0 l=1 rev1=1 rx=0x7fb86400c000 tx=0x7fb864006df0).stop 2022-07-20T03:15:40.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86ed9d700 1 -- 172.21.15.61:0/689495128 reap_dead start 2022-07-20T03:15:40.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/689495128 shutdown_connections 2022-07-20T03:15:40.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/689495128 >> 172.21.15.61:0/689495128 conn(0x7fb8700634c0 msgr2=0x7fb8700b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:40.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/689495128 shutdown_connections 2022-07-20T03:15:40.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/689495128 wait complete. 2022-07-20T03:15:40.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 Processor -- start 2022-07-20T03:15:40.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- start start 2022-07-20T03:15:40.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb854003e20 0x7fb870125240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 0x7fb870143f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb870144460 0x7fb8701448e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8701430b0 con 0x7fb870140bc0 2022-07-20T03:15:40.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb870127d80 con 0x7fb854003e20 2022-07-20T03:15:40.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb870142b10 con 0x7fb870144460 2022-07-20T03:15:40.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 0x7fb870143f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 0x7fb870143f70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48966/0 (socket says 172.21.15.61:48966) 2022-07-20T03:15:40.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 -- 172.21.15.61:0/1488786849 learned_addr learned my addr 172.21.15.61:0/1488786849 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:40.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86ed9d700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb870144460 0x7fb8701448e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86e59c700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb854003e20 0x7fb870125240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 -- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb870144460 msgr2=0x7fb8701448e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb870144460 0x7fb8701448e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 -- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb854003e20 msgr2=0x7fb870125240 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb854003e20 0x7fb870125240 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 -- 172.21.15.61:0/1488786849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8701418d0 con 0x7fb870140bc0 2022-07-20T03:15:40.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86e59c700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb854003e20 0x7fb870125240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:40.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86dd9b700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 0x7fb870143f70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fb86400c000 tx=0x7fb864019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:40.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86401b820 con 0x7fb870140bc0 2022-07-20T03:15:40.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb86401b980 con 0x7fb870140bc0 2022-07-20T03:15:40.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb870141b70 con 0x7fb870140bc0 2022-07-20T03:15:40.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb86401bce0 con 0x7fb870140bc0 2022-07-20T03:15:40.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.104+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8700639a0 con 0x7fb870140bc0 2022-07-20T03:15:40.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.108+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8700574e0 con 0x7fb870140bc0 2022-07-20T03:15:40.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.108+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb86402d070 con 0x7fb870140bc0 2022-07-20T03:15:40.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.108+0000 7fb85f7fe700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb84c053af0 0x7fb84c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.108+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb86406d1d0 con 0x7fb870140bc0 2022-07-20T03:15:40.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.108+0000 7fb86e59c700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb84c053af0 0x7fb84c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.112+0000 7fb86e59c700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb84c053af0 0x7fb84c055fb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fb868005000 tx=0x7fb868009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:40.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.112+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb864044090 con 0x7fb870140bc0 2022-07-20T03:15:40.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.240+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fb870141d00 con 0x7fb870140bc0 2022-07-20T03:15:40.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.240+0000 7fb85f7fe700 1 -- 172.21.15.61:0/1488786849 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+10 (secure 0 0 0) 0x7fb8640476b0 con 0x7fb870140bc0 2022-07-20T03:15:40.248 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:40.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.240+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb84c053af0 msgr2=0x7fb84c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.240+0000 7fb86f59e700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb84c053af0 0x7fb84c055fb0 secure :-1 s=READY pgs=513 cs=0 l=1 rev1=1 rx=0x7fb868005000 tx=0x7fb868009040).stop 2022-07-20T03:15:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.240+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 msgr2=0x7fb870143f70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.240+0000 7fb86f59e700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 0x7fb870143f70 secure :-1 s=READY pgs=1054 cs=0 l=1 rev1=1 rx=0x7fb86400c000 tx=0x7fb864019000).stop 2022-07-20T03:15:40.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 shutdown_connections 2022-07-20T03:15:40.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb84c053af0 0x7fb84c055fb0 unknown :-1 s=CLOSED pgs=513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb870144460 0x7fb8701448e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb870140bc0 0x7fb870143f70 unknown :-1 s=CLOSED pgs=1054 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 --2- 172.21.15.61:0/1488786849 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb854003e20 0x7fb870125240 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 >> 172.21.15.61:0/1488786849 conn(0x7fb8700634c0 msgr2=0x7fb8700b2e80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:40.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 shutdown_connections 2022-07-20T03:15:40.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.244+0000 7fb86f59e700 1 -- 172.21.15.61:0/1488786849 wait complete. 2022-07-20T03:15:40.261 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:15:40.262 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:40.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 Processor -- start 2022-07-20T03:15:40.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 -- start start 2022-07-20T03:15:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07e80b1ef0 0x7f07e8142f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80ac3a0 0x7f07e8143470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f07e8060980 con 0x7f07e8143960 2022-07-20T03:15:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f07e80b22d0 con 0x7f07e80b1ef0 2022-07-20T03:15:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07ec827700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8098810 con 0x7f07e80ac3a0 2022-07-20T03:15:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07e80b1ef0 0x7f07e8142f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f07e80b1ef0 0x7f07e8142f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48972/0 (socket says 172.21.15.61:48972) 2022-07-20T03:15:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 -- 172.21.15.61:0/2020454207 learned_addr learned my addr 172.21.15.61:0/2020454207 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e6ffd700 1 --2- 172.21.15.61:0/2020454207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80ac3a0 0x7f07e8143470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 -- 172.21.15.61:0/2020454207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80ac3a0 msgr2=0x7f07e8143470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:40.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 --2- 172.21.15.61:0/2020454207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80ac3a0 0x7f07e8143470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 -- 172.21.15.61:0/2020454207 >> v1:172.21.15.117:6789/0 conn(0x7f07e8143960 legacy=0x7f07e80ab180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 -- 172.21.15.61:0/2020454207 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07e8060bb0 con 0x7f07e80b1ef0 2022-07-20T03:15:40.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e77fe700 1 --2- 172.21.15.61:0/2020454207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80b1ef0 0x7f07e8142f80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f07d8002f70 tx=0x7f07d8006ee0).ready entity=mon.0 client_cookie=1a7bf2f69528bb92 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e67fc700 1 -- 172.21.15.61:0/2020454207 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d800bc80 con 0x7f07e80b1ef0 2022-07-20T03:15:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e67fc700 1 -- 172.21.15.61:0/2020454207 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d8000e90 con 0x7f07e80b1ef0 2022-07-20T03:15:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.364+0000 7f07e67fc700 1 -- 172.21.15.61:0/2020454207 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d800de90 con 0x7f07e80b1ef0 2022-07-20T03:15:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 -- 172.21.15.61:0/2020454207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80b1ef0 msgr2=0x7f07e8142f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 --2- 172.21.15.61:0/2020454207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80b1ef0 0x7f07e8142f80 secure :-1 s=READY pgs=1055 cs=0 l=1 rev1=1 rx=0x7f07d8002f70 tx=0x7f07d8006ee0).stop 2022-07-20T03:15:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 -- 172.21.15.61:0/2020454207 shutdown_connections 2022-07-20T03:15:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 --2- 172.21.15.61:0/2020454207 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80ac3a0 0x7f07e8143470 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 --2- 172.21.15.61:0/2020454207 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80b1ef0 0x7f07e8142f80 unknown :-1 s=CLOSED pgs=1055 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 -- 172.21.15.61:0/2020454207 >> 172.21.15.61:0/2020454207 conn(0x7f07e80b2ca0 msgr2=0x7f07e80b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 -- 172.21.15.61:0/2020454207 shutdown_connections 2022-07-20T03:15:40.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 -- 172.21.15.61:0/2020454207 wait complete. 2022-07-20T03:15:40.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 Processor -- start 2022-07-20T03:15:40.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.368+0000 7f07ec827700 1 -- start start 2022-07-20T03:15:40.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 0x7f07e809b290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80b1ef0 0x7f07e809b780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07e8143960 0x7f07e8097c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e8065680 con 0x7f07e80ac3a0 2022-07-20T03:15:40.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f07e809a520 con 0x7f07e8143960 2022-07-20T03:15:40.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f07e809a7c0 con 0x7f07e80b1ef0 2022-07-20T03:15:40.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 0x7f07e809b290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 0x7f07e809b290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48978/0 (socket says 172.21.15.61:48978) 2022-07-20T03:15:40.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 -- 172.21.15.61:0/3327170977 learned_addr learned my addr 172.21.15.61:0/3327170977 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:40.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e6ffd700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80b1ef0 0x7f07e809b780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e7fff700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07e8143960 0x7f07e8097c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 -- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80b1ef0 msgr2=0x7f07e809b780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80b1ef0 0x7f07e809b780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 -- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07e8143960 msgr2=0x7f07e8097c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07e8143960 0x7f07e8097c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 -- 172.21.15.61:0/3327170977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f07d800d410 con 0x7f07e80ac3a0 2022-07-20T03:15:40.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e77fe700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 0x7f07e809b290 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f07d8002f70 tx=0x7f07e8061f90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d801a840 con 0x7f07e80ac3a0 2022-07-20T03:15:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f07d801a9a0 con 0x7f07e80ac3a0 2022-07-20T03:15:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f07e8065130 con 0x7f07e80ac3a0 2022-07-20T03:15:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f07d801ad00 con 0x7f07e80ac3a0 2022-07-20T03:15:40.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f07e80999a0 con 0x7f07e80ac3a0 2022-07-20T03:15:40.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.372+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f07e80574e0 con 0x7f07e80ac3a0 2022-07-20T03:15:40.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.376+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f07d8018400 con 0x7f07e80ac3a0 2022-07-20T03:15:40.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.376+0000 7f07e4ff9700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07c80539b0 0x7f07c8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:40.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.376+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f07d806dfa0 con 0x7f07e80ac3a0 2022-07-20T03:15:40.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.376+0000 7f07e6ffd700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07c80539b0 0x7f07c8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:40.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.376+0000 7f07e6ffd700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07c80539b0 0x7f07c8055e70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f07dc00c000 tx=0x7f07dc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:40.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.380+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f07d8044ea0 con 0x7f07e80ac3a0 2022-07-20T03:15:40.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.540+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f07e80447e0 con 0x7f07e80ac3a0 2022-07-20T03:15:40.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07e4ff9700 1 -- 172.21.15.61:0/3327170977 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f07d801ae60 con 0x7f07e80ac3a0 2022-07-20T03:15:40.565 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:40.566 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:35.686626+0000","modified":"2022-07-20T03:15:36.689415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7335":{"gid":7335,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2834789489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2834789489},{"type":"v1","addr":"172.21.15.61:6833","nonce":2834789489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T03:15:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07c80539b0 msgr2=0x7f07c8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07c80539b0 0x7f07c8055e70 secure :-1 s=READY pgs=514 cs=0 l=1 rev1=1 rx=0x7f07dc00c000 tx=0x7f07dc008040).stop 2022-07-20T03:15:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 msgr2=0x7f07e809b290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 0x7f07e809b290 secure :-1 s=READY pgs=1056 cs=0 l=1 rev1=1 rx=0x7f07d8002f70 tx=0x7f07e8061f90).stop 2022-07-20T03:15:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 shutdown_connections 2022-07-20T03:15:40.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f07c80539b0 0x7f07c8055e70 unknown :-1 s=CLOSED pgs=514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f07e8143960 0x7f07e8097c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f07e80b1ef0 0x7f07e809b780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 --2- 172.21.15.61:0/3327170977 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f07e80ac3a0 0x7f07e809b290 unknown :-1 s=CLOSED pgs=1056 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:40.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 >> 172.21.15.61:0/3327170977 conn(0x7f07e80b2ca0 msgr2=0x7f07e813a960 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:40.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 shutdown_connections 2022-07-20T03:15:40.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:40.544+0000 7f07ec827700 1 -- 172.21.15.61:0/3327170977 wait complete. 2022-07-20T03:15:40.571 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-20T03:15:40.774 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:35.686626+0000', 'modified': '2022-07-20T03:15:36.689415+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7335}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7335': {'gid': 7335, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2834789489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2834789489}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2834789489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:15:40.774 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:15:40.775 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:15:40.896 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7335,"want_state":"up:active","state":"up:active","rank_uptime":5.1961199689999997,"mdsmap_epoch":56,"osdmap_epoch":101,"osdmap_epoch_barrier":100,"uptime":9.6672966769999995} 2022-07-20T03:15:40.897 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7335, "mdsmap_epoch": 56, "osdmap_epoch": 101, "osdmap_epoch_barrier": 100, "rank_uptime": 5.196119969, "state": "up:active", "uptime": 9.667296677, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:15:40.898 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:15:40.899 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:15:40.899 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:15:40.900 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:40.900 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:40.910 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:15:40.976 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:40.977 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:40.995 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:40.997 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:40.997 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:40.998 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:15:41.048 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:15:41.049 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:41.052 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:15:41.072 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:41.073 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:41.087 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:41.088 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:41.090 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:41.091 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:15:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:15:41.121+0000 7f834b14b080 -1 init, newargv = 0x5566c5f532f0 newargc=9 2022-07-20T03:15:41.125 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23246]: starting ceph client 2022-07-20T03:15:41.133 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23246]: starting fuse 2022-07-20T03:15:42.237 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:15:42.264 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:42.265 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:42.284 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:42.286 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:42.287 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:42.288 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:15:42.337 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:15:42.337 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:15:42.338 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:42.338 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:15:42.339 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:15:42.339 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:15:42.339 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:15:42.340 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:15:42.340 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:42.341 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:15:42.341 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:15:42.346 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:15:42.347 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:42.347 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:15:42.348 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:15:42.348 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:15:42.349 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:15:42.349 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:42.349 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:15:42.354 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:15:42.354 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:15:42.355 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:15:42.355 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:15:42.356 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:15:42.356 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:15:42.357 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:15:42.413 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23246.asok 2022-07-20T03:15:42.421 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23246.asok 2022-07-20T03:15:42.422 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:42.423 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23246.asok status 2022-07-20T03:15:42.565 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:15:42.566 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:15:42.567 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:15:42.567 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:15:42.568 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:15:42.568 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:15:42.569 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:15:42.569 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23246", 2022-07-20T03:15:42.569 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:15:42.570 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:42.570 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:15:42.571 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:15:42.571 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 7500, 2022-07-20T03:15:42.572 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:15:42.572 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:15:42.573 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:15:42.573 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 7500 2022-07-20T03:15:42.573 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:42.575 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:15:42.576 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:15:42.576 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:15:42.577 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 4123695182 2022-07-20T03:15:42.577 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:15:42.578 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:42.578 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:15:42.579 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:15:42.579 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:15:42.580 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 4123695182 2022-07-20T03:15:42.580 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:15:42.580 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.7500 172.21.15.124:0/4123695182", 2022-07-20T03:15:42.581 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/4123695182", 2022-07-20T03:15:42.581 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:15:42.598 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 56, 2022-07-20T03:15:42.599 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 101, 2022-07-20T03:15:42.599 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:15:42.600 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:15:42.600 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:15:42.600 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:15:42.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:42.604 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:42.605 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:42.624 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:15:42.625 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:42.626 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:42.627 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:42.700 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:42.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 Processor -- start 2022-07-20T03:15:42.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- start start 2022-07-20T03:15:42.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7efc13c790 0x7f7efc0b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:42.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7efc13cb70 0x7f7efc143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:42.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7efc142ba0 con 0x7f7efc0b3ca0 2022-07-20T03:15:42.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc1429b0 con 0x7f7efc13c790 2022-07-20T03:15:42.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7efc146330 con 0x7f7efc13cb70 2022-07-20T03:15:42.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7efc13c790 0x7f7efc0b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:42.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7efc13c790 0x7f7efc0b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48986/0 (socket says 172.21.15.61:48986) 2022-07-20T03:15:42.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 -- 172.21.15.61:0/324481525 learned_addr learned my addr 172.21.15.61:0/324481525 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:42.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efb7fe700 1 --2- 172.21.15.61:0/324481525 >> v2:172.21.15.117:3300/0 conn(0x7f7efc13cb70 0x7f7efc143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:42.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 -- 172.21.15.61:0/324481525 >> v2:172.21.15.117:3300/0 conn(0x7f7efc13cb70 msgr2=0x7f7efc143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:42.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 --2- 172.21.15.61:0/324481525 >> v2:172.21.15.117:3300/0 conn(0x7f7efc13cb70 0x7f7efc143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 -- 172.21.15.61:0/324481525 >> v1:172.21.15.117:6789/0 conn(0x7f7efc0b3ca0 legacy=0x7f7efc0b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:42.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 -- 172.21.15.61:0/324481525 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7efc0b4c00 con 0x7f7efc13c790 2022-07-20T03:15:42.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efaffd700 1 --2- 172.21.15.61:0/324481525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13c790 0x7f7efc0b4710 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7f7ef40032f0 tx=0x7f7ef4009290).ready entity=mon.0 client_cookie=b26e39f17b861886 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:42.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7ef9ffb700 1 -- 172.21.15.61:0/324481525 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ef400b870 con 0x7f7efc13c790 2022-07-20T03:15:42.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7ef9ffb700 1 -- 172.21.15.61:0/324481525 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7ef400b9d0 con 0x7f7efc13c790 2022-07-20T03:15:42.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7ef9ffb700 1 -- 172.21.15.61:0/324481525 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7ef400bb40 con 0x7f7efc13c790 2022-07-20T03:15:42.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- 172.21.15.61:0/324481525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13c790 msgr2=0x7f7efc0b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:42.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 --2- 172.21.15.61:0/324481525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13c790 0x7f7efc0b4710 secure :-1 s=READY pgs=1059 cs=0 l=1 rev1=1 rx=0x7f7ef40032f0 tx=0x7f7ef4009290).stop 2022-07-20T03:15:42.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- 172.21.15.61:0/324481525 shutdown_connections 2022-07-20T03:15:42.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 --2- 172.21.15.61:0/324481525 >> v2:172.21.15.117:3300/0 conn(0x7f7efc13cb70 0x7f7efc143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 --2- 172.21.15.61:0/324481525 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13c790 0x7f7efc0b4710 unknown :-1 s=CLOSED pgs=1059 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- 172.21.15.61:0/324481525 >> 172.21.15.61:0/324481525 conn(0x7f7efc0678b0 msgr2=0x7f7efc0b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:42.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- 172.21.15.61:0/324481525 shutdown_connections 2022-07-20T03:15:42.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.820+0000 7f7efbfff700 1 -- 172.21.15.61:0/324481525 wait complete. 2022-07-20T03:15:42.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 Processor -- start 2022-07-20T03:15:42.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 -- start start 2022-07-20T03:15:42.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7efc0b3ca0 0x7f7efc1555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:42.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7efc13c790 0x7f7efc1419b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:42.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 0x7f7efc0b0410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:42.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc13e1f0 con 0x7f7efc13cb70 2022-07-20T03:15:42.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc13ffa0 con 0x7f7efc13c790 2022-07-20T03:15:42.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efbfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7efc0affb0 con 0x7f7efc0b3ca0 2022-07-20T03:15:42.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 0x7f7efc0b0410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:42.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 0x7f7efc0b0410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:48996/0 (socket says 172.21.15.61:48996) 2022-07-20T03:15:42.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 -- 172.21.15.61:0/3196218912 learned_addr learned my addr 172.21.15.61:0/3196218912 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:42.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efa7fc700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7efc13c790 0x7f7efc1419b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:42.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efaffd700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7efc0b3ca0 0x7f7efc1555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:42.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 -- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7efc0b3ca0 msgr2=0x7f7efc1555d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:42.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7efc0b3ca0 0x7f7efc1555d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 -- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7efc13c790 msgr2=0x7f7efc1419b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:42.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7efc13c790 0x7f7efc1419b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 -- 172.21.15.61:0/3196218912 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ef4009530 con 0x7f7efc13cb70 2022-07-20T03:15:42.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7efb7fe700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 0x7f7efc0b0410 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f7eec00c000 tx=0x7f7eec006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:42.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.824+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eec011750 con 0x7f7efc13cb70 2022-07-20T03:15:42.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eec0118b0 con 0x7f7efc13cb70 2022-07-20T03:15:42.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7efc1401d0 con 0x7f7efc13cb70 2022-07-20T03:15:42.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eec011c10 con 0x7f7efc13cb70 2022-07-20T03:15:42.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7efc13f490 con 0x7f7efc13cb70 2022-07-20T03:15:42.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7efc0574e0 con 0x7f7efc13cb70 2022-07-20T03:15:42.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7eec011d70 con 0x7f7efc13cb70 2022-07-20T03:15:42.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7ef3fff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ed4053940 0x7f7ed4055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:42.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7eec06c0d0 con 0x7f7efc13cb70 2022-07-20T03:15:42.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7efaffd700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ed4053940 0x7f7ed4055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:42.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.828+0000 7f7efaffd700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ed4053940 0x7f7ed4055e00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f7ef401b000 tx=0x7f7ef4016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:42.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.836+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7eec043030 con 0x7f7efc13cb70 2022-07-20T03:15:42.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.960+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7efc0b0140 con 0x7f7efc13cb70 2022-07-20T03:15:42.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7ef3fff700 1 -- 172.21.15.61:0/3196218912 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v101) v1 ==== 74+0+16092 (secure 0 0 0) 0x7f7eec046650 con 0x7f7efc13cb70 2022-07-20T03:15:42.972 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:42.972 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":101,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:36.690047+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":19,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:34.173179+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"100","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":20,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:34.697342+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"101","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:42.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ed4053940 msgr2=0x7f7ed4055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:42.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ed4053940 0x7f7ed4055e00 secure :-1 s=READY pgs=515 cs=0 l=1 rev1=1 rx=0x7f7ef401b000 tx=0x7f7ef4016000).stop 2022-07-20T03:15:42.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 msgr2=0x7f7efc0b0410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:42.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 0x7f7efc0b0410 secure :-1 s=READY pgs=1060 cs=0 l=1 rev1=1 rx=0x7f7eec00c000 tx=0x7f7eec006df0).stop 2022-07-20T03:15:42.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 shutdown_connections 2022-07-20T03:15:42.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ed4053940 0x7f7ed4055e00 unknown :-1 s=CLOSED pgs=515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7efc13c790 0x7f7efc1419b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7efc13cb70 0x7f7efc0b0410 unknown :-1 s=CLOSED pgs=1060 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 --2- 172.21.15.61:0/3196218912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7efc0b3ca0 0x7f7efc1555d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:42.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 >> 172.21.15.61:0/3196218912 conn(0x7f7efc0678b0 msgr2=0x7f7efc0b7200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:42.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 shutdown_connections 2022-07-20T03:15:42.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:42.964+0000 7f7efbfff700 1 -- 172.21.15.61:0/3196218912 wait complete. 2022-07-20T03:15:42.989 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 19, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:15:34.173179+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '100', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 20, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:15:34.697342+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '101', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:15:42.990 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:43.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 Processor -- start 2022-07-20T03:15:43.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 -- start start 2022-07-20T03:15:43.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 0x7efd40142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7efd40142f40 0x7efd401419a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd400ba180 con 0x7efd400b9da0 2022-07-20T03:15:43.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd400ad5b0 con 0x7efd40142f40 2022-07-20T03:15:43.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd45fec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd401461b0 con 0x7efd400b3cf0 2022-07-20T03:15:43.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd44fea700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efd400b9da0 0x7efd401468f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45598/0 (socket says 172.21.15.61:45598) 2022-07-20T03:15:43.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd44fea700 1 -- 172.21.15.61:0/1425346460 learned_addr learned my addr 172.21.15.61:0/1425346460 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:43.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd457eb700 1 --2- 172.21.15.61:0/1425346460 >> v2:172.21.15.117:3300/0 conn(0x7efd40142f40 0x7efd401419a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd37fff700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 0x7efd40142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 161413037 0 0) 0x7efd400ba180 con 0x7efd400b9da0 2022-07-20T03:15:43.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.096+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efd2c003260 con 0x7efd400b9da0 2022-07-20T03:15:43.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2709658611 0 0) 0x7efd2c003260 con 0x7efd400b9da0 2022-07-20T03:15:43.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 msgr2=0x7efd40142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 0x7efd40142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 >> v2:172.21.15.117:3300/0 conn(0x7efd40142f40 msgr2=0x7efd401419a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 --2- 172.21.15.61:0/1425346460 >> v2:172.21.15.117:3300/0 conn(0x7efd40142f40 0x7efd401419a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40146450 con 0x7efd400b9da0 2022-07-20T03:15:43.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7efd3c001660 con 0x7efd400b9da0 2022-07-20T03:15:43.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 >> v1:172.21.15.61:6789/0 conn(0x7efd400b9da0 legacy=0x7efd401468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd2c003750 0x7efd2c003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 0x7efd2c004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd2c0049f0 0x7efd2c004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd400ad5b0 con 0x7efd2c003750 2022-07-20T03:15:43.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd401461b0 con 0x7efd2c0049f0 2022-07-20T03:15:43.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd2c00b750 con 0x7efd400b3cf0 2022-07-20T03:15:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd2c003750 0x7efd2c003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd37fff700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 0x7efd2c004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd457eb700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd2c0049f0 0x7efd2c004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 -- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 msgr2=0x7efd2c004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b3cf0 0x7efd2c004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 -- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd2c0049f0 msgr2=0x7efd2c004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd2c0049f0 0x7efd2c004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 -- 172.21.15.61:0/1425346460 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd2c003260 con 0x7efd2c003750 2022-07-20T03:15:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd457eb700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd2c0049f0 0x7efd2c004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd44fea700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd2c003750 0x7efd2c003b70 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7efd3c001370 tx=0x7efd4005d250).ready entity=mon.0 client_cookie=8915365efaf6ac9e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd3c0070d0 con 0x7efd2c003750 2022-07-20T03:15:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3c007230 con 0x7efd2c003750 2022-07-20T03:15:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd377fe700 1 -- 172.21.15.61:0/1425346460 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd3c00c8f0 con 0x7efd2c003750 2022-07-20T03:15:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- 172.21.15.61:0/1425346460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd2c003750 msgr2=0x7efd2c003b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 --2- 172.21.15.61:0/1425346460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd2c003750 0x7efd2c003b70 secure :-1 s=READY pgs=1062 cs=0 l=1 rev1=1 rx=0x7efd3c001370 tx=0x7efd4005d250).stop 2022-07-20T03:15:43.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd457eb700 1 -- 172.21.15.61:0/1425346460 reap_dead start 2022-07-20T03:15:43.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- 172.21.15.61:0/1425346460 shutdown_connections 2022-07-20T03:15:43.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- 172.21.15.61:0/1425346460 >> 172.21.15.61:0/1425346460 conn(0x7efd400678c0 msgr2=0x7efd400b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- 172.21.15.61:0/1425346460 shutdown_connections 2022-07-20T03:15:43.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- 172.21.15.61:0/1425346460 wait complete. 2022-07-20T03:15:43.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 Processor -- start 2022-07-20T03:15:43.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- start start 2022-07-20T03:15:43.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b11f0 0x7efd400b15d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 0x7efd40115180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd40115670 0x7efd40112aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd40141760 con 0x7efd400b25d0 2022-07-20T03:15:43.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd401146d0 con 0x7efd40115670 2022-07-20T03:15:43.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.100+0000 7efd45fec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd401529b0 con 0x7efd400b11f0 2022-07-20T03:15:43.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 0x7efd40115180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 0x7efd40115180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49016/0 (socket says 172.21.15.61:49016) 2022-07-20T03:15:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 -- 172.21.15.61:0/3943191221 learned_addr learned my addr 172.21.15.61:0/3943191221 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd457eb700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd40115670 0x7efd40112aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd44fea700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b11f0 0x7efd400b15d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 -- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b11f0 msgr2=0x7efd400b15d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b11f0 0x7efd400b15d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 -- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd40115670 msgr2=0x7efd40112aa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd40115670 0x7efd40112aa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 -- 172.21.15.61:0/3943191221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd40115de0 con 0x7efd400b25d0 2022-07-20T03:15:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd44fea700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b11f0 0x7efd400b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:43.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd37fff700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 0x7efd40115180 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7efd3000b000 tx=0x7efd30006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd3000f8e0 con 0x7efd400b25d0 2022-07-20T03:15:43.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd3000fa40 con 0x7efd400b25d0 2022-07-20T03:15:43.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd40115b40 con 0x7efd400b25d0 2022-07-20T03:15:43.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd3000fda0 con 0x7efd400b25d0 2022-07-20T03:15:43.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd40112540 con 0x7efd400b25d0 2022-07-20T03:15:43.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd400667f0 con 0x7efd400b25d0 2022-07-20T03:15:43.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efd3001e460 con 0x7efd400b25d0 2022-07-20T03:15:43.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd35ffb700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efd20053940 0x7efd20055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efd3006ca90 con 0x7efd400b25d0 2022-07-20T03:15:43.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.104+0000 7efd44fea700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efd20053940 0x7efd20055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.108+0000 7efd44fea700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efd20053940 0x7efd20055e00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7efd3c006000 tx=0x7efd3c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.112+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd300438f0 con 0x7efd400b25d0 2022-07-20T03:15:43.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.236+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7efd400574e0 con 0x7efd400b25d0 2022-07-20T03:15:43.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.236+0000 7efd35ffb700 1 -- 172.21.15.61:0/3943191221 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v101) v1 ==== 110+0+10 (secure 0 0 0) 0x7efd30046f10 con 0x7efd400b25d0 2022-07-20T03:15:43.245 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:43.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efd20053940 msgr2=0x7efd20055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efd20053940 0x7efd20055e00 secure :-1 s=READY pgs=516 cs=0 l=1 rev1=1 rx=0x7efd3c006000 tx=0x7efd3c015000).stop 2022-07-20T03:15:43.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 msgr2=0x7efd40115180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 0x7efd40115180 secure :-1 s=READY pgs=1063 cs=0 l=1 rev1=1 rx=0x7efd3000b000 tx=0x7efd30006df0).stop 2022-07-20T03:15:43.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 shutdown_connections 2022-07-20T03:15:43.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efd20053940 0x7efd20055e00 unknown :-1 s=CLOSED pgs=516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd40115670 0x7efd40112aa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd400b25d0 0x7efd40115180 unknown :-1 s=CLOSED pgs=1063 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 --2- 172.21.15.61:0/3943191221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd400b11f0 0x7efd400b15d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 >> 172.21.15.61:0/3943191221 conn(0x7efd400678c0 msgr2=0x7efd40140080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 shutdown_connections 2022-07-20T03:15:43.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.240+0000 7efd45fec700 1 -- 172.21.15.61:0/3943191221 wait complete. 2022-07-20T03:15:43.256 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:43.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.368+0000 7f42a99af700 1 Processor -- start 2022-07-20T03:15:43.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.368+0000 7f42a99af700 1 -- start start 2022-07-20T03:15:43.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a99af700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 0x7f429c0d7820 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a99af700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f429c0d3f10 0x7f429c0d7d10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a99af700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f429c006550 con 0x7f429c03d750 2022-07-20T03:15:43.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a99af700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f429c0d4ec0 con 0x7f429c0d3f10 2022-07-20T03:15:43.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a99af700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c03f210 con 0x7f429c047d00 2022-07-20T03:15:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a91ae700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f429c03d750 0x7f429c03dbc0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45618/0 (socket says 172.21.15.61:45618) 2022-07-20T03:15:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a91ae700 1 -- 172.21.15.61:0/2446486874 learned_addr learned my addr 172.21.15.61:0/2446486874 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2446486874 >> v2:172.21.15.117:3300/0 conn(0x7f429c0d3f10 0x7f429c0d7d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a89ad700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 0x7f429c0d7820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3173818687 0 0) 0x7f429c006550 con 0x7f429c03d750 2022-07-20T03:15:43.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4288003260 con 0x7f429c03d750 2022-07-20T03:15:43.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3156959117 0 0) 0x7f4288003260 con 0x7f429c03d750 2022-07-20T03:15:43.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 msgr2=0x7f429c0d7820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 0x7f429c0d7820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 >> v2:172.21.15.117:3300/0 conn(0x7f429c0d3f10 msgr2=0x7f429c0d7d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 --2- 172.21.15.61:0/2446486874 >> v2:172.21.15.117:3300/0 conn(0x7f429c0d3f10 0x7f429c0d7d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0062b0 con 0x7f429c03d750 2022-07-20T03:15:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f42a405ad40 con 0x7f429c03d750 2022-07-20T03:15:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 >> v1:172.21.15.61:6789/0 conn(0x7f429c03d750 legacy=0x7f429c03dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 0x7f4288003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4288003c80 0x7f4288004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42880049f0 0x7f4288004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0d4ec0 con 0x7f4288003c80 2022-07-20T03:15:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c03f210 con 0x7f42880049f0 2022-07-20T03:15:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f428800b750 con 0x7f429c047d00 2022-07-20T03:15:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4288003c80 0x7f4288004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a89ad700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 0x7f4288003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a91ae700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42880049f0 0x7f4288004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 -- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 msgr2=0x7f4288003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c047d00 0x7f4288003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 -- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42880049f0 msgr2=0x7f4288004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f42880049f0 0x7f4288004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 -- 172.21.15.61:0/2446486874 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4288003260 con 0x7f4288003c80 2022-07-20T03:15:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4288003c80 0x7f4288004500 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f429000c000 tx=0x7f4290006df0).ready entity=mon.0 client_cookie=ca5a289c20daf125 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f429000e830 con 0x7f4288003c80 2022-07-20T03:15:43.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f429000e990 con 0x7f4288003c80 2022-07-20T03:15:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.372+0000 7f42a37fe700 1 -- 172.21.15.61:0/2446486874 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f429000ecf0 con 0x7f4288003c80 2022-07-20T03:15:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2446486874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4288003c80 msgr2=0x7f4288004500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 --2- 172.21.15.61:0/2446486874 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4288003c80 0x7f4288004500 secure :-1 s=READY pgs=1065 cs=0 l=1 rev1=1 rx=0x7f429000c000 tx=0x7f4290006df0).stop 2022-07-20T03:15:43.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a91ae700 1 -- 172.21.15.61:0/2446486874 reap_dead start 2022-07-20T03:15:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2446486874 shutdown_connections 2022-07-20T03:15:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2446486874 >> 172.21.15.61:0/2446486874 conn(0x7f429c00e2a0 msgr2=0x7f429c047450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2446486874 shutdown_connections 2022-07-20T03:15:43.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2446486874 wait complete. 2022-07-20T03:15:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 Processor -- start 2022-07-20T03:15:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- start start 2022-07-20T03:15:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4288003c80 0x7f429c0d94d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 0x7f429c0e6800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c03adb0 0x7f429c03b230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0d5d40 con 0x7f429c0da4e0 2022-07-20T03:15:43.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0d6210 con 0x7f4288003c80 2022-07-20T03:15:43.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f429c0e4440 con 0x7f429c03adb0 2022-07-20T03:15:43.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 0x7f429c0e6800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 0x7f429c0e6800 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49034/0 (socket says 172.21.15.61:49034) 2022-07-20T03:15:43.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 -- 172.21.15.61:0/2878937684 learned_addr learned my addr 172.21.15.61:0/2878937684 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:43.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a89ad700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4288003c80 0x7f429c0d94d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a91ae700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c03adb0 0x7f429c03b230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 -- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c03adb0 msgr2=0x7f429c03b230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c03adb0 0x7f429c03b230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 -- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4288003c80 msgr2=0x7f429c0d94d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4288003c80 0x7f429c0d94d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 -- 172.21.15.61:0/2878937684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f429c0d5f70 con 0x7f429c0da4e0 2022-07-20T03:15:43.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a89ad700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4288003c80 0x7f429c0d94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:43.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a3fff700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 0x7f429c0e6800 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f429c0e7860 tx=0x7f4290019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f429001b820 con 0x7f429c0da4e0 2022-07-20T03:15:43.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4290000e90 con 0x7f429c0da4e0 2022-07-20T03:15:43.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f429c03a910 con 0x7f429c0da4e0 2022-07-20T03:15:43.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f429001bb70 con 0x7f429c0da4e0 2022-07-20T03:15:43.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.376+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f429c03d3b0 con 0x7f429c0da4e0 2022-07-20T03:15:43.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.380+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f429002c070 con 0x7f429c0da4e0 2022-07-20T03:15:43.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.380+0000 7f42a1ffb700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4284053a40 0x7f4284055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.380+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f429c03ef70 con 0x7f429c0da4e0 2022-07-20T03:15:43.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.380+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f429006cfc0 con 0x7f429c0da4e0 2022-07-20T03:15:43.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.380+0000 7f42a89ad700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4284053a40 0x7f4284055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.380+0000 7f42a89ad700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4284053a40 0x7f4284055f00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f429800b000 tx=0x7f4298008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.384+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f42900478f0 con 0x7f429c0da4e0 2022-07-20T03:15:43.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.508+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f429c005240 con 0x7f429c0da4e0 2022-07-20T03:15:43.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a1ffb700 1 -- 172.21.15.61:0/2878937684 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v101) v1 ==== 92+0+10 (secure 0 0 0) 0x7f42900474e0 con 0x7f429c0da4e0 2022-07-20T03:15:43.525 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:43.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4284053a40 msgr2=0x7f4284055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4284053a40 0x7f4284055f00 secure :-1 s=READY pgs=517 cs=0 l=1 rev1=1 rx=0x7f429800b000 tx=0x7f4298008040).stop 2022-07-20T03:15:43.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 msgr2=0x7f429c0e6800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 0x7f429c0e6800 secure :-1 s=READY pgs=1066 cs=0 l=1 rev1=1 rx=0x7f429c0e7860 tx=0x7f4290019000).stop 2022-07-20T03:15:43.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 shutdown_connections 2022-07-20T03:15:43.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4284053a40 0x7f4284055f00 unknown :-1 s=CLOSED pgs=517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f429c03adb0 0x7f429c03b230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f429c0da4e0 0x7f429c0e6800 unknown :-1 s=CLOSED pgs=1066 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 --2- 172.21.15.61:0/2878937684 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4288003c80 0x7f429c0d94d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 >> 172.21.15.61:0/2878937684 conn(0x7f429c00e2a0 msgr2=0x7f429c047430 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 shutdown_connections 2022-07-20T03:15:43.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.512+0000 7f42a99af700 1 -- 172.21.15.61:0/2878937684 wait complete. 2022-07-20T03:15:43.536 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:15:43.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.644+0000 7fb8eeb8f700 1 Processor -- start 2022-07-20T03:15:43.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.644+0000 7fb8eeb8f700 1 -- start start 2022-07-20T03:15:43.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e80b59f0 0x7fb8e81407e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb8e8140cd0 0x7fb8e81445a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb8e8144a90 0x7fb8e80af940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e80b1750 con 0x7fb8e8144a90 2022-07-20T03:15:43.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb8e80b5dd0 con 0x7fb8e8140cd0 2022-07-20T03:15:43.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e80652a0 con 0x7fb8e80b59f0 2022-07-20T03:15:43.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb8e8144a90 0x7fb8e80af940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb8e8144a90 0x7fb8e80af940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49044/0 (socket says 172.21.15.61:49044) 2022-07-20T03:15:43.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 -- 172.21.15.61:0/1569911566 learned_addr learned my addr 172.21.15.61:0/1569911566 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:43.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8edb8d700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e80b59f0 0x7fb8e81407e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ed38c700 1 --2- 172.21.15.61:0/1569911566 >> v2:172.21.15.117:3300/0 conn(0x7fb8e8140cd0 0x7fb8e81445a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 -- 172.21.15.61:0/1569911566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e80b59f0 msgr2=0x7fb8e81407e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e80b59f0 0x7fb8e81407e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 -- 172.21.15.61:0/1569911566 >> v2:172.21.15.117:3300/0 conn(0x7fb8e8140cd0 msgr2=0x7fb8e81445a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 --2- 172.21.15.61:0/1569911566 >> v2:172.21.15.117:3300/0 conn(0x7fb8e8140cd0 0x7fb8e81445a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 -- 172.21.15.61:0/1569911566 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e8143e60 con 0x7fb8e8144a90 2022-07-20T03:15:43.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ed38c700 1 --2- 172.21.15.61:0/1569911566 >> v2:172.21.15.117:3300/0 conn(0x7fb8e8140cd0 0x7fb8e81445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:43.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8edb8d700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e80b59f0 0x7fb8e81407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:43.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ee38e700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e80af940 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7fb8e0002f70 tx=0x7fb8e0006eb0).ready entity=mon.0 client_cookie=f1ea28a0407daa2d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ecb8b700 1 -- 172.21.15.61:0/1569911566 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e000d570 con 0x7fb8e8144a90 2022-07-20T03:15:43.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ecb8b700 1 -- 172.21.15.61:0/1569911566 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e0000e90 con 0x7fb8e8144a90 2022-07-20T03:15:43.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8ecb8b700 1 -- 172.21.15.61:0/1569911566 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e000d900 con 0x7fb8e8144a90 2022-07-20T03:15:43.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/1569911566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 msgr2=0x7fb8e80af940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e80af940 secure :-1 s=READY pgs=1067 cs=0 l=1 rev1=1 rx=0x7fb8e0002f70 tx=0x7fb8e0006eb0).stop 2022-07-20T03:15:43.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/1569911566 shutdown_connections 2022-07-20T03:15:43.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e80af940 unknown :-1 s=CLOSED pgs=1067 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/1569911566 >> v2:172.21.15.117:3300/0 conn(0x7fb8e8140cd0 0x7fb8e81445a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/1569911566 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e80b59f0 0x7fb8e81407e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/1569911566 >> 172.21.15.61:0/1569911566 conn(0x7fb8e8063480 msgr2=0x7fb8e80b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/1569911566 shutdown_connections 2022-07-20T03:15:43.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.648+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/1569911566 wait complete. 2022-07-20T03:15:43.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 Processor -- start 2022-07-20T03:15:43.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- start start 2022-07-20T03:15:43.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8e80b59f0 0x7fb8e80b4490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e8140cd0 0x7fb8e8128330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e812a770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e81535a0 con 0x7fb8e8144a90 2022-07-20T03:15:43.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8153840 con 0x7fb8e80b59f0 2022-07-20T03:15:43.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb8e8153ae0 con 0x7fb8e8140cd0 2022-07-20T03:15:43.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e812a770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e812a770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49050/0 (socket says 172.21.15.61:49050) 2022-07-20T03:15:43.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 -- 172.21.15.61:0/2386943619 learned_addr learned my addr 172.21.15.61:0/2386943619 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:43.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8edb8d700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8e80b59f0 0x7fb8e80b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ed38c700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e8140cd0 0x7fb8e8128330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 -- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e8140cd0 msgr2=0x7fb8e8128330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e8140cd0 0x7fb8e8128330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 -- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8e80b59f0 msgr2=0x7fb8e80b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8e80b59f0 0x7fb8e80b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 -- 172.21.15.61:0/2386943619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8e000b1e0 con 0x7fb8e8144a90 2022-07-20T03:15:43.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8ee38e700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e812a770 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7fb8e0002f70 tx=0x7fb8e0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e00117e0 con 0x7fb8e8144a90 2022-07-20T03:15:43.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb8e0011940 con 0x7fb8e8144a90 2022-07-20T03:15:43.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8e80b4980 con 0x7fb8e8144a90 2022-07-20T03:15:43.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb8e0011ca0 con 0x7fb8e8144a90 2022-07-20T03:15:43.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8e80b5390 con 0x7fb8e8144a90 2022-07-20T03:15:43.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.652+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8e8144100 con 0x7fb8e8144a90 2022-07-20T03:15:43.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.656+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb8e0025030 con 0x7fb8e8144a90 2022-07-20T03:15:43.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.656+0000 7fb8daffd700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8c8053940 0x7fb8c8055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.656+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb8e006d030 con 0x7fb8e8144a90 2022-07-20T03:15:43.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.656+0000 7fb8edb8d700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8c8053940 0x7fb8c8055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.656+0000 7fb8edb8d700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8c8053940 0x7fb8c8055e00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fb8dc00d000 tx=0x7fb8dc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.664+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb8e0043f90 con 0x7fb8e8144a90 2022-07-20T03:15:43.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb8e80574e0 con 0x7fb8e8144a90 2022-07-20T03:15:43.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8daffd700 1 -- 172.21.15.61:0/2386943619 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v101) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb8e00230a0 con 0x7fb8e8144a90 2022-07-20T03:15:43.800 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:43.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8c8053940 msgr2=0x7fb8c8055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8c8053940 0x7fb8c8055e00 secure :-1 s=READY pgs=518 cs=0 l=1 rev1=1 rx=0x7fb8dc00d000 tx=0x7fb8dc008040).stop 2022-07-20T03:15:43.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 msgr2=0x7fb8e812a770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e812a770 secure :-1 s=READY pgs=1068 cs=0 l=1 rev1=1 rx=0x7fb8e0002f70 tx=0x7fb8e0015000).stop 2022-07-20T03:15:43.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 shutdown_connections 2022-07-20T03:15:43.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb8c8053940 0x7fb8c8055e00 unknown :-1 s=CLOSED pgs=518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb8e8144a90 0x7fb8e812a770 unknown :-1 s=CLOSED pgs=1068 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb8e8140cd0 0x7fb8e8128330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 --2- 172.21.15.61:0/2386943619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb8e80b59f0 0x7fb8e80b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 >> 172.21.15.61:0/2386943619 conn(0x7fb8e8063480 msgr2=0x7fb8e80654a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 shutdown_connections 2022-07-20T03:15:43.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.792+0000 7fb8eeb8f700 1 -- 172.21.15.61:0/2386943619 wait complete. 2022-07-20T03:15:43.809 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:15:43.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 Processor -- start 2022-07-20T03:15:43.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- start start 2022-07-20T03:15:43.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7efe840db470 0x7efe84041000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efe8403f9e0 0x7efe840db950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efe8403daf0 con 0x7efe8403d710 2022-07-20T03:15:43.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe840d95b0 con 0x7efe8403f9e0 2022-07-20T03:15:43.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efe8403e3a0 con 0x7efe840db470 2022-07-20T03:15:43.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efe8403f9e0 0x7efe840db950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efe8403f9e0 0x7efe840db950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49058/0 (socket says 172.21.15.61:49058) 2022-07-20T03:15:43.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 -- 172.21.15.61:0/2156956626 learned_addr learned my addr 172.21.15.61:0/2156956626 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:43.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe8b7fe700 1 --2- 172.21.15.61:0/2156956626 >> v2:172.21.15.117:3300/0 conn(0x7efe840db470 0x7efe84041000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 -- 172.21.15.61:0/2156956626 >> v2:172.21.15.117:3300/0 conn(0x7efe840db470 msgr2=0x7efe84041000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 --2- 172.21.15.61:0/2156956626 >> v2:172.21.15.117:3300/0 conn(0x7efe840db470 0x7efe84041000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 -- 172.21.15.61:0/2156956626 >> v1:172.21.15.117:6789/0 conn(0x7efe8403d710 legacy=0x7efe8403c8d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:43.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 -- 172.21.15.61:0/2156956626 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840daa90 con 0x7efe8403f9e0 2022-07-20T03:15:43.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe90aa4700 1 --2- 172.21.15.61:0/2156956626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403f9e0 0x7efe840db950 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7efe84040a50 tx=0x7efe8c05f750).ready entity=mon.0 client_cookie=d09182983fe73003 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe8affd700 1 -- 172.21.15.61:0/2156956626 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe8c084030 con 0x7efe8403f9e0 2022-07-20T03:15:43.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe8affd700 1 -- 172.21.15.61:0/2156956626 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe8c05af90 con 0x7efe8403f9e0 2022-07-20T03:15:43.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe8affd700 1 -- 172.21.15.61:0/2156956626 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe8c06a910 con 0x7efe8403f9e0 2022-07-20T03:15:43.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- 172.21.15.61:0/2156956626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403f9e0 msgr2=0x7efe840db950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 --2- 172.21.15.61:0/2156956626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403f9e0 0x7efe840db950 secure :-1 s=READY pgs=1069 cs=0 l=1 rev1=1 rx=0x7efe84040a50 tx=0x7efe8c05f750).stop 2022-07-20T03:15:43.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- 172.21.15.61:0/2156956626 shutdown_connections 2022-07-20T03:15:43.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 --2- 172.21.15.61:0/2156956626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403f9e0 0x7efe840db950 unknown :-1 s=CLOSED pgs=1069 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 --2- 172.21.15.61:0/2156956626 >> v2:172.21.15.117:3300/0 conn(0x7efe840db470 0x7efe84041000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- 172.21.15.61:0/2156956626 >> 172.21.15.61:0/2156956626 conn(0x7efe84046330 msgr2=0x7efe84046710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:43.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- 172.21.15.61:0/2156956626 shutdown_connections 2022-07-20T03:15:43.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.916+0000 7efe912a5700 1 -- 172.21.15.61:0/2156956626 wait complete. 2022-07-20T03:15:43.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 Processor -- start 2022-07-20T03:15:43.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 -- start start 2022-07-20T03:15:43.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 0x7efe840a2300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efe8403f9e0 0x7efe840ac2e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe840db470 0x7efe840e7970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840acbb0 con 0x7efe8403d710 2022-07-20T03:15:43.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efe840ace50 con 0x7efe840db470 2022-07-20T03:15:43.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe912a5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efe840a1d40 con 0x7efe8403f9e0 2022-07-20T03:15:43.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 0x7efe840a2300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 0x7efe840a2300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49060/0 (socket says 172.21.15.61:49060) 2022-07-20T03:15:43.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 -- 172.21.15.61:0/3803123270 learned_addr learned my addr 172.21.15.61:0/3803123270 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:43.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe90aa4700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe840db470 0x7efe840e7970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8b7fe700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efe8403f9e0 0x7efe840ac2e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 -- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efe8403f9e0 msgr2=0x7efe840ac2e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efe8403f9e0 0x7efe840ac2e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 -- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe840db470 msgr2=0x7efe840e7970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:43.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe840db470 0x7efe840e7970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:43.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 -- 172.21.15.61:0/3803123270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efe840ac7d0 con 0x7efe8403d710 2022-07-20T03:15:43.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe90aa4700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe840db470 0x7efe840e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:43.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8b7fe700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efe8403f9e0 0x7efe840ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:43.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.920+0000 7efe8bfff700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 0x7efe840a2300 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7efe78002f30 tx=0x7efe78006d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe78012410 con 0x7efe8403d710 2022-07-20T03:15:43.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efe78012570 con 0x7efe8403d710 2022-07-20T03:15:43.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efe840a3000 con 0x7efe8403d710 2022-07-20T03:15:43.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efe780128d0 con 0x7efe8403d710 2022-07-20T03:15:43.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efe840e8080 con 0x7efe8403d710 2022-07-20T03:15:43.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efe840aca20 con 0x7efe8403d710 2022-07-20T03:15:43.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efe7800f030 con 0x7efe8403d710 2022-07-20T03:15:43.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe897fa700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe6c053af0 0x7efe6c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:43.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efe7806b9a0 con 0x7efe8403d710 2022-07-20T03:15:43.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.924+0000 7efe8b7fe700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe6c053af0 0x7efe6c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:43.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.928+0000 7efe8b7fe700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe6c053af0 0x7efe6c055fb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7efe8000c000 tx=0x7efe80008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:43.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:43.932+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efe78042900 con 0x7efe8403d710 2022-07-20T03:15:44.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.056+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7efe840dad30 con 0x7efe8403d710 2022-07-20T03:15:44.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.056+0000 7efe897fa700 1 -- 172.21.15.61:0/3803123270 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v101) v1 ==== 100+0+10 (secure 0 0 0) 0x7efe780182b0 con 0x7efe8403d710 2022-07-20T03:15:44.065 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:44.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe6c053af0 msgr2=0x7efe6c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe6c053af0 0x7efe6c055fb0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7efe8000c000 tx=0x7efe80008040).stop 2022-07-20T03:15:44.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 msgr2=0x7efe840a2300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 0x7efe840a2300 secure :-1 s=READY pgs=1070 cs=0 l=1 rev1=1 rx=0x7efe78002f30 tx=0x7efe78006d70).stop 2022-07-20T03:15:44.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 shutdown_connections 2022-07-20T03:15:44.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe6c053af0 0x7efe6c055fb0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efe840db470 0x7efe840e7970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efe8403f9e0 0x7efe840ac2e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 --2- 172.21.15.61:0/3803123270 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efe8403d710 0x7efe840a2300 unknown :-1 s=CLOSED pgs=1070 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 >> 172.21.15.61:0/3803123270 conn(0x7efe84046330 msgr2=0x7efe84045460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:44.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 shutdown_connections 2022-07-20T03:15:44.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.060+0000 7efe912a5700 1 -- 172.21.15.61:0/3803123270 wait complete. 2022-07-20T03:15:44.080 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:15:44.081 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:44.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6dba8f5700 1 Processor -- start 2022-07-20T03:15:44.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6dba8f5700 1 -- start start 2022-07-20T03:15:44.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6dba8f5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6db405c3f0 0x7f6db414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:44.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6dba8f5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db405d480 con 0x7f6db40acef0 2022-07-20T03:15:44.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6dba8f5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6db40b21e0 con 0x7f6db40b1e00 2022-07-20T03:15:44.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6dba8f5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6db405cf40 con 0x7f6db405c3f0 2022-07-20T03:15:44.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6db8e92700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6db40acef0 0x7f6db4146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45670/0 (socket says 172.21.15.61:45670) 2022-07-20T03:15:44.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6db8e92700 1 -- 172.21.15.61:0/1013965358 learned_addr learned my addr 172.21.15.61:0/1013965358 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:44.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6db37fe700 1 --2- 172.21.15.61:0/1013965358 >> v2:172.21.15.117:3300/0 conn(0x7f6db405c3f0 0x7f6db414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:44.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6db2ffd700 1 -- 172.21.15.61:0/1013965358 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 456386668 0 0) 0x7f6db405d480 con 0x7f6db40acef0 2022-07-20T03:15:44.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.180+0000 7f6db2ffd700 1 -- 172.21.15.61:0/1013965358 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6da0003260 con 0x7f6db40acef0 2022-07-20T03:15:44.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db37fe700 1 -- 172.21.15.61:0/1013965358 >> v1:172.21.15.117:6789/0 conn(0x7f6db40b1e00 legacy=0x7f6db40ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db37fe700 1 -- 172.21.15.61:0/1013965358 >> v1:172.21.15.61:6789/0 conn(0x7f6db40acef0 legacy=0x7f6db4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db37fe700 1 -- 172.21.15.61:0/1013965358 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6db405d1e0 con 0x7f6db405c3f0 2022-07-20T03:15:44.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db37fe700 1 --2- 172.21.15.61:0/1013965358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db405c3f0 0x7f6db414a6d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f6da40032f0 tx=0x7f6da40091d0).ready entity=mon.1 client_cookie=1638046a1673dd15 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:44.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db2ffd700 1 -- 172.21.15.61:0/1013965358 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da400b6c0 con 0x7f6db405c3f0 2022-07-20T03:15:44.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db2ffd700 1 -- 172.21.15.61:0/1013965358 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6da400b820 con 0x7f6db405c3f0 2022-07-20T03:15:44.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db2ffd700 1 -- 172.21.15.61:0/1013965358 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6da400b990 con 0x7f6db405c3f0 2022-07-20T03:15:44.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- 172.21.15.61:0/1013965358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db405c3f0 msgr2=0x7f6db414a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/1013965358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db405c3f0 0x7f6db414a6d0 secure :-1 s=READY pgs=489 cs=0 l=1 rev1=1 rx=0x7f6da40032f0 tx=0x7f6da40091d0).stop 2022-07-20T03:15:44.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- 172.21.15.61:0/1013965358 shutdown_connections 2022-07-20T03:15:44.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/1013965358 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db405c3f0 0x7f6db414a6d0 unknown :-1 s=CLOSED pgs=489 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- 172.21.15.61:0/1013965358 >> 172.21.15.61:0/1013965358 conn(0x7f6db40611c0 msgr2=0x7f6db4062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:44.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- 172.21.15.61:0/1013965358 shutdown_connections 2022-07-20T03:15:44.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- 172.21.15.61:0/1013965358 wait complete. 2022-07-20T03:15:44.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 Processor -- start 2022-07-20T03:15:44.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- start start 2022-07-20T03:15:44.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db40acef0 0x7f6db4146650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:44.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db40b1e00 0x7f6db4143490 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:44.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 0x7f6db41566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:44.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40a1c40 con 0x7f6db4143980 2022-07-20T03:15:44.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40b5420 con 0x7f6db40b1e00 2022-07-20T03:15:44.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6dba8f5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6db40b5180 con 0x7f6db40acef0 2022-07-20T03:15:44.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db8e92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 0x7f6db41566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:44.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db8e92700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 0x7f6db41566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49078/0 (socket says 172.21.15.61:49078) 2022-07-20T03:15:44.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db8e92700 1 -- 172.21.15.61:0/3730082376 learned_addr learned my addr 172.21.15.61:0/3730082376 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:44.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db37fe700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db40b1e00 0x7f6db4143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:44.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db3fff700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db40acef0 0x7f6db4146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:44.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.184+0000 7f6db8e92700 1 -- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db40acef0 msgr2=0x7f6db4146650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db8e92700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db40acef0 0x7f6db4146650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db8e92700 1 -- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db40b1e00 msgr2=0x7f6db4143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db8e92700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db40b1e00 0x7f6db4143490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db8e92700 1 -- 172.21.15.61:0/3730082376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6da4009470 con 0x7f6db4143980 2022-07-20T03:15:44.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db8e92700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 0x7f6db41566d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f6dac001110 tx=0x7f6dac007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:44.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dac00ce80 con 0x7f6db4143980 2022-07-20T03:15:44.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6dac0070d0 con 0x7f6db4143980 2022-07-20T03:15:44.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6db40a19a0 con 0x7f6db4143980 2022-07-20T03:15:44.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6dac007430 con 0x7f6db4143980 2022-07-20T03:15:44.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6db4157870 con 0x7f6db4143980 2022-07-20T03:15:44.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6dac010070 con 0x7f6db4143980 2022-07-20T03:15:44.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6db40b04b0 con 0x7f6db4143980 2022-07-20T03:15:44.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db17fa700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d94053a40 0x7f6d94055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:44.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6dac06b520 con 0x7f6db4143980 2022-07-20T03:15:44.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.188+0000 7f6db3fff700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d94053a40 0x7f6d94055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:44.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.192+0000 7f6db3fff700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d94053a40 0x7f6d94055f00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f6da800d000 tx=0x7f6da8007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:44.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.196+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6dac0423e0 con 0x7f6db4143980 2022-07-20T03:15:44.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.360+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6db40574e0 con 0x7f6db4143980 2022-07-20T03:15:44.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6db17fa700 1 -- 172.21.15.61:0/3730082376 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f6dac007bc0 con 0x7f6db4143980 2022-07-20T03:15:44.368 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:44.369 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:35.686626+0000","modified":"2022-07-20T03:15:36.689415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7335":{"gid":7335,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2834789489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2834789489},{"type":"v1","addr":"172.21.15.61:6833","nonce":2834789489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T03:15:44.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d94053a40 msgr2=0x7f6d94055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d94053a40 0x7f6d94055f00 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f6da800d000 tx=0x7f6da8007b70).stop 2022-07-20T03:15:44.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 msgr2=0x7f6db41566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:44.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 0x7f6db41566d0 secure :-1 s=READY pgs=1072 cs=0 l=1 rev1=1 rx=0x7f6dac001110 tx=0x7f6dac007a80).stop 2022-07-20T03:15:44.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 shutdown_connections 2022-07-20T03:15:44.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6d94053a40 0x7f6d94055f00 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6db40b1e00 0x7f6db4143490 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6db4143980 0x7f6db41566d0 unknown :-1 s=CLOSED pgs=1072 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 --2- 172.21.15.61:0/3730082376 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6db40acef0 0x7f6db4146650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:44.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 >> 172.21.15.61:0/3730082376 conn(0x7f6db405c3f0 msgr2=0x7f6db414ae10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:44.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 shutdown_connections 2022-07-20T03:15:44.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:44.364+0000 7f6dba8f5700 1 -- 172.21.15.61:0/3730082376 wait complete. 2022-07-20T03:15:44.381 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-20T03:15:44.590 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 56, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:35.686626+0000', 'modified': '2022-07-20T03:15:36.689415+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7335}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7335': {'gid': 7335, 'name': 'a', 'rank': 0, 'incarnation': 55, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/2834789489', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 2834789489}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 2834789489}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [20], 'metadata_pool': 19, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:15:44.591 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:15:44.592 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:15:44.710 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7335,"want_state":"up:active","state":"up:active","rank_uptime":9.0102461559999991,"mdsmap_epoch":56,"osdmap_epoch":101,"osdmap_epoch_barrier":100,"uptime":13.481438044000001} 2022-07-20T03:15:44.711 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7335, "mdsmap_epoch": 56, "osdmap_epoch": 101, "osdmap_epoch_barrier": 100, "rank_uptime": 9.010246156, "state": "up:active", "uptime": 13.481438044, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:15:44.712 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:15:44.713 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:15:44.715 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:15:44.715 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:44.716 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:44.750 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:15:44.775 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:44.777 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:44.840 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:44.842 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:44.843 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:44.844 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:15:44.896 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:15:44.897 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:44.907 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:15:44.928 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:15:44.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:15:44.979 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22686]: starting ceph client 2022-07-20T03:15:44.980 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:15:44.976+0000 7f3acd0ac080 -1 init, newargv = 0x55b187b5e290 newargc=9 2022-07-20T03:15:44.989 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22686]: starting fuse 2022-07-20T03:15:44.990 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:15:44.992 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:15:44.992 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:44.994 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:15:45.001 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:15:45.001 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:15:45.003 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:45.004 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:15:45.005 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:15:45.006 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:15:45.006 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:15:45.007 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:15:45.008 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:45.010 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:15:45.011 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:15:45.012 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:15:45.012 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:45.013 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:15:45.014 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:15:45.015 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:15:45.015 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:15:45.016 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:45.017 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:15:45.018 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:15:45.018 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:15:45.019 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:15:45.020 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:15:45.021 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:15:45.022 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:15:45.023 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:15:45.069 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22686.asok 2022-07-20T03:15:45.075 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22686.asok 2022-07-20T03:15:45.076 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:45.077 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22686.asok status 2022-07-20T03:15:45.214 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:15:45.215 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:15:45.215 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:15:45.215 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:15:45.215 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:15:45.216 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:15:45.216 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:15:45.216 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22686", 2022-07-20T03:15:45.216 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:15:45.216 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:45.217 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:15:45.217 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:15:45.217 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 7545, 2022-07-20T03:15:45.217 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:15:45.218 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:15:45.218 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:15:45.218 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 7545 2022-07-20T03:15:45.218 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:15:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:15:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:15:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 753285495 2022-07-20T03:15:45.219 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:15:45.220 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:45.220 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:15:45.220 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:15:45.220 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:15:45.221 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 753285495 2022-07-20T03:15:45.221 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:15:45.221 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.7545 172.21.15.163:0/753285495", 2022-07-20T03:15:45.221 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/753285495", 2022-07-20T03:15:45.221 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:15:45.222 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 56, 2022-07-20T03:15:45.222 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 101, 2022-07-20T03:15:45.222 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:15:45.222 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:15:45.223 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:15:45.223 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:15:45.224 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:45.225 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:45.227 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:45.238 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:15:45.239 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:45.239 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:45.240 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:45.266 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-20T03:15:45.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 Processor -- start 2022-07-20T03:15:45.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 -- start start 2022-07-20T03:15:45.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11800b1e00 0x7f118014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f118005c3f0 0x7f11800ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f118005d480 con 0x7f118005bd90 2022-07-20T03:15:45.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f11800b21e0 con 0x7f118005c3f0 2022-07-20T03:15:45.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f1186178700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118005cf40 con 0x7f11800b1e00 2022-07-20T03:15:45.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f117effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f118005c3f0 0x7f11800ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f117effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f118005c3f0 0x7f11800ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49084/0 (socket says 172.21.15.61:49084) 2022-07-20T03:15:45.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f117f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11800b1e00 0x7f118014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.372+0000 7f117effd700 1 -- 172.21.15.61:0/1993821394 learned_addr learned my addr 172.21.15.61:0/1993821394 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:45.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117effd700 1 -- 172.21.15.61:0/1993821394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11800b1e00 msgr2=0x7f118014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:45.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117effd700 1 --2- 172.21.15.61:0/1993821394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11800b1e00 0x7f118014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117effd700 1 -- 172.21.15.61:0/1993821394 >> v1:172.21.15.117:6789/0 conn(0x7f118005bd90 legacy=0x7f1180146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117e7fc700 1 -- 172.21.15.61:0/1993821394 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710154119 0 0) 0x7f118005d480 con 0x7f118005bd90 2022-07-20T03:15:45.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117effd700 1 -- 172.21.15.61:0/1993821394 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f118005d1e0 con 0x7f118005c3f0 2022-07-20T03:15:45.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117f7fe700 1 --2- 172.21.15.61:0/1993821394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11800b1e00 0x7f118014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:45.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117effd700 1 --2- 172.21.15.61:0/1993821394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118005c3f0 0x7f11800ac7e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f1168002f70 tx=0x7f1168006eb0).ready entity=mon.0 client_cookie=fa442e8b3b8aa9ac server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:45.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117e7fc700 1 -- 172.21.15.61:0/1993821394 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116800d570 con 0x7f118005c3f0 2022-07-20T03:15:45.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117e7fc700 1 -- 172.21.15.61:0/1993821394 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1168000e90 con 0x7f118005c3f0 2022-07-20T03:15:45.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f117e7fc700 1 -- 172.21.15.61:0/1993821394 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f116800d8c0 con 0x7f118005c3f0 2022-07-20T03:15:45.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 -- 172.21.15.61:0/1993821394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118005c3f0 msgr2=0x7f11800ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 --2- 172.21.15.61:0/1993821394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118005c3f0 0x7f11800ac7e0 secure :-1 s=READY pgs=1076 cs=0 l=1 rev1=1 rx=0x7f1168002f70 tx=0x7f1168006eb0).stop 2022-07-20T03:15:45.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 -- 172.21.15.61:0/1993821394 shutdown_connections 2022-07-20T03:15:45.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 --2- 172.21.15.61:0/1993821394 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f118005c3f0 0x7f11800ac7e0 unknown :-1 s=CLOSED pgs=1076 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 --2- 172.21.15.61:0/1993821394 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11800b1e00 0x7f118014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 -- 172.21.15.61:0/1993821394 >> 172.21.15.61:0/1993821394 conn(0x7f11800611c0 msgr2=0x7f1180062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:45.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 -- 172.21.15.61:0/1993821394 shutdown_connections 2022-07-20T03:15:45.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.376+0000 7f1186178700 1 -- 172.21.15.61:0/1993821394 wait complete. 2022-07-20T03:15:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 Processor -- start 2022-07-20T03:15:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 -- start start 2022-07-20T03:15:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f118005c3f0 0x7f11800a18c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 0x7f1180156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1180159a90 0x7f1180159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f118009ee20 con 0x7f11800b1e00 2022-07-20T03:15:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f11801573f0 con 0x7f1180159a90 2022-07-20T03:15:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f1186178700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f118009e080 con 0x7f118005c3f0 2022-07-20T03:15:45.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 0x7f1180156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f118005c3f0 0x7f11800a18c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 0x7f1180156f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49088/0 (socket says 172.21.15.61:49088) 2022-07-20T03:15:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1180159a90 0x7f1180159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117effd700 1 -- 172.21.15.61:0/2111302052 learned_addr learned my addr 172.21.15.61:0/2111302052 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117f7fe700 1 -- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1180159a90 msgr2=0x7f1180159f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117f7fe700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1180159a90 0x7f1180159f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117f7fe700 1 -- 172.21.15.61:0/2111302052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 msgr2=0x7f1180156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117f7fe700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 0x7f1180156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117f7fe700 1 -- 172.21.15.61:0/2111302052 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1178009470 con 0x7f118005c3f0 2022-07-20T03:15:45.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117ffff700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1180159a90 0x7f1180159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.380+0000 7f117effd700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 0x7f1180156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f117f7fe700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f118005c3f0 0x7f11800a18c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f117800c000 tx=0x7f11780091d0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:45.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f117800c770 con 0x7f118005c3f0 2022-07-20T03:15:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117800c8d0 con 0x7f118005c3f0 2022-07-20T03:15:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f116800b1e0 con 0x7f118005c3f0 2022-07-20T03:15:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f117800cc30 con 0x7f118005c3f0 2022-07-20T03:15:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f11800b4fc0 con 0x7f118005c3f0 2022-07-20T03:15:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f11800b04b0 con 0x7f118005c3f0 2022-07-20T03:15:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.384+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1178022070 con 0x7f118005c3f0 2022-07-20T03:15:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.388+0000 7f117cff9700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11640539d0 0x7f1164055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.388+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f117806c570 con 0x7f118005c3f0 2022-07-20T03:15:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.388+0000 7f117effd700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11640539d0 0x7f1164055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.388+0000 7f117effd700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11640539d0 0x7f1164055e90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f116801c000 tx=0x7f1168015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:45.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.396+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1178043430 con 0x7f118005c3f0 2022-07-20T03:15:45.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.564+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f11800574e0 con 0x7f118005c3f0 2022-07-20T03:15:45.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f117cff9700 1 -- 172.21.15.61:0/2111302052 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1940 (secure 0 0 0) 0x7f1178046a50 con 0x7f118005c3f0 2022-07-20T03:15:45.580 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:45.580 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:45.580 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:15:45.580 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-20T03:15:45.581 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761054093312, 2022-07-20T03:15:45.581 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6531203072, 2022-07-20T03:15:45.581 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6531227648, 2022-07-20T03:15:45.581 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085087977349758148, 2022-07-20T03:15:45.582 INFO:teuthology.orchestra.run.smithi061.stdout: "num_osds": 8, 2022-07-20T03:15:45.582 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_osds": 0, 2022-07-20T03:15:45.582 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_omap_osds": 0 2022-07-20T03:15:45.583 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:45.583 INFO:teuthology.orchestra.run.smithi061.stdout: "stats_by_class": { 2022-07-20T03:15:45.583 INFO:teuthology.orchestra.run.smithi061.stdout: "ssd": { 2022-07-20T03:15:45.583 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-20T03:15:45.584 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761054093312, 2022-07-20T03:15:45.584 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6531203072, 2022-07-20T03:15:45.584 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6531227648, 2022-07-20T03:15:45.584 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085087977349758148 2022-07-20T03:15:45.587 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:45.587 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:45.588 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:15:45.590 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:45.590 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "device_health_metrics", 2022-07-20T03:15:45.591 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 1, 2022-07-20T03:15:45.591 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:15:45.592 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:15:45.592 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:15:45.593 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:15:45.593 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:15:45.593 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:15:45.594 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337225216 2022-07-20T03:15:45.594 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:45.595 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:45.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:45.596 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "rbd", 2022-07-20T03:15:45.596 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 2, 2022-07-20T03:15:45.597 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:15:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:15:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:15:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:15:45.600 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:15:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:15:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337225216 2022-07-20T03:15:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:45.601 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_metadata", 2022-07-20T03:15:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 19, 2022-07-20T03:15:45.602 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:15:45.603 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 2304, 2022-07-20T03:15:45.603 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 21, 2022-07-20T03:15:45.603 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 3, 2022-07-20T03:15:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 2304, 2022-07-20T03:15:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 3.1881575157655107e-09, 2022-07-20T03:15:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337225216 2022-07-20T03:15:45.604 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:45.605 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:45.605 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:45.605 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_data", 2022-07-20T03:15:45.605 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 20, 2022-07-20T03:15:45.606 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:15:45.606 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:15:45.606 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:15:45.606 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:15:45.607 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:15:45.607 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:15:45.607 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361337225216 2022-07-20T03:15:45.607 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:45.608 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:45.608 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:45.608 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:45.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11640539d0 msgr2=0x7f1164055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11640539d0 0x7f1164055e90 secure :-1 s=READY pgs=521 cs=0 l=1 rev1=1 rx=0x7f116801c000 tx=0x7f1168015000).stop 2022-07-20T03:15:45.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f118005c3f0 msgr2=0x7f11800a18c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f118005c3f0 0x7f11800a18c0 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f117800c000 tx=0x7f11780091d0).stop 2022-07-20T03:15:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 shutdown_connections 2022-07-20T03:15:45.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f11640539d0 0x7f1164055e90 unknown :-1 s=CLOSED pgs=521 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1180159a90 0x7f1180159f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f11800b1e00 0x7f1180156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 --2- 172.21.15.61:0/2111302052 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f118005c3f0 0x7f11800a18c0 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 >> 172.21.15.61:0/2111302052 conn(0x7f118005bd90 msgr2=0x7f11801461e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:45.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 shutdown_connections 2022-07-20T03:15:45.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.568+0000 7f1186178700 1 -- 172.21.15.61:0/2111302052 wait complete. 2022-07-20T03:15:45.625 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data size -f json-pretty 2022-07-20T03:15:45.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.736+0000 7fb4aad48700 1 Processor -- start 2022-07-20T03:15:45.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.736+0000 7fb4aad48700 1 -- start start 2022-07-20T03:15:45.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb4a40b4b10 0x7fb4a40af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4a40b0890 0x7fb4a40ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4a40aabe0 con 0x7fb4a40b0350 2022-07-20T03:15:45.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a40b4ef0 con 0x7fb4a40b0890 2022-07-20T03:15:45.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4a40aae10 con 0x7fb4a40b4b10 2022-07-20T03:15:45.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9545700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb4a40b0890 0x7fb4a40ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aa547700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb4a40b0350 0x7fb4a40b56e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45696/0 (socket says 172.21.15.61:45696) 2022-07-20T03:15:45.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aa547700 1 -- 172.21.15.61:0/3462200947 learned_addr learned my addr 172.21.15.61:0/3462200947 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:45.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9d46700 1 --2- 172.21.15.61:0/3462200947 >> v2:172.21.15.117:3300/0 conn(0x7fb4a40b4b10 0x7fb4a40af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a8d44700 1 -- 172.21.15.61:0/3462200947 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1993980546 0 0) 0x7fb4a40aabe0 con 0x7fb4a40b0350 2022-07-20T03:15:45.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a8d44700 1 -- 172.21.15.61:0/3462200947 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb48c003260 con 0x7fb4a40b0350 2022-07-20T03:15:45.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9545700 1 -- 172.21.15.61:0/3462200947 >> v2:172.21.15.117:3300/0 conn(0x7fb4a40b4b10 msgr2=0x7fb4a40af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:45.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9545700 1 --2- 172.21.15.61:0/3462200947 >> v2:172.21.15.117:3300/0 conn(0x7fb4a40b4b10 0x7fb4a40af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9545700 1 -- 172.21.15.61:0/3462200947 >> v1:172.21.15.61:6789/0 conn(0x7fb4a40b0350 legacy=0x7fb4a40b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9545700 1 -- 172.21.15.61:0/3462200947 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a40ab0b0 con 0x7fb4a40b0890 2022-07-20T03:15:45.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9d46700 1 --2- 172.21.15.61:0/3462200947 >> v2:172.21.15.117:3300/0 conn(0x7fb4a40b4b10 0x7fb4a40af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:45.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a9545700 1 --2- 172.21.15.61:0/3462200947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0890 0x7fb4a40ac1e0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7fb498002f70 tx=0x7fb498006df0).ready entity=mon.0 client_cookie=a96347a0445c4f30 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:45.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a8d44700 1 -- 172.21.15.61:0/3462200947 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb49800d4f0 con 0x7fb4a40b0890 2022-07-20T03:15:45.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a8d44700 1 -- 172.21.15.61:0/3462200947 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb498000e90 con 0x7fb4a40b0890 2022-07-20T03:15:45.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4a8d44700 1 -- 172.21.15.61:0/3462200947 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb49800d840 con 0x7fb4a40b0890 2022-07-20T03:15:45.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- 172.21.15.61:0/3462200947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0890 msgr2=0x7fb4a40ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 --2- 172.21.15.61:0/3462200947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0890 0x7fb4a40ac1e0 secure :-1 s=READY pgs=1078 cs=0 l=1 rev1=1 rx=0x7fb498002f70 tx=0x7fb498006df0).stop 2022-07-20T03:15:45.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- 172.21.15.61:0/3462200947 shutdown_connections 2022-07-20T03:15:45.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 --2- 172.21.15.61:0/3462200947 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0890 0x7fb4a40ac1e0 unknown :-1 s=CLOSED pgs=1078 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 --2- 172.21.15.61:0/3462200947 >> v2:172.21.15.117:3300/0 conn(0x7fb4a40b4b10 0x7fb4a40af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- 172.21.15.61:0/3462200947 >> 172.21.15.61:0/3462200947 conn(0x7fb4a40632b0 msgr2=0x7fb4a4063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:45.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- 172.21.15.61:0/3462200947 shutdown_connections 2022-07-20T03:15:45.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.740+0000 7fb4aad48700 1 -- 172.21.15.61:0/3462200947 wait complete. 2022-07-20T03:15:45.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 Processor -- start 2022-07-20T03:15:45.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 -- start start 2022-07-20T03:15:45.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 0x7fb4a40b5700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4a40b0890 0x7fb4a4150d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a40b4b10 0x7fb4a4151230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4140d00 con 0x7fb4a40b0350 2022-07-20T03:15:45.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a4142bb0 con 0x7fb4a40b4b10 2022-07-20T03:15:45.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aad48700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4a413da20 con 0x7fb4a40b0890 2022-07-20T03:15:45.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 0x7fb4a40b5700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 0x7fb4a40b5700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49102/0 (socket says 172.21.15.61:49102) 2022-07-20T03:15:45.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 -- 172.21.15.61:0/2470509844 learned_addr learned my addr 172.21.15.61:0/2470509844 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:45.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 -- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4a40b0890 msgr2=0x7fb4a4150d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4a40b0890 0x7fb4a4150d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:15:45.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 -- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a40b4b10 msgr2=0x7fb4a4151230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aa547700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a40b4b10 0x7fb4a4151230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a40b4b10 0x7fb4a4151230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 -- 172.21.15.61:0/2470509844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4a000b180 con 0x7fb4a40b0350 2022-07-20T03:15:45.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4aa547700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a40b4b10 0x7fb4a4151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:45.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb4a9d46700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 0x7fb4a40b5700 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fb4a000c000 tx=0x7fb4a0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:45.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.744+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4a000c860 con 0x7fb4a40b0350 2022-07-20T03:15:45.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.748+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4a000c9c0 con 0x7fb4a40b0350 2022-07-20T03:15:45.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.748+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb49800b180 con 0x7fb4a40b0350 2022-07-20T03:15:45.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.748+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4a000cd20 con 0x7fb4a40b0350 2022-07-20T03:15:45.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.748+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4a413d4a0 con 0x7fb4a40b0350 2022-07-20T03:15:45.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.748+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4a413d890 con 0x7fb4a40b0350 2022-07-20T03:15:45.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.752+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb4a001c460 con 0x7fb4a40b0350 2022-07-20T03:15:45.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.752+0000 7fb496ffd700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb484053a40 0x7fb484055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.752+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4a006d0e0 con 0x7fb4a40b0350 2022-07-20T03:15:45.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.752+0000 7fb4a9545700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb484053a40 0x7fb484055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:45.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.752+0000 7fb4a9545700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb484053a40 0x7fb484055f00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fb498002f70 tx=0x7fb498014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:45.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.756+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4a0043ed0 con 0x7fb4a40b0350 2022-07-20T03:15:45.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.884+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"} v 0) v1 -- 0x7fb4a413dbe0 con 0x7fb4a40b0350 2022-07-20T03:15:45.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb496ffd700 1 -- 172.21.15.61:0/2470509844 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "size", "format": "json-pretty"}]=0 v101) v1 ==== 123+0+64 (secure 0 0 0) 0x7fb4a413dbe0 con 0x7fb4a40b0350 2022-07-20T03:15:45.892 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:45.892 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": "cephfs_data", 2022-07-20T03:15:45.893 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_id": 20, 2022-07-20T03:15:45.894 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2 2022-07-20T03:15:45.894 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:45.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb484053a40 msgr2=0x7fb484055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb484053a40 0x7fb484055f00 secure :-1 s=READY pgs=522 cs=0 l=1 rev1=1 rx=0x7fb498002f70 tx=0x7fb498014000).stop 2022-07-20T03:15:45.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 msgr2=0x7fb4a40b5700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:45.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 0x7fb4a40b5700 secure :-1 s=READY pgs=1079 cs=0 l=1 rev1=1 rx=0x7fb4a000c000 tx=0x7fb4a0006df0).stop 2022-07-20T03:15:45.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 shutdown_connections 2022-07-20T03:15:45.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb484053a40 0x7fb484055f00 unknown :-1 s=CLOSED pgs=522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4a40b4b10 0x7fb4a4151230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4a40b0890 0x7fb4a4150d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2022-07-20T03:15:45.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 --2- 172.21.15.61:0/2470509844 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4a40b0350 0x7fb4a40b5700 unknown :-1 s=CLOSED pgs=1079 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:45.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 >> 172.21.15.61:0/2470509844 conn(0x7fb4a40632b0 msgr2=0x7fb4a40b0c90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:45.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 shutdown_connections 2022-07-20T03:15:45.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:45.888+0000 7fb4aad48700 1 -- 172.21.15.61:0/2470509844 wait complete. 2022-07-20T03:15:45.908 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T03:15:45.909 DEBUG:teuthology.orchestra.run.smithi124:> cd /home/ubuntu/cephtest/mnt.0 && sudo df . 2022-07-20T03:15:45.939 INFO:teuthology.orchestra.run.smithi124.stdout:Filesystem 1K-blocks Used Available Use% Mounted on 2022-07-20T03:15:45.940 INFO:teuthology.orchestra.run.smithi124.stdout:ceph-fuse 352866304 0 352866304 0% /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:45.941 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:15:45.942 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:45.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:45.990 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:46.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:46.036 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:46.061 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:15:46.062 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:46.063 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:46.063 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:46.077 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:15:46.077 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:46.078 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:15:46.079 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:15:46.080 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:46.100 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23246]: fuse finished with error 0 and tester_r 0 2022-07-20T03:15:46.102 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:46.103 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:46.104 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:46.125 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:15:46.126 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:15:46.133 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:15:46.135 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:46.174 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:46.175 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:46.176 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:46.195 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:15:46.197 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:46.199 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:15:46.199 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:15:46.199 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:46.257 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22686]: fuse finished with error 0 and tester_r 0 2022-07-20T03:15:46.259 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:46.260 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:46.261 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:46.271 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:15:46.272 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:15:46.374 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:15:46.375 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:46.391 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:46.392 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:46.393 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:46.416 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:46.422 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:46.423 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:15:46.424 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:15:46.425 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:46.435 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:46.443 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:46.444 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:15:46.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.552+0000 7f24b6324700 1 Processor -- start 2022-07-20T03:15:46.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.552+0000 7f24b6324700 1 -- start start 2022-07-20T03:15:46.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.552+0000 7f24b6324700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b00ac260 0x7f24b00ac640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:46.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.552+0000 7f24b6324700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24b00b3620 0x7f24b0096890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:46.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.552+0000 7f24b6324700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f24b00b22d0 con 0x7f24b0141200 2022-07-20T03:15:46.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.552+0000 7f24b6324700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f24b00b26d0 con 0x7f24b00b3620 2022-07-20T03:15:46.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0140ac0 con 0x7f24b00ac260 2022-07-20T03:15:46.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24b00b3620 0x7f24b0096890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:46.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f24b00b3620 0x7f24b0096890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49110/0 (socket says 172.21.15.61:49110) 2022-07-20T03:15:46.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 -- 172.21.15.61:0/820162094 learned_addr learned my addr 172.21.15.61:0/820162094 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:46.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24affff700 1 --2- 172.21.15.61:0/820162094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b00ac260 0x7f24b00ac640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:46.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 -- 172.21.15.61:0/820162094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b00ac260 msgr2=0x7f24b00ac640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:46.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 --2- 172.21.15.61:0/820162094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b00ac260 0x7f24b00ac640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:46.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 -- 172.21.15.61:0/820162094 >> v1:172.21.15.61:6789/0 conn(0x7f24b0141200 legacy=0x7f24b01400c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:46.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24aeffd700 1 -- 172.21.15.61:0/820162094 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 420965890 0 0) 0x7f24b00b22d0 con 0x7f24b0141200 2022-07-20T03:15:46.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 -- 172.21.15.61:0/820162094 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24b0140d60 con 0x7f24b00b3620 2022-07-20T03:15:46.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24af7fe700 1 --2- 172.21.15.61:0/820162094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00b3620 0x7f24b0096890 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f24a0002f70 tx=0x7f24a0006e50).ready entity=mon.0 client_cookie=c4e6862b614e23e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:46.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24aeffd700 1 -- 172.21.15.61:0/820162094 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24a000d5c0 con 0x7f24b00b3620 2022-07-20T03:15:46.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24aeffd700 1 -- 172.21.15.61:0/820162094 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a0000e90 con 0x7f24b00b3620 2022-07-20T03:15:46.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24aeffd700 1 -- 172.21.15.61:0/820162094 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24a000d910 con 0x7f24b00b3620 2022-07-20T03:15:46.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 -- 172.21.15.61:0/820162094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00b3620 msgr2=0x7f24b0096890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:46.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 --2- 172.21.15.61:0/820162094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00b3620 0x7f24b0096890 secure :-1 s=READY pgs=1081 cs=0 l=1 rev1=1 rx=0x7f24a0002f70 tx=0x7f24a0006e50).stop 2022-07-20T03:15:46.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 -- 172.21.15.61:0/820162094 shutdown_connections 2022-07-20T03:15:46.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 --2- 172.21.15.61:0/820162094 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00b3620 0x7f24b0096890 unknown :-1 s=CLOSED pgs=1081 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:46.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 --2- 172.21.15.61:0/820162094 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b00ac260 0x7f24b00ac640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:46.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 -- 172.21.15.61:0/820162094 >> 172.21.15.61:0/820162094 conn(0x7f24b00b47d0 msgr2=0x7f24b005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:46.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 -- 172.21.15.61:0/820162094 shutdown_connections 2022-07-20T03:15:46.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.556+0000 7f24b6324700 1 -- 172.21.15.61:0/820162094 wait complete. 2022-07-20T03:15:46.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 Processor -- start 2022-07-20T03:15:46.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 -- start start 2022-07-20T03:15:46.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 0x7f24b014fc90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:46.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b00b3620 0x7f24b0150180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:46.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b0141200 0x7f24b0065a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:46.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b0150c50 con 0x7f24b00ac260 2022-07-20T03:15:46.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24b01509b0 con 0x7f24b00b3620 2022-07-20T03:15:46.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f24b00aef70 con 0x7f24b0141200 2022-07-20T03:15:46.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 0x7f24b014fc90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:46.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 0x7f24b014fc90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49120/0 (socket says 172.21.15.61:49120) 2022-07-20T03:15:46.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 -- 172.21.15.61:0/1552389037 learned_addr learned my addr 172.21.15.61:0/1552389037 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:46.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b48c1700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b0141200 0x7f24b0065a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:46.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24af7fe700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b00b3620 0x7f24b0150180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:46.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 -- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b0141200 msgr2=0x7f24b0065a00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:46.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b0141200 0x7f24b0065a00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:46.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 -- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b00b3620 msgr2=0x7f24b0150180 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:46.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b00b3620 0x7f24b0150180 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:46.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 -- 172.21.15.61:0/1552389037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f24a000b1e0 con 0x7f24b00ac260 2022-07-20T03:15:46.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24affff700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 0x7f24b014fc90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f24a400c000 tx=0x7f24a4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:46.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24a4011750 con 0x7f24b00ac260 2022-07-20T03:15:46.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f24a40118b0 con 0x7f24b00ac260 2022-07-20T03:15:46.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.560+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24b00af410 con 0x7f24b00ac260 2022-07-20T03:15:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f24a4011c10 con 0x7f24b00ac260 2022-07-20T03:15:46.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24b00aea60 con 0x7f24b00ac260 2022-07-20T03:15:46.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24b0150810 con 0x7f24b00ac260 2022-07-20T03:15:46.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f24a4011d70 con 0x7f24b00ac260 2022-07-20T03:15:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24ad7fa700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2490053a40 0x7f2490055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:46.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f24a406c310 con 0x7f24b00ac260 2022-07-20T03:15:46.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24af7fe700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2490053a40 0x7f2490055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:46.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.564+0000 7f24af7fe700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2490053a40 0x7f2490055f00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f24a001a000 tx=0x7f24a0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:46.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.568+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f24b0150810 con 0x7f24b00ac260 2022-07-20T03:15:46.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.692+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f24b0150810 con 0x7f24b00ac260 2022-07-20T03:15:46.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.696+0000 7f24ad7fa700 1 -- 172.21.15.61:0/1552389037 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v101) v1 ==== 81+0+33699 (secure 0 0 0) 0x7f24b0150810 con 0x7f24b00ac260 2022-07-20T03:15:46.718 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:46.719 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 101, 2022-07-20T03:15:46.727 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:15:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:15:46.728 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:15:36.690047+0000", 2022-07-20T03:15:46.729 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:46.734 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:15:46.735 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:15:46.736 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:15:46.737 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:15:46.737 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:15:46.738 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:15:46.738 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:15:46.739 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:46.743 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:15:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:15:46.744 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:15:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:15:46.745 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:15:46.747 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 20, 2022-07-20T03:15:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:15:46.748 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:15:46.749 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:15:46.750 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:15:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:15:46.751 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:15:46.752 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:15:46.753 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:15:46.753 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:46.754 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:46.770 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:46.771 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:46.772 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:46.773 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:46.773 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:15:46.774 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:15:46.774 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:15:46.775 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:15:46.775 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:15:46.776 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:46.776 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:46.780 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:46.781 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:46.786 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.787 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:15:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:46.788 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:46.789 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:46.790 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:46.791 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:46.796 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:46.797 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:46.799 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:46.800 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:46.801 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:46.801 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:46.805 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:46.806 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:46.806 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:46.806 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.807 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:46.808 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:46.809 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:46.810 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:15:46.811 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.811 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:46.811 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:15:46.811 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.812 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.812 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:15:46.812 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:15:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:15:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:46.813 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:46.820 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:46.830 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:46.830 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:46.831 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:46.831 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:46.832 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:46.832 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:46.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:46.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:46.833 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:46.834 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:46.834 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:46.835 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:46.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:46.836 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:46.836 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:46.837 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:46.837 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.837 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:15:46.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:46.838 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:46.839 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:46.839 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:46.840 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:46.840 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:46.841 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:46.841 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:46.842 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:46.843 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:46.844 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:46.844 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:46.844 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:46.845 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:46.845 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:46.846 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:46.846 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:46.847 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:46.848 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:46.849 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:46.850 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:46.851 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:46.852 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:46.853 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:15:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.854 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 19, 2022-07-20T03:15:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:15:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:15:34.173179+0000", 2022-07-20T03:15:46.855 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:46.856 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:46.857 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:46.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:46.859 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:46.860 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "100", 2022-07-20T03:15:46.861 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:46.862 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:46.863 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:46.864 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:46.865 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:46.865 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:46.866 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:46.866 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:46.867 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:46.867 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:46.868 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:46.868 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:46.869 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:46.869 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:46.870 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:46.870 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:46.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:46.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:46.872 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:46.872 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:46.873 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:46.873 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:46.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.884 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:46.885 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:46.885 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:46.886 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:46.886 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:46.887 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:46.887 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:46.888 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:46.888 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:46.889 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:46.889 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:46.900 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:46.901 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:15:46.901 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:15:46.902 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:15:46.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.903 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:46.903 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:15:46.904 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:15:46.904 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.905 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.905 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.905 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.906 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 20, 2022-07-20T03:15:46.906 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:15:46.907 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:15:34.697342+0000", 2022-07-20T03:15:46.908 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:46.908 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:46.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:46.909 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:46.909 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:46.910 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:46.910 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:46.911 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:46.911 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:46.911 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:46.912 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:46.912 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:46.913 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:46.913 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:46.913 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:46.914 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:46.914 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:46.915 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:46.915 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:46.916 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:46.916 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.917 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "101", 2022-07-20T03:15:46.917 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:46.918 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:46.918 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:46.919 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:46.919 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:46.920 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:46.920 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:46.921 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:46.921 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:46.922 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:46.922 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:46.923 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:46.923 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:46.924 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:46.924 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:46.925 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:46.925 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:46.926 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:46.926 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:46.927 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:46.927 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:46.928 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:46.928 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:46.928 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:46.929 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:46.929 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:46.930 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.930 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:46.931 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:46.931 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:46.932 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:46.932 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:46.932 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:46.933 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:46.933 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:46.934 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:46.935 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:46.935 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:46.936 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:46.936 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:46.937 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:15:46.937 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:15:46.938 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.938 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.939 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.939 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:46.940 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:15:46.940 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.941 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:46.941 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:15:46.942 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:46.943 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:46.943 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:46.943 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:46.944 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:46.944 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:46.945 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:46.945 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:46.946 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:46.946 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:46.946 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:46.946 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:46.947 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.947 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:46.947 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:15:46.947 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.948 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.948 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.948 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:46.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:15:46.948 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.949 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.949 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:46.949 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.950 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:46.950 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:46.950 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.951 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:46.951 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:15:46.951 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.951 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.952 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.952 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:46.952 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:15:46.952 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.952 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.953 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:46.953 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.953 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:46.954 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:46.954 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.954 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:46.955 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:15:46.955 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.967 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.968 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.968 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:46.969 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:15:46.969 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.970 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.970 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:46.971 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.971 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:46.971 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:46.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.972 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:46.972 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:15:46.972 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.972 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.973 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.973 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:46.973 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:15:46.973 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:46.974 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.974 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:46.975 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.975 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:15:46.975 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:15:46.975 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:15:46.976 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:15:46.976 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:46.976 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:46.976 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:46.977 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:46.977 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.977 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.977 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:46.978 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:15:46.978 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:46.979 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:46.979 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:46.980 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:46.980 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:46.981 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:46.982 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:46.982 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:46.983 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:46.983 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:46.984 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:46.984 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:46.985 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.985 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:46.985 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:15:46.986 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:46.986 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.987 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.987 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:46.987 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:15:46.988 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:46.988 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:46.989 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:46.989 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.989 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:46.990 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:46.990 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.991 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:46.991 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:15:46.992 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:46.992 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:46.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:46.998 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:46.999 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:15:46.999 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:47.000 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.000 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.000 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.001 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.001 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.002 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.002 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:15:47.003 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:47.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.005 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.005 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:15:47.006 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:47.006 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.007 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.007 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.008 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.008 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.009 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.009 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.010 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:15:47.010 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:47.010 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.011 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.011 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.012 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:15:47.012 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:47.012 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.013 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.013 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.014 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:15:47.014 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:15:47.015 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:15:47.015 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:15:47.016 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.016 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.017 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.017 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.024 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.025 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:47.025 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:15:47.026 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:47.026 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:47.027 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:47.027 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:47.028 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:47.028 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:47.029 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:47.029 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-20T03:15:47.030 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:47.030 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:47.031 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:47.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.032 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.033 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:15:47.033 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.036 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:15:47.036 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.037 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.037 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.038 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.038 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:47.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.039 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:15:47.041 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:15:47.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.044 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.044 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.045 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.046 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:15:47.048 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.049 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:15:47.050 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.050 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.051 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.051 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.052 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.053 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.053 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:15:47.054 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.054 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.054 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.055 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.055 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:15:47.056 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:47.056 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.057 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.057 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.057 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:15:47.058 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:15:47.059 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:15:47.059 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:15:47.060 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.060 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.062 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.065 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.066 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:47.067 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:15:47.067 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:47.067 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:47.069 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:47.070 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:47.071 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:47.071 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:47.071 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:47.073 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:47.074 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:47.075 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:47.075 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:47.076 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.076 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.077 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.077 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:15:47.078 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.078 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.079 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.079 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.080 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:15:47.082 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.083 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.083 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.084 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:47.084 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.085 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.085 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.086 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:15:47.086 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.087 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.088 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:15:47.088 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.089 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.089 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.090 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.090 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.091 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.091 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.091 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.092 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:15:47.092 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.093 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.093 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.094 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.094 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:15:47.095 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.095 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.095 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.096 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.096 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.098 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:15:47.099 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.100 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.101 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.101 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:15:47.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:47.102 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.102 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.103 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.103 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:15:47.104 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:15:47.104 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:15:47.105 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:15:47.105 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.105 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.106 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.106 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.107 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.107 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.108 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:47.108 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:15:47.109 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:47.109 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:47.109 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:47.110 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:47.110 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:47.111 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:47.111 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:47.112 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-20T03:15:47.113 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:47.113 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:47.114 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:47.115 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.115 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.116 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:15:47.116 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.117 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.118 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:15:47.119 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.120 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.120 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:47.121 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.122 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.122 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.123 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:15:47.123 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.124 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.124 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.125 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:15:47.126 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.126 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.127 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.127 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.128 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.128 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.129 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.130 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:15:47.130 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.131 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.131 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.132 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.132 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:15:47.132 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.133 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.133 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.134 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.134 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.135 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.136 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:15:47.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.137 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.138 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:15:47.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:47.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.141 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:15:47.142 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:15:47.142 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:15:47.143 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:15:47.143 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.144 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.144 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.145 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.146 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:47.147 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:15:47.147 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:47.148 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:47.148 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:47.149 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:47.149 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:47.150 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:47.150 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:47.151 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-20T03:15:47.151 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:47.152 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:47.152 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:47.152 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.153 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.153 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.154 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:15:47.155 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.155 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.156 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.157 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:15:47.157 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.158 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.158 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.159 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.159 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:47.160 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.160 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.161 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.161 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:15:47.161 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.166 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.167 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.167 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:15:47.168 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.169 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.169 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.171 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.176 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:15:47.177 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.177 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.178 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:15:47.179 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.180 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.180 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.181 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.181 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.182 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.183 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.183 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:15:47.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.184 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.186 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.186 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.187 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:15:47.188 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:47.188 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.189 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.189 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.190 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:15:47.191 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:15:47.191 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:15:47.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:15:47.193 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.193 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.194 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.194 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.196 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.197 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:47.198 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:15:47.199 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:47.200 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:47.200 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:47.200 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:47.201 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:47.201 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:47.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:47.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:47.202 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:47.202 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:47.202 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:47.202 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.203 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.206 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:15:47.207 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.207 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.208 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:15:47.208 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.209 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.209 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.210 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:47.210 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:15:47.211 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.211 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.212 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.212 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:15:47.212 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.213 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.213 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.214 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.214 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.215 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:15:47.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.216 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:15:47.216 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.217 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.217 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.217 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.218 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.218 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.218 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.219 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:15:47.219 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.219 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.220 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:15:47.220 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:47.221 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.221 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.221 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.221 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:15:47.222 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:15:47.222 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:15:47.222 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:15:47.223 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.223 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.223 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.223 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.224 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.224 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:47.224 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:15:47.225 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:47.225 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:47.225 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:47.226 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:47.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:47.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:47.226 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:47.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:47.227 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:47.227 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:47.227 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:47.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.228 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.228 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:15:47.230 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.230 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.231 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:15:47.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.233 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.233 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.234 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:47.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.235 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.235 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:15:47.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.238 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:15:47.239 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.240 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.240 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.241 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.241 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:47.242 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.242 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.243 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:15:47.244 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:15:47.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.247 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.247 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.248 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:47.249 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:47.249 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.250 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:47.250 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:15:47.251 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.253 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:47.254 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:15:47.254 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:47.255 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.255 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.256 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:15:47.257 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:15:47.257 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:15:47.258 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:15:47.258 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:47.259 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:47.259 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:47.260 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:47.260 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.261 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:47.261 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:15:47.262 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.262 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:47.263 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.263 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.264 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.265 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.265 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.266 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:15:47.267 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.267 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.268 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.268 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:47.269 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.269 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.269 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.270 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.272 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.272 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:15:47.273 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.273 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.274 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:47.274 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.275 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.275 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.275 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.276 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.276 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:15:47.277 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.278 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:47.279 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.279 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.280 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.280 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.281 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:15:47.282 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.282 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.283 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:47.283 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.284 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.284 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.285 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.285 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.286 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:15:47.286 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.287 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.288 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:47.288 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.288 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.289 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.289 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.290 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.290 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:15:47.291 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.292 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:47.293 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.293 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.293 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.294 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.295 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:15:47.296 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.296 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:47.297 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:47.297 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:47.298 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:47.298 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:47.299 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:47.299 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:47.299 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:15:47.300 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:47.300 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.301 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:47.301 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:15:47.301 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:15:47.302 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:15:47.302 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:15:47.303 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:15:47.303 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:15:47.304 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:15:47.304 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:15:47.304 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:15:47.305 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:15:47.305 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:15:47.306 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:15:47.306 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:15:47.307 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:47.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:47.307 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:15:47.308 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:15:47.308 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:15:47.309 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:15:47.309 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:15:47.310 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:47.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.696+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2490053a40 msgr2=0x7f2490055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.696+0000 7f24b6324700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2490053a40 0x7f2490055f00 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7f24a001a000 tx=0x7f24a0014000).stop 2022-07-20T03:15:47.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.696+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 msgr2=0x7f24b014fc90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.696+0000 7f24b6324700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 0x7f24b014fc90 secure :-1 s=READY pgs=1082 cs=0 l=1 rev1=1 rx=0x7f24a400c000 tx=0x7f24a4006df0).stop 2022-07-20T03:15:47.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 shutdown_connections 2022-07-20T03:15:47.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2490053a40 0x7f2490055f00 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f24b00b3620 0x7f24b0150180 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f24b0141200 0x7f24b0065a00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 --2- 172.21.15.61:0/1552389037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f24b00ac260 0x7f24b014fc90 unknown :-1 s=CLOSED pgs=1082 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 >> 172.21.15.61:0/1552389037 conn(0x7f24b00b47d0 msgr2=0x7f24b013dea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:47.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 shutdown_connections 2022-07-20T03:15:47.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:46.700+0000 7f24b6324700 1 -- 172.21.15.61:0/1552389037 wait complete. 2022-07-20T03:15:47.321 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:47.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.432+0000 7f6f5de0c700 1 Processor -- start 2022-07-20T03:15:47.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.432+0000 7f6f5de0c700 1 -- start start 2022-07-20T03:15:47.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5de0c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f581424e0 0x7f6f580b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5de0c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6f580aea10 0x7f6f580ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5de0c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6f581433d0 con 0x7f6f580b0570 2022-07-20T03:15:47.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5de0c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6f58143b10 con 0x7f6f580aea10 2022-07-20T03:15:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5de0c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58143670 con 0x7f6f581424e0 2022-07-20T03:15:47.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f57fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f581424e0 0x7f6f580b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6f580aea10 0x7f6f580ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6f580aea10 0x7f6f580ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58806/0 (socket says 172.21.15.61:58806) 2022-07-20T03:15:47.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f57fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f581424e0 0x7f6f580b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35798/0 (socket says 172.21.15.61:35798) 2022-07-20T03:15:47.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 -- 172.21.15.61:0/3776884545 learned_addr learned my addr 172.21.15.61:0/3776884545 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:47.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5ce0a700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f6f580b0570 0x7f6f58141dd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:54226/0 (socket says 172.21.15.61:54226) 2022-07-20T03:15:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 -- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f581424e0 msgr2=0x7f6f580b0950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:47.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 --2- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f581424e0 0x7f6f580b0950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 -- 172.21.15.61:0/3776884545 >> v1:172.21.15.117:6789/0 conn(0x7f6f580b0570 legacy=0x7f6f58141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 -- 172.21.15.61:0/3776884545 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f580b1010 con 0x7f6f580aea10 2022-07-20T03:15:47.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f5d60b700 1 --2- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f580ad350 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f6f500032f0 tx=0x7f6f50009290).ready entity=mon.1 client_cookie=24e2b196aa680223 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f577fe700 1 -- 172.21.15.61:0/3776884545 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f5000b8a0 con 0x7f6f580aea10 2022-07-20T03:15:47.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f577fe700 1 -- 172.21.15.61:0/3776884545 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f5000ba00 con 0x7f6f580aea10 2022-07-20T03:15:47.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.436+0000 7f6f577fe700 1 -- 172.21.15.61:0/3776884545 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f5000bb70 con 0x7f6f580aea10 2022-07-20T03:15:47.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 -- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 msgr2=0x7f6f580ad350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f580ad350 secure :-1 s=READY pgs=492 cs=0 l=1 rev1=1 rx=0x7f6f500032f0 tx=0x7f6f50009290).stop 2022-07-20T03:15:47.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 -- 172.21.15.61:0/3776884545 shutdown_connections 2022-07-20T03:15:47.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f580ad350 unknown :-1 s=CLOSED pgs=492 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/3776884545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f581424e0 0x7f6f580b0950 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 -- 172.21.15.61:0/3776884545 >> 172.21.15.61:0/3776884545 conn(0x7f6f580b9360 msgr2=0x7f6f580b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:47.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 -- 172.21.15.61:0/3776884545 shutdown_connections 2022-07-20T03:15:47.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 -- 172.21.15.61:0/3776884545 wait complete. 2022-07-20T03:15:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 Processor -- start 2022-07-20T03:15:47.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.440+0000 7f6f5de0c700 1 -- start start 2022-07-20T03:15:47.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f581126f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f580b0570 0x7f6f58155520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 0x7f6f58155a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58114b90 con 0x7f6f581424e0 2022-07-20T03:15:47.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f58114e30 con 0x7f6f580aea10 2022-07-20T03:15:47.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6f580b3500 con 0x7f6f580b0570 2022-07-20T03:15:47.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 0x7f6f58155a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5ce0a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f581126f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 0x7f6f58155a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49134/0 (socket says 172.21.15.61:49134) 2022-07-20T03:15:47.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 -- 172.21.15.61:0/2639105410 learned_addr learned my addr 172.21.15.61:0/2639105410 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:47.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f57fff700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f580b0570 0x7f6f58155520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 -- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f580b0570 msgr2=0x7f6f58155520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f580b0570 0x7f6f58155520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 -- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 msgr2=0x7f6f581126f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f581126f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 -- 172.21.15.61:0/2639105410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f50009530 con 0x7f6f581424e0 2022-07-20T03:15:47.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5d60b700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 0x7f6f58155a10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f6f500032f0 tx=0x7f6f50016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:47.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f5000b8a0 con 0x7f6f581424e0 2022-07-20T03:15:47.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6f50021630 con 0x7f6f581424e0 2022-07-20T03:15:47.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f58112f60 con 0x7f6f581424e0 2022-07-20T03:15:47.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6f50021990 con 0x7f6f581424e0 2022-07-20T03:15:47.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.444+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f580b39a0 con 0x7f6f581424e0 2022-07-20T03:15:47.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.448+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f580b36c0 con 0x7f6f581424e0 2022-07-20T03:15:47.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.448+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6f50027020 con 0x7f6f581424e0 2022-07-20T03:15:47.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.448+0000 7f6f55ffb700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f38053920 0x7f6f38055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.448+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f6f5006d150 con 0x7f6f581424e0 2022-07-20T03:15:47.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.448+0000 7f6f5ce0a700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f38053920 0x7f6f38055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.448+0000 7f6f5ce0a700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f38053920 0x7f6f38055de0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f6f4c00d000 tx=0x7f6f4c007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:47.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.456+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6f50044160 con 0x7f6f581424e0 2022-07-20T03:15:47.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.612+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f6f580ace30 con 0x7f6f581424e0 2022-07-20T03:15:47.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.612+0000 7f6f55ffb700 1 -- 172.21.15.61:0/2639105410 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 56 v56) v1 ==== 94+0+2143 (secure 0 0 0) 0x7f6f50015020 con 0x7f6f581424e0 2022-07-20T03:15:47.620 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:47.620 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":56,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54}],"filesystems":[{"mdsmap":{"epoch":56,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:35.686626+0000","modified":"2022-07-20T03:15:36.689415+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7335},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7335":{"gid":7335,"name":"a","rank":0,"incarnation":55,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/2834789489","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":2834789489},{"type":"v1","addr":"172.21.15.61:6833","nonce":2834789489}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T03:15:47.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f38053920 msgr2=0x7f6f38055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f38053920 0x7f6f38055de0 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7f6f4c00d000 tx=0x7f6f4c007b40).stop 2022-07-20T03:15:47.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 msgr2=0x7f6f58155a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 0x7f6f58155a10 secure :-1 s=READY pgs=1083 cs=0 l=1 rev1=1 rx=0x7f6f500032f0 tx=0x7f6f50016000).stop 2022-07-20T03:15:47.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 shutdown_connections 2022-07-20T03:15:47.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6f38053920 0x7f6f38055de0 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6f581424e0 0x7f6f58155a10 unknown :-1 s=CLOSED pgs=1083 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6f580b0570 0x7f6f58155520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 --2- 172.21.15.61:0/2639105410 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6f580aea10 0x7f6f581126f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 >> 172.21.15.61:0/2639105410 conn(0x7f6f580b9360 msgr2=0x7f6f5813f1e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:47.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 shutdown_connections 2022-07-20T03:15:47.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.616+0000 7f6f5de0c700 1 -- 172.21.15.61:0/2639105410 wait complete. 2022-07-20T03:15:47.628 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 56 2022-07-20T03:15:47.637 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:15:47.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.748+0000 7feb34bfb700 1 Processor -- start 2022-07-20T03:15:47.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.748+0000 7feb34bfb700 1 -- start start 2022-07-20T03:15:47.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7feb300af3e0 0x7feb30146780 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb3005d210 0x7feb3014a360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb30146c70 con 0x7feb300b5990 2022-07-20T03:15:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300ad050 con 0x7feb3005d210 2022-07-20T03:15:47.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb300b17e0 con 0x7feb300af3e0 2022-07-20T03:15:47.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb3005d210 0x7feb3014a360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb3005d210 0x7feb3014a360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49140/0 (socket says 172.21.15.61:49140) 2022-07-20T03:15:47.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 -- 172.21.15.61:0/2621096150 learned_addr learned my addr 172.21.15.61:0/2621096150 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:47.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2effd700 1 --2- 172.21.15.61:0/2621096150 >> v2:172.21.15.117:3300/0 conn(0x7feb300af3e0 0x7feb30146780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 -- 172.21.15.61:0/2621096150 >> v2:172.21.15.117:3300/0 conn(0x7feb300af3e0 msgr2=0x7feb30146780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:47.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 --2- 172.21.15.61:0/2621096150 >> v2:172.21.15.117:3300/0 conn(0x7feb300af3e0 0x7feb30146780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 -- 172.21.15.61:0/2621096150 >> v1:172.21.15.117:6789/0 conn(0x7feb300b5990 legacy=0x7feb30146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 -- 172.21.15.61:0/2621096150 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb300b5d70 con 0x7feb3005d210 2022-07-20T03:15:47.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2ffff700 1 --2- 172.21.15.61:0/2621096150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb3005d210 0x7feb3014a360 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7feb28002f70 tx=0x7feb28006eb0).ready entity=mon.0 client_cookie=190fcf0d55861dc3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:47.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2e7fc700 1 -- 172.21.15.61:0/2621096150 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb2800d5c0 con 0x7feb3005d210 2022-07-20T03:15:47.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2e7fc700 1 -- 172.21.15.61:0/2621096150 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb28000e90 con 0x7feb3005d210 2022-07-20T03:15:47.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb2e7fc700 1 -- 172.21.15.61:0/2621096150 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb2800d950 con 0x7feb3005d210 2022-07-20T03:15:47.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- 172.21.15.61:0/2621096150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb3005d210 msgr2=0x7feb3014a360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 --2- 172.21.15.61:0/2621096150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb3005d210 0x7feb3014a360 secure :-1 s=READY pgs=1084 cs=0 l=1 rev1=1 rx=0x7feb28002f70 tx=0x7feb28006eb0).stop 2022-07-20T03:15:47.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- 172.21.15.61:0/2621096150 shutdown_connections 2022-07-20T03:15:47.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 --2- 172.21.15.61:0/2621096150 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb3005d210 0x7feb3014a360 unknown :-1 s=CLOSED pgs=1084 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 --2- 172.21.15.61:0/2621096150 >> v2:172.21.15.117:3300/0 conn(0x7feb300af3e0 0x7feb30146780 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- 172.21.15.61:0/2621096150 >> 172.21.15.61:0/2621096150 conn(0x7feb30063480 msgr2=0x7feb300b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:47.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- 172.21.15.61:0/2621096150 shutdown_connections 2022-07-20T03:15:47.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- 172.21.15.61:0/2621096150 wait complete. 2022-07-20T03:15:47.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 Processor -- start 2022-07-20T03:15:47.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.752+0000 7feb34bfb700 1 -- start start 2022-07-20T03:15:47.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb3005d210 0x7feb30147140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb300af3e0 0x7feb30159c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 0x7feb30066d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb3014b030 con 0x7feb300b5990 2022-07-20T03:15:47.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb30065c80 con 0x7feb300af3e0 2022-07-20T03:15:47.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb300645b0 con 0x7feb3005d210 2022-07-20T03:15:47.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 0x7feb30066d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 0x7feb30066d80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49148/0 (socket says 172.21.15.61:49148) 2022-07-20T03:15:47.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 -- 172.21.15.61:0/3512582936 learned_addr learned my addr 172.21.15.61:0/3512582936 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:47.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2effd700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb300af3e0 0x7feb30159c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2f7fe700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb3005d210 0x7feb30147140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 -- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb3005d210 msgr2=0x7feb30147140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb3005d210 0x7feb30147140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 -- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb300af3e0 msgr2=0x7feb30159c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:47.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb300af3e0 0x7feb30159c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:47.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 -- 172.21.15.61:0/3512582936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb2800b1e0 con 0x7feb300b5990 2022-07-20T03:15:47.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2effd700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb300af3e0 0x7feb30159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:47.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2ffff700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 0x7feb30066d80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7feb28002f70 tx=0x7feb28015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:47.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb28011710 con 0x7feb300b5990 2022-07-20T03:15:47.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb3014acb0 con 0x7feb300b5990 2022-07-20T03:15:47.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb28011870 con 0x7feb300b5990 2022-07-20T03:15:47.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb28011bd0 con 0x7feb300b5990 2022-07-20T03:15:47.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb3014aa50 con 0x7feb300b5990 2022-07-20T03:15:47.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.756+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb30066a90 con 0x7feb300b5990 2022-07-20T03:15:47.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.760+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feb2801a040 con 0x7feb300b5990 2022-07-20T03:15:47.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.760+0000 7feb2cff9700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb14053a40 0x7feb14055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:47.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.760+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(101..101 src has 1..101) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feb2806cdb0 con 0x7feb300b5990 2022-07-20T03:15:47.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.760+0000 7feb2f7fe700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb14053a40 0x7feb14055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:47.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.760+0000 7feb2f7fe700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb14053a40 0x7feb14055f00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7feb2000b000 tx=0x7feb20009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:47.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.768+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feb28043cb0 con 0x7feb300b5990 2022-07-20T03:15:47.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:47.928+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7feb3014ae40 con 0x7feb300b5990 2022-07-20T03:15:48.113 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:15:48.108+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:15:48.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.124+0000 7feb2cff9700 1 -- 172.21.15.61:0/3512582936 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v57) v1 ==== 161+0+0 (secure 0 0 0) 0x7feb28023020 con 0x7feb300b5990 2022-07-20T03:15:48.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.124+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb14053a40 msgr2=0x7feb14055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.124+0000 7feb34bfb700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb14053a40 0x7feb14055f00 secure :-1 s=READY pgs=525 cs=0 l=1 rev1=1 rx=0x7feb2000b000 tx=0x7feb20009040).stop 2022-07-20T03:15:48.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.124+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 msgr2=0x7feb30066d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.124+0000 7feb34bfb700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 0x7feb30066d80 secure :-1 s=READY pgs=1085 cs=0 l=1 rev1=1 rx=0x7feb28002f70 tx=0x7feb28015000).stop 2022-07-20T03:15:48.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 shutdown_connections 2022-07-20T03:15:48.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb14053a40 0x7feb14055f00 unknown :-1 s=CLOSED pgs=525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb300af3e0 0x7feb30159c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb300b5990 0x7feb30066d80 unknown :-1 s=CLOSED pgs=1085 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 --2- 172.21.15.61:0/3512582936 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb3005d210 0x7feb30147140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 >> 172.21.15.61:0/3512582936 conn(0x7feb30063480 msgr2=0x7feb300b1e50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:48.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 shutdown_connections 2022-07-20T03:15:48.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.128+0000 7feb34bfb700 1 -- 172.21.15.61:0/3512582936 wait complete. 2022-07-20T03:15:48.134 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:15:48.147 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:48.153 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:15:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 Processor -- start 2022-07-20T03:15:48.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 -- start start 2022-07-20T03:15:48.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0928061a50 0x7f0928061e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f092809aa30 0x7f09280b2470 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f09280acba0 con 0x7f092805ce60 2022-07-20T03:15:48.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09280b1d30 con 0x7f092809aa30 2022-07-20T03:15:48.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092eb0d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f09280b1fd0 con 0x7f0928061a50 2022-07-20T03:15:48.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092d30a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f092809aa30 0x7f09280b2470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092e30c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f092805ce60 0x7f092809b010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45766/0 (socket says 172.21.15.61:45766) 2022-07-20T03:15:48.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092e30c700 1 -- 172.21.15.61:0/1646627626 learned_addr learned my addr 172.21.15.61:0/1646627626 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:48.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.252+0000 7f092db0b700 1 --2- 172.21.15.61:0/1646627626 >> v2:172.21.15.117:3300/0 conn(0x7f0928061a50 0x7f0928061e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092d30a700 1 -- 172.21.15.61:0/1646627626 >> v2:172.21.15.117:3300/0 conn(0x7f0928061a50 msgr2=0x7f0928061e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:48.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092d30a700 1 --2- 172.21.15.61:0/1646627626 >> v2:172.21.15.117:3300/0 conn(0x7f0928061a50 0x7f0928061e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092d30a700 1 -- 172.21.15.61:0/1646627626 >> v1:172.21.15.61:6789/0 conn(0x7f092805ce60 legacy=0x7f092809b010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092d30a700 1 -- 172.21.15.61:0/1646627626 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f092809b720 con 0x7f092809aa30 2022-07-20T03:15:48.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092d30a700 1 --2- 172.21.15.61:0/1646627626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f092809aa30 0x7f09280b2470 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f0928099800 tx=0x7f0918006df0).ready entity=mon.0 client_cookie=9a2ab5166b0545af server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:48.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092cb09700 1 -- 172.21.15.61:0/1646627626 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091800d5c0 con 0x7f092809aa30 2022-07-20T03:15:48.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092cb09700 1 -- 172.21.15.61:0/1646627626 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0918000e90 con 0x7f092809aa30 2022-07-20T03:15:48.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092cb09700 1 -- 172.21.15.61:0/1646627626 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091800d910 con 0x7f092809aa30 2022-07-20T03:15:48.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- 172.21.15.61:0/1646627626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f092809aa30 msgr2=0x7f09280b2470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 --2- 172.21.15.61:0/1646627626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f092809aa30 0x7f09280b2470 secure :-1 s=READY pgs=1090 cs=0 l=1 rev1=1 rx=0x7f0928099800 tx=0x7f0918006df0).stop 2022-07-20T03:15:48.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- 172.21.15.61:0/1646627626 shutdown_connections 2022-07-20T03:15:48.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 --2- 172.21.15.61:0/1646627626 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f092809aa30 0x7f09280b2470 unknown :-1 s=CLOSED pgs=1090 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 --2- 172.21.15.61:0/1646627626 >> v2:172.21.15.117:3300/0 conn(0x7f0928061a50 0x7f0928061e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- 172.21.15.61:0/1646627626 >> 172.21.15.61:0/1646627626 conn(0x7f09280b2ca0 msgr2=0x7f09280b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:48.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- 172.21.15.61:0/1646627626 shutdown_connections 2022-07-20T03:15:48.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- 172.21.15.61:0/1646627626 wait complete. 2022-07-20T03:15:48.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 Processor -- start 2022-07-20T03:15:48.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- start start 2022-07-20T03:15:48.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f092805ce60 0x7f0928142250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 0x7f0928063c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092809aa30 0x7f0928064130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f09280af310 con 0x7f0928061a50 2022-07-20T03:15:48.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0928143f40 con 0x7f092809aa30 2022-07-20T03:15:48.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.256+0000 7f092eb0d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0928142e00 con 0x7f092805ce60 2022-07-20T03:15:48.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 0x7f0928063c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 0x7f0928063c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49172/0 (socket says 172.21.15.61:49172) 2022-07-20T03:15:48.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 -- 172.21.15.61:0/2937246603 learned_addr learned my addr 172.21.15.61:0/2937246603 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:48.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092e30c700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092809aa30 0x7f0928064130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092db0b700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f092805ce60 0x7f0928142250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 -- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f092805ce60 msgr2=0x7f0928142250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f092805ce60 0x7f0928142250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 -- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092809aa30 msgr2=0x7f0928064130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092809aa30 0x7f0928064130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 -- 172.21.15.61:0/2937246603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f091800b180 con 0x7f0928061a50 2022-07-20T03:15:48.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092d30a700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 0x7f0928063c40 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f0928099800 tx=0x7f0918014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:48.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0918013e90 con 0x7f0928061a50 2022-07-20T03:15:48.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0918002f70 con 0x7f0928061a50 2022-07-20T03:15:48.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f09280af540 con 0x7f0928061a50 2022-07-20T03:15:48.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f091801e5f0 con 0x7f0928061a50 2022-07-20T03:15:48.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09281430a0 con 0x7f0928061a50 2022-07-20T03:15:48.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f092805cbb0 con 0x7f0928061a50 2022-07-20T03:15:48.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f091801c070 con 0x7f0928061a50 2022-07-20T03:15:48.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f0916ffd700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0908053a40 0x7f0908055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.260+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f091806ca10 con 0x7f0928061a50 2022-07-20T03:15:48.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.264+0000 7f092db0b700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0908053a40 0x7f0908055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.264+0000 7f092db0b700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0908053a40 0x7f0908055f00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f092400b000 tx=0x7f0924006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:48.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.268+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0918043880 con 0x7f0928061a50 2022-07-20T03:15:48.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.432+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f09280574e0 con 0x7f0928061a50 2022-07-20T03:15:48.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f0916ffd700 1 -- 172.21.15.61:0/2937246603 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 57 v57) v1 ==== 94+0+1783 (secure 0 0 0) 0x7f0918023070 con 0x7f0928061a50 2022-07-20T03:15:48.443 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:48.443 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":57,"default_fscid":9,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54}],"filesystems":[{"mdsmap":{"epoch":57,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:35.686626+0000","modified":"2022-07-20T03:15:48.113400+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":102,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[20],"metadata_pool":19,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":9}]} 2022-07-20T03:15:48.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0908053a40 msgr2=0x7f0908055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0908053a40 0x7f0908055f00 secure :-1 s=READY pgs=526 cs=0 l=1 rev1=1 rx=0x7f092400b000 tx=0x7f0924006df0).stop 2022-07-20T03:15:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 msgr2=0x7f0928063c40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 0x7f0928063c40 secure :-1 s=READY pgs=1091 cs=0 l=1 rev1=1 rx=0x7f0928099800 tx=0x7f0918014000).stop 2022-07-20T03:15:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 shutdown_connections 2022-07-20T03:15:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0908053a40 0x7f0908055f00 unknown :-1 s=CLOSED pgs=526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f092809aa30 0x7f0928064130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0928061a50 0x7f0928063c40 unknown :-1 s=CLOSED pgs=1091 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 --2- 172.21.15.61:0/2937246603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f092805ce60 0x7f0928142250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 >> 172.21.15.61:0/2937246603 conn(0x7f09280b2ca0 msgr2=0x7f09280b4310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:48.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 shutdown_connections 2022-07-20T03:15:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.436+0000 7f092eb0d700 1 -- 172.21.15.61:0/2937246603 wait complete. 2022-07-20T03:15:48.460 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 57 2022-07-20T03:15:48.464 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:15:48.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.576+0000 7f3736b4b700 1 Processor -- start 2022-07-20T03:15:48.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.576+0000 7f3736b4b700 1 -- start start 2022-07-20T03:15:48.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3730148580 0x7f373014a970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3730096930 con 0x7f37300acc10 2022-07-20T03:15:48.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37300abde0 con 0x7f37300acff0 2022-07-20T03:15:48.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3730096c80 con 0x7f3730148580 2022-07-20T03:15:48.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3735b49700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f37300acc10 0x7f3730095b10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45776/0 (socket says 172.21.15.61:45776) 2022-07-20T03:15:48.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3735b49700 1 -- 172.21.15.61:0/2048263753 learned_addr learned my addr 172.21.15.61:0/2048263753 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:48.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f373634a700 1 --2- 172.21.15.61:0/2048263753 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3730148580 0x7f373014a970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072180901 0 0) 0x7f3730096930 con 0x7f37300acc10 2022-07-20T03:15:48.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3718003260 con 0x7f37300acc10 2022-07-20T03:15:48.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3495492285 0 0) 0x7f37300abde0 con 0x7f37300acff0 2022-07-20T03:15:48.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3730096930 con 0x7f37300acff0 2022-07-20T03:15:48.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f373634a700 1 -- 172.21.15.61:0/2048263753 >> v1:172.21.15.117:6789/0 conn(0x7f37300acff0 legacy=0x7f3730096220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f373634a700 1 -- 172.21.15.61:0/2048263753 >> v1:172.21.15.61:6789/0 conn(0x7f37300acc10 legacy=0x7f3730095b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f373634a700 1 -- 172.21.15.61:0/2048263753 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3730098b20 con 0x7f3730148580 2022-07-20T03:15:48.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f373634a700 1 --2- 172.21.15.61:0/2048263753 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3730148580 0x7f373014a970 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f37280032f0 tx=0x7f37280091d0).ready entity=mon.2 client_cookie=412ebd75c2776fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:48.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f372800b640 con 0x7f3730148580 2022-07-20T03:15:48.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f372800b7a0 con 0x7f3730148580 2022-07-20T03:15:48.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3734b47700 1 -- 172.21.15.61:0/2048263753 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f372800b910 con 0x7f3730148580 2022-07-20T03:15:48.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- 172.21.15.61:0/2048263753 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3730148580 msgr2=0x7f373014a970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 --2- 172.21.15.61:0/2048263753 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3730148580 0x7f373014a970 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f37280032f0 tx=0x7f37280091d0).stop 2022-07-20T03:15:48.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- 172.21.15.61:0/2048263753 shutdown_connections 2022-07-20T03:15:48.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 --2- 172.21.15.61:0/2048263753 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3730148580 0x7f373014a970 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- 172.21.15.61:0/2048263753 >> 172.21.15.61:0/2048263753 conn(0x7f37300b6210 msgr2=0x7f37300b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:48.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- 172.21.15.61:0/2048263753 shutdown_connections 2022-07-20T03:15:48.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.580+0000 7f3736b4b700 1 -- 172.21.15.61:0/2048263753 wait complete. 2022-07-20T03:15:48.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 Processor -- start 2022-07-20T03:15:48.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 -- start start 2022-07-20T03:15:48.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37300acc10 0x7f373005c960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37300acff0 0x7f373013d670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 0x7f373013db60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3730140320 con 0x7f3730148580 2022-07-20T03:15:48.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3730093ba0 con 0x7f37300acc10 2022-07-20T03:15:48.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f373013f470 con 0x7f37300acff0 2022-07-20T03:15:48.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 0x7f373013db60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 0x7f373013db60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49188/0 (socket says 172.21.15.61:49188) 2022-07-20T03:15:48.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 -- 172.21.15.61:0/4253495773 learned_addr learned my addr 172.21.15.61:0/4253495773 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:48.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3735b49700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37300acc10 0x7f373005c960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3735348700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37300acff0 0x7f373013d670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 -- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37300acff0 msgr2=0x7f373013d670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37300acff0 0x7f373013d670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 -- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37300acc10 msgr2=0x7f373005c960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:48.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37300acc10 0x7f373005c960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:48.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 -- 172.21.15.61:0/4253495773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3728009470 con 0x7f3730148580 2022-07-20T03:15:48.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f373634a700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 0x7f373013db60 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f37280032f0 tx=0x7f3728016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:48.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f372800b910 con 0x7f3730148580 2022-07-20T03:15:48.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3728011de0 con 0x7f3730148580 2022-07-20T03:15:48.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3730095260 con 0x7f3730148580 2022-07-20T03:15:48.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f372800b640 con 0x7f3730148580 2022-07-20T03:15:48.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.584+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f373013f8d0 con 0x7f3730148580 2022-07-20T03:15:48.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.588+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37301413b0 con 0x7f3730148580 2022-07-20T03:15:48.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.588+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f372800f070 con 0x7f3730148580 2022-07-20T03:15:48.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.588+0000 7f371effd700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37100539b0 0x7f3710055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:48.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.588+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f372806c7e0 con 0x7f3730148580 2022-07-20T03:15:48.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.588+0000 7f3735b49700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37100539b0 0x7f3710055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:48.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.592+0000 7f3735b49700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37100539b0 0x7f3710055e70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f372c00d000 tx=0x7f372c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:48.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.592+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f37301413b0 con 0x7f3730148580 2022-07-20T03:15:48.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:48.756+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f37301413b0 con 0x7f3730148580 2022-07-20T03:15:49.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.124+0000 7f371effd700 1 -- 172.21.15.61:0/4253495773 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v58) v1 ==== 104+0+0 (secure 0 0 0) 0x7f37301413b0 con 0x7f3730148580 2022-07-20T03:15:49.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37100539b0 msgr2=0x7f3710055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:49.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37100539b0 0x7f3710055e70 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7f372c00d000 tx=0x7f372c009040).stop 2022-07-20T03:15:49.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 msgr2=0x7f373013db60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:49.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 0x7f373013db60 secure :-1 s=READY pgs=1093 cs=0 l=1 rev1=1 rx=0x7f37280032f0 tx=0x7f3728016000).stop 2022-07-20T03:15:49.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 shutdown_connections 2022-07-20T03:15:49.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37100539b0 0x7f3710055e70 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3730148580 0x7f373013db60 unknown :-1 s=CLOSED pgs=1093 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37300acff0 0x7f373013d670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 --2- 172.21.15.61:0/4253495773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37300acc10 0x7f373005c960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.128+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 >> 172.21.15.61:0/4253495773 conn(0x7f37300b6210 msgr2=0x7f3730141990 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:49.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.132+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 shutdown_connections 2022-07-20T03:15:49.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.132+0000 7f3736b4b700 1 -- 172.21.15.61:0/4253495773 wait complete. 2022-07-20T03:15:49.151 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:15:49.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.260+0000 7fac115b2700 1 Processor -- start 2022-07-20T03:15:49.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.260+0000 7fac115b2700 1 -- start start 2022-07-20T03:15:49.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fac0c0b6840 0x7fac0c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:49.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fac0c0b4bc0 0x7fac0c141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:49.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fac0c066b10 con 0x7fac0c0b3cf0 2022-07-20T03:15:49.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c0b6c20 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fac0c0aead0 con 0x7fac0c0b6840 2022-07-20T03:15:49.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fac0c0b4bc0 0x7fac0c141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:49.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fac0c0b4bc0 0x7fac0c141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49198/0 (socket says 172.21.15.61:49198) 2022-07-20T03:15:49.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 -- 172.21.15.61:0/3488754216 learned_addr learned my addr 172.21.15.61:0/3488754216 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:49.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac0bfff700 1 --2- 172.21.15.61:0/3488754216 >> v2:172.21.15.117:3300/0 conn(0x7fac0c0b6840 0x7fac0c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:49.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 -- 172.21.15.61:0/3488754216 >> v2:172.21.15.117:3300/0 conn(0x7fac0c0b6840 msgr2=0x7fac0c0b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:49.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 --2- 172.21.15.61:0/3488754216 >> v2:172.21.15.117:3300/0 conn(0x7fac0c0b6840 0x7fac0c0b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 -- 172.21.15.61:0/3488754216 >> v1:172.21.15.117:6789/0 conn(0x7fac0c0b3cf0 legacy=0x7fac0c146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:49.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac0affd700 1 -- 172.21.15.61:0/3488754216 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3009369166 0 0) 0x7fac0c066b10 con 0x7fac0c0b3cf0 2022-07-20T03:15:49.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 -- 172.21.15.61:0/3488754216 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac0c0aed70 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac10db1700 1 --2- 172.21.15.61:0/3488754216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c141a90 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fac040032f0 tx=0x7fac04009290).ready entity=mon.0 client_cookie=8e78542241d81582 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:49.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac0affd700 1 -- 172.21.15.61:0/3488754216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac0400b870 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac0affd700 1 -- 172.21.15.61:0/3488754216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fac0400b9d0 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac0affd700 1 -- 172.21.15.61:0/3488754216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fac0400bb40 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- 172.21.15.61:0/3488754216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 msgr2=0x7fac0c141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:49.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 --2- 172.21.15.61:0/3488754216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c141a90 secure :-1 s=READY pgs=1094 cs=0 l=1 rev1=1 rx=0x7fac040032f0 tx=0x7fac04009290).stop 2022-07-20T03:15:49.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- 172.21.15.61:0/3488754216 shutdown_connections 2022-07-20T03:15:49.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 --2- 172.21.15.61:0/3488754216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c141a90 unknown :-1 s=CLOSED pgs=1094 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 --2- 172.21.15.61:0/3488754216 >> v2:172.21.15.117:3300/0 conn(0x7fac0c0b6840 0x7fac0c0b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- 172.21.15.61:0/3488754216 >> 172.21.15.61:0/3488754216 conn(0x7fac0c0b9e10 msgr2=0x7fac0c0b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:49.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- 172.21.15.61:0/3488754216 shutdown_connections 2022-07-20T03:15:49.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.264+0000 7fac115b2700 1 -- 172.21.15.61:0/3488754216 wait complete. 2022-07-20T03:15:49.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 Processor -- start 2022-07-20T03:15:49.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 -- start start 2022-07-20T03:15:49.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac0c0b3cf0 0x7fac0c153010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:49.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c112a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:49.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac0c0b6840 0x7fac0c0afe60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:49.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0af280 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0b1cb0 con 0x7fac0c0b3cf0 2022-07-20T03:15:49.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac115b2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fac0c0b2bb0 con 0x7fac0c0b6840 2022-07-20T03:15:49.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c112a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:49.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac0c0b3cf0 0x7fac0c153010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:49.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c112a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49202/0 (socket says 172.21.15.61:49202) 2022-07-20T03:15:49.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 -- 172.21.15.61:0/227377105 learned_addr learned my addr 172.21.15.61:0/227377105 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:49.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac10db1700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac0c0b6840 0x7fac0c0afe60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:49.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 -- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac0c0b6840 msgr2=0x7fac0c0afe60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:49.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac0c0b6840 0x7fac0c0afe60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 -- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac0c0b3cf0 msgr2=0x7fac0c153010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:49.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac0c0b3cf0 0x7fac0c153010 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:49.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 -- 172.21.15.61:0/227377105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac04009530 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.268+0000 7fac0b7fe700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c112a70 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7fabf4000ed0 tx=0x7fabf4007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:49.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf4012400 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabf4012560 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabf40128c0 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac0c0b1a10 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac0c0b20c0 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac0c0aefc0 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fabf4019030 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac097fa700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fabf00539d0 0x7fabf0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:49.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(102..102 src has 1..102) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fabf406b060 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.272+0000 7fac0bfff700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fabf00539d0 0x7fabf0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:49.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.276+0000 7fac0bfff700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fabf00539d0 0x7fabf0055e90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fabfc00b000 tx=0x7fabfc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:49.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.284+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fac0c0aefc0 con 0x7fac0c0b4bc0 2022-07-20T03:15:49.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:49.408+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fac0c0aef70 con 0x7fac0c0b4bc0 2022-07-20T03:15:50.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.148+0000 7fac097fa700 1 -- 172.21.15.61:0/227377105 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v103) v1 ==== 185+0+0 (secure 0 0 0) 0x7fabf4042020 con 0x7fac0c0b4bc0 2022-07-20T03:15:50.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fabf00539d0 msgr2=0x7fabf0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:50.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fabf00539d0 0x7fabf0055e90 secure :-1 s=READY pgs=529 cs=0 l=1 rev1=1 rx=0x7fabfc00b000 tx=0x7fabfc006df0).stop 2022-07-20T03:15:50.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 msgr2=0x7fac0c112a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:50.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c112a70 secure :-1 s=READY pgs=1095 cs=0 l=1 rev1=1 rx=0x7fabf4000ed0 tx=0x7fabf4007a80).stop 2022-07-20T03:15:50.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 shutdown_connections 2022-07-20T03:15:50.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fabf00539d0 0x7fabf0055e90 unknown :-1 s=CLOSED pgs=529 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fac0c0b6840 0x7fac0c0afe60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fac0c0b4bc0 0x7fac0c112a70 unknown :-1 s=CLOSED pgs=1095 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 --2- 172.21.15.61:0/227377105 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fac0c0b3cf0 0x7fac0c153010 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 >> 172.21.15.61:0/227377105 conn(0x7fac0c0b9e10 msgr2=0x7fac0c0b31a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:50.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 shutdown_connections 2022-07-20T03:15:50.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.152+0000 7fac115b2700 1 -- 172.21.15.61:0/227377105 wait complete. 2022-07-20T03:15:50.178 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:15:50.180 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:15:50.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 Processor -- start 2022-07-20T03:15:50.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 -- start start 2022-07-20T03:15:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa1700b6690 0x7fa1700b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:50.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1700b59a0 0x7fa17013c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1700acae0 con 0x7fa17013ce60 2022-07-20T03:15:50.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa170147100 con 0x7fa1700b59a0 2022-07-20T03:15:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.308+0000 7fa176a3f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1700acd10 con 0x7fa1700b6690 2022-07-20T03:15:50.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1700b59a0 0x7fa17013c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1700b59a0 0x7fa17013c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49210/0 (socket says 172.21.15.61:49210) 2022-07-20T03:15:50.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 -- 172.21.15.61:0/2089087076 learned_addr learned my addr 172.21.15.61:0/2089087076 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:50.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16ffff700 1 --2- 172.21.15.61:0/2089087076 >> v2:172.21.15.117:3300/0 conn(0x7fa1700b6690 0x7fa1700b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:50.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 -- 172.21.15.61:0/2089087076 >> v2:172.21.15.117:3300/0 conn(0x7fa1700b6690 msgr2=0x7fa1700b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:50.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 --2- 172.21.15.61:0/2089087076 >> v2:172.21.15.117:3300/0 conn(0x7fa1700b6690 0x7fa1700b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 -- 172.21.15.61:0/2089087076 >> v1:172.21.15.117:6789/0 conn(0x7fa17013ce60 legacy=0x7fa1700ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:50.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 -- 172.21.15.61:0/2089087076 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1700b5df0 con 0x7fa1700b59a0 2022-07-20T03:15:50.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16f7fe700 1 --2- 172.21.15.61:0/2089087076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa17013c970 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7fa160002f70 tx=0x7fa160006eb0).ready entity=mon.0 client_cookie=847c0a73c5367353 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:50.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16effd700 1 -- 172.21.15.61:0/2089087076 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16000d610 con 0x7fa1700b59a0 2022-07-20T03:15:50.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16effd700 1 -- 172.21.15.61:0/2089087076 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa160000e90 con 0x7fa1700b59a0 2022-07-20T03:15:50.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa16effd700 1 -- 172.21.15.61:0/2089087076 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16000d9a0 con 0x7fa1700b59a0 2022-07-20T03:15:50.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 -- 172.21.15.61:0/2089087076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 msgr2=0x7fa17013c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 --2- 172.21.15.61:0/2089087076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa17013c970 secure :-1 s=READY pgs=1096 cs=0 l=1 rev1=1 rx=0x7fa160002f70 tx=0x7fa160006eb0).stop 2022-07-20T03:15:50.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 -- 172.21.15.61:0/2089087076 shutdown_connections 2022-07-20T03:15:50.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 --2- 172.21.15.61:0/2089087076 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa17013c970 unknown :-1 s=CLOSED pgs=1096 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 --2- 172.21.15.61:0/2089087076 >> v2:172.21.15.117:3300/0 conn(0x7fa1700b6690 0x7fa1700b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 -- 172.21.15.61:0/2089087076 >> 172.21.15.61:0/2089087076 conn(0x7fa1700b9f70 msgr2=0x7fa1700b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:50.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 -- 172.21.15.61:0/2089087076 shutdown_connections 2022-07-20T03:15:50.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 -- 172.21.15.61:0/2089087076 wait complete. 2022-07-20T03:15:50.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 Processor -- start 2022-07-20T03:15:50.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.312+0000 7fa176a3f700 1 -- start start 2022-07-20T03:15:50.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa1700b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:50.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1700b6690 0x7fa170155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:50.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa17013ce60 0x7fa1701129e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:50.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17013f930 con 0x7fa1700b59a0 2022-07-20T03:15:50.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa17013f3f0 con 0x7fa1700b6690 2022-07-20T03:15:50.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa17013f690 con 0x7fa17013ce60 2022-07-20T03:15:50.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa1700b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:50.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa1700b3190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49214/0 (socket says 172.21.15.61:49214) 2022-07-20T03:15:50.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 -- 172.21.15.61:0/1236919142 learned_addr learned my addr 172.21.15.61:0/1236919142 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:50.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16f7fe700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1700b6690 0x7fa170155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:50.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa174fdc700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa17013ce60 0x7fa1701129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:50.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 -- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa17013ce60 msgr2=0x7fa1701129e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:50.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa17013ce60 0x7fa1701129e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 -- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1700b6690 msgr2=0x7fa170155b90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:50.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1700b6690 0x7fa170155b90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:50.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 -- 172.21.15.61:0/1236919142 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa16000b1e0 con 0x7fa1700b59a0 2022-07-20T03:15:50.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16ffff700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa1700b3190 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fa1580030b0 tx=0x7fa158008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:50.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa158011780 con 0x7fa1700b59a0 2022-07-20T03:15:50.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1580118e0 con 0x7fa1700b59a0 2022-07-20T03:15:50.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa170141490 con 0x7fa1700b59a0 2022-07-20T03:15:50.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa158011c40 con 0x7fa1700b59a0 2022-07-20T03:15:50.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.316+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1701402d0 con 0x7fa1700b59a0 2022-07-20T03:15:50.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.320+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1700acfb0 con 0x7fa1700b59a0 2022-07-20T03:15:50.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.320+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa15801c460 con 0x7fa1700b59a0 2022-07-20T03:15:50.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.320+0000 7fa16d7fa700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa154053a60 0x7fa154055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:50.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.320+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(103..103 src has 1..103) v4 ==== 5721+0+0 (secure 0 0 0) 0x7fa15806be50 con 0x7fa1700b59a0 2022-07-20T03:15:50.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.320+0000 7fa16f7fe700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa154053a60 0x7fa154055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:50.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.320+0000 7fa16f7fe700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa154053a60 0x7fa154055f20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fa16001c000 tx=0x7fa160015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:50.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.324+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa158042e30 con 0x7fa1700b59a0 2022-07-20T03:15:50.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:50.452+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fa1700574e0 con 0x7fa1700b59a0 2022-07-20T03:15:51.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.152+0000 7fa16d7fa700 1 -- 172.21.15.61:0/1236919142 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v104) v1 ==== 173+0+0 (secure 0 0 0) 0x7fa158046450 con 0x7fa1700b59a0 2022-07-20T03:15:51.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa154053a60 msgr2=0x7fa154055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa154053a60 0x7fa154055f20 secure :-1 s=READY pgs=530 cs=0 l=1 rev1=1 rx=0x7fa16001c000 tx=0x7fa160015000).stop 2022-07-20T03:15:51.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 msgr2=0x7fa1700b3190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa1700b3190 secure :-1 s=READY pgs=1097 cs=0 l=1 rev1=1 rx=0x7fa1580030b0 tx=0x7fa158008040).stop 2022-07-20T03:15:51.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 shutdown_connections 2022-07-20T03:15:51.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa154053a60 0x7fa154055f20 unknown :-1 s=CLOSED pgs=530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1700b6690 0x7fa170155b90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa17013ce60 0x7fa1701129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 --2- 172.21.15.61:0/1236919142 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1700b59a0 0x7fa1700b3190 unknown :-1 s=CLOSED pgs=1097 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 >> 172.21.15.61:0/1236919142 conn(0x7fa1700b9f70 msgr2=0x7fa1700b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:51.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 shutdown_connections 2022-07-20T03:15:51.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.156+0000 7fa176a3f700 1 -- 172.21.15.61:0/1236919142 wait complete. 2022-07-20T03:15:51.164 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:15:51.174 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:15:51.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 Processor -- start 2022-07-20T03:15:51.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 -- start start 2022-07-20T03:15:51.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6bb80aec90 0x7f6bb80acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 0x7f6bb813e880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bb8140060 con 0x7f6bb80affa0 2022-07-20T03:15:51.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bb80b0ff0 con 0x7f6bb80aec90 2022-07-20T03:15:51.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbdc99700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb80af070 con 0x7f6bb813e430 2022-07-20T03:15:51.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbd498700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 0x7f6bb813e880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbcc97700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6bb80aec90 0x7f6bb80acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbd498700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 0x7f6bb813e880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35898/0 (socket says 172.21.15.61:35898) 2022-07-20T03:15:51.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbcc97700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6bb80aec90 0x7f6bb80acae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58902/0 (socket says 172.21.15.61:58902) 2022-07-20T03:15:51.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bb7fff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f6bb80affa0 0x7f6bb813f950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:54326/0 (socket says 172.21.15.61:54326) 2022-07-20T03:15:51.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.280+0000 7f6bbd498700 1 -- 172.21.15.61:0/1807690287 learned_addr learned my addr 172.21.15.61:0/1807690287 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:51.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbcc97700 1 -- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 msgr2=0x7f6bb813e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:51.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbcc97700 1 --2- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 0x7f6bb813e880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbcc97700 1 -- 172.21.15.61:0/1807690287 >> v1:172.21.15.117:6789/0 conn(0x7f6bb80affa0 legacy=0x7f6bb813f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbcc97700 1 -- 172.21.15.61:0/1807690287 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb813ed70 con 0x7f6bb80aec90 2022-07-20T03:15:51.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbd498700 1 --2- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 0x7f6bb813e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:51.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbcc97700 1 --2- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb80acae0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f6ba0002f70 tx=0x7f6ba0006d90).ready entity=mon.1 client_cookie=278b7fdfd0c300f7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:51.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bb77fe700 1 -- 172.21.15.61:0/1807690287 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ba000d4a0 con 0x7f6bb80aec90 2022-07-20T03:15:51.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bb77fe700 1 -- 172.21.15.61:0/1807690287 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba0000e90 con 0x7f6bb80aec90 2022-07-20T03:15:51.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bb77fe700 1 -- 172.21.15.61:0/1807690287 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ba000d7f0 con 0x7f6bb80aec90 2022-07-20T03:15:51.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 -- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 msgr2=0x7f6bb80acae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb80acae0 secure :-1 s=READY pgs=498 cs=0 l=1 rev1=1 rx=0x7f6ba0002f70 tx=0x7f6ba0006d90).stop 2022-07-20T03:15:51.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 -- 172.21.15.61:0/1807690287 shutdown_connections 2022-07-20T03:15:51.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb813e430 0x7f6bb813e880 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/1807690287 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb80acae0 unknown :-1 s=CLOSED pgs=498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 -- 172.21.15.61:0/1807690287 >> 172.21.15.61:0/1807690287 conn(0x7f6bb8063480 msgr2=0x7f6bb80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:51.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 -- 172.21.15.61:0/1807690287 shutdown_connections 2022-07-20T03:15:51.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 -- 172.21.15.61:0/1807690287 wait complete. 2022-07-20T03:15:51.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.284+0000 7f6bbdc99700 1 Processor -- start 2022-07-20T03:15:51.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 -- start start 2022-07-20T03:15:51.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb8065880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb80affa0 0x7f6bb8065d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 0x7f6bb81522a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8066350 con 0x7f6bb813e430 2022-07-20T03:15:51.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8065340 con 0x7f6bb80aec90 2022-07-20T03:15:51.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bb8063c30 con 0x7f6bb80affa0 2022-07-20T03:15:51.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 0x7f6bb81522a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 0x7f6bb81522a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49232/0 (socket says 172.21.15.61:49232) 2022-07-20T03:15:51.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 -- 172.21.15.61:0/2091925978 learned_addr learned my addr 172.21.15.61:0/2091925978 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:51.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bb7fff700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb80affa0 0x7f6bb8065d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbcc97700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb8065880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 -- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb80affa0 msgr2=0x7f6bb8065d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb80affa0 0x7f6bb8065d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 -- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 msgr2=0x7f6bb8065880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb8065880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 -- 172.21.15.61:0/2091925978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bb0009530 con 0x7f6bb813e430 2022-07-20T03:15:51.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bb7fff700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb80affa0 0x7f6bb8065d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:51.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbd498700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 0x7f6bb81522a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f6bb0006ec0 tx=0x7f6bb0009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:51.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb000c850 con 0x7f6bb813e430 2022-07-20T03:15:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6ba000b120 con 0x7f6bb813e430 2022-07-20T03:15:51.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bb000c9b0 con 0x7f6bb813e430 2022-07-20T03:15:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bb000cd10 con 0x7f6bb813e430 2022-07-20T03:15:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.288+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bb80641d0 con 0x7f6bb813e430 2022-07-20T03:15:51.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.292+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bb8065540 con 0x7f6bb813e430 2022-07-20T03:15:51.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.292+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6bb0022070 con 0x7f6bb813e430 2022-07-20T03:15:51.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.292+0000 7f6bb5ffb700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6b9c053a40 0x7f6b9c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.292+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f6bb006c4f0 con 0x7f6bb813e430 2022-07-20T03:15:51.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.292+0000 7f6bbcc97700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6b9c053a40 0x7f6b9c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.292+0000 7f6bbcc97700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6b9c053a40 0x7f6b9c055f00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f6ba001a000 tx=0x7f6ba0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:51.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.296+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6bb8065540 con 0x7f6bb813e430 2022-07-20T03:15:51.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.428+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f6bb8065540 con 0x7f6bb813e430 2022-07-20T03:15:51.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bb5ffb700 1 -- 172.21.15.61:0/2091925978 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v17) v1 ==== 104+0+0 (secure 0 0 0) 0x7f6bb8065540 con 0x7f6bb813e430 2022-07-20T03:15:51.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6b9c053a40 msgr2=0x7f6b9c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6b9c053a40 0x7f6b9c055f00 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7f6ba001a000 tx=0x7f6ba0014000).stop 2022-07-20T03:15:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 msgr2=0x7f6bb81522a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 0x7f6bb81522a0 secure :-1 s=READY pgs=1098 cs=0 l=1 rev1=1 rx=0x7f6bb0006ec0 tx=0x7f6bb0009290).stop 2022-07-20T03:15:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 shutdown_connections 2022-07-20T03:15:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6b9c053a40 0x7f6b9c055f00 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bb813e430 0x7f6bb81522a0 unknown :-1 s=CLOSED pgs=1098 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bb80affa0 0x7f6bb8065d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 --2- 172.21.15.61:0/2091925978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bb80aec90 0x7f6bb8065880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.436+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 >> 172.21.15.61:0/2091925978 conn(0x7f6bb8063480 msgr2=0x7f6bb805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.440+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 shutdown_connections 2022-07-20T03:15:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.440+0000 7f6bbdc99700 1 -- 172.21.15.61:0/2091925978 wait complete. 2022-07-20T03:15:51.457 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df' 2022-07-20T03:15:51.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.560+0000 7f5644fff700 1 Processor -- start 2022-07-20T03:15:51.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.560+0000 7f5644fff700 1 -- start start 2022-07-20T03:15:51.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f564005c7c0 0x7f5640060c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5640147860 0x7f5640147c90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5640098d50 con 0x7f56400b21b0 2022-07-20T03:15:51.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f56400604f0 con 0x7f564005c7c0 2022-07-20T03:15:51.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5640060790 con 0x7f5640147860 2022-07-20T03:15:51.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5636ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f56400b21b0 0x7f564014aa40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45834/0 (socket says 172.21.15.61:45834) 2022-07-20T03:15:51.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5636ffd700 1 -- 172.21.15.61:0/3190966603 learned_addr learned my addr 172.21.15.61:0/3190966603 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:51.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563ffff700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5640147860 0x7f5640147c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3190966603 >> v2:172.21.15.117:3300/0 conn(0x7f564005c7c0 0x7f5640060c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563effd700 1 -- 172.21.15.61:0/3190966603 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3572824722 0 0) 0x7f5640098d50 con 0x7f56400b21b0 2022-07-20T03:15:51.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563effd700 1 -- 172.21.15.61:0/3190966603 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5620003260 con 0x7f56400b21b0 2022-07-20T03:15:51.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563f7fe700 1 -- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5640147860 msgr2=0x7f5640147c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:51.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5640147860 0x7f5640147c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563f7fe700 1 -- 172.21.15.61:0/3190966603 >> v1:172.21.15.61:6789/0 conn(0x7f56400b21b0 legacy=0x7f564014aa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563f7fe700 1 -- 172.21.15.61:0/3190966603 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f564014b150 con 0x7f564005c7c0 2022-07-20T03:15:51.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563ffff700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5640147860 0x7f5640147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:51.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640060c30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f5630002f70 tx=0x7f5630006df0).ready entity=mon.1 client_cookie=6323ec680a600e3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:51.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563effd700 1 -- 172.21.15.61:0/3190966603 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f563000d4a0 con 0x7f564005c7c0 2022-07-20T03:15:51.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563effd700 1 -- 172.21.15.61:0/3190966603 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5630000e90 con 0x7f564005c7c0 2022-07-20T03:15:51.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f563effd700 1 -- 172.21.15.61:0/3190966603 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f563000d7f0 con 0x7f564005c7c0 2022-07-20T03:15:51.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 msgr2=0x7f5640060c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640060c30 secure :-1 s=READY pgs=499 cs=0 l=1 rev1=1 rx=0x7f5630002f70 tx=0x7f5630006df0).stop 2022-07-20T03:15:51.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- 172.21.15.61:0/3190966603 shutdown_connections 2022-07-20T03:15:51.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5640147860 0x7f5640147c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 --2- 172.21.15.61:0/3190966603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640060c30 unknown :-1 s=CLOSED pgs=499 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- 172.21.15.61:0/3190966603 >> 172.21.15.61:0/3190966603 conn(0x7f56400b61f0 msgr2=0x7f56400b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:51.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- 172.21.15.61:0/3190966603 shutdown_connections 2022-07-20T03:15:51.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.564+0000 7f5644fff700 1 -- 172.21.15.61:0/3190966603 wait complete. 2022-07-20T03:15:51.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 Processor -- start 2022-07-20T03:15:51.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- start start 2022-07-20T03:15:51.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640157280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56400b21b0 0x7f56400660a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 0x7f56400991c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f564009bef0 con 0x7f5640147860 2022-07-20T03:15:51.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5640065830 con 0x7f564005c7c0 2022-07-20T03:15:51.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5640065590 con 0x7f56400b21b0 2022-07-20T03:15:51.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 0x7f56400991c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 0x7f56400991c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49246/0 (socket says 172.21.15.61:49246) 2022-07-20T03:15:51.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 -- 172.21.15.61:0/3998071546 learned_addr learned my addr 172.21.15.61:0/3998071546 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:51.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640157280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5636ffd700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56400b21b0 0x7f56400660a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 -- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56400b21b0 msgr2=0x7f56400660a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56400b21b0 0x7f56400660a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 -- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 msgr2=0x7f5640157280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:51.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640157280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:51.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 -- 172.21.15.61:0/3998071546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5638009470 con 0x7f5640147860 2022-07-20T03:15:51.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5636ffd700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56400b21b0 0x7f56400660a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:51.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:51.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563ffff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 0x7f56400991c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f563800c000 tx=0x7f56380091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:51.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f563800c850 con 0x7f5640147860 2022-07-20T03:15:51.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f563800c9b0 con 0x7f5640147860 2022-07-20T03:15:51.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f563000b180 con 0x7f5640147860 2022-07-20T03:15:51.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f563800cd10 con 0x7f5640147860 2022-07-20T03:15:51.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f564009bbe0 con 0x7f5640147860 2022-07-20T03:15:51.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.568+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5640098a40 con 0x7f5640147860 2022-07-20T03:15:51.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.572+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5638022070 con 0x7f5640147860 2022-07-20T03:15:51.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.572+0000 7f563d7fa700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f56240539b0 0x7f5624055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:51.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.572+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f563806c3b0 con 0x7f5640147860 2022-07-20T03:15:51.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.572+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f56240539b0 0x7f5624055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:51.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.572+0000 7f563f7fe700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f56240539b0 0x7f5624055e70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f5630018000 tx=0x7f5630014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:51.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.576+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f56380435f0 con 0x7f5640147860 2022-07-20T03:15:51.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:51.740+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]} v 0) v1 -- 0x7f56400574e0 con 0x7f5640147860 2022-07-20T03:15:52.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.168+0000 7f563d7fa700 1 -- 172.21.15.61:0/3998071546 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_filtered_df"]}]=0 v400) v1 ==== 127+0+0 (secure 0 0 0) 0x7f5638046c10 con 0x7f5640147860 2022-07-20T03:15:52.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f56240539b0 msgr2=0x7f5624055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f56240539b0 0x7f5624055e70 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7f5630018000 tx=0x7f5630014000).stop 2022-07-20T03:15:52.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 msgr2=0x7f56400991c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 0x7f56400991c0 secure :-1 s=READY pgs=1100 cs=0 l=1 rev1=1 rx=0x7f563800c000 tx=0x7f56380091d0).stop 2022-07-20T03:15:52.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 shutdown_connections 2022-07-20T03:15:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f56240539b0 0x7f5624055e70 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5640147860 0x7f56400991c0 unknown :-1 s=CLOSED pgs=1100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f56400b21b0 0x7f56400660a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 --2- 172.21.15.61:0/3998071546 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f564005c7c0 0x7f5640157280 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 >> 172.21.15.61:0/3998071546 conn(0x7f56400b61f0 msgr2=0x7f56400b2760 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:52.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 shutdown_connections 2022-07-20T03:15:52.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.172+0000 7f5644fff700 1 -- 172.21.15.61:0/3998071546 wait complete. 2022-07-20T03:15:52.193 INFO:tasks.cephfs_test_runner:test_filtered_df (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:15:52.194 INFO:tasks.cephfs_test_runner:Starting test: test_fs_new (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:15:52.195 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-20T03:15:52.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 Processor -- start 2022-07-20T03:15:52.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- start start 2022-07-20T03:15:52.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb76c0b1e00 0x7fb76c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:52.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c05d480 con 0x7fb76c05c3f0 2022-07-20T03:15:52.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb76c0b21e0 con 0x7fb76c0acef0 2022-07-20T03:15:52.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb76c05cf40 con 0x7fb76c0b1e00 2022-07-20T03:15:52.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb76c0b1e00 0x7fb76c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:52.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76a7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb76c05c3f0 0x7fb76c0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45850/0 (socket says 172.21.15.61:45850) 2022-07-20T03:15:52.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76a7fc700 1 -- 172.21.15.61:0/51438760 learned_addr learned my addr 172.21.15.61:0/51438760 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:52.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76affd700 1 -- 172.21.15.61:0/51438760 >> v1:172.21.15.117:6789/0 conn(0x7fb76c0acef0 legacy=0x7fb76c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76affd700 1 -- 172.21.15.61:0/51438760 >> v1:172.21.15.61:6789/0 conn(0x7fb76c05c3f0 legacy=0x7fb76c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76affd700 1 -- 172.21.15.61:0/51438760 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb76c05d1e0 con 0x7fb76c0b1e00 2022-07-20T03:15:52.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb76affd700 1 --2- 172.21.15.61:0/51438760 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c0b1e00 0x7fb76c14a6d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7fb75c002f70 tx=0x7fb75c006df0).ready entity=mon.0 client_cookie=978bf8636e931e41 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:52.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb769ffb700 1 -- 172.21.15.61:0/51438760 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c00d610 con 0x7fb76c0b1e00 2022-07-20T03:15:52.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb769ffb700 1 -- 172.21.15.61:0/51438760 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb75c000e90 con 0x7fb76c0b1e00 2022-07-20T03:15:52.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb769ffb700 1 -- 172.21.15.61:0/51438760 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb75c00d960 con 0x7fb76c0b1e00 2022-07-20T03:15:52.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- 172.21.15.61:0/51438760 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c0b1e00 msgr2=0x7fb76c14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 --2- 172.21.15.61:0/51438760 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c0b1e00 0x7fb76c14a6d0 secure :-1 s=READY pgs=1102 cs=0 l=1 rev1=1 rx=0x7fb75c002f70 tx=0x7fb75c006df0).stop 2022-07-20T03:15:52.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- 172.21.15.61:0/51438760 shutdown_connections 2022-07-20T03:15:52.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 --2- 172.21.15.61:0/51438760 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c0b1e00 0x7fb76c14a6d0 unknown :-1 s=CLOSED pgs=1102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- 172.21.15.61:0/51438760 >> 172.21.15.61:0/51438760 conn(0x7fb76c0611c0 msgr2=0x7fb76c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:52.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- 172.21.15.61:0/51438760 shutdown_connections 2022-07-20T03:15:52.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.308+0000 7fb77180f700 1 -- 172.21.15.61:0/51438760 wait complete. 2022-07-20T03:15:52.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 Processor -- start 2022-07-20T03:15:52.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 -- start start 2022-07-20T03:15:52.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 0x7fb76c064590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:52.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb76c0b1e00 0x7fb76c0b47b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:52.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c09fcf0 0x7fb76c159ee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:52.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0a0c40 con 0x7fb76c09fcf0 2022-07-20T03:15:52.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0a0ee0 con 0x7fb76c0acef0 2022-07-20T03:15:52.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb76c0a0320 con 0x7fb76c0b1e00 2022-07-20T03:15:52.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 0x7fb76c064590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:52.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c09fcf0 0x7fb76c159ee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:52.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 0x7fb76c064590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:58936/0 (socket says 172.21.15.61:58936) 2022-07-20T03:15:52.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76a7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb76c0b1e00 0x7fb76c0b47b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:52.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76affd700 1 -- 172.21.15.61:0/48672914 learned_addr learned my addr 172.21.15.61:0/48672914 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:52.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 -- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb76c0b1e00 msgr2=0x7fb76c0b47b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb76c0b1e00 0x7fb76c0b47b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 -- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 msgr2=0x7fb76c064590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:52.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 0x7fb76c064590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:52.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 -- 172.21.15.61:0/48672914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb75c00b180 con 0x7fb76c09fcf0 2022-07-20T03:15:52.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76affd700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 0x7fb76c064590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:52.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb76b7fe700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c09fcf0 0x7fb76c159ee0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7fb76c05f3e0 tx=0x7fb764007eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:52.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb76400f470 con 0x7fb76c09fcf0 2022-07-20T03:15:52.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb76400f5d0 con 0x7fb76c09fcf0 2022-07-20T03:15:52.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb76c0a07c0 con 0x7fb76c09fcf0 2022-07-20T03:15:52.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb76400f930 con 0x7fb76c09fcf0 2022-07-20T03:15:52.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.312+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb76c09ebc0 con 0x7fb76c09fcf0 2022-07-20T03:15:52.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.316+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb764010070 con 0x7fb76c09fcf0 2022-07-20T03:15:52.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.316+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb76c0b04b0 con 0x7fb76c09fcf0 2022-07-20T03:15:52.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.316+0000 7fb75bfff700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb748053a40 0x7fb748055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:52.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.316+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb76406b190 con 0x7fb76c09fcf0 2022-07-20T03:15:52.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.316+0000 7fb76affd700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb748053a40 0x7fb748055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:52.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.320+0000 7fb76affd700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb748053a40 0x7fb748055f00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fb75c01c000 tx=0x7fb75c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:52.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.324+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb7640422c0 con 0x7fb76c09fcf0 2022-07-20T03:15:52.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:52.488+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7fb76c0574e0 con 0x7fb76c09fcf0 2022-07-20T03:15:53.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.172+0000 7fb75bfff700 1 -- 172.21.15.61:0/48672914 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v401) v1 ==== 125+0+0 (secure 0 0 0) 0x7fb7640458e0 con 0x7fb76c09fcf0 2022-07-20T03:15:53.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb748053a40 msgr2=0x7fb748055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb748053a40 0x7fb748055f00 secure :-1 s=READY pgs=533 cs=0 l=1 rev1=1 rx=0x7fb75c01c000 tx=0x7fb75c015000).stop 2022-07-20T03:15:53.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c09fcf0 msgr2=0x7fb76c159ee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c09fcf0 0x7fb76c159ee0 secure :-1 s=READY pgs=1103 cs=0 l=1 rev1=1 rx=0x7fb76c05f3e0 tx=0x7fb764007eb0).stop 2022-07-20T03:15:53.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 shutdown_connections 2022-07-20T03:15:53.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb748053a40 0x7fb748055f00 unknown :-1 s=CLOSED pgs=533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb76c09fcf0 0x7fb76c159ee0 unknown :-1 s=CLOSED pgs=1103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb76c0b1e00 0x7fb76c0b47b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 --2- 172.21.15.61:0/48672914 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb76c0acef0 0x7fb76c064590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 >> 172.21.15.61:0/48672914 conn(0x7fb76c05c3f0 msgr2=0x7fb76c143680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:53.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 shutdown_connections 2022-07-20T03:15:53.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.176+0000 7fb77180f700 1 -- 172.21.15.61:0/48672914 wait complete. 2022-07-20T03:15:53.197 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:15:53.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.324+0000 7fa978a42700 1 Processor -- start 2022-07-20T03:15:53.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.324+0000 7fa978a42700 1 -- start start 2022-07-20T03:15:53.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa9740b59d0 0x7fa9740b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9740b14f0 0x7fa974147050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa9740afbc0 con 0x7fa97414ada0 2022-07-20T03:15:53.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa974148080 con 0x7fa9740b14f0 2022-07-20T03:15:53.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa9740b00e0 con 0x7fa9740b59d0 2022-07-20T03:15:53.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa973fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa9740b14f0 0x7fa974147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa972ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa97414ada0 0x7fa97405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45862/0 (socket says 172.21.15.61:45862) 2022-07-20T03:15:53.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa972ffd700 1 -- 172.21.15.61:0/3523563839 learned_addr learned my addr 172.21.15.61:0/3523563839 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:53.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa9737fe700 1 --2- 172.21.15.61:0/3523563839 >> v2:172.21.15.117:3300/0 conn(0x7fa9740b59d0 0x7fa9740b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa973fff700 1 -- 172.21.15.61:0/3523563839 >> v2:172.21.15.117:3300/0 conn(0x7fa9740b59d0 msgr2=0x7fa9740b07e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:53.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa973fff700 1 --2- 172.21.15.61:0/3523563839 >> v2:172.21.15.117:3300/0 conn(0x7fa9740b59d0 0x7fa9740b07e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa973fff700 1 -- 172.21.15.61:0/3523563839 >> v1:172.21.15.61:6789/0 conn(0x7fa97414ada0 legacy=0x7fa97405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa9727fc700 1 -- 172.21.15.61:0/3523563839 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2540178259 0 0) 0x7fa9740afbc0 con 0x7fa97414ada0 2022-07-20T03:15:53.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa973fff700 1 -- 172.21.15.61:0/3523563839 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa9740b5db0 con 0x7fa9740b14f0 2022-07-20T03:15:53.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa9737fe700 1 --2- 172.21.15.61:0/3523563839 >> v2:172.21.15.117:3300/0 conn(0x7fa9740b59d0 0x7fa9740b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:53.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa973fff700 1 --2- 172.21.15.61:0/3523563839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974147050 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7fa96c002f70 tx=0x7fa96c006df0).ready entity=mon.0 client_cookie=d57f5cd9e357d080 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:53.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa9727fc700 1 -- 172.21.15.61:0/3523563839 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c00d5c0 con 0x7fa9740b14f0 2022-07-20T03:15:53.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa9727fc700 1 -- 172.21.15.61:0/3523563839 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96c000e90 con 0x7fa9740b14f0 2022-07-20T03:15:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa9727fc700 1 -- 172.21.15.61:0/3523563839 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96c00d910 con 0x7fa9740b14f0 2022-07-20T03:15:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- 172.21.15.61:0/3523563839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 msgr2=0x7fa974147050 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 --2- 172.21.15.61:0/3523563839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974147050 secure :-1 s=READY pgs=1105 cs=0 l=1 rev1=1 rx=0x7fa96c002f70 tx=0x7fa96c006df0).stop 2022-07-20T03:15:53.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- 172.21.15.61:0/3523563839 shutdown_connections 2022-07-20T03:15:53.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 --2- 172.21.15.61:0/3523563839 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974147050 unknown :-1 s=CLOSED pgs=1105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 --2- 172.21.15.61:0/3523563839 >> v2:172.21.15.117:3300/0 conn(0x7fa9740b59d0 0x7fa9740b07e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- 172.21.15.61:0/3523563839 >> 172.21.15.61:0/3523563839 conn(0x7fa9740634c0 msgr2=0x7fa9740b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:53.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- 172.21.15.61:0/3523563839 shutdown_connections 2022-07-20T03:15:53.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.328+0000 7fa978a42700 1 -- 172.21.15.61:0/3523563839 wait complete. 2022-07-20T03:15:53.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 Processor -- start 2022-07-20T03:15:53.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 -- start start 2022-07-20T03:15:53.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974149840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740b59d0 0x7fa974155fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa97414ada0 0x7fa974159ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa974064270 con 0x7fa9740b14f0 2022-07-20T03:15:53.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa974156490 con 0x7fa9740b59d0 2022-07-20T03:15:53.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa974111480 con 0x7fa97414ada0 2022-07-20T03:15:53.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974149840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974149840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49270/0 (socket says 172.21.15.61:49270) 2022-07-20T03:15:53.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 -- 172.21.15.61:0/1760811828 learned_addr learned my addr 172.21.15.61:0/1760811828 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:53.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa972ffd700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740b59d0 0x7fa974155fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa973fff700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa97414ada0 0x7fa974159ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 -- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa97414ada0 msgr2=0x7fa974159ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa97414ada0 0x7fa974159ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 -- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740b59d0 msgr2=0x7fa974155fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740b59d0 0x7fa974155fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 -- 172.21.15.61:0/1760811828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa96c00b180 con 0x7fa9740b14f0 2022-07-20T03:15:53.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa972ffd700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740b59d0 0x7fa974155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:53.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa9737fe700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974149840 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7fa964006cc0 tx=0x7fa964008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:53.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96400c7d0 con 0x7fa9740b14f0 2022-07-20T03:15:53.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa96400c930 con 0x7fa9740b14f0 2022-07-20T03:15:53.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa9741111e0 con 0x7fa9740b14f0 2022-07-20T03:15:53.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa96400cc90 con 0x7fa9740b14f0 2022-07-20T03:15:53.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.332+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa97410fa30 con 0x7fa9740b14f0 2022-07-20T03:15:53.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.336+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa96401d460 con 0x7fa9740b14f0 2022-07-20T03:15:53.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.336+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa97414b2e0 con 0x7fa9740b14f0 2022-07-20T03:15:53.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.336+0000 7fa970ff9700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa958053ad0 0x7fa958055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.336+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa96406c0f0 con 0x7fa9740b14f0 2022-07-20T03:15:53.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.336+0000 7fa972ffd700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa958053ad0 0x7fa958055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.336+0000 7fa972ffd700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa958053ad0 0x7fa958055f90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fa95c00b000 tx=0x7fa95c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:53.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.340+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa964043510 con 0x7fa9740b14f0 2022-07-20T03:15:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.468+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fa9740574e0 con 0x7fa9740b14f0 2022-07-20T03:15:53.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.476+0000 7fa970ff9700 1 -- 172.21.15.61:0/1760811828 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v18)=0 v18) v1 ==== 122+0+0 (secure 0 0 0) 0x7fa964046760 con 0x7fa9740b14f0 2022-07-20T03:15:53.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa958053ad0 msgr2=0x7fa958055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa958053ad0 0x7fa958055f90 secure :-1 s=READY pgs=534 cs=0 l=1 rev1=1 rx=0x7fa95c00b000 tx=0x7fa95c009040).stop 2022-07-20T03:15:53.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 msgr2=0x7fa974149840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974149840 secure :-1 s=READY pgs=1106 cs=0 l=1 rev1=1 rx=0x7fa964006cc0 tx=0x7fa964008040).stop 2022-07-20T03:15:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 shutdown_connections 2022-07-20T03:15:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa958053ad0 0x7fa958055f90 unknown :-1 s=CLOSED pgs=534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa9740b59d0 0x7fa974155fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa97414ada0 0x7fa974159ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 --2- 172.21.15.61:0/1760811828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa9740b14f0 0x7fa974149840 unknown :-1 s=CLOSED pgs=1106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 >> 172.21.15.61:0/1760811828 conn(0x7fa9740634c0 msgr2=0x7fa9740b3b00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:53.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 shutdown_connections 2022-07-20T03:15:53.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.480+0000 7fa978a42700 1 -- 172.21.15.61:0/1760811828 wait complete. 2022-07-20T03:15:53.515 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:15:53.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:53.543 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:53.568 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:53.592 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:53.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:15:53.644 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:15:53.645 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:53.646 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:53.647 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:15:53.655 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:15:53.657 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:53.658 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:15:53.659 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:53.660 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:53.661 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:15:53.670 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:53.671 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:15:53.672 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:15:53.673 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:15:53.673 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:15:53.674 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:15:53.682 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:15:53.683 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:15:53.686 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:15:53.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:15:53.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 Processor -- start 2022-07-20T03:15:53.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 -- start start 2022-07-20T03:15:53.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 0x7f0cc0140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0cc00b1a70 0x7f0cc00b1f00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0cc00aee00 con 0x7f0cc00675f0 2022-07-20T03:15:53.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0cc0144470 con 0x7f0cc00b1a70 2022-07-20T03:15:53.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc7fa0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0144710 con 0x7f0cc00aea20 2022-07-20T03:15:53.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 0x7f0cc0140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc653d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0cc00b1a70 0x7f0cc00b1f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 0x7f0cc0140a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:35950/0 (socket says 172.21.15.61:35950) 2022-07-20T03:15:53.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 -- 172.21.15.61:0/2371791468 learned_addr learned my addr 172.21.15.61:0/2371791468 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:53.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc4d3a700 1 -- 172.21.15.61:0/2371791468 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 970583964 0 0) 0x7f0cc00aee00 con 0x7f0cc00675f0 2022-07-20T03:15:53.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 -- 172.21.15.61:0/2371791468 >> v2:172.21.15.117:3300/0 conn(0x7f0cc00b1a70 msgr2=0x7f0cc00b1f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:53.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 --2- 172.21.15.61:0/2371791468 >> v2:172.21.15.117:3300/0 conn(0x7f0cc00b1a70 0x7f0cc00b1f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 -- 172.21.15.61:0/2371791468 >> v1:172.21.15.117:6789/0 conn(0x7f0cc00675f0 legacy=0x7f0cc0140f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 -- 172.21.15.61:0/2371791468 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cc01449b0 con 0x7f0cc00aea20 2022-07-20T03:15:53.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc5d3c700 1 --2- 172.21.15.61:0/2371791468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 0x7f0cc0140a90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0cb00032f0 tx=0x7f0cb0009290).ready entity=mon.2 client_cookie=66da1ea258bb7292 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:53.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc4d3a700 1 -- 172.21.15.61:0/2371791468 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cb000b8a0 con 0x7f0cc00aea20 2022-07-20T03:15:53.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc4d3a700 1 -- 172.21.15.61:0/2371791468 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb000ba00 con 0x7f0cc00aea20 2022-07-20T03:15:53.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.796+0000 7f0cc4d3a700 1 -- 172.21.15.61:0/2371791468 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cb000bb70 con 0x7f0cc00aea20 2022-07-20T03:15:53.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/2371791468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 msgr2=0x7f0cc0140a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/2371791468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 0x7f0cc0140a90 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f0cb00032f0 tx=0x7f0cb0009290).stop 2022-07-20T03:15:53.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/2371791468 shutdown_connections 2022-07-20T03:15:53.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/2371791468 >> v2:172.21.15.117:3300/0 conn(0x7f0cc00b1a70 0x7f0cc00b1f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/2371791468 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00aea20 0x7f0cc0140a90 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/2371791468 >> 172.21.15.61:0/2371791468 conn(0x7f0cc00611c0 msgr2=0x7f0cc0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/2371791468 shutdown_connections 2022-07-20T03:15:53.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/2371791468 wait complete. 2022-07-20T03:15:53.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 Processor -- start 2022-07-20T03:15:53.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- start start 2022-07-20T03:15:53.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc00aea20 0x7f0cc0145e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00b1a70 0x7f0cc0065ce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 0x7f0cc00661d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0141520 con 0x7f0cc013f600 2022-07-20T03:15:53.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc00666c0 con 0x7f0cc00aea20 2022-07-20T03:15:53.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc7fa0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0cc0064190 con 0x7f0cc00b1a70 2022-07-20T03:15:53.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc653d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 0x7f0cc00661d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc653d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 0x7f0cc00661d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49288/0 (socket says 172.21.15.61:49288) 2022-07-20T03:15:53.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc653d700 1 -- 172.21.15.61:0/1673096633 learned_addr learned my addr 172.21.15.61:0/1673096633 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:53.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc5d3c700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc00aea20 0x7f0cc0145e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc553b700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00b1a70 0x7f0cc0065ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc653d700 1 -- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00b1a70 msgr2=0x7f0cc0065ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.800+0000 7f0cc653d700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00b1a70 0x7f0cc0065ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc653d700 1 -- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc00aea20 msgr2=0x7f0cc0145e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:53.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc653d700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc00aea20 0x7f0cc0145e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:53.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc653d700 1 -- 172.21.15.61:0/1673096633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cb0009530 con 0x7f0cc013f600 2022-07-20T03:15:53.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc653d700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 0x7f0cc00661d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f0cc0141c90 tx=0x7f0cb8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:53.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cb8011750 con 0x7f0cc013f600 2022-07-20T03:15:53.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0cb80118b0 con 0x7f0cc013f600 2022-07-20T03:15:53.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cc0146710 con 0x7f0cc013f600 2022-07-20T03:15:53.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0cb8011c10 con 0x7f0cc013f600 2022-07-20T03:15:53.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cc0064670 con 0x7f0cc013f600 2022-07-20T03:15:53.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cc0064fb0 con 0x7f0cc013f600 2022-07-20T03:15:53.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0cb8011d70 con 0x7f0cc013f600 2022-07-20T03:15:53.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cbeffd700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ca0053a60 0x7f0ca0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:53.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.804+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0cb806be90 con 0x7f0cc013f600 2022-07-20T03:15:53.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.808+0000 7f0cc5d3c700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ca0053a60 0x7f0ca0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:53.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.808+0000 7f0cc5d3c700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ca0053a60 0x7f0ca0055f20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f0cb001b000 tx=0x7f0cb0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:53.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.816+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0cb8043070 con 0x7f0cc013f600 2022-07-20T03:15:53.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.944+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0cc0144e50 con 0x7f0cc013f600 2022-07-20T03:15:53.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cbeffd700 1 -- 172.21.15.61:0/1673096633 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v104) v1 ==== 81+0+28877 (secure 0 0 0) 0x7f0cb8046690 con 0x7f0cc013f600 2022-07-20T03:15:53.953 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:53.954 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:53.954 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 104, 2022-07-20T03:15:53.955 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:15:53.955 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:15:53.956 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:15:51.145246+0000", 2022-07-20T03:15:53.957 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:53.961 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:15:53.962 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:15:53.963 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:15:53.963 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:15:53.964 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:15:53.964 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:15:53.965 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:15:53.965 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:15:53.966 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:53.966 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:15:53.966 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:15:53.966 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:15:53.967 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:15:53.967 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:15:53.967 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 20, 2022-07-20T03:15:53.967 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:15:53.968 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:15:53.968 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:15:53.968 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:15:53.968 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:15:53.969 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:53.969 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:15:53.970 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:15:53.970 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:15:53.970 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:53.971 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:53.971 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:53.971 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:53.976 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:53.976 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:53.977 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:53.977 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:53.977 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:15:53.978 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:15:53.978 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:15:53.979 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:15:53.979 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:15:53.980 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:53.980 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:53.982 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:53.982 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:53.983 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:53.983 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:53.984 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:53.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:53.985 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:15:53.986 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:53.986 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:53.986 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:53.987 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:53.987 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:53.987 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:53.987 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:53.988 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:53.988 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:53.988 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:53.988 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:53.989 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:53.989 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:53.989 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:53.989 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:53.989 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:53.990 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:53.990 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:53.990 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:53.991 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:53.991 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:53.991 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:53.991 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:53.991 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:53.992 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:53.992 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:53.992 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:53.992 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:53.993 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:53.993 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:53.993 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:53.993 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:53.994 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:53.994 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:53.994 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:53.994 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:53.995 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:53.995 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:53.995 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:15:53.995 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:15:53.995 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:53.996 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:53.996 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:15:53.996 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:53.997 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:53.997 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:53.997 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:15:53.998 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:15:53.998 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:15:53.998 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:15:53.998 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:15:53.999 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:15:53.999 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:15:53.999 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:15:53.999 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:15:54.000 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:15:54.000 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:15:54.000 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:15:54.000 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:15:54.001 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:15:54.001 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:15:54.001 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:15:54.001 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:15:54.002 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:15:54.002 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:15:54.002 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:15:54.003 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:15:54.003 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:15:54.004 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:15:54.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.004 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:15:54.004 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:15:54.005 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:15:54.005 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:15:54.005 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:15:54.005 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:15:54.006 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:15:54.006 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:15:54.006 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:15:54.006 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:15:54.007 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:15:54.007 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:15:54.007 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:15:54.007 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:15:54.007 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:15:54.008 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:15:54.008 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:15:54.008 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:15:54.008 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:15:54.009 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:15:54.009 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:15:54.009 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:15:54.009 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:15:54.010 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:15:54.010 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:15:54.010 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:15:54.010 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:15:54.011 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.011 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:15:54.011 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:15:54.011 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:15:54.012 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:15:54.012 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:15:54.012 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:15:54.012 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:15:54.013 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:15:54.013 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:15:54.013 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:15:54.014 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:15:54.014 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:15:54.014 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:15:54.015 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:15:54.015 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.015 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.015 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:54.015 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:15:54.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.016 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:54.016 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:15:54.016 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.017 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.017 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.017 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.017 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.018 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.018 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.019 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:54.019 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.019 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.019 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.020 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.020 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:15:54.021 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.021 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.021 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.021 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.022 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:15:54.022 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.022 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.022 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.023 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.023 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.023 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.024 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.024 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:15:54.024 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.024 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.025 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.025 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.025 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:15:54.025 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.025 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.026 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.026 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.026 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.026 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.027 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.027 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:15:54.027 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.028 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.028 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.028 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.028 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:15:54.028 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.029 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.029 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.029 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.029 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.030 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.030 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.030 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.030 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:15:54.031 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.031 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.031 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.031 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:15:54.032 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:15:54.032 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.032 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.032 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.033 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:15:54.033 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:15:54.033 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:15:54.033 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:15:54.034 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.034 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.034 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.034 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.035 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.035 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:54.035 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:15:54.036 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.036 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.036 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.036 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.037 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.037 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.037 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:54.038 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.038 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.038 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.039 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.039 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:15:54.039 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.040 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:15:54.040 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.041 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.041 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.041 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.042 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.042 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.042 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:15:54.043 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.043 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:15:54.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.044 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.044 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.045 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.045 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.045 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.046 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.046 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:15:54.046 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.047 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.047 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.047 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:15:54.047 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.048 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.048 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.048 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.048 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.049 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.049 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.049 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:15:54.049 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.050 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.050 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.050 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.050 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:15:54.051 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:15:54.051 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.051 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.051 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.052 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:15:54.052 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:15:54.052 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:15:54.052 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:15:54.052 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.053 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.053 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.053 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.054 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.054 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:54.054 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:15:54.054 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.055 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.055 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.055 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.055 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.055 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.056 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.056 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-20T03:15:54.056 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.056 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.057 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.057 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.057 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.057 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:15:54.058 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.058 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.058 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.058 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:15:54.059 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.059 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.059 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.060 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.060 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.061 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:15:54.062 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:15:54.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.064 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.065 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.066 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.067 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:15:54.067 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.067 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.068 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.069 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:15:54.069 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.070 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.070 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.071 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.071 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.072 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.072 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.073 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.073 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:15:54.073 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.075 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.075 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:15:54.075 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:15:54.076 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.076 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.077 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.077 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:15:54.078 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:15:54.078 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:15:54.079 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:15:54.079 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:15:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.083 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.083 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.084 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.084 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.085 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:54.085 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.086 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.086 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.088 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:15:54.088 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:15:54.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.090 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.090 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.090 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:15:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.093 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:15:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.095 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.095 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.095 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.095 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:15:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:15:54.098 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.098 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.098 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.099 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.099 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.099 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.100 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.100 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.100 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:15:54.100 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:15:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:15:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:15:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:15:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:15:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:15:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:15:54.106 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.106 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.106 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.106 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-20T03:15:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.109 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.109 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.109 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.110 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:15:54.110 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.111 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.112 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.112 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:15:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.114 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.114 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.115 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.115 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.115 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.116 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:15:54.116 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.116 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.118 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.118 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:15:54.119 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.119 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.120 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.120 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.120 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.120 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.121 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:15:54.121 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.121 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.121 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.122 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.122 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:15:54.122 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.123 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.123 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.123 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.123 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.123 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.124 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:15:54.124 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.125 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.125 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.125 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:15:54.126 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:15:54.126 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.126 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.126 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.126 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:15:54.127 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:15:54.127 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:15:54.127 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:15:54.127 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.128 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.128 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.128 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.130 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:54.130 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:15:54.130 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.131 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.131 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.132 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.132 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.133 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.133 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.134 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 98, 2022-07-20T03:15:54.134 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.134 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.135 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:15:54.137 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.138 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.138 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.139 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.139 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:15:54.139 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.140 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.141 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.141 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.141 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.142 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.142 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:15:54.143 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.143 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.144 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:15:54.145 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.145 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.145 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.146 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.146 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.147 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.147 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:15:54.148 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.148 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.149 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:15:54.149 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.149 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.150 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.150 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.150 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.151 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.151 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:15:54.152 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.152 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.152 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.153 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:15:54.153 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:15:54.153 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.153 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.154 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.154 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:15:54.154 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:15:54.154 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:15:54.155 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:15:54.155 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.155 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.155 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.156 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.156 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.157 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:54.157 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:15:54.157 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.158 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.158 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.159 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.159 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.159 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.160 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.160 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:54.161 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.161 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.162 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.162 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.162 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.163 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.164 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:15:54.165 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.165 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.165 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.165 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.166 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:15:54.166 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.167 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.167 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.168 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.168 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.169 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.170 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:15:54.171 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.171 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.172 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:15:54.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.175 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.175 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.176 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:15:54.176 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.177 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:15:54.178 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.178 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.179 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.179 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.180 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.181 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.181 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:15:54.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.181 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.182 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.182 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:15:54.182 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:15:54.183 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.183 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.183 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.183 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:15:54.184 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:15:54.184 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:15:54.184 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:15:54.184 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.185 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.185 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.185 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.186 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.186 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.186 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:54.187 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:15:54.187 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:15:54.187 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:15:54.187 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:15:54.188 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:15:54.188 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:15:54.189 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:15:54.189 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:15:54.189 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 99, 2022-07-20T03:15:54.189 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:15:54.190 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:15:54.190 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:15:54.191 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.191 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.192 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.192 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:15:54.193 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.194 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.195 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.195 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:15:54.196 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.196 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.197 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.197 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.198 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:15:54.198 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.199 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.199 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.199 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:15:54.200 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.200 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.201 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.202 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:15:54.202 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.203 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.203 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.204 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.204 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:15:54.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.205 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:15:54.206 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.207 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.207 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.208 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:15:54.209 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.209 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.209 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.210 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.210 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:15:54.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:15:54.211 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.212 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:15:54.212 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:15:54.213 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.213 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.214 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.214 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:15:54.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:15:54.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:15:54.216 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.216 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.217 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:15:54.217 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:15:54.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:15:54.218 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:15:54.219 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:15:54.219 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:15:54.220 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:15:54.220 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:54.223 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.223 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:54.223 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:15:54.224 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.224 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:15:54.224 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.225 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.225 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.225 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.226 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:15:54.226 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.227 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.227 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.227 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:15:54.227 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.227 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.228 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.228 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.228 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.228 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:15:54.229 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.229 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:15:54.230 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.230 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.230 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.230 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.231 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.231 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:15:54.231 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.231 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.232 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.232 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:15:54.232 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.232 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.233 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.233 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.233 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.233 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:15:54.234 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.234 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.234 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.235 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:15:54.235 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.235 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.235 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.236 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.236 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.236 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:15:54.236 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.237 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:15:54.237 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.238 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.238 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.238 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.239 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.239 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:15:54.239 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.240 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:15:54.241 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.241 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.242 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.242 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.243 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.243 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:15:54.244 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:54.245 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:15:54.246 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:15:54.246 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:15:54.247 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:15:54.247 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:15:54.248 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:15:54.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:15:54.249 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:15:54.249 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.250 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:15:54.250 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:15:54.251 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:15:54.251 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:15:54.252 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:15:54.252 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:15:54.253 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/2834789489": "2022-07-21T03:15:47.933433+0000", 2022-07-20T03:15:54.253 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/2834789489": "2022-07-21T03:15:47.933433+0000" 2022-07-20T03:15:54.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.254 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:15:54.255 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:15:54.255 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:15:54.256 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:15:54.256 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:15:54.257 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:15:54.257 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:15:54.258 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:15:54.258 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:54.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:54.259 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:15:54.260 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:15:54.260 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:15:54.261 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:15:54.261 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:15:54.262 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:54.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ca0053a60 msgr2=0x7f0ca0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ca0053a60 0x7f0ca0055f20 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7f0cb001b000 tx=0x7f0cb0016000).stop 2022-07-20T03:15:54.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 msgr2=0x7f0cc00661d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 0x7f0cc00661d0 secure :-1 s=READY pgs=1107 cs=0 l=1 rev1=1 rx=0x7f0cc0141c90 tx=0x7f0cb8006df0).stop 2022-07-20T03:15:54.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 shutdown_connections 2022-07-20T03:15:54.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0ca0053a60 0x7f0ca0055f20 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0cc013f600 0x7f0cc00661d0 unknown :-1 s=CLOSED pgs=1107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0cc00b1a70 0x7f0cc0065ce0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 --2- 172.21.15.61:0/1673096633 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0cc00aea20 0x7f0cc0145e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 >> 172.21.15.61:0/1673096633 conn(0x7f0cc00675f0 msgr2=0x7f0cc0143900 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:54.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 shutdown_connections 2022-07-20T03:15:54.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:53.948+0000 7f0cc7fa0700 1 -- 172.21.15.61:0/1673096633 wait complete. 2022-07-20T03:15:54.273 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:54.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd130f8a700 1 Processor -- start 2022-07-20T03:15:54.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd130f8a700 1 -- start start 2022-07-20T03:15:54.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd130f8a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd12c0aebb0 0x7fd12c0b0690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd130f8a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0aef90 con 0x7fd12c13dc80 2022-07-20T03:15:54.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd130f8a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd12c0b0270 con 0x7fd12c0acae0 2022-07-20T03:15:54.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd130f8a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd12c0af200 con 0x7fd12c0aebb0 2022-07-20T03:15:54.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd12c13dc80 0x7fd12c13e0d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45892/0 (socket says 172.21.15.61:45892) 2022-07-20T03:15:54.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12bfff700 1 -- 172.21.15.61:0/3748579851 learned_addr learned my addr 172.21.15.61:0/3748579851 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:54.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12b7fe700 1 --2- 172.21.15.61:0/3748579851 >> v2:172.21.15.117:3300/0 conn(0x7fd12c0aebb0 0x7fd12c0b0690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12a7fc700 1 -- 172.21.15.61:0/3748579851 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3052575172 0 0) 0x7fd12c0aef90 con 0x7fd12c13dc80 2022-07-20T03:15:54.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12a7fc700 1 -- 172.21.15.61:0/3748579851 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd118003260 con 0x7fd12c13dc80 2022-07-20T03:15:54.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12b7fe700 1 -- 172.21.15.61:0/3748579851 >> v1:172.21.15.117:6789/0 conn(0x7fd12c0acae0 legacy=0x7fd12c13f870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12b7fe700 1 -- 172.21.15.61:0/3748579851 >> v1:172.21.15.61:6789/0 conn(0x7fd12c13dc80 legacy=0x7fd12c13e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12b7fe700 1 -- 172.21.15.61:0/3748579851 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd12c0af4a0 con 0x7fd12c0aebb0 2022-07-20T03:15:54.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.388+0000 7fd12b7fe700 1 --2- 172.21.15.61:0/3748579851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0aebb0 0x7fd12c0b0690 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fd11c002f70 tx=0x7fd11c006df0).ready entity=mon.1 client_cookie=efcb2376c3e1ed81 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:54.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12a7fc700 1 -- 172.21.15.61:0/3748579851 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd11c00d610 con 0x7fd12c0aebb0 2022-07-20T03:15:54.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12a7fc700 1 -- 172.21.15.61:0/3748579851 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd11c000e90 con 0x7fd12c0aebb0 2022-07-20T03:15:54.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12a7fc700 1 -- 172.21.15.61:0/3748579851 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd11c00d960 con 0x7fd12c0aebb0 2022-07-20T03:15:54.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- 172.21.15.61:0/3748579851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0aebb0 msgr2=0x7fd12c0b0690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 --2- 172.21.15.61:0/3748579851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0aebb0 0x7fd12c0b0690 secure :-1 s=READY pgs=503 cs=0 l=1 rev1=1 rx=0x7fd11c002f70 tx=0x7fd11c006df0).stop 2022-07-20T03:15:54.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- 172.21.15.61:0/3748579851 shutdown_connections 2022-07-20T03:15:54.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 --2- 172.21.15.61:0/3748579851 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0aebb0 0x7fd12c0b0690 unknown :-1 s=CLOSED pgs=503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- 172.21.15.61:0/3748579851 >> 172.21.15.61:0/3748579851 conn(0x7fd12c0634e0 msgr2=0x7fd12c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:54.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- 172.21.15.61:0/3748579851 shutdown_connections 2022-07-20T03:15:54.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- 172.21.15.61:0/3748579851 wait complete. 2022-07-20T03:15:54.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 Processor -- start 2022-07-20T03:15:54.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- start start 2022-07-20T03:15:54.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0acae0 0x7fd12c1437c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 0x7fd12c147930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd12c13dc80 0x7fd12c147e20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c066440 con 0x7fd12c0aebb0 2022-07-20T03:15:54.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c1461f0 con 0x7fd12c0acae0 2022-07-20T03:15:54.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd130f8a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd12c148310 con 0x7fd12c13dc80 2022-07-20T03:15:54.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 0x7fd12c147930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 0x7fd12c147930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49302/0 (socket says 172.21.15.61:49302) 2022-07-20T03:15:54.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 -- 172.21.15.61:0/2788543462 learned_addr learned my addr 172.21.15.61:0/2788543462 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:54.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12b7fe700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0acae0 0x7fd12c1437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12bfff700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd12c13dc80 0x7fd12c147e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 -- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd12c13dc80 msgr2=0x7fd12c147e20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd12c13dc80 0x7fd12c147e20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 -- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0acae0 msgr2=0x7fd12c1437c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0acae0 0x7fd12c1437c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12affd700 1 -- 172.21.15.61:0/2788543462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd11c00b180 con 0x7fd12c0aebb0 2022-07-20T03:15:54.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.392+0000 7fd12bfff700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd12c13dc80 0x7fd12c147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:54.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd12affd700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 0x7fd12c147930 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7fd12c060b30 tx=0x7fd120007ba0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:54.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd12000cea0 con 0x7fd12c0aebb0 2022-07-20T03:15:54.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd120012400 con 0x7fd12c0aebb0 2022-07-20T03:15:54.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd12c145f00 con 0x7fd12c0aebb0 2022-07-20T03:15:54.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd120012760 con 0x7fd12c0aebb0 2022-07-20T03:15:54.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd12c066040 con 0x7fd12c0aebb0 2022-07-20T03:15:54.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd12c13ff80 con 0x7fd12c0aebb0 2022-07-20T03:15:54.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd120019030 con 0x7fd12c0aebb0 2022-07-20T03:15:54.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd128ff9700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd10c053920 0x7fd10c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.396+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd12006ae70 con 0x7fd12c0aebb0 2022-07-20T03:15:54.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.400+0000 7fd12b7fe700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd10c053920 0x7fd10c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.400+0000 7fd12b7fe700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd10c053920 0x7fd10c055de0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fd11c01a000 tx=0x7fd11c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:54.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.404+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd1200420b0 con 0x7fd12c0aebb0 2022-07-20T03:15:54.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fd12c0574e0 con 0x7fd12c0aebb0 2022-07-20T03:15:54.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd128ff9700 1 -- 172.21.15.61:0/2788543462 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1212 (secure 0 0 0) 0x7fd1200456d0 con 0x7fd12c0aebb0 2022-07-20T03:15:54.573 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:54.573 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54},{"gid":7587,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3393349352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3393349352},{"type":"v1","addr":"172.21.15.61:6833","nonce":3393349352}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":58}],"filesystems":[]} 2022-07-20T03:15:54.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd10c053920 msgr2=0x7fd10c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd10c053920 0x7fd10c055de0 secure :-1 s=READY pgs=536 cs=0 l=1 rev1=1 rx=0x7fd11c01a000 tx=0x7fd11c014000).stop 2022-07-20T03:15:54.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 msgr2=0x7fd12c147930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 0x7fd12c147930 secure :-1 s=READY pgs=1109 cs=0 l=1 rev1=1 rx=0x7fd12c060b30 tx=0x7fd120007ba0).stop 2022-07-20T03:15:54.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 shutdown_connections 2022-07-20T03:15:54.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd10c053920 0x7fd10c055de0 unknown :-1 s=CLOSED pgs=536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd12c13dc80 0x7fd12c147e20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd12c0aebb0 0x7fd12c147930 unknown :-1 s=CLOSED pgs=1109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 --2- 172.21.15.61:0/2788543462 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd12c0acae0 0x7fd12c1437c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 >> 172.21.15.61:0/2788543462 conn(0x7fd12c0634e0 msgr2=0x7fd12c140350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:54.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.568+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 shutdown_connections 2022-07-20T03:15:54.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.572+0000 7fd130f8a700 1 -- 172.21.15.61:0/2788543462 wait complete. 2022-07-20T03:15:54.580 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 58 2022-07-20T03:15:54.586 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:15:54.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 Processor -- start 2022-07-20T03:15:54.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 -- start start 2022-07-20T03:15:54.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5dd805c3f0 0x7f5dd80ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5dd805bd90 0x7f5dd80accd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd805d480 con 0x7f5dd805c3f0 2022-07-20T03:15:54.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5dd80ad1c0 con 0x7f5dd805bd90 2022-07-20T03:15:54.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5ddd49a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80b21e0 con 0x7f5dd80b1e00 2022-07-20T03:15:54.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.688+0000 7f5dd67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5dd805c3f0 0x7f5dd80ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5dd805c3f0 0x7f5dd80ac7e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49306/0 (socket says 172.21.15.61:49306) 2022-07-20T03:15:54.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 -- 172.21.15.61:0/2966555689 learned_addr learned my addr 172.21.15.61:0/2966555689 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:54.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd77fe700 1 --2- 172.21.15.61:0/2966555689 >> v2:172.21.15.117:3300/0 conn(0x7f5dd805bd90 0x7f5dd80accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 -- 172.21.15.61:0/2966555689 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 msgr2=0x7f5dd814a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:54.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd6ffd700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd814a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 -- 172.21.15.61:0/2966555689 >> v2:172.21.15.117:3300/0 conn(0x7f5dd805bd90 msgr2=0x7f5dd80accd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:54.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 --2- 172.21.15.61:0/2966555689 >> v2:172.21.15.117:3300/0 conn(0x7f5dd805bd90 0x7f5dd80accd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 -- 172.21.15.61:0/2966555689 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dd805cf40 con 0x7f5dd805c3f0 2022-07-20T03:15:54.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd77fe700 1 --2- 172.21.15.61:0/2966555689 >> v2:172.21.15.117:3300/0 conn(0x7f5dd805bd90 0x7f5dd80accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:54.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd6ffd700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:15:54.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd80ac7e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f5dc8002f70 tx=0x7f5dc8006eb0).ready entity=mon.0 client_cookie=8ebe369e18f101a2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:54.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd5ffb700 1 -- 172.21.15.61:0/2966555689 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc800d570 con 0x7f5dd805c3f0 2022-07-20T03:15:54.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd5ffb700 1 -- 172.21.15.61:0/2966555689 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dc8000e90 con 0x7f5dd805c3f0 2022-07-20T03:15:54.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd5ffb700 1 -- 172.21.15.61:0/2966555689 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dc800d900 con 0x7f5dd805c3f0 2022-07-20T03:15:54.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- 172.21.15.61:0/2966555689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 msgr2=0x7f5dd80ac7e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd80ac7e0 secure :-1 s=READY pgs=1110 cs=0 l=1 rev1=1 rx=0x7f5dc8002f70 tx=0x7f5dc8006eb0).stop 2022-07-20T03:15:54.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- 172.21.15.61:0/2966555689 shutdown_connections 2022-07-20T03:15:54.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/2966555689 >> v2:172.21.15.117:3300/0 conn(0x7f5dd805bd90 0x7f5dd80accd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd80ac7e0 unknown :-1 s=CLOSED pgs=1110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/2966555689 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd814a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- 172.21.15.61:0/2966555689 >> 172.21.15.61:0/2966555689 conn(0x7f5dd80611c0 msgr2=0x7f5dd8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:54.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- 172.21.15.61:0/2966555689 shutdown_connections 2022-07-20T03:15:54.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- 172.21.15.61:0/2966555689 wait complete. 2022-07-20T03:15:54.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 Processor -- start 2022-07-20T03:15:54.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- start start 2022-07-20T03:15:54.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd8148e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd8143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5dd814a5b0 0x7f5dd8143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80b5e60 con 0x7f5dd805c3f0 2022-07-20T03:15:54.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80b4560 con 0x7f5dd814a5b0 2022-07-20T03:15:54.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5ddd49a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5dd80b4800 con 0x7f5dd80b1e00 2022-07-20T03:15:54.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd8148e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd8148e00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49316/0 (socket says 172.21.15.61:49316) 2022-07-20T03:15:54.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd77fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5dd814a5b0 0x7f5dd8143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd6ffd700 1 -- 172.21.15.61:0/3999202876 learned_addr learned my addr 172.21.15.61:0/3999202876 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:54.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.692+0000 7f5dd67fc700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd8143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd6ffd700 1 -- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 msgr2=0x7f5dd8143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd6ffd700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd8143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd6ffd700 1 -- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5dd814a5b0 msgr2=0x7f5dd8143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd6ffd700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5dd814a5b0 0x7f5dd8143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd6ffd700 1 -- 172.21.15.61:0/3999202876 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5dc800b1e0 con 0x7f5dd805c3f0 2022-07-20T03:15:54.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd6ffd700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd8148e00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f5dcc007000 tx=0x7f5dcc008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:54.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dcc00ce80 con 0x7f5dd805c3f0 2022-07-20T03:15:54.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5dcc010580 con 0x7f5dd805c3f0 2022-07-20T03:15:54.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5dd80b4aa0 con 0x7f5dd805c3f0 2022-07-20T03:15:54.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5dcc0108e0 con 0x7f5dd805c3f0 2022-07-20T03:15:54.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5dd80b4f40 con 0x7f5dd805c3f0 2022-07-20T03:15:54.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5dd805d1e0 con 0x7f5dd805c3f0 2022-07-20T03:15:54.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5dcc010a40 con 0x7f5dd805c3f0 2022-07-20T03:15:54.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dc7fff700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5db4053ad0 0x7f5db4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:54.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(104..104 src has 1..104) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f5dcc06b920 con 0x7f5dd805c3f0 2022-07-20T03:15:54.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.696+0000 7f5dd67fc700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5db4053ad0 0x7f5db4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:54.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.700+0000 7f5dd67fc700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5db4053ad0 0x7f5db4055f90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f5dc801c000 tx=0x7f5dc8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:54.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.704+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5dcc042b60 con 0x7f5dd805c3f0 2022-07-20T03:15:54.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.876+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5dd80574e0 con 0x7f5dd805c3f0 2022-07-20T03:15:54.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.876+0000 7f5dc7fff700 1 -- 172.21.15.61:0/3999202876 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 58 v58) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f5dcc046180 con 0x7f5dd805c3f0 2022-07-20T03:15:54.881 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:54.881 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":58,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7267,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/1955139228","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1955139228},{"type":"v1","addr":"172.21.15.117:6835","nonce":1955139228}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":54},{"gid":7587,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3393349352","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3393349352},{"type":"v1","addr":"172.21.15.61:6833","nonce":3393349352}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":58}],"filesystems":[]} 2022-07-20T03:15:54.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5db4053ad0 msgr2=0x7f5db4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5db4053ad0 0x7f5db4055f90 secure :-1 s=READY pgs=537 cs=0 l=1 rev1=1 rx=0x7f5dc801c000 tx=0x7f5dc8015000).stop 2022-07-20T03:15:54.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 msgr2=0x7f5dd8148e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:54.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd8148e00 secure :-1 s=READY pgs=1111 cs=0 l=1 rev1=1 rx=0x7f5dcc007000 tx=0x7f5dcc008040).stop 2022-07-20T03:15:54.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 shutdown_connections 2022-07-20T03:15:54.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5db4053ad0 0x7f5db4055f90 unknown :-1 s=CLOSED pgs=537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5dd814a5b0 0x7f5dd8143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5dd80b1e00 0x7f5dd8143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 --2- 172.21.15.61:0/3999202876 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5dd805c3f0 0x7f5dd8148e00 unknown :-1 s=CLOSED pgs=1111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:54.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 >> 172.21.15.61:0/3999202876 conn(0x7f5dd805bd90 msgr2=0x7f5dd80ac700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:54.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 shutdown_connections 2022-07-20T03:15:54.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:54.880+0000 7f5ddd49a700 1 -- 172.21.15.61:0/3999202876 wait complete. 2022-07-20T03:15:54.889 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 58 2022-07-20T03:15:54.895 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:15:54.896 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:15:54.896 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:15:54.896 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:15:54.897 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:15:54.897 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:15:54.898 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:15:54.898 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:15:54.922 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:15:54.923 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:15:54.928 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:15:54.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:15:54.941 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:15:54.942 DEBUG:teuthology.parallel:result is None 2022-07-20T03:15:54.943 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:15:54.944 DEBUG:teuthology.parallel:result is None 2022-07-20T03:15:54.945 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:15:55.011 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:15:55.016 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:15:55.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.044+0000 7f74d5c1d700 1 Processor -- start 2022-07-20T03:15:55.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.044+0000 7f74d5c1d700 1 -- start start 2022-07-20T03:15:55.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d5c1d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f74d00b6060 0x7f74d0144710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:55.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d5c1d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d00b0ff0 con 0x7f74d0144c00 2022-07-20T03:15:55.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d5c1d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74d00b4440 con 0x7f74d00b51e0 2022-07-20T03:15:55.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d5c1d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74d0143fd0 con 0x7f74d00b6060 2022-07-20T03:15:55.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f74d0144c00 0x7f74d00b01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45938/0 (socket says 172.21.15.61:45938) 2022-07-20T03:15:55.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cffff700 1 -- 172.21.15.61:0/1893493571 learned_addr learned my addr 172.21.15.61:0/1893493571 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:55.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d4c1b700 1 --2- 172.21.15.61:0/1893493571 >> v2:172.21.15.117:3300/0 conn(0x7f74d00b6060 0x7f74d0144710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:55.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cf7fe700 1 -- 172.21.15.61:0/1893493571 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2864980772 0 0) 0x7f74d00b0ff0 con 0x7f74d0144c00 2022-07-20T03:15:55.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cf7fe700 1 -- 172.21.15.61:0/1893493571 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74bc003260 con 0x7f74d0144c00 2022-07-20T03:15:55.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d4c1b700 1 -- 172.21.15.61:0/1893493571 >> v1:172.21.15.117:6789/0 conn(0x7f74d00b51e0 legacy=0x7f74d00b08e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:55.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d4c1b700 1 -- 172.21.15.61:0/1893493571 >> v1:172.21.15.61:6789/0 conn(0x7f74d0144c00 legacy=0x7f74d00b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:55.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d4c1b700 1 -- 172.21.15.61:0/1893493571 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74d0144270 con 0x7f74d00b6060 2022-07-20T03:15:55.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74d4c1b700 1 --2- 172.21.15.61:0/1893493571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0144710 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f74c0002f70 tx=0x7f74c0006df0).ready entity=mon.1 client_cookie=17353b20f2a553ff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:55.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cf7fe700 1 -- 172.21.15.61:0/1893493571 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74c000d610 con 0x7f74d00b6060 2022-07-20T03:15:55.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cf7fe700 1 -- 172.21.15.61:0/1893493571 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c0000e90 con 0x7f74d00b6060 2022-07-20T03:15:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.048+0000 7f74cf7fe700 1 -- 172.21.15.61:0/1893493571 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74c000d960 con 0x7f74d00b6060 2022-07-20T03:15:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1893493571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 msgr2=0x7f74d0144710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:55.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1893493571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0144710 secure :-1 s=READY pgs=507 cs=0 l=1 rev1=1 rx=0x7f74c0002f70 tx=0x7f74c0006df0).stop 2022-07-20T03:15:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1893493571 shutdown_connections 2022-07-20T03:15:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1893493571 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0144710 unknown :-1 s=CLOSED pgs=507 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:55.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1893493571 >> 172.21.15.61:0/1893493571 conn(0x7f74d0063d10 msgr2=0x7f74d00b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:55.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1893493571 shutdown_connections 2022-07-20T03:15:55.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1893493571 wait complete. 2022-07-20T03:15:55.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 Processor -- start 2022-07-20T03:15:55.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.052+0000 7f74d5c1d700 1 -- start start 2022-07-20T03:15:55.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74d00b51e0 0x7f74d01432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:55.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:55.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 0x7f74d0152570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:55.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d0065850 con 0x7f74d0144c00 2022-07-20T03:15:55.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74d009ba90 con 0x7f74d00b6060 2022-07-20T03:15:55.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74d00aba10 con 0x7f74d00b51e0 2022-07-20T03:15:55.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 0x7f74d0152570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:55.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 0x7f74d0152570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49348/0 (socket says 172.21.15.61:49348) 2022-07-20T03:15:55.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 -- 172.21.15.61:0/1122839875 learned_addr learned my addr 172.21.15.61:0/1122839875 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:55.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d4c1b700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74d00b51e0 0x7f74d01432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:55.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74cffff700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:55.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 -- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74d00b51e0 msgr2=0x7f74d01432f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:55.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74d00b51e0 0x7f74d01432f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:55.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 -- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 msgr2=0x7f74d0152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:55.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:55.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 -- 172.21.15.61:0/1122839875 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74c000b180 con 0x7f74d0144c00 2022-07-20T03:15:55.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d541c700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 0x7f74d0152570 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f74d006ad30 tx=0x7f74c8007f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:55.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74c800f470 con 0x7f74d0144c00 2022-07-20T03:15:55.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74c800f5d0 con 0x7f74d0144c00 2022-07-20T03:15:55.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74d00670a0 con 0x7f74d0144c00 2022-07-20T03:15:55.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74c800f930 con 0x7f74d0144c00 2022-07-20T03:15:55.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74d0152a60 con 0x7f74d0144c00 2022-07-20T03:15:55.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.056+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74d00b40c0 con 0x7f74d0144c00 2022-07-20T03:15:55.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.060+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f74c8019030 con 0x7f74d0144c00 2022-07-20T03:15:55.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.060+0000 7f74cdffb700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74b0053a40 0x7f74b0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:55.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.060+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(106..106 src has 1..106) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f74c806b2a0 con 0x7f74d0144c00 2022-07-20T03:15:55.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.060+0000 7f74d4c1b700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74b0053a40 0x7f74b0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:55.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.060+0000 7f74d4c1b700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74b0053a40 0x7f74b0055f00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f74c001a000 tx=0x7f74c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:55.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.068+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74c80423d0 con 0x7f74d0144c00 2022-07-20T03:15:55.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:55.208+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f74d00574e0 con 0x7f74d0144c00 2022-07-20T03:15:56.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.208+0000 7f74cdffb700 1 -- 172.21.15.61:0/1122839875 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v107) v1 ==== 97+0+0 (secure 0 0 0) 0x7f74c80459f0 con 0x7f74d0144c00 2022-07-20T03:15:56.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74b0053a40 msgr2=0x7f74b0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74b0053a40 0x7f74b0055f00 secure :-1 s=READY pgs=538 cs=0 l=1 rev1=1 rx=0x7f74c001a000 tx=0x7f74c0014000).stop 2022-07-20T03:15:56.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 msgr2=0x7f74d0152570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 0x7f74d0152570 secure :-1 s=READY pgs=1118 cs=0 l=1 rev1=1 rx=0x7f74d006ad30 tx=0x7f74c8007f40).stop 2022-07-20T03:15:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 shutdown_connections 2022-07-20T03:15:56.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74b0053a40 0x7f74b0055f00 unknown :-1 s=CLOSED pgs=538 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74d00b6060 0x7f74d0152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74d0144c00 0x7f74d0152570 unknown :-1 s=CLOSED pgs=1118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 --2- 172.21.15.61:0/1122839875 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74d00b51e0 0x7f74d01432f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 >> 172.21.15.61:0/1122839875 conn(0x7f74d0063d10 msgr2=0x7f74d01460c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:56.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 shutdown_connections 2022-07-20T03:15:56.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.212+0000 7f74d5c1d700 1 -- 172.21.15.61:0/1122839875 wait complete. 2022-07-20T03:15:56.223 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:15:56.233 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:15:56.234 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:15:56.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.340+0000 7fa30eea3700 1 Processor -- start 2022-07-20T03:15:56.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.340+0000 7fa30eea3700 1 -- start start 2022-07-20T03:15:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa3080b1200 0x7fa3080b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa30805bd90 0x7fa308142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080aea30 con 0x7fa3080b5650 2022-07-20T03:15:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa308142970 con 0x7fa3080b1200 2022-07-20T03:15:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080aec60 con 0x7fa30805bd90 2022-07-20T03:15:56.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa3080b1200 0x7fa3080b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa3080b1200 0x7fa3080b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49354/0 (socket says 172.21.15.61:49354) 2022-07-20T03:15:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 -- 172.21.15.61:0/4017781462 learned_addr learned my addr 172.21.15.61:0/4017781462 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30e6a2700 1 --2- 172.21.15.61:0/4017781462 >> v2:172.21.15.117:3300/0 conn(0x7fa30805bd90 0x7fa308142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 -- 172.21.15.61:0/4017781462 >> v2:172.21.15.117:3300/0 conn(0x7fa30805bd90 msgr2=0x7fa308142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 --2- 172.21.15.61:0/4017781462 >> v2:172.21.15.117:3300/0 conn(0x7fa30805bd90 0x7fa308142d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 -- 172.21.15.61:0/4017781462 >> v1:172.21.15.117:6789/0 conn(0x7fa3080b5650 legacy=0x7fa30805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 -- 172.21.15.61:0/4017781462 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080af180 con 0x7fa3080b1200 2022-07-20T03:15:56.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30d6a0700 1 --2- 172.21.15.61:0/4017781462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b1200 0x7fa3080b5a30 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7fa304002f70 tx=0x7fa304006e50).ready entity=mon.0 client_cookie=1c146f5e92e62ceb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:56.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30ce9f700 1 -- 172.21.15.61:0/4017781462 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa30400d610 con 0x7fa3080b1200 2022-07-20T03:15:56.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30ce9f700 1 -- 172.21.15.61:0/4017781462 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa304000e90 con 0x7fa3080b1200 2022-07-20T03:15:56.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30ce9f700 1 -- 172.21.15.61:0/4017781462 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa30400d9a0 con 0x7fa3080b1200 2022-07-20T03:15:56.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- 172.21.15.61:0/4017781462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b1200 msgr2=0x7fa3080b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 --2- 172.21.15.61:0/4017781462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b1200 0x7fa3080b5a30 secure :-1 s=READY pgs=1119 cs=0 l=1 rev1=1 rx=0x7fa304002f70 tx=0x7fa304006e50).stop 2022-07-20T03:15:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- 172.21.15.61:0/4017781462 shutdown_connections 2022-07-20T03:15:56.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 --2- 172.21.15.61:0/4017781462 >> v2:172.21.15.117:3300/0 conn(0x7fa30805bd90 0x7fa308142d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 --2- 172.21.15.61:0/4017781462 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b1200 0x7fa3080b5a30 unknown :-1 s=CLOSED pgs=1119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- 172.21.15.61:0/4017781462 >> 172.21.15.61:0/4017781462 conn(0x7fa3080634e0 msgr2=0x7fa3080b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- 172.21.15.61:0/4017781462 shutdown_connections 2022-07-20T03:15:56.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.344+0000 7fa30eea3700 1 -- 172.21.15.61:0/4017781462 wait complete. 2022-07-20T03:15:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 Processor -- start 2022-07-20T03:15:56.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 -- start start 2022-07-20T03:15:56.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa30805bd90 0x7fa308144530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080b1200 0x7fa30813d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 0x7fa308151f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa30813e420 con 0x7fa3080b5650 2022-07-20T03:15:56.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa308142720 con 0x7fa3080b1200 2022-07-20T03:15:56.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30eea3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30813cff0 con 0x7fa30805bd90 2022-07-20T03:15:56.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30e6a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 0x7fa308151f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30e6a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 0x7fa308151f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49364/0 (socket says 172.21.15.61:49364) 2022-07-20T03:15:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30e6a2700 1 -- 172.21.15.61:0/1749134879 learned_addr learned my addr 172.21.15.61:0/1749134879 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30dea1700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa30805bd90 0x7fa308144530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.348+0000 7fa30d6a0700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080b1200 0x7fa30813d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30e6a2700 1 -- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa30805bd90 msgr2=0x7fa308144530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30e6a2700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa30805bd90 0x7fa308144530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30e6a2700 1 -- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080b1200 msgr2=0x7fa30813d730 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30e6a2700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080b1200 0x7fa30813d730 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30e6a2700 1 -- 172.21.15.61:0/1749134879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa30400b1e0 con 0x7fa3080b5650 2022-07-20T03:15:56.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30e6a2700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 0x7fa308151f40 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7fa30000a000 tx=0x7fa300008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:56.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa30000c070 con 0x7fa3080b5650 2022-07-20T03:15:56.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa3000133f0 con 0x7fa3080b5650 2022-07-20T03:15:56.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30813d290 con 0x7fa3080b5650 2022-07-20T03:15:56.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa30813dc20 con 0x7fa3080b5650 2022-07-20T03:15:56.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa300013750 con 0x7fa3080b5650 2022-07-20T03:15:56.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa3000138b0 con 0x7fa3080b5650 2022-07-20T03:15:56.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3080574e0 con 0x7fa3080b5650 2022-07-20T03:15:56.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa2f6ffd700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e8053ad0 0x7fa2e8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.352+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa30006ba20 con 0x7fa3080b5650 2022-07-20T03:15:56.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.356+0000 7fa30dea1700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e8053ad0 0x7fa2e8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.356+0000 7fa30dea1700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e8053ad0 0x7fa2e8055f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fa2f800d000 tx=0x7fa2f8007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:56.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.360+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa300042ba0 con 0x7fa3080b5650 2022-07-20T03:15:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.528+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7fa308067ea0 con 0x7fa3080b5650 2022-07-20T03:15:56.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa2f6ffd700 1 -- 172.21.15.61:0/1749134879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v37) v1 ==== 80+0+4242 (secure 0 0 0) 0x7fa30000f040 con 0x7fa3080b5650 2022-07-20T03:15:56.537 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:56.538 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:15:56.539 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:15:56.539 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.539 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:15:56.540 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:15:56.540 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.541 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:56.541 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.541 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.542 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.542 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.542 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.543 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.543 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:15:56.543 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:15:56.543 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:56.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.545 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.545 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.545 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.545 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:15:56.546 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:15:56.546 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.546 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.547 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.547 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.547 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.548 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.548 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:15:56.548 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:15:56.549 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.549 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.550 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.551 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:15:56.552 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:15:56.552 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.553 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.553 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.554 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.554 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.555 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:15:56.555 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:15:56.555 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.556 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.557 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.557 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.557 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.558 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:15:56.559 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:15:56.559 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.560 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.561 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.561 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.562 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.562 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.563 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:15:56.563 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:15:56.563 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.564 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.564 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.565 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.565 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.567 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.568 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.569 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:15:56.570 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:15:56.571 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.571 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.572 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.572 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.572 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.573 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.573 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.574 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:15:56.574 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:15:56.574 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.575 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.575 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.576 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.576 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.577 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.577 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.577 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:15:56.578 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:15:56.578 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.579 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:56.579 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:56.580 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:56.587 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.587 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.588 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.588 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:15:56.588 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:15:56.589 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.589 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:56.596 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:56.596 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:56.597 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.598 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.598 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.599 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:15:56.599 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:15:56.600 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.600 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:15:56.600 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:15:56.601 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:15:56.601 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.602 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.603 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.604 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:15:56.604 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:15:56.605 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.605 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:56.606 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:15:56.606 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:15:56.607 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.607 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.608 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.609 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.609 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:15:56.610 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:15:56.611 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.611 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:56.612 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:15:56.612 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.613 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.614 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:56.615 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:56.615 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:15:56.616 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:15:56.616 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:15:56.617 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:15:56.617 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:15:56.618 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:15:56.618 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.619 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:56.619 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:15:56.620 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:15:56.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e8053ad0 msgr2=0x7fa2e8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e8053ad0 0x7fa2e8055f90 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x7fa2f800d000 tx=0x7fa2f8007b90).stop 2022-07-20T03:15:56.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 msgr2=0x7fa308151f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 0x7fa308151f40 secure :-1 s=READY pgs=1120 cs=0 l=1 rev1=1 rx=0x7fa30000a000 tx=0x7fa300008040).stop 2022-07-20T03:15:56.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 shutdown_connections 2022-07-20T03:15:56.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e8053ad0 0x7fa2e8055f90 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080b1200 0x7fa30813d730 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080b5650 0x7fa308151f40 unknown :-1 s=CLOSED pgs=1120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 --2- 172.21.15.61:0/1749134879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa30805bd90 0x7fa308144530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 >> 172.21.15.61:0/1749134879 conn(0x7fa3080634e0 msgr2=0x7fa3080b3b50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:56.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 shutdown_connections 2022-07-20T03:15:56.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.532+0000 7fa30eea3700 1 -- 172.21.15.61:0/1749134879 wait complete. 2022-07-20T03:15:56.630 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:56.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 Processor -- start 2022-07-20T03:15:56.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 -- start start 2022-07-20T03:15:56.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcd501427a0 0x7fcd50061130 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcd50061620 0x7fcd50061ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcd500b22d0 con 0x7fcd500b1ef0 2022-07-20T03:15:56.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd501472a0 con 0x7fcd50061620 2022-07-20T03:15:56.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4f59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcd50142c50 con 0x7fcd501427a0 2022-07-20T03:15:56.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4e59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcd500b1ef0 0x7fcd50142090 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45966/0 (socket says 172.21.15.61:45966) 2022-07-20T03:15:56.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4e59c700 1 -- 172.21.15.61:0/4222507234 learned_addr learned my addr 172.21.15.61:0/4222507234 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:56.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4ed9d700 1 --2- 172.21.15.61:0/4222507234 >> v2:172.21.15.61:3300/0 conn(0x7fcd50061620 0x7fcd50061ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4dd9b700 1 --2- 172.21.15.61:0/4222507234 >> v2:172.21.15.117:3300/0 conn(0x7fcd501427a0 0x7fcd50061130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4d59a700 1 -- 172.21.15.61:0/4222507234 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3763731758 0 0) 0x7fcd500b22d0 con 0x7fcd500b1ef0 2022-07-20T03:15:56.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4d59a700 1 -- 172.21.15.61:0/4222507234 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcd30003260 con 0x7fcd500b1ef0 2022-07-20T03:15:56.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4d59a700 1 -- 172.21.15.61:0/4222507234 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1133219661 0 0) 0x7fcd30003260 con 0x7fcd500b1ef0 2022-07-20T03:15:56.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4ed9d700 1 -- 172.21.15.61:0/4222507234 >> v2:172.21.15.117:3300/0 conn(0x7fcd501427a0 msgr2=0x7fcd50061130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4ed9d700 1 --2- 172.21.15.61:0/4222507234 >> v2:172.21.15.117:3300/0 conn(0x7fcd501427a0 0x7fcd50061130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4ed9d700 1 -- 172.21.15.61:0/4222507234 >> v1:172.21.15.61:6789/0 conn(0x7fcd500b1ef0 legacy=0x7fcd50142090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.736+0000 7fcd4ed9d700 1 -- 172.21.15.61:0/4222507234 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd500ac0c0 con 0x7fcd50061620 2022-07-20T03:15:56.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4ed9d700 1 --2- 172.21.15.61:0/4222507234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd50061ab0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7fcd40001d00 tx=0x7fcd48006df0).ready entity=mon.0 client_cookie=37c9623b2d7b9439 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4d59a700 1 -- 172.21.15.61:0/4222507234 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd40001b60 con 0x7fcd50061620 2022-07-20T03:15:56.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4d59a700 1 -- 172.21.15.61:0/4222507234 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd48000e90 con 0x7fcd50061620 2022-07-20T03:15:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4d59a700 1 -- 172.21.15.61:0/4222507234 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd48012460 con 0x7fcd50061620 2022-07-20T03:15:56.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- 172.21.15.61:0/4222507234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 msgr2=0x7fcd50061ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/4222507234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd50061ab0 secure :-1 s=READY pgs=1122 cs=0 l=1 rev1=1 rx=0x7fcd40001d00 tx=0x7fcd48006df0).stop 2022-07-20T03:15:56.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- 172.21.15.61:0/4222507234 shutdown_connections 2022-07-20T03:15:56.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/4222507234 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd50061ab0 unknown :-1 s=CLOSED pgs=1122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/4222507234 >> v2:172.21.15.117:3300/0 conn(0x7fcd501427a0 0x7fcd50061130 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- 172.21.15.61:0/4222507234 >> 172.21.15.61:0/4222507234 conn(0x7fcd500b2ca0 msgr2=0x7fcd500b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:56.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- 172.21.15.61:0/4222507234 shutdown_connections 2022-07-20T03:15:56.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- 172.21.15.61:0/4222507234 wait complete. 2022-07-20T03:15:56.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 Processor -- start 2022-07-20T03:15:56.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- start start 2022-07-20T03:15:56.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd5009ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd500b1ef0 0x7fcd5013f060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd501427a0 0x7fcd5013f550 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.740+0000 7fcd4f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd50149020 con 0x7fcd50061620 2022-07-20T03:15:56.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd501492c0 con 0x7fcd501427a0 2022-07-20T03:15:56.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5009a2f0 con 0x7fcd500b1ef0 2022-07-20T03:15:56.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd5009ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd5009ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49374/0 (socket says 172.21.15.61:49374) 2022-07-20T03:15:56.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 -- 172.21.15.61:0/1889809285 learned_addr learned my addr 172.21.15.61:0/1889809285 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:56.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4dd9b700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd500b1ef0 0x7fcd5013f060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4ed9d700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd501427a0 0x7fcd5013f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 -- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd500b1ef0 msgr2=0x7fcd5013f060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd500b1ef0 0x7fcd5013f060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 -- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd501427a0 msgr2=0x7fcd5013f550 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd501427a0 0x7fcd5013f550 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 -- 172.21.15.61:0/1889809285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4800b180 con 0x7fcd50061620 2022-07-20T03:15:56.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4e59c700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd5009ba90 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7fcd40001dc0 tx=0x7fcd400017c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:56.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd40007320 con 0x7fcd50061620 2022-07-20T03:15:56.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcd40007480 con 0x7fcd50061620 2022-07-20T03:15:56.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd500b58f0 con 0x7fcd50061620 2022-07-20T03:15:56.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcd40005860 con 0x7fcd50061620 2022-07-20T03:15:56.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5013fa40 con 0x7fcd50061620 2022-07-20T03:15:56.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd500ad980 con 0x7fcd50061620 2022-07-20T03:15:56.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.744+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcd40010030 con 0x7fcd50061620 2022-07-20T03:15:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.748+0000 7fcd477fe700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd34053a40 0x7fcd34055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:56.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.748+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fcd4006bd10 con 0x7fcd50061620 2022-07-20T03:15:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.748+0000 7fcd4dd9b700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd34053a40 0x7fcd34055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:56.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.748+0000 7fcd4dd9b700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd34053a40 0x7fcd34055f00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fcd3800b000 tx=0x7fcd38006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:56.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.752+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcd500ad980 con 0x7fcd50061620 2022-07-20T03:15:56.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.876+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fcd5009a150 con 0x7fcd50061620 2022-07-20T03:15:56.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd477fe700 1 -- 172.21.15.61:0/1889809285 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v107) v1 ==== 74+0+13231 (secure 0 0 0) 0x7fcd5009a150 con 0x7fcd50061620 2022-07-20T03:15:56.885 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:56.885 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":107,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:56.196326+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":20,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":98,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:56.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd34053a40 msgr2=0x7fcd34055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd34053a40 0x7fcd34055f00 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x7fcd3800b000 tx=0x7fcd38006df0).stop 2022-07-20T03:15:56.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 msgr2=0x7fcd5009ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:56.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd5009ba90 secure :-1 s=READY pgs=1123 cs=0 l=1 rev1=1 rx=0x7fcd40001dc0 tx=0x7fcd400017c0).stop 2022-07-20T03:15:56.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 shutdown_connections 2022-07-20T03:15:56.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcd34053a40 0x7fcd34055f00 unknown :-1 s=CLOSED pgs=542 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcd501427a0 0x7fcd5013f550 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcd500b1ef0 0x7fcd5013f060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 --2- 172.21.15.61:0/1889809285 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcd50061620 0x7fcd5009ba90 unknown :-1 s=CLOSED pgs=1123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:56.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 >> 172.21.15.61:0/1889809285 conn(0x7fcd500b2ca0 msgr2=0x7fcd501465b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 shutdown_connections 2022-07-20T03:15:56.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:56.880+0000 7fcd4f59e700 1 -- 172.21.15.61:0/1889809285 wait complete. 2022-07-20T03:15:56.919 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:15:56.920 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:15:57.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5fd95700 1 Processor -- start 2022-07-20T03:15:57.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5fd95700 1 -- start start 2022-07-20T03:15:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5fd95700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5c580678f0 0x7f5c580b1bf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5fd95700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580b3670 con 0x7f5c5805cf40 2022-07-20T03:15:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5fd95700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5c580b3910 con 0x7f5c580b2e10 2022-07-20T03:15:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5fd95700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5c580b2310 con 0x7f5c580678f0 2022-07-20T03:15:57.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5c5805cf40 0x7f5c580b2700 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:45980/0 (socket says 172.21.15.61:45980) 2022-07-20T03:15:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 -- 172.21.15.61:0/582827562 learned_addr learned my addr 172.21.15.61:0/582827562 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5d330700 1 --2- 172.21.15.61:0/582827562 >> v2:172.21.15.117:3300/0 conn(0x7f5c580678f0 0x7f5c580b1bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3816062527 0 0) 0x7f5c580b3670 con 0x7f5c5805cf40 2022-07-20T03:15:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c3c003260 con 0x7f5c5805cf40 2022-07-20T03:15:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2029106596 0 0) 0x7f5c580b3910 con 0x7f5c580b2e10 2022-07-20T03:15:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5c580b3670 con 0x7f5c580b2e10 2022-07-20T03:15:57.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 935579844 0 0) 0x7f5c3c003260 con 0x7f5c5805cf40 2022-07-20T03:15:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 >> v2:172.21.15.117:3300/0 conn(0x7f5c580678f0 msgr2=0x7f5c580b1bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 --2- 172.21.15.61:0/582827562 >> v2:172.21.15.117:3300/0 conn(0x7f5c580678f0 0x7f5c580b1bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 >> v1:172.21.15.117:6789/0 conn(0x7f5c580b2e10 legacy=0x7f5c58062310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5805d320 con 0x7f5c5805cf40 2022-07-20T03:15:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f5c54001660 con 0x7f5c5805cf40 2022-07-20T03:15:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 >> v1:172.21.15.61:6789/0 conn(0x7f5c5805cf40 legacy=0x7f5c580b2700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c3c0038e0 0x7f5c3c003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c004500 0x7f5c3c004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c3c004e70 0x7f5c3c00b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c00bcf0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c00be30 con 0x7f5c3c004e70 2022-07-20T03:15:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c3c00bff0 con 0x7f5c3c004500 2022-07-20T03:15:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c3c0038e0 0x7f5c3c003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5e332700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c3c004e70 0x7f5c3c00b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5d330700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c004500 0x7f5c3c004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 -- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c004500 msgr2=0x7f5c3c004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:57.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c004500 0x7f5c3c004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 -- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c3c004e70 msgr2=0x7f5c3c00b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:57.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.032+0000 7f5c5db31700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c3c004e70 0x7f5c3c00b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5e332700 1 -- 172.21.15.61:0/582827562 reap_dead start 2022-07-20T03:15:57.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5db31700 1 -- 172.21.15.61:0/582827562 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c3c003260 con 0x7f5c3c0038e0 2022-07-20T03:15:57.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5db31700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c3c0038e0 0x7f5c3c003ce0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f5c54001430 tx=0x7f5c54001dd0).ready entity=mon.0 client_cookie=f87ed10b39a520db server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c540070d0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54007230 con 0x7f5c3c0038e0 2022-07-20T03:15:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5cb2f700 1 -- 172.21.15.61:0/582827562 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c5400c8c0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- 172.21.15.61:0/582827562 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c3c0038e0 msgr2=0x7f5c3c003ce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c3c0038e0 0x7f5c3c003ce0 secure :-1 s=READY pgs=1125 cs=0 l=1 rev1=1 rx=0x7f5c54001430 tx=0x7f5c54001dd0).stop 2022-07-20T03:15:57.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- 172.21.15.61:0/582827562 shutdown_connections 2022-07-20T03:15:57.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/582827562 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c3c0038e0 0x7f5c3c003ce0 unknown :-1 s=CLOSED pgs=1125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- 172.21.15.61:0/582827562 >> 172.21.15.61:0/582827562 conn(0x7f5c580b7d20 msgr2=0x7f5c580b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- 172.21.15.61:0/582827562 shutdown_connections 2022-07-20T03:15:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- 172.21.15.61:0/582827562 wait complete. 2022-07-20T03:15:57.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 Processor -- start 2022-07-20T03:15:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- start start 2022-07-20T03:15:57.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c0038e0 0x7f5c5813f810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c580ac550 0x7f5c5814e1d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 0x7f5c5809b310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5814d870 con 0x7f5c58141eb0 2022-07-20T03:15:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5814da90 con 0x7f5c580ac550 2022-07-20T03:15:57.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5fd95700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5c5814dd30 con 0x7f5c3c0038e0 2022-07-20T03:15:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5e332700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 0x7f5c5809b310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5db31700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c0038e0 0x7f5c5813f810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5e332700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 0x7f5c5809b310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49398/0 (socket says 172.21.15.61:49398) 2022-07-20T03:15:57.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5e332700 1 -- 172.21.15.61:0/2701416460 learned_addr learned my addr 172.21.15.61:0/2701416460 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.036+0000 7f5c5d330700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c580ac550 0x7f5c5814e1d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5db31700 1 -- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c580ac550 msgr2=0x7f5c5814e1d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5db31700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c580ac550 0x7f5c5814e1d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5db31700 1 -- 172.21.15.61:0/2701416460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 msgr2=0x7f5c5809b310 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5db31700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 0x7f5c5809b310 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5db31700 1 -- 172.21.15.61:0/2701416460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5813fd00 con 0x7f5c3c0038e0 2022-07-20T03:15:57.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5e332700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 0x7f5c5809b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:57.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5db31700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c0038e0 0x7f5c5813f810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f5c54001430 tx=0x7f5c54015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c540070d0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5c54000e90 con 0x7f5c3c0038e0 2022-07-20T03:15:57.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c581423c0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5c54004710 con 0x7f5c3c0038e0 2022-07-20T03:15:57.070 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c5809b800 con 0x7f5c3c0038e0 2022-07-20T03:15:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c5814def0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5c54010070 con 0x7f5c3c0038e0 2022-07-20T03:15:57.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c4b7fe700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c34053920 0x7f5c34055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f5c5406c350 con 0x7f5c3c0038e0 2022-07-20T03:15:57.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.040+0000 7f5c5d330700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c34053920 0x7f5c34055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.044+0000 7f5c5d330700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c34053920 0x7f5c34055de0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f5c4c00b000 tx=0x7f5c4c012040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.052+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5c54043450 con 0x7f5c3c0038e0 2022-07-20T03:15:57.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.180+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f5c580574e0 con 0x7f5c3c0038e0 2022-07-20T03:15:57.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.180+0000 7f5c4b7fe700 1 -- 172.21.15.61:0/2701416460 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v107) v1 ==== 110+0+10 (secure 0 0 0) 0x7f5c54024070 con 0x7f5c3c0038e0 2022-07-20T03:15:57.185 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:15:57.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c34053920 msgr2=0x7f5c34055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c34053920 0x7f5c34055de0 secure :-1 s=READY pgs=543 cs=0 l=1 rev1=1 rx=0x7f5c4c00b000 tx=0x7f5c4c012040).stop 2022-07-20T03:15:57.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c0038e0 msgr2=0x7f5c5813f810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c0038e0 0x7f5c5813f810 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f5c54001430 tx=0x7f5c54015000).stop 2022-07-20T03:15:57.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 shutdown_connections 2022-07-20T03:15:57.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5c34053920 0x7f5c34055de0 unknown :-1 s=CLOSED pgs=543 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5c580ac550 0x7f5c5814e1d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5c58141eb0 0x7f5c5809b310 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 --2- 172.21.15.61:0/2701416460 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5c3c0038e0 0x7f5c5813f810 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 >> 172.21.15.61:0/2701416460 conn(0x7f5c580b7d20 msgr2=0x7f5c58145280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 shutdown_connections 2022-07-20T03:15:57.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.184+0000 7f5c5fd95700 1 -- 172.21.15.61:0/2701416460 wait complete. 2022-07-20T03:15:57.205 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:15:57.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebea9b700 1 Processor -- start 2022-07-20T03:15:57.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebea9b700 1 -- start start 2022-07-20T03:15:57.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebea9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efeb809b6d0 0x7efeb809bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebea9b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb8147030 con 0x7efeb80b5960 2022-07-20T03:15:57.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebea9b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efeb805d4c0 con 0x7efeb805d0e0 2022-07-20T03:15:57.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebea9b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efeb81461e0 con 0x7efeb809b6d0 2022-07-20T03:15:57.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebd038700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efeb809b6d0 0x7efeb809bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efeb7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efeb80b5960 0x7efeb8142b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46002/0 (socket says 172.21.15.61:46002) 2022-07-20T03:15:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efeb7fff700 1 -- 172.21.15.61:0/3717835468 learned_addr learned my addr 172.21.15.61:0/3717835468 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:57.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efeb77fe700 1 -- 172.21.15.61:0/3717835468 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 849941221 0 0) 0x7efeb8147030 con 0x7efeb80b5960 2022-07-20T03:15:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebd038700 1 -- 172.21.15.61:0/3717835468 >> v1:172.21.15.117:6789/0 conn(0x7efeb805d0e0 legacy=0x7efeb8146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebd038700 1 -- 172.21.15.61:0/3717835468 >> v1:172.21.15.61:6789/0 conn(0x7efeb80b5960 legacy=0x7efeb8142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebd038700 1 -- 172.21.15.61:0/3717835468 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb8146480 con 0x7efeb809b6d0 2022-07-20T03:15:57.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efebd038700 1 --2- 172.21.15.61:0/3717835468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb809b6d0 0x7efeb809bb00 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7efeac001110 tx=0x7efeb00091d0).ready entity=mon.0 client_cookie=585208c90236c64b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efeb77fe700 1 -- 172.21.15.61:0/3717835468 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeb000b600 con 0x7efeb809b6d0 2022-07-20T03:15:57.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efeb77fe700 1 -- 172.21.15.61:0/3717835468 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb000b760 con 0x7efeb809b6d0 2022-07-20T03:15:57.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.312+0000 7efeb77fe700 1 -- 172.21.15.61:0/3717835468 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeb000b8d0 con 0x7efeb809b6d0 2022-07-20T03:15:57.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- 172.21.15.61:0/3717835468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb809b6d0 msgr2=0x7efeb809bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 --2- 172.21.15.61:0/3717835468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb809b6d0 0x7efeb809bb00 secure :-1 s=READY pgs=1127 cs=0 l=1 rev1=1 rx=0x7efeac001110 tx=0x7efeb00091d0).stop 2022-07-20T03:15:57.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- 172.21.15.61:0/3717835468 shutdown_connections 2022-07-20T03:15:57.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 --2- 172.21.15.61:0/3717835468 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb809b6d0 0x7efeb809bb00 unknown :-1 s=CLOSED pgs=1127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- 172.21.15.61:0/3717835468 >> 172.21.15.61:0/3717835468 conn(0x7efeb80b6a20 msgr2=0x7efeb8066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- 172.21.15.61:0/3717835468 shutdown_connections 2022-07-20T03:15:57.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- 172.21.15.61:0/3717835468 wait complete. 2022-07-20T03:15:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 Processor -- start 2022-07-20T03:15:57.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- start start 2022-07-20T03:15:57.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efeb805d0e0 0x7efeb80b4610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efeb809b6d0 0x7efeb8151dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 0x7efeb8140e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb80b04a0 con 0x7efeb80b5960 2022-07-20T03:15:57.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efeb8140740 con 0x7efeb805d0e0 2022-07-20T03:15:57.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebea9b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efeb809d220 con 0x7efeb809b6d0 2022-07-20T03:15:57.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebd038700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 0x7efeb8140e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebd038700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 0x7efeb8140e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49412/0 (socket says 172.21.15.61:49412) 2022-07-20T03:15:57.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebd038700 1 -- 172.21.15.61:0/3005287788 learned_addr learned my addr 172.21.15.61:0/3005287788 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:57.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efeb7fff700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efeb809b6d0 0x7efeb8151dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.316+0000 7efebc837700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efeb805d0e0 0x7efeb80b4610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebd038700 1 -- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efeb809b6d0 msgr2=0x7efeb8151dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebd038700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efeb809b6d0 0x7efeb8151dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebd038700 1 -- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efeb805d0e0 msgr2=0x7efeb80b4610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebd038700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efeb805d0e0 0x7efeb80b4610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebd038700 1 -- 172.21.15.61:0/3005287788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efeb0009470 con 0x7efeb80b5960 2022-07-20T03:15:57.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebc837700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efeb805d0e0 0x7efeb80b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:57.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebd038700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 0x7efeb8140e10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7efeb80b27a0 tx=0x7efeb0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeb000b600 con 0x7efeb80b5960 2022-07-20T03:15:57.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efeb00032d0 con 0x7efeb80b5960 2022-07-20T03:15:57.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efeb8140970 con 0x7efeb80b5960 2022-07-20T03:15:57.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efeb001f5e0 con 0x7efeb80b5960 2022-07-20T03:15:57.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efeb81522b0 con 0x7efeb80b5960 2022-07-20T03:15:57.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efeb8140b00 con 0x7efeb80b5960 2022-07-20T03:15:57.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.320+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efeb0026030 con 0x7efeb80b5960 2022-07-20T03:15:57.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.324+0000 7efeb5ffb700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe98053a40 0x7efe98055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.324+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efeb006c880 con 0x7efeb80b5960 2022-07-20T03:15:57.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.324+0000 7efebc837700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe98053a40 0x7efe98055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.324+0000 7efebc837700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe98053a40 0x7efe98055f00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7efeac00b000 tx=0x7efeac009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.328+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efeb0043ab0 con 0x7efeb80b5960 2022-07-20T03:15:57.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7efeb8143280 con 0x7efeb80b5960 2022-07-20T03:15:57.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efeb5ffb700 1 -- 172.21.15.61:0/3005287788 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v107) v1 ==== 92+0+10 (secure 0 0 0) 0x7efeb00150a0 con 0x7efeb80b5960 2022-07-20T03:15:57.461 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:15:57.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe98053a40 msgr2=0x7efe98055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe98053a40 0x7efe98055f00 secure :-1 s=READY pgs=544 cs=0 l=1 rev1=1 rx=0x7efeac00b000 tx=0x7efeac009040).stop 2022-07-20T03:15:57.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 msgr2=0x7efeb8140e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 0x7efeb8140e10 secure :-1 s=READY pgs=1128 cs=0 l=1 rev1=1 rx=0x7efeb80b27a0 tx=0x7efeb0016000).stop 2022-07-20T03:15:57.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 shutdown_connections 2022-07-20T03:15:57.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efe98053a40 0x7efe98055f00 unknown :-1 s=CLOSED pgs=544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efeb80b5960 0x7efeb8140e10 unknown :-1 s=CLOSED pgs=1128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efeb809b6d0 0x7efeb8151dc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 --2- 172.21.15.61:0/3005287788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efeb805d0e0 0x7efeb80b4610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 >> 172.21.15.61:0/3005287788 conn(0x7efeb80b6a20 msgr2=0x7efeb8066fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 shutdown_connections 2022-07-20T03:15:57.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.456+0000 7efebea9b700 1 -- 172.21.15.61:0/3005287788 wait complete. 2022-07-20T03:15:57.474 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:15:57.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.576+0000 7f751e8ba700 1 Processor -- start 2022-07-20T03:15:57.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.576+0000 7f751e8ba700 1 -- start start 2022-07-20T03:15:57.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f75180b31e0 0x7f751813df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f75180b1ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f75180b7f50 con 0x7f75180b7b70 2022-07-20T03:15:57.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f75180b2800 con 0x7f75180b31e0 2022-07-20T03:15:57.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75180b2aa0 con 0x7f7518066160 2022-07-20T03:15:57.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d8b8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f75180b7b70 0x7f751813d810 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46014/0 (socket says 172.21.15.61:46014) 2022-07-20T03:15:57.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d8b8700 1 -- 172.21.15.61:0/4073885355 learned_addr learned my addr 172.21.15.61:0/4073885355 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:57.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d0b7700 1 --2- 172.21.15.61:0/4073885355 >> v2:172.21.15.61:3300/0 conn(0x7f75180b31e0 0x7f751813df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e0b9700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f75180b1ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d0b7700 1 -- 172.21.15.61:0/4073885355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 msgr2=0x7f75180b1ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:57.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751c8b6700 1 -- 172.21.15.61:0/4073885355 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1796100287 0 0) 0x7f75180b7f50 con 0x7f75180b7b70 2022-07-20T03:15:57.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d0b7700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f75180b1ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d0b7700 1 -- 172.21.15.61:0/4073885355 >> v1:172.21.15.61:6789/0 conn(0x7f75180b7b70 legacy=0x7f751813d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d0b7700 1 -- 172.21.15.61:0/4073885355 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75180b2d40 con 0x7f75180b31e0 2022-07-20T03:15:57.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e0b9700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f75180b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:57.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751d0b7700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b31e0 0x7f751813df20 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f7508002f70 tx=0x7f7508006df0).ready entity=mon.0 client_cookie=2ed02a228dbc88ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751c8b6700 1 -- 172.21.15.61:0/4073885355 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f750800d570 con 0x7f75180b31e0 2022-07-20T03:15:57.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751c8b6700 1 -- 172.21.15.61:0/4073885355 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7508000e90 con 0x7f75180b31e0 2022-07-20T03:15:57.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751c8b6700 1 -- 172.21.15.61:0/4073885355 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f750800d8c0 con 0x7f75180b31e0 2022-07-20T03:15:57.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- 172.21.15.61:0/4073885355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b31e0 msgr2=0x7f751813df20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b31e0 0x7f751813df20 secure :-1 s=READY pgs=1130 cs=0 l=1 rev1=1 rx=0x7f7508002f70 tx=0x7f7508006df0).stop 2022-07-20T03:15:57.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- 172.21.15.61:0/4073885355 shutdown_connections 2022-07-20T03:15:57.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f75180b1ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 --2- 172.21.15.61:0/4073885355 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b31e0 0x7f751813df20 unknown :-1 s=CLOSED pgs=1130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- 172.21.15.61:0/4073885355 >> 172.21.15.61:0/4073885355 conn(0x7f75180656b0 msgr2=0x7f75180b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- 172.21.15.61:0/4073885355 shutdown_connections 2022-07-20T03:15:57.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.580+0000 7f751e8ba700 1 -- 172.21.15.61:0/4073885355 wait complete. 2022-07-20T03:15:57.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 Processor -- start 2022-07-20T03:15:57.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 -- start start 2022-07-20T03:15:57.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f7518110b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b31e0 0x7f7518111f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 0x7f7518113f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7518150f90 con 0x7f75180b7b70 2022-07-20T03:15:57.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f751813e780 con 0x7f75180b31e0 2022-07-20T03:15:57.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f75181129e0 con 0x7f7518066160 2022-07-20T03:15:57.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 0x7f7518113f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 0x7f7518113f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49424/0 (socket says 172.21.15.61:49424) 2022-07-20T03:15:57.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 -- 172.21.15.61:0/441436186 learned_addr learned my addr 172.21.15.61:0/441436186 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:57.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751d0b7700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b31e0 0x7f7518111f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751d8b8700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f7518110b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 -- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 msgr2=0x7f7518110b50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f7518110b50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 -- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b31e0 msgr2=0x7f7518111f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b31e0 0x7f7518111f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 -- 172.21.15.61:0/441436186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f750800b180 con 0x7f75180b7b70 2022-07-20T03:15:57.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751d0b7700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b31e0 0x7f7518111f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:15:57.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e0b9700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 0x7f7518113f00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f7510006cc0 tx=0x7f7510008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7510013620 con 0x7f75180b7b70 2022-07-20T03:15:57.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7510013780 con 0x7f75180b7b70 2022-07-20T03:15:57.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7518114a90 con 0x7f75180b7b70 2022-07-20T03:15:57.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7510013ae0 con 0x7f75180b7b70 2022-07-20T03:15:57.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.584+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75181105d0 con 0x7f75180b7b70 2022-07-20T03:15:57.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.588+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75180574e0 con 0x7f75180b7b70 2022-07-20T03:15:57.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.588+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7510013c40 con 0x7f75180b7b70 2022-07-20T03:15:57.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.588+0000 7f7506ffd700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74f8053a60 0x7f74f8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.588+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f751006bd60 con 0x7f75180b7b70 2022-07-20T03:15:57.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.588+0000 7f751d8b8700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74f8053a60 0x7f74f8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.588+0000 7f751d8b8700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74f8053a60 0x7f74f8055f20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f751400d000 tx=0x7f7514009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.596+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7510042f90 con 0x7f75180b7b70 2022-07-20T03:15:57.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f7518068630 con 0x7f75180b7b70 2022-07-20T03:15:57.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f7506ffd700 1 -- 172.21.15.61:0/441436186 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v107) v1 ==== 84+0+143 (secure 0 0 0) 0x7f75100465b0 con 0x7f75180b7b70 2022-07-20T03:15:57.729 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:57.729 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:15:57.730 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:57.730 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:15:57.730 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:15:57.730 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:15:57.731 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:15:57.731 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:15:57.731 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:15:57.731 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:15:57.732 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:15:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74f8053a60 msgr2=0x7f74f8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74f8053a60 0x7f74f8055f20 secure :-1 s=READY pgs=545 cs=0 l=1 rev1=1 rx=0x7f751400d000 tx=0x7f7514009040).stop 2022-07-20T03:15:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 msgr2=0x7f7518113f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 0x7f7518113f00 secure :-1 s=READY pgs=1131 cs=0 l=1 rev1=1 rx=0x7f7510006cc0 tx=0x7f7510008040).stop 2022-07-20T03:15:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 shutdown_connections 2022-07-20T03:15:57.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f74f8053a60 0x7f74f8055f20 unknown :-1 s=CLOSED pgs=545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f75180b31e0 0x7f7518111f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f75180b7b70 0x7f7518113f00 unknown :-1 s=CLOSED pgs=1131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 --2- 172.21.15.61:0/441436186 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7518066160 0x7f7518110b50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 >> 172.21.15.61:0/441436186 conn(0x7f75180656b0 msgr2=0x7f75180b7030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 shutdown_connections 2022-07-20T03:15:57.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.724+0000 7f751e8ba700 1 -- 172.21.15.61:0/441436186 wait complete. 2022-07-20T03:15:57.744 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:15:57.745 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:15:57.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 Processor -- start 2022-07-20T03:15:57.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 -- start start 2022-07-20T03:15:57.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380b59f0 0x7fb9381473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb9380b1400 0x7fb938147890 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb93805cc70 con 0x7fb9380af940 2022-07-20T03:15:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb9380b1070 con 0x7fb9380b1400 2022-07-20T03:15:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb937fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb9380b5dd0 con 0x7fb9380b59f0 2022-07-20T03:15:57.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9367fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb9380af940 0x7fb93814a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46032/0 (socket says 172.21.15.61:46032) 2022-07-20T03:15:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9367fc700 1 -- 172.21.15.61:0/451508975 learned_addr learned my addr 172.21.15.61:0/451508975 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:57.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9377fe700 1 --2- 172.21.15.61:0/451508975 >> v2:172.21.15.117:3300/0 conn(0x7fb9380b1400 0x7fb938147890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb936ffd700 1 --2- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380b59f0 0x7fb9381473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9377fe700 1 -- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380b59f0 msgr2=0x7fb9381473a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9377fe700 1 --2- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380b59f0 0x7fb9381473a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9377fe700 1 -- 172.21.15.61:0/451508975 >> v1:172.21.15.61:6789/0 conn(0x7fb9380af940 legacy=0x7fb93814a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb935ffb700 1 -- 172.21.15.61:0/451508975 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2156037618 0 0) 0x7fb93805cc70 con 0x7fb9380af940 2022-07-20T03:15:57.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9377fe700 1 -- 172.21.15.61:0/451508975 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb93805c760 con 0x7fb9380b1400 2022-07-20T03:15:57.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb9377fe700 1 --2- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b1400 0x7fb938147890 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fb9300032f0 tx=0x7fb9300091d0).ready entity=mon.1 client_cookie=ac09f0a0d104af33 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb935ffb700 1 -- 172.21.15.61:0/451508975 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb93000b6c0 con 0x7fb9380b1400 2022-07-20T03:15:57.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb935ffb700 1 -- 172.21.15.61:0/451508975 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb93000b820 con 0x7fb9380b1400 2022-07-20T03:15:57.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.856+0000 7fb935ffb700 1 -- 172.21.15.61:0/451508975 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb93000b990 con 0x7fb9380b1400 2022-07-20T03:15:57.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b1400 msgr2=0x7fb938147890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 --2- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b1400 0x7fb938147890 secure :-1 s=READY pgs=511 cs=0 l=1 rev1=1 rx=0x7fb9300032f0 tx=0x7fb9300091d0).stop 2022-07-20T03:15:57.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- 172.21.15.61:0/451508975 shutdown_connections 2022-07-20T03:15:57.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 --2- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b1400 0x7fb938147890 unknown :-1 s=CLOSED pgs=511 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 --2- 172.21.15.61:0/451508975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380b59f0 0x7fb9381473a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- 172.21.15.61:0/451508975 >> 172.21.15.61:0/451508975 conn(0x7fb938063480 msgr2=0x7fb9380b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:57.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- 172.21.15.61:0/451508975 shutdown_connections 2022-07-20T03:15:57.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- 172.21.15.61:0/451508975 wait complete. 2022-07-20T03:15:57.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 Processor -- start 2022-07-20T03:15:57.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- start start 2022-07-20T03:15:57.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380af940 0x7fb938111ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9380b1400 0x7fb938067530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 0x7fb93815a4a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938065b60 con 0x7fb9380b1400 2022-07-20T03:15:57.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb938159ac0 con 0x7fb9380b59f0 2022-07-20T03:15:57.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb937fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb938159d60 con 0x7fb9380af940 2022-07-20T03:15:57.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb9377fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 0x7fb93815a4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb9367fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9380b1400 0x7fb938067530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb9377fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 0x7fb93815a4a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59118/0 (socket says 172.21.15.61:59118) 2022-07-20T03:15:57.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb9377fe700 1 -- 172.21.15.61:0/1612325557 learned_addr learned my addr 172.21.15.61:0/1612325557 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:57.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.860+0000 7fb936ffd700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380af940 0x7fb938111ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb9377fe700 1 -- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380af940 msgr2=0x7fb938111ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb9377fe700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380af940 0x7fb938111ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb9377fe700 1 -- 172.21.15.61:0/1612325557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9380b1400 msgr2=0x7fb938067530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:57.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb9377fe700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9380b1400 0x7fb938067530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:57.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb9377fe700 1 -- 172.21.15.61:0/1612325557 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb930009470 con 0x7fb9380b59f0 2022-07-20T03:15:57.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb9377fe700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 0x7fb93815a4a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fb9300032f0 tx=0x7fb930016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb93000b990 con 0x7fb9380b59f0 2022-07-20T03:15:57.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb930021630 con 0x7fb9380b59f0 2022-07-20T03:15:57.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb93815a000 con 0x7fb9380b59f0 2022-07-20T03:15:57.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb93000b6c0 con 0x7fb9380b59f0 2022-07-20T03:15:57.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb938064190 con 0x7fb9380b59f0 2022-07-20T03:15:57.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb93815a1c0 con 0x7fb9380b59f0 2022-07-20T03:15:57.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb930027020 con 0x7fb9380b59f0 2022-07-20T03:15:57.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb927fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb914053a60 0x7fb914055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:57.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(107..107 src has 1..107) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb93006d620 con 0x7fb9380b59f0 2022-07-20T03:15:57.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.864+0000 7fb936ffd700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb914053a60 0x7fb914055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:57.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.868+0000 7fb936ffd700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb914053a60 0x7fb914055f20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fb92800b000 tx=0x7fb928006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:57.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.872+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb930044960 con 0x7fb9380b59f0 2022-07-20T03:15:58.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:57.996+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb93805c9d0 con 0x7fb9380b59f0 2022-07-20T03:15:58.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.240+0000 7fb927fff700 1 -- 172.21.15.61:0/1612325557 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v108) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb930015020 con 0x7fb9380b59f0 2022-07-20T03:15:58.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb914053a60 msgr2=0x7fb914055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb914053a60 0x7fb914055f20 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7fb92800b000 tx=0x7fb928006df0).stop 2022-07-20T03:15:58.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 msgr2=0x7fb93815a4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 0x7fb93815a4a0 secure :-1 s=READY pgs=512 cs=0 l=1 rev1=1 rx=0x7fb9300032f0 tx=0x7fb930016000).stop 2022-07-20T03:15:58.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 shutdown_connections 2022-07-20T03:15:58.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb914053a60 0x7fb914055f20 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb9380b59f0 0x7fb93815a4a0 unknown :-1 s=CLOSED pgs=512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb9380b1400 0x7fb938067530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 --2- 172.21.15.61:0/1612325557 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb9380af940 0x7fb938111ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 >> 172.21.15.61:0/1612325557 conn(0x7fb938063480 msgr2=0x7fb9380b2230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:58.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 shutdown_connections 2022-07-20T03:15:58.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.244+0000 7fb937fff700 1 -- 172.21.15.61:0/1612325557 wait complete. 2022-07-20T03:15:58.253 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:15:58.264 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:15:58.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.364+0000 7f010b57d700 1 Processor -- start 2022-07-20T03:15:58.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.364+0000 7f010b57d700 1 -- start start 2022-07-20T03:15:58.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f01040b2210 0x7f0104140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:58.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f010413fe80 con 0x7f01040ac680 2022-07-20T03:15:58.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0104140120 con 0x7f01040967e0 2022-07-20T03:15:58.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f01041403c0 con 0x7f01040b2210 2022-07-20T03:15:58.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f01040b2210 0x7f0104140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:58.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f01040b2210 0x7f0104140860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49444/0 (socket says 172.21.15.61:49444) 2022-07-20T03:15:58.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 -- 172.21.15.61:0/2896305290 learned_addr learned my addr 172.21.15.61:0/2896305290 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:58.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 -- 172.21.15.61:0/2896305290 >> v1:172.21.15.117:6789/0 conn(0x7f01040967e0 legacy=0x7f01040ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 -- 172.21.15.61:0/2896305290 >> v1:172.21.15.61:6789/0 conn(0x7f01040ac680 legacy=0x7f010413efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 -- 172.21.15.61:0/2896305290 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f01040b25f0 con 0x7f01040b2210 2022-07-20T03:15:58.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f0109319700 1 --2- 172.21.15.61:0/2896305290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040b2210 0x7f0104140860 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f00f4002f70 tx=0x7f00f4006eb0).ready entity=mon.0 client_cookie=266187710080a455 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:58.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f00f3fff700 1 -- 172.21.15.61:0/2896305290 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00f400d660 con 0x7f01040b2210 2022-07-20T03:15:58.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f00f3fff700 1 -- 172.21.15.61:0/2896305290 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f00f4000e90 con 0x7f01040b2210 2022-07-20T03:15:58.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f00f3fff700 1 -- 172.21.15.61:0/2896305290 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f00f400d9f0 con 0x7f01040b2210 2022-07-20T03:15:58.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- 172.21.15.61:0/2896305290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040b2210 msgr2=0x7f0104140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 --2- 172.21.15.61:0/2896305290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040b2210 0x7f0104140860 secure :-1 s=READY pgs=1134 cs=0 l=1 rev1=1 rx=0x7f00f4002f70 tx=0x7f00f4006eb0).stop 2022-07-20T03:15:58.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- 172.21.15.61:0/2896305290 shutdown_connections 2022-07-20T03:15:58.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 --2- 172.21.15.61:0/2896305290 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040b2210 0x7f0104140860 unknown :-1 s=CLOSED pgs=1134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- 172.21.15.61:0/2896305290 >> 172.21.15.61:0/2896305290 conn(0x7f01040b36c0 msgr2=0x7f01040b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:58.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- 172.21.15.61:0/2896305290 shutdown_connections 2022-07-20T03:15:58.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.368+0000 7f010b57d700 1 -- 172.21.15.61:0/2896305290 wait complete. 2022-07-20T03:15:58.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 Processor -- start 2022-07-20T03:15:58.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 -- start start 2022-07-20T03:15:58.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f01040967e0 0x7f0104063d80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:58.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 0x7f0104136f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:58.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01040b2210 0x7f0104137420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:58.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0104142ab0 con 0x7f01040ac680 2022-07-20T03:15:58.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f01040affb0 con 0x7f01040b2210 2022-07-20T03:15:58.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f01040645d0 con 0x7f01040967e0 2022-07-20T03:15:58.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 0x7f0104136f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:58.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 0x7f0104136f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49452/0 (socket says 172.21.15.61:49452) 2022-07-20T03:15:58.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 -- 172.21.15.61:0/2822692257 learned_addr learned my addr 172.21.15.61:0/2822692257 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:58.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0109b1a700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01040b2210 0x7f0104137420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:58.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0109319700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f01040967e0 0x7f0104063d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:58.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 -- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f01040967e0 msgr2=0x7f0104063d80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f01040967e0 0x7f0104063d80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 -- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01040b2210 msgr2=0x7f0104137420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:58.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01040b2210 0x7f0104137420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:58.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 -- 172.21.15.61:0/2822692257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00f400b1e0 con 0x7f01040ac680 2022-07-20T03:15:58.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f0108b18700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 0x7f0104136f30 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f01040632d0 tx=0x7f0100007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:58.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0100007030 con 0x7f01040ac680 2022-07-20T03:15:58.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0100007190 con 0x7f01040ac680 2022-07-20T03:15:58.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f01040b01a0 con 0x7f01040ac680 2022-07-20T03:15:58.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0100006510 con 0x7f01040ac680 2022-07-20T03:15:58.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.372+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f01040af0d0 con 0x7f01040ac680 2022-07-20T03:15:58.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.376+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0104137ab0 con 0x7f01040ac680 2022-07-20T03:15:58.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.376+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0100010030 con 0x7f01040ac680 2022-07-20T03:15:58.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.376+0000 7f00f27fc700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00e4053920 0x7f00e4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:58.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.376+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(108..108 src has 1..108) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f010006b7c0 con 0x7f01040ac680 2022-07-20T03:15:58.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.376+0000 7f0109319700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00e4053920 0x7f00e4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:58.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.376+0000 7f0109319700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00e4053920 0x7f00e4055de0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f00f401c000 tx=0x7f00f4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:58.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.384+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f01000428f0 con 0x7f01040ac680 2022-07-20T03:15:58.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:58.508+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f0104064410 con 0x7f01040ac680 2022-07-20T03:15:59.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.248+0000 7f00f27fc700 1 -- 172.21.15.61:0/2822692257 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v109) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0100045f10 con 0x7f01040ac680 2022-07-20T03:15:59.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00e4053920 msgr2=0x7f00e4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00e4053920 0x7f00e4055de0 secure :-1 s=READY pgs=547 cs=0 l=1 rev1=1 rx=0x7f00f401c000 tx=0x7f00f4015000).stop 2022-07-20T03:15:59.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 msgr2=0x7f0104136f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 0x7f0104136f30 secure :-1 s=READY pgs=1135 cs=0 l=1 rev1=1 rx=0x7f01040632d0 tx=0x7f0100007be0).stop 2022-07-20T03:15:59.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 shutdown_connections 2022-07-20T03:15:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f00e4053920 0x7f00e4055de0 unknown :-1 s=CLOSED pgs=547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f01040b2210 0x7f0104137420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f01040ac680 0x7f0104136f30 unknown :-1 s=CLOSED pgs=1135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 --2- 172.21.15.61:0/2822692257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f01040967e0 0x7f0104063d80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 >> 172.21.15.61:0/2822692257 conn(0x7f01040b36c0 msgr2=0x7f010409b170 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:59.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 shutdown_connections 2022-07-20T03:15:59.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.256+0000 7f010b57d700 1 -- 172.21.15.61:0/2822692257 wait complete. 2022-07-20T03:15:59.268 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:15:59.273 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:15:59.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 Processor -- start 2022-07-20T03:15:59.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 -- start start 2022-07-20T03:15:59.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1e4003fe90 0x7f1e400d45f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1e4003d8a0 0x7f1e4003f550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1e40041a10 con 0x7f1e4003f170 2022-07-20T03:15:59.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e4003e3a0 con 0x7f1e4003d8a0 2022-07-20T03:15:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4eb04700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1e40042800 con 0x7f1e4003fe90 2022-07-20T03:15:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4e303700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1e4003d8a0 0x7f1e4003f550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4e303700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1e4003d8a0 0x7f1e4003f550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49462/0 (socket says 172.21.15.61:49462) 2022-07-20T03:15:59.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4e303700 1 -- 172.21.15.61:0/590430728 learned_addr learned my addr 172.21.15.61:0/590430728 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:59.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.380+0000 7f1e4d301700 1 --2- 172.21.15.61:0/590430728 >> v2:172.21.15.117:3300/0 conn(0x7f1e4003fe90 0x7f1e400d45f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4e303700 1 -- 172.21.15.61:0/590430728 >> v2:172.21.15.117:3300/0 conn(0x7f1e4003fe90 msgr2=0x7f1e400d45f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:59.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4e303700 1 --2- 172.21.15.61:0/590430728 >> v2:172.21.15.117:3300/0 conn(0x7f1e4003fe90 0x7f1e400d45f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4cb00700 1 -- 172.21.15.61:0/590430728 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 307636740 0 0) 0x7f1e40041a10 con 0x7f1e4003f170 2022-07-20T03:15:59.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4e303700 1 -- 172.21.15.61:0/590430728 >> v1:172.21.15.117:6789/0 conn(0x7f1e4003f170 legacy=0x7f1e400d3ee0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4e303700 1 -- 172.21.15.61:0/590430728 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e40042aa0 con 0x7f1e4003d8a0 2022-07-20T03:15:59.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4e303700 1 --2- 172.21.15.61:0/590430728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003d8a0 0x7f1e4003f550 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7f1e48044dc0 tx=0x7f1e4805f750).ready entity=mon.0 client_cookie=a85e063ec8628375 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4cb00700 1 -- 172.21.15.61:0/590430728 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e48084030 con 0x7f1e4003d8a0 2022-07-20T03:15:59.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4cb00700 1 -- 172.21.15.61:0/590430728 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e4806b200 con 0x7f1e4003d8a0 2022-07-20T03:15:59.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4cb00700 1 -- 172.21.15.61:0/590430728 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e4806b370 con 0x7f1e4003d8a0 2022-07-20T03:15:59.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 -- 172.21.15.61:0/590430728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003d8a0 msgr2=0x7f1e4003f550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/590430728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003d8a0 0x7f1e4003f550 secure :-1 s=READY pgs=1136 cs=0 l=1 rev1=1 rx=0x7f1e48044dc0 tx=0x7f1e4805f750).stop 2022-07-20T03:15:59.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 -- 172.21.15.61:0/590430728 shutdown_connections 2022-07-20T03:15:59.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/590430728 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003d8a0 0x7f1e4003f550 unknown :-1 s=CLOSED pgs=1136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/590430728 >> v2:172.21.15.117:3300/0 conn(0x7f1e4003fe90 0x7f1e400d45f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 -- 172.21.15.61:0/590430728 >> 172.21.15.61:0/590430728 conn(0x7f1e40046330 msgr2=0x7f1e40046710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:59.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 -- 172.21.15.61:0/590430728 shutdown_connections 2022-07-20T03:15:59.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 -- 172.21.15.61:0/590430728 wait complete. 2022-07-20T03:15:59.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 Processor -- start 2022-07-20T03:15:59.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.384+0000 7f1e4eb04700 1 -- start start 2022-07-20T03:15:59.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e4003d8a0 0x7f1e400d7f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e4003f170 0x7f1e400d8cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 0x7f1e4003b740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e400d7890 con 0x7f1e4003fe90 2022-07-20T03:15:59.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e400d9970 con 0x7f1e4003d8a0 2022-07-20T03:15:59.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1e400cff30 con 0x7f1e4003f170 2022-07-20T03:15:59.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 0x7f1e4003b740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 0x7f1e4003b740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49468/0 (socket says 172.21.15.61:49468) 2022-07-20T03:15:59.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 -- 172.21.15.61:0/3247303527 learned_addr learned my addr 172.21.15.61:0/3247303527 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:59.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4db02700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e4003d8a0 0x7f1e400d7f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4d301700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e4003f170 0x7f1e400d8cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 -- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e4003f170 msgr2=0x7f1e400d8cc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e4003f170 0x7f1e400d8cc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 -- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e4003d8a0 msgr2=0x7f1e400d7f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e4003d8a0 0x7f1e400d7f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 -- 172.21.15.61:0/3247303527 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e400d7ac0 con 0x7f1e4003fe90 2022-07-20T03:15:59.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4d301700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e4003f170 0x7f1e400d8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:59.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4db02700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e4003d8a0 0x7f1e400d7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:15:59.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4e303700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 0x7f1e4003b740 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f1e48044dc0 tx=0x7f1e48066000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e48084070 con 0x7f1e4003fe90 2022-07-20T03:15:59.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1e48026300 con 0x7f1e4003fe90 2022-07-20T03:15:59.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e4003a0a0 con 0x7f1e4003fe90 2022-07-20T03:15:59.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1e4805d3d0 con 0x7f1e4003fe90 2022-07-20T03:15:59.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.388+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e4003c720 con 0x7f1e4003fe90 2022-07-20T03:15:59.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.392+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e400d7c50 con 0x7f1e4003fe90 2022-07-20T03:15:59.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.392+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1e480683d0 con 0x7f1e4003fe90 2022-07-20T03:15:59.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.392+0000 7f1e36ffd700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e28053ad0 0x7f1e28055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.392+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(109..109 src has 1..109) v4 ==== 5957+0+0 (secure 0 0 0) 0x7f1e48085070 con 0x7f1e4003fe90 2022-07-20T03:15:59.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.392+0000 7f1e4db02700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e28053ad0 0x7f1e28055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.392+0000 7f1e4db02700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e28053ad0 0x7f1e28055f90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f1e4400d000 tx=0x7f1e44009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.396+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1e480b2cd0 con 0x7f1e4003fe90 2022-07-20T03:15:59.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.560+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f1e4003fa40 con 0x7f1e4003fe90 2022-07-20T03:15:59.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.572+0000 7f1e36ffd700 1 -- 172.21.15.61:0/3247303527 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 21 and data pool 22 v60) v1 ==== 189+0+0 (secure 0 0 0) 0x7f1e480b62f0 con 0x7f1e4003fe90 2022-07-20T03:15:59.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e28053ad0 msgr2=0x7f1e28055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e28053ad0 0x7f1e28055f90 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f1e4400d000 tx=0x7f1e44009040).stop 2022-07-20T03:15:59.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 msgr2=0x7f1e4003b740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 0x7f1e4003b740 secure :-1 s=READY pgs=1137 cs=0 l=1 rev1=1 rx=0x7f1e48044dc0 tx=0x7f1e48066000).stop 2022-07-20T03:15:59.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 shutdown_connections 2022-07-20T03:15:59.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1e28053ad0 0x7f1e28055f90 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1e4003fe90 0x7f1e4003b740 unknown :-1 s=CLOSED pgs=1137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1e4003f170 0x7f1e400d8cc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 --2- 172.21.15.61:0/3247303527 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1e4003d8a0 0x7f1e400d7f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 >> 172.21.15.61:0/3247303527 conn(0x7f1e40046330 msgr2=0x7f1e400d4b60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:59.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 shutdown_connections 2022-07-20T03:15:59.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.576+0000 7f1e4eb04700 1 -- 172.21.15.61:0/3247303527 wait complete. 2022-07-20T03:15:59.584 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 21 and data pool 22 2022-07-20T03:15:59.596 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:59.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd720b700 1 Processor -- start 2022-07-20T03:15:59.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd720b700 1 -- start start 2022-07-20T03:15:59.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd720b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7dc80426b0 0x7f7dc802a110 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd720b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc80290d0 con 0x7f7dc8028cf0 2022-07-20T03:15:59.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd720b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7dc8043040 con 0x7f7dc8041f60 2022-07-20T03:15:59.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd720b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7dc80432e0 con 0x7f7dc80426b0 2022-07-20T03:15:59.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd6209700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7dc8028cf0 0x7f7dc80d5390 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46070/0 (socket says 172.21.15.61:46070) 2022-07-20T03:15:59.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd6209700 1 -- 172.21.15.61:0/2250383851 learned_addr learned my addr 172.21.15.61:0/2250383851 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:15:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd5a08700 1 --2- 172.21.15.61:0/2250383851 >> v2:172.21.15.61:3300/0 conn(0x7f7dc80426b0 0x7f7dc802a110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd5a08700 1 -- 172.21.15.61:0/2250383851 >> v1:172.21.15.117:6789/0 conn(0x7f7dc8041f60 legacy=0x7f7dc8025bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd5a08700 1 -- 172.21.15.61:0/2250383851 >> v1:172.21.15.61:6789/0 conn(0x7f7dc8028cf0 legacy=0x7f7dc80d5390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd5207700 1 -- 172.21.15.61:0/2250383851 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2915886244 0 0) 0x7f7dc80290d0 con 0x7f7dc8028cf0 2022-07-20T03:15:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.692+0000 7f7dd5a08700 1 -- 172.21.15.61:0/2250383851 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc8029c70 con 0x7f7dc80426b0 2022-07-20T03:15:59.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd5a08700 1 --2- 172.21.15.61:0/2250383851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc80426b0 0x7f7dc802a110 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f7dc0002f70 tx=0x7f7dc0006df0).ready entity=mon.0 client_cookie=2d4f293c9afc92c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd5207700 1 -- 172.21.15.61:0/2250383851 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dc000d610 con 0x7f7dc80426b0 2022-07-20T03:15:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd5207700 1 -- 172.21.15.61:0/2250383851 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc0000e90 con 0x7f7dc80426b0 2022-07-20T03:15:59.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd5207700 1 -- 172.21.15.61:0/2250383851 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dc000d960 con 0x7f7dc80426b0 2022-07-20T03:15:59.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 -- 172.21.15.61:0/2250383851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc80426b0 msgr2=0x7f7dc802a110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 --2- 172.21.15.61:0/2250383851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc80426b0 0x7f7dc802a110 secure :-1 s=READY pgs=1139 cs=0 l=1 rev1=1 rx=0x7f7dc0002f70 tx=0x7f7dc0006df0).stop 2022-07-20T03:15:59.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 -- 172.21.15.61:0/2250383851 shutdown_connections 2022-07-20T03:15:59.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 --2- 172.21.15.61:0/2250383851 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc80426b0 0x7f7dc802a110 unknown :-1 s=CLOSED pgs=1139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 -- 172.21.15.61:0/2250383851 >> 172.21.15.61:0/2250383851 conn(0x7f7dc8046bf0 msgr2=0x7f7dc8046fd0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:59.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 -- 172.21.15.61:0/2250383851 shutdown_connections 2022-07-20T03:15:59.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 -- 172.21.15.61:0/2250383851 wait complete. 2022-07-20T03:15:59.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 Processor -- start 2022-07-20T03:15:59.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.696+0000 7f7dd720b700 1 -- start start 2022-07-20T03:15:59.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7dc8028cf0 0x7f7dc80d4750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 0x7f7dc80d8100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7dc80e66d0 0x7f7dc80279f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc801dbe0 con 0x7f7dc8041f60 2022-07-20T03:15:59.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc801de10 con 0x7f7dc8028cf0 2022-07-20T03:15:59.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7dc80402f0 con 0x7f7dc80e66d0 2022-07-20T03:15:59.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 0x7f7dc80d8100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd6209700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7dc8028cf0 0x7f7dc80d4750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 0x7f7dc80d8100 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49480/0 (socket says 172.21.15.61:49480) 2022-07-20T03:15:59.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 -- 172.21.15.61:0/3190508096 learned_addr learned my addr 172.21.15.61:0/3190508096 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:59.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd6a0a700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7dc80e66d0 0x7f7dc80279f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 -- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7dc80e66d0 msgr2=0x7f7dc80279f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7dc80e66d0 0x7f7dc80279f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 -- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7dc8028cf0 msgr2=0x7f7dc80d4750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7dc8028cf0 0x7f7dc80d4750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 -- 172.21.15.61:0/3190508096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc000b180 con 0x7f7dc8041f60 2022-07-20T03:15:59.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd5a08700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 0x7f7dc80d8100 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f7dc0002f70 tx=0x7f7dc0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dc00119a0 con 0x7f7dc8041f60 2022-07-20T03:15:59.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7dc0011b00 con 0x7f7dc8041f60 2022-07-20T03:15:59.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dc80d85f0 con 0x7f7dc8041f60 2022-07-20T03:15:59.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7dc0011e60 con 0x7f7dc8041f60 2022-07-20T03:15:59.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dc80d7860 con 0x7f7dc8041f60 2022-07-20T03:15:59.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dc801dfd0 con 0x7f7dc8041f60 2022-07-20T03:15:59.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.700+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7dc0025030 con 0x7f7dc8041f60 2022-07-20T03:15:59.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.704+0000 7f7dbf7fe700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7db0053a60 0x7f7db0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.704+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7dc006d1e0 con 0x7f7dc8041f60 2022-07-20T03:15:59.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.704+0000 7f7dd6209700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7db0053a60 0x7f7db0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.704+0000 7f7dd6209700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7db0053a60 0x7f7db0055f20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f7dcc00d000 tx=0x7f7dcc007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.708+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7dc0044030 con 0x7f7dc8041f60 2022-07-20T03:15:59.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.836+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f7dc8005270 con 0x7f7dc8041f60 2022-07-20T03:15:59.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.836+0000 7f7dbf7fe700 1 -- 172.21.15.61:0/3190508096 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16112 (secure 0 0 0) 0x7f7dc0023090 con 0x7f7dc8041f60 2022-07-20T03:15:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:15:59.843 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":110,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:59.567265+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:58.052517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:58.563360+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:15:59.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7db0053a60 msgr2=0x7f7db0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7db0053a60 0x7f7db0055f20 secure :-1 s=READY pgs=549 cs=0 l=1 rev1=1 rx=0x7f7dcc00d000 tx=0x7f7dcc007a00).stop 2022-07-20T03:15:59.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 msgr2=0x7f7dc80d8100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 0x7f7dc80d8100 secure :-1 s=READY pgs=1140 cs=0 l=1 rev1=1 rx=0x7f7dc0002f70 tx=0x7f7dc0015000).stop 2022-07-20T03:15:59.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 shutdown_connections 2022-07-20T03:15:59.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7db0053a60 0x7f7db0055f20 unknown :-1 s=CLOSED pgs=549 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7dc80e66d0 0x7f7dc80279f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7dc8041f60 0x7f7dc80d8100 unknown :-1 s=CLOSED pgs=1140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 --2- 172.21.15.61:0/3190508096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7dc8028cf0 0x7f7dc80d4750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 >> 172.21.15.61:0/3190508096 conn(0x7f7dc8046bf0 msgr2=0x7f7dc802c260 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:15:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 shutdown_connections 2022-07-20T03:15:59.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.840+0000 7f7dd720b700 1 -- 172.21.15.61:0/3190508096 wait complete. 2022-07-20T03:15:59.857 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:15:59.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 Processor -- start 2022-07-20T03:15:59.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 -- start start 2022-07-20T03:15:59.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5d5c0b0570 0x7f5d5c0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c0aea10 0x7f5d5c141dd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:15:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5d5c142750 con 0x7f5d5c1422c0 2022-07-20T03:15:59.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5d5c143b10 con 0x7f5d5c0b0570 2022-07-20T03:15:59.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d60d6c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1433d0 con 0x7f5d5c0aea10 2022-07-20T03:15:59.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5d5c0b0570 0x7f5d5c0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5d5c0b0570 0x7f5d5c0b0950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49486/0 (socket says 172.21.15.61:49486) 2022-07-20T03:15:59.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 -- 172.21.15.61:0/818174801 learned_addr learned my addr 172.21.15.61:0/818174801 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:15:59.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d53fff700 1 --2- 172.21.15.61:0/818174801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c0aea10 0x7f5d5c141dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:15:59.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 -- 172.21.15.61:0/818174801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c0aea10 msgr2=0x7f5d5c141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:15:59.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5affd700 1 -- 172.21.15.61:0/818174801 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2051468949 0 0) 0x7f5d5c142750 con 0x7f5d5c1422c0 2022-07-20T03:15:59.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 --2- 172.21.15.61:0/818174801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c0aea10 0x7f5d5c141dd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:15:59.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 -- 172.21.15.61:0/818174801 >> v1:172.21.15.117:6789/0 conn(0x7f5d5c1422c0 legacy=0x7f5d5c0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 -- 172.21.15.61:0/818174801 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d5c143670 con 0x7f5d5c0b0570 2022-07-20T03:15:59.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5b7fe700 1 --2- 172.21.15.61:0/818174801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c0b0570 0x7f5d5c0b0950 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f5d4c002f70 tx=0x7f5d4c006eb0).ready entity=mon.0 client_cookie=50b5d5a060e23b48 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:15:59.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5affd700 1 -- 172.21.15.61:0/818174801 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d4c00d570 con 0x7f5d5c0b0570 2022-07-20T03:15:59.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.960+0000 7f5d5affd700 1 -- 172.21.15.61:0/818174801 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d4c000e90 con 0x7f5d5c0b0570 2022-07-20T03:15:59.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d5affd700 1 -- 172.21.15.61:0/818174801 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d4c00d8c0 con 0x7f5d5c0b0570 2022-07-20T03:15:59.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- 172.21.15.61:0/818174801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c0b0570 msgr2=0x7f5d5c0b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:15:59.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/818174801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c0b0570 0x7f5d5c0b0950 secure :-1 s=READY pgs=1141 cs=0 l=1 rev1=1 rx=0x7f5d4c002f70 tx=0x7f5d4c006eb0).stop 2022-07-20T03:15:59.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- 172.21.15.61:0/818174801 shutdown_connections 2022-07-20T03:15:59.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/818174801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c0aea10 0x7f5d5c141dd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/818174801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c0b0570 0x7f5d5c0b0950 unknown :-1 s=CLOSED pgs=1141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- 172.21.15.61:0/818174801 >> 172.21.15.61:0/818174801 conn(0x7f5d5c0b9360 msgr2=0x7f5d5c0b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- 172.21.15.61:0/818174801 shutdown_connections 2022-07-20T03:16:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- 172.21.15.61:0/818174801 wait complete. 2022-07-20T03:16:00.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 Processor -- start 2022-07-20T03:16:00.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- start start 2022-07-20T03:16:00.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d5c0aea10 0x7f5d5c0b1850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c1422c0 0x7f5d5c13e530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 0x7f5d5c13ea20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c145e60 con 0x7f5d5c1467c0 2022-07-20T03:16:00.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c0b4ee0 con 0x7f5d5c0aea10 2022-07-20T03:16:00.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d60d6c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5d5c1459c0 con 0x7f5d5c1422c0 2022-07-20T03:16:00.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 0x7f5d5c13ea20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d53fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c1422c0 0x7f5d5c13e530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d5bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 0x7f5d5c13ea20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49496/0 (socket says 172.21.15.61:49496) 2022-07-20T03:16:00.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d5bfff700 1 -- 172.21.15.61:0/2885005952 learned_addr learned my addr 172.21.15.61:0/2885005952 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:00.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.964+0000 7f5d5b7fe700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d5c0aea10 0x7f5d5c0b1850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5bfff700 1 -- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c1422c0 msgr2=0x7f5d5c13e530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5bfff700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c1422c0 0x7f5d5c13e530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5bfff700 1 -- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d5c0aea10 msgr2=0x7f5d5c0b1850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5bfff700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d5c0aea10 0x7f5d5c0b1850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5bfff700 1 -- 172.21.15.61:0/2885005952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d4c00b1e0 con 0x7f5d5c1467c0 2022-07-20T03:16:00.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5bfff700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 0x7f5d5c13ea20 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f5d54001250 tx=0x7f5d5400c7f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:00.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d54006aa0 con 0x7f5d5c1467c0 2022-07-20T03:16:00.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5d54006c00 con 0x7f5d5c1467c0 2022-07-20T03:16:00.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d5c13f4c0 con 0x7f5d5c1467c0 2022-07-20T03:16:00.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5d54007030 con 0x7f5d5c1467c0 2022-07-20T03:16:00.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d5c0b1e70 con 0x7f5d5c1467c0 2022-07-20T03:16:00.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5d5c0b4d30 con 0x7f5d5c1467c0 2022-07-20T03:16:00.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5d54010030 con 0x7f5d5c1467c0 2022-07-20T03:16:00.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d597fa700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d400539d0 0x7f5d40055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5d5406bdb0 con 0x7f5d5c1467c0 2022-07-20T03:16:00.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.968+0000 7f5d5b7fe700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d400539d0 0x7f5d40055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.972+0000 7f5d5b7fe700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d400539d0 0x7f5d40055e90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f5d4c01c000 tx=0x7f5d4c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:00.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:15:59.976+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5d54042d10 con 0x7f5d5c1467c0 2022-07-20T03:16:00.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.104+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f5d5c13f0b0 con 0x7f5d5c1467c0 2022-07-20T03:16:00.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.104+0000 7f5d597fa700 1 -- 172.21.15.61:0/2885005952 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v110) v1 ==== 74+0+16112 (secure 0 0 0) 0x7f5d54007190 con 0x7f5d5c1467c0 2022-07-20T03:16:00.109 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:00.109 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":110,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:15:59.567265+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:58.052517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:58.563360+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.104+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d400539d0 msgr2=0x7f5d40055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.104+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d400539d0 0x7f5d40055e90 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f5d4c01c000 tx=0x7f5d4c015000).stop 2022-07-20T03:16:00.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.104+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 msgr2=0x7f5d5c13ea20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.104+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 0x7f5d5c13ea20 secure :-1 s=READY pgs=1142 cs=0 l=1 rev1=1 rx=0x7f5d54001250 tx=0x7f5d5400c7f0).stop 2022-07-20T03:16:00.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 shutdown_connections 2022-07-20T03:16:00.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5d400539d0 0x7f5d40055e90 unknown :-1 s=CLOSED pgs=550 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5d5c1467c0 0x7f5d5c13ea20 unknown :-1 s=CLOSED pgs=1142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5d5c1422c0 0x7f5d5c13e530 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 --2- 172.21.15.61:0/2885005952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5d5c0aea10 0x7f5d5c0b1850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 >> 172.21.15.61:0/2885005952 conn(0x7f5d5c0b9360 msgr2=0x7f5d5c140450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:00.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 shutdown_connections 2022-07-20T03:16:00.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.108+0000 7f5d60d6c700 1 -- 172.21.15.61:0/2885005952 wait complete. 2022-07-20T03:16:00.122 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:16:00.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.220+0000 7f36d9807700 1 Processor -- start 2022-07-20T03:16:00.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.220+0000 7f36d9807700 1 -- start start 2022-07-20T03:16:00.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f36d40b3ed0 0x7f36d4143530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f36d4143a20 0x7f36d413fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f36d40b6680 con 0x7f36d40b62a0 2022-07-20T03:16:00.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d40b4af0 con 0x7f36d4143a20 2022-07-20T03:16:00.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f36d41465c0 con 0x7f36d40b3ed0 2022-07-20T03:16:00.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f36d4143a20 0x7f36d413fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f36d4143a20 0x7f36d413fcc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49504/0 (socket says 172.21.15.61:49504) 2022-07-20T03:16:00.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 -- 172.21.15.61:0/179865878 learned_addr learned my addr 172.21.15.61:0/179865878 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:00.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d27fc700 1 --2- 172.21.15.61:0/179865878 >> v2:172.21.15.117:3300/0 conn(0x7f36d40b3ed0 0x7f36d4143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 -- 172.21.15.61:0/179865878 >> v2:172.21.15.117:3300/0 conn(0x7f36d40b3ed0 msgr2=0x7f36d4143530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:00.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 --2- 172.21.15.61:0/179865878 >> v2:172.21.15.117:3300/0 conn(0x7f36d40b3ed0 0x7f36d4143530 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 -- 172.21.15.61:0/179865878 >> v1:172.21.15.117:6789/0 conn(0x7f36d40b62a0 legacy=0x7f36d40b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 -- 172.21.15.61:0/179865878 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36d4142df0 con 0x7f36d4143a20 2022-07-20T03:16:00.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d37fe700 1 --2- 172.21.15.61:0/179865878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413fcc0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f36cc0032f0 tx=0x7f36cc009290).ready entity=mon.0 client_cookie=14fee22fff6c4910 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:00.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d1ffb700 1 -- 172.21.15.61:0/179865878 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36cc00b870 con 0x7f36d4143a20 2022-07-20T03:16:00.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d1ffb700 1 -- 172.21.15.61:0/179865878 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc00b9d0 con 0x7f36d4143a20 2022-07-20T03:16:00.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d1ffb700 1 -- 172.21.15.61:0/179865878 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36cc00bb40 con 0x7f36d4143a20 2022-07-20T03:16:00.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- 172.21.15.61:0/179865878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 msgr2=0x7f36d413fcc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 --2- 172.21.15.61:0/179865878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413fcc0 secure :-1 s=READY pgs=1143 cs=0 l=1 rev1=1 rx=0x7f36cc0032f0 tx=0x7f36cc009290).stop 2022-07-20T03:16:00.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- 172.21.15.61:0/179865878 shutdown_connections 2022-07-20T03:16:00.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 --2- 172.21.15.61:0/179865878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413fcc0 unknown :-1 s=CLOSED pgs=1143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 --2- 172.21.15.61:0/179865878 >> v2:172.21.15.117:3300/0 conn(0x7f36d40b3ed0 0x7f36d4143530 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- 172.21.15.61:0/179865878 >> 172.21.15.61:0/179865878 conn(0x7f36d4066f00 msgr2=0x7f36d40672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:00.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- 172.21.15.61:0/179865878 shutdown_connections 2022-07-20T03:16:00.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.224+0000 7f36d9807700 1 -- 172.21.15.61:0/179865878 wait complete. 2022-07-20T03:16:00.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 Processor -- start 2022-07-20T03:16:00.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 -- start start 2022-07-20T03:16:00.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36d40b3ed0 0x7f36d4147840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36d40b62a0 0x7f36d4147d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413cbf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4146a80 con 0x7f36d4143a20 2022-07-20T03:16:00.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f36d41482e0 con 0x7f36d40b62a0 2022-07-20T03:16:00.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d9807700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f36d4152900 con 0x7f36d40b3ed0 2022-07-20T03:16:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413cbf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d27fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36d40b62a0 0x7f36d4147d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d2ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36d40b3ed0 0x7f36d4147840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413cbf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49510/0 (socket says 172.21.15.61:49510) 2022-07-20T03:16:00.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 -- 172.21.15.61:0/4123288714 learned_addr learned my addr 172.21.15.61:0/4123288714 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:00.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 -- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36d40b3ed0 msgr2=0x7f36d4147840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36d40b3ed0 0x7f36d4147840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 -- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36d40b62a0 msgr2=0x7f36d4147d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:00.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36d40b62a0 0x7f36d4147d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:00.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.228+0000 7f36d37fe700 1 -- 172.21.15.61:0/4123288714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f36cc009530 con 0x7f36d4143a20 2022-07-20T03:16:00.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36d37fe700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413cbf0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f36cc0032f0 tx=0x7f36cc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:00.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36cc00b870 con 0x7f36d4143a20 2022-07-20T03:16:00.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f36cc021440 con 0x7f36d4143a20 2022-07-20T03:16:00.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f36d4152ba0 con 0x7f36d4143a20 2022-07-20T03:16:00.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f36cc0217a0 con 0x7f36d4143a20 2022-07-20T03:16:00.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f36d41403c0 con 0x7f36d4143a20 2022-07-20T03:16:00.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f36d4152d30 con 0x7f36d4143a20 2022-07-20T03:16:00.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f36cc027020 con 0x7f36d4143a20 2022-07-20T03:16:00.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36cbfff700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f36b80539b0 0x7f36b8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:00.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(110..110 src has 1..110) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f36cc06d080 con 0x7f36d4143a20 2022-07-20T03:16:00.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.232+0000 7f36d2ffd700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f36b80539b0 0x7f36b8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:00.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.236+0000 7f36d2ffd700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f36b80539b0 0x7f36b8055e70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f36c400d000 tx=0x7f36c4007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:00.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.240+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f36cc044090 con 0x7f36d4143a20 2022-07-20T03:16:00.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:00.400+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f36d4143090 con 0x7f36d4143a20 2022-07-20T03:16:01.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.260+0000 7f36cbfff700 1 -- 172.21.15.61:0/4123288714 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v62) v1 ==== 118+0+0 (secure 0 0 0) 0x7f36cc015020 con 0x7f36d4143a20 2022-07-20T03:16:01.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f36b80539b0 msgr2=0x7f36b8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f36b80539b0 0x7f36b8055e70 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f36c400d000 tx=0x7f36c4007b90).stop 2022-07-20T03:16:01.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 msgr2=0x7f36d413cbf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413cbf0 secure :-1 s=READY pgs=1144 cs=0 l=1 rev1=1 rx=0x7f36cc0032f0 tx=0x7f36cc016000).stop 2022-07-20T03:16:01.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 shutdown_connections 2022-07-20T03:16:01.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f36b80539b0 0x7f36b8055e70 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f36d40b62a0 0x7f36d4147d30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f36d4143a20 0x7f36d413cbf0 unknown :-1 s=CLOSED pgs=1144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 --2- 172.21.15.61:0/4123288714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f36d40b3ed0 0x7f36d4147840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 >> 172.21.15.61:0/4123288714 conn(0x7f36d4066f00 msgr2=0x7f36d405c580 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:01.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 shutdown_connections 2022-07-20T03:16:01.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.264+0000 7f36d9807700 1 -- 172.21.15.61:0/4123288714 wait complete. 2022-07-20T03:16:01.286 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa30d2b9700 1 Processor -- start 2022-07-20T03:16:01.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa30d2b9700 1 -- start start 2022-07-20T03:16:01.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa30d2b9700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa3080afa10 0x7fa3080b46d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa30d2b9700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa3080b42a0 con 0x7fa3080608b0 2022-07-20T03:16:01.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa30d2b9700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa308060c90 con 0x7fa3080b3de0 2022-07-20T03:16:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa30d2b9700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa3080af410 con 0x7fa3080afa10 2022-07-20T03:16:01.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa306ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa3080608b0 0x7fa30805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46112/0 (socket says 172.21.15.61:46112) 2022-07-20T03:16:01.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa306ffd700 1 -- 172.21.15.61:0/1936868962 learned_addr learned my addr 172.21.15.61:0/1936868962 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:01.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa3067fc700 1 --2- 172.21.15.61:0/1936868962 >> v2:172.21.15.117:3300/0 conn(0x7fa3080afa10 0x7fa3080b46d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa305ffb700 1 -- 172.21.15.61:0/1936868962 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3653585412 0 0) 0x7fa3080b42a0 con 0x7fa3080608b0 2022-07-20T03:16:01.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa305ffb700 1 -- 172.21.15.61:0/1936868962 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa2ec003260 con 0x7fa3080608b0 2022-07-20T03:16:01.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa305ffb700 1 -- 172.21.15.61:0/1936868962 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2512197985 0 0) 0x7fa308060c90 con 0x7fa3080b3de0 2022-07-20T03:16:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa3067fc700 1 -- 172.21.15.61:0/1936868962 >> v1:172.21.15.117:6789/0 conn(0x7fa3080b3de0 legacy=0x7fa3081452d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa3067fc700 1 -- 172.21.15.61:0/1936868962 >> v1:172.21.15.61:6789/0 conn(0x7fa3080608b0 legacy=0x7fa30805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa3067fc700 1 -- 172.21.15.61:0/1936868962 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3080af6b0 con 0x7fa3080afa10 2022-07-20T03:16:01.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa3067fc700 1 --2- 172.21.15.61:0/1936868962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080afa10 0x7fa3080b46d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fa2f80032f0 tx=0x7fa2f80091d0).ready entity=mon.1 client_cookie=5df94684b925d870 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:01.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa305ffb700 1 -- 172.21.15.61:0/1936868962 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f800b6c0 con 0x7fa3080afa10 2022-07-20T03:16:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa305ffb700 1 -- 172.21.15.61:0/1936868962 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f800b820 con 0x7fa3080afa10 2022-07-20T03:16:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.392+0000 7fa305ffb700 1 -- 172.21.15.61:0/1936868962 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f800b990 con 0x7fa3080afa10 2022-07-20T03:16:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1936868962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080afa10 msgr2=0x7fa3080b46d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1936868962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080afa10 0x7fa3080b46d0 secure :-1 s=READY pgs=519 cs=0 l=1 rev1=1 rx=0x7fa2f80032f0 tx=0x7fa2f80091d0).stop 2022-07-20T03:16:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1936868962 shutdown_connections 2022-07-20T03:16:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1936868962 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080afa10 0x7fa3080b46d0 unknown :-1 s=CLOSED pgs=519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1936868962 >> 172.21.15.61:0/1936868962 conn(0x7fa3080b5ee0 msgr2=0x7fa3080b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1936868962 shutdown_connections 2022-07-20T03:16:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1936868962 wait complete. 2022-07-20T03:16:01.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 Processor -- start 2022-07-20T03:16:01.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- start start 2022-07-20T03:16:01.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080608b0 0x7fa3080903f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 0x7fa308094020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3080b3de0 0x7fa308094510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3081561e0 con 0x7fa3080afa10 2022-07-20T03:16:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa308155d80 con 0x7fa3080608b0 2022-07-20T03:16:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa30d2b9700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa30809bb50 con 0x7fa3080b3de0 2022-07-20T03:16:01.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa3067fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 0x7fa308094020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa3067fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 0x7fa308094020 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49522/0 (socket says 172.21.15.61:49522) 2022-07-20T03:16:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa3067fc700 1 -- 172.21.15.61:0/1128537766 learned_addr learned my addr 172.21.15.61:0/1128537766 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.396+0000 7fa3077fe700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3080b3de0 0x7fa308094510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa306ffd700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080608b0 0x7fa3080903f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa3067fc700 1 -- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3080b3de0 msgr2=0x7fa308094510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa3067fc700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3080b3de0 0x7fa308094510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa3067fc700 1 -- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080608b0 msgr2=0x7fa3080903f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa3067fc700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080608b0 0x7fa3080903f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa3067fc700 1 -- 172.21.15.61:0/1128537766 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2f8009470 con 0x7fa3080afa10 2022-07-20T03:16:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa3067fc700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 0x7fa308094020 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fa2f80032f0 tx=0x7fa2f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f800b6c0 con 0x7fa3080afa10 2022-07-20T03:16:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa2f801f3f0 con 0x7fa3080afa10 2022-07-20T03:16:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa30809b610 con 0x7fa3080afa10 2022-07-20T03:16:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2f801f750 con 0x7fa3080afa10 2022-07-20T03:16:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa308094a00 con 0x7fa3080afa10 2022-07-20T03:16:01.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3080574e0 con 0x7fa3080afa10 2022-07-20T03:16:01.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa2f800f070 con 0x7fa3080afa10 2022-07-20T03:16:01.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa2fffff700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e00539b0 0x7fa2e0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa2f806ca30 con 0x7fa3080afa10 2022-07-20T03:16:01.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.400+0000 7fa306ffd700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e00539b0 0x7fa2e0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.404+0000 7fa306ffd700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e00539b0 0x7fa2e0055e70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fa2f000d000 tx=0x7fa2f0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:01.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.408+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa2f8043990 con 0x7fa3080afa10 2022-07-20T03:16:01.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.568+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa308069a40 con 0x7fa3080afa10 2022-07-20T03:16:01.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.568+0000 7fa2fffff700 1 -- 172.21.15.61:0/1128537766 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7fa2f8015020 con 0x7fa3080afa10 2022-07-20T03:16:01.577 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:01.577 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7662,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3960371515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3960371515},{"type":"v1","addr":"172.21.15.61:6833","nonce":3960371515}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:59.567631+0000","modified":"2022-07-20T03:16:01.250517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"b","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/2884785802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2884785802},{"type":"v1","addr":"172.21.15.117:6835","nonce":2884785802}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T03:16:01.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e00539b0 msgr2=0x7fa2e0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e00539b0 0x7fa2e0055e70 secure :-1 s=READY pgs=552 cs=0 l=1 rev1=1 rx=0x7fa2f000d000 tx=0x7fa2f0009040).stop 2022-07-20T03:16:01.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 msgr2=0x7fa308094020 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 0x7fa308094020 secure :-1 s=READY pgs=1146 cs=0 l=1 rev1=1 rx=0x7fa2f80032f0 tx=0x7fa2f8016000).stop 2022-07-20T03:16:01.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 shutdown_connections 2022-07-20T03:16:01.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa2e00539b0 0x7fa2e0055e70 unknown :-1 s=CLOSED pgs=552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa3080b3de0 0x7fa308094510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa3080afa10 0x7fa308094020 unknown :-1 s=CLOSED pgs=1146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 --2- 172.21.15.61:0/1128537766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa3080608b0 0x7fa3080903f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 >> 172.21.15.61:0/1128537766 conn(0x7fa3080b5ee0 msgr2=0x7fa3080b8740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:01.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 shutdown_connections 2022-07-20T03:16:01.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.572+0000 7fa30d2b9700 1 -- 172.21.15.61:0/1128537766 wait complete. 2022-07-20T03:16:01.588 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-20T03:16:01.590 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:01.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 Processor -- start 2022-07-20T03:16:01.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 -- start start 2022-07-20T03:16:01.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f920005c3f0 0x7f920014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f92000acef0 0x7f9200146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f920005d480 con 0x7f92000b1e00 2022-07-20T03:16:01.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f92000b21e0 con 0x7f92000acef0 2022-07-20T03:16:01.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f920005cf40 con 0x7f920005c3f0 2022-07-20T03:16:01.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91feffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f92000b1e00 0x7f92000ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46126/0 (socket says 172.21.15.61:46126) 2022-07-20T03:16:01.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91feffd700 1 -- 172.21.15.61:0/2883551659 learned_addr learned my addr 172.21.15.61:0/2883551659 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:01.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91ff7fe700 1 --2- 172.21.15.61:0/2883551659 >> v2:172.21.15.117:3300/0 conn(0x7f92000acef0 0x7f9200146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fe7fc700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f920005c3f0 0x7f920014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fdffb700 1 -- 172.21.15.61:0/2883551659 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3754817855 0 0) 0x7f920005d480 con 0x7f92000b1e00 2022-07-20T03:16:01.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fdffb700 1 -- 172.21.15.61:0/2883551659 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f91e0003260 con 0x7f92000b1e00 2022-07-20T03:16:01.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91ff7fe700 1 -- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f920005c3f0 msgr2=0x7f920014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:01.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91ff7fe700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f920005c3f0 0x7f920014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91ff7fe700 1 -- 172.21.15.61:0/2883551659 >> v1:172.21.15.61:6789/0 conn(0x7f92000b1e00 legacy=0x7f92000ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91ff7fe700 1 -- 172.21.15.61:0/2883551659 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f920005d1e0 con 0x7f92000acef0 2022-07-20T03:16:01.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fe7fc700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f920005c3f0 0x7f920014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:01.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91ff7fe700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146410 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f91f80032f0 tx=0x7f91f80091d0).ready entity=mon.1 client_cookie=c77a8de6600d1396 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:01.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fdffb700 1 -- 172.21.15.61:0/2883551659 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f800b600 con 0x7f92000acef0 2022-07-20T03:16:01.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fdffb700 1 -- 172.21.15.61:0/2883551659 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f800b760 con 0x7f92000acef0 2022-07-20T03:16:01.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f91fdffb700 1 -- 172.21.15.61:0/2883551659 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f800b8d0 con 0x7f92000acef0 2022-07-20T03:16:01.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.692+0000 7f920584b700 1 -- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 msgr2=0x7f9200146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146410 secure :-1 s=READY pgs=520 cs=0 l=1 rev1=1 rx=0x7f91f80032f0 tx=0x7f91f80091d0).stop 2022-07-20T03:16:01.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- 172.21.15.61:0/2883551659 shutdown_connections 2022-07-20T03:16:01.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146410 unknown :-1 s=CLOSED pgs=520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 --2- 172.21.15.61:0/2883551659 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f920005c3f0 0x7f920014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- 172.21.15.61:0/2883551659 >> 172.21.15.61:0/2883551659 conn(0x7f92000611c0 msgr2=0x7f9200062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:01.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- 172.21.15.61:0/2883551659 shutdown_connections 2022-07-20T03:16:01.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- 172.21.15.61:0/2883551659 wait complete. 2022-07-20T03:16:01.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 Processor -- start 2022-07-20T03:16:01.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- start start 2022-07-20T03:16:01.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92000b1e00 0x7f9200066cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 0x7f92000671c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9200149710 con 0x7f92001432a0 2022-07-20T03:16:01.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f920014a650 con 0x7f92000acef0 2022-07-20T03:16:01.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9200148190 con 0x7f92000b1e00 2022-07-20T03:16:01.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 0x7f92000671c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 0x7f92000671c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49536/0 (socket says 172.21.15.61:49536) 2022-07-20T03:16:01.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 -- 172.21.15.61:0/3980375853 learned_addr learned my addr 172.21.15.61:0/3980375853 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:01.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91feffd700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91fe7fc700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92000b1e00 0x7f9200066cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 -- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92000b1e00 msgr2=0x7f9200066cd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92000b1e00 0x7f9200066cd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 -- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 msgr2=0x7f9200146980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 -- 172.21.15.61:0/3980375853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91f000b180 con 0x7f92001432a0 2022-07-20T03:16:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91ff7fe700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 0x7f92000671c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f91f80032f0 tx=0x7f91f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f800b8d0 con 0x7f92001432a0 2022-07-20T03:16:01.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f91f801f5e0 con 0x7f92001432a0 2022-07-20T03:16:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f91f8009470 con 0x7f92001432a0 2022-07-20T03:16:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.696+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f91f800b600 con 0x7f92001432a0 2022-07-20T03:16:01.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.700+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9200147560 con 0x7f92001432a0 2022-07-20T03:16:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.700+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92000b04b0 con 0x7f92001432a0 2022-07-20T03:16:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.700+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f91f8026030 con 0x7f92001432a0 2022-07-20T03:16:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.700+0000 7f91f7fff700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f91e4053a40 0x7f91e4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:01.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.704+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f91f806cc30 con 0x7f92001432a0 2022-07-20T03:16:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.704+0000 7f91feffd700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f91e4053a40 0x7f91e4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:01.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.704+0000 7f91feffd700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f91e4053a40 0x7f91e4055f00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f91e800b000 tx=0x7f91e8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:01.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.708+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f91f8043b90 con 0x7f92001432a0 2022-07-20T03:16:01.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.832+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f92000574e0 con 0x7f92001432a0 2022-07-20T03:16:01.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f91f7fff700 1 -- 172.21.15.61:0/3980375853 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f91f8015070 con 0x7f92001432a0 2022-07-20T03:16:01.840 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:01.840 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":111,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:00.570268+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:58.052517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:58.563360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:01.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f91e4053a40 msgr2=0x7f91e4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f91e4053a40 0x7f91e4055f00 secure :-1 s=READY pgs=553 cs=0 l=1 rev1=1 rx=0x7f91e800b000 tx=0x7f91e8009040).stop 2022-07-20T03:16:01.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 msgr2=0x7f92000671c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:01.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 0x7f92000671c0 secure :-1 s=READY pgs=1148 cs=0 l=1 rev1=1 rx=0x7f91f80032f0 tx=0x7f91f8016000).stop 2022-07-20T03:16:01.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 shutdown_connections 2022-07-20T03:16:01.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f91e4053a40 0x7f91e4055f00 unknown :-1 s=CLOSED pgs=553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f92001432a0 0x7f92000671c0 unknown :-1 s=CLOSED pgs=1148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f92000b1e00 0x7f9200066cd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 --2- 172.21.15.61:0/3980375853 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f92000acef0 0x7f9200146980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:01.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 >> 172.21.15.61:0/3980375853 conn(0x7f920005c3f0 msgr2=0x7f9200069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:01.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 shutdown_connections 2022-07-20T03:16:01.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.836+0000 7f920584b700 1 -- 172.21.15.61:0/3980375853 wait complete. 2022-07-20T03:16:01.855 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:16:02.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 Processor -- start 2022-07-20T03:16:02.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 -- start start 2022-07-20T03:16:02.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f84940b3e60 0x7f849405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f849405d280 0x7f849405c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8494060c20 0x7f84940610b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8494067c40 con 0x7f849405d280 2022-07-20T03:16:02.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84940b32b0 con 0x7f84940b3e60 2022-07-20T03:16:02.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8498874700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940b4240 con 0x7f8494060c20 2022-07-20T03:16:02.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f849405d280 0x7f849405c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f849405d280 0x7f849405c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49544/0 (socket says 172.21.15.61:49544) 2022-07-20T03:16:02.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 -- 172.21.15.61:0/317476687 learned_addr learned my addr 172.21.15.61:0/317476687 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:02.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f8493fff700 1 --2- 172.21.15.61:0/317476687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8494060c20 0x7f84940610b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f84937fe700 1 --2- 172.21.15.61:0/317476687 >> v2:172.21.15.117:3300/0 conn(0x7f84940b3e60 0x7f849405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 -- 172.21.15.61:0/317476687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8494060c20 msgr2=0x7f84940610b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:02.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 --2- 172.21.15.61:0/317476687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8494060c20 0x7f84940610b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 -- 172.21.15.61:0/317476687 >> v2:172.21.15.117:3300/0 conn(0x7f84940b3e60 msgr2=0x7f849405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:02.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 --2- 172.21.15.61:0/317476687 >> v2:172.21.15.117:3300/0 conn(0x7f84940b3e60 0x7f849405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 -- 172.21.15.61:0/317476687 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84940b34a0 con 0x7f849405d280 2022-07-20T03:16:02.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f84937fe700 1 --2- 172.21.15.61:0/317476687 >> v2:172.21.15.117:3300/0 conn(0x7f84940b3e60 0x7f849405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:02.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.968+0000 7f848bfff700 1 --2- 172.21.15.61:0/317476687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f849405c280 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f8494066620 tx=0x7f847c006eb0).ready entity=mon.0 client_cookie=a26f021042277c6e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8492ffd700 1 -- 172.21.15.61:0/317476687 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f847c00d570 con 0x7f849405d280 2022-07-20T03:16:02.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8492ffd700 1 -- 172.21.15.61:0/317476687 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f847c000e90 con 0x7f849405d280 2022-07-20T03:16:02.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8492ffd700 1 -- 172.21.15.61:0/317476687 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f847c00d900 con 0x7f849405d280 2022-07-20T03:16:02.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 -- 172.21.15.61:0/317476687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 msgr2=0x7f849405c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 --2- 172.21.15.61:0/317476687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f849405c280 secure :-1 s=READY pgs=1149 cs=0 l=1 rev1=1 rx=0x7f8494066620 tx=0x7f847c006eb0).stop 2022-07-20T03:16:02.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 -- 172.21.15.61:0/317476687 shutdown_connections 2022-07-20T03:16:02.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 --2- 172.21.15.61:0/317476687 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8494060c20 0x7f84940610b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 --2- 172.21.15.61:0/317476687 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f849405c280 unknown :-1 s=CLOSED pgs=1149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 --2- 172.21.15.61:0/317476687 >> v2:172.21.15.117:3300/0 conn(0x7f84940b3e60 0x7f849405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 -- 172.21.15.61:0/317476687 >> 172.21.15.61:0/317476687 conn(0x7f8494064c00 msgr2=0x7f8494064fe0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 -- 172.21.15.61:0/317476687 shutdown_connections 2022-07-20T03:16:02.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 -- 172.21.15.61:0/317476687 wait complete. 2022-07-20T03:16:02.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 Processor -- start 2022-07-20T03:16:02.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.972+0000 7f8498874700 1 -- start start 2022-07-20T03:16:02.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f84940b7a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8494060c20 0x7f8494143690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84940b3e60 0x7f8494143b80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8494149f60 con 0x7f849405d280 2022-07-20T03:16:02.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84940ad960 con 0x7f8494060c20 2022-07-20T03:16:02.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84940adc00 con 0x7f84940b3e60 2022-07-20T03:16:02.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f84940b7a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f84940b7a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49548/0 (socket says 172.21.15.61:49548) 2022-07-20T03:16:02.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 -- 172.21.15.61:0/1271353501 learned_addr learned my addr 172.21.15.61:0/1271353501 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:02.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f848bfff700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8494060c20 0x7f8494143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8493fff700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84940b3e60 0x7f8494143b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 -- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84940b3e60 msgr2=0x7f8494143b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84940b3e60 0x7f8494143b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 -- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8494060c20 msgr2=0x7f8494143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8494060c20 0x7f8494143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 -- 172.21.15.61:0/1271353501 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f847c00b1e0 con 0x7f849405d280 2022-07-20T03:16:02.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84937fe700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f84940b7a10 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f8484006940 tx=0x7f8484008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8484011780 con 0x7f849405d280 2022-07-20T03:16:02.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84840118e0 con 0x7f849405d280 2022-07-20T03:16:02.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84940adea0 con 0x7f849405d280 2022-07-20T03:16:02.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8484011c40 con 0x7f849405d280 2022-07-20T03:16:02.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84940ad3e0 con 0x7f849405d280 2022-07-20T03:16:02.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84940ad770 con 0x7f849405d280 2022-07-20T03:16:02.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.976+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8484012420 con 0x7f849405d280 2022-07-20T03:16:02.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.980+0000 7f84917fa700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8478053a40 0x7f8478055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.980+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f848406bb50 con 0x7f849405d280 2022-07-20T03:16:02.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.980+0000 7f848bfff700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8478053a40 0x7f8478055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.980+0000 7f848bfff700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8478053a40 0x7f8478055f00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f847c01b000 tx=0x7f847c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:01.984+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8484042a50 con 0x7f849405d280 2022-07-20T03:16:02.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.148+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f84940574e0 con 0x7f849405d280 2022-07-20T03:16:02.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.156+0000 7f84917fa700 1 -- 172.21.15.61:0/1271353501 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v38) v1 ==== 184+0+0 (secure 0 0 0) 0x7f8484046070 con 0x7f849405d280 2022-07-20T03:16:02.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8478053a40 msgr2=0x7f8478055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8478053a40 0x7f8478055f00 secure :-1 s=READY pgs=554 cs=0 l=1 rev1=1 rx=0x7f847c01b000 tx=0x7f847c015000).stop 2022-07-20T03:16:02.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 msgr2=0x7f84940b7a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f84940b7a10 secure :-1 s=READY pgs=1150 cs=0 l=1 rev1=1 rx=0x7f8484006940 tx=0x7f8484008040).stop 2022-07-20T03:16:02.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 shutdown_connections 2022-07-20T03:16:02.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8478053a40 0x7f8478055f00 unknown :-1 s=CLOSED pgs=554 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8494060c20 0x7f8494143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84940b3e60 0x7f8494143b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 --2- 172.21.15.61:0/1271353501 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f849405d280 0x7f84940b7a10 unknown :-1 s=CLOSED pgs=1150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 >> 172.21.15.61:0/1271353501 conn(0x7f8494064c00 msgr2=0x7f84940b67a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 shutdown_connections 2022-07-20T03:16:02.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.160+0000 7f8498874700 1 -- 172.21.15.61:0/1271353501 wait complete. 2022-07-20T03:16:02.167 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:16:02.177 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:16:02.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b350b5700 1 Processor -- start 2022-07-20T03:16:02.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b350b5700 1 -- start start 2022-07-20T03:16:02.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b350b5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7b300af2c0 0x7f7b300af6f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b350b5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b30147120 con 0x7f7b300b5830 2022-07-20T03:16:02.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b350b5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7b300b0130 con 0x7f7b3005cf40 2022-07-20T03:16:02.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b350b5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7b301462d0 con 0x7f7b300af2c0 2022-07-20T03:16:02.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b2f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7b300af2c0 0x7f7b300af6f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b2e59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7b300b5830 0x7f7b30142c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46156/0 (socket says 172.21.15.61:46156) 2022-07-20T03:16:02.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.284+0000 7f7b2e59c700 1 -- 172.21.15.61:0/3884330834 learned_addr learned my addr 172.21.15.61:0/3884330834 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:02.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2f59e700 1 -- 172.21.15.61:0/3884330834 >> v1:172.21.15.117:6789/0 conn(0x7f7b3005cf40 legacy=0x7f7b30146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2f59e700 1 -- 172.21.15.61:0/3884330834 >> v1:172.21.15.61:6789/0 conn(0x7f7b300b5830 legacy=0x7f7b30142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2f59e700 1 -- 172.21.15.61:0/3884330834 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b30146570 con 0x7f7b300af2c0 2022-07-20T03:16:02.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2f59e700 1 --2- 172.21.15.61:0/3884330834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b300af2c0 0x7f7b300af6f0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f7b28002f70 tx=0x7f7b28006df0).ready entity=mon.0 client_cookie=1edecf56708193d2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2dd9b700 1 -- 172.21.15.61:0/3884330834 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b2800d610 con 0x7f7b300af2c0 2022-07-20T03:16:02.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2dd9b700 1 -- 172.21.15.61:0/3884330834 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b28000e90 con 0x7f7b300af2c0 2022-07-20T03:16:02.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b2dd9b700 1 -- 172.21.15.61:0/3884330834 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b2800d960 con 0x7f7b300af2c0 2022-07-20T03:16:02.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- 172.21.15.61:0/3884330834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b300af2c0 msgr2=0x7f7b300af6f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 --2- 172.21.15.61:0/3884330834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b300af2c0 0x7f7b300af6f0 secure :-1 s=READY pgs=1152 cs=0 l=1 rev1=1 rx=0x7f7b28002f70 tx=0x7f7b28006df0).stop 2022-07-20T03:16:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- 172.21.15.61:0/3884330834 shutdown_connections 2022-07-20T03:16:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 --2- 172.21.15.61:0/3884330834 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b300af2c0 0x7f7b300af6f0 unknown :-1 s=CLOSED pgs=1152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- 172.21.15.61:0/3884330834 >> 172.21.15.61:0/3884330834 conn(0x7f7b300b69e0 msgr2=0x7f7b30066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- 172.21.15.61:0/3884330834 shutdown_connections 2022-07-20T03:16:02.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- 172.21.15.61:0/3884330834 wait complete. 2022-07-20T03:16:02.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 Processor -- start 2022-07-20T03:16:02.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- start start 2022-07-20T03:16:02.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 0x7f7b30142560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b300af2c0 0x7f7b30152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b300b5830 0x7f7b3009cfd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301564f0 con 0x7f7b3005cf40 2022-07-20T03:16:02.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7b301521c0 con 0x7f7b300b5830 2022-07-20T03:16:02.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.288+0000 7f7b350b5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7b30152460 con 0x7f7b300af2c0 2022-07-20T03:16:02.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 0x7f7b30142560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 0x7f7b30142560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49562/0 (socket says 172.21.15.61:49562) 2022-07-20T03:16:02.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 -- 172.21.15.61:0/101865618 learned_addr learned my addr 172.21.15.61:0/101865618 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:02.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2e59c700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b300af2c0 0x7f7b30152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2f59e700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b300b5830 0x7f7b3009cfd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 -- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b300af2c0 msgr2=0x7f7b30152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b300af2c0 0x7f7b30152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 -- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b300b5830 msgr2=0x7f7b3009cfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b300b5830 0x7f7b3009cfd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 -- 172.21.15.61:0/101865618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7b2800b180 con 0x7f7b3005cf40 2022-07-20T03:16:02.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2ed9d700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 0x7f7b30142560 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f7b3006ad40 tx=0x7f7b200014a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b2000e4a0 con 0x7f7b3005cf40 2022-07-20T03:16:02.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7b2000e600 con 0x7f7b3005cf40 2022-07-20T03:16:02.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7b30152700 con 0x7f7b3005cf40 2022-07-20T03:16:02.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7b2000e960 con 0x7f7b3005cf40 2022-07-20T03:16:02.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7b3009d4c0 con 0x7f7b3005cf40 2022-07-20T03:16:02.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7b20019030 con 0x7f7b3005cf40 2022-07-20T03:16:02.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7b30152010 con 0x7f7b3005cf40 2022-07-20T03:16:02.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b27fff700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b140539d0 0x7f7b14055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7b2006b650 con 0x7f7b3005cf40 2022-07-20T03:16:02.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.292+0000 7f7b2e59c700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b140539d0 0x7f7b14055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.296+0000 7f7b2e59c700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b140539d0 0x7f7b14055e90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f7b1800d000 tx=0x7f7b18007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.300+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7b20042450 con 0x7f7b3005cf40 2022-07-20T03:16:02.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.464+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f7b30143340 con 0x7f7b3005cf40 2022-07-20T03:16:02.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b27fff700 1 -- 172.21.15.61:0/101865618 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v39) v1 ==== 184+0+0 (secure 0 0 0) 0x7f7b20017020 con 0x7f7b3005cf40 2022-07-20T03:16:02.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b140539d0 msgr2=0x7f7b14055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b140539d0 0x7f7b14055e90 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7f7b1800d000 tx=0x7f7b18007be0).stop 2022-07-20T03:16:02.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 msgr2=0x7f7b30142560 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 0x7f7b30142560 secure :-1 s=READY pgs=1153 cs=0 l=1 rev1=1 rx=0x7f7b3006ad40 tx=0x7f7b200014a0).stop 2022-07-20T03:16:02.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 shutdown_connections 2022-07-20T03:16:02.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7b140539d0 0x7f7b14055e90 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7b300b5830 0x7f7b3009cfd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7b300af2c0 0x7f7b30152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 --2- 172.21.15.61:0/101865618 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7b3005cf40 0x7f7b30142560 unknown :-1 s=CLOSED pgs=1153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.472+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 >> 172.21.15.61:0/101865618 conn(0x7f7b300b69e0 msgr2=0x7f7b300b88e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.476+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 shutdown_connections 2022-07-20T03:16:02.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.476+0000 7f7b350b5700 1 -- 172.21.15.61:0/101865618 wait complete. 2022-07-20T03:16:02.485 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:16:02.495 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:16:02.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.600+0000 7fb67f59e700 1 Processor -- start 2022-07-20T03:16:02.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.600+0000 7fb67f59e700 1 -- start start 2022-07-20T03:16:02.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb6800b6800 0x7fb6800aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67f59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6800ae280 con 0x7fb6800ad0e0 2022-07-20T03:16:02.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67f59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb6800b6be0 con 0x7fb6800b3cb0 2022-07-20T03:16:02.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb6800ae4b0 con 0x7fb6800b6800 2022-07-20T03:16:02.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb6800ad0e0 0x7fb6801418f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46168/0 (socket says 172.21.15.61:46168) 2022-07-20T03:16:02.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67ed9d700 1 -- 172.21.15.61:0/3941927458 learned_addr learned my addr 172.21.15.61:0/3941927458 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:02.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67e59c700 1 --2- 172.21.15.61:0/3941927458 >> v2:172.21.15.61:3300/0 conn(0x7fb6800b6800 0x7fb6800aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1716579473 0 0) 0x7fb6800ae280 con 0x7fb6800ad0e0 2022-07-20T03:16:02.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb664003260 con 0x7fb6800ad0e0 2022-07-20T03:16:02.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1316340182 0 0) 0x7fb664003260 con 0x7fb6800ad0e0 2022-07-20T03:16:02.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 >> v2:172.21.15.61:3300/0 conn(0x7fb6800b6800 msgr2=0x7fb6800aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:02.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 --2- 172.21.15.61:0/3941927458 >> v2:172.21.15.61:3300/0 conn(0x7fb6800b6800 0x7fb6800aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 >> v1:172.21.15.117:6789/0 conn(0x7fb6800b3cb0 legacy=0x7fb680145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6800ae750 con 0x7fb6800ad0e0 2022-07-20T03:16:02.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fb6780019e0 con 0x7fb6800ad0e0 2022-07-20T03:16:02.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 >> v1:172.21.15.61:6789/0 conn(0x7fb6800ad0e0 legacy=0x7fb6801418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb664003750 0x7fb664003b30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb664004500 0x7fb664004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004e70 0x7fb66400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6800ae4b0 con 0x7fb664004e70 2022-07-20T03:16:02.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb66400bc70 con 0x7fb664004500 2022-07-20T03:16:02.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb66400be30 con 0x7fb664003750 2022-07-20T03:16:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67ed9d700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004e70 0x7fb66400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb664004500 0x7fb664004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67e59c700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb664003750 0x7fb664003b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 -- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb664003750 msgr2=0x7fb664003b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:02.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb664003750 0x7fb664003b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 -- 172.21.15.61:0/3941927458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004e70 msgr2=0x7fb66400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:02.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004e70 0x7fb66400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 -- 172.21.15.61:0/3941927458 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb664003260 con 0x7fb664004500 2022-07-20T03:16:02.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67ed9d700 1 -- 172.21.15.61:0/3941927458 reap_dead start 2022-07-20T03:16:02.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb664004500 0x7fb664004980 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fb670001020 tx=0x7fb6700079e0).ready entity=mon.1 client_cookie=8aa895ba595c112f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670007cf0 con 0x7fb664004500 2022-07-20T03:16:02.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb670007030 con 0x7fb664004500 2022-07-20T03:16:02.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.604+0000 7fb67d59a700 1 -- 172.21.15.61:0/3941927458 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670007390 con 0x7fb664004500 2022-07-20T03:16:02.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 -- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb664004500 msgr2=0x7fb664004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb664004500 0x7fb664004980 secure :-1 s=READY pgs=523 cs=0 l=1 rev1=1 rx=0x7fb670001020 tx=0x7fb6700079e0).stop 2022-07-20T03:16:02.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 -- 172.21.15.61:0/3941927458 shutdown_connections 2022-07-20T03:16:02.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 --2- 172.21.15.61:0/3941927458 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb664004500 0x7fb664004980 unknown :-1 s=CLOSED pgs=523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 -- 172.21.15.61:0/3941927458 >> 172.21.15.61:0/3941927458 conn(0x7fb6800b9dd0 msgr2=0x7fb6800b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 -- 172.21.15.61:0/3941927458 shutdown_connections 2022-07-20T03:16:02.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 -- 172.21.15.61:0/3941927458 wait complete. 2022-07-20T03:16:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 Processor -- start 2022-07-20T03:16:02.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.608+0000 7fb67f59e700 1 -- start start 2022-07-20T03:16:02.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 0x7fb6800b2510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6801151d0 0x7fb680140a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb6800b34b0 0x7fb680140f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6801127e0 con 0x7fb664004500 2022-07-20T03:16:02.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb680112990 con 0x7fb6801151d0 2022-07-20T03:16:02.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb680112c30 con 0x7fb6800b34b0 2022-07-20T03:16:02.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 0x7fb6800b2510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6801151d0 0x7fb680140a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67e59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 0x7fb6800b2510 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49582/0 (socket says 172.21.15.61:49582) 2022-07-20T03:16:02.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67e59c700 1 -- 172.21.15.61:0/4041248878 learned_addr learned my addr 172.21.15.61:0/4041248878 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:02.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67ed9d700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb6800b34b0 0x7fb680140f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 -- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb6800b34b0 msgr2=0x7fb680140f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb6800b34b0 0x7fb680140f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 -- 172.21.15.61:0/4041248878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 msgr2=0x7fb6800b2510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 0x7fb6800b2510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 -- 172.21.15.61:0/4041248878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb680112ed0 con 0x7fb6801151d0 2022-07-20T03:16:02.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67e59c700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 0x7fb6800b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:02.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67dd9b700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6801151d0 0x7fb680140a80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fb6800b1f90 tx=0x7fb670015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670020600 con 0x7fb6801151d0 2022-07-20T03:16:02.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb670020760 con 0x7fb6801151d0 2022-07-20T03:16:02.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb680113370 con 0x7fb6801151d0 2022-07-20T03:16:02.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb670020ac0 con 0x7fb6801151d0 2022-07-20T03:16:02.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.612+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb680151bf0 con 0x7fb6801151d0 2022-07-20T03:16:02.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.616+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb670020c20 con 0x7fb6801151d0 2022-07-20T03:16:02.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.616+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb680113060 con 0x7fb6801151d0 2022-07-20T03:16:02.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.616+0000 7fb66f7fe700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb65c053a60 0x7fb65c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.616+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb67001c040 con 0x7fb6801151d0 2022-07-20T03:16:02.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.616+0000 7fb67e59c700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb65c053a60 0x7fb65c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.616+0000 7fb67e59c700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb65c053a60 0x7fb65c055f20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fb67400b000 tx=0x7fb674006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.624+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb670043910 con 0x7fb6801151d0 2022-07-20T03:16:02.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.796+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb6800b5530 con 0x7fb6801151d0 2022-07-20T03:16:02.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.808+0000 7fb66f7fe700 1 -- 172.21.15.61:0/4041248878 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v40) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb670046f30 con 0x7fb6801151d0 2022-07-20T03:16:02.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb65c053a60 msgr2=0x7fb65c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb65c053a60 0x7fb65c055f20 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fb67400b000 tx=0x7fb674006df0).stop 2022-07-20T03:16:02.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6801151d0 msgr2=0x7fb680140a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6801151d0 0x7fb680140a80 secure :-1 s=READY pgs=524 cs=0 l=1 rev1=1 rx=0x7fb6800b1f90 tx=0x7fb670015000).stop 2022-07-20T03:16:02.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 shutdown_connections 2022-07-20T03:16:02.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb65c053a60 0x7fb65c055f20 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb6801151d0 0x7fb680140a80 unknown :-1 s=CLOSED pgs=524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb6800b34b0 0x7fb680140f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 --2- 172.21.15.61:0/4041248878 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb664004500 0x7fb6800b2510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 >> 172.21.15.61:0/4041248878 conn(0x7fb6800b9dd0 msgr2=0x7fb6800b7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 shutdown_connections 2022-07-20T03:16:02.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.812+0000 7fb67f59e700 1 -- 172.21.15.61:0/4041248878 wait complete. 2022-07-20T03:16:02.820 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:16:02.831 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:02.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 Processor -- start 2022-07-20T03:16:02.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- start start 2022-07-20T03:16:02.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400adff0 0x7f89400b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8940147010 con 0x7f89400af8e0 2022-07-20T03:16:02.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f89401472b0 con 0x7f89400b5650 2022-07-20T03:16:02.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8940146d40 con 0x7f89400adff0 2022-07-20T03:16:02.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f89400af8e0 0x7f894005bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46192/0 (socket says 172.21.15.61:46192) 2022-07-20T03:16:02.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893f7fe700 1 -- 172.21.15.61:0/967289126 learned_addr learned my addr 172.21.15.61:0/967289126 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:02.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f89367fc700 1 --2- 172.21.15.61:0/967289126 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400adff0 0x7f89400b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1118443753 0 0) 0x7f8940147010 con 0x7f89400af8e0 2022-07-20T03:16:02.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8920003260 con 0x7f89400af8e0 2022-07-20T03:16:02.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2496693236 0 0) 0x7f89401472b0 con 0x7f89400b5650 2022-07-20T03:16:02.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f8940147010 con 0x7f89400b5650 2022-07-20T03:16:02.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f89367fc700 1 -- 172.21.15.61:0/967289126 >> v1:172.21.15.117:6789/0 conn(0x7f89400b5650 legacy=0x7f8940147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f89367fc700 1 -- 172.21.15.61:0/967289126 >> v1:172.21.15.61:6789/0 conn(0x7f89400af8e0 legacy=0x7f894005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f89367fc700 1 -- 172.21.15.61:0/967289126 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f894005c4a0 con 0x7f89400adff0 2022-07-20T03:16:02.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f89367fc700 1 --2- 172.21.15.61:0/967289126 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400adff0 0x7f89400b5a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8928002f70 tx=0x7f8928006d90).ready entity=mon.2 client_cookie=70b4b5090ce1ab74 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f892800d4a0 con 0x7f89400adff0 2022-07-20T03:16:02.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8928000e90 con 0x7f89400adff0 2022-07-20T03:16:02.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893e7fc700 1 -- 172.21.15.61:0/967289126 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f892800d7f0 con 0x7f89400adff0 2022-07-20T03:16:02.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- 172.21.15.61:0/967289126 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400adff0 msgr2=0x7f89400b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 --2- 172.21.15.61:0/967289126 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400adff0 0x7f89400b5a30 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f8928002f70 tx=0x7f8928006d90).stop 2022-07-20T03:16:02.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- 172.21.15.61:0/967289126 shutdown_connections 2022-07-20T03:16:02.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 --2- 172.21.15.61:0/967289126 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400adff0 0x7f89400b5a30 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- 172.21.15.61:0/967289126 >> 172.21.15.61:0/967289126 conn(0x7f89400634e0 msgr2=0x7f89400b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:02.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- 172.21.15.61:0/967289126 shutdown_connections 2022-07-20T03:16:02.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.936+0000 7f893ffff700 1 -- 172.21.15.61:0/967289126 wait complete. 2022-07-20T03:16:02.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 Processor -- start 2022-07-20T03:16:02.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- start start 2022-07-20T03:16:02.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89400adff0 0x7f8940145140 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400af8e0 0x7f8940065320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 0x7f8940063a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8940110370 con 0x7f89400b5650 2022-07-20T03:16:02.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f89401434f0 con 0x7f89400adff0 2022-07-20T03:16:02.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8940143790 con 0x7f89400af8e0 2022-07-20T03:16:02.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 0x7f8940063a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 0x7f8940063a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49600/0 (socket says 172.21.15.61:49600) 2022-07-20T03:16:02.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 -- 172.21.15.61:0/2471600934 learned_addr learned my addr 172.21.15.61:0/2471600934 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:02.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893effd700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89400adff0 0x7f8940145140 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f89367fc700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400af8e0 0x7f8940065320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 -- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400af8e0 msgr2=0x7f8940065320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400af8e0 0x7f8940065320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 -- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89400adff0 msgr2=0x7f8940145140 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:02.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89400adff0 0x7f8940145140 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:02.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 -- 172.21.15.61:0/2471600934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f892800b120 con 0x7f89400b5650 2022-07-20T03:16:02.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893f7fe700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 0x7f8940063a00 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f8938001490 tx=0x7f8938007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8938005ba0 con 0x7f89400b5650 2022-07-20T03:16:02.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8938005d00 con 0x7f89400b5650 2022-07-20T03:16:02.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8940110060 con 0x7f89400b5650 2022-07-20T03:16:02.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f89380072c0 con 0x7f89400b5650 2022-07-20T03:16:02.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f89401117e0 con 0x7f89400b5650 2022-07-20T03:16:02.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.940+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8940143950 con 0x7f89400b5650 2022-07-20T03:16:02.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.944+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f893801a060 con 0x7f89400b5650 2022-07-20T03:16:02.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.944+0000 7f893cff9700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:02.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.944+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f893806bad0 con 0x7f89400b5650 2022-07-20T03:16:02.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.944+0000 7f893effd700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:02.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.944+0000 7f893effd700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f893000b000 tx=0x7f8930009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:02.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:02.952+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8938042a30 con 0x7f89400b5650 2022-07-20T03:16:03.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f89400574e0 con 0x7f89400b5650 2022-07-20T03:16:03.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893cff9700 1 -- 172.21.15.61:0/2471600934 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f8938018070 con 0x7f89400b5650 2022-07-20T03:16:03.124 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:03.124 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7662,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3960371515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3960371515},{"type":"v1","addr":"172.21.15.61:6833","nonce":3960371515}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:59.567631+0000","modified":"2022-07-20T03:16:01.250517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"b","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/2884785802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2884785802},{"type":"v1","addr":"172.21.15.117:6835","nonce":2884785802}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T03:16:03.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 msgr2=0x7f8924055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 secure :-1 s=READY pgs=557 cs=0 l=1 rev1=1 rx=0x7f893000b000 tx=0x7f8930009040).stop 2022-07-20T03:16:03.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 msgr2=0x7f8940063a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 0x7f8940063a00 secure :-1 s=READY pgs=1156 cs=0 l=1 rev1=1 rx=0x7f8938001490 tx=0x7f8938007e00).stop 2022-07-20T03:16:03.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 shutdown_connections 2022-07-20T03:16:03.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8924053a40 0x7f8924055f00 unknown :-1 s=CLOSED pgs=557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f89400b5650 0x7f8940063a00 unknown :-1 s=CLOSED pgs=1156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f89400af8e0 0x7f8940065320 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 --2- 172.21.15.61:0/2471600934 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f89400adff0 0x7f8940145140 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 >> 172.21.15.61:0/2471600934 conn(0x7f89400634e0 msgr2=0x7f89400b3c10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:03.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 shutdown_connections 2022-07-20T03:16:03.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.120+0000 7f893ffff700 1 -- 172.21.15.61:0/2471600934 wait complete. 2022-07-20T03:16:03.130 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-20T03:16:03.342 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:59.567631+0000', 'modified': '2022-07-20T03:16:01.250517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7668}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7668': {'gid': 7668, 'name': 'b', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/2884785802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 2884785802}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 2884785802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:03.343 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:03.344 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:16:03.464 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7668,"want_state":"up:active","state":"up:active","rank_uptime":3.8833759840000002,"mdsmap_epoch":62,"osdmap_epoch":111,"osdmap_epoch_barrier":111,"uptime":8.4358533760000007} 2022-07-20T03:16:03.469 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7668, "mdsmap_epoch": 62, "osdmap_epoch": 111, "osdmap_epoch_barrier": 111, "rank_uptime": 3.883375984, "state": "up:active", "uptime": 8.435853376, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:03.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:03.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.580+0000 7fd4dc816700 1 Processor -- start 2022-07-20T03:16:03.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.580+0000 7fd4dc816700 1 -- start start 2022-07-20T03:16:03.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd4d80adff0 0x7fd4d80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d8147010 con 0x7fd4d80af8e0 2022-07-20T03:16:03.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d81472b0 con 0x7fd4d80b5650 2022-07-20T03:16:03.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd4d8146d40 con 0x7fd4d80adff0 2022-07-20T03:16:03.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d759e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd4d80af8e0 0x7fd4d805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46206/0 (socket says 172.21.15.61:46206) 2022-07-20T03:16:03.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d759e700 1 -- 172.21.15.61:0/3433841354 learned_addr learned my addr 172.21.15.61:0/3433841354 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:03.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d659c700 1 --2- 172.21.15.61:0/3433841354 >> v2:172.21.15.61:3300/0 conn(0x7fd4d80adff0 0x7fd4d80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d5d9b700 1 -- 172.21.15.61:0/3433841354 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602010426 0 0) 0x7fd4d81472b0 con 0x7fd4d80b5650 2022-07-20T03:16:03.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d659c700 1 -- 172.21.15.61:0/3433841354 >> v1:172.21.15.117:6789/0 conn(0x7fd4d80b5650 legacy=0x7fd4d8147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d659c700 1 -- 172.21.15.61:0/3433841354 >> v1:172.21.15.61:6789/0 conn(0x7fd4d80af8e0 legacy=0x7fd4d805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d659c700 1 -- 172.21.15.61:0/3433841354 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d805c4a0 con 0x7fd4d80adff0 2022-07-20T03:16:03.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d659c700 1 --2- 172.21.15.61:0/3433841354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80adff0 0x7fd4d80b5a30 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7fd4c8001110 tx=0x7fd4c0006d90).ready entity=mon.0 client_cookie=47c8e4e08d325aa0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:03.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d5d9b700 1 -- 172.21.15.61:0/3433841354 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c000d4a0 con 0x7fd4d80adff0 2022-07-20T03:16:03.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d5d9b700 1 -- 172.21.15.61:0/3433841354 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c0000e90 con 0x7fd4d80adff0 2022-07-20T03:16:03.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4d5d9b700 1 -- 172.21.15.61:0/3433841354 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c000d7f0 con 0x7fd4d80adff0 2022-07-20T03:16:03.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- 172.21.15.61:0/3433841354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80adff0 msgr2=0x7fd4d80b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 --2- 172.21.15.61:0/3433841354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80adff0 0x7fd4d80b5a30 secure :-1 s=READY pgs=1158 cs=0 l=1 rev1=1 rx=0x7fd4c8001110 tx=0x7fd4c0006d90).stop 2022-07-20T03:16:03.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- 172.21.15.61:0/3433841354 shutdown_connections 2022-07-20T03:16:03.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 --2- 172.21.15.61:0/3433841354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80adff0 0x7fd4d80b5a30 unknown :-1 s=CLOSED pgs=1158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- 172.21.15.61:0/3433841354 >> 172.21.15.61:0/3433841354 conn(0x7fd4d80634e0 msgr2=0x7fd4d80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:03.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- 172.21.15.61:0/3433841354 shutdown_connections 2022-07-20T03:16:03.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.584+0000 7fd4dc816700 1 -- 172.21.15.61:0/3433841354 wait complete. 2022-07-20T03:16:03.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 Processor -- start 2022-07-20T03:16:03.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 -- start start 2022-07-20T03:16:03.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d80adff0 0x7fd4d8063cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 0x7fd4d80641c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d80b5650 0x7fd4d81563c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d81411e0 con 0x7fd4d80af8e0 2022-07-20T03:16:03.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d810e9d0 con 0x7fd4d80adff0 2022-07-20T03:16:03.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d8157010 con 0x7fd4d80b5650 2022-07-20T03:16:03.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 0x7fd4d80641c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d80adff0 0x7fd4d8063cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 0x7fd4d80641c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49612/0 (socket says 172.21.15.61:49612) 2022-07-20T03:16:03.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d659c700 1 -- 172.21.15.61:0/310501451 learned_addr learned my addr 172.21.15.61:0/310501451 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:03.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d759e700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d80b5650 0x7fd4d81563c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 -- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d80b5650 msgr2=0x7fd4d81563c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d80b5650 0x7fd4d81563c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 -- 172.21.15.61:0/310501451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 msgr2=0x7fd4d80641c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 0x7fd4d80641c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 -- 172.21.15.61:0/310501451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c000b120 con 0x7fd4d80adff0 2022-07-20T03:16:03.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d659c700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 0x7fd4d80641c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:03.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4d6d9d700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d80adff0 0x7fd4d8063cd0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fd4d0001250 tx=0x7fd4d0007f10).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:03.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4d000f470 con 0x7fd4d80adff0 2022-07-20T03:16:03.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4d000f5d0 con 0x7fd4d80adff0 2022-07-20T03:16:03.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d814adc0 con 0x7fd4d80adff0 2022-07-20T03:16:03.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4d000f930 con 0x7fd4d80adff0 2022-07-20T03:16:03.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.588+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d810ee00 con 0x7fd4d80adff0 2022-07-20T03:16:03.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.592+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d80574e0 con 0x7fd4d80adff0 2022-07-20T03:16:03.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.592+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd4d0019030 con 0x7fd4d80adff0 2022-07-20T03:16:03.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.592+0000 7fd4cffff700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4bc053a40 0x7fd4bc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.592+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd4d006b730 con 0x7fd4d80adff0 2022-07-20T03:16:03.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.592+0000 7fd4d659c700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4bc053a40 0x7fd4bc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.592+0000 7fd4d659c700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4bc053a40 0x7fd4bc055f00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fd4c001a000 tx=0x7fd4c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:03.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.600+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4d0042690 con 0x7fd4d80adff0 2022-07-20T03:16:03.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.728+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd4d8067ea0 con 0x7fd4d80adff0 2022-07-20T03:16:03.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4cffff700 1 -- 172.21.15.61:0/310501451 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16099 (secure 0 0 0) 0x7fd4d0045cb0 con 0x7fd4d80adff0 2022-07-20T03:16:03.743 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:03.744 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":111,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:00.570268+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:58.052517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:58.563360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:03.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4bc053a40 msgr2=0x7fd4bc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4bc053a40 0x7fd4bc055f00 secure :-1 s=READY pgs=558 cs=0 l=1 rev1=1 rx=0x7fd4c001a000 tx=0x7fd4c0014000).stop 2022-07-20T03:16:03.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d80adff0 msgr2=0x7fd4d8063cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d80adff0 0x7fd4d8063cd0 secure :-1 s=READY pgs=527 cs=0 l=1 rev1=1 rx=0x7fd4d0001250 tx=0x7fd4d0007f10).stop 2022-07-20T03:16:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 shutdown_connections 2022-07-20T03:16:03.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4bc053a40 0x7fd4bc055f00 unknown :-1 s=CLOSED pgs=558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d80b5650 0x7fd4d81563c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d80af8e0 0x7fd4d80641c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 --2- 172.21.15.61:0/310501451 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d80adff0 0x7fd4d8063cd0 unknown :-1 s=CLOSED pgs=527 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 >> 172.21.15.61:0/310501451 conn(0x7fd4d80634e0 msgr2=0x7fd4d81438a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.732+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 shutdown_connections 2022-07-20T03:16:03.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.736+0000 7fd4dc816700 1 -- 172.21.15.61:0/310501451 wait complete. 2022-07-20T03:16:03.756 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:15:58.052517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:15:58.563360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:16:03.758 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:16:03.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 Processor -- start 2022-07-20T03:16:03.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 -- start start 2022-07-20T03:16:03.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f948c05cf40 0x7f948c0adb20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f948c062ec0 0x7f948c142b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f948c0ad3e0 con 0x7f948c143070 2022-07-20T03:16:03.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c0b3b10 con 0x7f948c05cf40 2022-07-20T03:16:03.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.868+0000 7f9491091700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f948c0ad680 con 0x7f948c062ec0 2022-07-20T03:16:03.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f948c05cf40 0x7f948c0adb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948b59e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f948c143070 0x7f948c146580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46218/0 (socket says 172.21.15.61:46218) 2022-07-20T03:16:03.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948b59e700 1 -- 172.21.15.61:0/1109042565 learned_addr learned my addr 172.21.15.61:0/1109042565 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:03.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f948c05cf40 0x7f948c0adb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49618/0 (socket says 172.21.15.61:49618) 2022-07-20T03:16:03.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948259c700 1 --2- 172.21.15.61:0/1109042565 >> v2:172.21.15.117:3300/0 conn(0x7f948c062ec0 0x7f948c142b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 -- 172.21.15.61:0/1109042565 >> v2:172.21.15.117:3300/0 conn(0x7f948c062ec0 msgr2=0x7f948c142b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:03.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 --2- 172.21.15.61:0/1109042565 >> v2:172.21.15.117:3300/0 conn(0x7f948c062ec0 0x7f948c142b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 -- 172.21.15.61:0/1109042565 >> v1:172.21.15.61:6789/0 conn(0x7f948c143070 legacy=0x7f948c146580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948a59c700 1 -- 172.21.15.61:0/1109042565 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2869180350 0 0) 0x7f948c0ad3e0 con 0x7f948c143070 2022-07-20T03:16:03.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 -- 172.21.15.61:0/1109042565 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f948c064810 con 0x7f948c05cf40 2022-07-20T03:16:03.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948259c700 1 --2- 172.21.15.61:0/1109042565 >> v2:172.21.15.117:3300/0 conn(0x7f948c062ec0 0x7f948c142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:03.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948ad9d700 1 --2- 172.21.15.61:0/1109042565 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c05cf40 0x7f948c0adb20 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7f947c002f70 tx=0x7f947c006df0).ready entity=mon.0 client_cookie=1d59ebee7fb99e33 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:03.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948a59c700 1 -- 172.21.15.61:0/1109042565 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f947c00d5c0 con 0x7f948c05cf40 2022-07-20T03:16:03.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948a59c700 1 -- 172.21.15.61:0/1109042565 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f947c000e90 con 0x7f948c05cf40 2022-07-20T03:16:03.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f948a59c700 1 -- 172.21.15.61:0/1109042565 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f947c00d910 con 0x7f948c05cf40 2022-07-20T03:16:03.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 -- 172.21.15.61:0/1109042565 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c05cf40 msgr2=0x7f948c0adb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 --2- 172.21.15.61:0/1109042565 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c05cf40 0x7f948c0adb20 secure :-1 s=READY pgs=1160 cs=0 l=1 rev1=1 rx=0x7f947c002f70 tx=0x7f947c006df0).stop 2022-07-20T03:16:03.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 -- 172.21.15.61:0/1109042565 shutdown_connections 2022-07-20T03:16:03.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 --2- 172.21.15.61:0/1109042565 >> v2:172.21.15.117:3300/0 conn(0x7f948c062ec0 0x7f948c142b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 --2- 172.21.15.61:0/1109042565 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c05cf40 0x7f948c0adb20 unknown :-1 s=CLOSED pgs=1160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 -- 172.21.15.61:0/1109042565 >> 172.21.15.61:0/1109042565 conn(0x7f948c0b7d30 msgr2=0x7f948c0b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:03.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 -- 172.21.15.61:0/1109042565 shutdown_connections 2022-07-20T03:16:03.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 -- 172.21.15.61:0/1109042565 wait complete. 2022-07-20T03:16:03.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 Processor -- start 2022-07-20T03:16:03.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.872+0000 7f9491091700 1 -- start start 2022-07-20T03:16:03.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f948c05cf40 0x7f948c0ae360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 0x7f948c0b5a70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f948c143070 0x7f948c155370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c1455c0 con 0x7f948c062ec0 2022-07-20T03:16:03.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f948c148e70 con 0x7f948c143070 2022-07-20T03:16:03.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f948c149310 con 0x7f948c05cf40 2022-07-20T03:16:03.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 0x7f948c0b5a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 0x7f948c0b5a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49626/0 (socket says 172.21.15.61:49626) 2022-07-20T03:16:03.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 -- 172.21.15.61:0/2576730726 learned_addr learned my addr 172.21.15.61:0/2576730726 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:03.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948b59e700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f948c143070 0x7f948c155370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948ad9d700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f948c05cf40 0x7f948c0ae360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 -- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f948c05cf40 msgr2=0x7f948c0ae360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f948c05cf40 0x7f948c0ae360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 -- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f948c143070 msgr2=0x7f948c155370 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:03.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f948c143070 0x7f948c155370 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:03.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 -- 172.21.15.61:0/2576730726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f947c00b180 con 0x7f948c062ec0 2022-07-20T03:16:03.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948ad9d700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f948c05cf40 0x7f948c0ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:03.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f948259c700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 0x7f948c0b5a70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7f9474006940 tx=0x7f9474008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:03.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9474011780 con 0x7f948c062ec0 2022-07-20T03:16:03.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f94740118e0 con 0x7f948c062ec0 2022-07-20T03:16:03.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f948c0af1e0 con 0x7f948c062ec0 2022-07-20T03:16:03.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9474011c40 con 0x7f948c062ec0 2022-07-20T03:16:03.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.876+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f948c155a80 con 0x7f948c062ec0 2022-07-20T03:16:03.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.880+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f948c148cd0 con 0x7f948c062ec0 2022-07-20T03:16:03.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.880+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9474012420 con 0x7f948c062ec0 2022-07-20T03:16:03.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.880+0000 7f9488d99700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9464053a60 0x7f9464055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:03.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.880+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f947406bd70 con 0x7f948c062ec0 2022-07-20T03:16:03.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.880+0000 7f948ad9d700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9464053a60 0x7f9464055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:03.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.880+0000 7f948ad9d700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9464053a60 0x7f9464055f20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f947c018000 tx=0x7f947c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:03.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:03.888+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f948c148cd0 con 0x7f948c062ec0 2022-07-20T03:16:04.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.012+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f948c148cd0 con 0x7f948c062ec0 2022-07-20T03:16:04.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.012+0000 7f9488d99700 1 -- 172.21.15.61:0/2576730726 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v111) v1 ==== 110+0+10 (secure 0 0 0) 0x7f948c148cd0 con 0x7f948c062ec0 2022-07-20T03:16:04.017 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:16:04.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9464053a60 msgr2=0x7f9464055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9464053a60 0x7f9464055f20 secure :-1 s=READY pgs=559 cs=0 l=1 rev1=1 rx=0x7f947c018000 tx=0x7f947c014000).stop 2022-07-20T03:16:04.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 msgr2=0x7f948c0b5a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 0x7f948c0b5a70 secure :-1 s=READY pgs=1161 cs=0 l=1 rev1=1 rx=0x7f9474006940 tx=0x7f9474008040).stop 2022-07-20T03:16:04.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 shutdown_connections 2022-07-20T03:16:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9464053a60 0x7f9464055f20 unknown :-1 s=CLOSED pgs=559 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f948c143070 0x7f948c155370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f948c062ec0 0x7f948c0b5a70 unknown :-1 s=CLOSED pgs=1161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 --2- 172.21.15.61:0/2576730726 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f948c05cf40 0x7f948c0ae360 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 >> 172.21.15.61:0/2576730726 conn(0x7f948c0b7d30 msgr2=0x7f948c1497c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 shutdown_connections 2022-07-20T03:16:04.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.016+0000 7f9491091700 1 -- 172.21.15.61:0/2576730726 wait complete. 2022-07-20T03:16:04.050 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:16:04.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbeacbba700 1 Processor -- start 2022-07-20T03:16:04.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbeacbba700 1 -- start start 2022-07-20T03:16:04.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbeacbba700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbea8063270 0x7fbea80675f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbeacbba700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea80b16f0 con 0x7fbea8140860 2022-07-20T03:16:04.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbeacbba700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbea8140d30 con 0x7fbea805cda0 2022-07-20T03:16:04.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbeacbba700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbea80624b0 con 0x7fbea8063270 2022-07-20T03:16:04.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea5d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbea8140860 0x7fbea8141470 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46232/0 (socket says 172.21.15.61:46232) 2022-07-20T03:16:04.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea5d9b700 1 -- 172.21.15.61:0/851315505 learned_addr learned my addr 172.21.15.61:0/851315505 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:04.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea6d9d700 1 --2- 172.21.15.61:0/851315505 >> v2:172.21.15.61:3300/0 conn(0x7fbea8063270 0x7fbea80675f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2927332684 0 0) 0x7fbea80b16f0 con 0x7fbea8140860 2022-07-20T03:16:04.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbe8c003260 con 0x7fbea8140860 2022-07-20T03:16:04.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1854253943 0 0) 0x7fbea8140d30 con 0x7fbea805cda0 2022-07-20T03:16:04.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbea80b16f0 con 0x7fbea805cda0 2022-07-20T03:16:04.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3494708928 0 0) 0x7fbe8c003260 con 0x7fbea8140860 2022-07-20T03:16:04.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea6d9d700 1 -- 172.21.15.61:0/851315505 >> v1:172.21.15.117:6789/0 conn(0x7fbea805cda0 legacy=0x7fbea8061da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea6d9d700 1 -- 172.21.15.61:0/851315505 >> v1:172.21.15.61:6789/0 conn(0x7fbea8140860 legacy=0x7fbea8141470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea6d9d700 1 -- 172.21.15.61:0/851315505 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea8140fd0 con 0x7fbea8063270 2022-07-20T03:16:04.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea6d9d700 1 --2- 172.21.15.61:0/851315505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea8063270 0x7fbea80675f0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7fbe9c001110 tx=0x7fbea00091d0).ready entity=mon.0 client_cookie=ec431c20a2ce8760 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe9c001660 con 0x7fbea8063270 2022-07-20T03:16:04.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbea000f510 con 0x7fbea8063270 2022-07-20T03:16:04.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.164+0000 7fbea559a700 1 -- 172.21.15.61:0/851315505 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbea000f680 con 0x7fbea8063270 2022-07-20T03:16:04.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- 172.21.15.61:0/851315505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea8063270 msgr2=0x7fbea80675f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 --2- 172.21.15.61:0/851315505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea8063270 0x7fbea80675f0 secure :-1 s=READY pgs=1163 cs=0 l=1 rev1=1 rx=0x7fbe9c001110 tx=0x7fbea00091d0).stop 2022-07-20T03:16:04.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- 172.21.15.61:0/851315505 shutdown_connections 2022-07-20T03:16:04.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 --2- 172.21.15.61:0/851315505 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea8063270 0x7fbea80675f0 unknown :-1 s=CLOSED pgs=1163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- 172.21.15.61:0/851315505 >> 172.21.15.61:0/851315505 conn(0x7fbea80b7da0 msgr2=0x7fbea80b3d70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- 172.21.15.61:0/851315505 shutdown_connections 2022-07-20T03:16:04.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- 172.21.15.61:0/851315505 wait complete. 2022-07-20T03:16:04.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 Processor -- start 2022-07-20T03:16:04.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- start start 2022-07-20T03:16:04.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 0x7fbea8143ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbea8063270 0x7fbea80b2f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea8140860 0x7fbea813f510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea80992e0 con 0x7fbea805cda0 2022-07-20T03:16:04.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbea81434e0 con 0x7fbea8140860 2022-07-20T03:16:04.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbeacbba700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbea8143780 con 0x7fbea8063270 2022-07-20T03:16:04.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbea659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 0x7fbea8143ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbea659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 0x7fbea8143ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49640/0 (socket says 172.21.15.61:49640) 2022-07-20T03:16:04.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbea659c700 1 -- 172.21.15.61:0/1909326004 learned_addr learned my addr 172.21.15.61:0/1909326004 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:04.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbea5d9b700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbea8063270 0x7fbea80b2f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.168+0000 7fbea6d9d700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea8140860 0x7fbea813f510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea659c700 1 -- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbea8063270 msgr2=0x7fbea80b2f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea659c700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbea8063270 0x7fbea80b2f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea659c700 1 -- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea8140860 msgr2=0x7fbea813f510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea659c700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea8140860 0x7fbea813f510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea659c700 1 -- 172.21.15.61:0/1909326004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbea0009470 con 0x7fbea805cda0 2022-07-20T03:16:04.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea5d9b700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbea8063270 0x7fbea80b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:04.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbea659c700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 0x7fbea8143ec0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7fbe98001110 tx=0x7fbe98007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe98012400 con 0x7fbea805cda0 2022-07-20T03:16:04.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbe98012560 con 0x7fbea805cda0 2022-07-20T03:16:04.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbea8143a20 con 0x7fbea805cda0 2022-07-20T03:16:04.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbe980128c0 con 0x7fbea805cda0 2022-07-20T03:16:04.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbea8143140 con 0x7fbea805cda0 2022-07-20T03:16:04.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbea81421b0 con 0x7fbea805cda0 2022-07-20T03:16:04.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.172+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbe98019030 con 0x7fbea805cda0 2022-07-20T03:16:04.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.176+0000 7fbe977fe700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe84053a40 0x7fbe84055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.176+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbe9806b270 con 0x7fbea805cda0 2022-07-20T03:16:04.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.176+0000 7fbea5d9b700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe84053a40 0x7fbe84055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.176+0000 7fbea5d9b700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe84053a40 0x7fbe84055f00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fbe9c00d000 tx=0x7fbe9c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.180+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbea81421b0 con 0x7fbea805cda0 2022-07-20T03:16:04.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.308+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fbea81421b0 con 0x7fbea805cda0 2022-07-20T03:16:04.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.308+0000 7fbe977fe700 1 -- 172.21.15.61:0/1909326004 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v111) v1 ==== 92+0+10 (secure 0 0 0) 0x7fbea81421b0 con 0x7fbea805cda0 2022-07-20T03:16:04.315 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:04.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe84053a40 msgr2=0x7fbe84055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe84053a40 0x7fbe84055f00 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7fbe9c00d000 tx=0x7fbe9c009040).stop 2022-07-20T03:16:04.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 msgr2=0x7fbea8143ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 0x7fbea8143ec0 secure :-1 s=READY pgs=1164 cs=0 l=1 rev1=1 rx=0x7fbe98001110 tx=0x7fbe98007ae0).stop 2022-07-20T03:16:04.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 shutdown_connections 2022-07-20T03:16:04.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbe84053a40 0x7fbe84055f00 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbea8140860 0x7fbea813f510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbea8063270 0x7fbea80b2f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 --2- 172.21.15.61:0/1909326004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbea805cda0 0x7fbea8143ec0 unknown :-1 s=CLOSED pgs=1164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 >> 172.21.15.61:0/1909326004 conn(0x7fbea80b7da0 msgr2=0x7fbea80625f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 shutdown_connections 2022-07-20T03:16:04.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.312+0000 7fbeacbba700 1 -- 172.21.15.61:0/1909326004 wait complete. 2022-07-20T03:16:04.327 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:16:04.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 Processor -- start 2022-07-20T03:16:04.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- start start 2022-07-20T03:16:04.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f84f409df40 0x7f84f4145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84f40b4d60 0x7f84f4145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f4060ee0 0x7f84f4146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f4060d20 con 0x7f84f40b4d60 2022-07-20T03:16:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f84f40607e0 con 0x7f84f409df40 2022-07-20T03:16:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4060a80 con 0x7f84f4060ee0 2022-07-20T03:16:04.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84f40b4d60 0x7f84f4145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f84f40b4d60 0x7f84f4145b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49650/0 (socket says 172.21.15.61:49650) 2022-07-20T03:16:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 -- 172.21.15.61:0/281012115 learned_addr learned my addr 172.21.15.61:0/281012115 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f2d9d700 1 --2- 172.21.15.61:0/281012115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f4060ee0 0x7f84f4146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f259c700 1 --2- 172.21.15.61:0/281012115 >> v2:172.21.15.117:3300/0 conn(0x7f84f409df40 0x7f84f4145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 -- 172.21.15.61:0/281012115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f4060ee0 msgr2=0x7f84f4146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/281012115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f4060ee0 0x7f84f4146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 -- 172.21.15.61:0/281012115 >> v2:172.21.15.117:3300/0 conn(0x7f84f409df40 msgr2=0x7f84f4145680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/281012115 >> v2:172.21.15.117:3300/0 conn(0x7f84f409df40 0x7f84f4145680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 -- 172.21.15.61:0/281012115 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84f409e320 con 0x7f84f40b4d60 2022-07-20T03:16:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/281012115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f40b4d60 0x7f84f4145b70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f84e4002f70 tx=0x7f84e4006eb0).ready entity=mon.0 client_cookie=94c62caaea26ce94 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f159a700 1 -- 172.21.15.61:0/281012115 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84e400d570 con 0x7f84f40b4d60 2022-07-20T03:16:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f159a700 1 -- 172.21.15.61:0/281012115 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e4000e90 con 0x7f84f40b4d60 2022-07-20T03:16:04.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f159a700 1 -- 172.21.15.61:0/281012115 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84e400d900 con 0x7f84f40b4d60 2022-07-20T03:16:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- 172.21.15.61:0/281012115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f40b4d60 msgr2=0x7f84f4145b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/281012115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f40b4d60 0x7f84f4145b70 secure :-1 s=READY pgs=1165 cs=0 l=1 rev1=1 rx=0x7f84e4002f70 tx=0x7f84e4006eb0).stop 2022-07-20T03:16:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- 172.21.15.61:0/281012115 shutdown_connections 2022-07-20T03:16:04.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/281012115 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f4060ee0 0x7f84f4146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/281012115 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f40b4d60 0x7f84f4145b70 unknown :-1 s=CLOSED pgs=1165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/281012115 >> v2:172.21.15.117:3300/0 conn(0x7f84f409df40 0x7f84f4145680 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- 172.21.15.61:0/281012115 >> 172.21.15.61:0/281012115 conn(0x7f84f40b5f30 msgr2=0x7f84f40b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- 172.21.15.61:0/281012115 shutdown_connections 2022-07-20T03:16:04.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.436+0000 7f84f8fb6700 1 -- 172.21.15.61:0/281012115 wait complete. 2022-07-20T03:16:04.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 Processor -- start 2022-07-20T03:16:04.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 -- start start 2022-07-20T03:16:04.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 0x7f84f40b1ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84f409df40 0x7f84f408fab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f40b4d60 0x7f84f408ffa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4090970 con 0x7f84f4060ee0 2022-07-20T03:16:04.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4151f10 con 0x7f84f409df40 2022-07-20T03:16:04.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f8fb6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f84f4094d30 con 0x7f84f40b4d60 2022-07-20T03:16:04.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 0x7f84f40b1ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 0x7f84f40b1ec0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49652/0 (socket says 172.21.15.61:49652) 2022-07-20T03:16:04.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f259c700 1 -- 172.21.15.61:0/395291795 learned_addr learned my addr 172.21.15.61:0/395291795 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:04.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f2d9d700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f40b4d60 0x7f84f408ffa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.440+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84f409df40 0x7f84f408fab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f259c700 1 -- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f40b4d60 msgr2=0x7f84f408ffa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f259c700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f40b4d60 0x7f84f408ffa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f259c700 1 -- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84f409df40 msgr2=0x7f84f408fab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f259c700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84f409df40 0x7f84f408fab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f259c700 1 -- 172.21.15.61:0/395291795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84e400b1e0 con 0x7f84f4060ee0 2022-07-20T03:16:04.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f2d9d700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f40b4d60 0x7f84f408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:04.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84f409df40 0x7f84f408fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:04.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f259c700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 0x7f84f40b1ec0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f84e800a000 tx=0x7f84e8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84e8011780 con 0x7f84f4060ee0 2022-07-20T03:16:04.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f84e80118e0 con 0x7f84f4060ee0 2022-07-20T03:16:04.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84f4094fd0 con 0x7f84f4060ee0 2022-07-20T03:16:04.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f84e8011c40 con 0x7f84f4060ee0 2022-07-20T03:16:04.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84f4090510 con 0x7f84f4060ee0 2022-07-20T03:16:04.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84f4146060 con 0x7f84f4060ee0 2022-07-20T03:16:04.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.444+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f84e801c460 con 0x7f84f4060ee0 2022-07-20T03:16:04.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.448+0000 7f84e37fe700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84d0053920 0x7f84d0055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.448+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f84e806c270 con 0x7f84f4060ee0 2022-07-20T03:16:04.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.448+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84d0053920 0x7f84d0055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.448+0000 7f84f1d9b700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84d0053920 0x7f84d0055de0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f84e401c000 tx=0x7f84e4015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.456+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f84e8043170 con 0x7f84f4060ee0 2022-07-20T03:16:04.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.580+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f84f40574e0 con 0x7f84f4060ee0 2022-07-20T03:16:04.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.580+0000 7f84e37fe700 1 -- 172.21.15.61:0/395291795 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+10 (secure 0 0 0) 0x7f84e8046790 con 0x7f84f4060ee0 2022-07-20T03:16:04.589 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:04.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84d0053920 msgr2=0x7f84d0055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84d0053920 0x7f84d0055de0 secure :-1 s=READY pgs=561 cs=0 l=1 rev1=1 rx=0x7f84e401c000 tx=0x7f84e4015000).stop 2022-07-20T03:16:04.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 msgr2=0x7f84f40b1ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 0x7f84f40b1ec0 secure :-1 s=READY pgs=1166 cs=0 l=1 rev1=1 rx=0x7f84e800a000 tx=0x7f84e8008040).stop 2022-07-20T03:16:04.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 shutdown_connections 2022-07-20T03:16:04.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f84d0053920 0x7f84d0055de0 unknown :-1 s=CLOSED pgs=561 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f84f409df40 0x7f84f408fab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f84f40b4d60 0x7f84f408ffa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 --2- 172.21.15.61:0/395291795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f84f4060ee0 0x7f84f40b1ec0 unknown :-1 s=CLOSED pgs=1166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 >> 172.21.15.61:0/395291795 conn(0x7f84f40b5f30 msgr2=0x7f84f40b8b30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 shutdown_connections 2022-07-20T03:16:04.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.584+0000 7f84f8fb6700 1 -- 172.21.15.61:0/395291795 wait complete. 2022-07-20T03:16:04.599 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:16:04.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 Processor -- start 2022-07-20T03:16:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 -- start start 2022-07-20T03:16:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2b8c05cf40 0x7f2b8c0afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b8c0b1220 0x7f2b8c14a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2b8c1472b0 con 0x7f2b8c147c90 2022-07-20T03:16:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c0b0920 con 0x7f2b8c0b1220 2022-07-20T03:16:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b924dc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2b8c147550 con 0x7f2b8c05cf40 2022-07-20T03:16:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b8c0b1220 0x7f2b8c14a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2b8c0b1220 0x7f2b8c14a570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49664/0 (socket says 172.21.15.61:49664) 2022-07-20T03:16:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 -- 172.21.15.61:0/1407357823 learned_addr learned my addr 172.21.15.61:0/1407357823 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:04.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b8bfff700 1 --2- 172.21.15.61:0/1407357823 >> v2:172.21.15.117:3300/0 conn(0x7f2b8c05cf40 0x7f2b8c0afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 -- 172.21.15.61:0/1407357823 >> v2:172.21.15.117:3300/0 conn(0x7f2b8c05cf40 msgr2=0x7f2b8c0afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 --2- 172.21.15.61:0/1407357823 >> v2:172.21.15.117:3300/0 conn(0x7f2b8c05cf40 0x7f2b8c0afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 -- 172.21.15.61:0/1407357823 >> v1:172.21.15.117:6789/0 conn(0x7f2b8c147c90 legacy=0x7f2b8c0b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 -- 172.21.15.61:0/1407357823 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b8c1477f0 con 0x7f2b8c0b1220 2022-07-20T03:16:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b90a79700 1 --2- 172.21.15.61:0/1407357823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c14a570 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f2b840032f0 tx=0x7f2b84009290).ready entity=mon.0 client_cookie=94c682cc966b78 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b8b7fe700 1 -- 172.21.15.61:0/1407357823 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b8400b870 con 0x7f2b8c0b1220 2022-07-20T03:16:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b8b7fe700 1 -- 172.21.15.61:0/1407357823 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b8400b9d0 con 0x7f2b8c0b1220 2022-07-20T03:16:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.708+0000 7f2b8b7fe700 1 -- 172.21.15.61:0/1407357823 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b8400bb40 con 0x7f2b8c0b1220 2022-07-20T03:16:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- 172.21.15.61:0/1407357823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 msgr2=0x7f2b8c14a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 --2- 172.21.15.61:0/1407357823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c14a570 secure :-1 s=READY pgs=1167 cs=0 l=1 rev1=1 rx=0x7f2b840032f0 tx=0x7f2b84009290).stop 2022-07-20T03:16:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- 172.21.15.61:0/1407357823 shutdown_connections 2022-07-20T03:16:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 --2- 172.21.15.61:0/1407357823 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c14a570 unknown :-1 s=CLOSED pgs=1167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 --2- 172.21.15.61:0/1407357823 >> v2:172.21.15.117:3300/0 conn(0x7f2b8c05cf40 0x7f2b8c0afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- 172.21.15.61:0/1407357823 >> 172.21.15.61:0/1407357823 conn(0x7f2b8c0b5b70 msgr2=0x7f2b8c0b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- 172.21.15.61:0/1407357823 shutdown_connections 2022-07-20T03:16:04.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- 172.21.15.61:0/1407357823 wait complete. 2022-07-20T03:16:04.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 Processor -- start 2022-07-20T03:16:04.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- start start 2022-07-20T03:16:04.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 0x7f2b8c0b3bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c066860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b8c147c90 0x7f2b8c1566e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c065ef0 con 0x7f2b8c0b1220 2022-07-20T03:16:04.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c066120 con 0x7f2b8c05cf40 2022-07-20T03:16:04.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b924dc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2b8c0663c0 con 0x7f2b8c147c90 2022-07-20T03:16:04.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c066860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b8bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 0x7f2b8c0b3bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b90a79700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b8c147c90 0x7f2b8c1566e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b83fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c066860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49670/0 (socket says 172.21.15.61:49670) 2022-07-20T03:16:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b8bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 0x7f2b8c0b3bf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59346/0 (socket says 172.21.15.61:59346) 2022-07-20T03:16:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.712+0000 7f2b83fff700 1 -- 172.21.15.61:0/4190542267 learned_addr learned my addr 172.21.15.61:0/4190542267 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b83fff700 1 -- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b8c147c90 msgr2=0x7f2b8c1566e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b83fff700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b8c147c90 0x7f2b8c1566e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b83fff700 1 -- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 msgr2=0x7f2b8c0b3bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b83fff700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 0x7f2b8c0b3bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b83fff700 1 -- 172.21.15.61:0/4190542267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b84009530 con 0x7f2b8c0b1220 2022-07-20T03:16:04.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b8bfff700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 0x7f2b8c0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:04.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b83fff700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c066860 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f2b8c0626f0 tx=0x7f2b74007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b74007e70 con 0x7f2b8c0b1220 2022-07-20T03:16:04.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2b74007030 con 0x7f2b8c0b1220 2022-07-20T03:16:04.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b8c0673f0 con 0x7f2b8c0b1220 2022-07-20T03:16:04.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2b74007390 con 0x7f2b8c0b1220 2022-07-20T03:16:04.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b8c064e90 con 0x7f2b8c0b1220 2022-07-20T03:16:04.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b8c146fa0 con 0x7f2b8c0b1220 2022-07-20T03:16:04.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2b74010030 con 0x7f2b8c0b1220 2022-07-20T03:16:04.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b89ffb700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b70053a40 0x7f2b70055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.716+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2b7406ba80 con 0x7f2b8c0b1220 2022-07-20T03:16:04.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.720+0000 7f2b8bfff700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b70053a40 0x7f2b70055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:04.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.720+0000 7f2b8bfff700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b70053a40 0x7f2b70055f00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f2b7c00b000 tx=0x7f2b7c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:04.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.728+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2b74042940 con 0x7f2b8c0b1220 2022-07-20T03:16:04.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.852+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f2b8c0574e0 con 0x7f2b8c0b1220 2022-07-20T03:16:04.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.852+0000 7f2b89ffb700 1 -- 172.21.15.61:0/4190542267 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+10 (secure 0 0 0) 0x7f2b74045f60 con 0x7f2b8c0b1220 2022-07-20T03:16:04.858 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:04.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b70053a40 msgr2=0x7f2b70055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b70053a40 0x7f2b70055f00 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f2b7c00b000 tx=0x7f2b7c006df0).stop 2022-07-20T03:16:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 msgr2=0x7f2b8c066860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:04.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c066860 secure :-1 s=READY pgs=1168 cs=0 l=1 rev1=1 rx=0x7f2b8c0626f0 tx=0x7f2b74007c00).stop 2022-07-20T03:16:04.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 shutdown_connections 2022-07-20T03:16:04.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2b70053a40 0x7f2b70055f00 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2b8c147c90 0x7f2b8c1566e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2b8c0b1220 0x7f2b8c066860 unknown :-1 s=CLOSED pgs=1168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 --2- 172.21.15.61:0/4190542267 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2b8c05cf40 0x7f2b8c0b3bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:04.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 >> 172.21.15.61:0/4190542267 conn(0x7f2b8c0b5b70 msgr2=0x7f2b8c13f9b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:04.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 shutdown_connections 2022-07-20T03:16:04.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.856+0000 7f2b924dc700 1 -- 172.21.15.61:0/4190542267 wait complete. 2022-07-20T03:16:04.871 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:16:04.872 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:04.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b4a539700 1 Processor -- start 2022-07-20T03:16:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b4a539700 1 -- start start 2022-07-20T03:16:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b4a539700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 0x7f4b440ab0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b4a539700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b4405ff30 con 0x7f4b440b1620 2022-07-20T03:16:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b4a539700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b440b1a00 con 0x7f4b440616b0 2022-07-20T03:16:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b4a539700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440ad500 con 0x7f4b44061b40 2022-07-20T03:16:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b43fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4b440b1620 0x7f4b440acdf0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46272/0 (socket says 172.21.15.61:46272) 2022-07-20T03:16:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b43fff700 1 -- 172.21.15.61:0/3731212477 learned_addr learned my addr 172.21.15.61:0/3731212477 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b48ad6700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 0x7f4b440ab0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4111299457 0 0) 0x7f4b440b1a00 con 0x7f4b440616b0 2022-07-20T03:16:05.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b24003260 con 0x7f4b440616b0 2022-07-20T03:16:05.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2284655362 0 0) 0x7f4b4405ff30 con 0x7f4b440b1620 2022-07-20T03:16:05.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4b440b1a00 con 0x7f4b440b1620 2022-07-20T03:16:05.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.1 v1:172.21.15.117:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 825583418 0 0) 0x7f4b24003260 con 0x7f4b440616b0 2022-07-20T03:16:05.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 msgr2=0x7f4b440ab0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:05.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 0x7f4b440ab0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 >> v1:172.21.15.61:6789/0 conn(0x7f4b440b1620 legacy=0x7f4b440acdf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 --> v1:172.21.15.117:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4405c0d0 con 0x7f4b440616b0 2022-07-20T03:16:05.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.1 v1:172.21.15.117:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f4b3c0019e0 con 0x7f4b440616b0 2022-07-20T03:16:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 >> v1:172.21.15.117:6789/0 conn(0x7f4b440616b0 legacy=0x7f4b440aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 0x7f4b240038e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b24003df0 0x7f4b24004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b240049f0 0x7f4b24004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440ad500 con 0x7f4b240049f0 2022-07-20T03:16:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2400b7b0 con 0x7f4b24003df0 2022-07-20T03:16:05.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b2400b970 con 0x7f4b44061b40 2022-07-20T03:16:05.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b48ad6700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b240049f0 0x7f4b24004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b43fff700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 0x7f4b240038e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b437fe700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b24003df0 0x7f4b24004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b48ad6700 1 -- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 msgr2=0x7f4b240038e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:05.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b48ad6700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b44061b40 0x7f4b240038e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b48ad6700 1 -- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b24003df0 msgr2=0x7f4b24004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.972+0000 7f4b48ad6700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b24003df0 0x7f4b24004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b48ad6700 1 -- 172.21.15.61:0/3731212477 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b24003260 con 0x7f4b240049f0 2022-07-20T03:16:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b48ad6700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b240049f0 0x7f4b24004e70 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f4b3400c000 tx=0x7f4b34006df0).ready entity=mon.0 client_cookie=dab8418402ccecff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b3400e830 con 0x7f4b240049f0 2022-07-20T03:16:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3400e990 con 0x7f4b240049f0 2022-07-20T03:16:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b42ffd700 1 -- 172.21.15.61:0/3731212477 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b3400ecf0 con 0x7f4b240049f0 2022-07-20T03:16:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 -- 172.21.15.61:0/3731212477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b240049f0 msgr2=0x7f4b24004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 --2- 172.21.15.61:0/3731212477 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b240049f0 0x7f4b24004e70 secure :-1 s=READY pgs=1170 cs=0 l=1 rev1=1 rx=0x7f4b3400c000 tx=0x7f4b34006df0).stop 2022-07-20T03:16:05.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b48ad6700 1 -- 172.21.15.61:0/3731212477 reap_dead start 2022-07-20T03:16:05.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 -- 172.21.15.61:0/3731212477 shutdown_connections 2022-07-20T03:16:05.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 -- 172.21.15.61:0/3731212477 >> 172.21.15.61:0/3731212477 conn(0x7f4b440b23d0 msgr2=0x7f4b440b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:05.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 -- 172.21.15.61:0/3731212477 shutdown_connections 2022-07-20T03:16:05.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 -- 172.21.15.61:0/3731212477 wait complete. 2022-07-20T03:16:05.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 Processor -- start 2022-07-20T03:16:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.976+0000 7f4b4a539700 1 -- start start 2022-07-20T03:16:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b240049f0 0x7f4b44159910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 0x7f4b440635c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b441487d0 0x7f4b44148c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b44149140 con 0x7f4b44099c30 2022-07-20T03:16:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440aad50 con 0x7f4b240049f0 2022-07-20T03:16:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b440aaff0 con 0x7f4b441487d0 2022-07-20T03:16:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 0x7f4b440635c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 0x7f4b440635c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49686/0 (socket says 172.21.15.61:49686) 2022-07-20T03:16:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b48ad6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b441487d0 0x7f4b44148c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 -- 172.21.15.61:0/1229382324 learned_addr learned my addr 172.21.15.61:0/1229382324 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:05.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b43fff700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b240049f0 0x7f4b44159910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 -- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b441487d0 msgr2=0x7f4b44148c50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b441487d0 0x7f4b44148c50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 -- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b240049f0 msgr2=0x7f4b44159910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b240049f0 0x7f4b44159910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 -- 172.21.15.61:0/1229382324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b4409acb0 con 0x7f4b44099c30 2022-07-20T03:16:05.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b437fe700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 0x7f4b440635c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f4b3c00d000 tx=0x7f4b4405fe20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:05.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b3c00d8d0 con 0x7f4b44099c30 2022-07-20T03:16:05.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b3c00da30 con 0x7f4b44099c30 2022-07-20T03:16:05.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b440992e0 con 0x7f4b44099c30 2022-07-20T03:16:05.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b3c00dd90 con 0x7f4b44099c30 2022-07-20T03:16:05.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b44064f40 con 0x7f4b44099c30 2022-07-20T03:16:05.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.980+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b440ab1b0 con 0x7f4b44099c30 2022-07-20T03:16:05.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.984+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4b3c01b030 con 0x7f4b44099c30 2022-07-20T03:16:05.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.984+0000 7f4b417fa700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b28053af0 0x7f4b28055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:05.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.984+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4b3c06c0b0 con 0x7f4b44099c30 2022-07-20T03:16:05.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.984+0000 7f4b43fff700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b28053af0 0x7f4b28055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:05.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.984+0000 7f4b43fff700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b28053af0 0x7f4b28055fb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f4b2c007000 tx=0x7f4b2c007ad0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:05.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:04.988+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b3c043020 con 0x7f4b44099c30 2022-07-20T03:16:05.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.152+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4b440aab70 con 0x7f4b44099c30 2022-07-20T03:16:05.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.152+0000 7f4b417fa700 1 -- 172.21.15.61:0/1229382324 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f4b3c046640 con 0x7f4b44099c30 2022-07-20T03:16:05.164 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:05.164 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7662,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3960371515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3960371515},{"type":"v1","addr":"172.21.15.61:6833","nonce":3960371515}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:59.567631+0000","modified":"2022-07-20T03:16:01.250517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"b","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/2884785802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2884785802},{"type":"v1","addr":"172.21.15.117:6835","nonce":2884785802}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T03:16:05.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b28053af0 msgr2=0x7f4b28055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b28053af0 0x7f4b28055fb0 secure :-1 s=READY pgs=563 cs=0 l=1 rev1=1 rx=0x7f4b2c007000 tx=0x7f4b2c007ad0).stop 2022-07-20T03:16:05.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 msgr2=0x7f4b440635c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:05.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 0x7f4b440635c0 secure :-1 s=READY pgs=1171 cs=0 l=1 rev1=1 rx=0x7f4b3c00d000 tx=0x7f4b4405fe20).stop 2022-07-20T03:16:05.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 shutdown_connections 2022-07-20T03:16:05.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b28053af0 0x7f4b28055fb0 unknown :-1 s=CLOSED pgs=563 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b441487d0 0x7f4b44148c50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b44099c30 0x7f4b440635c0 unknown :-1 s=CLOSED pgs=1171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 --2- 172.21.15.61:0/1229382324 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b240049f0 0x7f4b44159910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:05.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 >> 172.21.15.61:0/1229382324 conn(0x7f4b440b23d0 msgr2=0x7f4b440b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:05.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 shutdown_connections 2022-07-20T03:16:05.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:05.156+0000 7f4b4a539700 1 -- 172.21.15.61:0/1229382324 wait complete. 2022-07-20T03:16:05.174 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-20T03:16:05.377 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:59.567631+0000', 'modified': '2022-07-20T03:16:01.250517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7668}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7668': {'gid': 7668, 'name': 'b', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/2884785802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 2884785802}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 2884785802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:05.387 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:05.387 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:16:05.502 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7668,"want_state":"up:active","state":"up:active","rank_uptime":5.9217971739999999,"mdsmap_epoch":62,"osdmap_epoch":111,"osdmap_epoch_barrier":111,"uptime":10.474264136} 2022-07-20T03:16:05.504 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7668, "mdsmap_epoch": 62, "osdmap_epoch": 111, "osdmap_epoch_barrier": 111, "rank_uptime": 5.921797174, "state": "up:active", "uptime": 10.474264136, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:05.505 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:16:05.505 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:16:05.506 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:16:05.507 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:05.509 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:05.521 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:16:05.581 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:05.582 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:05.597 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:05.601 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:05.603 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:05.603 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:16:05.617 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:16:05.618 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:05.660 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:16:05.678 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:05.679 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:05.735 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:05.737 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:05.742 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:05.743 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:16:05.744 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23330]: starting ceph client 2022-07-20T03:16:05.745 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:16:05.733+0000 7f33894c5080 -1 init, newargv = 0x557ab50ef230 newargc=9 2022-07-20T03:16:05.765 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23330]: starting fuse 2022-07-20T03:16:05.773 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:16:05.774 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:16:05.775 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:05.783 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:16:05.785 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:16:05.791 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:16:05.792 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:16:05.792 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:16:05.793 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:05.798 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:16:05.800 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:16:05.800 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:16:05.802 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:05.803 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:16:05.805 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:16:05.805 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:16:05.806 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:16:05.807 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:05.808 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:16:05.808 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:16:05.809 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:16:05.810 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:16:05.811 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:16:05.812 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:05.813 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:16:05.814 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:16:05.869 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23330.asok 2022-07-20T03:16:05.874 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23330.asok 2022-07-20T03:16:05.875 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:05.875 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23330.asok status 2022-07-20T03:16:05.984 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:16:05.984 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:16:05.985 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:16:05.985 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:16:05.986 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:16:05.986 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:16:05.986 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:16:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23330", 2022-07-20T03:16:05.987 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:16:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:16:05.988 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:16:05.989 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 7848, 2022-07-20T03:16:05.989 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:16:05.989 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:16:05.990 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:16:05.990 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 7848 2022-07-20T03:16:05.991 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:05.992 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:16:05.993 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:16:05.993 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:16:05.994 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2883630570 2022-07-20T03:16:05.994 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:16:05.995 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:05.995 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:16:05.996 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:16:05.996 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:16:05.997 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2883630570 2022-07-20T03:16:05.997 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:05.998 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.7848 172.21.15.124:0/2883630570", 2022-07-20T03:16:05.998 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/2883630570", 2022-07-20T03:16:05.999 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:16:05.999 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 62, 2022-07-20T03:16:06.000 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 111, 2022-07-20T03:16:06.000 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:16:06.000 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:16:06.001 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:16:06.001 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:16:06.004 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:06.005 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:06.006 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:06.016 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:16:06.019 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:06.019 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:06.020 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:06.040 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:06.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 Processor -- start 2022-07-20T03:16:06.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 -- start start 2022-07-20T03:16:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73980b9ef0 0x7f73980b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f73980b2960 0x7f7398142210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73980ba2d0 con 0x7f739805d160 2022-07-20T03:16:06.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f73980b4560 con 0x7f73980b2960 2022-07-20T03:16:06.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f73a0108700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980b4800 con 0x7f73980b9ef0 2022-07-20T03:16:06.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f73980b2960 0x7f7398142210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f73980b2960 0x7f7398142210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49696/0 (socket says 172.21.15.61:49696) 2022-07-20T03:16:06.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 -- 172.21.15.61:0/2453525795 learned_addr learned my addr 172.21.15.61:0/2453525795 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:06.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739d6a3700 1 --1- 172.21.15.61:0/2453525795 >> v1:172.21.15.61:6789/0 conn(0x7f739805d160 0x7f73980b2250 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46296/0 (socket says 172.21.15.61:46296) 2022-07-20T03:16:06.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739dea4700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73980b9ef0 0x7f73980b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 -- 172.21.15.61:0/2453525795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73980b9ef0 msgr2=0x7f73980b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:06.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73980b9ef0 0x7f73980b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 -- 172.21.15.61:0/2453525795 >> v1:172.21.15.61:6789/0 conn(0x7f739805d160 legacy=0x7f73980b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739cea2700 1 -- 172.21.15.61:0/2453525795 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 163855120 0 0) 0x7f73980ba2d0 con 0x7f739805d160 2022-07-20T03:16:06.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 -- 172.21.15.61:0/2453525795 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73980b1db0 con 0x7f73980b2960 2022-07-20T03:16:06.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739dea4700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73980b9ef0 0x7f73980b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:06.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739e6a5700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f7398142210 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f7388002f70 tx=0x7f7388006eb0).ready entity=mon.0 client_cookie=fe90db1b36023fb5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739cea2700 1 -- 172.21.15.61:0/2453525795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f738800d570 con 0x7f73980b2960 2022-07-20T03:16:06.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739cea2700 1 -- 172.21.15.61:0/2453525795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7388000e90 con 0x7f73980b2960 2022-07-20T03:16:06.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.164+0000 7f739cea2700 1 -- 172.21.15.61:0/2453525795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f738800d8c0 con 0x7f73980b2960 2022-07-20T03:16:06.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- 172.21.15.61:0/2453525795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 msgr2=0x7f7398142210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f7398142210 secure :-1 s=READY pgs=1175 cs=0 l=1 rev1=1 rx=0x7f7388002f70 tx=0x7f7388006eb0).stop 2022-07-20T03:16:06.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- 172.21.15.61:0/2453525795 shutdown_connections 2022-07-20T03:16:06.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f7398142210 unknown :-1 s=CLOSED pgs=1175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 --2- 172.21.15.61:0/2453525795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73980b9ef0 0x7f73980b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- 172.21.15.61:0/2453525795 >> 172.21.15.61:0/2453525795 conn(0x7f73980679d0 msgr2=0x7f73980b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:06.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- 172.21.15.61:0/2453525795 shutdown_connections 2022-07-20T03:16:06.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- 172.21.15.61:0/2453525795 wait complete. 2022-07-20T03:16:06.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 Processor -- start 2022-07-20T03:16:06.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- start start 2022-07-20T03:16:06.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f739805d160 0x7f73980aeb90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f73981423c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73980b9ef0 0x7f7398151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73980ae450 con 0x7f73980b2960 2022-07-20T03:16:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f73980ae6f0 con 0x7f73980b9ef0 2022-07-20T03:16:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f73a0108700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f73980af870 con 0x7f739805d160 2022-07-20T03:16:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f739d6a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f73981423c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f739d6a3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f73981423c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49702/0 (socket says 172.21.15.61:49702) 2022-07-20T03:16:06.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f739dea4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f739805d160 0x7f73980aeb90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f739d6a3700 1 -- 172.21.15.61:0/1241211360 learned_addr learned my addr 172.21.15.61:0/1241211360 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:06.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.168+0000 7f739e6a5700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73980b9ef0 0x7f7398151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739d6a3700 1 -- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f739805d160 msgr2=0x7f73980aeb90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739d6a3700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f739805d160 0x7f73980aeb90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739d6a3700 1 -- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73980b9ef0 msgr2=0x7f7398151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739d6a3700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73980b9ef0 0x7f7398151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739d6a3700 1 -- 172.21.15.61:0/1241211360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7390009470 con 0x7f73980b2960 2022-07-20T03:16:06.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739dea4700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f739805d160 0x7f73980aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:06.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f739d6a3700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f73981423c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f7380000bc0 tx=0x7f7380007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f73800070d0 con 0x7f73980b2960 2022-07-20T03:16:06.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7380007230 con 0x7f73980b2960 2022-07-20T03:16:06.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f738800b1e0 con 0x7f73980b2960 2022-07-20T03:16:06.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7380005740 con 0x7f73980b2960 2022-07-20T03:16:06.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73980af2a0 con 0x7f73980b2960 2022-07-20T03:16:06.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73980ae2b0 con 0x7f73980b2960 2022-07-20T03:16:06.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.172+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f738001a060 con 0x7f73980b2960 2022-07-20T03:16:06.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.176+0000 7f7396ffd700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f737c053a60 0x7f737c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.176+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f738006b160 con 0x7f73980b2960 2022-07-20T03:16:06.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.176+0000 7f739dea4700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f737c053a60 0x7f737c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.176+0000 7f739dea4700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f737c053a60 0x7f737c055f20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f739000b000 tx=0x7f73900091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.180+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f73980ae2b0 con 0x7f73980b2960 2022-07-20T03:16:06.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.308+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f73980ae2b0 con 0x7f73980b2960 2022-07-20T03:16:06.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f7396ffd700 1 -- 172.21.15.61:0/1241211360 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v111) v1 ==== 74+0+16099 (secure 0 0 0) 0x7f73980ae2b0 con 0x7f73980b2960 2022-07-20T03:16:06.351 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:06.352 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":111,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:00.570268+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":22,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":21,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:15:58.052517+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"110","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":22,"pool_name":"cephfs_data","create_time":"2022-07-20T03:15:58.563360+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"111","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":99,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":109,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":108,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:06.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f737c053a60 msgr2=0x7f737c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f737c053a60 0x7f737c055f20 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7f739000b000 tx=0x7f73900091d0).stop 2022-07-20T03:16:06.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 msgr2=0x7f73981423c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f73981423c0 secure :-1 s=READY pgs=1176 cs=0 l=1 rev1=1 rx=0x7f7380000bc0 tx=0x7f7380007a60).stop 2022-07-20T03:16:06.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 shutdown_connections 2022-07-20T03:16:06.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f737c053a60 0x7f737c055f20 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73980b9ef0 0x7f7398151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f73980b2960 0x7f73981423c0 unknown :-1 s=CLOSED pgs=1176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 --2- 172.21.15.61:0/1241211360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f739805d160 0x7f73980aeb90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 >> 172.21.15.61:0/1241211360 conn(0x7f73980679d0 msgr2=0x7f73980b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:06.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 shutdown_connections 2022-07-20T03:16:06.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.312+0000 7f73a0108700 1 -- 172.21.15.61:0/1241211360 wait complete. 2022-07-20T03:16:06.357 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 21, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:15:58.052517+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '110', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 22, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:15:58.563360+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '111', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:16:06.358 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:16:06.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.476+0000 7fa5164c0700 1 Processor -- start 2022-07-20T03:16:06.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.476+0000 7fa5164c0700 1 -- start start 2022-07-20T03:16:06.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 0x7fa5100b1fa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa5101460c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa51005cc50 0x7fa51005d0e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51005bd90 con 0x7fa5100b7bb0 2022-07-20T03:16:06.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa51005c820 con 0x7fa51005cc50 2022-07-20T03:16:06.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100b7f90 con 0x7fa5100b3770 2022-07-20T03:16:06.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5154be700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 0x7fa5100b1fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa51005cc50 0x7fa51005d0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5154be700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 0x7fa5100b1fa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49710/0 (socket says 172.21.15.61:49710) 2022-07-20T03:16:06.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5154be700 1 -- 172.21.15.61:0/1783375239 learned_addr learned my addr 172.21.15.61:0/1783375239 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:06.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa514cbd700 1 --2- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa5101460c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 -- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 msgr2=0x7fa5101460c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:06.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 --2- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa5101460c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 -- 172.21.15.61:0/1783375239 >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 msgr2=0x7fa5100b1fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 --2- 172.21.15.61:0/1783375239 >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 0x7fa5100b1fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 -- 172.21.15.61:0/1783375239 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51005bfc0 con 0x7fa51005cc50 2022-07-20T03:16:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5154be700 1 --2- 172.21.15.61:0/1783375239 >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 0x7fa5100b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa515cbf700 1 --2- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa51005cc50 0x7fa51005d0e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fa5080032f0 tx=0x7fa5080091d0).ready entity=mon.1 client_cookie=51c8992aca5ea1eb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa503fff700 1 -- 172.21.15.61:0/1783375239 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa50800b600 con 0x7fa51005cc50 2022-07-20T03:16:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa503fff700 1 -- 172.21.15.61:0/1783375239 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50800b760 con 0x7fa51005cc50 2022-07-20T03:16:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa503fff700 1 -- 172.21.15.61:0/1783375239 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa50800b8d0 con 0x7fa51005cc50 2022-07-20T03:16:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa51005cc50 msgr2=0x7fa51005d0e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa51005cc50 0x7fa51005d0e0 secure :-1 s=READY pgs=531 cs=0 l=1 rev1=1 rx=0x7fa5080032f0 tx=0x7fa5080091d0).stop 2022-07-20T03:16:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- 172.21.15.61:0/1783375239 shutdown_connections 2022-07-20T03:16:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa51005cc50 0x7fa51005d0e0 unknown :-1 s=CLOSED pgs=531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1783375239 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa5101460c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1783375239 >> v2:172.21.15.61:3300/0 conn(0x7fa5100b7bb0 0x7fa5100b1fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- 172.21.15.61:0/1783375239 >> 172.21.15.61:0/1783375239 conn(0x7fa510065720 msgr2=0x7fa5100b7160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- 172.21.15.61:0/1783375239 shutdown_connections 2022-07-20T03:16:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.480+0000 7fa5164c0700 1 -- 172.21.15.61:0/1783375239 wait complete. 2022-07-20T03:16:06.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 Processor -- start 2022-07-20T03:16:06.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 -- start start 2022-07-20T03:16:06.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa51005cc50 0x7fa510151f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa510066fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 0x7fa5100b08d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5100ae5b0 con 0x7fa51005cc50 2022-07-20T03:16:06.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa510144910 con 0x7fa5100b7bb0 2022-07-20T03:16:06.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa510144bb0 con 0x7fa5100b3770 2022-07-20T03:16:06.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5154be700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa51005cc50 0x7fa510151f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 0x7fa5100b08d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5154be700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa51005cc50 0x7fa510151f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49714/0 (socket says 172.21.15.61:49714) 2022-07-20T03:16:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 0x7fa5100b08d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59398/0 (socket says 172.21.15.61:59398) 2022-07-20T03:16:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5154be700 1 -- 172.21.15.61:0/1176882768 learned_addr learned my addr 172.21.15.61:0/1176882768 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa514cbd700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa510066fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 -- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 msgr2=0x7fa510066fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa510066fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 -- 172.21.15.61:0/1176882768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa51005cc50 msgr2=0x7fa510151f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa51005cc50 0x7fa510151f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 -- 172.21.15.61:0/1176882768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa50400b1e0 con 0x7fa5100b7bb0 2022-07-20T03:16:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa515cbf700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 0x7fa5100b08d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fa5080032f0 tx=0x7fa508016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa50800b600 con 0x7fa5100b7bb0 2022-07-20T03:16:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa50800be60 con 0x7fa5100b7bb0 2022-07-20T03:16:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa508009470 con 0x7fa5100b7bb0 2022-07-20T03:16:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa50801f5e0 con 0x7fa5100b7bb0 2022-07-20T03:16:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.484+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa510145180 con 0x7fa5100b7bb0 2022-07-20T03:16:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.488+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa51005c260 con 0x7fa5100b7bb0 2022-07-20T03:16:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.488+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa508026030 con 0x7fa5100b7bb0 2022-07-20T03:16:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.488+0000 7fa5027fc700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4f0053a40 0x7fa4f0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.488+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa50806c780 con 0x7fa5100b7bb0 2022-07-20T03:16:06.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.488+0000 7fa5154be700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4f0053a40 0x7fa4f0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.488+0000 7fa5154be700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4f0053a40 0x7fa4f0055f00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fa50400b000 tx=0x7fa504006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.496+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa508043680 con 0x7fa5100b7bb0 2022-07-20T03:16:06.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.624+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa5100574e0 con 0x7fa5100b7bb0 2022-07-20T03:16:06.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5027fc700 1 -- 172.21.15.61:0/1176882768 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v111) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa508024020 con 0x7fa5100b7bb0 2022-07-20T03:16:06.631 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:16:06.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4f0053a40 msgr2=0x7fa4f0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4f0053a40 0x7fa4f0055f00 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fa50400b000 tx=0x7fa504006eb0).stop 2022-07-20T03:16:06.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 msgr2=0x7fa5100b08d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 0x7fa5100b08d0 secure :-1 s=READY pgs=532 cs=0 l=1 rev1=1 rx=0x7fa5080032f0 tx=0x7fa508016000).stop 2022-07-20T03:16:06.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 shutdown_connections 2022-07-20T03:16:06.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4f0053a40 0x7fa4f0055f00 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa5100b7bb0 0x7fa5100b08d0 unknown :-1 s=CLOSED pgs=532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa5100b3770 0x7fa510066fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 --2- 172.21.15.61:0/1176882768 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa51005cc50 0x7fa510151f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 >> 172.21.15.61:0/1176882768 conn(0x7fa510065720 msgr2=0x7fa5100b6370 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 shutdown_connections 2022-07-20T03:16:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.628+0000 7fa5164c0700 1 -- 172.21.15.61:0/1176882768 wait complete. 2022-07-20T03:16:06.644 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:16:06.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 Processor -- start 2022-07-20T03:16:06.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 -- start start 2022-07-20T03:16:06.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f700988b0 0x7f9f701422c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9f701427b0 0x7f9f70060170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f700afdb0 con 0x7f9f701462a0 2022-07-20T03:16:06.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f700b0050 con 0x7f9f700988b0 2022-07-20T03:16:06.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f75aed700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f7005fcd0 con 0x7f9f701427b0 2022-07-20T03:16:06.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9f700988b0 0x7f9f701422c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6f7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9f701462a0 0x7f9f70146680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46320/0 (socket says 172.21.15.61:46320) 2022-07-20T03:16:06.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6f7fe700 1 -- 172.21.15.61:0/3360358581 learned_addr learned my addr 172.21.15.61:0/3360358581 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:06.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6ffff700 1 --2- 172.21.15.61:0/3360358581 >> v2:172.21.15.117:3300/0 conn(0x7f9f701427b0 0x7f9f70060170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6e7fc700 1 -- 172.21.15.61:0/3360358581 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1362064690 0 0) 0x7f9f700afdb0 con 0x7f9f701462a0 2022-07-20T03:16:06.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6e7fc700 1 -- 172.21.15.61:0/3360358581 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f5c003260 con 0x7f9f701462a0 2022-07-20T03:16:06.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6effd700 1 -- 172.21.15.61:0/3360358581 >> v2:172.21.15.117:3300/0 conn(0x7f9f701427b0 msgr2=0x7f9f70060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:06.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6effd700 1 --2- 172.21.15.61:0/3360358581 >> v2:172.21.15.117:3300/0 conn(0x7f9f701427b0 0x7f9f70060170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6effd700 1 -- 172.21.15.61:0/3360358581 >> v1:172.21.15.61:6789/0 conn(0x7f9f701462a0 legacy=0x7f9f70146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6effd700 1 -- 172.21.15.61:0/3360358581 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f700afae0 con 0x7f9f700988b0 2022-07-20T03:16:06.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.752+0000 7f9f6effd700 1 --2- 172.21.15.61:0/3360358581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f700988b0 0x7f9f701422c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f9f64002f70 tx=0x7f9f64006df0).ready entity=mon.0 client_cookie=26986c9627a0d03f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f6e7fc700 1 -- 172.21.15.61:0/3360358581 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f6400d4f0 con 0x7f9f700988b0 2022-07-20T03:16:06.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f6e7fc700 1 -- 172.21.15.61:0/3360358581 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f64000e90 con 0x7f9f700988b0 2022-07-20T03:16:06.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f6e7fc700 1 -- 172.21.15.61:0/3360358581 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f6400d840 con 0x7f9f700988b0 2022-07-20T03:16:06.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- 172.21.15.61:0/3360358581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f700988b0 msgr2=0x7f9f701422c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 --2- 172.21.15.61:0/3360358581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f700988b0 0x7f9f701422c0 secure :-1 s=READY pgs=1178 cs=0 l=1 rev1=1 rx=0x7f9f64002f70 tx=0x7f9f64006df0).stop 2022-07-20T03:16:06.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- 172.21.15.61:0/3360358581 shutdown_connections 2022-07-20T03:16:06.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 --2- 172.21.15.61:0/3360358581 >> v2:172.21.15.117:3300/0 conn(0x7f9f701427b0 0x7f9f70060170 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 --2- 172.21.15.61:0/3360358581 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f700988b0 0x7f9f701422c0 unknown :-1 s=CLOSED pgs=1178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- 172.21.15.61:0/3360358581 >> 172.21.15.61:0/3360358581 conn(0x7f9f700b5f00 msgr2=0x7f9f700b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:06.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- 172.21.15.61:0/3360358581 shutdown_connections 2022-07-20T03:16:06.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- 172.21.15.61:0/3360358581 wait complete. 2022-07-20T03:16:06.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 Processor -- start 2022-07-20T03:16:06.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- start start 2022-07-20T03:16:06.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f700988b0 0x7f9f70140650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 0x7f9f70152aa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f701462a0 0x7f9f700b2840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f7013f450 con 0x7f9f701427b0 2022-07-20T03:16:06.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f701520c0 con 0x7f9f701462a0 2022-07-20T03:16:06.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f75aed700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f70152360 con 0x7f9f700988b0 2022-07-20T03:16:06.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f6effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 0x7f9f70152aa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f6effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 0x7f9f70152aa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49730/0 (socket says 172.21.15.61:49730) 2022-07-20T03:16:06.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.756+0000 7f9f6effd700 1 -- 172.21.15.61:0/2716413009 learned_addr learned my addr 172.21.15.61:0/2716413009 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:06.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6ffff700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f701462a0 0x7f9f700b2840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6f7fe700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f700988b0 0x7f9f70140650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6effd700 1 -- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f700988b0 msgr2=0x7f9f70140650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6effd700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f700988b0 0x7f9f70140650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6effd700 1 -- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f701462a0 msgr2=0x7f9f700b2840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6effd700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f701462a0 0x7f9f700b2840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6effd700 1 -- 172.21.15.61:0/2716413009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f6400b180 con 0x7f9f701427b0 2022-07-20T03:16:06.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6effd700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 0x7f9f70152aa0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f9f64002f70 tx=0x7f9f64014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f64016070 con 0x7f9f701427b0 2022-07-20T03:16:06.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f640138d0 con 0x7f9f701427b0 2022-07-20T03:16:06.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f70152600 con 0x7f9f701427b0 2022-07-20T03:16:06.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f64013c30 con 0x7f9f701427b0 2022-07-20T03:16:06.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f700b0ee0 con 0x7f9f701427b0 2022-07-20T03:16:06.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f7009d450 con 0x7f9f701427b0 2022-07-20T03:16:06.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f64025030 con 0x7f9f701427b0 2022-07-20T03:16:06.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6cff9700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f50053ad0 0x7f9f50055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:06.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f6406cc70 con 0x7f9f701427b0 2022-07-20T03:16:06.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.760+0000 7f9f6f7fe700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f50053ad0 0x7f9f50055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:06.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.764+0000 7f9f6f7fe700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f50053ad0 0x7f9f50055f90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f9f6000b000 tx=0x7f9f60009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:06.790 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:06.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.768+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f7009d450 con 0x7f9f701427b0 2022-07-20T03:16:06.793 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:06.796 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:06.799 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:06.803 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:06.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9f7009d450 con 0x7f9f701427b0 2022-07-20T03:16:06.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f6cff9700 1 -- 172.21.15.61:0/2716413009 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v111) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9f7009d450 con 0x7f9f701427b0 2022-07-20T03:16:06.896 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:06.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f50053ad0 msgr2=0x7f9f50055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f50053ad0 0x7f9f50055f90 secure :-1 s=READY pgs=566 cs=0 l=1 rev1=1 rx=0x7f9f6000b000 tx=0x7f9f60009040).stop 2022-07-20T03:16:06.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 msgr2=0x7f9f70152aa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:06.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 0x7f9f70152aa0 secure :-1 s=READY pgs=1179 cs=0 l=1 rev1=1 rx=0x7f9f64002f70 tx=0x7f9f64014000).stop 2022-07-20T03:16:06.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 shutdown_connections 2022-07-20T03:16:06.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f50053ad0 0x7f9f50055f90 unknown :-1 s=CLOSED pgs=566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f701462a0 0x7f9f700b2840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f701427b0 0x7f9f70152aa0 unknown :-1 s=CLOSED pgs=1179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 --2- 172.21.15.61:0/2716413009 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f700988b0 0x7f9f70140650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 >> 172.21.15.61:0/2716413009 conn(0x7f9f700b5f00 msgr2=0x7f9f70141770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:06.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 shutdown_connections 2022-07-20T03:16:06.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:06.892+0000 7f9f75aed700 1 -- 172.21.15.61:0/2716413009 wait complete. 2022-07-20T03:16:06.917 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:16:07.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 Processor -- start 2022-07-20T03:16:07.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 -- start start 2022-07-20T03:16:07.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5900b9da0 0x7fb5901468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb5901419a0 0x7fb590141e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5900ba180 con 0x7fb5900b3cf0 2022-07-20T03:16:07.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5900ad5b0 con 0x7fb5900b9da0 2022-07-20T03:16:07.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb596274700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5901461b0 con 0x7fb5901419a0 2022-07-20T03:16:07.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5900b9da0 0x7fb5901468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5900b9da0 0x7fb5901468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49736/0 (socket says 172.21.15.61:49736) 2022-07-20T03:16:07.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 -- 172.21.15.61:0/138797463 learned_addr learned my addr 172.21.15.61:0/138797463 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:07.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595a73700 1 --2- 172.21.15.61:0/138797463 >> v2:172.21.15.117:3300/0 conn(0x7fb5901419a0 0x7fb590141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 -- 172.21.15.61:0/138797463 >> v2:172.21.15.117:3300/0 conn(0x7fb5901419a0 msgr2=0x7fb590141e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:07.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 --2- 172.21.15.61:0/138797463 >> v2:172.21.15.117:3300/0 conn(0x7fb5901419a0 0x7fb590141e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 -- 172.21.15.61:0/138797463 >> v1:172.21.15.117:6789/0 conn(0x7fb5900b3cf0 legacy=0x7fb590142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 -- 172.21.15.61:0/138797463 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb590146450 con 0x7fb5900b9da0 2022-07-20T03:16:07.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb595272700 1 --2- 172.21.15.61:0/138797463 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb5901468f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7fb58c002f70 tx=0x7fb58c006eb0).ready entity=mon.0 client_cookie=e04d60326e9b1caa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb587fff700 1 -- 172.21.15.61:0/138797463 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58c00d610 con 0x7fb5900b9da0 2022-07-20T03:16:07.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb587fff700 1 -- 172.21.15.61:0/138797463 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb58c000e90 con 0x7fb5900b9da0 2022-07-20T03:16:07.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.028+0000 7fb587fff700 1 -- 172.21.15.61:0/138797463 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb58c00d9a0 con 0x7fb5900b9da0 2022-07-20T03:16:07.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 -- 172.21.15.61:0/138797463 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 msgr2=0x7fb5901468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 --2- 172.21.15.61:0/138797463 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb5901468f0 secure :-1 s=READY pgs=1180 cs=0 l=1 rev1=1 rx=0x7fb58c002f70 tx=0x7fb58c006eb0).stop 2022-07-20T03:16:07.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 -- 172.21.15.61:0/138797463 shutdown_connections 2022-07-20T03:16:07.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 --2- 172.21.15.61:0/138797463 >> v2:172.21.15.117:3300/0 conn(0x7fb5901419a0 0x7fb590141e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 --2- 172.21.15.61:0/138797463 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb5901468f0 unknown :-1 s=CLOSED pgs=1180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 -- 172.21.15.61:0/138797463 >> 172.21.15.61:0/138797463 conn(0x7fb5900678c0 msgr2=0x7fb5900b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:07.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 -- 172.21.15.61:0/138797463 shutdown_connections 2022-07-20T03:16:07.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 -- 172.21.15.61:0/138797463 wait complete. 2022-07-20T03:16:07.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 Processor -- start 2022-07-20T03:16:07.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.032+0000 7fb596274700 1 -- start start 2022-07-20T03:16:07.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5900b3cf0 0x7fb590112f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb590113400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5901419a0 0x7fb590155420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5900b28b0 con 0x7fb5900b9da0 2022-07-20T03:16:07.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb590115ae0 con 0x7fb5901419a0 2022-07-20T03:16:07.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb590115d70 con 0x7fb5900b3cf0 2022-07-20T03:16:07.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb590113400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb590113400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49744/0 (socket says 172.21.15.61:49744) 2022-07-20T03:16:07.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 -- 172.21.15.61:0/2612062259 learned_addr learned my addr 172.21.15.61:0/2612062259 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:07.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb595272700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5900b3cf0 0x7fb590112f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb595a73700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5901419a0 0x7fb590155420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 -- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5900b3cf0 msgr2=0x7fb590112f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5900b3cf0 0x7fb590112f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 -- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5901419a0 msgr2=0x7fb590155420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5901419a0 0x7fb590155420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 -- 172.21.15.61:0/2612062259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb58c00b1e0 con 0x7fb5900b9da0 2022-07-20T03:16:07.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb595a73700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5901419a0 0x7fb590155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:07.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb594a71700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb590113400 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fb590066cd0 tx=0x7fb580007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb580012400 con 0x7fb5900b9da0 2022-07-20T03:16:07.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb580012560 con 0x7fb5900b9da0 2022-07-20T03:16:07.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5900b2be0 con 0x7fb5900b9da0 2022-07-20T03:16:07.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5800128c0 con 0x7fb5900b9da0 2022-07-20T03:16:07.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.036+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb590112ac0 con 0x7fb5900b9da0 2022-07-20T03:16:07.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.040+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5900574e0 con 0x7fb5900b9da0 2022-07-20T03:16:07.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.040+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb580019030 con 0x7fb5900b9da0 2022-07-20T03:16:07.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.040+0000 7fb5867fc700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb578053ad0 0x7fb578055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.040+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb58006b210 con 0x7fb5900b9da0 2022-07-20T03:16:07.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.040+0000 7fb595272700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb578053ad0 0x7fb578055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.040+0000 7fb595272700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb578053ad0 0x7fb578055f90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7fb58c01c000 tx=0x7fb58c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.048+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5800420d0 con 0x7fb5900b9da0 2022-07-20T03:16:07.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.172+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fb5900b5a70 con 0x7fb5900b9da0 2022-07-20T03:16:07.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb5867fc700 1 -- 172.21.15.61:0/2612062259 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v111) v1 ==== 104+0+10 (secure 0 0 0) 0x7fb5800456f0 con 0x7fb5900b9da0 2022-07-20T03:16:07.180 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:07.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb578053ad0 msgr2=0x7fb578055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb578053ad0 0x7fb578055f90 secure :-1 s=READY pgs=567 cs=0 l=1 rev1=1 rx=0x7fb58c01c000 tx=0x7fb58c015000).stop 2022-07-20T03:16:07.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 msgr2=0x7fb590113400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb590113400 secure :-1 s=READY pgs=1181 cs=0 l=1 rev1=1 rx=0x7fb590066cd0 tx=0x7fb580007c20).stop 2022-07-20T03:16:07.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 shutdown_connections 2022-07-20T03:16:07.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb578053ad0 0x7fb578055f90 unknown :-1 s=CLOSED pgs=567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5901419a0 0x7fb590155420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5900b9da0 0x7fb590113400 unknown :-1 s=CLOSED pgs=1181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 --2- 172.21.15.61:0/2612062259 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5900b3cf0 0x7fb590112f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 >> 172.21.15.61:0/2612062259 conn(0x7fb5900678c0 msgr2=0x7fb590140e90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:07.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 shutdown_connections 2022-07-20T03:16:07.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.176+0000 7fb596274700 1 -- 172.21.15.61:0/2612062259 wait complete. 2022-07-20T03:16:07.193 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:16:07.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.296+0000 7fdaef59e700 1 Processor -- start 2022-07-20T03:16:07.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.296+0000 7fdaef59e700 1 -- start start 2022-07-20T03:16:07.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaef59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdaf00b4b10 0x7fdaf00af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaef59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf00aabe0 con 0x7fdaf00b0350 2022-07-20T03:16:07.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaef59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdaf00b4ef0 con 0x7fdaf00b0890 2022-07-20T03:16:07.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaef59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdaf00aae10 con 0x7fdaf00b4b10 2022-07-20T03:16:07.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaeed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdaf00b0350 0x7fdaf013e880 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46352/0 (socket says 172.21.15.61:46352) 2022-07-20T03:16:07.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaeed9d700 1 -- 172.21.15.61:0/657731100 learned_addr learned my addr 172.21.15.61:0/657731100 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:07.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaee59c700 1 --2- 172.21.15.61:0/657731100 >> v2:172.21.15.117:3300/0 conn(0x7fdaf00b4b10 0x7fdaf00af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaedd9b700 1 -- 172.21.15.61:0/657731100 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1933922588 0 0) 0x7fdaf00aabe0 con 0x7fdaf00b0350 2022-07-20T03:16:07.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaedd9b700 1 -- 172.21.15.61:0/657731100 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdad0003260 con 0x7fdaf00b0350 2022-07-20T03:16:07.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaee59c700 1 -- 172.21.15.61:0/657731100 >> v1:172.21.15.117:6789/0 conn(0x7fdaf00b0890 legacy=0x7fdaf00ac1e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaee59c700 1 -- 172.21.15.61:0/657731100 >> v1:172.21.15.61:6789/0 conn(0x7fdaf00b0350 legacy=0x7fdaf013e880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaee59c700 1 -- 172.21.15.61:0/657731100 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdaf00ab0b0 con 0x7fdaf00b4b10 2022-07-20T03:16:07.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaee59c700 1 --2- 172.21.15.61:0/657731100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b4b10 0x7fdaf00af8b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fdae0002f70 tx=0x7fdae0006df0).ready entity=mon.1 client_cookie=ef4353821d357123 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaedd9b700 1 -- 172.21.15.61:0/657731100 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae000d610 con 0x7fdaf00b4b10 2022-07-20T03:16:07.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaedd9b700 1 -- 172.21.15.61:0/657731100 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae0000e90 con 0x7fdaf00b4b10 2022-07-20T03:16:07.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaedd9b700 1 -- 172.21.15.61:0/657731100 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae000d960 con 0x7fdaf00b4b10 2022-07-20T03:16:07.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaef59e700 1 -- 172.21.15.61:0/657731100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b4b10 msgr2=0x7fdaf00af8b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.300+0000 7fdaef59e700 1 --2- 172.21.15.61:0/657731100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b4b10 0x7fdaf00af8b0 secure :-1 s=READY pgs=535 cs=0 l=1 rev1=1 rx=0x7fdae0002f70 tx=0x7fdae0006df0).stop 2022-07-20T03:16:07.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 -- 172.21.15.61:0/657731100 shutdown_connections 2022-07-20T03:16:07.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 --2- 172.21.15.61:0/657731100 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b4b10 0x7fdaf00af8b0 unknown :-1 s=CLOSED pgs=535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 -- 172.21.15.61:0/657731100 >> 172.21.15.61:0/657731100 conn(0x7fdaf00632b0 msgr2=0x7fdaf0063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:07.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 -- 172.21.15.61:0/657731100 shutdown_connections 2022-07-20T03:16:07.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 -- 172.21.15.61:0/657731100 wait complete. 2022-07-20T03:16:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 Processor -- start 2022-07-20T03:16:07.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.304+0000 7fdaef59e700 1 -- start start 2022-07-20T03:16:07.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 0x7fdaf0064290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b0890 0x7fdaf0153060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdaf00b4b10 0x7fdaf0143e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf0064780 con 0x7fdaf00b0350 2022-07-20T03:16:07.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf0064c40 con 0x7fdaf00b0890 2022-07-20T03:16:07.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdaf00649a0 con 0x7fdaf00b4b10 2022-07-20T03:16:07.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 0x7fdaf0064290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 0x7fdaf0064290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49756/0 (socket says 172.21.15.61:49756) 2022-07-20T03:16:07.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 -- 172.21.15.61:0/1506214180 learned_addr learned my addr 172.21.15.61:0/1506214180 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:07.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdae7fff700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b0890 0x7fdaf0153060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaeed9d700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdaf00b4b10 0x7fdaf0143e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 -- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdaf00b4b10 msgr2=0x7fdaf0143e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdaf00b4b10 0x7fdaf0143e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 -- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b0890 msgr2=0x7fdaf0153060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b0890 0x7fdaf0153060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 -- 172.21.15.61:0/1506214180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdae000b180 con 0x7fdaf00b0350 2022-07-20T03:16:07.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdae7fff700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b0890 0x7fdaf0153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:07.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaee59c700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 0x7fdaf0064290 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7fdaf00648c0 tx=0x7fdae0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae0016070 con 0x7fdaf00b0350 2022-07-20T03:16:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdae000de90 con 0x7fdaf00b0350 2022-07-20T03:16:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdaf0141fb0 con 0x7fdaf00b0350 2022-07-20T03:16:07.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdae001d3f0 con 0x7fdaf00b0350 2022-07-20T03:16:07.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdaf0144370 con 0x7fdaf00b0350 2022-07-20T03:16:07.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.308+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdaf0141660 con 0x7fdaf00b0350 2022-07-20T03:16:07.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.312+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdae0024030 con 0x7fdaf00b0350 2022-07-20T03:16:07.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.312+0000 7fdae77fe700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdad4053ad0 0x7fdad4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.312+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fdae006cc70 con 0x7fdaf00b0350 2022-07-20T03:16:07.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.312+0000 7fdae7fff700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdad4053ad0 0x7fdad4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.312+0000 7fdae7fff700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdad4053ad0 0x7fdad4055f90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7fdad800b000 tx=0x7fdad8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.316+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdaf0141660 con 0x7fdaf00b0350 2022-07-20T03:16:07.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.444+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fdaf0141660 con 0x7fdaf00b0350 2022-07-20T03:16:07.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.444+0000 7fdae77fe700 1 -- 172.21.15.61:0/1506214180 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v111) v1 ==== 100+0+10 (secure 0 0 0) 0x7fdaf0141660 con 0x7fdaf00b0350 2022-07-20T03:16:07.450 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:07.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdad4053ad0 msgr2=0x7fdad4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdad4053ad0 0x7fdad4055f90 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7fdad800b000 tx=0x7fdad8009040).stop 2022-07-20T03:16:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 msgr2=0x7fdaf0064290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 0x7fdaf0064290 secure :-1 s=READY pgs=1183 cs=0 l=1 rev1=1 rx=0x7fdaf00648c0 tx=0x7fdae0014000).stop 2022-07-20T03:16:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 shutdown_connections 2022-07-20T03:16:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdad4053ad0 0x7fdad4055f90 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdaf00b0890 0x7fdaf0153060 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdaf00b4b10 0x7fdaf0143e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 --2- 172.21.15.61:0/1506214180 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdaf00b0350 0x7fdaf0064290 unknown :-1 s=CLOSED pgs=1183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 >> 172.21.15.61:0/1506214180 conn(0x7fdaf00632b0 msgr2=0x7fdaf00b60b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:07.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 shutdown_connections 2022-07-20T03:16:07.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.448+0000 7fdaef59e700 1 -- 172.21.15.61:0/1506214180 wait complete. 2022-07-20T03:16:07.464 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:16:07.465 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:07.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 Processor -- start 2022-07-20T03:16:07.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 -- start start 2022-07-20T03:16:07.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64140b5650 0x7f64140b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f641405bd90 0x7f6414142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64140aea30 con 0x7f64140b1200 2022-07-20T03:16:07.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6414142970 con 0x7f641405bd90 2022-07-20T03:16:07.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.568+0000 7f6419408700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64140aec60 con 0x7f64140b5650 2022-07-20T03:16:07.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f641405bd90 0x7f6414142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f641405bd90 0x7f6414142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49768/0 (socket says 172.21.15.61:49768) 2022-07-20T03:16:07.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 -- 172.21.15.61:0/1415305534 learned_addr learned my addr 172.21.15.61:0/1415305534 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:07.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6413fff700 1 --2- 172.21.15.61:0/1415305534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64140b5650 0x7f64140b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 -- 172.21.15.61:0/1415305534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64140b5650 msgr2=0x7f64140b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:07.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6412ffd700 1 -- 172.21.15.61:0/1415305534 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 929255871 0 0) 0x7f64140aea30 con 0x7f64140b1200 2022-07-20T03:16:07.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 --2- 172.21.15.61:0/1415305534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64140b5650 0x7f64140b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 -- 172.21.15.61:0/1415305534 >> v1:172.21.15.117:6789/0 conn(0x7f64140b1200 legacy=0x7f641405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 -- 172.21.15.61:0/1415305534 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64140af180 con 0x7f641405bd90 2022-07-20T03:16:07.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6418c07700 1 --2- 172.21.15.61:0/1415305534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f641405bd90 0x7f6414142d60 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f640c002f70 tx=0x7f640c006eb0).ready entity=mon.0 client_cookie=77143add45fa1eeb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6412ffd700 1 -- 172.21.15.61:0/1415305534 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f640c00d570 con 0x7f641405bd90 2022-07-20T03:16:07.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6412ffd700 1 -- 172.21.15.61:0/1415305534 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c000e90 con 0x7f641405bd90 2022-07-20T03:16:07.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6412ffd700 1 -- 172.21.15.61:0/1415305534 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f640c00d8c0 con 0x7f641405bd90 2022-07-20T03:16:07.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 -- 172.21.15.61:0/1415305534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f641405bd90 msgr2=0x7f6414142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 --2- 172.21.15.61:0/1415305534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f641405bd90 0x7f6414142d60 secure :-1 s=READY pgs=1184 cs=0 l=1 rev1=1 rx=0x7f640c002f70 tx=0x7f640c006eb0).stop 2022-07-20T03:16:07.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 -- 172.21.15.61:0/1415305534 shutdown_connections 2022-07-20T03:16:07.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 --2- 172.21.15.61:0/1415305534 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f641405bd90 0x7f6414142d60 unknown :-1 s=CLOSED pgs=1184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 --2- 172.21.15.61:0/1415305534 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64140b5650 0x7f64140b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 -- 172.21.15.61:0/1415305534 >> 172.21.15.61:0/1415305534 conn(0x7f64140634e0 msgr2=0x7f64140b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:07.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 -- 172.21.15.61:0/1415305534 shutdown_connections 2022-07-20T03:16:07.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.572+0000 7f6419408700 1 -- 172.21.15.61:0/1415305534 wait complete. 2022-07-20T03:16:07.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 Processor -- start 2022-07-20T03:16:07.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 -- start start 2022-07-20T03:16:07.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f641405bd90 0x7f6414155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64140b1200 0x7f6414144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 0x7f64141387a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6414155230 con 0x7f64140b5650 2022-07-20T03:16:07.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6414155460 con 0x7f64140b1200 2022-07-20T03:16:07.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6419408700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6414155700 con 0x7f641405bd90 2022-07-20T03:16:07.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6418c07700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 0x7f64141387a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6418c07700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 0x7f64141387a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49774/0 (socket says 172.21.15.61:49774) 2022-07-20T03:16:07.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.576+0000 7f6418c07700 1 -- 172.21.15.61:0/3197284512 learned_addr learned my addr 172.21.15.61:0/3197284512 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:07.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f64137fe700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64140b1200 0x7f6414144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6413fff700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f641405bd90 0x7f6414155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6418c07700 1 -- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f641405bd90 msgr2=0x7f6414155e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6418c07700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f641405bd90 0x7f6414155e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6418c07700 1 -- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64140b1200 msgr2=0x7f6414144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6418c07700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64140b1200 0x7f6414144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6418c07700 1 -- 172.21.15.61:0/3197284512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f640c00b1e0 con 0x7f64140b5650 2022-07-20T03:16:07.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6418c07700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 0x7f64141387a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f640c002f70 tx=0x7f640c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f640c011860 con 0x7f64140b5650 2022-07-20T03:16:07.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f640c0119c0 con 0x7f64140b5650 2022-07-20T03:16:07.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f640c011d20 con 0x7f64140b5650 2022-07-20T03:16:07.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64141559a0 con 0x7f64140b5650 2022-07-20T03:16:07.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6414145210 con 0x7f64140b5650 2022-07-20T03:16:07.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.580+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6414155b30 con 0x7f64140b5650 2022-07-20T03:16:07.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.584+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f640c01a040 con 0x7f64140b5650 2022-07-20T03:16:07.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.584+0000 7f64117fa700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63f80539b0 0x7f63f8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:07.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.584+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f640c06ccd0 con 0x7f64140b5650 2022-07-20T03:16:07.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.584+0000 7f6413fff700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63f80539b0 0x7f63f8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:07.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.584+0000 7f6413fff700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63f80539b0 0x7f63f8055e70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f640400b000 tx=0x7f6404006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:07.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.588+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f640c043c30 con 0x7f64140b5650 2022-07-20T03:16:07.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.748+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f64140574e0 con 0x7f64140b5650 2022-07-20T03:16:07.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f64117fa700 1 -- 172.21.15.61:0/3197284512 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 62 v62) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f640c023020 con 0x7f64140b5650 2022-07-20T03:16:07.756 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:07.757 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":62,"default_fscid":10,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7662,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/3960371515","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3960371515},{"type":"v1","addr":"172.21.15.61:6833","nonce":3960371515}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":59}],"filesystems":[{"mdsmap":{"epoch":62,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:15:59.567631+0000","modified":"2022-07-20T03:16:01.250517+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":7668},"failed":[],"damaged":[],"stopped":[],"info":{"gid_7668":{"gid":7668,"name":"b","rank":0,"incarnation":60,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/2884785802","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2884785802},{"type":"v1","addr":"172.21.15.117:6835","nonce":2884785802}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[22],"metadata_pool":21,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":10}]} 2022-07-20T03:16:07.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63f80539b0 msgr2=0x7f63f8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63f80539b0 0x7f63f8055e70 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7f640400b000 tx=0x7f6404006df0).stop 2022-07-20T03:16:07.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 msgr2=0x7f64141387a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 0x7f64141387a0 secure :-1 s=READY pgs=1185 cs=0 l=1 rev1=1 rx=0x7f640c002f70 tx=0x7f640c015000).stop 2022-07-20T03:16:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 shutdown_connections 2022-07-20T03:16:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f63f80539b0 0x7f63f8055e70 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64140b1200 0x7f6414144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64140b5650 0x7f64141387a0 unknown :-1 s=CLOSED pgs=1185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 --2- 172.21.15.61:0/3197284512 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f641405bd90 0x7f6414155e40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 >> 172.21.15.61:0/3197284512 conn(0x7f64140634e0 msgr2=0x7f64140b3920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:07.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 shutdown_connections 2022-07-20T03:16:07.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:07.752+0000 7f6419408700 1 -- 172.21.15.61:0/3197284512 wait complete. 2022-07-20T03:16:07.762 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 62 2022-07-20T03:16:07.970 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 62, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:15:59.567631+0000', 'modified': '2022-07-20T03:16:01.250517+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 7668}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_7668': {'gid': 7668, 'name': 'b', 'rank': 0, 'incarnation': 60, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/2884785802', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 2884785802}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 2884785802}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [22], 'metadata_pool': 21, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:07.978 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:07.979 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:16:08.096 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":7668,"want_state":"up:active","state":"up:active","rank_uptime":8.5150202349999997,"mdsmap_epoch":62,"osdmap_epoch":111,"osdmap_epoch_barrier":111,"uptime":13.067487673} 2022-07-20T03:16:08.097 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 7668, "mdsmap_epoch": 62, "osdmap_epoch": 111, "osdmap_epoch_barrier": 111, "rank_uptime": 8.515020235, "state": "up:active", "uptime": 13.067487673, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:08.098 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:16:08.099 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:16:08.100 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:16:08.101 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.102 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.114 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:08.139 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:08.140 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:08.197 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:08.198 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:08.199 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.200 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:08.207 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:16:08.207 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.221 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:08.240 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:08.241 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:08.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[2022-07-20T03:16:08.283+0000 7fa82813e080 -1 init, newargv = 0x55698aca3200 newargc=9 2022-07-20T03:16:08.288 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:22766]: starting ceph client 2022-07-20T03:16:08.291 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:08.293 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:08.295 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.295 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:08.303 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22766]: starting fuse 2022-07-20T03:16:08.306 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:16:08.307 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:16:08.308 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.310 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:16:08.312 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:16:08.312 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:16:08.314 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:16:08.314 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:16:08.315 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:08.316 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:16:08.317 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:16:08.319 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:16:08.319 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:08.321 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:16:08.321 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:16:08.322 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:16:08.322 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:16:08.323 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:08.323 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:16:08.324 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:16:08.324 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:16:08.325 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:16:08.325 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:16:08.326 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:08.327 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:16:08.327 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:16:08.404 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22766.asok 2022-07-20T03:16:08.408 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22766.asok 2022-07-20T03:16:08.409 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22766.asok status 2022-07-20T03:16:08.506 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:16:08.506 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:16:08.507 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:16:08.508 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:16:08.508 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:16:08.509 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:16:08.509 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:16:08.509 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22766", 2022-07-20T03:16:08.510 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:16:08.510 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:08.511 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:16:08.511 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:16:08.512 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 7893, 2022-07-20T03:16:08.512 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:16:08.512 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:16:08.513 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:16:08.513 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 7893 2022-07-20T03:16:08.514 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:08.515 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:16:08.515 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:16:08.516 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:16:08.516 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3472661818 2022-07-20T03:16:08.516 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:16:08.517 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:08.517 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:16:08.518 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:16:08.518 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:16:08.518 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 3472661818 2022-07-20T03:16:08.519 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:08.519 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.7893 172.21.15.163:0/3472661818", 2022-07-20T03:16:08.520 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/3472661818", 2022-07-20T03:16:08.520 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:16:08.521 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 62, 2022-07-20T03:16:08.521 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 111, 2022-07-20T03:16:08.521 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:16:08.522 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:16:08.522 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:16:08.523 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:16:08.525 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.526 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.527 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.537 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:08.539 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:08.540 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.541 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.559 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.561 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.561 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:08.570 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:16:08.571 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:08.572 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.572 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.573 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:08.621 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:16:08.623 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:08.623 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:16:08.624 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:16:08.624 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:08.640 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23330]: fuse finished with error 0 and tester_r 0 2022-07-20T03:16:08.642 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.643 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.643 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:08.651 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:16:08.652 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:16:08.652 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:16:08.735 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:08.735 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:08.744 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.745 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.745 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.755 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:08.755 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:08.756 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.756 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.757 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.765 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:08.766 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:08.767 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:16:08.767 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:16:08.768 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.784 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22766]: fuse finished with error 0 and tester_r 0 2022-07-20T03:16:08.787 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:08.787 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:08.788 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.838 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:16:08.839 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:16:08.839 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:16:08.882 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:08.882 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:08.891 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:16:08.891 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:16:08.892 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:16:08.892 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:16:08.910 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:16:08.910 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:16:08.911 DEBUG:teuthology.parallel:result is None 2022-07-20T03:16:08.912 DEBUG:teuthology.parallel:result is None 2022-07-20T03:16:08.913 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail a 2022-07-20T03:16:08.916 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph mds fail b 2022-07-20T03:16:09.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 Processor -- start 2022-07-20T03:16:09.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 -- start start 2022-07-20T03:16:09.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c0b9eb0 0x7f5a7c05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5a7c05c990 0x7f5a7c146430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5a7c0b0c20 con 0x7f5a7c0b5320 2022-07-20T03:16:09.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5a7c0ba290 con 0x7f5a7c05c990 2022-07-20T03:16:09.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a80f8a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c142e00 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5a7c05c990 0x7f5a7c146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5a7c05c990 0x7f5a7c146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49782/0 (socket says 172.21.15.61:49782) 2022-07-20T03:16:09.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/1169436961 learned_addr learned my addr 172.21.15.61:0/1169436961 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:09.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a79d9b700 1 --1- 172.21.15.61:0/1169436961 >> v1:172.21.15.61:6789/0 conn(0x7f5a7c0b5320 0x7f5a7c05c280 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46378/0 (socket says 172.21.15.61:46378) 2022-07-20T03:16:09.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7a59c700 1 --2- 172.21.15.61:0/1169436961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c0b9eb0 0x7f5a7c05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7959a700 1 -- 172.21.15.61:0/1169436961 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 26184979 0 0) 0x7f5a7c0b0c20 con 0x7f5a7c0b5320 2022-07-20T03:16:09.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/1169436961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c0b9eb0 msgr2=0x7f5a7c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:09.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 --2- 172.21.15.61:0/1169436961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c0b9eb0 0x7f5a7c05bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/1169436961 >> v1:172.21.15.61:6789/0 conn(0x7f5a7c0b5320 legacy=0x7f5a7c05c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.024+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/1169436961 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a7c0afed0 con 0x7f5a7c05c990 2022-07-20T03:16:09.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a7ad9d700 1 --2- 172.21.15.61:0/1169436961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c05c990 0x7f5a7c146430 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f5a740032f0 tx=0x7f5a74009290).ready entity=mon.0 client_cookie=bde058192ff153d4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a7959a700 1 -- 172.21.15.61:0/1169436961 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a7400b8a0 con 0x7f5a7c05c990 2022-07-20T03:16:09.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a7959a700 1 -- 172.21.15.61:0/1169436961 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a7400ba00 con 0x7f5a7c05c990 2022-07-20T03:16:09.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a7959a700 1 -- 172.21.15.61:0/1169436961 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a7400bb70 con 0x7f5a7c05c990 2022-07-20T03:16:09.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- 172.21.15.61:0/1169436961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c05c990 msgr2=0x7f5a7c146430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/1169436961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c05c990 0x7f5a7c146430 secure :-1 s=READY pgs=1189 cs=0 l=1 rev1=1 rx=0x7f5a740032f0 tx=0x7f5a74009290).stop 2022-07-20T03:16:09.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- 172.21.15.61:0/1169436961 shutdown_connections 2022-07-20T03:16:09.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/1169436961 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c05c990 0x7f5a7c146430 unknown :-1 s=CLOSED pgs=1189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 Processor -- start 2022-07-20T03:16:09.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/1169436961 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c0b9eb0 0x7f5a7c05bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- 172.21.15.61:0/1169436961 >> 172.21.15.61:0/1169436961 conn(0x7f5a7c067980 msgr2=0x7f5a7c0b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:09.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- start start 2022-07-20T03:16:09.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- 172.21.15.61:0/1169436961 shutdown_connections 2022-07-20T03:16:09.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- 172.21.15.61:0/1169436961 wait complete. 2022-07-20T03:16:09.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4380af300 0x7fc43813f530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc4380aff10 0x7fc43813fa20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc4380b17b0 con 0x7fc4380ace90 2022-07-20T03:16:09.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4380b09a0 con 0x7fc4380af300 2022-07-20T03:16:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43813f090 con 0x7fc4380aff10 2022-07-20T03:16:09.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4380af300 0x7fc43813f530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc4380af300 0x7fc43813f530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49786/0 (socket says 172.21.15.61:49786) 2022-07-20T03:16:09.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 -- 172.21.15.61:0/4112489034 learned_addr learned my addr 172.21.15.61:0/4112489034 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:09.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 Processor -- start 2022-07-20T03:16:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4377fe700 1 --2- 172.21.15.61:0/4112489034 >> v2:172.21.15.117:3300/0 conn(0x7fc4380aff10 0x7fc43813fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 -- 172.21.15.61:0/4112489034 >> v2:172.21.15.117:3300/0 conn(0x7fc4380aff10 msgr2=0x7fc43813fa20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:09.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- start start 2022-07-20T03:16:09.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 --2- 172.21.15.61:0/4112489034 >> v2:172.21.15.117:3300/0 conn(0x7fc4380aff10 0x7fc43813fa20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 -- 172.21.15.61:0/4112489034 >> v1:172.21.15.117:6789/0 conn(0x7fc4380ace90 legacy=0x7fc43813e410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:09.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 -- 172.21.15.61:0/4112489034 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4380b0c40 con 0x7fc4380af300 2022-07-20T03:16:09.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4377fe700 1 --2- 172.21.15.61:0/4112489034 >> v2:172.21.15.117:3300/0 conn(0x7fc4380aff10 0x7fc43813fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:16:09.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc4367fc700 1 --2- 172.21.15.61:0/4112489034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc43813f530 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7fc438060b20 tx=0x7fc42c006eb0).ready entity=mon.0 client_cookie=9557ff97333f5d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc435ffb700 1 -- 172.21.15.61:0/4112489034 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42c00d5c0 con 0x7fc4380af300 2022-07-20T03:16:09.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc435ffb700 1 -- 172.21.15.61:0/4112489034 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c000e90 con 0x7fc4380af300 2022-07-20T03:16:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc435ffb700 1 -- 172.21.15.61:0/4112489034 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42c00d910 con 0x7fc4380af300 2022-07-20T03:16:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- 172.21.15.61:0/4112489034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 msgr2=0x7fc43813f530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 --2- 172.21.15.61:0/4112489034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc43813f530 secure :-1 s=READY pgs=1190 cs=0 l=1 rev1=1 rx=0x7fc438060b20 tx=0x7fc42c006eb0).stop 2022-07-20T03:16:09.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- 172.21.15.61:0/4112489034 shutdown_connections 2022-07-20T03:16:09.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 --2- 172.21.15.61:0/4112489034 >> v2:172.21.15.117:3300/0 conn(0x7fc4380aff10 0x7fc43813fa20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 --2- 172.21.15.61:0/4112489034 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc43813f530 unknown :-1 s=CLOSED pgs=1190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- 172.21.15.61:0/4112489034 >> 172.21.15.61:0/4112489034 conn(0x7fc438063490 msgr2=0x7fc4380b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:09.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- 172.21.15.61:0/4112489034 shutdown_connections 2022-07-20T03:16:09.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7fc437fff700 1 -- 172.21.15.61:0/4112489034 wait complete. 2022-07-20T03:16:09.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c05c990 0x7f5a7c148c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a7c0b5320 0x7f5a7c13d900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 0x7f5a7c13ddf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c146230 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c156410 con 0x7f5a7c0b5320 2022-07-20T03:16:09.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.028+0000 7f5a80f8a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5a7c0b8490 con 0x7f5a7c05c990 2022-07-20T03:16:09.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 0x7f5a7c13ddf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 0x7f5a7c13ddf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49794/0 (socket says 172.21.15.61:49794) 2022-07-20T03:16:09.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/315964176 learned_addr learned my addr 172.21.15.61:0/315964176 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:09.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a79d9b700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a7c0b5320 0x7f5a7c13d900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7a59c700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c05c990 0x7f5a7c148c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c05c990 msgr2=0x7f5a7c148c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c05c990 0x7f5a7c148c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a7c0b5320 msgr2=0x7f5a7c13d900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a7c0b5320 0x7f5a7c13d900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 -- 172.21.15.61:0/315964176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5a74009530 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a79d9b700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a7c0b5320 0x7f5a7c13d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:09.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7a59c700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c05c990 0x7f5a7c148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:09.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a7ad9d700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 0x7f5a7c13ddf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f5a740032f0 tx=0x7f5a74016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 Processor -- start 2022-07-20T03:16:09.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 -- start start 2022-07-20T03:16:09.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a7400b8a0 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5a74021630 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5a7c1554e0 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5a7c0b8960 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5a74021990 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4380ace90 0x7fc438155e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc438134130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4380aff10 0x7fc438151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc438156370 con 0x7fc4380af300 2022-07-20T03:16:09.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4381554a0 con 0x7fc4380ace90 2022-07-20T03:16:09.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc438155740 con 0x7fc4380aff10 2022-07-20T03:16:09.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc438134130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc438134130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49800/0 (socket says 172.21.15.61:49800) 2022-07-20T03:16:09.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 -- 172.21.15.61:0/2678883662 learned_addr learned my addr 172.21.15.61:0/2678883662 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:09.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc436ffd700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4380ace90 0x7fc438155e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4377fe700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4380aff10 0x7fc438151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 -- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4380aff10 msgr2=0x7fc438151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4380aff10 0x7fc438151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 -- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4380ace90 msgr2=0x7fc438155e80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4380ace90 0x7fc438155e80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 -- 172.21.15.61:0/2678883662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c00b1e0 con 0x7fc4380af300 2022-07-20T03:16:09.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc4367fc700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc438134130 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7fc438060b20 tx=0x7fc42c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5a7c0b0170 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42c002f50 con 0x7fc4380af300 2022-07-20T03:16:09.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c00d940 con 0x7fc4380af300 2022-07-20T03:16:09.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4381559e0 con 0x7fc4380af300 2022-07-20T03:16:09.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42c011d70 con 0x7fc4380af300 2022-07-20T03:16:09.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4381346a0 con 0x7fc4380af300 2022-07-20T03:16:09.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5a74027020 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.032+0000 7f5a6b7fe700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a58053a40 0x7f5a58055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5a7406d350 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7f5a7a59c700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a58053a40 0x7f5a58055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4380574e0 con 0x7fc4380af300 2022-07-20T03:16:09.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7f5a7a59c700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a58053a40 0x7f5a58055f00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f5a6c00c000 tx=0x7f5a6c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc42c025030 con 0x7fc4380af300 2022-07-20T03:16:09.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7fc427fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4140539b0 0x7fc414055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(111..111 src has 1..111) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc42c06cf70 con 0x7fc4380af300 2022-07-20T03:16:09.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7fc436ffd700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4140539b0 0x7fc414055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.036+0000 7fc436ffd700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4140539b0 0x7fc414055e70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fc42800d000 tx=0x7fc428007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.040+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5a740442b0 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.044+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc42c043ed0 con 0x7fc4380af300 2022-07-20T03:16:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.204+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "b"} v 0) v1 -- 0x7f5a7c0574e0 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.212+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "mds fail", "role_or_gid": "a"} v 0) v1 -- 0x7fc43805db10 con 0x7fc4380af300 2022-07-20T03:16:09.288 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:16:09.284+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:16:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.300+0000 7fc427fff700 1 -- 172.21.15.61:0/2678883662 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "a"}]=0 failed mds gid 7662 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7fc42c023020 con 0x7fc4380af300 2022-07-20T03:16:09.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.300+0000 7f5a6b7fe700 1 -- 172.21.15.61:0/315964176 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "mds fail", "role_or_gid": "b"}]=0 failed mds gid 7668 v63) v1 ==== 95+0+0 (secure 0 0 0) 0x7f5a74015090 con 0x7f5a7c0b9eb0 2022-07-20T03:16:09.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a58053a40 msgr2=0x7f5a58055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a58053a40 0x7f5a58055f00 secure :-1 s=READY pgs=570 cs=0 l=1 rev1=1 rx=0x7f5a6c00c000 tx=0x7f5a6c008040).stop 2022-07-20T03:16:09.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 msgr2=0x7f5a7c13ddf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 0x7f5a7c13ddf0 secure :-1 s=READY pgs=1191 cs=0 l=1 rev1=1 rx=0x7f5a740032f0 tx=0x7f5a74016000).stop 2022-07-20T03:16:09.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 shutdown_connections 2022-07-20T03:16:09.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5a58053a40 0x7f5a58055f00 unknown :-1 s=CLOSED pgs=570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5a7c0b5320 0x7f5a7c13d900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5a7c0b9eb0 0x7f5a7c13ddf0 unknown :-1 s=CLOSED pgs=1191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 --2- 172.21.15.61:0/315964176 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5a7c05c990 0x7f5a7c148c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 >> 172.21.15.61:0/315964176 conn(0x7f5a7c067980 msgr2=0x7f5a7c1424b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:09.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4140539b0 msgr2=0x7fc414055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4140539b0 0x7fc414055e70 secure :-1 s=READY pgs=571 cs=0 l=1 rev1=1 rx=0x7fc42800d000 tx=0x7fc428007b70).stop 2022-07-20T03:16:09.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 msgr2=0x7fc438134130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc438134130 secure :-1 s=READY pgs=1192 cs=0 l=1 rev1=1 rx=0x7fc438060b20 tx=0x7fc42c015000).stop 2022-07-20T03:16:09.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 shutdown_connections 2022-07-20T03:16:09.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7f5a80f8a700 1 -- 172.21.15.61:0/315964176 wait complete. 2022-07-20T03:16:09.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 shutdown_connections 2022-07-20T03:16:09.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc4140539b0 0x7fc414055e70 unknown :-1 s=CLOSED pgs=571 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc4380aff10 0x7fc438151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc4380af300 0x7fc438134130 unknown :-1 s=CLOSED pgs=1192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 --2- 172.21.15.61:0/2678883662 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4380ace90 0x7fc438155e80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 >> 172.21.15.61:0/2678883662 conn(0x7fc438063490 msgr2=0x7fc4380b45c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:09.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 shutdown_connections 2022-07-20T03:16:09.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.304+0000 7fc437fff700 1 -- 172.21.15.61:0/2678883662 wait complete. 2022-07-20T03:16:09.321 INFO:teuthology.orchestra.run.smithi061.stderr:failed mds gid 7668 2022-07-20T03:16:09.323 INFO:teuthology.orchestra.run.smithi061.stderr:failed mds gid 7662 2022-07-20T03:16:09.325 DEBUG:teuthology.parallel:result is '' 2022-07-20T03:16:09.327 DEBUG:teuthology.parallel:result is '' 2022-07-20T03:16:09.328 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:16:09.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 Processor -- start 2022-07-20T03:16:09.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 -- start start 2022-07-20T03:16:09.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbde80b9eb0 0x7fbde805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbde80b5320 0x7fbde805c280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde805c770 0x7fbde8141d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde80b0c20 con 0x7fbde80b5320 2022-07-20T03:16:09.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbde80ba290 con 0x7fbde80b9eb0 2022-07-20T03:16:09.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbdedebd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde805cbc0 con 0x7fbde805c770 2022-07-20T03:16:09.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbde6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbde80b5320 0x7fbde805c280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbde6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbde80b5320 0x7fbde805c280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49808/0 (socket says 172.21.15.61:49808) 2022-07-20T03:16:09.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbde6ffd700 1 -- 172.21.15.61:0/3896155027 learned_addr learned my addr 172.21.15.61:0/3896155027 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:09.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.448+0000 7fbde77fe700 1 --2- 172.21.15.61:0/3896155027 >> v2:172.21.15.117:3300/0 conn(0x7fbde80b9eb0 0x7fbde805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde7fff700 1 --2- 172.21.15.61:0/3896155027 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde805c770 0x7fbde8141d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde6ffd700 1 -- 172.21.15.61:0/3896155027 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde805c770 msgr2=0x7fbde8141d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:09.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde6ffd700 1 --2- 172.21.15.61:0/3896155027 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde805c770 0x7fbde8141d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde6ffd700 1 -- 172.21.15.61:0/3896155027 >> v2:172.21.15.117:3300/0 conn(0x7fbde80b9eb0 msgr2=0x7fbde805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:09.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde6ffd700 1 --2- 172.21.15.61:0/3896155027 >> v2:172.21.15.117:3300/0 conn(0x7fbde80b9eb0 0x7fbde805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde6ffd700 1 -- 172.21.15.61:0/3896155027 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbde8142e00 con 0x7fbde80b5320 2022-07-20T03:16:09.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde77fe700 1 --2- 172.21.15.61:0/3896155027 >> v2:172.21.15.117:3300/0 conn(0x7fbde80b9eb0 0x7fbde805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:09.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde6ffd700 1 --2- 172.21.15.61:0/3896155027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b5320 0x7fbde805c280 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fbddc002f70 tx=0x7fbddc006eb0).ready entity=mon.0 client_cookie=9720fb931047580e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde67fc700 1 -- 172.21.15.61:0/3896155027 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbddc00d570 con 0x7fbde80b5320 2022-07-20T03:16:09.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde67fc700 1 -- 172.21.15.61:0/3896155027 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbddc000e90 con 0x7fbde80b5320 2022-07-20T03:16:09.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbde67fc700 1 -- 172.21.15.61:0/3896155027 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbddc00d900 con 0x7fbde80b5320 2022-07-20T03:16:09.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 -- 172.21.15.61:0/3896155027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b5320 msgr2=0x7fbde805c280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3896155027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b5320 0x7fbde805c280 secure :-1 s=READY pgs=1193 cs=0 l=1 rev1=1 rx=0x7fbddc002f70 tx=0x7fbddc006eb0).stop 2022-07-20T03:16:09.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 -- 172.21.15.61:0/3896155027 shutdown_connections 2022-07-20T03:16:09.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3896155027 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde805c770 0x7fbde8141d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3896155027 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b5320 0x7fbde805c280 unknown :-1 s=CLOSED pgs=1193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3896155027 >> v2:172.21.15.117:3300/0 conn(0x7fbde80b9eb0 0x7fbde805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 -- 172.21.15.61:0/3896155027 >> 172.21.15.61:0/3896155027 conn(0x7fbde8067980 msgr2=0x7fbde80b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:09.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 -- 172.21.15.61:0/3896155027 shutdown_connections 2022-07-20T03:16:09.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.452+0000 7fbdedebd700 1 -- 172.21.15.61:0/3896155027 wait complete. 2022-07-20T03:16:09.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 Processor -- start 2022-07-20T03:16:09.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 -- start start 2022-07-20T03:16:09.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbde805c770 0x7fbde8118ea0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde80b5320 0x7fbde81166a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 0x7fbde8152090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8117900 con 0x7fbde80b9eb0 2022-07-20T03:16:09.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbde8116d60 con 0x7fbde805c770 2022-07-20T03:16:09.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbde81151f0 con 0x7fbde80b5320 2022-07-20T03:16:09.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 0x7fbde8152090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 0x7fbde8152090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49816/0 (socket says 172.21.15.61:49816) 2022-07-20T03:16:09.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 -- 172.21.15.61:0/3024619539 learned_addr learned my addr 172.21.15.61:0/3024619539 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:09.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde6ffd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde80b5320 0x7fbde81166a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde77fe700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbde805c770 0x7fbde8118ea0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 -- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde80b5320 msgr2=0x7fbde81166a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde80b5320 0x7fbde81166a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 -- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbde805c770 msgr2=0x7fbde8118ea0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:09.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbde805c770 0x7fbde8118ea0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:09.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 -- 172.21.15.61:0/3024619539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdd8009470 con 0x7fbde80b9eb0 2022-07-20T03:16:09.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde7fff700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 0x7fbde8152090 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7fbde000c000 tx=0x7fbde0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbde0011750 con 0x7fbde80b9eb0 2022-07-20T03:16:09.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbde00118b0 con 0x7fbde80b9eb0 2022-07-20T03:16:09.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc00b1e0 con 0x7fbde80b9eb0 2022-07-20T03:16:09.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.456+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbde0011c10 con 0x7fbde80b9eb0 2022-07-20T03:16:09.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.460+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbde8114a80 con 0x7fbde80b9eb0 2022-07-20T03:16:09.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.460+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbde8116f60 con 0x7fbde80b9eb0 2022-07-20T03:16:09.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.460+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbde0011d70 con 0x7fbde80b9eb0 2022-07-20T03:16:09.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.460+0000 7fbde4ff9700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdc8053a40 0x7fbdc8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:09.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.460+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbde006c240 con 0x7fbde80b9eb0 2022-07-20T03:16:09.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.460+0000 7fbde77fe700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdc8053a40 0x7fbdc8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:09.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.464+0000 7fbde77fe700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdc8053a40 0x7fbdc8055f00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fbdd800b000 tx=0x7fbdd80091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:09.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.468+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbde0043040 con 0x7fbde80b9eb0 2022-07-20T03:16:09.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:09.628+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fbde8116b90 con 0x7fbde80b9eb0 2022-07-20T03:16:10.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbde4ff9700 1 -- 172.21.15.61:0/3024619539 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v64) v1 ==== 104+0+0 (secure 0 0 0) 0x7fbde0046660 con 0x7fbde80b9eb0 2022-07-20T03:16:10.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdc8053a40 msgr2=0x7fbdc8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:10.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdc8053a40 0x7fbdc8055f00 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7fbdd800b000 tx=0x7fbdd80091d0).stop 2022-07-20T03:16:10.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 msgr2=0x7fbde8152090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:10.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 0x7fbde8152090 secure :-1 s=READY pgs=1194 cs=0 l=1 rev1=1 rx=0x7fbde000c000 tx=0x7fbde0006df0).stop 2022-07-20T03:16:10.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 shutdown_connections 2022-07-20T03:16:10.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdc8053a40 0x7fbdc8055f00 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbde80b9eb0 0x7fbde8152090 unknown :-1 s=CLOSED pgs=1194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbde80b5320 0x7fbde81166a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 --2- 172.21.15.61:0/3024619539 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbde805c770 0x7fbde8118ea0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.304+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 >> 172.21.15.61:0/3024619539 conn(0x7fbde8067980 msgr2=0x7fbde80b3290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:10.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.308+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 shutdown_connections 2022-07-20T03:16:10.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.308+0000 7fbdedebd700 1 -- 172.21.15.61:0/3024619539 wait complete. 2022-07-20T03:16:10.327 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:16:10.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 Processor -- start 2022-07-20T03:16:10.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 -- start start 2022-07-20T03:16:10.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b6690 0x7f22880b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:10.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22880ae670 0x7f22880aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:10.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f22880acae0 con 0x7f22880b59a0 2022-07-20T03:16:10.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2288147100 con 0x7f22880ae670 2022-07-20T03:16:10.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f22901ef700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f22880b5dc0 con 0x7f22880b6690 2022-07-20T03:16:10.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22880ae670 0x7f22880aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:10.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f22880ae670 0x7f22880aeb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49824/0 (socket says 172.21.15.61:49824) 2022-07-20T03:16:10.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 -- 172.21.15.61:0/3229057001 learned_addr learned my addr 172.21.15.61:0/3229057001 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:10.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228d78a700 1 --1- 172.21.15.61:0/3229057001 >> v1:172.21.15.117:6789/0 conn(0x7f22880b59a0 0x7f228813c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:54924/0 (socket says 172.21.15.61:54924) 2022-07-20T03:16:10.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228df8b700 1 --2- 172.21.15.61:0/3229057001 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b6690 0x7f22880b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:10.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 -- 172.21.15.61:0/3229057001 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b6690 msgr2=0x7f22880b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:10.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 --2- 172.21.15.61:0/3229057001 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b6690 0x7f22880b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228cf89700 1 -- 172.21.15.61:0/3229057001 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 382437015 0 0) 0x7f22880acae0 con 0x7f22880b59a0 2022-07-20T03:16:10.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 -- 172.21.15.61:0/3229057001 >> v1:172.21.15.117:6789/0 conn(0x7f22880b59a0 legacy=0x7f228813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 -- 172.21.15.61:0/3229057001 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f22880acd10 con 0x7f22880ae670 2022-07-20T03:16:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228e78c700 1 --2- 172.21.15.61:0/3229057001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880aeb00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f22800032f0 tx=0x7f2280009290).ready entity=mon.0 client_cookie=1a1ab4b5b631bb0a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228cf89700 1 -- 172.21.15.61:0/3229057001 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228000b8a0 con 0x7f22880ae670 2022-07-20T03:16:10.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228cf89700 1 -- 172.21.15.61:0/3229057001 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f228000ba00 con 0x7f22880ae670 2022-07-20T03:16:10.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.452+0000 7f228cf89700 1 -- 172.21.15.61:0/3229057001 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f228000bb70 con 0x7f22880ae670 2022-07-20T03:16:10.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- 172.21.15.61:0/3229057001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 msgr2=0x7f22880aeb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:10.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 --2- 172.21.15.61:0/3229057001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880aeb00 secure :-1 s=READY pgs=1195 cs=0 l=1 rev1=1 rx=0x7f22800032f0 tx=0x7f2280009290).stop 2022-07-20T03:16:10.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- 172.21.15.61:0/3229057001 shutdown_connections 2022-07-20T03:16:10.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 --2- 172.21.15.61:0/3229057001 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880aeb00 unknown :-1 s=CLOSED pgs=1195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 --2- 172.21.15.61:0/3229057001 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b6690 0x7f22880b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- 172.21.15.61:0/3229057001 >> 172.21.15.61:0/3229057001 conn(0x7f22880b9f70 msgr2=0x7f22880b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:10.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- 172.21.15.61:0/3229057001 shutdown_connections 2022-07-20T03:16:10.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- 172.21.15.61:0/3229057001 wait complete. 2022-07-20T03:16:10.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 Processor -- start 2022-07-20T03:16:10.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- start start 2022-07-20T03:16:10.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880b3470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:10.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b59a0 0x7f228813d950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:10.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880b6690 0x7f228813be20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:10.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22880b2d10 con 0x7f22880ae670 2022-07-20T03:16:10.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f22880b2a70 con 0x7f22880b6690 2022-07-20T03:16:10.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f22901ef700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2288153490 con 0x7f22880b59a0 2022-07-20T03:16:10.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f228df8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880b3470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:10.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f228df8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880b3470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49828/0 (socket says 172.21.15.61:49828) 2022-07-20T03:16:10.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f228df8b700 1 -- 172.21.15.61:0/3932909541 learned_addr learned my addr 172.21.15.61:0/3932909541 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:10.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f228d78a700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b59a0 0x7f228813d950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:10.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.456+0000 7f228e78c700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880b6690 0x7f228813be20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:10.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f228d78a700 1 -- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880b6690 msgr2=0x7f228813be20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f228d78a700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880b6690 0x7f228813be20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f228d78a700 1 -- 172.21.15.61:0/3932909541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 msgr2=0x7f22880b3470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f228d78a700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880b3470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f228d78a700 1 -- 172.21.15.61:0/3932909541 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2280009530 con 0x7f22880b59a0 2022-07-20T03:16:10.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f228d78a700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b59a0 0x7f228813d950 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f2278000ed0 tx=0x7f2278007a00).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:10.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f22780070d0 con 0x7f22880b59a0 2022-07-20T03:16:10.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2278007230 con 0x7f22880b59a0 2022-07-20T03:16:10.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f22880b1ac0 con 0x7f22880b59a0 2022-07-20T03:16:10.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2278005550 con 0x7f22880b59a0 2022-07-20T03:16:10.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f22880b2610 con 0x7f22880b59a0 2022-07-20T03:16:10.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2288153650 con 0x7f22880b59a0 2022-07-20T03:16:10.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f227801a060 con 0x7f22880b59a0 2022-07-20T03:16:10.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f2276ffd700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2268053a60 0x7f2268055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:10.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.460+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(113..113 src has 1..113) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f227806b500 con 0x7f22880b59a0 2022-07-20T03:16:10.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.464+0000 7f228df8b700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2268053a60 0x7f2268055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:10.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.464+0000 7f228df8b700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2268053a60 0x7f2268055f20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f228400b000 tx=0x7f2284006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:10.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.472+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2278042410 con 0x7f22880b59a0 2022-07-20T03:16:10.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:10.604+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f22880acfb0 con 0x7f22880b59a0 2022-07-20T03:16:11.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.320+0000 7f2276ffd700 1 -- 172.21.15.61:0/3932909541 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v114) v1 ==== 185+0+0 (secure 0 0 0) 0x7f2278018020 con 0x7f22880b59a0 2022-07-20T03:16:11.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2268053a60 msgr2=0x7f2268055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:11.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2268053a60 0x7f2268055f20 secure :-1 s=READY pgs=573 cs=0 l=1 rev1=1 rx=0x7f228400b000 tx=0x7f2284006df0).stop 2022-07-20T03:16:11.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b59a0 msgr2=0x7f228813d950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:11.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b59a0 0x7f228813d950 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f2278000ed0 tx=0x7f2278007a00).stop 2022-07-20T03:16:11.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 shutdown_connections 2022-07-20T03:16:11.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2268053a60 0x7f2268055f20 unknown :-1 s=CLOSED pgs=573 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f22880b6690 0x7f228813be20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f22880b59a0 0x7f228813d950 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 --2- 172.21.15.61:0/3932909541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f22880ae670 0x7f22880b3470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 >> 172.21.15.61:0/3932909541 conn(0x7f22880b9f70 msgr2=0x7f22880b7df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:11.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 shutdown_connections 2022-07-20T03:16:11.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.324+0000 7f22901ef700 1 -- 172.21.15.61:0/3932909541 wait complete. 2022-07-20T03:16:11.336 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:16:11.349 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:16:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.476+0000 7f77b3fff700 1 Processor -- start 2022-07-20T03:16:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.476+0000 7f77b3fff700 1 -- start start 2022-07-20T03:16:11.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 0x7f77b40ab360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f77b405f840 0x7f77b4146ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b405ca70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b40ab850 con 0x7f77b40acbe0 2022-07-20T03:16:11.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77b41460e0 con 0x7f77b405f840 2022-07-20T03:16:11.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4146380 con 0x7f77b405c620 2022-07-20T03:16:11.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 0x7f77b40ab360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b405ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77aa7fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f77b405f840 0x7f77b4146ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b2ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 0x7f77b40ab360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49834/0 (socket says 172.21.15.61:49834) 2022-07-20T03:16:11.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b2ffd700 1 -- 172.21.15.61:0/4224039864 learned_addr learned my addr 172.21.15.61:0/4224039864 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:11.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 -- 172.21.15.61:0/4224039864 >> v2:172.21.15.117:3300/0 conn(0x7f77b405f840 msgr2=0x7f77b4146ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:11.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 --2- 172.21.15.61:0/4224039864 >> v2:172.21.15.117:3300/0 conn(0x7f77b405f840 0x7f77b4146ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 -- 172.21.15.61:0/4224039864 >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 msgr2=0x7f77b40ab360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:11.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 --2- 172.21.15.61:0/4224039864 >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 0x7f77b40ab360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 -- 172.21.15.61:0/4224039864 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77b4146620 con 0x7f77b405c620 2022-07-20T03:16:11.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77aa7fc700 1 --2- 172.21.15.61:0/4224039864 >> v2:172.21.15.117:3300/0 conn(0x7f77b405f840 0x7f77b4146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:11.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b2ffd700 1 --2- 172.21.15.61:0/4224039864 >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 0x7f77b40ab360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:11.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b37fe700 1 --2- 172.21.15.61:0/4224039864 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b405ca70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f77ac0032f0 tx=0x7f77ac0091d0).ready entity=mon.2 client_cookie=e57e35c86c04e2ff server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:11.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b27fc700 1 -- 172.21.15.61:0/4224039864 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77ac00b600 con 0x7f77b405c620 2022-07-20T03:16:11.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b27fc700 1 -- 172.21.15.61:0/4224039864 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f77ac00b760 con 0x7f77b405c620 2022-07-20T03:16:11.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b27fc700 1 -- 172.21.15.61:0/4224039864 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f77ac00b8d0 con 0x7f77b405c620 2022-07-20T03:16:11.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- 172.21.15.61:0/4224039864 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 msgr2=0x7f77b405ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:11.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- 172.21.15.61:0/4224039864 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b405ca70 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f77ac0032f0 tx=0x7f77ac0091d0).stop 2022-07-20T03:16:11.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- 172.21.15.61:0/4224039864 shutdown_connections 2022-07-20T03:16:11.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- 172.21.15.61:0/4224039864 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b405ca70 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- 172.21.15.61:0/4224039864 >> v2:172.21.15.117:3300/0 conn(0x7f77b405f840 0x7f77b4146ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 --2- 172.21.15.61:0/4224039864 >> v2:172.21.15.61:3300/0 conn(0x7f77b40acbe0 0x7f77b40ab360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- 172.21.15.61:0/4224039864 >> 172.21.15.61:0/4224039864 conn(0x7f77b40b2af0 msgr2=0x7f77b40b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:11.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- 172.21.15.61:0/4224039864 shutdown_connections 2022-07-20T03:16:11.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.480+0000 7f77b3fff700 1 -- 172.21.15.61:0/4224039864 wait complete. 2022-07-20T03:16:11.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 Processor -- start 2022-07-20T03:16:11.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 -- start start 2022-07-20T03:16:11.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b40af680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 0x7f77b41554e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77b40acbe0 0x7f77b41559d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4099160 con 0x7f77b405f840 2022-07-20T03:16:11.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4099390 con 0x7f77b40acbe0 2022-07-20T03:16:11.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77b4099630 con 0x7f77b405c620 2022-07-20T03:16:11.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 0x7f77b41554e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 0x7f77b41554e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49842/0 (socket says 172.21.15.61:49842) 2022-07-20T03:16:11.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b37fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77b40acbe0 0x7f77b41559d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 -- 172.21.15.61:0/194832153 learned_addr learned my addr 172.21.15.61:0/194832153 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:11.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b2ffd700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b40af680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 -- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 msgr2=0x7f77b40af680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:11.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b40af680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 -- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77b40acbe0 msgr2=0x7f77b41559d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:11.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77b40acbe0 0x7f77b41559d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:11.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 -- 172.21.15.61:0/194832153 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779c00b180 con 0x7f77b405f840 2022-07-20T03:16:11.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b2ffd700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b40af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:11.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77aa7fc700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 0x7f77b41554e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f779c00c000 tx=0x7f779c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:11.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779c00c890 con 0x7f77b405f840 2022-07-20T03:16:11.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779c00c9f0 con 0x7f77b405f840 2022-07-20T03:16:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a400b1e0 con 0x7f77b405f840 2022-07-20T03:16:11.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.484+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779c00cd50 con 0x7f77b405f840 2022-07-20T03:16:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.488+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77ac009470 con 0x7f77b405f840 2022-07-20T03:16:11.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.488+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77b4146fb0 con 0x7f77b405f840 2022-07-20T03:16:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.488+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f779c01c460 con 0x7f77b405f840 2022-07-20T03:16:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.488+0000 7f77b0ff9700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77980539b0 0x7f7798055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:11.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.488+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(114..114 src has 1..114) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f779c06c3e0 con 0x7f77b405f840 2022-07-20T03:16:11.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.488+0000 7f77b2ffd700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77980539b0 0x7f7798055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:11.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.492+0000 7f77b2ffd700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77980539b0 0x7f7798055e70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f77a400b000 tx=0x7f77a4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:11.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.496+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f779c0433a0 con 0x7f77b405f840 2022-07-20T03:16:11.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:11.628+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f77b40574e0 con 0x7f77b405f840 2022-07-20T03:16:12.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.320+0000 7f77b0ff9700 1 -- 172.21.15.61:0/194832153 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v115) v1 ==== 133+0+0 (secure 0 0 0) 0x7f779c0469c0 con 0x7f77b405f840 2022-07-20T03:16:12.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77980539b0 msgr2=0x7f7798055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:12.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77980539b0 0x7f7798055e70 secure :-1 s=READY pgs=574 cs=0 l=1 rev1=1 rx=0x7f77a400b000 tx=0x7f77a4008040).stop 2022-07-20T03:16:12.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 msgr2=0x7f77b41554e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:12.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 0x7f77b41554e0 secure :-1 s=READY pgs=1196 cs=0 l=1 rev1=1 rx=0x7f779c00c000 tx=0x7f779c006df0).stop 2022-07-20T03:16:12.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 shutdown_connections 2022-07-20T03:16:12.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f77980539b0 0x7f7798055e70 unknown :-1 s=CLOSED pgs=574 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77b40acbe0 0x7f77b41559d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77b405f840 0x7f77b41554e0 unknown :-1 s=CLOSED pgs=1196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 --2- 172.21.15.61:0/194832153 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77b405c620 0x7f77b40af680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 >> 172.21.15.61:0/194832153 conn(0x7f77b40b2af0 msgr2=0x7f77b4061d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:12.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 shutdown_connections 2022-07-20T03:16:12.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:12.328+0000 7f77b3fff700 1 -- 172.21.15.61:0/194832153 wait complete. 2022-07-20T03:16:12.363 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:16:12.368 DEBUG:teuthology.orchestra.run.smithi124:> sudo rados -p cephfs_metadata put key /etc/fstab 2022-07-20T03:16:12.425 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 Processor -- start 2022-07-20T03:16:12.425 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 -- start start 2022-07-20T03:16:12.426 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 0x55639a274bd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.426 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26ded0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.426 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2752d0 0x55639a26e3c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.426 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x55639a169810 con 0x55639a2752d0 2022-07-20T03:16:12.427 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55639a1596f0 con 0x55639a2666f0 2022-07-20T03:16:12.427 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.405+0000 7f724e8c0b80 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x55639a26ea20 con 0x55639a266fc0 2022-07-20T03:16:12.427 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 0x55639a274bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.428 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724c4fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26ded0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.428 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724d4fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2752d0 0x55639a26e3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.428 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 0x55639a274bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.124:60040/0 (socket says 172.21.15.124:60040) 2022-07-20T03:16:12.428 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 -- 172.21.15.124:0/3641116086 learned_addr learned my addr 172.21.15.124:0/3641116086 (peer_addr_for_me v2:172.21.15.124:0/0) 2022-07-20T03:16:12.429 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 -- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 msgr2=0x55639a26ded0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:12.429 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26ded0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.429 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 -- 172.21.15.124:0/3641116086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2752d0 msgr2=0x55639a26e3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:12.429 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2752d0 0x55639a26e3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.439 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 -- 172.21.15.124:0/3641116086 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f723c00a0e0 con 0x55639a2666f0 2022-07-20T03:16:12.440 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 0x55639a274bd0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x55639a275290 tx=0x7f723c014000).ready entity=mon.1 client_cookie=3d6ff1540adac23f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:12.440 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724acf9700 1 -- 172.21.15.124:0/3641116086 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723c005ea0 con 0x55639a2666f0 2022-07-20T03:16:12.440 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724acf9700 1 -- 172.21.15.124:0/3641116086 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f723c00d0f0 con 0x55639a2666f0 2022-07-20T03:16:12.440 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724acf9700 1 -- 172.21.15.124:0/3641116086 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f723c00d740 con 0x55639a2666f0 2022-07-20T03:16:12.441 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 msgr2=0x55639a274bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:12.441 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 0x55639a274bd0 secure :-1 s=READY pgs=541 cs=0 l=1 rev1=1 rx=0x55639a275290 tx=0x7f723c014000).stop 2022-07-20T03:16:12.441 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- 172.21.15.124:0/3641116086 shutdown_connections 2022-07-20T03:16:12.442 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2752d0 0x55639a26e3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.442 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26ded0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.442 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/3641116086 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2666f0 0x55639a274bd0 unknown :-1 s=CLOSED pgs=541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.443 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- 172.21.15.124:0/3641116086 >> 172.21.15.124:0/3641116086 conn(0x55639a262200 msgr2=0x55639a276270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:12.443 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- 172.21.15.124:0/3641116086 shutdown_connections 2022-07-20T03:16:12.444 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- 172.21.15.124:0/3641116086 wait complete. 2022-07-20T03:16:12.444 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 Processor -- start 2022-07-20T03:16:12.445 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- start start 2022-07-20T03:16:12.445 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2666f0 0x55639a2747f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.445 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26f950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.445 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2752d0 0x55639a26fe40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.446 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x55639a265650 con 0x55639a2666f0 2022-07-20T03:16:12.446 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x55639a270330 con 0x55639a2752d0 2022-07-20T03:16:12.447 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724e8c0b80 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x55639a2704f0 con 0x55639a266fc0 2022-07-20T03:16:12.447 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724d4fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2752d0 0x55639a26fe40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.447 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724c4fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.447 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724ccfd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2666f0 0x55639a2747f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.448 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724d4fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2752d0 0x55639a26fe40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.124:60050/0 (socket says 172.21.15.124:60050) 2022-07-20T03:16:12.448 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.409+0000 7f724d4fe700 1 -- 172.21.15.124:0/2099097582 learned_addr learned my addr 172.21.15.124:0/2099097582 (peer_addr_for_me v2:172.21.15.124:0/0) 2022-07-20T03:16:12.448 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724d4fe700 1 -- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 msgr2=0x55639a26f950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:12.448 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724d4fe700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x55639a266fc0 0x55639a26f950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.449 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724d4fe700 1 -- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2666f0 msgr2=0x55639a2747f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:12.449 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724d4fe700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x55639a2666f0 0x55639a2747f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:12.449 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724d4fe700 1 -- 172.21.15.124:0/2099097582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x55639a17e5d0 con 0x55639a2752d0 2022-07-20T03:16:12.449 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724d4fe700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2752d0 0x55639a26fe40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x55639a270f50 tx=0x7f7244001680).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:12.457 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724a4f8700 1 -- 172.21.15.124:0/2099097582 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f724400f450 con 0x55639a2752d0 2022-07-20T03:16:12.457 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724a4f8700 1 -- 172.21.15.124:0/2099097582 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f724400f5b0 con 0x55639a2752d0 2022-07-20T03:16:12.457 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x55639a152270 con 0x55639a2752d0 2022-07-20T03:16:12.458 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724a4f8700 1 -- 172.21.15.124:0/2099097582 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f724400f910 con 0x55639a2752d0 2022-07-20T03:16:12.458 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x55639a286680 con 0x55639a2752d0 2022-07-20T03:16:12.458 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.413+0000 7f724a4f8700 1 -- 172.21.15.124:0/2099097582 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f724401f070 con 0x55639a2752d0 2022-07-20T03:16:12.458 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724a4f8700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f722c0525c0 0x7f722c054a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.459 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724a4f8700 1 -- 172.21.15.124:0/2099097582 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(115..115 src has 1..115) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f724406b990 con 0x55639a2752d0 2022-07-20T03:16:12.459 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724ccfd700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f722c0525c0 0x7f722c054a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.459 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x55639a287490 0x55639a289880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:12.459 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 --> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] -- osd_op(unknown.0.0:1 23.0 23:0620e6f9:::key:head [writefull 0~617 in=617b] snapc 0=[] ondisk+write+known_if_redirected e115) v8 -- 0x55639a289e10 con 0x55639a287490 2022-07-20T03:16:12.460 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724c4fc700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x55639a287490 0x55639a289880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:12.460 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724ccfd700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f722c0525c0 0x7f722c054a80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f723c022000 tx=0x7f723c01b000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:12.460 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:12.417+0000 7f724c4fc700 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x55639a287490 0x55639a289880 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:13.788 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.784+0000 7f724a4f8700 1 -- 172.21.15.124:0/2099097582 <== osd.0 v2:172.21.15.61:6800/22894 1 ==== osd_map(116..116 src has 1..116) v4 ==== 320+0+0 (crc 0 0 0) 0x7f723400f9d0 con 0x55639a287490 2022-07-20T03:16:13.800 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724c4fc700 1 -- 172.21.15.124:0/2099097582 <== osd.0 v2:172.21.15.61:6800/22894 2 ==== osd_op_reply(1 key [writefull 0~617] v116'1 uv1 ondisk = 0) v8 ==== 147+0+0 (crc 0 0 0) 0x7f723400fb90 con 0x55639a287490 2022-07-20T03:16:13.801 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x55639a287490 msgr2=0x55639a289880 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:13.801 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x55639a287490 0x55639a289880 crc :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f722c0525c0 msgr2=0x7f722c054a80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f722c0525c0 0x7f722c054a80 secure :-1 s=READY pgs=575 cs=0 l=1 rev1=1 rx=0x7f723c022000 tx=0x7f723c01b000).stop 2022-07-20T03:16:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2752d0 msgr2=0x55639a26fe40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:13.802 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 --2- 172.21.15.124:0/2099097582 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x55639a2752d0 0x55639a26fe40 secure :-1 s=READY pgs=542 cs=0 l=1 rev1=1 rx=0x55639a270f50 tx=0x7f7244001680).stop 2022-07-20T03:16:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724d4fe700 1 -- 172.21.15.124:0/2099097582 reap_dead start 2022-07-20T03:16:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 shutdown_connections 2022-07-20T03:16:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 >> 172.21.15.124:0/2099097582 conn(0x55639a262200 msgr2=0x55639a276270 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:13.803 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 shutdown_connections 2022-07-20T03:16:13.804 INFO:teuthology.orchestra.run.smithi124.stderr:2022-07-20T03:16:13.796+0000 7f724e8c0b80 1 -- 172.21.15.124:0/2099097582 wait complete. 2022-07-20T03:16:13.813 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-20T03:16:13.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 Processor -- start 2022-07-20T03:16:13.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 -- start start 2022-07-20T03:16:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6bbc0adff0 0x7f6bbc0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6bbc0af8e0 0x7f6bbc05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:13.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6bbc147010 con 0x7f6bbc0b5650 2022-07-20T03:16:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc1472b0 con 0x7f6bbc0adff0 2022-07-20T03:16:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbb59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6bbc146d40 con 0x7f6bbc0af8e0 2022-07-20T03:16:13.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6bbc0adff0 0x7f6bbc0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6bbc0adff0 0x7f6bbc0b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49848/0 (socket says 172.21.15.61:49848) 2022-07-20T03:16:13.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 -- 172.21.15.61:0/4080169579 learned_addr learned my addr 172.21.15.61:0/4080169579 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:13.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bbad9d700 1 --2- 172.21.15.61:0/4080169579 >> v2:172.21.15.117:3300/0 conn(0x7f6bbc0af8e0 0x7f6bbc05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:13.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 -- 172.21.15.61:0/4080169579 >> v2:172.21.15.117:3300/0 conn(0x7f6bbc0af8e0 msgr2=0x7f6bbc05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:13.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 --2- 172.21.15.61:0/4080169579 >> v2:172.21.15.117:3300/0 conn(0x7f6bbc0af8e0 0x7f6bbc05bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:13.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 -- 172.21.15.61:0/4080169579 >> v1:172.21.15.117:6789/0 conn(0x7f6bbc0b5650 legacy=0x7f6bbc147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:13.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 -- 172.21.15.61:0/4080169579 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bbc0acae0 con 0x7f6bbc0adff0 2022-07-20T03:16:13.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb9d9b700 1 --2- 172.21.15.61:0/4080169579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b5a30 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f6bbc060b70 tx=0x7f6bac009250).ready entity=mon.0 client_cookie=821ff3d6f7059085 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:13.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb959a700 1 -- 172.21.15.61:0/4080169579 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bac00b830 con 0x7f6bbc0adff0 2022-07-20T03:16:13.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb959a700 1 -- 172.21.15.61:0/4080169579 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bac00b990 con 0x7f6bbc0adff0 2022-07-20T03:16:13.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.920+0000 7f6bb959a700 1 -- 172.21.15.61:0/4080169579 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bac00bb00 con 0x7f6bbc0adff0 2022-07-20T03:16:13.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/4080169579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 msgr2=0x7f6bbc0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:13.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/4080169579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b5a30 secure :-1 s=READY pgs=1198 cs=0 l=1 rev1=1 rx=0x7f6bbc060b70 tx=0x7f6bac009250).stop 2022-07-20T03:16:13.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/4080169579 shutdown_connections 2022-07-20T03:16:13.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/4080169579 >> v2:172.21.15.117:3300/0 conn(0x7f6bbc0af8e0 0x7f6bbc05bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:13.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/4080169579 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b5a30 unknown :-1 s=CLOSED pgs=1198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:13.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/4080169579 >> 172.21.15.61:0/4080169579 conn(0x7f6bbc0634e0 msgr2=0x7f6bbc0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:13.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/4080169579 shutdown_connections 2022-07-20T03:16:13.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/4080169579 wait complete. 2022-07-20T03:16:13.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 Processor -- start 2022-07-20T03:16:13.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- start start 2022-07-20T03:16:13.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b45d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:13.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bbc0af8e0 0x7f6bbc13d660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:13.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bbc0b5650 0x7f6bbc13db50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:13.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc111660 con 0x7f6bbc0adff0 2022-07-20T03:16:13.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc15a7e0 con 0x7f6bbc0af8e0 2022-07-20T03:16:13.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6bbc110f20 con 0x7f6bbc0b5650 2022-07-20T03:16:13.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b45d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:13.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b45d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49854/0 (socket says 172.21.15.61:49854) 2022-07-20T03:16:13.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 -- 172.21.15.61:0/2327554696 learned_addr learned my addr 172.21.15.61:0/2327554696 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:13.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbad9d700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bbc0b5650 0x7f6bbc13db50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:13.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bb9d9b700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bbc0af8e0 0x7f6bbc13d660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:13.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 -- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bbc0b5650 msgr2=0x7f6bbc13db50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:13.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bbc0b5650 0x7f6bbc13db50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:13.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 -- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bbc0af8e0 msgr2=0x7f6bbc13d660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:13.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bbc0af8e0 0x7f6bbc13d660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:13.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 -- 172.21.15.61:0/2327554696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bac0094d0 con 0x7f6bbc0adff0 2022-07-20T03:16:13.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bba59c700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b45d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f6ba4002f30 tx=0x7f6ba4006d70).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:13.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ba4002320 con 0x7f6bbc0adff0 2022-07-20T03:16:13.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ba4002480 con 0x7f6bbc0adff0 2022-07-20T03:16:13.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ba40027e0 con 0x7f6bbc0adff0 2022-07-20T03:16:13.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6bbc1111c0 con 0x7f6bbc0adff0 2022-07-20T03:16:13.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.924+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6bbc111d60 con 0x7f6bbc0adff0 2022-07-20T03:16:13.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.928+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6bbc111350 con 0x7f6bbc0adff0 2022-07-20T03:16:13.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.928+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6ba400f030 con 0x7f6bbc0adff0 2022-07-20T03:16:13.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.928+0000 7f6bb37fe700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ba00539b0 0x7f6ba0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:13.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.928+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(116..116 src has 1..116) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f6ba406c2e0 con 0x7f6bbc0adff0 2022-07-20T03:16:13.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.928+0000 7f6bb9d9b700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ba00539b0 0x7f6ba0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:13.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.928+0000 7f6bb9d9b700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ba00539b0 0x7f6ba0055e70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f6bac019000 tx=0x7f6bac016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:13.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:13.936+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6ba4043240 con 0x7f6bbc0adff0 2022-07-20T03:16:14.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7f6bbc0574e0 con 0x7f6bbc0adff0 2022-07-20T03:16:14.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bb37fe700 1 -- 172.21.15.61:0/2327554696 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1912 (secure 0 0 0) 0x7f6ba4046860 con 0x7f6bbc0adff0 2022-07-20T03:16:14.108 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:14.109 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:16:14.109 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:14.109 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-20T03:16:14.109 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761052430336, 2022-07-20T03:16:14.110 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6532755456, 2022-07-20T03:16:14.110 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6532890624, 2022-07-20T03:16:14.110 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085109639912843704, 2022-07-20T03:16:14.110 INFO:teuthology.orchestra.run.smithi061.stdout: "num_osds": 8, 2022-07-20T03:16:14.111 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_osds": 0, 2022-07-20T03:16:14.111 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_omap_osds": 0 2022-07-20T03:16:14.111 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:14.111 INFO:teuthology.orchestra.run.smithi061.stdout: "stats_by_class": { 2022-07-20T03:16:14.112 INFO:teuthology.orchestra.run.smithi061.stdout: "ssd": { 2022-07-20T03:16:14.112 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-20T03:16:14.112 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761052430336, 2022-07-20T03:16:14.112 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6532755456, 2022-07-20T03:16:14.112 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6532890624, 2022-07-20T03:16:14.113 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085109639912843704 2022-07-20T03:16:14.113 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:14.113 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:14.113 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:16:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "device_health_metrics", 2022-07-20T03:16:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 1, 2022-07-20T03:16:14.114 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:14.115 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361336307712 2022-07-20T03:16:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:14.116 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "rbd", 2022-07-20T03:16:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 2, 2022-07-20T03:16:14.117 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:14.118 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:14.118 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:14.118 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:14.118 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361336307712 2022-07-20T03:16:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:14.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_data", 2022-07-20T03:16:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 22, 2022-07-20T03:16:14.120 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:14.121 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:14.121 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:14.121 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:14.121 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:14.122 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:14.122 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361336307712 2022-07-20T03:16:14.122 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:14.122 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:14.123 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:14.123 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_metadata", 2022-07-20T03:16:14.123 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 23, 2022-07-20T03:16:14.123 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:14.124 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:14.124 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:14.124 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:14.124 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:14.125 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:14.125 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361336307712 2022-07-20T03:16:14.125 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:14.125 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:14.126 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:14.126 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:16:14.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ba00539b0 msgr2=0x7f6ba0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:14.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ba00539b0 0x7f6ba0055e70 secure :-1 s=READY pgs=576 cs=0 l=1 rev1=1 rx=0x7f6bac019000 tx=0x7f6bac016000).stop 2022-07-20T03:16:14.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 msgr2=0x7f6bbc0b45d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:14.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b45d0 secure :-1 s=READY pgs=1199 cs=0 l=1 rev1=1 rx=0x7f6ba4002f30 tx=0x7f6ba4006d70).stop 2022-07-20T03:16:14.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 shutdown_connections 2022-07-20T03:16:14.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6ba00539b0 0x7f6ba0055e70 unknown :-1 s=CLOSED pgs=576 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:14.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6bbc0af8e0 0x7f6bbc13d660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:14.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6bbc0b5650 0x7f6bbc13db50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:14.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.104+0000 7f6bbb59e700 1 --2- 172.21.15.61:0/2327554696 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6bbc0adff0 0x7f6bbc0b45d0 unknown :-1 s=CLOSED pgs=1199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:14.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.108+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 >> 172.21.15.61:0/2327554696 conn(0x7f6bbc0634e0 msgr2=0x7f6bbc0afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:14.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.108+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 shutdown_connections 2022-07-20T03:16:14.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:14.108+0000 7f6bbb59e700 1 -- 172.21.15.61:0/2327554696 wait complete. 2022-07-20T03:16:14.136 DEBUG:tasks.ceph_test_case:wait_until_true: waiting (timeout=30)... 2022-07-20T03:16:19.138 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph df --format=json-pretty 2022-07-20T03:16:19.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc10670f700 1 Processor -- start 2022-07-20T03:16:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc10670f700 1 -- start start 2022-07-20T03:16:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc10670f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc1000aff80 0x7fc100140240 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc10670f700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1000b5d70 con 0x7fc1000b5990 2022-07-20T03:16:19.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc10670f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc1000651c0 con 0x7fc1000aecf0 2022-07-20T03:16:19.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc10670f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc1000b0690 con 0x7fc1000aff80 2022-07-20T03:16:19.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc1000aff80 0x7fc100140240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc1000aff80 0x7fc100140240 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49866/0 (socket says 172.21.15.61:49866) 2022-07-20T03:16:19.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 -- 172.21.15.61:0/497353253 learned_addr learned my addr 172.21.15.61:0/497353253 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:19.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 -- 172.21.15.61:0/497353253 >> v1:172.21.15.117:6789/0 conn(0x7fc1000aecf0 legacy=0x7fc10005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc0effff700 1 -- 172.21.15.61:0/497353253 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1477152768 0 0) 0x7fc1000651c0 con 0x7fc1000aecf0 2022-07-20T03:16:19.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 -- 172.21.15.61:0/497353253 >> v1:172.21.15.61:6789/0 conn(0x7fc1000b5990 legacy=0x7fc10013fb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 -- 172.21.15.61:0/497353253 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc1000b0930 con 0x7fc1000aff80 2022-07-20T03:16:19.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.248+0000 7fc105f0e700 1 --2- 172.21.15.61:0/497353253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aff80 0x7fc100140240 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7fc0f0001110 tx=0x7fc0f8009290).ready entity=mon.0 client_cookie=e96b5e386a43e766 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc0effff700 1 -- 172.21.15.61:0/497353253 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f800b760 con 0x7fc1000aff80 2022-07-20T03:16:19.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc0effff700 1 -- 172.21.15.61:0/497353253 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f800b8c0 con 0x7fc1000aff80 2022-07-20T03:16:19.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc0effff700 1 -- 172.21.15.61:0/497353253 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f800ba30 con 0x7fc1000aff80 2022-07-20T03:16:19.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 -- 172.21.15.61:0/497353253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aff80 msgr2=0x7fc100140240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 --2- 172.21.15.61:0/497353253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aff80 0x7fc100140240 secure :-1 s=READY pgs=1201 cs=0 l=1 rev1=1 rx=0x7fc0f0001110 tx=0x7fc0f8009290).stop 2022-07-20T03:16:19.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 -- 172.21.15.61:0/497353253 shutdown_connections 2022-07-20T03:16:19.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 --2- 172.21.15.61:0/497353253 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aff80 0x7fc100140240 unknown :-1 s=CLOSED pgs=1201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 -- 172.21.15.61:0/497353253 >> 172.21.15.61:0/497353253 conn(0x7fc1000634e0 msgr2=0x7fc1000b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:19.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 -- 172.21.15.61:0/497353253 shutdown_connections 2022-07-20T03:16:19.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 -- 172.21.15.61:0/497353253 wait complete. 2022-07-20T03:16:19.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 Processor -- start 2022-07-20T03:16:19.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.252+0000 7fc10670f700 1 -- start start 2022-07-20T03:16:19.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 0x7fc100155e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc1000aff80 0x7fc100146250 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1000b5990 0x7fc100152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc100112e00 con 0x7fc1000aecf0 2022-07-20T03:16:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc100155700 con 0x7fc1000b5990 2022-07-20T03:16:19.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc1001559a0 con 0x7fc1000aff80 2022-07-20T03:16:19.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 0x7fc100155e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 0x7fc100155e40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49868/0 (socket says 172.21.15.61:49868) 2022-07-20T03:16:19.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 -- 172.21.15.61:0/3335538586 learned_addr learned my addr 172.21.15.61:0/3335538586 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:19.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc105f0e700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1000b5990 0x7fc100152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc104f0c700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc1000aff80 0x7fc100146250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 -- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc1000aff80 msgr2=0x7fc100146250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc1000aff80 0x7fc100146250 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 -- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1000b5990 msgr2=0x7fc100152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1000b5990 0x7fc100152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 -- 172.21.15.61:0/3335538586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8009530 con 0x7fc1000aecf0 2022-07-20T03:16:19.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc104f0c700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc1000aff80 0x7fc100146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:19.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10570d700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 0x7fc100155e40 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7fc0f0000bc0 tx=0x7fc0f0007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f000ce40 con 0x7fc1000aecf0 2022-07-20T03:16:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc0f0007020 con 0x7fc1000aecf0 2022-07-20T03:16:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc100146bd0 con 0x7fc1000aecf0 2022-07-20T03:16:19.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc0f0007380 con 0x7fc1000aecf0 2022-07-20T03:16:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc100156330 con 0x7fc1000aecf0 2022-07-20T03:16:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc1000574e0 con 0x7fc1000aecf0 2022-07-20T03:16:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.256+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc0f0010070 con 0x7fc1000aecf0 2022-07-20T03:16:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.260+0000 7fc0ee7fc700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0e0053a40 0x7fc0e0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.260+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6075+0+0 (secure 0 0 0) 0x7fc0f006b780 con 0x7fc1000aecf0 2022-07-20T03:16:19.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.260+0000 7fc104f0c700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0e0053a40 0x7fc0e0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.260+0000 7fc104f0c700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0e0053a40 0x7fc0e0055f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fc0fc00b000 tx=0x7fc0fc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.264+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc0f00426e0 con 0x7fc1000aecf0 2022-07-20T03:16:19.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.432+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "df", "format": "json-pretty"} v 0) v1 -- 0x7fc100067ea0 con 0x7fc1000aecf0 2022-07-20T03:16:19.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.432+0000 7fc0ee7fc700 1 -- 172.21.15.61:0/3335538586 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "df", "format": "json-pretty"}]=0 v0) v1 ==== 75+0+1937 (secure 0 0 0) 0x7fc0f0020020 con 0x7fc1000aecf0 2022-07-20T03:16:19.437 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:19.437 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:16:19.438 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:19.438 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-20T03:16:19.438 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761049804800, 2022-07-20T03:16:19.439 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6534832128, 2022-07-20T03:16:19.439 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6535516160, 2022-07-20T03:16:19.439 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085143847391009331, 2022-07-20T03:16:19.439 INFO:teuthology.orchestra.run.smithi061.stdout: "num_osds": 8, 2022-07-20T03:16:19.440 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_osds": 0, 2022-07-20T03:16:19.440 INFO:teuthology.orchestra.run.smithi061.stdout: "num_per_pool_omap_osds": 0 2022-07-20T03:16:19.441 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:19.441 INFO:teuthology.orchestra.run.smithi061.stdout: "stats_by_class": { 2022-07-20T03:16:19.441 INFO:teuthology.orchestra.run.smithi061.stdout: "ssd": { 2022-07-20T03:16:19.441 INFO:teuthology.orchestra.run.smithi061.stdout: "total_bytes": 767585320960, 2022-07-20T03:16:19.443 INFO:teuthology.orchestra.run.smithi061.stdout: "total_avail_bytes": 761049804800, 2022-07-20T03:16:19.444 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_bytes": 6534832128, 2022-07-20T03:16:19.444 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_bytes": 6535516160, 2022-07-20T03:16:19.444 INFO:teuthology.orchestra.run.smithi061.stdout: "total_used_raw_ratio": 0.0085143847391009331 2022-07-20T03:16:19.444 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:19.445 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:19.445 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:16:19.445 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:19.445 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "device_health_metrics", 2022-07-20T03:16:19.445 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 1, 2022-07-20T03:16:19.446 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:19.446 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:19.446 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:19.446 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:19.447 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:19.447 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:19.450 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335128064 2022-07-20T03:16:19.450 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:19.451 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:19.451 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:19.452 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "rbd", 2022-07-20T03:16:19.452 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 2, 2022-07-20T03:16:19.452 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:19.453 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:19.453 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:19.454 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:19.454 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:19.455 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:19.455 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335128064 2022-07-20T03:16:19.455 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:19.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:19.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:19.457 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_data", 2022-07-20T03:16:19.457 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 22, 2022-07-20T03:16:19.458 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:19.458 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 0, 2022-07-20T03:16:19.459 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 0, 2022-07-20T03:16:19.460 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 0, 2022-07-20T03:16:19.460 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 0, 2022-07-20T03:16:19.461 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 0, 2022-07-20T03:16:19.461 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335128064 2022-07-20T03:16:19.462 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:19.462 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:19.463 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:19.463 INFO:teuthology.orchestra.run.smithi061.stdout: "name": "cephfs_metadata", 2022-07-20T03:16:19.464 INFO:teuthology.orchestra.run.smithi061.stdout: "id": 23, 2022-07-20T03:16:19.464 INFO:teuthology.orchestra.run.smithi061.stdout: "stats": { 2022-07-20T03:16:19.469 INFO:teuthology.orchestra.run.smithi061.stdout: "stored": 617, 2022-07-20T03:16:19.472 INFO:teuthology.orchestra.run.smithi061.stdout: "objects": 1, 2022-07-20T03:16:19.472 INFO:teuthology.orchestra.run.smithi061.stdout: "kb_used": 1, 2022-07-20T03:16:19.472 INFO:teuthology.orchestra.run.smithi061.stdout: "bytes_used": 617, 2022-07-20T03:16:19.473 INFO:teuthology.orchestra.run.smithi061.stdout: "percent_used": 8.5377804825981229e-10, 2022-07-20T03:16:19.473 INFO:teuthology.orchestra.run.smithi061.stdout: "max_avail": 361335128064 2022-07-20T03:16:19.473 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:19.473 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:19.474 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:19.474 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:16:19.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.432+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0e0053a40 msgr2=0x7fc0e0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.432+0000 7fc10670f700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0e0053a40 0x7fc0e0055f00 secure :-1 s=READY pgs=577 cs=0 l=1 rev1=1 rx=0x7fc0fc00b000 tx=0x7fc0fc009040).stop 2022-07-20T03:16:19.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.432+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 msgr2=0x7fc100155e40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.432+0000 7fc10670f700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 0x7fc100155e40 secure :-1 s=READY pgs=1202 cs=0 l=1 rev1=1 rx=0x7fc0f0000bc0 tx=0x7fc0f0007bf0).stop 2022-07-20T03:16:19.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 shutdown_connections 2022-07-20T03:16:19.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc0e0053a40 0x7fc0e0055f00 unknown :-1 s=CLOSED pgs=577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc1000b5990 0x7fc100152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc1000aff80 0x7fc100146250 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 --2- 172.21.15.61:0/3335538586 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc1000aecf0 0x7fc100155e40 unknown :-1 s=CLOSED pgs=1202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 >> 172.21.15.61:0/3335538586 conn(0x7fc1000634e0 msgr2=0x7fc100145a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:19.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 shutdown_connections 2022-07-20T03:16:19.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.436+0000 7fc10670f700 1 -- 172.21.15.61:0/3335538586 wait complete. 2022-07-20T03:16:19.487 DEBUG:tasks.ceph_test_case:wait_until_true: success in 5s 2022-07-20T03:16:19.488 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-20T03:16:19.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.596+0000 7feba1c5c700 1 Processor -- start 2022-07-20T03:16:19.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.596+0000 7feba1c5c700 1 -- start start 2022-07-20T03:16:19.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb9c0b59f0 0x7feb9c1473a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c05cc70 con 0x7feb9c0af940 2022-07-20T03:16:19.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feb9c0b5dd0 con 0x7feb9c0b1400 2022-07-20T03:16:19.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feb9c0b1070 con 0x7feb9c0b59f0 2022-07-20T03:16:19.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba0c5a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7feb9c0b59f0 0x7feb9c1473a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feb9bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feb9c0af940 0x7feb9c14a3c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46474/0 (socket says 172.21.15.61:46474) 2022-07-20T03:16:19.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feb9bfff700 1 -- 172.21.15.61:0/19839495 learned_addr learned my addr 172.21.15.61:0/19839495 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:19.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba0c5a700 1 -- 172.21.15.61:0/19839495 >> v1:172.21.15.117:6789/0 conn(0x7feb9c0b1400 legacy=0x7feb9c14aad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba0c5a700 1 -- 172.21.15.61:0/19839495 >> v1:172.21.15.61:6789/0 conn(0x7feb9c0af940 legacy=0x7feb9c14a3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba0c5a700 1 -- 172.21.15.61:0/19839495 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb9c05c760 con 0x7feb9c0b59f0 2022-07-20T03:16:19.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feb9b7fe700 1 -- 172.21.15.61:0/19839495 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4077657409 0 0) 0x7feb9c05cc70 con 0x7feb9c0af940 2022-07-20T03:16:19.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba0c5a700 1 --2- 172.21.15.61:0/19839495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b59f0 0x7feb9c1473a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7feb940032f0 tx=0x7feb940091d0).ready entity=mon.0 client_cookie=1004d44f1a0a5fa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feb9b7fe700 1 -- 172.21.15.61:0/19839495 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb94009ea0 con 0x7feb9c0b59f0 2022-07-20T03:16:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feb9b7fe700 1 -- 172.21.15.61:0/19839495 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb9400b8a0 con 0x7feb9c0b59f0 2022-07-20T03:16:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feb9b7fe700 1 -- 172.21.15.61:0/19839495 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb9400ba10 con 0x7feb9c0b59f0 2022-07-20T03:16:19.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- 172.21.15.61:0/19839495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b59f0 msgr2=0x7feb9c1473a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 --2- 172.21.15.61:0/19839495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b59f0 0x7feb9c1473a0 secure :-1 s=READY pgs=1204 cs=0 l=1 rev1=1 rx=0x7feb940032f0 tx=0x7feb940091d0).stop 2022-07-20T03:16:19.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- 172.21.15.61:0/19839495 shutdown_connections 2022-07-20T03:16:19.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 --2- 172.21.15.61:0/19839495 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b59f0 0x7feb9c1473a0 unknown :-1 s=CLOSED pgs=1204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- 172.21.15.61:0/19839495 >> 172.21.15.61:0/19839495 conn(0x7feb9c063480 msgr2=0x7feb9c0b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:19.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- 172.21.15.61:0/19839495 shutdown_connections 2022-07-20T03:16:19.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.600+0000 7feba1c5c700 1 -- 172.21.15.61:0/19839495 wait complete. 2022-07-20T03:16:19.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.604+0000 7feba1c5c700 1 Processor -- start 2022-07-20T03:16:19.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.604+0000 7feba1c5c700 1 -- start start 2022-07-20T03:16:19.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb9c0af940 0x7feb9c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b1400 0x7feb9c159b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb9c0b59f0 0x7feb9c111a50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c10e190 con 0x7feb9c0b1400 2022-07-20T03:16:19.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c10f3f0 con 0x7feb9c0af940 2022-07-20T03:16:19.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b1400 0x7feb9c159b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feb9c10f150 con 0x7feb9c0b59f0 2022-07-20T03:16:19.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b1400 0x7feb9c159b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49884/0 (socket says 172.21.15.61:49884) 2022-07-20T03:16:19.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 -- 172.21.15.61:0/978411879 learned_addr learned my addr 172.21.15.61:0/978411879 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:19.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba0c5a700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb9c0af940 0x7feb9c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba145b700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb9c0b59f0 0x7feb9c111a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 -- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb9c0b59f0 msgr2=0x7feb9c111a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feb9c0b59f0 0x7feb9c111a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 -- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb9c0af940 msgr2=0x7feb9c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb9c0af940 0x7feb9c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 -- 172.21.15.61:0/978411879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb94009470 con 0x7feb9c0b1400 2022-07-20T03:16:19.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba0c5a700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feb9c0af940 0x7feb9c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:19.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb9bfff700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feb9c0b1400 0x7feb9c159b70 secure :-1 s=READY pgs=1205 cs=0 l=1 rev1=1 rx=0x7feb9c10f580 tx=0x7feb84007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb8400ce40 con 0x7feb9c0b1400 2022-07-20T03:16:19.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feb840072e0 con 0x7feb9c0b1400 2022-07-20T03:16:19.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 -- 172.21.15.61:0/978411879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb9c10e3c0 con 0x7feb9c0b1400 2022-07-20T03:16:19.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feb84005800 con 0x7feb9c0b1400 2022-07-20T03:16:19.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 -- 172.21.15.61:0/978411879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb9c110230 con 0x7feb9c0b1400 2022-07-20T03:16:19.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feb84010070 con 0x7feb9c0b1400 2022-07-20T03:16:19.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.608+0000 7feba1c5c700 1 -- 172.21.15.61:0/978411879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb9c067970 con 0x7feb9c0b1400 2022-07-20T03:16:19.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.612+0000 7feb99ffb700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb80053a60 0x7feb80055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.612+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6075+0+0 (secure 0 0 0) 0x7feb8406b670 con 0x7feb9c0b1400 2022-07-20T03:16:19.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.612+0000 7feba0c5a700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb80053a60 0x7feb80055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.612+0000 7feba0c5a700 1 --2- 172.21.15.61:0/978411879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7feb80053a60 0x7feb80055f20 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7feb94019000 tx=0x7feb94016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.616+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feb840424d0 con 0x7feb9c0b1400 2022-07-20T03:16:19.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.784+0000 7feba1c5c700 1 -- 172.21.15.61:0/978411879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7feb9c05c9d0 con 0x7feb9c0b1400 2022-07-20T03:16:19.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.784+0000 7feb99ffb700 1 -- 172.21.15.61:0/978411879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=-22 pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. v64) v1 ==== 209+0+0 (secure 0 0 0) 0x7feb84045af0 con 0x7feb9c0b1400 2022-07-20T03:16:19.788 INFO:teuthology.orchestra.run.smithi061.stderr:Error EINVAL: pool 'cephfs_metadata' already contains some objects. Use an empty pool instead. 2022-07-20T03:16:19.792 DEBUG:teuthology.orchestra.run:got remote process result: 22 2022-07-20T03:16:19.794 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:16:19.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 Processor -- start 2022-07-20T03:16:19.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 -- start start 2022-07-20T03:16:19.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 0x7f8ed0062f30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed00b3880 0x7f8ed00b24d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8ed00b29c0 0x7f8ed013a060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed00b1e00 con 0x7f8ed005cf40 2022-07-20T03:16:19.924 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8ed00add50 con 0x7f8ed00b29c0 2022-07-20T03:16:19.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed7c7f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed00adff0 con 0x7f8ed00b3880 2022-07-20T03:16:19.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed5a1b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 0x7f8ed0062f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed5a1b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 0x7f8ed0062f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49890/0 (socket says 172.21.15.61:49890) 2022-07-20T03:16:19.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed621c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8ed00b29c0 0x7f8ed013a060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed5a1b700 1 -- 172.21.15.61:0/1971687364 learned_addr learned my addr 172.21.15.61:0/1971687364 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:19.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.900+0000 7f8ed521a700 1 --2- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed00b3880 0x7f8ed00b24d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed621c700 1 -- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed00b3880 msgr2=0x7f8ed00b24d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed621c700 1 --2- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed00b3880 0x7f8ed00b24d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed621c700 1 -- 172.21.15.61:0/1971687364 >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 msgr2=0x7f8ed0062f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed621c700 1 --2- 172.21.15.61:0/1971687364 >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 0x7f8ed0062f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed621c700 1 -- 172.21.15.61:0/1971687364 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ed005d320 con 0x7f8ed00b29c0 2022-07-20T03:16:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed5a1b700 1 --2- 172.21.15.61:0/1971687364 >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 0x7f8ed0062f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed621c700 1 --2- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b29c0 0x7f8ed013a060 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f8ec80032f0 tx=0x7f8ec80091d0).ready entity=mon.1 client_cookie=c0a0d945fdb0f650 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed4a19700 1 -- 172.21.15.61:0/1971687364 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ec800b600 con 0x7f8ed00b29c0 2022-07-20T03:16:19.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed4a19700 1 -- 172.21.15.61:0/1971687364 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ec800b760 con 0x7f8ed00b29c0 2022-07-20T03:16:19.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed4a19700 1 -- 172.21.15.61:0/1971687364 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ec800b8d0 con 0x7f8ed00b29c0 2022-07-20T03:16:19.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b29c0 msgr2=0x7f8ed013a060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b29c0 0x7f8ed013a060 secure :-1 s=READY pgs=546 cs=0 l=1 rev1=1 rx=0x7f8ec80032f0 tx=0x7f8ec80091d0).stop 2022-07-20T03:16:19.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/1971687364 shutdown_connections 2022-07-20T03:16:19.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b29c0 0x7f8ed013a060 unknown :-1 s=CLOSED pgs=546 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/1971687364 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed00b3880 0x7f8ed00b24d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/1971687364 >> v2:172.21.15.61:3300/0 conn(0x7f8ed005cf40 0x7f8ed0062f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/1971687364 >> 172.21.15.61:0/1971687364 conn(0x7f8ed00b7d00 msgr2=0x7f8ed00b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:19.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/1971687364 shutdown_connections 2022-07-20T03:16:19.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.904+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/1971687364 wait complete. 2022-07-20T03:16:19.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 Processor -- start 2022-07-20T03:16:19.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 -- start start 2022-07-20T03:16:19.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed005cf40 0x7f8ed013dcd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 0x7f8ed009ba90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b3880 0x7f8ed013b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed013e300 con 0x7f8ed00b29c0 2022-07-20T03:16:19.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed014fdb0 con 0x7f8ed00b3880 2022-07-20T03:16:19.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed7c7f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8ed0150050 con 0x7f8ed005cf40 2022-07-20T03:16:19.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 0x7f8ed009ba90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 0x7f8ed009ba90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49898/0 (socket says 172.21.15.61:49898) 2022-07-20T03:16:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 -- 172.21.15.61:0/3455335401 learned_addr learned my addr 172.21.15.61:0/3455335401 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:19.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed621c700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b3880 0x7f8ed013b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed5a1b700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed005cf40 0x7f8ed013dcd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 -- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed005cf40 msgr2=0x7f8ed013dcd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed005cf40 0x7f8ed013dcd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 -- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b3880 msgr2=0x7f8ed013b070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:19.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b3880 0x7f8ed013b070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:19.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 -- 172.21.15.61:0/3455335401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ec000b1e0 con 0x7f8ed00b29c0 2022-07-20T03:16:19.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.908+0000 7f8ed521a700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 0x7f8ed009ba90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f8ecc00c000 tx=0x7f8ecc006d90).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ecc011750 con 0x7f8ed00b29c0 2022-07-20T03:16:19.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ecc0118b0 con 0x7f8ed00b29c0 2022-07-20T03:16:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ec8009470 con 0x7f8ed00b29c0 2022-07-20T03:16:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ecc011c10 con 0x7f8ed00b29c0 2022-07-20T03:16:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ed013d6e0 con 0x7f8ed00b29c0 2022-07-20T03:16:19.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8ecc011d70 con 0x7f8ed00b29c0 2022-07-20T03:16:19.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ed0150210 con 0x7f8ed00b29c0 2022-07-20T03:16:19.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ebeffd700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8eb00539d0 0x7f8eb0055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:19.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(117..117 src has 1..117) v4 ==== 6075+0+0 (secure 0 0 0) 0x7f8ecc06bbc0 con 0x7f8ed00b29c0 2022-07-20T03:16:19.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.912+0000 7f8ed5a1b700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8eb00539d0 0x7f8eb0055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:19.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.916+0000 7f8ed5a1b700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8eb00539d0 0x7f8eb0055e90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f8ec000b000 tx=0x7f8ec0006eb0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:19.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:19.920+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8ecc042a20 con 0x7f8ed00b29c0 2022-07-20T03:16:20.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.084+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7f8ed014fbe0 con 0x7f8ed00b29c0 2022-07-20T03:16:20.089 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:16:20.084+0000 7f2534a18700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:16:20.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.092+0000 7f8ebeffd700 1 -- 172.21.15.61:0/3455335401 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 23 and data pool 22 v65) v1 ==== 189+0+0 (secure 0 0 0) 0x7f8ecc046040 con 0x7f8ed00b29c0 2022-07-20T03:16:20.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.092+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8eb00539d0 msgr2=0x7f8eb0055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.092+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8eb00539d0 0x7f8eb0055e90 secure :-1 s=READY pgs=579 cs=0 l=1 rev1=1 rx=0x7f8ec000b000 tx=0x7f8ec0006eb0).stop 2022-07-20T03:16:20.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 msgr2=0x7f8ed009ba90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 0x7f8ed009ba90 secure :-1 s=READY pgs=1206 cs=0 l=1 rev1=1 rx=0x7f8ecc00c000 tx=0x7f8ecc006d90).stop 2022-07-20T03:16:20.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 shutdown_connections 2022-07-20T03:16:20.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8eb00539d0 0x7f8eb0055e90 unknown :-1 s=CLOSED pgs=579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8ed00b3880 0x7f8ed013b070 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8ed00b29c0 0x7f8ed009ba90 unknown :-1 s=CLOSED pgs=1206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 --2- 172.21.15.61:0/3455335401 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8ed005cf40 0x7f8ed013dcd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 >> 172.21.15.61:0/3455335401 conn(0x7f8ed00b7d00 msgr2=0x7f8ed00b5160 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:20.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 shutdown_connections 2022-07-20T03:16:20.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.096+0000 7f8ed7c7f700 1 -- 172.21.15.61:0/3455335401 wait complete. 2022-07-20T03:16:20.108 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 23 and data pool 22 2022-07-20T03:16:20.112 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:16:20.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.216+0000 7f4ca22fc700 1 Processor -- start 2022-07-20T03:16:20.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.216+0000 7f4ca22fc700 1 -- start start 2022-07-20T03:16:20.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4c9c0b2170 0x7f4c9c0b51e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:20.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c0b5020 con 0x7f4c9c0b2550 2022-07-20T03:16:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4c9c0adad0 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c9c0b3640 con 0x7f4c9c0b2170 2022-07-20T03:16:20.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca0af9700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4c9c0b2550 0x7f4c9c0aeae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46504/0 (socket says 172.21.15.61:46504) 2022-07-20T03:16:20.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca0af9700 1 -- 172.21.15.61:0/2111538109 learned_addr learned my addr 172.21.15.61:0/2111538109 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:20.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca12fa700 1 --2- 172.21.15.61:0/2111538109 >> v2:172.21.15.117:3300/0 conn(0x7f4c9c0b2170 0x7f4c9c0b51e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:20.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4c9bfff700 1 -- 172.21.15.61:0/2111538109 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4072573863 0 0) 0x7f4c9c0b5020 con 0x7f4c9c0b2550 2022-07-20T03:16:20.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4c9bfff700 1 -- 172.21.15.61:0/2111538109 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4c88003260 con 0x7f4c9c0b2550 2022-07-20T03:16:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca12fa700 1 -- 172.21.15.61:0/2111538109 >> v1:172.21.15.117:6789/0 conn(0x7f4c9c0b3d40 legacy=0x7f4c9c13e5a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca12fa700 1 -- 172.21.15.61:0/2111538109 >> v1:172.21.15.61:6789/0 conn(0x7f4c9c0b2550 legacy=0x7f4c9c0aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca12fa700 1 -- 172.21.15.61:0/2111538109 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c9c0add00 con 0x7f4c9c0b2170 2022-07-20T03:16:20.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca12fa700 1 --2- 172.21.15.61:0/2111538109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2170 0x7f4c9c0b51e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f4c9c0b3c20 tx=0x7f4c8c0091d0).ready entity=mon.1 client_cookie=f69941edb50da7e2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:20.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4c9bfff700 1 -- 172.21.15.61:0/2111538109 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8c00f8a0 con 0x7f4c9c0b2170 2022-07-20T03:16:20.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4c9bfff700 1 -- 172.21.15.61:0/2111538109 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8c00fa00 con 0x7f4c9c0b2170 2022-07-20T03:16:20.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4c9bfff700 1 -- 172.21.15.61:0/2111538109 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8c00fd60 con 0x7f4c9c0b2170 2022-07-20T03:16:20.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- 172.21.15.61:0/2111538109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2170 msgr2=0x7f4c9c0b51e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/2111538109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2170 0x7f4c9c0b51e0 secure :-1 s=READY pgs=548 cs=0 l=1 rev1=1 rx=0x7f4c9c0b3c20 tx=0x7f4c8c0091d0).stop 2022-07-20T03:16:20.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- 172.21.15.61:0/2111538109 shutdown_connections 2022-07-20T03:16:20.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/2111538109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2170 0x7f4c9c0b51e0 unknown :-1 s=CLOSED pgs=548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- 172.21.15.61:0/2111538109 >> 172.21.15.61:0/2111538109 conn(0x7f4c9c063150 msgr2=0x7f4c9c063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- 172.21.15.61:0/2111538109 shutdown_connections 2022-07-20T03:16:20.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.220+0000 7f4ca22fc700 1 -- 172.21.15.61:0/2111538109 wait complete. 2022-07-20T03:16:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 Processor -- start 2022-07-20T03:16:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- start start 2022-07-20T03:16:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c9c0b2170 0x7f4c9c066be0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:20.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2550 0x7f4c9c142910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:20.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 0x7f4c9c064450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:20.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c142e00 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c064c30 con 0x7f4c9c0b2550 2022-07-20T03:16:20.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c9c064e60 con 0x7f4c9c0b2170 2022-07-20T03:16:20.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 0x7f4c9c064450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:20.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 0x7f4c9c064450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49914/0 (socket says 172.21.15.61:49914) 2022-07-20T03:16:20.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 -- 172.21.15.61:0/3906343474 learned_addr learned my addr 172.21.15.61:0/3906343474 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:20.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca12fa700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c9c0b2170 0x7f4c9c066be0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:20.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca0af9700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2550 0x7f4c9c142910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:20.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 -- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c9c0b2170 msgr2=0x7f4c9c066be0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c9c0b2170 0x7f4c9c066be0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 -- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2550 msgr2=0x7f4c9c142910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:20.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2550 0x7f4c9c142910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:20.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 -- 172.21.15.61:0/3906343474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c009470 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca1afb700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 0x7f4c9c064450 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f4c9c0b3530 tx=0x7f4c94007c20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:20.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c94007030 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c94007190 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c9c065100 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c94006510 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c9c0655a0 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.224+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c9c0b38a0 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.228+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4c94010030 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.228+0000 7f4c9a7fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c7c0539b0 0x7f4c7c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:20.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.228+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f4c9406b9f0 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.228+0000 7f4ca12fa700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c7c0539b0 0x7f4c7c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:20.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.228+0000 7f4ca12fa700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c7c0539b0 0x7f4c7c055e70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f4c8c01b000 tx=0x7f4c8c018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:20.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.236+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c9406de60 con 0x7f4c9c0b3d40 2022-07-20T03:16:20.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:20.400+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f4c9c065290 con 0x7f4c9c0b3d40 2022-07-20T03:16:21.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.308+0000 7f4c9a7fc700 1 -- 172.21.15.61:0/3906343474 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v66) v1 ==== 104+0+0 (secure 0 0 0) 0x7f4c94042900 con 0x7f4c9c0b3d40 2022-07-20T03:16:21.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c7c0539b0 msgr2=0x7f4c7c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c7c0539b0 0x7f4c7c055e70 secure :-1 s=READY pgs=580 cs=0 l=1 rev1=1 rx=0x7f4c8c01b000 tx=0x7f4c8c018000).stop 2022-07-20T03:16:21.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 msgr2=0x7f4c9c064450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 0x7f4c9c064450 secure :-1 s=READY pgs=1208 cs=0 l=1 rev1=1 rx=0x7f4c9c0b3530 tx=0x7f4c94007c20).stop 2022-07-20T03:16:21.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 shutdown_connections 2022-07-20T03:16:21.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c7c0539b0 0x7f4c7c055e70 unknown :-1 s=CLOSED pgs=580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c9c0b2550 0x7f4c9c142910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c9c0b3d40 0x7f4c9c064450 unknown :-1 s=CLOSED pgs=1208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 --2- 172.21.15.61:0/3906343474 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c9c0b2170 0x7f4c9c066be0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 >> 172.21.15.61:0/3906343474 conn(0x7f4c9c063150 msgr2=0x7f4c9c05f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:21.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 shutdown_connections 2022-07-20T03:16:21.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.312+0000 7f4ca22fc700 1 -- 172.21.15.61:0/3906343474 wait complete. 2022-07-20T03:16:21.333 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:16:21.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e21957700 1 Processor -- start 2022-07-20T03:16:21.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e21957700 1 -- start start 2022-07-20T03:16:21.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e21957700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 0x7f9e1c146200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e21957700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c0b2370 con 0x7f9e1c066650 2022-07-20T03:16:21.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e21957700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e1c0b2610 con 0x7f9e1c0b1330 2022-07-20T03:16:21.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e21957700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c0b1f00 con 0x7f9e1c05cf40 2022-07-20T03:16:21.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e1a7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9e1c066650 0x7f9e1c0b0c20 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46520/0 (socket says 172.21.15.61:46520) 2022-07-20T03:16:21.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.452+0000 7f9e1a7fc700 1 -- 172.21.15.61:0/2197631375 learned_addr learned my addr 172.21.15.61:0/2197631375 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:21.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1affd700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 0x7f9e1c146200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3908702219 0 0) 0x7f9e1c0b2370 con 0x7f9e1c066650 2022-07-20T03:16:21.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e00003260 con 0x7f9e1c066650 2022-07-20T03:16:21.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1009086343 0 0) 0x7f9e1c0b2610 con 0x7f9e1c0b1330 2022-07-20T03:16:21.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e1c0b2370 con 0x7f9e1c0b1330 2022-07-20T03:16:21.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3496454197 0 0) 0x7f9e00003260 con 0x7f9e1c066650 2022-07-20T03:16:21.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 msgr2=0x7f9e1c146200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:21.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 0x7f9e1c146200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 >> v1:172.21.15.117:6789/0 conn(0x7f9e1c0b1330 legacy=0x7f9e1c1420d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c0b0780 con 0x7f9e1c066650 2022-07-20T03:16:21.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f9e0c001660 con 0x7f9e1c066650 2022-07-20T03:16:21.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 >> v1:172.21.15.61:6789/0 conn(0x7f9e1c066650 legacy=0x7f9e1c0b0c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 0x7f9e00003910 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e00003e20 0x7f9e00004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e000049f0 0x7f9e00004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0000b800 con 0x7f9e00003e20 2022-07-20T03:16:21.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0000b940 con 0x7f9e000049f0 2022-07-20T03:16:21.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0000bb00 con 0x7f9e1c05cf40 2022-07-20T03:16:21.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1a7fc700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e00003e20 0x7f9e00004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e000049f0 0x7f9e00004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1affd700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 0x7f9e00003910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 msgr2=0x7f9e00003910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:21.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e1c05cf40 0x7f9e00003910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/2197631375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e00003e20 msgr2=0x7f9e00004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:21.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e00003e20 0x7f9e00004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/2197631375 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e00003260 con 0x7f9e000049f0 2022-07-20T03:16:21.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1a7fc700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e00003e20 0x7f9e00004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:21.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e000049f0 0x7f9e00004e70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f9e14001490 tx=0x7f9e14007d80).ready entity=mon.1 client_cookie=1e465306bcd0f5bb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:21.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e14001780 con 0x7f9e000049f0 2022-07-20T03:16:21.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e14006890 con 0x7f9e000049f0 2022-07-20T03:16:21.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e19ffb700 1 -- 172.21.15.61:0/2197631375 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e14006bf0 con 0x7f9e000049f0 2022-07-20T03:16:21.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e21957700 1 -- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e000049f0 msgr2=0x7f9e00004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e21957700 1 --2- 172.21.15.61:0/2197631375 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e000049f0 0x7f9e00004e70 secure :-1 s=READY pgs=550 cs=0 l=1 rev1=1 rx=0x7f9e14001490 tx=0x7f9e14007d80).stop 2022-07-20T03:16:21.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/2197631375 reap_dead start 2022-07-20T03:16:21.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e21957700 1 -- 172.21.15.61:0/2197631375 shutdown_connections 2022-07-20T03:16:21.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e21957700 1 -- 172.21.15.61:0/2197631375 >> 172.21.15.61:0/2197631375 conn(0x7f9e1c0b7cc0 msgr2=0x7f9e1c0b3cf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:21.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e21957700 1 -- 172.21.15.61:0/2197631375 shutdown_connections 2022-07-20T03:16:21.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.456+0000 7f9e21957700 1 -- 172.21.15.61:0/2197631375 wait complete. 2022-07-20T03:16:21.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 Processor -- start 2022-07-20T03:16:21.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 -- start start 2022-07-20T03:16:21.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000049f0 0x7f9e1c090040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e1c151ce0 0x7f9e1c097410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 0x7f9e1c09c7d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c090f60 con 0x7f9e1c151ce0 2022-07-20T03:16:21.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c1521f0 con 0x7f9e1c09c320 2022-07-20T03:16:21.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e1c152490 con 0x7f9e000049f0 2022-07-20T03:16:21.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 0x7f9e1c09c7d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 0x7f9e1c09c7d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59612/0 (socket says 172.21.15.61:59612) 2022-07-20T03:16:21.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/1324429766 learned_addr learned my addr 172.21.15.61:0/1324429766 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:21.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e1c151ce0 0x7f9e1c097410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1affd700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000049f0 0x7f9e1c090040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000049f0 msgr2=0x7f9e1c090040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000049f0 0x7f9e1c090040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/1324429766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e1c151ce0 msgr2=0x7f9e1c097410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:21.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e1c151ce0 0x7f9e1c097410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:21.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 -- 172.21.15.61:0/1324429766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e1c152730 con 0x7f9e1c09c320 2022-07-20T03:16:21.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e1b7fe700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 0x7f9e1c09c7d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f9e1c090530 tx=0x7f9e14016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:21.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e14001780 con 0x7f9e1c09c320 2022-07-20T03:16:21.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9e1400ced0 con 0x7f9e1c09c320 2022-07-20T03:16:21.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9e1400c3f0 con 0x7f9e1c09c320 2022-07-20T03:16:21.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e1c096820 con 0x7f9e1c09c320 2022-07-20T03:16:21.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.460+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e1c09ccc0 con 0x7f9e1c09c320 2022-07-20T03:16:21.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.464+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e1c1528c0 con 0x7f9e1c09c320 2022-07-20T03:16:21.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.464+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9e14025030 con 0x7f9e1c09c320 2022-07-20T03:16:21.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.464+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9df8053af0 0x7f9df8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:21.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.464+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(118..118 src has 1..118) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9e1406cff0 con 0x7f9e1c09c320 2022-07-20T03:16:21.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.464+0000 7f9e1affd700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9df8053af0 0x7f9df8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:21.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.464+0000 7f9e1affd700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9df8053af0 0x7f9df8055fb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f9e1000b000 tx=0x7f9e10006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:21.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.472+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9e14043e10 con 0x7f9e1c09c320 2022-07-20T03:16:21.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:21.600+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9e1c0574e0 con 0x7f9e1c09c320 2022-07-20T03:16:22.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.316+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1324429766 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v119) v1 ==== 185+0+0 (secure 0 0 0) 0x7f9e14023070 con 0x7f9e1c09c320 2022-07-20T03:16:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9df8053af0 msgr2=0x7f9df8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9df8053af0 0x7f9df8055fb0 secure :-1 s=READY pgs=581 cs=0 l=1 rev1=1 rx=0x7f9e1000b000 tx=0x7f9e10006df0).stop 2022-07-20T03:16:22.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 msgr2=0x7f9e1c09c7d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:22.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 0x7f9e1c09c7d0 secure :-1 s=READY pgs=551 cs=0 l=1 rev1=1 rx=0x7f9e1c090530 tx=0x7f9e14016000).stop 2022-07-20T03:16:22.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 shutdown_connections 2022-07-20T03:16:22.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9df8053af0 0x7f9df8055fb0 unknown :-1 s=CLOSED pgs=581 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e1c09c320 0x7f9e1c09c7d0 unknown :-1 s=CLOSED pgs=551 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e1c151ce0 0x7f9e1c097410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 --2- 172.21.15.61:0/1324429766 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e000049f0 0x7f9e1c090040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 >> 172.21.15.61:0/1324429766 conn(0x7f9e1c0b7cc0 msgr2=0x7f9e1c146110 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:22.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 shutdown_connections 2022-07-20T03:16:22.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.320+0000 7f9e21957700 1 -- 172.21.15.61:0/1324429766 wait complete. 2022-07-20T03:16:22.331 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:16:22.342 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:16:22.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 Processor -- start 2022-07-20T03:16:22.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 -- start start 2022-07-20T03:16:22.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe0139440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:22.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7fe0140070 0x7f7fe00acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:22.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7fe00b5dd0 con 0x7f7fe0139930 2022-07-20T03:16:22.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7fe00ae1f0 con 0x7f7fe0140070 2022-07-20T03:16:22.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.452+0000 7f7fe6b29700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe00ad6e0 con 0x7f7fe00b59f0 2022-07-20T03:16:22.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5326700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7fe0140070 0x7f7fe00acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:22.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5b27700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe0139440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:22.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5326700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7fe0140070 0x7f7fe00acae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59620/0 (socket says 172.21.15.61:59620) 2022-07-20T03:16:22.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5326700 1 -- 172.21.15.61:0/351930638 learned_addr learned my addr 172.21.15.61:0/351930638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:22.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5b27700 1 -- 172.21.15.61:0/351930638 >> v2:172.21.15.117:3300/0 conn(0x7f7fe0140070 msgr2=0x7f7fe00acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:22.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5b27700 1 --2- 172.21.15.61:0/351930638 >> v2:172.21.15.117:3300/0 conn(0x7f7fe0140070 0x7f7fe00acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5b27700 1 -- 172.21.15.61:0/351930638 >> v1:172.21.15.117:6789/0 conn(0x7f7fe0139930 legacy=0x7f7fe00acfd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:22.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5b27700 1 -- 172.21.15.61:0/351930638 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fe0140500 con 0x7f7fe00b59f0 2022-07-20T03:16:22.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe5b27700 1 --2- 172.21.15.61:0/351930638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe0139440 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7fd0002f70 tx=0x7f7fd0006e80).ready entity=mon.2 client_cookie=c8bbf466346835e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:22.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe4b25700 1 -- 172.21.15.61:0/351930638 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd000b7d0 con 0x7f7fe00b59f0 2022-07-20T03:16:22.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe4b25700 1 -- 172.21.15.61:0/351930638 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fd000b930 con 0x7f7fe00b59f0 2022-07-20T03:16:22.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe4b25700 1 -- 172.21.15.61:0/351930638 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fd000bc90 con 0x7f7fe00b59f0 2022-07-20T03:16:22.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 -- 172.21.15.61:0/351930638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 msgr2=0x7f7fe0139440 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:22.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/351930638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe0139440 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f7fd0002f70 tx=0x7f7fd0006e80).stop 2022-07-20T03:16:22.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 -- 172.21.15.61:0/351930638 shutdown_connections 2022-07-20T03:16:22.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/351930638 >> v2:172.21.15.117:3300/0 conn(0x7f7fe0140070 0x7f7fe00acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/351930638 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe0139440 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 -- 172.21.15.61:0/351930638 >> 172.21.15.61:0/351930638 conn(0x7f7fe0063480 msgr2=0x7f7fe00b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:22.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 -- 172.21.15.61:0/351930638 shutdown_connections 2022-07-20T03:16:22.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.456+0000 7f7fe6b29700 1 -- 172.21.15.61:0/351930638 wait complete. 2022-07-20T03:16:22.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 Processor -- start 2022-07-20T03:16:22.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 -- start start 2022-07-20T03:16:22.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe00654f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:22.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 0x7f7fe013d690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:22.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fe01530c0 0x7f7fe0136ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:22.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe01526e0 con 0x7f7fe0139930 2022-07-20T03:16:22.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0152980 con 0x7f7fe01530c0 2022-07-20T03:16:22.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7fe0152c20 con 0x7f7fe00b59f0 2022-07-20T03:16:22.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 0x7f7fe013d690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:22.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 0x7f7fe013d690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49946/0 (socket says 172.21.15.61:49946) 2022-07-20T03:16:22.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 -- 172.21.15.61:0/3398740060 learned_addr learned my addr 172.21.15.61:0/3398740060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:22.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6328700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fe01530c0 0x7f7fe0136ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:22.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5b27700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe00654f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:22.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 -- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 msgr2=0x7f7fe00654f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:22.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe00654f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 -- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fe01530c0 msgr2=0x7f7fe0136ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:22.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fe01530c0 0x7f7fe0136ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:22.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 -- 172.21.15.61:0/3398740060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7fd000d410 con 0x7f7fe0139930 2022-07-20T03:16:22.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe5326700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 0x7f7fe013d690 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f7fdc00c000 tx=0x7f7fdc009290).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:22.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fdc012600 con 0x7f7fe0139930 2022-07-20T03:16:22.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7fdc012760 con 0x7f7fe0139930 2022-07-20T03:16:22.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7fe0064010 con 0x7f7fe0139930 2022-07-20T03:16:22.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7fdc012ac0 con 0x7f7fe0139930 2022-07-20T03:16:22.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.460+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7fe01523c0 con 0x7f7fe0139930 2022-07-20T03:16:22.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.464+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7fe0152de0 con 0x7f7fe0139930 2022-07-20T03:16:22.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.464+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7fdc022070 con 0x7f7fe0139930 2022-07-20T03:16:22.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.464+0000 7f7fceffd700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053ad0 0x7f7fc0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:22.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.464+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(119..119 src has 1..119) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f7fdc06c3f0 con 0x7f7fe0139930 2022-07-20T03:16:22.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.464+0000 7f7fe5b27700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053ad0 0x7f7fc0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:22.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.464+0000 7f7fe5b27700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053ad0 0x7f7fc0055f90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f7fd001b000 tx=0x7f7fd0006e80).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:22.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.472+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7fdc043410 con 0x7f7fe0139930 2022-07-20T03:16:22.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:22.600+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f7fe0063ba0 con 0x7f7fe0139930 2022-07-20T03:16:23.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.328+0000 7f7fceffd700 1 -- 172.21.15.61:0/3398740060 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v120) v1 ==== 133+0+0 (secure 0 0 0) 0x7f7fdc016020 con 0x7f7fe0139930 2022-07-20T03:16:23.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053ad0 msgr2=0x7f7fc0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053ad0 0x7f7fc0055f90 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f7fd001b000 tx=0x7f7fd0006e80).stop 2022-07-20T03:16:23.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 msgr2=0x7f7fe013d690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 0x7f7fe013d690 secure :-1 s=READY pgs=1210 cs=0 l=1 rev1=1 rx=0x7f7fdc00c000 tx=0x7f7fdc009290).stop 2022-07-20T03:16:23.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 shutdown_connections 2022-07-20T03:16:23.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7fc0053ad0 0x7f7fc0055f90 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7fe01530c0 0x7f7fe0136ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7fe0139930 0x7f7fe013d690 unknown :-1 s=CLOSED pgs=1210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 --2- 172.21.15.61:0/3398740060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7fe00b59f0 0x7f7fe00654f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 >> 172.21.15.61:0/3398740060 conn(0x7f7fe0063480 msgr2=0x7f7fe00afe70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:23.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 shutdown_connections 2022-07-20T03:16:23.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.332+0000 7f7fe6b29700 1 -- 172.21.15.61:0/3398740060 wait complete. 2022-07-20T03:16:23.340 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:16:23.350 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data 2022-07-20T03:16:23.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe47d28c700 1 Processor -- start 2022-07-20T03:16:23.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe47d28c700 1 -- start start 2022-07-20T03:16:23.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe47d28c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe478061ba0 0x7fe478142720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe47d28c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47805f840 con 0x7fe4780ab180 2022-07-20T03:16:23.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe47d28c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe47805fa70 con 0x7fe4780b34d0 2022-07-20T03:16:23.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe47d28c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47805fd10 con 0x7fe478061ba0 2022-07-20T03:16:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe4767fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe4780ab180 0x7fe478142fd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46552/0 (socket says 172.21.15.61:46552) 2022-07-20T03:16:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe4767fc700 1 -- 172.21.15.61:0/1850738593 learned_addr learned my addr 172.21.15.61:0/1850738593 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:23.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe4777fe700 1 --2- 172.21.15.61:0/1850738593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe478061ba0 0x7fe478142720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2065996897 0 0) 0x7fe47805f840 con 0x7fe4780ab180 2022-07-20T03:16:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.464+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe45c003260 con 0x7fe4780ab180 2022-07-20T03:16:23.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3255810017 0 0) 0x7fe47805fa70 con 0x7fe4780b34d0 2022-07-20T03:16:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe47805f840 con 0x7fe4780b34d0 2022-07-20T03:16:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe4777fe700 1 -- 172.21.15.61:0/1850738593 >> v1:172.21.15.117:6789/0 conn(0x7fe4780b34d0 legacy=0x7fe478142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe4777fe700 1 -- 172.21.15.61:0/1850738593 >> v1:172.21.15.61:6789/0 conn(0x7fe4780ab180 legacy=0x7fe478142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe4777fe700 1 -- 172.21.15.61:0/1850738593 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe47805ffb0 con 0x7fe478061ba0 2022-07-20T03:16:23.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe4777fe700 1 --2- 172.21.15.61:0/1850738593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe478061ba0 0x7fe478142720 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe4700032f0 tx=0x7fe4700091d0).ready entity=mon.2 client_cookie=5d6da32c9c329871 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:23.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe47000b640 con 0x7fe478061ba0 2022-07-20T03:16:23.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe47000b7a0 con 0x7fe478061ba0 2022-07-20T03:16:23.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe475ffb700 1 -- 172.21.15.61:0/1850738593 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe47000b910 con 0x7fe478061ba0 2022-07-20T03:16:23.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 -- 172.21.15.61:0/1850738593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe478061ba0 msgr2=0x7fe478142720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1850738593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe478061ba0 0x7fe478142720 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fe4700032f0 tx=0x7fe4700091d0).stop 2022-07-20T03:16:23.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 -- 172.21.15.61:0/1850738593 shutdown_connections 2022-07-20T03:16:23.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1850738593 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe478061ba0 0x7fe478142720 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 -- 172.21.15.61:0/1850738593 >> 172.21.15.61:0/1850738593 conn(0x7fe4780b6c50 msgr2=0x7fe4780b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:23.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 -- 172.21.15.61:0/1850738593 shutdown_connections 2022-07-20T03:16:23.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.468+0000 7fe47d28c700 1 -- 172.21.15.61:0/1850738593 wait complete. 2022-07-20T03:16:23.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 Processor -- start 2022-07-20T03:16:23.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 -- start start 2022-07-20T03:16:23.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe478061ba0 0x7fe478142340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:23.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 0x7fe47809b1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:23.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4780b34d0 0x7fe4781553d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:23.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809a850 con 0x7fe4780ab180 2022-07-20T03:16:23.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809aab0 con 0x7fe478061ba0 2022-07-20T03:16:23.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe47809ad50 con 0x7fe4780b34d0 2022-07-20T03:16:23.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 0x7fe47809b1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:23.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 0x7fe47809b1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49960/0 (socket says 172.21.15.61:49960) 2022-07-20T03:16:23.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 -- 172.21.15.61:0/1231174305 learned_addr learned my addr 172.21.15.61:0/1231174305 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:23.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4777fe700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4780b34d0 0x7fe4781553d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:23.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe476ffd700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe478061ba0 0x7fe478142340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:23.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 -- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4780b34d0 msgr2=0x7fe4781553d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4780b34d0 0x7fe4781553d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 -- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe478061ba0 msgr2=0x7fe478142340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe478061ba0 0x7fe478142340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 -- 172.21.15.61:0/1231174305 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe470009470 con 0x7fe4780ab180 2022-07-20T03:16:23.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe476ffd700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe478061ba0 0x7fe478142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:23.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe4767fc700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 0x7fe47809b1f0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7fe46c0011d0 tx=0x7fe46c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:23.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46c0070d0 con 0x7fe4780ab180 2022-07-20T03:16:23.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe46c007230 con 0x7fe4780ab180 2022-07-20T03:16:23.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4780ae230 con 0x7fe4780ab180 2022-07-20T03:16:23.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe46c005610 con 0x7fe4780ab180 2022-07-20T03:16:23.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.472+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe47809be20 con 0x7fe4780ab180 2022-07-20T03:16:23.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.476+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4780b38b0 con 0x7fe4780ab180 2022-07-20T03:16:23.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.476+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe46c01a060 con 0x7fe4780ab180 2022-07-20T03:16:23.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.476+0000 7fe467fff700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe454053a40 0x7fe454055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:23.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.476+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(120..120 src has 1..120) v4 ==== 6075+0+0 (secure 0 0 0) 0x7fe46c06b5b0 con 0x7fe4780ab180 2022-07-20T03:16:23.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.476+0000 7fe476ffd700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe454053a40 0x7fe454055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:23.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.476+0000 7fe476ffd700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe454053a40 0x7fe454055f00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fe46800d000 tx=0x7fe468007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:23.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.484+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe46c042510 con 0x7fe4780ab180 2022-07-20T03:16:23.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.648+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"} v 0) v1 -- 0x7fe4780574e0 con 0x7fe4780ab180 2022-07-20T03:16:23.653 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:16:23.648+0000 7f2534a18700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:16:23.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.656+0000 7fe467fff700 1 -- 172.21.15.61:0/1231174305 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data"}]=0 new fs with metadata pool 24 and data pool 22 v67) v1 ==== 174+0+0 (secure 0 0 0) 0x7fe46c018020 con 0x7fe4780ab180 2022-07-20T03:16:23.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe454053a40 msgr2=0x7fe454055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe454053a40 0x7fe454055f00 secure :-1 s=READY pgs=583 cs=0 l=1 rev1=1 rx=0x7fe46800d000 tx=0x7fe468007a20).stop 2022-07-20T03:16:23.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 msgr2=0x7fe47809b1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:23.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 0x7fe47809b1f0 secure :-1 s=READY pgs=1212 cs=0 l=1 rev1=1 rx=0x7fe46c0011d0 tx=0x7fe46c007b50).stop 2022-07-20T03:16:23.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 shutdown_connections 2022-07-20T03:16:23.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe454053a40 0x7fe454055f00 unknown :-1 s=CLOSED pgs=583 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe4780b34d0 0x7fe4781553d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe4780ab180 0x7fe47809b1f0 unknown :-1 s=CLOSED pgs=1212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 --2- 172.21.15.61:0/1231174305 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe478061ba0 0x7fe478142340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:23.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 >> 172.21.15.61:0/1231174305 conn(0x7fe4780b6c50 msgr2=0x7fe4780b5690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 shutdown_connections 2022-07-20T03:16:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:23.660+0000 7fe47d28c700 1 -- 172.21.15.61:0/1231174305 wait complete. 2022-07-20T03:16:23.671 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 24 and data pool 22 2022-07-20T03:16:23.679 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:16:23.680 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:23.707 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:23.730 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:23.751 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:23.773 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:23.798 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:16:23.799 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:23.800 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:23.800 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:23.807 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:16:23.808 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:23.809 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:23.810 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:23.811 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:23.857 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:23.859 INFO:teuthology.orchestra.run.smithi124.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:16:23.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:23.860 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:23.860 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:23.869 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:16:23.869 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:23.870 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:23.871 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:23.871 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:23.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:23.929 INFO:teuthology.orchestra.run.smithi163.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:16:23.930 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:23.931 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:23.932 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:16:23.941 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:23.943 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:16:23.943 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:16:23.944 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:23.945 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:16:23.952 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:16:23.953 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:23.956 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:16:24.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 Processor -- start 2022-07-20T03:16:24.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- start start 2022-07-20T03:16:24.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780adff0 0x7f51780b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51780af8e0 0x7f517805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5178147010 con 0x7f51780b5650 2022-07-20T03:16:24.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f51781472b0 con 0x7f51780af8e0 2022-07-20T03:16:24.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5178146d40 con 0x7f51780adff0 2022-07-20T03:16:24.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e876700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51780af8e0 0x7f517805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517d874700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780adff0 0x7f51780b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e075700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f51780b5650 0x7f5178147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46564/0 (socket says 172.21.15.61:46564) 2022-07-20T03:16:24.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e075700 1 -- 172.21.15.61:0/1894257550 learned_addr learned my addr 172.21.15.61:0/1894257550 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:24.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e876700 1 -- 172.21.15.61:0/1894257550 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780adff0 msgr2=0x7f51780b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:24.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e876700 1 --2- 172.21.15.61:0/1894257550 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780adff0 0x7f51780b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e876700 1 -- 172.21.15.61:0/1894257550 >> v1:172.21.15.61:6789/0 conn(0x7f51780b5650 legacy=0x7f5178147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517d073700 1 -- 172.21.15.61:0/1894257550 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2410720545 0 0) 0x7f5178147010 con 0x7f51780b5650 2022-07-20T03:16:24.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e876700 1 -- 172.21.15.61:0/1894257550 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51780acae0 con 0x7f51780af8e0 2022-07-20T03:16:24.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517d874700 1 --2- 172.21.15.61:0/1894257550 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780adff0 0x7f51780b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:24.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517e876700 1 --2- 172.21.15.61:0/1894257550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780af8e0 0x7f517805bd90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f51700032f0 tx=0x7f51700091d0).ready entity=mon.0 client_cookie=f903117b8cd911c2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517d073700 1 -- 172.21.15.61:0/1894257550 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f517000b6c0 con 0x7f51780af8e0 2022-07-20T03:16:24.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517d073700 1 -- 172.21.15.61:0/1894257550 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f517000b820 con 0x7f51780af8e0 2022-07-20T03:16:24.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517d073700 1 -- 172.21.15.61:0/1894257550 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f517000b990 con 0x7f51780af8e0 2022-07-20T03:16:24.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- 172.21.15.61:0/1894257550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780af8e0 msgr2=0x7f517805bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 --2- 172.21.15.61:0/1894257550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780af8e0 0x7f517805bd90 secure :-1 s=READY pgs=1214 cs=0 l=1 rev1=1 rx=0x7f51700032f0 tx=0x7f51700091d0).stop 2022-07-20T03:16:24.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- 172.21.15.61:0/1894257550 shutdown_connections 2022-07-20T03:16:24.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 --2- 172.21.15.61:0/1894257550 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780af8e0 0x7f517805bd90 unknown :-1 s=CLOSED pgs=1214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 --2- 172.21.15.61:0/1894257550 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780adff0 0x7f51780b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- 172.21.15.61:0/1894257550 >> 172.21.15.61:0/1894257550 conn(0x7f51780634e0 msgr2=0x7f51780b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:24.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.068+0000 7f517f077700 1 -- 172.21.15.61:0/1894257550 shutdown_connections 2022-07-20T03:16:24.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.072+0000 7f517f077700 1 -- 172.21.15.61:0/1894257550 wait complete. 2022-07-20T03:16:24.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.072+0000 7f517f077700 1 Processor -- start 2022-07-20T03:16:24.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.072+0000 7f517f077700 1 -- start start 2022-07-20T03:16:24.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 0x7f51780b30b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51780af8e0 0x7f5178143330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780b5650 0x7f5178143820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178111b00 con 0x7f51780adff0 2022-07-20T03:16:24.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5178110240 con 0x7f51780af8e0 2022-07-20T03:16:24.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5178111860 con 0x7f51780b5650 2022-07-20T03:16:24.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 0x7f51780b30b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 0x7f51780b30b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:49972/0 (socket says 172.21.15.61:49972) 2022-07-20T03:16:24.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 -- 172.21.15.61:0/1786858964 learned_addr learned my addr 172.21.15.61:0/1786858964 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:24.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517d874700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51780af8e0 0x7f5178143330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e876700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780b5650 0x7f5178143820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 -- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780b5650 msgr2=0x7f5178143820 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780b5650 0x7f5178143820 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 -- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51780af8e0 msgr2=0x7f5178143330 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51780af8e0 0x7f5178143330 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 -- 172.21.15.61:0/1786858964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f516800b120 con 0x7f51780adff0 2022-07-20T03:16:24.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517e075700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 0x7f51780b30b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f5174000ed0 tx=0x7f5174007ce0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f51740070d0 con 0x7f51780adff0 2022-07-20T03:16:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5174007230 con 0x7f51780adff0 2022-07-20T03:16:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5170009470 con 0x7f51780adff0 2022-07-20T03:16:24.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5174005610 con 0x7f51780adff0 2022-07-20T03:16:24.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5178111610 con 0x7f51780adff0 2022-07-20T03:16:24.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.076+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f51780574e0 con 0x7f51780adff0 2022-07-20T03:16:24.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.080+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f517401a060 con 0x7f51780adff0 2022-07-20T03:16:24.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.080+0000 7f51677fe700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f51580539d0 0x7f5158055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.080+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(121..121 src has 1..121) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f517406b1d0 con 0x7f51780adff0 2022-07-20T03:16:24.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.080+0000 7f517d874700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f51580539d0 0x7f5158055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.080+0000 7f517d874700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f51580539d0 0x7f5158055e90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f5168008000 tx=0x7f5168008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.084+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f51740420e0 con 0x7f51780adff0 2022-07-20T03:16:24.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.212+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f5178064050 con 0x7f51780adff0 2022-07-20T03:16:24.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.212+0000 7f51677fe700 1 -- 172.21.15.61:0/1786858964 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v121) v1 ==== 81+0+33876 (secure 0 0 0) 0x7f5174018020 con 0x7f51780adff0 2022-07-20T03:16:24.218 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:24.218 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:16:24.218 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 121, 2022-07-20T03:16:24.219 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:16:24.219 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:16:24.219 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:16:23.652732+0000", 2022-07-20T03:16:24.220 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:16:24.220 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:16:24.221 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:16:24.221 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:16:24.222 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:16:24.222 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:16:24.222 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:16:24.223 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:16:24.223 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:16:24.224 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:24.224 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:16:24.224 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:16:24.225 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:16:24.225 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:16:24.226 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:16:24.226 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 24, 2022-07-20T03:16:24.226 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:16:24.226 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:16:24.227 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:16:24.227 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:16:24.227 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:16:24.227 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:16:24.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:16:24.228 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:16:24.229 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:24.229 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:24.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:24.230 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:24.230 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:24.231 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:24.231 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:24.232 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:24.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:16:24.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:16:24.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:16:24.234 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:16:24.234 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:16:24.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:24.235 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:24.235 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:24.236 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:24.236 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:24.237 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:24.237 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:24.238 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.238 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:16:24.239 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:24.239 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:24.239 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:24.239 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:24.240 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:24.240 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:24.241 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:24.241 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:24.241 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:24.242 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:24.242 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:24.242 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:24.243 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:24.243 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:24.244 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:24.244 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:24.245 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:24.245 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:24.246 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:24.246 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:24.246 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:24.247 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:24.247 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:24.247 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:24.247 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:24.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:24.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.248 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:24.248 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:24.249 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:24.249 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:24.249 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:24.249 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:24.250 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:24.250 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:24.250 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:24.250 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:24.251 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:24.251 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:16:24.252 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:16:24.252 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.252 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:24.253 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:16:24.254 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.254 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.255 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.255 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:16:24.255 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:16:24.255 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:16:24.256 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:24.256 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:24.256 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:24.256 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:24.257 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:24.257 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:24.257 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:24.257 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:24.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:24.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:24.259 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:24.259 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:24.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:24.260 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:24.261 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:24.261 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:24.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:24.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:24.263 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:24.263 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:24.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:16:24.266 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:24.267 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:24.268 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:24.269 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:24.269 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:24.270 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:24.270 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:24.271 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:24.272 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:24.272 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:24.273 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:24.273 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:24.274 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:24.275 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:24.275 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:24.276 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:24.277 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:24.277 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:24.278 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:24.279 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:24.279 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:24.280 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:24.281 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:24.281 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:24.282 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:24.283 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:24.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.284 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:24.284 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:24.285 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:24.285 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:24.286 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:24.286 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:24.287 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:24.287 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:24.288 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:24.288 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:24.289 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:24.289 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:16:24.290 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:24.290 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:16:24.291 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.292 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 22, 2022-07-20T03:16:24.292 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:16:24.293 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:15:58.563360+0000", 2022-07-20T03:16:24.293 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:24.293 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:24.293 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:24.294 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:24.294 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:24.294 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:24.295 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:24.295 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:24.296 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:24.296 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:24.297 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:24.297 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:24.298 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:24.299 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:24.299 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:24.300 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:24.300 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:24.301 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:24.301 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:24.302 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:24.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.303 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "121", 2022-07-20T03:16:24.303 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:24.303 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:24.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:24.304 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:24.305 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:24.305 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:24.306 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:24.306 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:24.306 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:24.307 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:24.307 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:24.307 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:24.308 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:24.308 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:24.308 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:24.308 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:24.309 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:24.309 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:24.309 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:24.309 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:24.310 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:24.310 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:24.310 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:24.310 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:24.311 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:24.311 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:24.311 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.311 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:24.311 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:24.312 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:24.312 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:24.312 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:24.312 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:24.313 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:24.313 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:24.313 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:24.313 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:24.314 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:24.314 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:16:24.314 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:24.315 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:16:24.315 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:16:24.315 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.315 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.316 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.316 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 24, 2022-07-20T03:16:24.316 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:16:24.317 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:16:22.657129+0000", 2022-07-20T03:16:24.317 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 32769, 2022-07-20T03:16:24.317 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool,creating", 2022-07-20T03:16:24.317 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:24.318 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:24.318 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:24.318 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:24.318 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:24.318 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:24.319 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:24.319 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:24.319 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:24.319 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:24.320 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:24.330 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:24.331 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:24.331 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:24.331 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:24.331 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:24.332 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:24.332 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:24.333 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.333 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "121", 2022-07-20T03:16:24.333 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:24.333 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:24.334 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:24.334 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:24.334 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:24.334 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:24.335 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:24.335 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:24.335 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:24.335 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:24.336 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:24.336 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:24.336 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:24.336 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:24.337 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:24.337 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:24.337 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:24.337 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:24.338 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:24.338 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:24.338 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:24.339 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:24.339 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:24.339 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:24.339 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:24.340 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:24.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.340 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:24.340 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:24.341 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:24.341 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:24.341 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:24.341 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:24.342 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:24.342 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:24.342 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:24.342 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:24.343 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:24.343 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:16:24.343 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:16:24.343 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:16:24.344 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:16:24.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.344 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:24.345 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:16:24.345 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:16:24.345 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.345 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.346 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.346 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:24.346 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:16:24.346 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.347 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:16:24.347 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:16:24.347 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.347 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.347 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.348 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.348 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.348 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.348 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.349 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:16:24.349 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.349 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.349 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.350 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.350 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:16:24.351 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.351 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.351 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.351 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.352 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:16:24.352 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.352 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.352 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.353 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.353 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.354 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.354 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:16:24.354 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.355 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.355 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.355 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:16:24.356 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.356 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.356 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.356 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.356 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:16:24.358 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.358 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:16:24.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.359 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.360 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.360 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.360 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.360 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:16:24.361 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.361 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.362 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:16:24.363 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:24.363 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.363 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.364 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:16:24.364 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:16:24.364 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:16:24.364 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:16:24.365 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.365 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.365 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.365 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.366 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.366 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:16:24.366 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:16:24.366 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.367 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.367 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.367 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.367 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.368 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:16:24.368 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.369 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.369 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.369 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.370 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.370 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:16:24.370 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.370 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.371 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.371 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:16:24.371 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.371 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.372 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.372 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.372 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.373 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.373 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.373 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:16:24.373 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.373 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.374 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:16:24.374 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.375 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.375 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.375 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.375 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.376 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.376 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.376 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.376 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:16:24.377 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.377 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.377 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:16:24.378 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.378 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.378 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.378 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.379 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.379 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.379 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.379 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.380 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:16:24.380 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.380 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.380 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.381 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:16:24.381 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:24.381 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.381 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.382 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.382 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:16:24.382 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:16:24.383 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:16:24.383 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:16:24.383 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.383 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.383 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.384 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.384 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.384 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.384 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:16:24.385 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:16:24.385 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.385 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.385 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.386 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.386 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.386 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.386 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.386 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 115, 2022-07-20T03:16:24.387 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.387 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.387 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.388 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.388 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:16:24.388 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.389 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.389 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:16:24.389 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.390 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.390 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.390 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.390 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.391 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.391 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:16:24.392 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.392 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.392 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:16:24.393 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.393 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.393 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.394 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.394 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.394 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.395 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:16:24.395 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.395 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.395 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.396 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.396 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:16:24.396 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.396 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.397 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.397 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.397 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.398 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.398 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:16:24.398 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.398 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:16:24.399 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:24.400 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.400 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.400 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:16:24.401 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:16:24.401 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:16:24.401 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:16:24.401 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.401 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.402 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.402 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.402 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.403 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:16:24.403 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:16:24.403 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.403 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.404 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.404 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.404 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.404 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.405 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.405 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 115, 2022-07-20T03:16:24.405 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.405 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.405 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.406 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.406 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.406 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.406 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:16:24.407 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.407 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.407 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.407 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.408 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:16:24.408 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.408 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.408 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.409 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.409 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.409 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.409 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.409 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.410 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:16:24.410 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.410 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.411 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:16:24.411 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.411 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.412 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.412 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.412 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.413 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:16:24.413 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.414 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:16:24.414 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.415 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.415 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.415 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.416 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.416 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.416 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.416 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:16:24.416 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.417 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:16:24.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:24.418 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.418 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.419 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:16:24.419 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:16:24.419 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:16:24.419 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:16:24.420 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.420 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.420 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.420 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.420 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.421 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.421 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:16:24.421 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:16:24.421 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.422 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.422 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.422 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.422 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.423 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.423 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.423 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 115, 2022-07-20T03:16:24.423 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.424 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.424 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.424 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.424 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.424 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:16:24.425 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.425 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.426 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.426 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:16:24.426 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.426 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.427 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.427 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.427 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.427 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.428 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:16:24.428 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.429 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.429 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.429 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:16:24.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.430 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.430 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.430 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.430 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.431 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.431 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:16:24.431 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.432 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.432 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:16:24.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.433 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.433 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.434 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.434 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.434 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:16:24.435 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.435 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.435 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:16:24.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:24.436 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.436 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.437 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:16:24.437 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:16:24.437 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:16:24.438 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:16:24.438 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.438 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.439 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.439 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.439 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.439 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.439 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:16:24.440 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:16:24.440 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.440 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.440 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.441 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.441 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.441 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.441 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.442 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:16:24.442 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.442 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.442 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.443 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.443 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.443 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.444 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:16:24.444 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.445 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.445 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.445 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.445 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:16:24.445 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.446 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.446 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.446 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.446 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.447 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.447 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.447 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.447 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:16:24.448 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.448 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.448 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.448 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.449 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:16:24.449 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.449 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.449 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.450 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.450 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.450 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.450 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.451 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:16:24.451 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.451 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.452 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.452 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.452 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:16:24.453 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.453 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.453 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.453 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.454 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.454 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.455 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:16:24.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.456 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:16:24.457 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:24.457 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.457 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.458 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.458 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:16:24.458 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:16:24.459 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:16:24.459 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:16:24.459 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.459 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.460 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.460 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.460 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.461 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.461 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:16:24.461 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:16:24.461 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.462 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.462 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.462 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.463 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.463 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.463 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.463 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 109, 2022-07-20T03:16:24.464 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.464 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.464 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.465 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.465 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.465 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.465 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:16:24.466 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.466 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.466 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.467 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.467 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:16:24.467 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.467 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.468 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.468 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.468 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.469 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.469 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.469 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.470 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:16:24.470 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.470 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.471 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.471 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.471 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:16:24.472 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.472 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.472 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.472 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.473 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.473 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.473 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.474 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.474 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:16:24.475 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.475 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.475 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.476 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.476 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:16:24.476 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.476 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.477 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.477 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.477 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.477 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.478 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.478 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.478 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:16:24.479 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.479 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.479 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.480 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.480 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:16:24.480 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:24.481 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.481 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.481 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.482 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:16:24.482 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:16:24.482 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:16:24.483 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:16:24.483 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.483 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.483 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.484 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.484 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.484 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.485 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:16:24.485 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:16:24.485 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:24.485 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:24.486 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:24.486 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:24.486 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:24.486 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:24.487 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:24.487 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 108, 2022-07-20T03:16:24.487 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:24.488 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:24.488 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:24.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.488 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.489 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.489 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:16:24.489 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.490 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.490 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.490 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.491 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:16:24.491 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.491 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.491 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.492 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.492 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:24.492 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.492 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.493 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.493 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:16:24.493 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.493 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.494 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.494 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.494 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:16:24.494 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.495 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.495 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.495 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.496 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:24.496 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.496 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.497 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.497 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:16:24.497 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.497 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.498 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.498 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.498 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:16:24.499 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.499 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.499 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.500 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.500 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:24.500 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:24.501 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.501 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:24.501 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:16:24.501 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.502 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.502 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.502 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:24.503 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:16:24.503 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:24.503 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.504 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.504 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.504 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:16:24.505 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:16:24.505 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:16:24.505 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:16:24.505 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:24.506 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:24.506 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:24.506 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:24.506 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.507 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:24.507 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:16:24.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.508 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:16:24.508 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.508 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.508 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.509 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.509 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.509 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:16:24.509 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.510 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.510 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.510 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:16:24.510 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.511 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.511 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.511 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.511 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.512 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:16:24.512 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.512 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.513 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.513 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:16:24.513 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.515 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.516 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.516 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.517 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.518 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:16:24.519 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.519 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.520 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.520 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:16:24.521 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.521 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.522 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.522 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.523 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.523 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:16:24.524 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.525 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.525 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.526 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:16:24.526 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.527 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.527 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.527 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.528 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.528 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:16:24.529 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.529 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.530 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:16:24.530 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.531 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.531 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.532 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.532 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.533 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:16:24.533 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.533 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.534 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:16:24.535 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.536 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.536 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.536 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.537 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.537 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:16:24.538 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.538 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.539 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:24.539 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:16:24.539 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:24.540 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:24.540 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:24.541 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:24.541 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:24.541 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:16:24.542 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:24.542 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.543 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:24.543 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:16:24.544 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:16:24.544 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:16:24.544 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:16:24.545 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:16:24.545 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6835/2884785802": "2022-07-21T03:16:09.208729+0000", 2022-07-20T03:16:24.546 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6834/2884785802": "2022-07-21T03:16:09.208729+0000" 2022-07-20T03:16:24.546 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.546 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:16:24.547 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:16:24.547 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:16:24.548 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:16:24.548 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:16:24.549 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:16:24.549 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:16:24.549 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:16:24.550 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:24.550 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:24.551 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:16:24.551 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:16:24.552 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:16:24.552 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:16:24.552 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:16:24.553 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:16:24.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f51580539d0 msgr2=0x7f5158055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f51580539d0 0x7f5158055e90 secure :-1 s=READY pgs=584 cs=0 l=1 rev1=1 rx=0x7f5168008000 tx=0x7f5168008040).stop 2022-07-20T03:16:24.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 msgr2=0x7f51780b30b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 0x7f51780b30b0 secure :-1 s=READY pgs=1215 cs=0 l=1 rev1=1 rx=0x7f5174000ed0 tx=0x7f5174007ce0).stop 2022-07-20T03:16:24.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 shutdown_connections 2022-07-20T03:16:24.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f51580539d0 0x7f5158055e90 unknown :-1 s=CLOSED pgs=584 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f51780af8e0 0x7f5178143330 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f51780b5650 0x7f5178143820 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 --2- 172.21.15.61:0/1786858964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51780adff0 0x7f51780b30b0 unknown :-1 s=CLOSED pgs=1215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 >> 172.21.15.61:0/1786858964 conn(0x7f51780634e0 msgr2=0x7f517814a250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 shutdown_connections 2022-07-20T03:16:24.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.216+0000 7f517f077700 1 -- 172.21.15.61:0/1786858964 wait complete. 2022-07-20T03:16:24.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:24.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe853e1d700 1 Processor -- start 2022-07-20T03:16:24.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe853e1d700 1 -- start start 2022-07-20T03:16:24.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe853e1d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe84c0b3110 0x7fe84c061d80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe853e1d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84c0acaf0 con 0x7fe84c13f940 2022-07-20T03:16:24.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe853e1d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe84c0b2190 con 0x7fe84c0b34f0 2022-07-20T03:16:24.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe853e1d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe84c0acd90 con 0x7fe84c0b3110 2022-07-20T03:16:24.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe8523ba700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe84c13f940 0x7fe84c13fdb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46582/0 (socket says 172.21.15.61:46582) 2022-07-20T03:16:24.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe8523ba700 1 -- 172.21.15.61:0/1686715535 learned_addr learned my addr 172.21.15.61:0/1686715535 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:24.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe851bb9700 1 --2- 172.21.15.61:0/1686715535 >> v2:172.21.15.117:3300/0 conn(0x7fe84c0b3110 0x7fe84c061d80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe850bb7700 1 -- 172.21.15.61:0/1686715535 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1520140135 0 0) 0x7fe84c0acaf0 con 0x7fe84c13f940 2022-07-20T03:16:24.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe851bb9700 1 -- 172.21.15.61:0/1686715535 >> v1:172.21.15.117:6789/0 conn(0x7fe84c0b34f0 legacy=0x7fe84c1414b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe851bb9700 1 -- 172.21.15.61:0/1686715535 >> v1:172.21.15.61:6789/0 conn(0x7fe84c13f940 legacy=0x7fe84c13fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe851bb9700 1 -- 172.21.15.61:0/1686715535 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe84c0b3940 con 0x7fe84c0b3110 2022-07-20T03:16:24.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.672+0000 7fe851bb9700 1 --2- 172.21.15.61:0/1686715535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c061d80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe84c0ad7d0 tx=0x7fe8440091d0).ready entity=mon.1 client_cookie=6faad8e534619bfb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe850bb7700 1 -- 172.21.15.61:0/1686715535 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84400b860 con 0x7fe84c0b3110 2022-07-20T03:16:24.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe850bb7700 1 -- 172.21.15.61:0/1686715535 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe84400b9c0 con 0x7fe84c0b3110 2022-07-20T03:16:24.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe850bb7700 1 -- 172.21.15.61:0/1686715535 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84400bb30 con 0x7fe84c0b3110 2022-07-20T03:16:24.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- 172.21.15.61:0/1686715535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 msgr2=0x7fe84c061d80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 --2- 172.21.15.61:0/1686715535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c061d80 secure :-1 s=READY pgs=555 cs=0 l=1 rev1=1 rx=0x7fe84c0ad7d0 tx=0x7fe8440091d0).stop 2022-07-20T03:16:24.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- 172.21.15.61:0/1686715535 shutdown_connections 2022-07-20T03:16:24.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 --2- 172.21.15.61:0/1686715535 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c061d80 unknown :-1 s=CLOSED pgs=555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- 172.21.15.61:0/1686715535 >> 172.21.15.61:0/1686715535 conn(0x7fe84c0b47e0 msgr2=0x7fe84c05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:24.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- 172.21.15.61:0/1686715535 shutdown_connections 2022-07-20T03:16:24.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- 172.21.15.61:0/1686715535 wait complete. 2022-07-20T03:16:24.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 Processor -- start 2022-07-20T03:16:24.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- start start 2022-07-20T03:16:24.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c0b34f0 0x7fe84c1558c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe84c13f940 0x7fe84c155db0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c095e00 con 0x7fe84c0b34f0 2022-07-20T03:16:24.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c096060 con 0x7fe84c0b3110 2022-07-20T03:16:24.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe853e1d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe84c093b70 con 0x7fe84c13f940 2022-07-20T03:16:24.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe8513b8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c0b34f0 0x7fe84c1558c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c143930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59666/0 (socket says 172.21.15.61:59666) 2022-07-20T03:16:24.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 -- 172.21.15.61:0/17667087 learned_addr learned my addr 172.21.15.61:0/17667087 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:24.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe8523ba700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe84c13f940 0x7fe84c155db0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 -- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe84c13f940 msgr2=0x7fe84c155db0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe84c13f940 0x7fe84c155db0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 -- 172.21.15.61:0/17667087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c0b34f0 msgr2=0x7fe84c1558c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c0b34f0 0x7fe84c1558c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 -- 172.21.15.61:0/17667087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe844009470 con 0x7fe84c0b3110 2022-07-20T03:16:24.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.676+0000 7fe851bb9700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c143930 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fe84c0ad7d0 tx=0x7fe844016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe844011ea0 con 0x7fe84c0b3110 2022-07-20T03:16:24.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe8440032f0 con 0x7fe84c0b3110 2022-07-20T03:16:24.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe84c1562a0 con 0x7fe84c0b3110 2022-07-20T03:16:24.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe84401f5e0 con 0x7fe84c0b3110 2022-07-20T03:16:24.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe84c094050 con 0x7fe84c0b3110 2022-07-20T03:16:24.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe84c096770 con 0x7fe84c0b3110 2022-07-20T03:16:24.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe844026030 con 0x7fe84c0b3110 2022-07-20T03:16:24.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe83affd700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe82c0539b0 0x7fe82c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe84406cae0 con 0x7fe84c0b3110 2022-07-20T03:16:24.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe8513b8700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe82c0539b0 0x7fe82c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.680+0000 7fe8513b8700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe82c0539b0 0x7fe82c055e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fe83c00d000 tx=0x7fe83c007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.688+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe84c096770 con 0x7fe84c0b3110 2022-07-20T03:16:24.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.852+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fe84c096770 con 0x7fe84c0b3110 2022-07-20T03:16:24.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.852+0000 7fe83affd700 1 -- 172.21.15.61:0/17667087 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 67 v67) v1 ==== 94+0+1426 (secure 0 0 0) 0x7fe84c096770 con 0x7fe84c0b3110 2022-07-20T03:16:24.858 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:24.859 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":67,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":67,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:23.653073+0000","modified":"2022-07-20T03:16:23.653191+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-20T03:16:24.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe82c0539b0 msgr2=0x7fe82c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe82c0539b0 0x7fe82c055e70 secure :-1 s=READY pgs=585 cs=0 l=1 rev1=1 rx=0x7fe83c00d000 tx=0x7fe83c007b90).stop 2022-07-20T03:16:24.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 msgr2=0x7fe84c143930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c143930 secure :-1 s=READY pgs=556 cs=0 l=1 rev1=1 rx=0x7fe84c0ad7d0 tx=0x7fe844016000).stop 2022-07-20T03:16:24.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 shutdown_connections 2022-07-20T03:16:24.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe82c0539b0 0x7fe82c055e70 unknown :-1 s=CLOSED pgs=585 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe84c13f940 0x7fe84c155db0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe84c0b34f0 0x7fe84c1558c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 --2- 172.21.15.61:0/17667087 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe84c0b3110 0x7fe84c143930 unknown :-1 s=CLOSED pgs=556 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 >> 172.21.15.61:0/17667087 conn(0x7fe84c0b47e0 msgr2=0x7fe84c05caf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:24.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 shutdown_connections 2022-07-20T03:16:24.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.856+0000 7fe853e1d700 1 -- 172.21.15.61:0/17667087 wait complete. 2022-07-20T03:16:24.866 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 67 2022-07-20T03:16:24.876 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:16:24.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc89387700 1 Processor -- start 2022-07-20T03:16:24.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc89387700 1 -- start start 2022-07-20T03:16:24.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc89387700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc8405d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:24.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc89387700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc84147090 con 0x7fbc84060bb0 2022-07-20T03:16:24.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc89387700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbc84099080 con 0x7fbc8405cda0 2022-07-20T03:16:24.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc89387700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc84060160 con 0x7fbc840b11b0 2022-07-20T03:16:24.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc837fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbc84060bb0 0x7fbc84147760 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46596/0 (socket says 172.21.15.61:46596) 2022-07-20T03:16:24.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc837fe700 1 -- 172.21.15.61:0/2905560427 learned_addr learned my addr 172.21.15.61:0/2905560427 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:24.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.976+0000 7fbc827fc700 1 --2- 172.21.15.61:0/2905560427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc8405d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:24.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3734278334 0 0) 0x7fbc84147090 con 0x7fbc84060bb0 2022-07-20T03:16:24.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc68003260 con 0x7fbc84060bb0 2022-07-20T03:16:24.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 153540379 0 0) 0x7fbc84099080 con 0x7fbc8405cda0 2022-07-20T03:16:24.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbc84147090 con 0x7fbc8405cda0 2022-07-20T03:16:24.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc827fc700 1 -- 172.21.15.61:0/2905560427 >> v1:172.21.15.117:6789/0 conn(0x7fbc8405cda0 legacy=0x7fbc8405fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc827fc700 1 -- 172.21.15.61:0/2905560427 >> v1:172.21.15.61:6789/0 conn(0x7fbc84060bb0 legacy=0x7fbc84147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc827fc700 1 -- 172.21.15.61:0/2905560427 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc841472c0 con 0x7fbc840b11b0 2022-07-20T03:16:24.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc827fc700 1 --2- 172.21.15.61:0/2905560427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc8405d180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fbc740032f0 tx=0x7fbc740091d0).ready entity=mon.2 client_cookie=93c92842a7c3148 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:24.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc7400b640 con 0x7fbc840b11b0 2022-07-20T03:16:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7400b7a0 con 0x7fbc840b11b0 2022-07-20T03:16:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc81ffb700 1 -- 172.21.15.61:0/2905560427 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc7400b910 con 0x7fbc840b11b0 2022-07-20T03:16:24.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 -- 172.21.15.61:0/2905560427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 msgr2=0x7fbc8405d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 --2- 172.21.15.61:0/2905560427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc8405d180 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fbc740032f0 tx=0x7fbc740091d0).stop 2022-07-20T03:16:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 -- 172.21.15.61:0/2905560427 shutdown_connections 2022-07-20T03:16:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 --2- 172.21.15.61:0/2905560427 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc8405d180 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:24.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 -- 172.21.15.61:0/2905560427 >> 172.21.15.61:0/2905560427 conn(0x7fbc840b5bb0 msgr2=0x7fbc840b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:24.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 -- 172.21.15.61:0/2905560427 shutdown_connections 2022-07-20T03:16:24.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 -- 172.21.15.61:0/2905560427 wait complete. 2022-07-20T03:16:24.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 Processor -- start 2022-07-20T03:16:24.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.980+0000 7fbc89387700 1 -- start start 2022-07-20T03:16:25.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc8405cda0 0x7fbc84064d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:25.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 0x7fbc840aea60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:25.000 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc841599e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:25.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840aa3a0 con 0x7fbc84060bb0 2022-07-20T03:16:25.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840aa100 con 0x7fbc8405cda0 2022-07-20T03:16:25.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbc840ae010 con 0x7fbc840b11b0 2022-07-20T03:16:25.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 0x7fbc840aea60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:25.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 0x7fbc840aea60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50004/0 (socket says 172.21.15.61:50004) 2022-07-20T03:16:25.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 -- 172.21.15.61:0/3244392891 learned_addr learned my addr 172.21.15.61:0/3244392891 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:25.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc837fe700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc841599e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:25.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc82ffd700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc8405cda0 0x7fbc84064d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:25.002 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 -- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 msgr2=0x7fbc841599e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:25.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc841599e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:25.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 -- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc8405cda0 msgr2=0x7fbc84064d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:25.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc8405cda0 0x7fbc84064d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:25.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 -- 172.21.15.61:0/3244392891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc74009470 con 0x7fbc84060bb0 2022-07-20T03:16:25.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc827fc700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 0x7fbc840aea60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7fbc740032f0 tx=0x7fbc74016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:25.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc7400b910 con 0x7fbc84060bb0 2022-07-20T03:16:25.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbc7401f3f0 con 0x7fbc84060bb0 2022-07-20T03:16:25.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbc7400b640 con 0x7fbc84060bb0 2022-07-20T03:16:25.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc840ae2b0 con 0x7fbc84060bb0 2022-07-20T03:16:25.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.984+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc840aa610 con 0x7fbc84060bb0 2022-07-20T03:16:25.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.988+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbc7400f070 con 0x7fbc84060bb0 2022-07-20T03:16:25.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.988+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc840ade80 con 0x7fbc84060bb0 2022-07-20T03:16:25.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.988+0000 7fbc73fff700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbc60053920 0x7fbc60055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:25.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.988+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(122..122 src has 1..122) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fbc7406c990 con 0x7fbc84060bb0 2022-07-20T03:16:25.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.988+0000 7fbc82ffd700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbc60053920 0x7fbc60055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:25.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.988+0000 7fbc82ffd700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbc60053920 0x7fbc60055de0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fbc7800d000 tx=0x7fbc78007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:25.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:24.996+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbc74043790 con 0x7fbc84060bb0 2022-07-20T03:16:25.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:25.160+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7fbc84147e70 con 0x7fbc84060bb0 2022-07-20T03:16:26.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.120+0000 7fbc73fff700 1 -- 172.21.15.61:0/3244392891 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v68) v1 ==== 161+0+0 (secure 0 0 0) 0x7fbc74015020 con 0x7fbc84060bb0 2022-07-20T03:16:26.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbc60053920 msgr2=0x7fbc60055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbc60053920 0x7fbc60055de0 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7fbc7800d000 tx=0x7fbc78007a00).stop 2022-07-20T03:16:26.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 msgr2=0x7fbc840aea60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 0x7fbc840aea60 secure :-1 s=READY pgs=1218 cs=0 l=1 rev1=1 rx=0x7fbc740032f0 tx=0x7fbc74016000).stop 2022-07-20T03:16:26.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 shutdown_connections 2022-07-20T03:16:26.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbc60053920 0x7fbc60055de0 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbc840b11b0 0x7fbc841599e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbc84060bb0 0x7fbc840aea60 unknown :-1 s=CLOSED pgs=1218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 --2- 172.21.15.61:0/3244392891 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbc8405cda0 0x7fbc84064d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 >> 172.21.15.61:0/3244392891 conn(0x7fbc840b5bb0 msgr2=0x7fbc840b31b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:26.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 shutdown_connections 2022-07-20T03:16:26.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.124+0000 7fbc89387700 1 -- 172.21.15.61:0/3244392891 wait complete. 2022-07-20T03:16:26.135 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:16:26.146 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:26.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 Processor -- start 2022-07-20T03:16:26.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- start start 2022-07-20T03:16:26.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c05c3f0 0x7f468c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f468c0acef0 0x7f468c146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f468c05d480 con 0x7f468c0b1e00 2022-07-20T03:16:26.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f468c0b21e0 con 0x7f468c0acef0 2022-07-20T03:16:26.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c05cf40 con 0x7f468c05c3f0 2022-07-20T03:16:26.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4692250700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f468c0acef0 0x7f468c146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f469124e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c05c3f0 0x7f468c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4691a4f700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f468c0b1e00 0x7f468c0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:55110/0 (socket says 172.21.15.61:55110) 2022-07-20T03:16:26.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4691a4f700 1 -- 172.21.15.61:0/2282308873 learned_addr learned my addr 172.21.15.61:0/2282308873 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:26.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f469124e700 1 -- 172.21.15.61:0/2282308873 >> v2:172.21.15.117:3300/0 conn(0x7f468c0acef0 msgr2=0x7f468c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:26.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f469124e700 1 --2- 172.21.15.61:0/2282308873 >> v2:172.21.15.117:3300/0 conn(0x7f468c0acef0 0x7f468c146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f469124e700 1 -- 172.21.15.61:0/2282308873 >> v1:172.21.15.117:6789/0 conn(0x7f468c0b1e00 legacy=0x7f468c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f469124e700 1 -- 172.21.15.61:0/2282308873 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f468c05d1e0 con 0x7f468c05c3f0 2022-07-20T03:16:26.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f469124e700 1 --2- 172.21.15.61:0/2282308873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c05c3f0 0x7f468c14a6d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f46840032f0 tx=0x7f46840091d0).ready entity=mon.2 client_cookie=7a8008a56d70f63d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:26.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4690a4d700 1 -- 172.21.15.61:0/2282308873 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f468400b6c0 con 0x7f468c05c3f0 2022-07-20T03:16:26.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4690a4d700 1 -- 172.21.15.61:0/2282308873 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f468400b820 con 0x7f468c05c3f0 2022-07-20T03:16:26.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4690a4d700 1 -- 172.21.15.61:0/2282308873 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f468400b990 con 0x7f468c05c3f0 2022-07-20T03:16:26.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- 172.21.15.61:0/2282308873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c05c3f0 msgr2=0x7f468c14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2282308873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c05c3f0 0x7f468c14a6d0 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f46840032f0 tx=0x7f46840091d0).stop 2022-07-20T03:16:26.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- 172.21.15.61:0/2282308873 shutdown_connections 2022-07-20T03:16:26.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2282308873 >> v2:172.21.15.117:3300/0 conn(0x7f468c0acef0 0x7f468c146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2282308873 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c05c3f0 0x7f468c14a6d0 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- 172.21.15.61:0/2282308873 >> 172.21.15.61:0/2282308873 conn(0x7f468c0611c0 msgr2=0x7f468c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:26.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- 172.21.15.61:0/2282308873 shutdown_connections 2022-07-20T03:16:26.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.252+0000 7f4693cb3700 1 -- 172.21.15.61:0/2282308873 wait complete. 2022-07-20T03:16:26.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 Processor -- start 2022-07-20T03:16:26.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 -- start start 2022-07-20T03:16:26.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f468c0acef0 0x7f468c15a220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c0b1e00 0x7f468c09ed60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 0x7f468c1442b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c09f250 con 0x7f468c0acef0 2022-07-20T03:16:26.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0b5470 con 0x7f468c0b5c00 2022-07-20T03:16:26.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f468c0b5710 con 0x7f468c0b1e00 2022-07-20T03:16:26.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f468c0acef0 0x7f468c15a220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f469124e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c0b1e00 0x7f468c09ed60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4692250700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 0x7f468c1442b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4692250700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 0x7f468c1442b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59698/0 (socket says 172.21.15.61:59698) 2022-07-20T03:16:26.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4692250700 1 -- 172.21.15.61:0/2190625798 learned_addr learned my addr 172.21.15.61:0/2190625798 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:26.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 -- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c0b1e00 msgr2=0x7f468c09ed60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c0b1e00 0x7f468c09ed60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 -- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 msgr2=0x7f468c1442b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 0x7f468c1442b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 -- 172.21.15.61:0/2190625798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4684009470 con 0x7f468c0acef0 2022-07-20T03:16:26.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4692250700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 0x7f468c1442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:26.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4691a4f700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f468c0acef0 0x7f468c15a220 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f468c0ac6c0 tx=0x7f4680007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:26.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4680007e50 con 0x7f468c0acef0 2022-07-20T03:16:26.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f46800070d0 con 0x7f468c0acef0 2022-07-20T03:16:26.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f468c15a870 con 0x7f468c0acef0 2022-07-20T03:16:26.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4680007430 con 0x7f468c0acef0 2022-07-20T03:16:26.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.256+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f468c0b50e0 con 0x7f468c0acef0 2022-07-20T03:16:26.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.260+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f468c15aa00 con 0x7f468c0acef0 2022-07-20T03:16:26.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.260+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4680010070 con 0x7f468c0acef0 2022-07-20T03:16:26.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.260+0000 7f467effd700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4674053a40 0x7f4674055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.260+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f468006b370 con 0x7f468c0acef0 2022-07-20T03:16:26.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.260+0000 7f469124e700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4674053a40 0x7f4674055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.264+0000 7f469124e700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4674053a40 0x7f4674055f00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f468401b000 tx=0x7f4684016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:26.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.272+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4680042290 con 0x7f468c0acef0 2022-07-20T03:16:26.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.436+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f468c0b0460 con 0x7f468c0acef0 2022-07-20T03:16:26.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.436+0000 7f467effd700 1 -- 172.21.15.61:0/2190625798 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 68 v68) v1 ==== 94+0+1426 (secure 0 0 0) 0x7f4680018030 con 0x7f468c0acef0 2022-07-20T03:16:26.444 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:26.445 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":68,"default_fscid":12,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[{"mdsmap":{"epoch":68,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:23.653073+0000","modified":"2022-07-20T03:16:26.112829+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[],"up":{},"failed":[],"damaged":[],"stopped":[],"info":{},"data_pools":[22],"metadata_pool":24,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":12}]} 2022-07-20T03:16:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4674053a40 msgr2=0x7f4674055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4674053a40 0x7f4674055f00 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f468401b000 tx=0x7f4684016000).stop 2022-07-20T03:16:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f468c0acef0 msgr2=0x7f468c15a220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f468c0acef0 0x7f468c15a220 secure :-1 s=READY pgs=1219 cs=0 l=1 rev1=1 rx=0x7f468c0ac6c0 tx=0x7f4680007be0).stop 2022-07-20T03:16:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 shutdown_connections 2022-07-20T03:16:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4674053a40 0x7f4674055f00 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f468c0b5c00 0x7f468c1442b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f468c0b1e00 0x7f468c09ed60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 --2- 172.21.15.61:0/2190625798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f468c0acef0 0x7f468c15a220 unknown :-1 s=CLOSED pgs=1219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 >> 172.21.15.61:0/2190625798 conn(0x7f468c05c3f0 msgr2=0x7f468c145810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 shutdown_connections 2022-07-20T03:16:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.440+0000 7f4693cb3700 1 -- 172.21.15.61:0/2190625798 wait complete. 2022-07-20T03:16:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 68 2022-07-20T03:16:26.459 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:16:26.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 Processor -- start 2022-07-20T03:16:26.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 -- start start 2022-07-20T03:16:26.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80b5970 0x7fb0e80ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0e80af8c0 0x7fb0e80ae5f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.569 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0e80aeae0 0x7fb0e805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e80b0b80 con 0x7fb0e80af8c0 2022-07-20T03:16:26.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb0e805d4d0 con 0x7fb0e80aeae0 2022-07-20T03:16:26.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.560+0000 7fb0ee4cc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e80b5d50 con 0x7fb0e80b5970 2022-07-20T03:16:26.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0e80af8c0 0x7fb0e80ae5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0e80af8c0 0x7fb0e80ae5f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50022/0 (socket says 172.21.15.61:50022) 2022-07-20T03:16:26.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 -- 172.21.15.61:0/3702953815 learned_addr learned my addr 172.21.15.61:0/3702953815 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:26.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0edccb700 1 --2- 172.21.15.61:0/3702953815 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80aeae0 0x7fb0e805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ed4ca700 1 --2- 172.21.15.61:0/3702953815 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80b5970 0x7fb0e80ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 -- 172.21.15.61:0/3702953815 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80b5970 msgr2=0x7fb0e80ae100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:26.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 --2- 172.21.15.61:0/3702953815 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80b5970 0x7fb0e80ae100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 -- 172.21.15.61:0/3702953815 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80aeae0 msgr2=0x7fb0e805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:26.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 --2- 172.21.15.61:0/3702953815 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80aeae0 0x7fb0e805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 -- 172.21.15.61:0/3702953815 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e80b0db0 con 0x7fb0e80af8c0 2022-07-20T03:16:26.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0eccc9700 1 --2- 172.21.15.61:0/3702953815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80af8c0 0x7fb0e80ae5f0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7fb0e4002f70 tx=0x7fb0e4006eb0).ready entity=mon.0 client_cookie=3fc789e58b3fd69 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:26.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0dffff700 1 -- 172.21.15.61:0/3702953815 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e400d570 con 0x7fb0e80af8c0 2022-07-20T03:16:26.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0dffff700 1 -- 172.21.15.61:0/3702953815 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e4000e90 con 0x7fb0e80af8c0 2022-07-20T03:16:26.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0dffff700 1 -- 172.21.15.61:0/3702953815 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e400d900 con 0x7fb0e80af8c0 2022-07-20T03:16:26.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/3702953815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80af8c0 msgr2=0x7fb0e80ae5f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/3702953815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80af8c0 0x7fb0e80ae5f0 secure :-1 s=READY pgs=1220 cs=0 l=1 rev1=1 rx=0x7fb0e4002f70 tx=0x7fb0e4006eb0).stop 2022-07-20T03:16:26.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/3702953815 shutdown_connections 2022-07-20T03:16:26.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/3702953815 >> v2:172.21.15.117:3300/0 conn(0x7fb0e80aeae0 0x7fb0e805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/3702953815 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80af8c0 0x7fb0e80ae5f0 unknown :-1 s=CLOSED pgs=1220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/3702953815 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80b5970 0x7fb0e80ae100 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/3702953815 >> 172.21.15.61:0/3702953815 conn(0x7fb0e8063490 msgr2=0x7fb0e80b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:26.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/3702953815 shutdown_connections 2022-07-20T03:16:26.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/3702953815 wait complete. 2022-07-20T03:16:26.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 Processor -- start 2022-07-20T03:16:26.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 -- start start 2022-07-20T03:16:26.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80aeae0 0x7fb0e8143980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e80af8c0 0x7fb0e810e450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.564+0000 7fb0ee4cc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 0x7fb0e8144000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ee4cc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8136f90 con 0x7fb0e80b5970 2022-07-20T03:16:26.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ee4cc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e8145020 con 0x7fb0e80af8c0 2022-07-20T03:16:26.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ee4cc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0e81560b0 con 0x7fb0e80aeae0 2022-07-20T03:16:26.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 0x7fb0e8144000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 0x7fb0e8144000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50032/0 (socket says 172.21.15.61:50032) 2022-07-20T03:16:26.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 -- 172.21.15.61:0/2458318852 learned_addr learned my addr 172.21.15.61:0/2458318852 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:26.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0eccc9700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e80af8c0 0x7fb0e810e450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ed4ca700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80aeae0 0x7fb0e8143980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 -- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80aeae0 msgr2=0x7fb0e8143980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80aeae0 0x7fb0e8143980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 -- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e80af8c0 msgr2=0x7fb0e810e450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:26.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e80af8c0 0x7fb0e810e450 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:26.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 -- 172.21.15.61:0/2458318852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0e400b1e0 con 0x7fb0e80b5970 2022-07-20T03:16:26.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0edccb700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 0x7fb0e8144000 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7fb0e0006e00 tx=0x7fb0e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:26.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e000eeb0 con 0x7fb0e80b5970 2022-07-20T03:16:26.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb0e000c850 con 0x7fb0e80b5970 2022-07-20T03:16:26.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb0e81452a0 con 0x7fb0e80b5970 2022-07-20T03:16:26.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb0e000cbb0 con 0x7fb0e80b5970 2022-07-20T03:16:26.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0e813c5f0 con 0x7fb0e80b5970 2022-07-20T03:16:26.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb0e81467f0 con 0x7fb0e80b5970 2022-07-20T03:16:26.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb0e000cd10 con 0x7fb0e80b5970 2022-07-20T03:16:26.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0de7fc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0d0053a40 0x7fb0d0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:26.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fb0e006c1a0 con 0x7fb0e80b5970 2022-07-20T03:16:26.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.568+0000 7fb0ed4ca700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0d0053a40 0x7fb0d0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:26.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.572+0000 7fb0ed4ca700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0d0053a40 0x7fb0d0055f00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fb0d800b000 tx=0x7fb0d8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:26.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.576+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0e81467f0 con 0x7fb0e80b5970 2022-07-20T03:16:26.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:26.740+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fb0e81467f0 con 0x7fb0e80b5970 2022-07-20T03:16:27.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.680+0000 7fb0de7fc700 1 -- 172.21.15.61:0/2458318852 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v69) v1 ==== 104+0+0 (secure 0 0 0) 0x7fb0e81467f0 con 0x7fb0e80b5970 2022-07-20T03:16:27.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0d0053a40 msgr2=0x7fb0d0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:27.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0d0053a40 0x7fb0d0055f00 secure :-1 s=READY pgs=588 cs=0 l=1 rev1=1 rx=0x7fb0d800b000 tx=0x7fb0d8006df0).stop 2022-07-20T03:16:27.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 msgr2=0x7fb0e8144000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:27.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 0x7fb0e8144000 secure :-1 s=READY pgs=1221 cs=0 l=1 rev1=1 rx=0x7fb0e0006e00 tx=0x7fb0e00091d0).stop 2022-07-20T03:16:27.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 shutdown_connections 2022-07-20T03:16:27.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb0d0053a40 0x7fb0d0055f00 unknown :-1 s=CLOSED pgs=588 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0e80af8c0 0x7fb0e810e450 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb0e80b5970 0x7fb0e8144000 unknown :-1 s=CLOSED pgs=1221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 --2- 172.21.15.61:0/2458318852 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0e80aeae0 0x7fb0e8143980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 >> 172.21.15.61:0/2458318852 conn(0x7fb0e8063490 msgr2=0x7fb0e80b37e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:27.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 shutdown_connections 2022-07-20T03:16:27.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.688+0000 7fb0ee4cc700 1 -- 172.21.15.61:0/2458318852 wait complete. 2022-07-20T03:16:27.708 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:16:27.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 Processor -- start 2022-07-20T03:16:27.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 -- start start 2022-07-20T03:16:27.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:27.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3f440b4bc0 0x7f3f44141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:27.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3f44066b10 con 0x7f3f440b3cf0 2022-07-20T03:16:27.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3f440b6c20 con 0x7f3f440b4bc0 2022-07-20T03:16:27.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.824+0000 7f3f4942e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440aead0 con 0x7f3f440b6840 2022-07-20T03:16:27.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3f440b4bc0 0x7f3f44141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:27.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f437fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3f440b3cf0 0x7f3f44146040 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46634/0 (socket says 172.21.15.61:46634) 2022-07-20T03:16:27.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f437fe700 1 -- 172.21.15.61:0/1217466228 learned_addr learned my addr 172.21.15.61:0/1217466228 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:27.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3f440b4bc0 0x7f3f44141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50040/0 (socket says 172.21.15.61:50040) 2022-07-20T03:16:27.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f43fff700 1 --2- 172.21.15.61:0/1217466228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:27.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 -- 172.21.15.61:0/1217466228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 msgr2=0x7f3f440b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:27.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 --2- 172.21.15.61:0/1217466228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 -- 172.21.15.61:0/1217466228 >> v1:172.21.15.61:6789/0 conn(0x7f3f440b3cf0 legacy=0x7f3f44146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:27.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f42ffd700 1 -- 172.21.15.61:0/1217466228 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3445199152 0 0) 0x7f3f44066b10 con 0x7f3f440b3cf0 2022-07-20T03:16:27.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 -- 172.21.15.61:0/1217466228 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f440aed70 con 0x7f3f440b4bc0 2022-07-20T03:16:27.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f48c2d700 1 --2- 172.21.15.61:0/1217466228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b4bc0 0x7f3f44141a90 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f3f3c0032f0 tx=0x7f3f3c0091d0).ready entity=mon.0 client_cookie=4615335c9b6a7cb9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:27.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f42ffd700 1 -- 172.21.15.61:0/1217466228 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f3c009ea0 con 0x7f3f440b4bc0 2022-07-20T03:16:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f42ffd700 1 -- 172.21.15.61:0/1217466228 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f3c00b8a0 con 0x7f3f440b4bc0 2022-07-20T03:16:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f42ffd700 1 -- 172.21.15.61:0/1217466228 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f3c00ba10 con 0x7f3f440b4bc0 2022-07-20T03:16:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 -- 172.21.15.61:0/1217466228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b4bc0 msgr2=0x7f3f44141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:27.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 --2- 172.21.15.61:0/1217466228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b4bc0 0x7f3f44141a90 secure :-1 s=READY pgs=1223 cs=0 l=1 rev1=1 rx=0x7f3f3c0032f0 tx=0x7f3f3c0091d0).stop 2022-07-20T03:16:27.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 -- 172.21.15.61:0/1217466228 shutdown_connections 2022-07-20T03:16:27.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 --2- 172.21.15.61:0/1217466228 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b4bc0 0x7f3f44141a90 unknown :-1 s=CLOSED pgs=1223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 --2- 172.21.15.61:0/1217466228 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 -- 172.21.15.61:0/1217466228 >> 172.21.15.61:0/1217466228 conn(0x7f3f440b9e10 msgr2=0x7f3f440b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:27.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 -- 172.21.15.61:0/1217466228 shutdown_connections 2022-07-20T03:16:27.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.828+0000 7f3f4942e700 1 -- 172.21.15.61:0/1217466228 wait complete. 2022-07-20T03:16:27.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 Processor -- start 2022-07-20T03:16:27.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 -- start start 2022-07-20T03:16:27.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 0x7f3f4413bb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:27.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f440b4bc0 0x7f3f441455d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:27.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b2040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:27.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4413a9c0 con 0x7f3f440b3cf0 2022-07-20T03:16:27.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3f4413c630 con 0x7f3f440b4bc0 2022-07-20T03:16:27.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3f440b1de0 con 0x7f3f440b6840 2022-07-20T03:16:27.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 0x7f3f4413bb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:27.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 0x7f3f4413bb10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50042/0 (socket says 172.21.15.61:50042) 2022-07-20T03:16:27.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 -- 172.21.15.61:0/4157270336 learned_addr learned my addr 172.21.15.61:0/4157270336 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:27.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f437fe700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f440b4bc0 0x7f3f441455d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:27.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f48c2d700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b2040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:27.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 -- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 msgr2=0x7f3f440b2040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:27.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b2040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 -- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f440b4bc0 msgr2=0x7f3f441455d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:27.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f440b4bc0 0x7f3f441455d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:27.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 -- 172.21.15.61:0/4157270336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3f3c009470 con 0x7f3f440b3cf0 2022-07-20T03:16:27.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f48c2d700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:27.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f43fff700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 0x7f3f4413bb10 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f3f340030b0 tx=0x7f3f34008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:27.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f34011780 con 0x7f3f440b3cf0 2022-07-20T03:16:27.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3f340118e0 con 0x7f3f440b3cf0 2022-07-20T03:16:27.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3f34011c40 con 0x7f3f440b3cf0 2022-07-20T03:16:27.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3f4413ad70 con 0x7f3f440b3cf0 2022-07-20T03:16:27.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.832+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3f440b26d0 con 0x7f3f440b3cf0 2022-07-20T03:16:27.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.836+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3f440b50e0 con 0x7f3f440b3cf0 2022-07-20T03:16:27.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.836+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3f3401c460 con 0x7f3f440b3cf0 2022-07-20T03:16:27.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.836+0000 7f3f417fa700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f24053a40 0x7f3f24055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:27.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.836+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(123..123 src has 1..123) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f3f3406c0b0 con 0x7f3f440b3cf0 2022-07-20T03:16:27.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.836+0000 7f3f437fe700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f24053a40 0x7f3f24055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:27.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.836+0000 7f3f437fe700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f24053a40 0x7f3f24055f00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f3f3800d000 tx=0x7f3f38007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:27.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.844+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3f34042ec0 con 0x7f3f440b3cf0 2022-07-20T03:16:27.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:27.968+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3f440574e0 con 0x7f3f440b3cf0 2022-07-20T03:16:28.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.688+0000 7f3f417fa700 1 -- 172.21.15.61:0/4157270336 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v124) v1 ==== 185+0+0 (secure 0 0 0) 0x7f3f340464e0 con 0x7f3f440b3cf0 2022-07-20T03:16:28.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.688+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f24053a40 msgr2=0x7f3f24055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:28.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f24053a40 0x7f3f24055f00 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7f3f3800d000 tx=0x7f3f38007a00).stop 2022-07-20T03:16:28.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 msgr2=0x7f3f4413bb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 0x7f3f4413bb10 secure :-1 s=READY pgs=1224 cs=0 l=1 rev1=1 rx=0x7f3f340030b0 tx=0x7f3f34008040).stop 2022-07-20T03:16:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 shutdown_connections 2022-07-20T03:16:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3f24053a40 0x7f3f24055f00 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3f440b4bc0 0x7f3f441455d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3f440b6840 0x7f3f440b2040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 --2- 172.21.15.61:0/4157270336 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3f440b3cf0 0x7f3f4413bb10 unknown :-1 s=CLOSED pgs=1224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 >> 172.21.15.61:0/4157270336 conn(0x7f3f440b9e10 msgr2=0x7f3f44066d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:28.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 shutdown_connections 2022-07-20T03:16:28.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.692+0000 7f3f4942e700 1 -- 172.21.15.61:0/4157270336 wait complete. 2022-07-20T03:16:28.701 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:16:28.707 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:16:28.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 Processor -- start 2022-07-20T03:16:28.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 -- start start 2022-07-20T03:16:28.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3c1c0b3560 0x7f3c1c146b10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:28.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c1c05cc40 0x7f3c1c0b1a20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:28.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3c1c146440 con 0x7f3c1c147000 2022-07-20T03:16:28.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b3940 con 0x7f3c1c05cc40 2022-07-20T03:16:28.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2359c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3c1c0ab490 con 0x7f3c1c0b3560 2022-07-20T03:16:28.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c21d99700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c1c05cc40 0x7f3c1c0b1a20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:28.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c21d99700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3c1c05cc40 0x7f3c1c0b1a20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50050/0 (socket says 172.21.15.61:50050) 2022-07-20T03:16:28.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c21d99700 1 -- 172.21.15.61:0/2164007262 learned_addr learned my addr 172.21.15.61:0/2164007262 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:28.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.816+0000 7f3c2259a700 1 --2- 172.21.15.61:0/2164007262 >> v2:172.21.15.117:3300/0 conn(0x7f3c1c0b3560 0x7f3c1c146b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:28.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21d99700 1 -- 172.21.15.61:0/2164007262 >> v2:172.21.15.117:3300/0 conn(0x7f3c1c0b3560 msgr2=0x7f3c1c146b10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:28.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21d99700 1 --2- 172.21.15.61:0/2164007262 >> v2:172.21.15.117:3300/0 conn(0x7f3c1c0b3560 0x7f3c1c146b10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21d99700 1 -- 172.21.15.61:0/2164007262 >> v1:172.21.15.117:6789/0 conn(0x7f3c1c147000 legacy=0x7f3c1c0b1f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:28.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21d99700 1 -- 172.21.15.61:0/2164007262 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c1c146670 con 0x7f3c1c05cc40 2022-07-20T03:16:28.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21598700 1 -- 172.21.15.61:0/2164007262 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1532885066 0 0) 0x7f3c1c146440 con 0x7f3c1c147000 2022-07-20T03:16:28.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21d99700 1 --2- 172.21.15.61:0/2164007262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b1a20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f3c0c002f70 tx=0x7f3c0c006eb0).ready entity=mon.0 client_cookie=103a3e297ea586cb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:28.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21598700 1 -- 172.21.15.61:0/2164007262 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c0c00d5c0 con 0x7f3c1c05cc40 2022-07-20T03:16:28.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21598700 1 -- 172.21.15.61:0/2164007262 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c0c000e90 con 0x7f3c1c05cc40 2022-07-20T03:16:28.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c21598700 1 -- 172.21.15.61:0/2164007262 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c0c00d910 con 0x7f3c1c05cc40 2022-07-20T03:16:28.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 -- 172.21.15.61:0/2164007262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 msgr2=0x7f3c1c0b1a20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:28.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 --2- 172.21.15.61:0/2164007262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b1a20 secure :-1 s=READY pgs=1225 cs=0 l=1 rev1=1 rx=0x7f3c0c002f70 tx=0x7f3c0c006eb0).stop 2022-07-20T03:16:28.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 -- 172.21.15.61:0/2164007262 shutdown_connections 2022-07-20T03:16:28.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 --2- 172.21.15.61:0/2164007262 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b1a20 unknown :-1 s=CLOSED pgs=1225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 --2- 172.21.15.61:0/2164007262 >> v2:172.21.15.117:3300/0 conn(0x7f3c1c0b3560 0x7f3c1c146b10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 -- 172.21.15.61:0/2164007262 >> 172.21.15.61:0/2164007262 conn(0x7f3c1c0b6b20 msgr2=0x7f3c1c0b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:28.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 -- 172.21.15.61:0/2164007262 shutdown_connections 2022-07-20T03:16:28.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 -- 172.21.15.61:0/2164007262 wait complete. 2022-07-20T03:16:28.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 Processor -- start 2022-07-20T03:16:28.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.820+0000 7f3c2359c700 1 -- start start 2022-07-20T03:16:28.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b0940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:28.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c1c0b3560 0x7f3c1c151e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:28.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c1c147000 0x7f3c1c13e640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:28.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c155410 con 0x7f3c1c05cc40 2022-07-20T03:16:28.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b2ad0 con 0x7f3c1c0b3560 2022-07-20T03:16:28.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3c1c0b1960 con 0x7f3c1c147000 2022-07-20T03:16:28.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b0940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:28.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b0940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50056/0 (socket says 172.21.15.61:50056) 2022-07-20T03:16:28.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 -- 172.21.15.61:0/4133242544 learned_addr learned my addr 172.21.15.61:0/4133242544 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:28.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c21d99700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c1c0b3560 0x7f3c1c151e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:28.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c22d9b700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c1c147000 0x7f3c1c13e640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:28.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 -- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c1c147000 msgr2=0x7f3c1c13e640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:28.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c1c147000 0x7f3c1c13e640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 -- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c1c0b3560 msgr2=0x7f3c1c151e60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:28.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c1c0b3560 0x7f3c1c151e60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:28.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 -- 172.21.15.61:0/4133242544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c0c00b1e0 con 0x7f3c1c05cc40 2022-07-20T03:16:28.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2259a700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b0940 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f3c1800c000 tx=0x7f3c18006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:28.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c18010780 con 0x7f3c1c05cc40 2022-07-20T03:16:28.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3c180108e0 con 0x7f3c1c05cc40 2022-07-20T03:16:28.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c1c0b1c00 con 0x7f3c1c05cc40 2022-07-20T03:16:28.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3c18010c40 con 0x7f3c1c05cc40 2022-07-20T03:16:28.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.824+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c1c146060 con 0x7f3c1c05cc40 2022-07-20T03:16:28.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.828+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3c1801c460 con 0x7f3c1c05cc40 2022-07-20T03:16:28.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.828+0000 7f3c0b7fe700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3bfc0539b0 0x7f3bfc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:28.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.828+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(124..124 src has 1..124) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f3c1806c0c0 con 0x7f3c1c05cc40 2022-07-20T03:16:28.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.828+0000 7f3c21d99700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3bfc0539b0 0x7f3bfc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:28.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.828+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c1c0ab110 con 0x7f3c1c05cc40 2022-07-20T03:16:28.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.828+0000 7f3c21d99700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3bfc0539b0 0x7f3bfc055e70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f3c0c01c000 tx=0x7f3c0c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:28.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.832+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3c18043120 con 0x7f3c1c05cc40 2022-07-20T03:16:28.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:28.968+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f3c1c0574e0 con 0x7f3c1c05cc40 2022-07-20T03:16:29.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.696+0000 7f3c0b7fe700 1 -- 172.21.15.61:0/4133242544 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v125) v1 ==== 173+0+0 (secure 0 0 0) 0x7f3c18046740 con 0x7f3c1c05cc40 2022-07-20T03:16:29.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3bfc0539b0 msgr2=0x7f3bfc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3bfc0539b0 0x7f3bfc055e70 secure :-1 s=READY pgs=590 cs=0 l=1 rev1=1 rx=0x7f3c0c01c000 tx=0x7f3c0c015000).stop 2022-07-20T03:16:29.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 msgr2=0x7f3c1c0b0940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b0940 secure :-1 s=READY pgs=1226 cs=0 l=1 rev1=1 rx=0x7f3c1800c000 tx=0x7f3c18006df0).stop 2022-07-20T03:16:29.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 shutdown_connections 2022-07-20T03:16:29.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3bfc0539b0 0x7f3bfc055e70 unknown :-1 s=CLOSED pgs=590 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3c1c0b3560 0x7f3c1c151e60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3c1c147000 0x7f3c1c13e640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 --2- 172.21.15.61:0/4133242544 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3c1c05cc40 0x7f3c1c0b0940 unknown :-1 s=CLOSED pgs=1226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 >> 172.21.15.61:0/4133242544 conn(0x7f3c1c0b6b20 msgr2=0x7f3c1c061290 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:29.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 shutdown_connections 2022-07-20T03:16:29.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.700+0000 7f3c2359c700 1 -- 172.21.15.61:0/4133242544 wait complete. 2022-07-20T03:16:29.711 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:16:29.720 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:16:29.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 Processor -- start 2022-07-20T03:16:29.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 -- start start 2022-07-20T03:16:29.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f58000b9ea0 0x7f58000b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:29.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58000b3df0 0x7f5800141c80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:29.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5800142dc0 con 0x7f58000b00f0 2022-07-20T03:16:29.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f58000ba280 con 0x7f58000b3df0 2022-07-20T03:16:29.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.828+0000 7f5805ad7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5800146280 con 0x7f58000b9ea0 2022-07-20T03:16:29.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f58000b3df0 0x7f5800141c80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:29.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57feffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f58000b00f0 0x7f5800143490 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46662/0 (socket says 172.21.15.61:46662) 2022-07-20T03:16:29.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57feffd700 1 -- 172.21.15.61:0/2396536403 learned_addr learned my addr 172.21.15.61:0/2396536403 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:29.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57ff7fe700 1 --2- 172.21.15.61:0/2396536403 >> v2:172.21.15.117:3300/0 conn(0x7f58000b9ea0 0x7f58000b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:29.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fe7fc700 1 -- 172.21.15.61:0/2396536403 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1664415970 0 0) 0x7f5800142dc0 con 0x7f58000b00f0 2022-07-20T03:16:29.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fe7fc700 1 -- 172.21.15.61:0/2396536403 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f57e0003260 con 0x7f58000b00f0 2022-07-20T03:16:29.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fffff700 1 -- 172.21.15.61:0/2396536403 >> v2:172.21.15.117:3300/0 conn(0x7f58000b9ea0 msgr2=0x7f58000b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:29.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fffff700 1 --2- 172.21.15.61:0/2396536403 >> v2:172.21.15.117:3300/0 conn(0x7f58000b9ea0 0x7f58000b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fffff700 1 -- 172.21.15.61:0/2396536403 >> v1:172.21.15.61:6789/0 conn(0x7f58000b00f0 legacy=0x7f5800143490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fffff700 1 -- 172.21.15.61:0/2396536403 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5800142ff0 con 0x7f58000b3df0 2022-07-20T03:16:29.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57ff7fe700 1 --2- 172.21.15.61:0/2396536403 >> v2:172.21.15.117:3300/0 conn(0x7f58000b9ea0 0x7f58000b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:29.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fffff700 1 --2- 172.21.15.61:0/2396536403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b3df0 0x7f5800141c80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f57f80032f0 tx=0x7f57f80091d0).ready entity=mon.0 client_cookie=555dfd19c5b0d5ae server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:29.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fe7fc700 1 -- 172.21.15.61:0/2396536403 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f800b600 con 0x7f58000b3df0 2022-07-20T03:16:29.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fe7fc700 1 -- 172.21.15.61:0/2396536403 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f800b760 con 0x7f58000b3df0 2022-07-20T03:16:29.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f57fe7fc700 1 -- 172.21.15.61:0/2396536403 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f800b8d0 con 0x7f58000b3df0 2022-07-20T03:16:29.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 -- 172.21.15.61:0/2396536403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b3df0 msgr2=0x7f5800141c80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2396536403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b3df0 0x7f5800141c80 secure :-1 s=READY pgs=1228 cs=0 l=1 rev1=1 rx=0x7f57f80032f0 tx=0x7f57f80091d0).stop 2022-07-20T03:16:29.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 -- 172.21.15.61:0/2396536403 shutdown_connections 2022-07-20T03:16:29.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2396536403 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b3df0 0x7f5800141c80 unknown :-1 s=CLOSED pgs=1228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2396536403 >> v2:172.21.15.117:3300/0 conn(0x7f58000b9ea0 0x7f58000b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 -- 172.21.15.61:0/2396536403 >> 172.21.15.61:0/2396536403 conn(0x7f5800067960 msgr2=0x7f58000b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:29.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 -- 172.21.15.61:0/2396536403 shutdown_connections 2022-07-20T03:16:29.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.832+0000 7f5805ad7700 1 -- 172.21.15.61:0/2396536403 wait complete. 2022-07-20T03:16:29.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 Processor -- start 2022-07-20T03:16:29.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 -- start start 2022-07-20T03:16:29.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58000b00f0 0x7f58000adad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:29.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58000b3df0 0x7f58000af130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:29.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b9ea0 0x7f5800152070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:29.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58001148a0 con 0x7f58000b9ea0 2022-07-20T03:16:29.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58000af7f0 con 0x7f58000b00f0 2022-07-20T03:16:29.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f58000afa90 con 0x7f58000b3df0 2022-07-20T03:16:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b9ea0 0x7f5800152070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57ff7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58000b00f0 0x7f58000adad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57ff7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58000b00f0 0x7f58000adad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59748/0 (socket says 172.21.15.61:59748) 2022-07-20T03:16:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57feffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58000b3df0 0x7f58000af130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:29.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57ff7fe700 1 -- 172.21.15.61:0/2925874508 learned_addr learned my addr 172.21.15.61:0/2925874508 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 -- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58000b3df0 msgr2=0x7f58000af130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58000b3df0 0x7f58000af130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 -- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58000b00f0 msgr2=0x7f58000adad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58000b00f0 0x7f58000adad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 -- 172.21.15.61:0/2925874508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57f000b180 con 0x7f58000b9ea0 2022-07-20T03:16:29.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fffff700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b9ea0 0x7f5800152070 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f57f80032f0 tx=0x7f57f8016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:29.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f8011b60 con 0x7f58000b9ea0 2022-07-20T03:16:29.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f57f8011cc0 con 0x7f58000b9ea0 2022-07-20T03:16:29.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57f8009470 con 0x7f58000b9ea0 2022-07-20T03:16:29.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57f801f5e0 con 0x7f58000b9ea0 2022-07-20T03:16:29.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.836+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5800114370 con 0x7f58000b9ea0 2022-07-20T03:16:29.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.840+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f57f8026030 con 0x7f58000b9ea0 2022-07-20T03:16:29.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.840+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f58000aeb30 con 0x7f58000b9ea0 2022-07-20T03:16:29.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.840+0000 7f57fcff9700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f57e4053a60 0x7f57e4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:29.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.840+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f57f806c320 con 0x7f58000b9ea0 2022-07-20T03:16:29.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.840+0000 7f57ff7fe700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f57e4053a60 0x7f57e4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:29.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.840+0000 7f57ff7fe700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f57e4053a60 0x7f57e4055f20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f57f000b000 tx=0x7f57f0006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:29.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.844+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f58000aeb30 con 0x7f58000b9ea0 2022-07-20T03:16:29.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.972+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f58000aeb30 con 0x7f58000b9ea0 2022-07-20T03:16:29.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f57fcff9700 1 -- 172.21.15.61:0/2925874508 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v19) v1 ==== 104+0+0 (secure 0 0 0) 0x7f58000aeb30 con 0x7f58000b9ea0 2022-07-20T03:16:29.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f57e4053a60 msgr2=0x7f57e4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f57e4053a60 0x7f57e4055f20 secure :-1 s=READY pgs=591 cs=0 l=1 rev1=1 rx=0x7f57f000b000 tx=0x7f57f0006df0).stop 2022-07-20T03:16:29.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b9ea0 msgr2=0x7f5800152070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:29.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b9ea0 0x7f5800152070 secure :-1 s=READY pgs=1229 cs=0 l=1 rev1=1 rx=0x7f57f80032f0 tx=0x7f57f8016000).stop 2022-07-20T03:16:29.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 shutdown_connections 2022-07-20T03:16:29.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f57e4053a60 0x7f57e4055f20 unknown :-1 s=CLOSED pgs=591 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f58000b9ea0 0x7f5800152070 unknown :-1 s=CLOSED pgs=1229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.980+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f58000b3df0 0x7f58000af130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.984+0000 7f5805ad7700 1 --2- 172.21.15.61:0/2925874508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f58000b00f0 0x7f58000adad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:29.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.984+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 >> 172.21.15.61:0/2925874508 conn(0x7f5800067960 msgr2=0x7f580005c4c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:29.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.984+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 shutdown_connections 2022-07-20T03:16:29.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:29.984+0000 7f5805ad7700 1 -- 172.21.15.61:0/2925874508 wait complete. 2022-07-20T03:16:29.998 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new' 2022-07-20T03:16:30.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.096+0000 7f7c91674700 1 Processor -- start 2022-07-20T03:16:30.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.096+0000 7f7c91674700 1 -- start start 2022-07-20T03:16:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7c8c0b9d50 0x7f7c8c14a490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c147b90 0x7f7c8c0b4490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7c8c0ba130 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7c8c0b5ac0 con 0x7f7c8c0b9d50 2022-07-20T03:16:30.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c14a980 con 0x7f7c8c147b90 2022-07-20T03:16:30.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8b7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7c8c0b3ca0 0x7f7c8c147480 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46674/0 (socket says 172.21.15.61:46674) 2022-07-20T03:16:30.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8b7fe700 1 -- 172.21.15.61:0/867111616 learned_addr learned my addr 172.21.15.61:0/867111616 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:30.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8bfff700 1 --2- 172.21.15.61:0/867111616 >> v2:172.21.15.117:3300/0 conn(0x7f7c8c0b9d50 0x7f7c8c14a490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c90e73700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c147b90 0x7f7c8c0b4490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8affd700 1 -- 172.21.15.61:0/867111616 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3479523793 0 0) 0x7f7c8c0ba130 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8affd700 1 -- 172.21.15.61:0/867111616 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7c78003260 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8bfff700 1 -- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c147b90 msgr2=0x7f7c8c0b4490 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:30.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8bfff700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c147b90 0x7f7c8c0b4490 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8bfff700 1 -- 172.21.15.61:0/867111616 >> v1:172.21.15.61:6789/0 conn(0x7f7c8c0b3ca0 legacy=0x7f7c8c147480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8bfff700 1 -- 172.21.15.61:0/867111616 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c14aba0 con 0x7f7c8c0b9d50 2022-07-20T03:16:30.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c90e73700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c147b90 0x7f7c8c0b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:30.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8bfff700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c0b9d50 0x7f7c8c14a490 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f7c8c14aac0 tx=0x7f7c80006df0).ready entity=mon.1 client_cookie=95b72c23028b103b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:30.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8affd700 1 -- 172.21.15.61:0/867111616 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c8000d570 con 0x7f7c8c0b9d50 2022-07-20T03:16:30.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8affd700 1 -- 172.21.15.61:0/867111616 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c80000e90 con 0x7f7c8c0b9d50 2022-07-20T03:16:30.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c8affd700 1 -- 172.21.15.61:0/867111616 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c8000d8c0 con 0x7f7c8c0b9d50 2022-07-20T03:16:30.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 -- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c0b9d50 msgr2=0x7f7c8c14a490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.100+0000 7f7c91674700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c0b9d50 0x7f7c8c14a490 secure :-1 s=READY pgs=560 cs=0 l=1 rev1=1 rx=0x7f7c8c14aac0 tx=0x7f7c80006df0).stop 2022-07-20T03:16:30.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 -- 172.21.15.61:0/867111616 shutdown_connections 2022-07-20T03:16:30.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c147b90 0x7f7c8c0b4490 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 --2- 172.21.15.61:0/867111616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c0b9d50 0x7f7c8c14a490 unknown :-1 s=CLOSED pgs=560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 -- 172.21.15.61:0/867111616 >> 172.21.15.61:0/867111616 conn(0x7f7c8c0678b0 msgr2=0x7f7c8c0b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:30.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 -- 172.21.15.61:0/867111616 shutdown_connections 2022-07-20T03:16:30.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 -- 172.21.15.61:0/867111616 wait complete. 2022-07-20T03:16:30.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 Processor -- start 2022-07-20T03:16:30.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.104+0000 7f7c91674700 1 -- start start 2022-07-20T03:16:30.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 0x7f7c8c156280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c0b9d50 0x7f7c8c0aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c147b90 0x7f7c8c062f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0ad4a0 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0af040 con 0x7f7c8c147b90 2022-07-20T03:16:30.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7c8c0649b0 con 0x7f7c8c0b9d50 2022-07-20T03:16:30.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 0x7f7c8c156280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 0x7f7c8c156280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50082/0 (socket says 172.21.15.61:50082) 2022-07-20T03:16:30.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 -- 172.21.15.61:0/3168666434 learned_addr learned my addr 172.21.15.61:0/3168666434 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:30.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8b7fe700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c0b9d50 0x7f7c8c0aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c90e73700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c147b90 0x7f7c8c062f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 -- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c0b9d50 msgr2=0x7f7c8c0aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c0b9d50 0x7f7c8c0aeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 -- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c147b90 msgr2=0x7f7c8c062f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c147b90 0x7f7c8c062f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 -- 172.21.15.61:0/3168666434 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c84009470 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c8bfff700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 0x7f7c8c156280 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7f7c8c14aac0 tx=0x7f7c80014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:30.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c80002f70 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7c8001e400 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c8000b180 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7c8001e760 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.108+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c8c0ae2c0 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.112+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c8c0acfa0 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.112+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7c80025030 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.112+0000 7f7c897fa700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7c6c053ad0 0x7f7c6c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.112+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7c8006d560 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.112+0000 7f7c8b7fe700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7c6c053ad0 0x7f7c6c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.112+0000 7f7c8b7fe700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7c6c053ad0 0x7f7c6c055f90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f7c7c00b000 tx=0x7f7c7c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:30.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.116+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7c80043b20 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.284+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]} v 0) v1 -- 0x7f7c8c14ae40 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.700+0000 7f7c897fa700 1 -- 172.21.15.61:0/3168666434 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_fs_new"]}]=0 v436) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7c80023070 con 0x7f7c8c0b3ca0 2022-07-20T03:16:30.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7c6c053ad0 msgr2=0x7f7c6c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7c6c053ad0 0x7f7c6c055f90 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7f7c7c00b000 tx=0x7f7c7c009040).stop 2022-07-20T03:16:30.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 msgr2=0x7f7c8c156280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 0x7f7c8c156280 secure :-1 s=READY pgs=1231 cs=0 l=1 rev1=1 rx=0x7f7c8c14aac0 tx=0x7f7c80014000).stop 2022-07-20T03:16:30.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 shutdown_connections 2022-07-20T03:16:30.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7c6c053ad0 0x7f7c6c055f90 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7c8c147b90 0x7f7c8c062f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7c8c0b9d50 0x7f7c8c0aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 --2- 172.21.15.61:0/3168666434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7c8c0b3ca0 0x7f7c8c156280 unknown :-1 s=CLOSED pgs=1231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 >> 172.21.15.61:0/3168666434 conn(0x7f7c8c0678b0 msgr2=0x7f7c8c149f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:30.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 shutdown_connections 2022-07-20T03:16:30.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.704+0000 7f7c91674700 1 -- 172.21.15.61:0/3168666434 wait complete. 2022-07-20T03:16:30.724 INFO:tasks.cephfs_test_runner:test_fs_new (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:16:30.725 INFO:tasks.cephfs_test_runner:Starting test: test_getattr_caps (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:16:30.726 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-20T03:16:30.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 Processor -- start 2022-07-20T03:16:30.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 -- start start 2022-07-20T03:16:30.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e80b1e00 0x7f79e814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e805c3f0 0x7f79e80ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f79e805d480 con 0x7f79e805bd90 2022-07-20T03:16:30.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f79e80b21e0 con 0x7f79e80b1e00 2022-07-20T03:16:30.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79eca0d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e805cf40 con 0x7f79e805c3f0 2022-07-20T03:16:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e80b1e00 0x7f79e814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f79e80b1e00 0x7f79e814a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50090/0 (socket says 172.21.15.61:50090) 2022-07-20T03:16:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 -- 172.21.15.61:0/2302782322 learned_addr learned my addr 172.21.15.61:0/2302782322 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e5d9b700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e805c3f0 0x7f79e80ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 -- 172.21.15.61:0/2302782322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e805c3f0 msgr2=0x7f79e80ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e805c3f0 0x7f79e80ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 -- 172.21.15.61:0/2302782322 >> v1:172.21.15.117:6789/0 conn(0x7f79e805bd90 legacy=0x7f79e8146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 -- 172.21.15.61:0/2302782322 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e805d1e0 con 0x7f79e80b1e00 2022-07-20T03:16:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e5d9b700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e805c3f0 0x7f79e80ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e659c700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b1e00 0x7f79e814a6d0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f79e00032f0 tx=0x7f79e0009290).ready entity=mon.0 client_cookie=fc1de794ae2a04d1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e559a700 1 -- 172.21.15.61:0/2302782322 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79e000b8a0 con 0x7f79e80b1e00 2022-07-20T03:16:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e559a700 1 -- 172.21.15.61:0/2302782322 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e000ba00 con 0x7f79e80b1e00 2022-07-20T03:16:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.832+0000 7f79e559a700 1 -- 172.21.15.61:0/2302782322 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79e000bb70 con 0x7f79e80b1e00 2022-07-20T03:16:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- 172.21.15.61:0/2302782322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b1e00 msgr2=0x7f79e814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b1e00 0x7f79e814a6d0 secure :-1 s=READY pgs=1232 cs=0 l=1 rev1=1 rx=0x7f79e00032f0 tx=0x7f79e0009290).stop 2022-07-20T03:16:30.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- 172.21.15.61:0/2302782322 shutdown_connections 2022-07-20T03:16:30.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e805c3f0 0x7f79e80ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2302782322 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e80b1e00 0x7f79e814a6d0 unknown :-1 s=CLOSED pgs=1232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- 172.21.15.61:0/2302782322 >> 172.21.15.61:0/2302782322 conn(0x7f79e80611c0 msgr2=0x7f79e8062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- 172.21.15.61:0/2302782322 shutdown_connections 2022-07-20T03:16:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- 172.21.15.61:0/2302782322 wait complete. 2022-07-20T03:16:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 Processor -- start 2022-07-20T03:16:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- start start 2022-07-20T03:16:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 0x7f79e809e220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b1e00 0x7f79e8149ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e8149fc0 0x7f79e81461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e80a05a0 con 0x7f79e805c3f0 2022-07-20T03:16:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8063b20 con 0x7f79e8149fc0 2022-07-20T03:16:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79eca0d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f79e815aa00 con 0x7f79e80b1e00 2022-07-20T03:16:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79e659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 0x7f79e809e220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79e659c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 0x7f79e809e220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50094/0 (socket says 172.21.15.61:50094) 2022-07-20T03:16:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79e659c700 1 -- 172.21.15.61:0/2247346868 learned_addr learned my addr 172.21.15.61:0/2247346868 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79e5d9b700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b1e00 0x7f79e8149ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.836+0000 7f79e6d9d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e8149fc0 0x7f79e81461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e659c700 1 -- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b1e00 msgr2=0x7f79e8149ad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e659c700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b1e00 0x7f79e8149ad0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e659c700 1 -- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e8149fc0 msgr2=0x7f79e81461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e659c700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e8149fc0 0x7f79e81461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e659c700 1 -- 172.21.15.61:0/2247346868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79e0009530 con 0x7f79e805c3f0 2022-07-20T03:16:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e659c700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 0x7f79e809e220 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f79e00032f0 tx=0x7f79e0016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79e000b8a0 con 0x7f79e805c3f0 2022-07-20T03:16:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f79e0021630 con 0x7f79e805c3f0 2022-07-20T03:16:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e809e710 con 0x7f79e805c3f0 2022-07-20T03:16:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f79e0021990 con 0x7f79e805c3f0 2022-07-20T03:16:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e815aca0 con 0x7f79e805c3f0 2022-07-20T03:16:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e80640f0 con 0x7f79e805c3f0 2022-07-20T03:16:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f79e0027020 con 0x7f79e805c3f0 2022-07-20T03:16:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79df7fe700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c0053ad0 0x7f79c0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f79e006cea0 con 0x7f79e805c3f0 2022-07-20T03:16:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.840+0000 7f79e5d9b700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c0053ad0 0x7f79c0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.844+0000 7f79e5d9b700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c0053ad0 0x7f79c0055f90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f79d800c000 tx=0x7f79d8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:30.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:30.848+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f79e0044120 con 0x7f79e805c3f0 2022-07-20T03:16:31.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.016+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f79e815a850 con 0x7f79e805c3f0 2022-07-20T03:16:31.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.708+0000 7f79df7fe700 1 -- 172.21.15.61:0/2247346868 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v437) v1 ==== 131+0+0 (secure 0 0 0) 0x7f79e0015020 con 0x7f79e805c3f0 2022-07-20T03:16:31.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c0053ad0 msgr2=0x7f79c0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:31.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c0053ad0 0x7f79c0055f90 secure :-1 s=READY pgs=593 cs=0 l=1 rev1=1 rx=0x7f79d800c000 tx=0x7f79d8008040).stop 2022-07-20T03:16:31.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 msgr2=0x7f79e809e220 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:31.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 0x7f79e809e220 secure :-1 s=READY pgs=1233 cs=0 l=1 rev1=1 rx=0x7f79e00032f0 tx=0x7f79e0016000).stop 2022-07-20T03:16:31.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 shutdown_connections 2022-07-20T03:16:31.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f79c0053ad0 0x7f79c0055f90 unknown :-1 s=CLOSED pgs=593 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f79e8149fc0 0x7f79e81461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f79e80b1e00 0x7f79e8149ad0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 --2- 172.21.15.61:0/2247346868 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f79e805c3f0 0x7f79e809e220 unknown :-1 s=CLOSED pgs=1233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 >> 172.21.15.61:0/2247346868 conn(0x7f79e805bd90 msgr2=0x7f79e814acf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:31.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 shutdown_connections 2022-07-20T03:16:31.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.712+0000 7f79eca0d700 1 -- 172.21.15.61:0/2247346868 wait complete. 2022-07-20T03:16:31.731 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:16:31.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.844+0000 7f9d2e96b700 1 Processor -- start 2022-07-20T03:16:31.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.844+0000 7f9d2e96b700 1 -- start start 2022-07-20T03:16:31.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d2805cf40 0x7f9d280afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:31.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d280b1220 0x7f9d2814a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:31.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d281472b0 con 0x7f9d28147c90 2022-07-20T03:16:31.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d280b0920 con 0x7f9d280b1220 2022-07-20T03:16:31.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28147550 con 0x7f9d2805cf40 2022-07-20T03:16:31.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d277fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9d28147c90 0x7f9d280b0210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46704/0 (socket says 172.21.15.61:46704) 2022-07-20T03:16:31.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d277fe700 1 -- 172.21.15.61:0/3364309787 learned_addr learned my addr 172.21.15.61:0/3364309787 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:31.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2cf08700 1 --2- 172.21.15.61:0/3364309787 >> v2:172.21.15.117:3300/0 conn(0x7f9d280b1220 0x7f9d2814a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:31.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d27fff700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d2805cf40 0x7f9d280afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:31.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d26ffd700 1 -- 172.21.15.61:0/3364309787 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 82058265 0 0) 0x7f9d281472b0 con 0x7f9d28147c90 2022-07-20T03:16:31.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d26ffd700 1 -- 172.21.15.61:0/3364309787 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9d14003260 con 0x7f9d28147c90 2022-07-20T03:16:31.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2cf08700 1 -- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d2805cf40 msgr2=0x7f9d280afd20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:31.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2cf08700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d2805cf40 0x7f9d280afd20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2cf08700 1 -- 172.21.15.61:0/3364309787 >> v1:172.21.15.61:6789/0 conn(0x7f9d28147c90 legacy=0x7f9d280b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:31.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2cf08700 1 -- 172.21.15.61:0/3364309787 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d281477f0 con 0x7f9d280b1220 2022-07-20T03:16:31.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d27fff700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d2805cf40 0x7f9d280afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:31.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2cf08700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d2814a570 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f9d20002f70 tx=0x7f9d20006df0).ready entity=mon.1 client_cookie=800bc5f189b41b32 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:31.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d26ffd700 1 -- 172.21.15.61:0/3364309787 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d2000d4a0 con 0x7f9d280b1220 2022-07-20T03:16:31.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d26ffd700 1 -- 172.21.15.61:0/3364309787 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d20000e90 con 0x7f9d280b1220 2022-07-20T03:16:31.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d26ffd700 1 -- 172.21.15.61:0/3364309787 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d2000d7f0 con 0x7f9d280b1220 2022-07-20T03:16:31.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 msgr2=0x7f9d2814a570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:31.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d2814a570 secure :-1 s=READY pgs=562 cs=0 l=1 rev1=1 rx=0x7f9d20002f70 tx=0x7f9d20006df0).stop 2022-07-20T03:16:31.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- 172.21.15.61:0/3364309787 shutdown_connections 2022-07-20T03:16:31.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d2814a570 unknown :-1 s=CLOSED pgs=562 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/3364309787 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d2805cf40 0x7f9d280afd20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- 172.21.15.61:0/3364309787 >> 172.21.15.61:0/3364309787 conn(0x7f9d280b5b70 msgr2=0x7f9d280b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:31.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- 172.21.15.61:0/3364309787 shutdown_connections 2022-07-20T03:16:31.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.848+0000 7f9d2e96b700 1 -- 172.21.15.61:0/3364309787 wait complete. 2022-07-20T03:16:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 Processor -- start 2022-07-20T03:16:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 -- start start 2022-07-20T03:16:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 0x7f9d28063c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:31.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d28148470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d28147c90 0x7f9d28148960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28144700 con 0x7f9d2805cf40 2022-07-20T03:16:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28157350 con 0x7f9d280b1220 2022-07-20T03:16:31.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d28064150 con 0x7f9d28147c90 2022-07-20T03:16:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 0x7f9d28063c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 0x7f9d28063c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50108/0 (socket says 172.21.15.61:50108) 2022-07-20T03:16:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 -- 172.21.15.61:0/2750499965 learned_addr learned my addr 172.21.15.61:0/2750499965 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:31.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2cf08700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d28147c90 0x7f9d28148960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d277fe700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d28148470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 -- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d28147c90 msgr2=0x7f9d28148960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d28147c90 0x7f9d28148960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 -- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 msgr2=0x7f9d28148470 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:31.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d28148470 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:31.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 -- 172.21.15.61:0/2750499965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d1c00b180 con 0x7f9d2805cf40 2022-07-20T03:16:31.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d27fff700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 0x7f9d28063c60 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f9d1c00c000 tx=0x7f9d1c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:31.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d1c010780 con 0x7f9d2805cf40 2022-07-20T03:16:31.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d1c0108e0 con 0x7f9d2805cf40 2022-07-20T03:16:31.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d2000b180 con 0x7f9d2805cf40 2022-07-20T03:16:31.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.852+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d1c010c40 con 0x7f9d2805cf40 2022-07-20T03:16:31.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.856+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d280678f0 con 0x7f9d2805cf40 2022-07-20T03:16:31.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.856+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d28144900 con 0x7f9d2805cf40 2022-07-20T03:16:31.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.856+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d1c01c460 con 0x7f9d2805cf40 2022-07-20T03:16:31.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.856+0000 7f9d257fa700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d080539b0 0x7f9d08055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:31.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.856+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9d1c06c090 con 0x7f9d2805cf40 2022-07-20T03:16:31.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.856+0000 7f9d277fe700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d080539b0 0x7f9d08055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:31.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.860+0000 7f9d277fe700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d080539b0 0x7f9d08055e70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f9d1800d000 tx=0x7f9d18009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:31.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.868+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d1c043270 con 0x7f9d2805cf40 2022-07-20T03:16:32.003 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:31.996+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f9d28146fa0 con 0x7f9d2805cf40 2022-07-20T03:16:32.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.004+0000 7f9d257fa700 1 -- 172.21.15.61:0/2750499965 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v20)=0 v20) v1 ==== 122+0+0 (secure 0 0 0) 0x7f9d1c046890 con 0x7f9d2805cf40 2022-07-20T03:16:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d080539b0 msgr2=0x7f9d08055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d080539b0 0x7f9d08055e70 secure :-1 s=READY pgs=594 cs=0 l=1 rev1=1 rx=0x7f9d1800d000 tx=0x7f9d18009040).stop 2022-07-20T03:16:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 msgr2=0x7f9d28063c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 0x7f9d28063c60 secure :-1 s=READY pgs=1235 cs=0 l=1 rev1=1 rx=0x7f9d1c00c000 tx=0x7f9d1c006df0).stop 2022-07-20T03:16:32.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 shutdown_connections 2022-07-20T03:16:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d080539b0 0x7f9d08055e70 unknown :-1 s=CLOSED pgs=594 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d280b1220 0x7f9d28148470 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d28147c90 0x7f9d28148960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 --2- 172.21.15.61:0/2750499965 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d2805cf40 0x7f9d28063c60 unknown :-1 s=CLOSED pgs=1235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 >> 172.21.15.61:0/2750499965 conn(0x7f9d280b5b70 msgr2=0x7f9d280afca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 shutdown_connections 2022-07-20T03:16:32.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.008+0000 7f9d2e96b700 1 -- 172.21.15.61:0/2750499965 wait complete. 2022-07-20T03:16:32.025 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:16:32.025 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:32.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:32.069 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:32.091 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:32.114 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:32.136 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:16:32.137 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:32.138 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:32.138 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:32.146 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:32.147 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:16:32.148 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:32.149 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:32.149 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:32.150 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:32.158 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:16:32.159 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:32.160 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:32.160 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:32.161 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:32.161 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:16:32.169 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:32.170 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:16:32.171 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:16:32.172 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:16:32.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 Processor -- start 2022-07-20T03:16:32.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 -- start start 2022-07-20T03:16:32.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2240adff0 0x7fa2240b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240af8e0 0x7fa22405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa224147010 con 0x7fa2240b5650 2022-07-20T03:16:32.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa2241472b0 con 0x7fa2240adff0 2022-07-20T03:16:32.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229956700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa224146d40 con 0x7fa2240af8e0 2022-07-20T03:16:32.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa223fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2240adff0 0x7fa2240b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa223fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa2240adff0 0x7fa2240b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50116/0 (socket says 172.21.15.61:50116) 2022-07-20T03:16:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa223fff700 1 -- 172.21.15.61:0/14413855 learned_addr learned my addr 172.21.15.61:0/14413855 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:32.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.280+0000 7fa229155700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240af8e0 0x7fa22405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa223fff700 1 -- 172.21.15.61:0/14413855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240af8e0 msgr2=0x7fa22405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:32.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa223fff700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240af8e0 0x7fa22405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa223fff700 1 -- 172.21.15.61:0/14413855 >> v1:172.21.15.61:6789/0 conn(0x7fa2240b5650 legacy=0x7fa224147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa2237fe700 1 -- 172.21.15.61:0/14413855 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2770826549 0 0) 0x7fa224147010 con 0x7fa2240b5650 2022-07-20T03:16:32.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa223fff700 1 -- 172.21.15.61:0/14413855 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa2240acae0 con 0x7fa2240adff0 2022-07-20T03:16:32.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229155700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240af8e0 0x7fa22405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa223fff700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240adff0 0x7fa2240b5a30 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7fa20c002f70 tx=0x7fa20c006e50).ready entity=mon.0 client_cookie=5ddf3c12de467d61 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:32.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa2237fe700 1 -- 172.21.15.61:0/14413855 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa20c00d5c0 con 0x7fa2240adff0 2022-07-20T03:16:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa2237fe700 1 -- 172.21.15.61:0/14413855 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa20c000e90 con 0x7fa2240adff0 2022-07-20T03:16:32.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa2237fe700 1 -- 172.21.15.61:0/14413855 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa20c00d910 con 0x7fa2240adff0 2022-07-20T03:16:32.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 -- 172.21.15.61:0/14413855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240adff0 msgr2=0x7fa2240b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240adff0 0x7fa2240b5a30 secure :-1 s=READY pgs=1237 cs=0 l=1 rev1=1 rx=0x7fa20c002f70 tx=0x7fa20c006e50).stop 2022-07-20T03:16:32.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 -- 172.21.15.61:0/14413855 shutdown_connections 2022-07-20T03:16:32.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240af8e0 0x7fa22405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 --2- 172.21.15.61:0/14413855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240adff0 0x7fa2240b5a30 unknown :-1 s=CLOSED pgs=1237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 -- 172.21.15.61:0/14413855 >> 172.21.15.61:0/14413855 conn(0x7fa2240634e0 msgr2=0x7fa2240b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:32.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 -- 172.21.15.61:0/14413855 shutdown_connections 2022-07-20T03:16:32.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 -- 172.21.15.61:0/14413855 wait complete. 2022-07-20T03:16:32.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 Processor -- start 2022-07-20T03:16:32.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.284+0000 7fa229956700 1 -- start start 2022-07-20T03:16:32.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 0x7fa224144ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2240af8e0 0x7fa224066c90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240b5650 0x7fa224156640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224067180 con 0x7fa2240b5650 2022-07-20T03:16:32.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa224111470 con 0x7fa2240af8e0 2022-07-20T03:16:32.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa224111710 con 0x7fa2240adff0 2022-07-20T03:16:32.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 0x7fa224144ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229155700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240b5650 0x7fa224156640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 0x7fa224144ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:36794/0 (socket says 172.21.15.61:36794) 2022-07-20T03:16:32.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 -- 172.21.15.61:0/3856326952 learned_addr learned my addr 172.21.15.61:0/3856326952 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:32.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa223fff700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2240af8e0 0x7fa224066c90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 -- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2240af8e0 msgr2=0x7fa224066c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2240af8e0 0x7fa224066c90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 -- 172.21.15.61:0/3856326952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240b5650 msgr2=0x7fa224156640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240b5650 0x7fa224156640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 -- 172.21.15.61:0/3856326952 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa20c00b1e0 con 0x7fa2240adff0 2022-07-20T03:16:32.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229155700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240b5650 0x7fa224156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:32.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa228954700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 0x7fa224144ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa214000ed0 tx=0x7fa214007c50).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:32.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa214012400 con 0x7fa2240adff0 2022-07-20T03:16:32.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa214012560 con 0x7fa2240adff0 2022-07-20T03:16:32.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa2241119b0 con 0x7fa2240adff0 2022-07-20T03:16:32.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa2140128c0 con 0x7fa2240adff0 2022-07-20T03:16:32.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.288+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa2241561a0 con 0x7fa2240adff0 2022-07-20T03:16:32.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.292+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa224111b40 con 0x7fa2240adff0 2022-07-20T03:16:32.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.292+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa214019030 con 0x7fa2240adff0 2022-07-20T03:16:32.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.292+0000 7fa221ffb700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053a40 0x7fa208055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.292+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa21406af30 con 0x7fa2240adff0 2022-07-20T03:16:32.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.292+0000 7fa223fff700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053a40 0x7fa208055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.292+0000 7fa223fff700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053a40 0x7fa208055f00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fa20c01c000 tx=0x7fa20c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:32.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.300+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa2140420d0 con 0x7fa2240adff0 2022-07-20T03:16:32.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.432+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fa2240574e0 con 0x7fa2240adff0 2022-07-20T03:16:32.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa221ffb700 1 -- 172.21.15.61:0/3856326952 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v125) v1 ==== 81+0+28887 (secure 0 0 0) 0x7fa2140456f0 con 0x7fa2240adff0 2022-07-20T03:16:32.450 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:32.451 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:16:32.455 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 125, 2022-07-20T03:16:32.455 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:16:32.456 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:16:32.457 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:16:29.685661+0000", 2022-07-20T03:16:32.457 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:16:32.458 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:16:32.458 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:16:32.459 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:16:32.459 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:16:32.460 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:16:32.466 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:16:32.467 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:16:32.468 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:16:32.469 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:32.473 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:16:32.474 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:16:32.474 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:16:32.475 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:16:32.475 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:16:32.475 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 24, 2022-07-20T03:16:32.475 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:16:32.476 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:16:32.476 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:16:32.477 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:16:32.477 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:16:32.478 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.478 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:16:32.479 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:16:32.479 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:16:32.480 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:32.480 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:32.481 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:32.481 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:32.481 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:32.482 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:32.482 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:32.483 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:32.483 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:16:32.484 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:16:32.484 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:16:32.484 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:16:32.484 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:16:32.485 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:32.485 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:32.485 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:32.486 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:32.486 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:32.486 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:32.487 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:32.487 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.487 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:16:32.488 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:32.488 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:32.488 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:32.488 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:32.489 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:32.489 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:32.489 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:32.490 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:32.490 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:32.490 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:32.491 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:32.491 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:32.491 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:32.491 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:32.492 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:32.492 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:32.492 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:32.493 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:32.493 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:32.493 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:32.493 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:32.494 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:32.494 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:32.495 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:32.495 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:32.495 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:32.496 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.496 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:32.496 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:32.497 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:32.497 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:32.498 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:32.498 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:32.498 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:32.499 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:32.499 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:32.500 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:32.500 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:32.500 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:16:32.501 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:16:32.501 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.501 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:32.501 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:16:32.502 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.502 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.502 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.503 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:16:32.503 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:16:32.503 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:16:32.504 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:32.504 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:32.505 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:32.505 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:32.505 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:32.505 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:32.506 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:32.506 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:32.506 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:32.507 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:32.507 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:32.507 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:32.507 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:32.508 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:32.508 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:32.509 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:32.509 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:32.509 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:32.511 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:32.511 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:32.511 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.512 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:16:32.512 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:32.512 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:32.513 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:32.513 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:32.513 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:32.514 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:32.514 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:32.515 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:32.515 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:32.515 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:32.516 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:32.516 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:32.517 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:32.517 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:32.518 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:32.518 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:32.518 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:32.519 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:32.519 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:32.520 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:32.520 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:32.520 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:32.521 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:32.521 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:32.522 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:32.522 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:32.523 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.523 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:32.523 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:32.524 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:32.524 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:32.525 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:32.525 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:32.525 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:32.526 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:32.526 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:32.527 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:32.527 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:32.528 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:16:32.528 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:32.528 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:16:32.529 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.529 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.529 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:32.530 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:16:32.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.530 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:16:32.531 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:16:32.531 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.531 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.531 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.532 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.532 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.532 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.533 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.533 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:16:32.534 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.534 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.534 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.535 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.535 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.536 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.536 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:16:32.537 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.537 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.537 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.538 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.538 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:16:32.539 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.539 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.539 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.540 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.540 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.540 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.541 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.541 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.541 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:16:32.542 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.542 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.542 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.543 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.543 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:16:32.544 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.544 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.545 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.545 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.545 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.546 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.546 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.547 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:16:32.547 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.547 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.548 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.548 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:16:32.549 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.549 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.549 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.550 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.550 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.551 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.551 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:16:32.552 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.552 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.552 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.553 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.553 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:16:32.554 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:32.554 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.554 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.555 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.555 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:16:32.556 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:16:32.556 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:16:32.556 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:16:32.557 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.557 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.557 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.558 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.558 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.558 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.559 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:16:32.559 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:16:32.560 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.560 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.561 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.561 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.561 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.562 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.562 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.563 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:16:32.563 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.563 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.564 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.564 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.565 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.565 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.565 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:16:32.566 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.566 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.567 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.567 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.567 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:16:32.568 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.568 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.569 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.569 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.569 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.570 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.570 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.570 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.571 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:16:32.571 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.572 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.572 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.573 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.573 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:16:32.573 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.574 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.574 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.575 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.576 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.576 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.576 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.577 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.577 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:16:32.578 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.578 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.579 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.580 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.580 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:16:32.580 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.581 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.582 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.582 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.582 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.582 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.583 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.583 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.583 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:16:32.584 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.584 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.585 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.585 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.585 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:16:32.586 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:32.586 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.586 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.587 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.587 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:16:32.587 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:16:32.588 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:16:32.588 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:16:32.588 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.589 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.589 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.590 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.590 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.590 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.591 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:16:32.591 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:16:32.592 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.592 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.592 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.593 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.593 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.594 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.594 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.594 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 115, 2022-07-20T03:16:32.595 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.595 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.596 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.596 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.596 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.597 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.598 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:16:32.598 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.598 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.599 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.599 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.600 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:16:32.600 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.600 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.601 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.601 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.601 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.602 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.602 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.603 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.603 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:16:32.603 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.603 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.604 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.604 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.604 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:16:32.605 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.605 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.605 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.605 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.606 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.606 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.606 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.607 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.607 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:16:32.607 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.608 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.608 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.608 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.608 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:16:32.609 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.609 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.609 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.610 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.610 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.611 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.611 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.611 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.612 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:16:32.612 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.613 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.613 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.614 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.614 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:16:32.614 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:32.615 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.615 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.616 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.616 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:16:32.617 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:16:32.617 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:16:32.617 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:16:32.618 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.618 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.619 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.619 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.620 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.620 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.621 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:16:32.621 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:16:32.622 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.622 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.623 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.623 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.624 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.624 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.624 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.625 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 121, 2022-07-20T03:16:32.625 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.626 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.626 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.627 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.627 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.627 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.628 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:16:32.628 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.629 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.629 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.630 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.630 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:16:32.630 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.631 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.631 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.631 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.632 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.632 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.632 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.632 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.633 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:16:32.633 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.633 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.634 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.634 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.634 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:16:32.635 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.635 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.636 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.636 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.637 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.637 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.637 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.638 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.638 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:16:32.639 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.639 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.640 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.640 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.640 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:16:32.641 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.641 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.642 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.642 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.643 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.643 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.643 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.644 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.644 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:16:32.645 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.645 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.646 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.646 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.646 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:16:32.647 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:32.647 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.648 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.648 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.649 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:16:32.649 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:16:32.649 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:16:32.650 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:16:32.650 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.651 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.651 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.652 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.652 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.653 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.653 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:16:32.653 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:16:32.654 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.654 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.655 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.655 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.656 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.656 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.657 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.657 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 115, 2022-07-20T03:16:32.658 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.658 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.658 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.659 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.659 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.660 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.660 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:16:32.661 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.661 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.661 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.662 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.662 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:16:32.663 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.663 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.664 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.664 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.664 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.665 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.665 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.666 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.666 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:16:32.667 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.667 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.667 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.668 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.669 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:16:32.669 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.669 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.669 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.670 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.670 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.670 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.671 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.671 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.672 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:16:32.672 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.672 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.673 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.673 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.674 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:16:32.674 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.675 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.675 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.675 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.676 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.676 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.677 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.677 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.678 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:16:32.678 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.679 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.679 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.680 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.680 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:16:32.681 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:32.681 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.682 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.682 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.682 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:16:32.683 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:16:32.683 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:16:32.684 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:16:32.684 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.685 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.685 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.686 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.686 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.687 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.687 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:16:32.688 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:16:32.688 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.689 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.689 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.689 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.691 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.692 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.692 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.693 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:16:32.693 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.694 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.694 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.694 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.695 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.696 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.696 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:16:32.697 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.697 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.697 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.698 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.698 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:16:32.700 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.701 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.701 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.702 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.702 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.703 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:16:32.705 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.705 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.707 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:16:32.707 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.710 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.710 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.711 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:16:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:16:32.715 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:16:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.722 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:16:32.722 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:16:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:16:32.725 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:16:32.725 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:16:32.725 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:16:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:16:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:16:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 121, 2022-07-20T03:16:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:16:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:16:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:16:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:16:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:16:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:16:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:16:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:16:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:16:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:16:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:16:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:16:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:16:32.750 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.750 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.751 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.751 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.752 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.752 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.752 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:16:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:16:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:32.753 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:32.754 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:32.754 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:32.754 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:32.754 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 121, 2022-07-20T03:16:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:32.755 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.756 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.756 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.756 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:16:32.756 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.757 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:16:32.758 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.758 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.758 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.758 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.759 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:32.759 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.759 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.759 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:16:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.760 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:16:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.761 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.762 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.762 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:32.762 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.763 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.764 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.764 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:16:32.764 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.764 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.765 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.765 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.765 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:16:32.765 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.766 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:32.767 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.768 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:32.768 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:16:32.768 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.768 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.769 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.769 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:32.769 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:16:32.769 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:16:32.769 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.770 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.770 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.770 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:16:32.770 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:16:32.771 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:16:32.771 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:16:32.771 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:32.771 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:32.772 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:32.772 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:32.772 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.772 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:32.772 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:16:32.773 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.773 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:16:32.773 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.773 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.774 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.774 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.774 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.774 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:16:32.775 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.775 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.775 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.775 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:16:32.776 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.776 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.776 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.776 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.777 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.777 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:16:32.777 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.777 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:16:32.778 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.779 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:16:32.780 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.781 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.781 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.781 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:16:32.781 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.782 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.782 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.782 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.783 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.783 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:16:32.783 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.783 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.784 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.784 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:16:32.784 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.784 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.784 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.785 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.785 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.785 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:16:32.786 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.786 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.786 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.786 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:16:32.787 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.788 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.788 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.788 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.789 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.789 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:16:32.790 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.790 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.790 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.791 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:16:32.791 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.792 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.792 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.792 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.793 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.793 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:16:32.794 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.794 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.795 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:32.795 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:16:32.796 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:16:32.796 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:16:32.797 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:16:32.798 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:16:32.799 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:16:32.799 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:16:32.799 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:16:32.800 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.800 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:32.801 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:16:32.801 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:16:32.802 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:16:32.802 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:16:32.803 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:16:32.803 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6835/2884785802": "2022-07-21T03:16:09.208729+0000", 2022-07-20T03:16:32.803 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.117:6834/2884785802": "2022-07-21T03:16:09.208729+0000" 2022-07-20T03:16:32.803 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.804 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:16:32.804 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:16:32.804 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:16:32.804 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:16:32.805 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:16:32.805 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:16:32.805 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:16:32.806 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:16:32.806 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:32.806 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:32.806 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:16:32.807 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:16:32.807 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:16:32.807 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:16:32.807 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:16:32.808 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:16:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053a40 msgr2=0x7fa208055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053a40 0x7fa208055f00 secure :-1 s=READY pgs=595 cs=0 l=1 rev1=1 rx=0x7fa20c01c000 tx=0x7fa20c015000).stop 2022-07-20T03:16:32.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 msgr2=0x7fa224144ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 0x7fa224144ba0 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7fa214000ed0 tx=0x7fa214007c50).stop 2022-07-20T03:16:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 shutdown_connections 2022-07-20T03:16:32.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa208053a40 0x7fa208055f00 unknown :-1 s=CLOSED pgs=595 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa2240af8e0 0x7fa224066c90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa2240b5650 0x7fa224156640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 --2- 172.21.15.61:0/3856326952 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa2240adff0 0x7fa224144ba0 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 >> 172.21.15.61:0/3856326952 conn(0x7fa2240634e0 msgr2=0x7fa224145450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:32.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 shutdown_connections 2022-07-20T03:16:32.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.436+0000 7fa229956700 1 -- 172.21.15.61:0/3856326952 wait complete. 2022-07-20T03:16:32.814 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabd530b700 1 Processor -- start 2022-07-20T03:16:32.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabd530b700 1 -- start start 2022-07-20T03:16:32.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabd530b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fabd00acef0 0x7fabd014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabd530b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd005d480 con 0x7fabd005c3f0 2022-07-20T03:16:32.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabd530b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fabd00b21e0 con 0x7fabd00b1e00 2022-07-20T03:16:32.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabd530b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fabd005cf40 con 0x7fabd00acef0 2022-07-20T03:16:32.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fabd005c3f0 0x7fabd0146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46730/0 (socket says 172.21.15.61:46730) 2022-07-20T03:16:32.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 -- 172.21.15.61:0/1066279714 learned_addr learned my addr 172.21.15.61:0/1066279714 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:32.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcf7fe700 1 --2- 172.21.15.61:0/1066279714 >> v2:172.21.15.117:3300/0 conn(0x7fabd00acef0 0x7fabd014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 361523081 0 0) 0x7fabd005d480 con 0x7fabd005c3f0 2022-07-20T03:16:32.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabb4003260 con 0x7fabd005c3f0 2022-07-20T03:16:32.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2359254939 0 0) 0x7fabd00b21e0 con 0x7fabd00b1e00 2022-07-20T03:16:32.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fabd005d480 con 0x7fabd00b1e00 2022-07-20T03:16:32.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3082666659 0 0) 0x7fabb4003260 con 0x7fabd005c3f0 2022-07-20T03:16:32.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 >> v2:172.21.15.117:3300/0 conn(0x7fabd00acef0 msgr2=0x7fabd014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:32.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 --2- 172.21.15.61:0/1066279714 >> v2:172.21.15.117:3300/0 conn(0x7fabd00acef0 0x7fabd014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 >> v1:172.21.15.117:6789/0 conn(0x7fabd00b1e00 legacy=0x7fabd00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd005d1e0 con 0x7fabd005c3f0 2022-07-20T03:16:32.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fabc80019e0 con 0x7fabd005c3f0 2022-07-20T03:16:32.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 >> v1:172.21.15.61:6789/0 conn(0x7fabd005c3f0 legacy=0x7fabd0146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabb40038e0 0x7fabb4003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4004500 0x7fabb4004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb4004ec0 0x7fabb400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd005cf40 con 0x7fabb4004ec0 2022-07-20T03:16:32.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabb400be30 con 0x7fabb4004500 2022-07-20T03:16:32.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabb400bff0 con 0x7fabb40038e0 2022-07-20T03:16:32.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4004500 0x7fabb4004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcf7fe700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb4004ec0 0x7fabb400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabceffd700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabb40038e0 0x7fabb4003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 -- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabb40038e0 msgr2=0x7fabb4003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:32.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabb40038e0 0x7fabb4003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 -- 172.21.15.61:0/1066279714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb4004ec0 msgr2=0x7fabb400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:32.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb4004ec0 0x7fabb400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabce7fc700 1 -- 172.21.15.61:0/1066279714 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabb4003260 con 0x7fabb4004500 2022-07-20T03:16:32.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcf7fe700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabb4004ec0 0x7fabb400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:32.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.924+0000 7fabcf7fe700 1 -- 172.21.15.61:0/1066279714 reap_dead start 2022-07-20T03:16:32.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabce7fc700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4004500 0x7fabb4004980 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7fabc80017b0 tx=0x7fabd00ac4d0).ready entity=mon.1 client_cookie=40bedf12e3543bdb server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:32.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc800cce0 con 0x7fabb4004500 2022-07-20T03:16:32.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc800ce40 con 0x7fabb4004500 2022-07-20T03:16:32.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabcdffb700 1 -- 172.21.15.61:0/1066279714 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc8005b80 con 0x7fabb4004500 2022-07-20T03:16:32.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 -- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4004500 msgr2=0x7fabb4004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4004500 0x7fabb4004980 secure :-1 s=READY pgs=564 cs=0 l=1 rev1=1 rx=0x7fabc80017b0 tx=0x7fabd00ac4d0).stop 2022-07-20T03:16:32.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 -- 172.21.15.61:0/1066279714 shutdown_connections 2022-07-20T03:16:32.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 --2- 172.21.15.61:0/1066279714 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabb4004500 0x7fabb4004980 unknown :-1 s=CLOSED pgs=564 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 -- 172.21.15.61:0/1066279714 >> 172.21.15.61:0/1066279714 conn(0x7fabd00611c0 msgr2=0x7fabd0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:32.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 -- 172.21.15.61:0/1066279714 shutdown_connections 2022-07-20T03:16:32.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 -- 172.21.15.61:0/1066279714 wait complete. 2022-07-20T03:16:32.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 Processor -- start 2022-07-20T03:16:32.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.928+0000 7fabd530b700 1 -- start start 2022-07-20T03:16:32.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabd00b4a80 0x7fabd00b2ec0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 0x7fabd0066340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabd0142ff0 0x7fabd0066830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0066eb0 con 0x7fabd00b5c30 2022-07-20T03:16:32.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0065ea0 con 0x7fabd00b4a80 2022-07-20T03:16:32.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fabd0143550 con 0x7fabd0142ff0 2022-07-20T03:16:32.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 0x7fabd0066340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 0x7fabd0066340 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50144/0 (socket says 172.21.15.61:50144) 2022-07-20T03:16:32.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 -- 172.21.15.61:0/2247448986 learned_addr learned my addr 172.21.15.61:0/2247448986 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:32.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabceffd700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabd00b4a80 0x7fabd00b2ec0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabcf7fe700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabd0142ff0 0x7fabd0066830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 -- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabd0142ff0 msgr2=0x7fabd0066830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabd0142ff0 0x7fabd0066830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 -- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabd00b4a80 msgr2=0x7fabd00b2ec0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:32.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabd00b4a80 0x7fabd00b2ec0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:32.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 -- 172.21.15.61:0/2247448986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fabd00b33b0 con 0x7fabd00b5c30 2022-07-20T03:16:32.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabce7fc700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 0x7fabd0066340 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7fabc80017b0 tx=0x7fabc8013000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:32.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc8001210 con 0x7fabd00b5c30 2022-07-20T03:16:32.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fabc8004ab0 con 0x7fabd00b5c30 2022-07-20T03:16:32.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fabd00b4e60 con 0x7fabd00b5c30 2022-07-20T03:16:32.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fabc8004e10 con 0x7fabd00b5c30 2022-07-20T03:16:32.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fabd0063c20 con 0x7fabd00b5c30 2022-07-20T03:16:32.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.932+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fabd0066d20 con 0x7fabd00b5c30 2022-07-20T03:16:32.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.936+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fabc801c070 con 0x7fabd00b5c30 2022-07-20T03:16:32.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.936+0000 7fabc7fff700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faba8053af0 0x7faba8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:32.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.936+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fabc806be30 con 0x7fabd00b5c30 2022-07-20T03:16:32.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.936+0000 7fabceffd700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faba8053af0 0x7faba8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:32.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.936+0000 7fabceffd700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faba8053af0 0x7faba8055fb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fabb8003000 tx=0x7fabb8007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:32.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:32.940+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fabc8042f60 con 0x7fabd00b5c30 2022-07-20T03:16:33.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.104+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fabd0065cf0 con 0x7fabd00b5c30 2022-07-20T03:16:33.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.104+0000 7fabc7fff700 1 -- 172.21.15.61:0/2247448986 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+504 (secure 0 0 0) 0x7fabc8046580 con 0x7fabd00b5c30 2022-07-20T03:16:33.109 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:33.110 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-20T03:16:33.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faba8053af0 msgr2=0x7faba8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faba8053af0 0x7faba8055fb0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fabb8003000 tx=0x7fabb8007a00).stop 2022-07-20T03:16:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 msgr2=0x7fabd0066340 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 0x7fabd0066340 secure :-1 s=READY pgs=1239 cs=0 l=1 rev1=1 rx=0x7fabc80017b0 tx=0x7fabc8013000).stop 2022-07-20T03:16:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 shutdown_connections 2022-07-20T03:16:33.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faba8053af0 0x7faba8055fb0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fabd0142ff0 0x7fabd0066830 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fabd00b5c30 0x7fabd0066340 unknown :-1 s=CLOSED pgs=1239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 --2- 172.21.15.61:0/2247448986 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fabd00b4a80 0x7fabd00b2ec0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 >> 172.21.15.61:0/2247448986 conn(0x7fabb4004500 msgr2=0x7fabd00afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:33.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 shutdown_connections 2022-07-20T03:16:33.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.108+0000 7fabd530b700 1 -- 172.21.15.61:0/2247448986 wait complete. 2022-07-20T03:16:33.114 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 69 2022-07-20T03:16:33.122 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:33.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.220+0000 7fc45359e700 1 Processor -- start 2022-07-20T03:16:33.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.220+0000 7fc45359e700 1 -- start start 2022-07-20T03:16:33.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.220+0000 7fc45359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413c790 0x7fc4540b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.220+0000 7fc45359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc45413cb70 0x7fc454143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc454142ba0 con 0x7fc4540b3ca0 2022-07-20T03:16:33.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc4541429b0 con 0x7fc45413cb70 2022-07-20T03:16:33.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc454146330 con 0x7fc45413c790 2022-07-20T03:16:33.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc451d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fc4540b3ca0 0x7fc4540b9a10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46748/0 (socket says 172.21.15.61:46748) 2022-07-20T03:16:33.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc451d9b700 1 -- 172.21.15.61:0/1432301580 learned_addr learned my addr 172.21.15.61:0/1432301580 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:33.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45259c700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413c790 0x7fc4540b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc452d9d700 1 --2- 172.21.15.61:0/1432301580 >> v2:172.21.15.117:3300/0 conn(0x7fc45413cb70 0x7fc454143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45159a700 1 -- 172.21.15.61:0/1432301580 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 5576696 0 0) 0x7fc454142ba0 con 0x7fc4540b3ca0 2022-07-20T03:16:33.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45159a700 1 -- 172.21.15.61:0/1432301580 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc438003260 con 0x7fc4540b3ca0 2022-07-20T03:16:33.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc452d9d700 1 -- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413c790 msgr2=0x7fc4540b4710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:33.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc452d9d700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413c790 0x7fc4540b4710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc452d9d700 1 -- 172.21.15.61:0/1432301580 >> v1:172.21.15.61:6789/0 conn(0x7fc4540b3ca0 legacy=0x7fc4540b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc452d9d700 1 -- 172.21.15.61:0/1432301580 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc4540b4c00 con 0x7fc45413cb70 2022-07-20T03:16:33.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45259c700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413c790 0x7fc4540b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:33.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc452d9d700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc45413cb70 0x7fc454143000 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fc44c0032f0 tx=0x7fc44c0091d0).ready entity=mon.1 client_cookie=522755235ff121a1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:33.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45159a700 1 -- 172.21.15.61:0/1432301580 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44c00b600 con 0x7fc45413cb70 2022-07-20T03:16:33.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45159a700 1 -- 172.21.15.61:0/1432301580 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc44c00b760 con 0x7fc45413cb70 2022-07-20T03:16:33.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45159a700 1 -- 172.21.15.61:0/1432301580 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44c00b8d0 con 0x7fc45413cb70 2022-07-20T03:16:33.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc45413cb70 msgr2=0x7fc454143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc45413cb70 0x7fc454143000 secure :-1 s=READY pgs=565 cs=0 l=1 rev1=1 rx=0x7fc44c0032f0 tx=0x7fc44c0091d0).stop 2022-07-20T03:16:33.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- 172.21.15.61:0/1432301580 shutdown_connections 2022-07-20T03:16:33.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc45413cb70 0x7fc454143000 unknown :-1 s=CLOSED pgs=565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 --2- 172.21.15.61:0/1432301580 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413c790 0x7fc4540b4710 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- 172.21.15.61:0/1432301580 >> 172.21.15.61:0/1432301580 conn(0x7fc4540678b0 msgr2=0x7fc4540b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:33.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- 172.21.15.61:0/1432301580 shutdown_connections 2022-07-20T03:16:33.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.224+0000 7fc45359e700 1 -- 172.21.15.61:0/1432301580 wait complete. 2022-07-20T03:16:33.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 Processor -- start 2022-07-20T03:16:33.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 -- start start 2022-07-20T03:16:33.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4540b3ca0 0x7fc454145340 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 0x7fc454155850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413cb70 0x7fc4540af410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4540aefe0 con 0x7fc45413c790 2022-07-20T03:16:33.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc45413ff60 con 0x7fc4540b3ca0 2022-07-20T03:16:33.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc454140200 con 0x7fc45413cb70 2022-07-20T03:16:33.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc451d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 0x7fc454155850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc451d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 0x7fc454155850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50156/0 (socket says 172.21.15.61:50156) 2022-07-20T03:16:33.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc451d9b700 1 -- 172.21.15.61:0/3414526312 learned_addr learned my addr 172.21.15.61:0/3414526312 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:33.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc45259c700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4540b3ca0 0x7fc454145340 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.228+0000 7fc452d9d700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413cb70 0x7fc4540af410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc451d9b700 1 -- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413cb70 msgr2=0x7fc4540af410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc451d9b700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413cb70 0x7fc4540af410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc451d9b700 1 -- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4540b3ca0 msgr2=0x7fc454145340 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc451d9b700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4540b3ca0 0x7fc454145340 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc451d9b700 1 -- 172.21.15.61:0/3414526312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc44800b180 con 0x7fc45413c790 2022-07-20T03:16:33.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc451d9b700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 0x7fc454155850 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fc444001110 tx=0x7fc444007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:33.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc44400cdb0 con 0x7fc45413c790 2022-07-20T03:16:33.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4440070d0 con 0x7fc45413c790 2022-07-20T03:16:33.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc44c009470 con 0x7fc45413c790 2022-07-20T03:16:33.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc444007430 con 0x7fc45413c790 2022-07-20T03:16:33.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc45413dae0 con 0x7fc45413c790 2022-07-20T03:16:33.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc4540574e0 con 0x7fc45413c790 2022-07-20T03:16:33.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.232+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc444010070 con 0x7fc45413c790 2022-07-20T03:16:33.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.236+0000 7fc4437fe700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc430053a40 0x7fc430055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.236+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fc44406b0d0 con 0x7fc45413c790 2022-07-20T03:16:33.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.236+0000 7fc45259c700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc430053a40 0x7fc430055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.236+0000 7fc45259c700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc430053a40 0x7fc430055f00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fc44800b000 tx=0x7fc448006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:33.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.240+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc444042290 con 0x7fc45413c790 2022-07-20T03:16:33.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.400+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc454142790 con 0x7fc45413c790 2022-07-20T03:16:33.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.400+0000 7fc4437fe700 1 -- 172.21.15.61:0/3414526312 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 69 v69) v1 ==== 94+0+504 (secure 0 0 0) 0x7fc4440458b0 con 0x7fc45413c790 2022-07-20T03:16:33.407 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:33.407 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":69,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[],"filesystems":[]} 2022-07-20T03:16:33.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc430053a40 msgr2=0x7fc430055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc430053a40 0x7fc430055f00 secure :-1 s=READY pgs=597 cs=0 l=1 rev1=1 rx=0x7fc44800b000 tx=0x7fc448006df0).stop 2022-07-20T03:16:33.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 msgr2=0x7fc454155850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 0x7fc454155850 secure :-1 s=READY pgs=1241 cs=0 l=1 rev1=1 rx=0x7fc444001110 tx=0x7fc444007a60).stop 2022-07-20T03:16:33.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 shutdown_connections 2022-07-20T03:16:33.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc430053a40 0x7fc430055f00 unknown :-1 s=CLOSED pgs=597 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc45413cb70 0x7fc4540af410 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc45413c790 0x7fc454155850 unknown :-1 s=CLOSED pgs=1241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 --2- 172.21.15.61:0/3414526312 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc4540b3ca0 0x7fc454145340 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 >> 172.21.15.61:0/3414526312 conn(0x7fc4540678b0 msgr2=0x7fc454142d60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:33.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 shutdown_connections 2022-07-20T03:16:33.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.404+0000 7fc45359e700 1 -- 172.21.15.61:0/3414526312 wait complete. 2022-07-20T03:16:33.412 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 69 2022-07-20T03:16:33.420 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:16:33.421 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:16:33.422 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:16:33.423 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:16:33.426 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:16:33.427 DEBUG:teuthology.parallel:result is None 2022-07-20T03:16:33.429 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:16:33.429 DEBUG:teuthology.parallel:result is None 2022-07-20T03:16:33.430 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:16:33.497 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:16:33.501 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:16:33.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 Processor -- start 2022-07-20T03:16:33.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 -- start start 2022-07-20T03:16:33.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f763c0b6840 0x7f763c0b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f763c0b3cf0 0x7f763c146040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f763c066b10 con 0x7f763c146530 2022-07-20T03:16:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763c0b6c20 con 0x7f763c0b3cf0 2022-07-20T03:16:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f7640d96700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f763c0aead0 con 0x7f763c0b6840 2022-07-20T03:16:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f763affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f763c0b3cf0 0x7f763c146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f763affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f763c0b3cf0 0x7f763c146040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50176/0 (socket says 172.21.15.61:50176) 2022-07-20T03:16:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f763affd700 1 -- 172.21.15.61:0/3572871870 learned_addr learned my addr 172.21.15.61:0/3572871870 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.528+0000 7f763b7fe700 1 --2- 172.21.15.61:0/3572871870 >> v2:172.21.15.117:3300/0 conn(0x7f763c0b6840 0x7f763c0b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763affd700 1 -- 172.21.15.61:0/3572871870 >> v2:172.21.15.117:3300/0 conn(0x7f763c0b6840 msgr2=0x7f763c0b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:33.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763affd700 1 --2- 172.21.15.61:0/3572871870 >> v2:172.21.15.117:3300/0 conn(0x7f763c0b6840 0x7f763c0b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763affd700 1 -- 172.21.15.61:0/3572871870 >> v1:172.21.15.117:6789/0 conn(0x7f763c146530 legacy=0x7f763c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763affd700 1 -- 172.21.15.61:0/3572871870 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f763c0aed70 con 0x7f763c0b3cf0 2022-07-20T03:16:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763affd700 1 --2- 172.21.15.61:0/3572871870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b3cf0 0x7f763c146040 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f7624002f70 tx=0x7f7624006eb0).ready entity=mon.0 client_cookie=e756f0635c27e50c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:33.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763a7fc700 1 -- 172.21.15.61:0/3572871870 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f762400d5c0 con 0x7f763c0b3cf0 2022-07-20T03:16:33.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763a7fc700 1 -- 172.21.15.61:0/3572871870 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7624000e90 con 0x7f763c0b3cf0 2022-07-20T03:16:33.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f763a7fc700 1 -- 172.21.15.61:0/3572871870 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f762400d950 con 0x7f763c0b3cf0 2022-07-20T03:16:33.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 -- 172.21.15.61:0/3572871870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b3cf0 msgr2=0x7f763c146040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 --2- 172.21.15.61:0/3572871870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b3cf0 0x7f763c146040 secure :-1 s=READY pgs=1245 cs=0 l=1 rev1=1 rx=0x7f7624002f70 tx=0x7f7624006eb0).stop 2022-07-20T03:16:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 -- 172.21.15.61:0/3572871870 shutdown_connections 2022-07-20T03:16:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 --2- 172.21.15.61:0/3572871870 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b3cf0 0x7f763c146040 unknown :-1 s=CLOSED pgs=1245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 --2- 172.21.15.61:0/3572871870 >> v2:172.21.15.117:3300/0 conn(0x7f763c0b6840 0x7f763c0b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 -- 172.21.15.61:0/3572871870 >> 172.21.15.61:0/3572871870 conn(0x7f763c0b9e10 msgr2=0x7f763c0b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 -- 172.21.15.61:0/3572871870 shutdown_connections 2022-07-20T03:16:33.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.532+0000 7f7640d96700 1 -- 172.21.15.61:0/3572871870 wait complete. 2022-07-20T03:16:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 Processor -- start 2022-07-20T03:16:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 -- start start 2022-07-20T03:16:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f763c0b3cf0 0x7f763c0af850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 0x7f763c1153a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f763c146530 0x7f763c115890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c0b2b20 con 0x7f763c0b6840 2022-07-20T03:16:33.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f763c114310 con 0x7f763c146530 2022-07-20T03:16:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f763c113c50 con 0x7f763c0b3cf0 2022-07-20T03:16:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 0x7f763c1153a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 0x7f763c1153a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50182/0 (socket says 172.21.15.61:50182) 2022-07-20T03:16:33.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 -- 172.21.15.61:0/2033906237 learned_addr learned my addr 172.21.15.61:0/2033906237 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:33.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763bfff700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f763c146530 0x7f763c115890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763b7fe700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f763c0b3cf0 0x7f763c0af850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 -- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f763c0b3cf0 msgr2=0x7f763c0af850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f763c0b3cf0 0x7f763c0af850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 -- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f763c146530 msgr2=0x7f763c115890 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:33.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f763c146530 0x7f763c115890 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:33.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 -- 172.21.15.61:0/2033906237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f762400b1e0 con 0x7f763c0b6840 2022-07-20T03:16:33.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f763affd700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 0x7f763c1153a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f7624002f70 tx=0x7f7624015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:33.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7624011760 con 0x7f763c0b6840 2022-07-20T03:16:33.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f76240118c0 con 0x7f763c0b6840 2022-07-20T03:16:33.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7624011c20 con 0x7f763c0b6840 2022-07-20T03:16:33.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f763c112ce0 con 0x7f763c0b6840 2022-07-20T03:16:33.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.536+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f763c112a50 con 0x7f763c0b6840 2022-07-20T03:16:33.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.540+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f763c1150e0 con 0x7f763c0b6840 2022-07-20T03:16:33.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.540+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f762401a040 con 0x7f763c0b6840 2022-07-20T03:16:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.540+0000 7f7638ff9700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76200539b0 0x7f7620055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.540+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(125..125 src has 1..125) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f762406c9d0 con 0x7f763c0b6840 2022-07-20T03:16:33.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.540+0000 7f763b7fe700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76200539b0 0x7f7620055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:33.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.540+0000 7f763b7fe700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76200539b0 0x7f7620055e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f763400b000 tx=0x7f76340091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:33.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.548+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f763c1150e0 con 0x7f763c0b6840 2022-07-20T03:16:33.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:33.684+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f763c1150e0 con 0x7f763c0b6840 2022-07-20T03:16:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.516+0000 7f7638ff9700 1 -- 172.21.15.61:0/2033906237 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v126) v1 ==== 97+0+0 (secure 0 0 0) 0x7f763c1150e0 con 0x7f763c0b6840 2022-07-20T03:16:34.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76200539b0 msgr2=0x7f7620055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76200539b0 0x7f7620055e70 secure :-1 s=READY pgs=598 cs=0 l=1 rev1=1 rx=0x7f763400b000 tx=0x7f76340091d0).stop 2022-07-20T03:16:34.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 msgr2=0x7f763c1153a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 0x7f763c1153a0 secure :-1 s=READY pgs=1246 cs=0 l=1 rev1=1 rx=0x7f7624002f70 tx=0x7f7624015000).stop 2022-07-20T03:16:34.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 shutdown_connections 2022-07-20T03:16:34.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f76200539b0 0x7f7620055e70 unknown :-1 s=CLOSED pgs=598 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f763c146530 0x7f763c115890 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f763c0b6840 0x7f763c1153a0 unknown :-1 s=CLOSED pgs=1246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 --2- 172.21.15.61:0/2033906237 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f763c0b3cf0 0x7f763c0af850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 >> 172.21.15.61:0/2033906237 conn(0x7f763c0b9e10 msgr2=0x7f763c0adb80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:34.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 shutdown_connections 2022-07-20T03:16:34.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.520+0000 7f7640d96700 1 -- 172.21.15.61:0/2033906237 wait complete. 2022-07-20T03:16:34.531 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:16:34.548 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:16:34.552 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:16:34.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 Processor -- start 2022-07-20T03:16:34.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 -- start start 2022-07-20T03:16:34.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0b5650 0x7efcdc0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:34.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efcdc0af8e0 0x7efcdc05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:34.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efcdc147010 con 0x7efcdc0adff0 2022-07-20T03:16:34.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efcdc1472b0 con 0x7efcdc0af8e0 2022-07-20T03:16:34.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce3475700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc146d40 con 0x7efcdc0b5650 2022-07-20T03:16:34.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce2c74700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efcdc0af8e0 0x7efcdc05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:34.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce1c72700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7efcdc0adff0 0x7efcdc147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46788/0 (socket says 172.21.15.61:46788) 2022-07-20T03:16:34.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce1c72700 1 -- 172.21.15.61:0/766741903 learned_addr learned my addr 172.21.15.61:0/766741903 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:34.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce2473700 1 --2- 172.21.15.61:0/766741903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0b5650 0x7efcdc0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:34.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce1471700 1 -- 172.21.15.61:0/766741903 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2454484723 0 0) 0x7efcdc147010 con 0x7efcdc0adff0 2022-07-20T03:16:34.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce1471700 1 -- 172.21.15.61:0/766741903 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7efcc4003260 con 0x7efcdc0adff0 2022-07-20T03:16:34.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce2c74700 1 -- 172.21.15.61:0/766741903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0b5650 msgr2=0x7efcdc0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:34.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce2c74700 1 --2- 172.21.15.61:0/766741903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0b5650 0x7efcdc0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce2c74700 1 -- 172.21.15.61:0/766741903 >> v1:172.21.15.61:6789/0 conn(0x7efcdc0adff0 legacy=0x7efcdc147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.672+0000 7efce2c74700 1 -- 172.21.15.61:0/766741903 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcdc0acae0 con 0x7efcdc0af8e0 2022-07-20T03:16:34.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce2c74700 1 --2- 172.21.15.61:0/766741903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0af8e0 0x7efcdc05bd90 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7efcd40032f0 tx=0x7efcd40091d0).ready entity=mon.0 client_cookie=67d2667fad469218 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:34.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce1471700 1 -- 172.21.15.61:0/766741903 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd400b600 con 0x7efcdc0af8e0 2022-07-20T03:16:34.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce1471700 1 -- 172.21.15.61:0/766741903 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd400b760 con 0x7efcdc0af8e0 2022-07-20T03:16:34.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce1471700 1 -- 172.21.15.61:0/766741903 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd400b8d0 con 0x7efcdc0af8e0 2022-07-20T03:16:34.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- 172.21.15.61:0/766741903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0af8e0 msgr2=0x7efcdc05bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 --2- 172.21.15.61:0/766741903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0af8e0 0x7efcdc05bd90 secure :-1 s=READY pgs=1248 cs=0 l=1 rev1=1 rx=0x7efcd40032f0 tx=0x7efcd40091d0).stop 2022-07-20T03:16:34.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- 172.21.15.61:0/766741903 shutdown_connections 2022-07-20T03:16:34.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 --2- 172.21.15.61:0/766741903 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0af8e0 0x7efcdc05bd90 unknown :-1 s=CLOSED pgs=1248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 --2- 172.21.15.61:0/766741903 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0b5650 0x7efcdc0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- 172.21.15.61:0/766741903 >> 172.21.15.61:0/766741903 conn(0x7efcdc0634e0 msgr2=0x7efcdc0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- 172.21.15.61:0/766741903 shutdown_connections 2022-07-20T03:16:34.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- 172.21.15.61:0/766741903 wait complete. 2022-07-20T03:16:34.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 Processor -- start 2022-07-20T03:16:34.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- start start 2022-07-20T03:16:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcdc0adff0 0x7efcdc1111d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:34.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0af8e0 0x7efcdc144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0b5650 0x7efcdc0648b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:34.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc110930 con 0x7efcdc0b5650 2022-07-20T03:16:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc110bd0 con 0x7efcdc0adff0 2022-07-20T03:16:34.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.676+0000 7efce3475700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efcdc110e70 con 0x7efcdc0af8e0 2022-07-20T03:16:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2c74700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0b5650 0x7efcdc0648b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcdc0adff0 0x7efcdc1111d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:34.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2c74700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0b5650 0x7efcdc0648b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50200/0 (socket says 172.21.15.61:50200) 2022-07-20T03:16:34.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce1c72700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0af8e0 0x7efcdc144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:34.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2c74700 1 -- 172.21.15.61:0/585953798 learned_addr learned my addr 172.21.15.61:0/585953798 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:34.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 -- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0af8e0 msgr2=0x7efcdc144ff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0af8e0 0x7efcdc144ff0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 -- 172.21.15.61:0/585953798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0b5650 msgr2=0x7efcdc0648b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0b5650 0x7efcdc0648b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 -- 172.21.15.61:0/585953798 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcd4009470 con 0x7efcdc0adff0 2022-07-20T03:16:34.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce2473700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcdc0adff0 0x7efcdc1111d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7efcd000c000 tx=0x7efcd0006df0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:34.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd0011750 con 0x7efcdc0adff0 2022-07-20T03:16:34.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcdc1105b0 con 0x7efcdc0adff0 2022-07-20T03:16:34.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efcd00118b0 con 0x7efcdc0adff0 2022-07-20T03:16:34.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efcd0011c10 con 0x7efcdc0adff0 2022-07-20T03:16:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcdc10f3f0 con 0x7efcdc0adff0 2022-07-20T03:16:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcdc159aa0 con 0x7efcdc0adff0 2022-07-20T03:16:34.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.680+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efcd0011d70 con 0x7efcdc0adff0 2022-07-20T03:16:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.684+0000 7efccf7fe700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcbc0539b0 0x7efcbc055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:34.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.684+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5253+0+0 (secure 0 0 0) 0x7efcd006bfd0 con 0x7efcdc0adff0 2022-07-20T03:16:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.684+0000 7efce1c72700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcbc0539b0 0x7efcbc055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:34.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.684+0000 7efce1c72700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcbc0539b0 0x7efcbc055e70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7efcd800d000 tx=0x7efcd80027f0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:34.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.692+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efcd0043070 con 0x7efcdc0adff0 2022-07-20T03:16:34.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.860+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7efcdc05c280 con 0x7efcdc0adff0 2022-07-20T03:16:34.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.860+0000 7efccf7fe700 1 -- 172.21.15.61:0/585953798 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v40) v1 ==== 80+0+4242 (secure 0 0 0) 0x7efcd0046690 con 0x7efcdc0adff0 2022-07-20T03:16:34.868 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:34.869 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:16:34.870 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:16:34.870 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.871 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:16:34.872 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:16:34.872 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.873 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:16:34.874 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.874 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.875 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.876 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.876 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.876 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.877 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:16:34.877 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:16:34.877 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.877 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:16:34.878 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.878 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.878 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.878 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.879 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.879 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.879 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:16:34.880 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:16:34.880 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.881 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.881 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.881 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.881 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.882 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.882 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.882 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:16:34.882 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:16:34.883 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.883 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.883 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.883 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.884 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.884 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.884 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.884 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:16:34.885 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:16:34.885 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.885 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.885 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.886 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.886 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.886 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.886 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.887 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:16:34.887 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:16:34.887 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.887 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.888 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.888 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.888 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.888 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.888 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.889 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:16:34.889 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:16:34.889 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.889 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.890 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.890 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.890 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.891 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.891 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.891 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:16:34.892 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:16:34.892 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.892 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.892 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.893 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.893 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.893 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.893 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.894 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:16:34.894 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:16:34.895 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.895 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.895 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.895 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.896 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.896 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.896 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:16:34.897 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:16:34.897 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.897 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.897 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.897 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.898 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.898 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.900 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.900 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:16:34.901 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:16:34.901 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.901 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:16:34.901 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:16:34.902 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:16:34.902 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.903 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.903 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:16:34.903 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:16:34.903 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.904 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:16:34.904 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:16:34.904 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:16:34.904 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.904 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.905 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.905 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:16:34.905 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:16:34.906 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.906 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:16:34.906 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:16:34.906 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:16:34.907 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.907 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.907 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.907 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:16:34.907 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:16:34.908 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.908 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:16:34.908 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:16:34.909 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:16:34.909 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.909 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.910 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.910 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:16:34.910 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:16:34.910 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.911 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:16:34.911 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:16:34.911 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.911 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.911 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:34.912 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:34.912 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:16:34.912 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:16:34.912 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:16:34.913 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:16:34.913 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:16:34.913 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:16:34.913 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.914 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:34.914 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:34.914 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:16:34.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcbc0539b0 msgr2=0x7efcbc055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcbc0539b0 0x7efcbc055e70 secure :-1 s=READY pgs=601 cs=0 l=1 rev1=1 rx=0x7efcd800d000 tx=0x7efcd80027f0).stop 2022-07-20T03:16:34.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcdc0adff0 msgr2=0x7efcdc1111d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:34.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcdc0adff0 0x7efcdc1111d0 secure :-1 s=READY pgs=568 cs=0 l=1 rev1=1 rx=0x7efcd000c000 tx=0x7efcd0006df0).stop 2022-07-20T03:16:34.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 shutdown_connections 2022-07-20T03:16:34.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcbc0539b0 0x7efcbc055e70 unknown :-1 s=CLOSED pgs=601 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efcdc0b5650 0x7efcdc0648b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efcdc0af8e0 0x7efcdc144ff0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 --2- 172.21.15.61:0/585953798 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efcdc0adff0 0x7efcdc1111d0 unknown :-1 s=CLOSED pgs=568 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:34.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 >> 172.21.15.61:0/585953798 conn(0x7efcdc0634e0 msgr2=0x7efcdc0ae840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:34.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 shutdown_connections 2022-07-20T03:16:34.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:34.864+0000 7efce3475700 1 -- 172.21.15.61:0/585953798 wait complete. 2022-07-20T03:16:34.920 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:35.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 Processor -- start 2022-07-20T03:16:35.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- start start 2022-07-20T03:16:35.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faa6005c3f0 0x7faa6014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7faa600acef0 0x7faa60146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faa6005d480 con 0x7faa600b1e00 2022-07-20T03:16:35.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa600b21e0 con 0x7faa6005c3f0 2022-07-20T03:16:35.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faa6005cf40 con 0x7faa600acef0 2022-07-20T03:16:35.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faa6005c3f0 0x7faa6014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7faa6005c3f0 0x7faa6014a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50206/0 (socket says 172.21.15.61:50206) 2022-07-20T03:16:35.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 -- 172.21.15.61:0/42147892 learned_addr learned my addr 172.21.15.61:0/42147892 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:35.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5f7fe700 1 --2- 172.21.15.61:0/42147892 >> v2:172.21.15.117:3300/0 conn(0x7faa600acef0 0x7faa60146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 -- 172.21.15.61:0/42147892 >> v2:172.21.15.117:3300/0 conn(0x7faa600acef0 msgr2=0x7faa60146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:35.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 --2- 172.21.15.61:0/42147892 >> v2:172.21.15.117:3300/0 conn(0x7faa600acef0 0x7faa60146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 -- 172.21.15.61:0/42147892 >> v1:172.21.15.61:6789/0 conn(0x7faa600b1e00 legacy=0x7faa600ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 -- 172.21.15.61:0/42147892 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa6005d1e0 con 0x7faa6005c3f0 2022-07-20T03:16:35.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5e7fc700 1 --2- 172.21.15.61:0/42147892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa6005c3f0 0x7faa6014a6d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7faa500032f0 tx=0x7faa50009290).ready entity=mon.0 client_cookie=33d4f108f2e60011 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5dffb700 1 -- 172.21.15.61:0/42147892 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa5000b870 con 0x7faa6005c3f0 2022-07-20T03:16:35.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5dffb700 1 -- 172.21.15.61:0/42147892 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa5000b9d0 con 0x7faa6005c3f0 2022-07-20T03:16:35.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa5dffb700 1 -- 172.21.15.61:0/42147892 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa5000bb40 con 0x7faa6005c3f0 2022-07-20T03:16:35.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- 172.21.15.61:0/42147892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa6005c3f0 msgr2=0x7faa6014a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 --2- 172.21.15.61:0/42147892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa6005c3f0 0x7faa6014a6d0 secure :-1 s=READY pgs=1250 cs=0 l=1 rev1=1 rx=0x7faa500032f0 tx=0x7faa50009290).stop 2022-07-20T03:16:35.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- 172.21.15.61:0/42147892 shutdown_connections 2022-07-20T03:16:35.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 --2- 172.21.15.61:0/42147892 >> v2:172.21.15.117:3300/0 conn(0x7faa600acef0 0x7faa60146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 --2- 172.21.15.61:0/42147892 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa6005c3f0 0x7faa6014a6d0 unknown :-1 s=CLOSED pgs=1250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- 172.21.15.61:0/42147892 >> 172.21.15.61:0/42147892 conn(0x7faa600611c0 msgr2=0x7faa60062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- 172.21.15.61:0/42147892 shutdown_connections 2022-07-20T03:16:35.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.024+0000 7faa6587d700 1 -- 172.21.15.61:0/42147892 wait complete. 2022-07-20T03:16:35.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 Processor -- start 2022-07-20T03:16:35.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 -- start start 2022-07-20T03:16:35.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa600acef0 0x7faa6009e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 0x7faa6009f650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 0x7faa60143400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6009fb40 con 0x7faa600b1e00 2022-07-20T03:16:35.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faa6009fd40 con 0x7faa600a1c00 2022-07-20T03:16:35.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faa6009ffe0 con 0x7faa600acef0 2022-07-20T03:16:35.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 0x7faa6009f650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5f7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 0x7faa60143400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5f7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 0x7faa60143400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59894/0 (socket says 172.21.15.61:59894) 2022-07-20T03:16:35.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 0x7faa6009f650 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50212/0 (socket says 172.21.15.61:50212) 2022-07-20T03:16:35.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5f7fe700 1 -- 172.21.15.61:0/2834142161 learned_addr learned my addr 172.21.15.61:0/2834142161 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:35.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5effd700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa600acef0 0x7faa6009e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 -- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa600acef0 msgr2=0x7faa6009e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa600acef0 0x7faa6009e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 -- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 msgr2=0x7faa60143400 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.047 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 0x7faa60143400 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 -- 172.21.15.61:0/2834142161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faa50009530 con 0x7faa600b1e00 2022-07-20T03:16:35.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5f7fe700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 0x7faa60143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:35.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa5e7fc700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 0x7faa6009f650 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7faa500032f0 tx=0x7faa50016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa5000b870 con 0x7faa600b1e00 2022-07-20T03:16:35.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7faa50021440 con 0x7faa600b1e00 2022-07-20T03:16:35.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.028+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faa600a0280 con 0x7faa600b1e00 2022-07-20T03:16:35.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7faa500217a0 con 0x7faa600b1e00 2022-07-20T03:16:35.049 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faa601438f0 con 0x7faa600b1e00 2022-07-20T03:16:35.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faa60143f10 con 0x7faa600b1e00 2022-07-20T03:16:35.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7faa50027020 con 0x7faa600b1e00 2022-07-20T03:16:35.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa57fff700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faa38053ad0 0x7faa38055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5253+0+0 (secure 0 0 0) 0x7faa5006ce70 con 0x7faa600b1e00 2022-07-20T03:16:35.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.032+0000 7faa5effd700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faa38053ad0 0x7faa38055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.036+0000 7faa5effd700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faa38053ad0 0x7faa38055f90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7faa4800d000 tx=0x7faa48007b70).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.040+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faa50044100 con 0x7faa600b1e00 2022-07-20T03:16:35.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.164+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7faa600b04b0 con 0x7faa600b1e00 2022-07-20T03:16:35.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa57fff700 1 -- 172.21.15.61:0/2834142161 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v126) v1 ==== 74+0+13239 (secure 0 0 0) 0x7faa50015070 con 0x7faa600b1e00 2022-07-20T03:16:35.172 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:35.173 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":126,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:34.511995+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":24,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":115,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":121,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:35.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faa38053ad0 msgr2=0x7faa38055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faa38053ad0 0x7faa38055f90 secure :-1 s=READY pgs=602 cs=0 l=1 rev1=1 rx=0x7faa4800d000 tx=0x7faa48007b70).stop 2022-07-20T03:16:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 msgr2=0x7faa6009f650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 0x7faa6009f650 secure :-1 s=READY pgs=1251 cs=0 l=1 rev1=1 rx=0x7faa500032f0 tx=0x7faa50016000).stop 2022-07-20T03:16:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 shutdown_connections 2022-07-20T03:16:35.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7faa38053ad0 0x7faa38055f90 unknown :-1 s=CLOSED pgs=602 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faa600a1c00 0x7faa60143400 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faa600b1e00 0x7faa6009f650 unknown :-1 s=CLOSED pgs=1251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 --2- 172.21.15.61:0/2834142161 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faa600acef0 0x7faa6009e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 >> 172.21.15.61:0/2834142161 conn(0x7faa6005c3f0 msgr2=0x7faa601458f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 shutdown_connections 2022-07-20T03:16:35.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.168+0000 7faa6587d700 1 -- 172.21.15.61:0/2834142161 wait complete. 2022-07-20T03:16:35.185 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:16:35.186 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:16:35.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 Processor -- start 2022-07-20T03:16:35.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 -- start start 2022-07-20T03:16:35.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0980b3ee0 0x7fb09805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0980b1b10 0x7fb0981474a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb09805d130 0x7fb098147990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb098147e80 con 0x7fb0980b3ee0 2022-07-20T03:16:35.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb098067c90 con 0x7fb0980b1b10 2022-07-20T03:16:35.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb0980b42c0 con 0x7fb09805d130 2022-07-20T03:16:35.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09c895700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb0980b3ee0 0x7fb09805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0980b1b10 0x7fb0981474a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb0980b1b10 0x7fb0981474a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:59900/0 (socket says 172.21.15.61:59900) 2022-07-20T03:16:35.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 -- 172.21.15.61:0/4228670639 learned_addr learned my addr 172.21.15.61:0/4228670639 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:35.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d096700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb09805d130 0x7fb098147990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 -- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb09805d130 msgr2=0x7fb098147990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:35.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 --2- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb09805d130 0x7fb098147990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 -- 172.21.15.61:0/4228670639 >> v2:172.21.15.61:3300/0 conn(0x7fb0980b3ee0 msgr2=0x7fb09805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:35.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 --2- 172.21.15.61:0/4228670639 >> v2:172.21.15.61:3300/0 conn(0x7fb0980b3ee0 0x7fb09805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 -- 172.21.15.61:0/4228670639 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb0980b1330 con 0x7fb0980b1b10 2022-07-20T03:16:35.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09c895700 1 --2- 172.21.15.61:0/4228670639 >> v2:172.21.15.61:3300/0 conn(0x7fb0980b3ee0 0x7fb09805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:35.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb097fff700 1 --2- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b1b10 0x7fb0981474a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fb088002f70 tx=0x7fb088006eb0).ready entity=mon.1 client_cookie=ea4f3ff5c8a6dff1 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb0977fe700 1 -- 172.21.15.61:0/4228670639 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb08800d570 con 0x7fb0980b1b10 2022-07-20T03:16:35.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb0977fe700 1 -- 172.21.15.61:0/4228670639 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb088000e90 con 0x7fb0980b1b10 2022-07-20T03:16:35.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb0977fe700 1 -- 172.21.15.61:0/4228670639 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb08800d900 con 0x7fb0980b1b10 2022-07-20T03:16:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.284+0000 7fb09d897700 1 -- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b1b10 msgr2=0x7fb0981474a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b1b10 0x7fb0981474a0 secure :-1 s=READY pgs=569 cs=0 l=1 rev1=1 rx=0x7fb088002f70 tx=0x7fb088006eb0).stop 2022-07-20T03:16:35.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- 172.21.15.61:0/4228670639 shutdown_connections 2022-07-20T03:16:35.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb09805d130 0x7fb098147990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- 172.21.15.61:0/4228670639 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b1b10 0x7fb0981474a0 unknown :-1 s=CLOSED pgs=569 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- 172.21.15.61:0/4228670639 >> v2:172.21.15.61:3300/0 conn(0x7fb0980b3ee0 0x7fb09805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- 172.21.15.61:0/4228670639 >> 172.21.15.61:0/4228670639 conn(0x7fb098064c50 msgr2=0x7fb098065030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- 172.21.15.61:0/4228670639 shutdown_connections 2022-07-20T03:16:35.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- 172.21.15.61:0/4228670639 wait complete. 2022-07-20T03:16:35.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 Processor -- start 2022-07-20T03:16:35.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- start start 2022-07-20T03:16:35.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 0x7fb098119c60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0980b1b10 0x7fb09814a3f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b3ee0 0x7fb0980af000 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098143f50 con 0x7fb09805d130 2022-07-20T03:16:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb098156570 con 0x7fb0980b3ee0 2022-07-20T03:16:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d897700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb098156810 con 0x7fb0980b1b10 2022-07-20T03:16:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 0x7fb098119c60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 0x7fb098119c60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50222/0 (socket says 172.21.15.61:50222) 2022-07-20T03:16:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 -- 172.21.15.61:0/701397154 learned_addr learned my addr 172.21.15.61:0/701397154 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb097fff700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0980b1b10 0x7fb09814a3f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09d096700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b3ee0 0x7fb0980af000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 -- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0980b1b10 msgr2=0x7fb09814a3f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0980b1b10 0x7fb09814a3f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 -- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b3ee0 msgr2=0x7fb0980af000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b3ee0 0x7fb0980af000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 -- 172.21.15.61:0/701397154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb080009470 con 0x7fb09805d130 2022-07-20T03:16:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.288+0000 7fb09c895700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 0x7fb098119c60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7fb08000c000 tx=0x7fb0800091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb08000c920 con 0x7fb09805d130 2022-07-20T03:16:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb08800b1e0 con 0x7fb09805d130 2022-07-20T03:16:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb08000ca80 con 0x7fb09805d130 2022-07-20T03:16:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb08000cde0 con 0x7fb09805d130 2022-07-20T03:16:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb098156f50 con 0x7fb09805d130 2022-07-20T03:16:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb098156370 con 0x7fb09805d130 2022-07-20T03:16:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb080022070 con 0x7fb09805d130 2022-07-20T03:16:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb095ffb700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb07c053a40 0x7fb07c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb08006c500 con 0x7fb09805d130 2022-07-20T03:16:35.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb097fff700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb07c053a40 0x7fb07c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.292+0000 7fb097fff700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb07c053a40 0x7fb07c055f00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fb08801c000 tx=0x7fb088015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.300+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb0800436f0 con 0x7fb09805d130 2022-07-20T03:16:35.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.428+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fb098147000 con 0x7fb09805d130 2022-07-20T03:16:35.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.428+0000 7fb095ffb700 1 -- 172.21.15.61:0/701397154 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v126) v1 ==== 110+0+10 (secure 0 0 0) 0x7fb080046d10 con 0x7fb09805d130 2022-07-20T03:16:35.433 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:16:35.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb07c053a40 msgr2=0x7fb07c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb07c053a40 0x7fb07c055f00 secure :-1 s=READY pgs=603 cs=0 l=1 rev1=1 rx=0x7fb08801c000 tx=0x7fb088015000).stop 2022-07-20T03:16:35.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 msgr2=0x7fb098119c60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 0x7fb098119c60 secure :-1 s=READY pgs=1252 cs=0 l=1 rev1=1 rx=0x7fb08000c000 tx=0x7fb0800091d0).stop 2022-07-20T03:16:35.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 shutdown_connections 2022-07-20T03:16:35.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb07c053a40 0x7fb07c055f00 unknown :-1 s=CLOSED pgs=603 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb0980b3ee0 0x7fb0980af000 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb0980b1b10 0x7fb09814a3f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 --2- 172.21.15.61:0/701397154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb09805d130 0x7fb098119c60 unknown :-1 s=CLOSED pgs=1252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 >> 172.21.15.61:0/701397154 conn(0x7fb098064c50 msgr2=0x7fb0980b65a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 shutdown_connections 2022-07-20T03:16:35.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.432+0000 7fb09d897700 1 -- 172.21.15.61:0/701397154 wait complete. 2022-07-20T03:16:35.455 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:16:35.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 Processor -- start 2022-07-20T03:16:35.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 -- start start 2022-07-20T03:16:35.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3d80b59d0 0x7fb3d80b07e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb3d80b14f0 0x7fb3d8147050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb3d80afbc0 con 0x7fb3d814ada0 2022-07-20T03:16:35.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d8148080 con 0x7fb3d80b59d0 2022-07-20T03:16:35.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.560+0000 7fb3de224700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3d80b00e0 con 0x7fb3d80b14f0 2022-07-20T03:16:35.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dd222700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3d80b59d0 0x7fb3d80b07e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dca21700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fb3d814ada0 0x7fb3d805bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:55332/0 (socket says 172.21.15.61:55332) 2022-07-20T03:16:35.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dca21700 1 -- 172.21.15.61:0/1142285236 learned_addr learned my addr 172.21.15.61:0/1142285236 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:35.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dda23700 1 --2- 172.21.15.61:0/1142285236 >> v2:172.21.15.117:3300/0 conn(0x7fb3d80b14f0 0x7fb3d8147050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3cbfff700 1 -- 172.21.15.61:0/1142285236 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1222770451 0 0) 0x7fb3d80afbc0 con 0x7fb3d814ada0 2022-07-20T03:16:35.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3cbfff700 1 -- 172.21.15.61:0/1142285236 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb3bc003260 con 0x7fb3d814ada0 2022-07-20T03:16:35.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dd222700 1 -- 172.21.15.61:0/1142285236 >> v2:172.21.15.117:3300/0 conn(0x7fb3d80b14f0 msgr2=0x7fb3d8147050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:35.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dd222700 1 --2- 172.21.15.61:0/1142285236 >> v2:172.21.15.117:3300/0 conn(0x7fb3d80b14f0 0x7fb3d8147050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dd222700 1 -- 172.21.15.61:0/1142285236 >> v1:172.21.15.117:6789/0 conn(0x7fb3d814ada0 legacy=0x7fb3d805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dd222700 1 -- 172.21.15.61:0/1142285236 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d80b5db0 con 0x7fb3d80b59d0 2022-07-20T03:16:35.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3dd222700 1 --2- 172.21.15.61:0/1142285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d80b07e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7fb3d4002f70 tx=0x7fb3d4006df0).ready entity=mon.0 client_cookie=e971ed45c75ce82e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3cbfff700 1 -- 172.21.15.61:0/1142285236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3d400d4f0 con 0x7fb3d80b59d0 2022-07-20T03:16:35.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3cbfff700 1 -- 172.21.15.61:0/1142285236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d4000e90 con 0x7fb3d80b59d0 2022-07-20T03:16:35.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3cbfff700 1 -- 172.21.15.61:0/1142285236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3d400d840 con 0x7fb3d80b59d0 2022-07-20T03:16:35.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 -- 172.21.15.61:0/1142285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 msgr2=0x7fb3d80b07e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 --2- 172.21.15.61:0/1142285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d80b07e0 secure :-1 s=READY pgs=1253 cs=0 l=1 rev1=1 rx=0x7fb3d4002f70 tx=0x7fb3d4006df0).stop 2022-07-20T03:16:35.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 -- 172.21.15.61:0/1142285236 shutdown_connections 2022-07-20T03:16:35.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 --2- 172.21.15.61:0/1142285236 >> v2:172.21.15.117:3300/0 conn(0x7fb3d80b14f0 0x7fb3d8147050 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 --2- 172.21.15.61:0/1142285236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d80b07e0 unknown :-1 s=CLOSED pgs=1253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 -- 172.21.15.61:0/1142285236 >> 172.21.15.61:0/1142285236 conn(0x7fb3d80634c0 msgr2=0x7fb3d80b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 -- 172.21.15.61:0/1142285236 shutdown_connections 2022-07-20T03:16:35.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 -- 172.21.15.61:0/1142285236 wait complete. 2022-07-20T03:16:35.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.564+0000 7fb3de224700 1 Processor -- start 2022-07-20T03:16:35.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- start start 2022-07-20T03:16:35.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3d80b14f0 0x7fb3d8145c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d8155f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3d814ada0 0x7fb3d810e950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8066410 con 0x7fb3d80b59d0 2022-07-20T03:16:35.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d80668a0 con 0x7fb3d814ada0 2022-07-20T03:16:35.592 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3d8066640 con 0x7fb3d80b14f0 2022-07-20T03:16:35.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d8155f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d8155f20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50236/0 (socket says 172.21.15.61:50236) 2022-07-20T03:16:35.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 -- 172.21.15.61:0/3383854221 learned_addr learned my addr 172.21.15.61:0/3383854221 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:35.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dd222700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3d80b14f0 0x7fb3d8145c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dda23700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3d814ada0 0x7fb3d810e950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 -- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3d80b14f0 msgr2=0x7fb3d8145c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3d80b14f0 0x7fb3d8145c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 -- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3d814ada0 msgr2=0x7fb3d810e950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3d814ada0 0x7fb3d810e950 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 -- 172.21.15.61:0/3383854221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3d400b180 con 0x7fb3d80b59d0 2022-07-20T03:16:35.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3dca21700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d8155f20 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7fb3cc001110 tx=0x7fb3cc007a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3cc007110 con 0x7fb3d80b59d0 2022-07-20T03:16:35.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3cc007270 con 0x7fb3d80b59d0 2022-07-20T03:16:35.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3d810e4b0 con 0x7fb3d80b59d0 2022-07-20T03:16:35.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3cc005030 con 0x7fb3d80b59d0 2022-07-20T03:16:35.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3d80af020 con 0x7fb3d80b59d0 2022-07-20T03:16:35.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.568+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3d810e670 con 0x7fb3d80b59d0 2022-07-20T03:16:35.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.572+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb3cc01b070 con 0x7fb3d80b59d0 2022-07-20T03:16:35.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.572+0000 7fb3ca7fc700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c00539b0 0x7fb3c0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.572+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb3cc06af60 con 0x7fb3d80b59d0 2022-07-20T03:16:35.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.572+0000 7fb3dd222700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c00539b0 0x7fb3c0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.572+0000 7fb3dd222700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c00539b0 0x7fb3c0055e70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fb3d401a000 tx=0x7fb3d4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.576+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3cc0421f0 con 0x7fb3d80b59d0 2022-07-20T03:16:35.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.700+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb3d80b0340 con 0x7fb3d80b59d0 2022-07-20T03:16:35.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3ca7fc700 1 -- 172.21.15.61:0/3383854221 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v126) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb3cc045810 con 0x7fb3d80b59d0 2022-07-20T03:16:35.709 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:35.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c00539b0 msgr2=0x7fb3c0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c00539b0 0x7fb3c0055e70 secure :-1 s=READY pgs=604 cs=0 l=1 rev1=1 rx=0x7fb3d401a000 tx=0x7fb3d4014000).stop 2022-07-20T03:16:35.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 msgr2=0x7fb3d8155f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d8155f20 secure :-1 s=READY pgs=1254 cs=0 l=1 rev1=1 rx=0x7fb3cc001110 tx=0x7fb3cc007a40).stop 2022-07-20T03:16:35.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 shutdown_connections 2022-07-20T03:16:35.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c00539b0 0x7fb3c0055e70 unknown :-1 s=CLOSED pgs=604 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3d814ada0 0x7fb3d810e950 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3d80b59d0 0x7fb3d8155f20 unknown :-1 s=CLOSED pgs=1254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 --2- 172.21.15.61:0/3383854221 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3d80b14f0 0x7fb3d8145c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 >> 172.21.15.61:0/3383854221 conn(0x7fb3d80634c0 msgr2=0x7fb3d814b4a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 shutdown_connections 2022-07-20T03:16:35.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.704+0000 7fb3de224700 1 -- 172.21.15.61:0/3383854221 wait complete. 2022-07-20T03:16:35.721 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:16:35.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf34a2700 1 Processor -- start 2022-07-20T03:16:35.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf34a2700 1 -- start start 2022-07-20T03:16:35.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf34a2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdcec05c3f0 0x7fdcec14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf34a2700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec05d480 con 0x7fdcec0b1e00 2022-07-20T03:16:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf34a2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdcec0b21e0 con 0x7fdcec0acef0 2022-07-20T03:16:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf34a2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdcec05cf40 con 0x7fdcec05c3f0 2022-07-20T03:16:35.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf123e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fdcec0b1e00 0x7fdcec0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46842/0 (socket says 172.21.15.61:46842) 2022-07-20T03:16:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf123e700 1 -- 172.21.15.61:0/3420580850 learned_addr learned my addr 172.21.15.61:0/3420580850 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcf0a3d700 1 --2- 172.21.15.61:0/3420580850 >> v2:172.21.15.61:3300/0 conn(0x7fdcec05c3f0 0x7fdcec14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcebfff700 1 -- 172.21.15.61:0/3420580850 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 660691262 0 0) 0x7fdcec05d480 con 0x7fdcec0b1e00 2022-07-20T03:16:35.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.816+0000 7fdcebfff700 1 -- 172.21.15.61:0/3420580850 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fdcd8003260 con 0x7fdcec0b1e00 2022-07-20T03:16:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf0a3d700 1 -- 172.21.15.61:0/3420580850 >> v1:172.21.15.117:6789/0 conn(0x7fdcec0acef0 legacy=0x7fdcec146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf0a3d700 1 -- 172.21.15.61:0/3420580850 >> v1:172.21.15.61:6789/0 conn(0x7fdcec0b1e00 legacy=0x7fdcec0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcebfff700 1 -- 172.21.15.61:0/3420580850 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3037594377 0 0) 0x7fdcd8003260 con 0x7fdcec0b1e00 2022-07-20T03:16:35.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf0a3d700 1 -- 172.21.15.61:0/3420580850 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdcec05d1e0 con 0x7fdcec05c3f0 2022-07-20T03:16:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf0a3d700 1 --2- 172.21.15.61:0/3420580850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec05c3f0 0x7fdcec14a6d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fdce4001b80 tx=0x7fdce0006df0).ready entity=mon.0 client_cookie=8295131007dd530 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcebfff700 1 -- 172.21.15.61:0/3420580850 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce40019e0 con 0x7fdcec05c3f0 2022-07-20T03:16:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcebfff700 1 -- 172.21.15.61:0/3420580850 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce0000e90 con 0x7fdcec05c3f0 2022-07-20T03:16:35.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcebfff700 1 -- 172.21.15.61:0/3420580850 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce000d4f0 con 0x7fdcec05c3f0 2022-07-20T03:16:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 -- 172.21.15.61:0/3420580850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec05c3f0 msgr2=0x7fdcec14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/3420580850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec05c3f0 0x7fdcec14a6d0 secure :-1 s=READY pgs=1256 cs=0 l=1 rev1=1 rx=0x7fdce4001b80 tx=0x7fdce0006df0).stop 2022-07-20T03:16:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 -- 172.21.15.61:0/3420580850 shutdown_connections 2022-07-20T03:16:35.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/3420580850 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec05c3f0 0x7fdcec14a6d0 unknown :-1 s=CLOSED pgs=1256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 -- 172.21.15.61:0/3420580850 >> 172.21.15.61:0/3420580850 conn(0x7fdcec0611c0 msgr2=0x7fdcec062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 -- 172.21.15.61:0/3420580850 shutdown_connections 2022-07-20T03:16:35.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 -- 172.21.15.61:0/3420580850 wait complete. 2022-07-20T03:16:35.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 Processor -- start 2022-07-20T03:16:35.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.820+0000 7fdcf34a2700 1 -- start start 2022-07-20T03:16:35.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 0x7fdcec14cc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcec0b1e00 0x7fdcec066ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcec09e530 0x7fdcec066fa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec0a1d10 con 0x7fdcec0acef0 2022-07-20T03:16:35.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec157290 con 0x7fdcec09e530 2022-07-20T03:16:35.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdcec067490 con 0x7fdcec0b1e00 2022-07-20T03:16:35.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 0x7fdcec14cc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 0x7fdcec14cc30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50250/0 (socket says 172.21.15.61:50250) 2022-07-20T03:16:35.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 -- 172.21.15.61:0/2879192039 learned_addr learned my addr 172.21.15.61:0/2879192039 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:35.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf0a3d700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcec0b1e00 0x7fdcec066ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf1a3f700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcec09e530 0x7fdcec066fa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 -- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcec0b1e00 msgr2=0x7fdcec066ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcec0b1e00 0x7fdcec066ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 -- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcec09e530 msgr2=0x7fdcec066fa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcec09e530 0x7fdcec066fa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 -- 172.21.15.61:0/2879192039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdce000b180 con 0x7fdcec0acef0 2022-07-20T03:16:35.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf0a3d700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcec0b1e00 0x7fdcec066ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:35.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf123e700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 0x7fdcec14cc30 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7fdce4001c40 tx=0x7fdce4002140).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce4004ab0 con 0x7fdcec0acef0 2022-07-20T03:16:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdce4004c10 con 0x7fdcec0acef0 2022-07-20T03:16:35.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdce4007170 con 0x7fdcec0acef0 2022-07-20T03:16:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdcec09e1e0 con 0x7fdcec0acef0 2022-07-20T03:16:35.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdcec067b50 con 0x7fdcec0acef0 2022-07-20T03:16:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.824+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdcec157490 con 0x7fdcec0acef0 2022-07-20T03:16:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.828+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdce4010070 con 0x7fdcec0acef0 2022-07-20T03:16:35.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.828+0000 7fdcea7fc700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdccc053a40 0x7fdccc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:35.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.828+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fdce406b780 con 0x7fdcec0acef0 2022-07-20T03:16:35.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.828+0000 7fdcf0a3d700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdccc053a40 0x7fdccc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:35.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.828+0000 7fdcf0a3d700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdccc053a40 0x7fdccc055f00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fdce0019000 tx=0x7fdce0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:35.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.836+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdcec157490 con 0x7fdcec0acef0 2022-07-20T03:16:35.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.964+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7fdcec1570d0 con 0x7fdcec0acef0 2022-07-20T03:16:35.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.964+0000 7fdcea7fc700 1 -- 172.21.15.61:0/2879192039 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v126) v1 ==== 84+0+143 (secure 0 0 0) 0x7fdce4019030 con 0x7fdcec0acef0 2022-07-20T03:16:35.970 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:35.971 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:16:35.971 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:35.972 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:16:35.972 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:16:35.972 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:35.973 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:35.973 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:16:35.973 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:16:35.973 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:35.975 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:16:35.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdccc053a40 msgr2=0x7fdccc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdccc053a40 0x7fdccc055f00 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7fdce0019000 tx=0x7fdce0014000).stop 2022-07-20T03:16:35.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 msgr2=0x7fdcec14cc30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:35.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 0x7fdcec14cc30 secure :-1 s=READY pgs=1257 cs=0 l=1 rev1=1 rx=0x7fdce4001c40 tx=0x7fdce4002140).stop 2022-07-20T03:16:35.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 shutdown_connections 2022-07-20T03:16:35.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdccc053a40 0x7fdccc055f00 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdcec09e530 0x7fdcec066fa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdcec0b1e00 0x7fdcec066ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 --2- 172.21.15.61:0/2879192039 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdcec0acef0 0x7fdcec14cc30 unknown :-1 s=CLOSED pgs=1257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 >> 172.21.15.61:0/2879192039 conn(0x7fdcec05c3f0 msgr2=0x7fdcec069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 shutdown_connections 2022-07-20T03:16:35.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:35.968+0000 7fdcf34a2700 1 -- 172.21.15.61:0/2879192039 wait complete. 2022-07-20T03:16:35.987 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:16:35.988 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:16:36.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 Processor -- start 2022-07-20T03:16:36.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 -- start start 2022-07-20T03:16:36.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f64cc13f5c0 0x7f64cc0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:36.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc143870 0x7f64cc13fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:36.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f64cc098ab0 con 0x7f64cc0b1a00 2022-07-20T03:16:36.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f64cc0ac480 con 0x7f64cc13f5c0 2022-07-20T03:16:36.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d338c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0ac720 con 0x7f64cc143870 2022-07-20T03:16:36.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d238a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f64cc0b1a00 0x7f64cc13eeb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46856/0 (socket says 172.21.15.61:46856) 2022-07-20T03:16:36.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d238a700 1 -- 172.21.15.61:0/2710428938 learned_addr learned my addr 172.21.15.61:0/2710428938 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:36.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d1b89700 1 --2- 172.21.15.61:0/2710428938 >> v2:172.21.15.117:3300/0 conn(0x7f64cc13f5c0 0x7f64cc0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:36.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.100+0000 7f64d2b8b700 1 --2- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc143870 0x7f64cc13fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:36.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1388700 1 -- 172.21.15.61:0/2710428938 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 973713704 0 0) 0x7f64c8001110 con 0x7f64cc0b1a00 2022-07-20T03:16:36.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1388700 1 -- 172.21.15.61:0/2710428938 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f64b4003260 con 0x7f64cc0b1a00 2022-07-20T03:16:36.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1b89700 1 -- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc143870 msgr2=0x7f64cc13fb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:36.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1b89700 1 --2- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc143870 0x7f64cc13fb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:36.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1b89700 1 -- 172.21.15.61:0/2710428938 >> v1:172.21.15.61:6789/0 conn(0x7f64cc0b1a00 legacy=0x7f64cc13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:36.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1b89700 1 -- 172.21.15.61:0/2710428938 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64cc143d20 con 0x7f64cc13f5c0 2022-07-20T03:16:36.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1b89700 1 --2- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc13f5c0 0x7f64cc0b1de0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f64cc061760 tx=0x7f64bc006df0).ready entity=mon.1 client_cookie=e57346c5d08d4315 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:36.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1388700 1 -- 172.21.15.61:0/2710428938 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64bc00d570 con 0x7f64cc13f5c0 2022-07-20T03:16:36.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1388700 1 -- 172.21.15.61:0/2710428938 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64bc000e90 con 0x7f64cc13f5c0 2022-07-20T03:16:36.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d1388700 1 -- 172.21.15.61:0/2710428938 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64bc00d8c0 con 0x7f64cc13f5c0 2022-07-20T03:16:36.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 -- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc13f5c0 msgr2=0x7f64cc0b1de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:36.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 --2- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc13f5c0 0x7f64cc0b1de0 secure :-1 s=READY pgs=572 cs=0 l=1 rev1=1 rx=0x7f64cc061760 tx=0x7f64bc006df0).stop 2022-07-20T03:16:36.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 -- 172.21.15.61:0/2710428938 shutdown_connections 2022-07-20T03:16:36.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 --2- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc143870 0x7f64cc13fb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:36.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 --2- 172.21.15.61:0/2710428938 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc13f5c0 0x7f64cc0b1de0 unknown :-1 s=CLOSED pgs=572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:36.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 -- 172.21.15.61:0/2710428938 >> 172.21.15.61:0/2710428938 conn(0x7f64cc0b2af0 msgr2=0x7f64cc0b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:36.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 -- 172.21.15.61:0/2710428938 shutdown_connections 2022-07-20T03:16:36.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.104+0000 7f64d338c700 1 -- 172.21.15.61:0/2710428938 wait complete. 2022-07-20T03:16:36.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 Processor -- start 2022-07-20T03:16:36.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 -- start start 2022-07-20T03:16:36.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc0b1a00 0x7f64cc0977f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:36.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc13f5c0 0x7f64cc09bb30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 0x7f64cc140da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc0992e0 con 0x7f64cc143870 2022-07-20T03:16:36.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc099580 con 0x7f64cc0b1a00 2022-07-20T03:16:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d338c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f64cc1428d0 con 0x7f64cc13f5c0 2022-07-20T03:16:36.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 0x7f64cc140da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:36.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 0x7f64cc140da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50268/0 (socket says 172.21.15.61:50268) 2022-07-20T03:16:36.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 -- 172.21.15.61:0/3115139365 learned_addr learned my addr 172.21.15.61:0/3115139365 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:36.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d238a700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc0b1a00 0x7f64cc0977f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:36.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d1b89700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc13f5c0 0x7f64cc09bb30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:36.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 -- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc13f5c0 msgr2=0x7f64cc09bb30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:36.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc13f5c0 0x7f64cc09bb30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:36.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 -- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc0b1a00 msgr2=0x7f64cc0977f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:36.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc0b1a00 0x7f64cc0977f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:36.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.108+0000 7f64d2b8b700 1 -- 172.21.15.61:0/3115139365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64bc00b180 con 0x7f64cc143870 2022-07-20T03:16:36.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64d2b8b700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 0x7f64cc140da0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f64c4006e00 tx=0x7f64c40091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:36.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64c4012600 con 0x7f64cc143870 2022-07-20T03:16:36.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f64c4012760 con 0x7f64cc143870 2022-07-20T03:16:36.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64cc09c020 con 0x7f64cc143870 2022-07-20T03:16:36.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f64c4012ac0 con 0x7f64cc143870 2022-07-20T03:16:36.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64cc1414b0 con 0x7f64cc143870 2022-07-20T03:16:36.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.112+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64cc060ea0 con 0x7f64cc143870 2022-07-20T03:16:36.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.116+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f64c4022070 con 0x7f64cc143870 2022-07-20T03:16:36.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.116+0000 7f64bb7fe700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f64ac053920 0x7f64ac055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:36.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.116+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(126..126 src has 1..126) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f64c406c640 con 0x7f64cc143870 2022-07-20T03:16:36.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.116+0000 7f64d238a700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f64ac053920 0x7f64ac055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:36.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.116+0000 7f64d238a700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f64ac053920 0x7f64ac055de0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f64c800d000 tx=0x7f64c8000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:36.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.124+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f64c40437c0 con 0x7f64cc143870 2022-07-20T03:16:36.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:36.248+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f64cc09c210 con 0x7f64cc143870 2022-07-20T03:16:36.855 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:36.858 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:36.861 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:36.863 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:36.866 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:16:37.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.132+0000 7f64bb7fe700 1 -- 172.21.15.61:0/3115139365 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v127) v1 ==== 133+0+0 (secure 0 0 0) 0x7f64cc09c210 con 0x7f64cc143870 2022-07-20T03:16:37.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f64ac053920 msgr2=0x7f64ac055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f64ac053920 0x7f64ac055de0 secure :-1 s=READY pgs=606 cs=0 l=1 rev1=1 rx=0x7f64c800d000 tx=0x7f64c8000bc0).stop 2022-07-20T03:16:37.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 msgr2=0x7f64cc140da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 0x7f64cc140da0 secure :-1 s=READY pgs=1259 cs=0 l=1 rev1=1 rx=0x7f64c4006e00 tx=0x7f64c40091d0).stop 2022-07-20T03:16:37.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 shutdown_connections 2022-07-20T03:16:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f64ac053920 0x7f64ac055de0 unknown :-1 s=CLOSED pgs=606 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f64cc143870 0x7f64cc140da0 unknown :-1 s=CLOSED pgs=1259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f64cc13f5c0 0x7f64cc09bb30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 --2- 172.21.15.61:0/3115139365 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f64cc0b1a00 0x7f64cc0977f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 >> 172.21.15.61:0/3115139365 conn(0x7f64cc0b2af0 msgr2=0x7f64cc143100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:37.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 shutdown_connections 2022-07-20T03:16:37.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.136+0000 7f64d338c700 1 -- 172.21.15.61:0/3115139365 wait complete. 2022-07-20T03:16:37.147 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:16:37.158 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:16:37.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 Processor -- start 2022-07-20T03:16:37.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 -- start start 2022-07-20T03:16:37.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1e00b1fe0 0x7fa1e00b2e20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:37.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e0142810 0x7fa1e0142e30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:37.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1e00ac980 con 0x7fa1e00ac5a0 2022-07-20T03:16:37.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1e00b33e0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1e48db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e00b3680 con 0x7fa1e0142810 2022-07-20T03:16:37.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1ddd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1e00b1fe0 0x7fa1e00b2e20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:37.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1ddd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1e00b1fe0 0x7fa1e00b2e20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50274/0 (socket says 172.21.15.61:50274) 2022-07-20T03:16:37.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/1261585146 learned_addr learned my addr 172.21.15.61:0/1261585146 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:37.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.272+0000 7fa1ded9d700 1 --2- 172.21.15.61:0/1261585146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e0142810 0x7fa1e0142e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:37.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/1261585146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e0142810 msgr2=0x7fa1e0142e30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:37.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1ddd9b700 1 --2- 172.21.15.61:0/1261585146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e0142810 0x7fa1e0142e30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/1261585146 >> v1:172.21.15.117:6789/0 conn(0x7fa1e00ac5a0 legacy=0x7fa1e0142100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:37.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/1261585146 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1e00b3920 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1dd59a700 1 -- 172.21.15.61:0/1261585146 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 612063525 0 0) 0x7fa1e00ac980 con 0x7fa1e00ac5a0 2022-07-20T03:16:37.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1ddd9b700 1 --2- 172.21.15.61:0/1261585146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e00b1fe0 0x7fa1e00b2e20 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7fa1d4002f70 tx=0x7fa1d4006eb0).ready entity=mon.0 client_cookie=1bbaafc88beac78d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1dd59a700 1 -- 172.21.15.61:0/1261585146 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1d400d570 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1dd59a700 1 -- 172.21.15.61:0/1261585146 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d4000e90 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1dd59a700 1 -- 172.21.15.61:0/1261585146 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1d400d8c0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 -- 172.21.15.61:0/1261585146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e00b1fe0 msgr2=0x7fa1e00b2e20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 --2- 172.21.15.61:0/1261585146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e00b1fe0 0x7fa1e00b2e20 secure :-1 s=READY pgs=1260 cs=0 l=1 rev1=1 rx=0x7fa1d4002f70 tx=0x7fa1d4006eb0).stop 2022-07-20T03:16:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 -- 172.21.15.61:0/1261585146 shutdown_connections 2022-07-20T03:16:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 --2- 172.21.15.61:0/1261585146 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e0142810 0x7fa1e0142e30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 --2- 172.21.15.61:0/1261585146 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e00b1fe0 0x7fa1e00b2e20 unknown :-1 s=CLOSED pgs=1260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 -- 172.21.15.61:0/1261585146 >> 172.21.15.61:0/1261585146 conn(0x7fa1e00b47c0 msgr2=0x7fa1e005bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 -- 172.21.15.61:0/1261585146 shutdown_connections 2022-07-20T03:16:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 -- 172.21.15.61:0/1261585146 wait complete. 2022-07-20T03:16:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 Processor -- start 2022-07-20T03:16:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.276+0000 7fa1e48db700 1 -- start start 2022-07-20T03:16:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1e00ac5a0 0x7fa1e005c900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e00b1fe0 0x7fa1e0155d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 0x7fa1e0156200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e01456a0 con 0x7fa1e0142810 2022-07-20T03:16:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0066830 con 0x7fa1e00ac5a0 2022-07-20T03:16:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1e0063490 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ded9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 0x7fa1e0156200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:37.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e00b1fe0 0x7fa1e0155d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:37.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ded9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 0x7fa1e0156200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50282/0 (socket says 172.21.15.61:50282) 2022-07-20T03:16:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ded9d700 1 -- 172.21.15.61:0/303489969 learned_addr learned my addr 172.21.15.61:0/303489969 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1de59c700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1e00ac5a0 0x7fa1e005c900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1e00ac5a0 msgr2=0x7fa1e005c900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1e00ac5a0 0x7fa1e005c900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/303489969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 msgr2=0x7fa1e0156200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 0x7fa1e0156200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 -- 172.21.15.61:0/303489969 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1d400b1e0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ded9d700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 0x7fa1e0156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1ddd9b700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e00b1fe0 0x7fa1e0155d10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa1e0152070 tx=0x7fa1d4015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1d40116e0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1d4011840 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1e00662f0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa1d4011ba0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1e005cdf0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.280+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1e01461c0 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.284+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa1d4025030 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.284+0000 7fa1cf7fe700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa1bc053a40 0x7fa1bc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:37.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.284+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(127..127 src has 1..127) v4 ==== 5607+0+0 (secure 0 0 0) 0x7fa1d406ce20 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.284+0000 7fa1de59c700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa1bc053a40 0x7fa1bc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:37.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.284+0000 7fa1de59c700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa1bc053a40 0x7fa1bc055f00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fa1d000d000 tx=0x7fa1d00079b0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:37.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.292+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1d4043e40 con 0x7fa1e00b1fe0 2022-07-20T03:16:37.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:37.420+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7fa1e0060230 con 0x7fa1e00b1fe0 2022-07-20T03:16:38.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.148+0000 7fa1cf7fe700 1 -- 172.21.15.61:0/303489969 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v128) v1 ==== 125+0+0 (secure 0 0 0) 0x7fa1d40230a0 con 0x7fa1e00b1fe0 2022-07-20T03:16:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa1bc053a40 msgr2=0x7fa1bc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa1bc053a40 0x7fa1bc055f00 secure :-1 s=READY pgs=607 cs=0 l=1 rev1=1 rx=0x7fa1d000d000 tx=0x7fa1d00079b0).stop 2022-07-20T03:16:38.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e00b1fe0 msgr2=0x7fa1e0155d10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e00b1fe0 0x7fa1e0155d10 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fa1e0152070 tx=0x7fa1d4015000).stop 2022-07-20T03:16:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 shutdown_connections 2022-07-20T03:16:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa1bc053a40 0x7fa1bc055f00 unknown :-1 s=CLOSED pgs=607 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1e0142810 0x7fa1e0156200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1e00b1fe0 0x7fa1e0155d10 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 --2- 172.21.15.61:0/303489969 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1e00ac5a0 0x7fa1e005c900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 >> 172.21.15.61:0/303489969 conn(0x7fa1e00b47c0 msgr2=0x7fa1e00adc70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:38.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 shutdown_connections 2022-07-20T03:16:38.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.156+0000 7fa1e48db700 1 -- 172.21.15.61:0/303489969 wait complete. 2022-07-20T03:16:38.165 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:16:38.174 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:16:38.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.276+0000 7fdd3e1d0700 1 Processor -- start 2022-07-20T03:16:38.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.276+0000 7fdd3e1d0700 1 -- start start 2022-07-20T03:16:38.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fdd380b6690 0x7fdd380b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd380b59a0 0x7fdd3813c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fdd380acae0 con 0x7fdd3813ce60 2022-07-20T03:16:38.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd38147100 con 0x7fdd380b59a0 2022-07-20T03:16:38.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fdd380acd10 con 0x7fdd380b6690 2022-07-20T03:16:38.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd380b59a0 0x7fdd3813c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fdd380b59a0 0x7fdd3813c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50286/0 (socket says 172.21.15.61:50286) 2022-07-20T03:16:38.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 -- 172.21.15.61:0/3010353522 learned_addr learned my addr 172.21.15.61:0/3010353522 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:38.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd377fe700 1 --2- 172.21.15.61:0/3010353522 >> v2:172.21.15.117:3300/0 conn(0x7fdd380b6690 0x7fdd380b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 -- 172.21.15.61:0/3010353522 >> v2:172.21.15.117:3300/0 conn(0x7fdd380b6690 msgr2=0x7fdd380b6a70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:38.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 --2- 172.21.15.61:0/3010353522 >> v2:172.21.15.117:3300/0 conn(0x7fdd380b6690 0x7fdd380b6a70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 -- 172.21.15.61:0/3010353522 >> v1:172.21.15.117:6789/0 conn(0x7fdd3813ce60 legacy=0x7fdd380ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 -- 172.21.15.61:0/3010353522 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd380b5df0 con 0x7fdd380b59a0 2022-07-20T03:16:38.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd36ffd700 1 --2- 172.21.15.61:0/3010353522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b59a0 0x7fdd3813c970 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7fdd20002f70 tx=0x7fdd20006eb0).ready entity=mon.0 client_cookie=3014756ede0273b4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd367fc700 1 -- 172.21.15.61:0/3010353522 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd2000d5c0 con 0x7fdd380b59a0 2022-07-20T03:16:38.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd367fc700 1 -- 172.21.15.61:0/3010353522 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd20000e90 con 0x7fdd380b59a0 2022-07-20T03:16:38.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd367fc700 1 -- 172.21.15.61:0/3010353522 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd2000d950 con 0x7fdd380b59a0 2022-07-20T03:16:38.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/3010353522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b59a0 msgr2=0x7fdd3813c970 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/3010353522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b59a0 0x7fdd3813c970 secure :-1 s=READY pgs=1261 cs=0 l=1 rev1=1 rx=0x7fdd20002f70 tx=0x7fdd20006eb0).stop 2022-07-20T03:16:38.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/3010353522 shutdown_connections 2022-07-20T03:16:38.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/3010353522 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b59a0 0x7fdd3813c970 unknown :-1 s=CLOSED pgs=1261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/3010353522 >> v2:172.21.15.117:3300/0 conn(0x7fdd380b6690 0x7fdd380b6a70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/3010353522 >> 172.21.15.61:0/3010353522 conn(0x7fdd380b9f70 msgr2=0x7fdd380b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:38.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/3010353522 shutdown_connections 2022-07-20T03:16:38.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.280+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/3010353522 wait complete. 2022-07-20T03:16:38.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 Processor -- start 2022-07-20T03:16:38.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 -- start start 2022-07-20T03:16:38.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd380b59a0 0x7fdd380b3190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 0x7fdd38155b90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd3813ce60 0x7fdd381129e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3813f930 con 0x7fdd380b6690 2022-07-20T03:16:38.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3813f3f0 con 0x7fdd3813ce60 2022-07-20T03:16:38.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fdd3813f690 con 0x7fdd380b59a0 2022-07-20T03:16:38.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 0x7fdd38155b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd377fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd380b59a0 0x7fdd380b3190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 0x7fdd38155b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50294/0 (socket says 172.21.15.61:50294) 2022-07-20T03:16:38.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 -- 172.21.15.61:0/2959514043 learned_addr learned my addr 172.21.15.61:0/2959514043 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:38.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd37fff700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd3813ce60 0x7fdd381129e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 -- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd380b59a0 msgr2=0x7fdd380b3190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd380b59a0 0x7fdd380b3190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 -- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd3813ce60 msgr2=0x7fdd381129e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd3813ce60 0x7fdd381129e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 -- 172.21.15.61:0/2959514043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdd2000b1e0 con 0x7fdd380b6690 2022-07-20T03:16:38.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd36ffd700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 0x7fdd38155b90 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7fdd20002f70 tx=0x7fdd20015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd20011760 con 0x7fdd380b6690 2022-07-20T03:16:38.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fdd200118c0 con 0x7fdd380b6690 2022-07-20T03:16:38.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdd38141490 con 0x7fdd380b6690 2022-07-20T03:16:38.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fdd20011c20 con 0x7fdd380b6690 2022-07-20T03:16:38.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.284+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdd381402d0 con 0x7fdd380b6690 2022-07-20T03:16:38.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.288+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdd381126d0 con 0x7fdd380b6690 2022-07-20T03:16:38.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.288+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fdd20025030 con 0x7fdd380b6690 2022-07-20T03:16:38.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.288+0000 7fdd34ff9700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd1c053a60 0x7fdd1c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.288+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(128..128 src has 1..128) v4 ==== 5957+0+0 (secure 0 0 0) 0x7fdd2006cf00 con 0x7fdd380b6690 2022-07-20T03:16:38.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.288+0000 7fdd377fe700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd1c053a60 0x7fdd1c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.292+0000 7fdd377fe700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd1c053a60 0x7fdd1c055f20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fdd2800b000 tx=0x7fdd28006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.296+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fdd20043ed0 con 0x7fdd380b6690 2022-07-20T03:16:38.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.460+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7fdd380acfb0 con 0x7fdd380b6690 2022-07-20T03:16:38.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.464+0000 7fdd34ff9700 1 -- 172.21.15.61:0/2959514043 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 25 and data pool 26 v71) v1 ==== 189+0+0 (secure 0 0 0) 0x7fdd20023020 con 0x7fdd380b6690 2022-07-20T03:16:38.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd1c053a60 msgr2=0x7fdd1c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd1c053a60 0x7fdd1c055f20 secure :-1 s=READY pgs=608 cs=0 l=1 rev1=1 rx=0x7fdd2800b000 tx=0x7fdd28006df0).stop 2022-07-20T03:16:38.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 msgr2=0x7fdd38155b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 0x7fdd38155b90 secure :-1 s=READY pgs=1262 cs=0 l=1 rev1=1 rx=0x7fdd20002f70 tx=0x7fdd20015000).stop 2022-07-20T03:16:38.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 shutdown_connections 2022-07-20T03:16:38.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fdd1c053a60 0x7fdd1c055f20 unknown :-1 s=CLOSED pgs=608 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fdd3813ce60 0x7fdd381129e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fdd380b6690 0x7fdd38155b90 unknown :-1 s=CLOSED pgs=1262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 --2- 172.21.15.61:0/2959514043 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fdd380b59a0 0x7fdd380b3190 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 >> 172.21.15.61:0/2959514043 conn(0x7fdd380b9f70 msgr2=0x7fdd380b7310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:38.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 shutdown_connections 2022-07-20T03:16:38.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.472+0000 7fdd3e1d0700 1 -- 172.21.15.61:0/2959514043 wait complete. 2022-07-20T03:16:38.483 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 25 and data pool 26 2022-07-20T03:16:38.491 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:38.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 Processor -- start 2022-07-20T03:16:38.583 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 -- start start 2022-07-20T03:16:38.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f802005f9e0 0x7f80200606a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f80200b1bb0 0x7f8020142fe0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8020062140 0x7f80201434d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8020061660 con 0x7f80200b1bb0 2022-07-20T03:16:38.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f80200b2050 con 0x7f802005f9e0 2022-07-20T03:16:38.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802686c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8020060200 con 0x7f8020062140 2022-07-20T03:16:38.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f80200b1bb0 0x7f8020142fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f80200b1bb0 0x7f8020142fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50312/0 (socket says 172.21.15.61:50312) 2022-07-20T03:16:38.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 -- 172.21.15.61:0/238278271 learned_addr learned my addr 172.21.15.61:0/238278271 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:38.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802606b700 1 --2- 172.21.15.61:0/238278271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8020062140 0x7f80201434d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f802586a700 1 --2- 172.21.15.61:0/238278271 >> v2:172.21.15.117:3300/0 conn(0x7f802005f9e0 0x7f80200606a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 -- 172.21.15.61:0/238278271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8020062140 msgr2=0x7f80201434d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:38.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 --2- 172.21.15.61:0/238278271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8020062140 0x7f80201434d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 -- 172.21.15.61:0/238278271 >> v2:172.21.15.117:3300/0 conn(0x7f802005f9e0 msgr2=0x7f80200606a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:38.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 --2- 172.21.15.61:0/238278271 >> v2:172.21.15.117:3300/0 conn(0x7f802005f9e0 0x7f80200606a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 -- 172.21.15.61:0/238278271 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8020098bf0 con 0x7f80200b1bb0 2022-07-20T03:16:38.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8025069700 1 --2- 172.21.15.61:0/238278271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80200b1bb0 0x7f8020142fe0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f8010002f70 tx=0x7f8010006f40).ready entity=mon.0 client_cookie=d0b148038b1c0211 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.580+0000 7f8024868700 1 -- 172.21.15.61:0/238278271 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f801000d9a0 con 0x7f80200b1bb0 2022-07-20T03:16:38.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f8024868700 1 -- 172.21.15.61:0/238278271 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010000e90 con 0x7f80200b1bb0 2022-07-20T03:16:38.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f8024868700 1 -- 172.21.15.61:0/238278271 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f801000dd30 con 0x7f80200b1bb0 2022-07-20T03:16:38.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 -- 172.21.15.61:0/238278271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80200b1bb0 msgr2=0x7f8020142fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 --2- 172.21.15.61:0/238278271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80200b1bb0 0x7f8020142fe0 secure :-1 s=READY pgs=1263 cs=0 l=1 rev1=1 rx=0x7f8010002f70 tx=0x7f8010006f40).stop 2022-07-20T03:16:38.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 -- 172.21.15.61:0/238278271 shutdown_connections 2022-07-20T03:16:38.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 --2- 172.21.15.61:0/238278271 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8020062140 0x7f80201434d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 --2- 172.21.15.61:0/238278271 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f80200b1bb0 0x7f8020142fe0 unknown :-1 s=CLOSED pgs=1263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 --2- 172.21.15.61:0/238278271 >> v2:172.21.15.117:3300/0 conn(0x7f802005f9e0 0x7f80200606a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 -- 172.21.15.61:0/238278271 >> 172.21.15.61:0/238278271 conn(0x7f80200b2ca0 msgr2=0x7f80200b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:38.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 -- 172.21.15.61:0/238278271 shutdown_connections 2022-07-20T03:16:38.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 -- 172.21.15.61:0/238278271 wait complete. 2022-07-20T03:16:38.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.584+0000 7f802686c700 1 Processor -- start 2022-07-20T03:16:38.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 -- start start 2022-07-20T03:16:38.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f802005f9e0 0x7f80200ac170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 0x7f8020151700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f80200b1bb0 0x7f8020151bf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80200af1f0 con 0x7f8020062140 2022-07-20T03:16:38.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80201512d0 con 0x7f802005f9e0 2022-07-20T03:16:38.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f802009b240 con 0x7f80200b1bb0 2022-07-20T03:16:38.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 0x7f8020151700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 0x7f8020151700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50318/0 (socket says 172.21.15.61:50318) 2022-07-20T03:16:38.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 -- 172.21.15.61:0/2094709508 learned_addr learned my addr 172.21.15.61:0/2094709508 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:38.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802586a700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f802005f9e0 0x7f80200ac170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802606b700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f80200b1bb0 0x7f8020151bf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 -- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f80200b1bb0 msgr2=0x7f8020151bf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f80200b1bb0 0x7f8020151bf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 -- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f802005f9e0 msgr2=0x7f80200ac170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f802005f9e0 0x7f80200ac170 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 -- 172.21.15.61:0/2094709508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f801000b460 con 0x7f8020062140 2022-07-20T03:16:38.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f8025069700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 0x7f8020151700 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f8010002f70 tx=0x7f8010006f40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f801000dd60 con 0x7f8020062140 2022-07-20T03:16:38.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8010024450 con 0x7f8020062140 2022-07-20T03:16:38.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f80100247b0 con 0x7f8020062140 2022-07-20T03:16:38.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80201520e0 con 0x7f8020062140 2022-07-20T03:16:38.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.588+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8020141220 con 0x7f8020062140 2022-07-20T03:16:38.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.592+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8010021070 con 0x7f8020062140 2022-07-20T03:16:38.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.592+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80200b22f0 con 0x7f8020062140 2022-07-20T03:16:38.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.592+0000 7f800effd700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8000053a40 0x7f8000055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.592+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f801006f620 con 0x7f8020062140 2022-07-20T03:16:38.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.592+0000 7f802586a700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8000053a40 0x7f8000055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.592+0000 7f802586a700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8000053a40 0x7f8000055f00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f801c006000 tx=0x7f801c006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.600+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f80100465c0 con 0x7f8020062140 2022-07-20T03:16:38.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.728+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f80200574e0 con 0x7f8020062140 2022-07-20T03:16:38.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.728+0000 7f800effd700 1 -- 172.21.15.61:0/2094709508 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f8010029070 con 0x7f8020062140 2022-07-20T03:16:38.734 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:38.734 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":129,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:38.464143+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:16:36.305597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T03:16:37.477547+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:38.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8000053a40 msgr2=0x7f8000055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8000053a40 0x7f8000055f00 secure :-1 s=READY pgs=609 cs=0 l=1 rev1=1 rx=0x7f801c006000 tx=0x7f801c006d90).stop 2022-07-20T03:16:38.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 msgr2=0x7f8020151700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 0x7f8020151700 secure :-1 s=READY pgs=1264 cs=0 l=1 rev1=1 rx=0x7f8010002f70 tx=0x7f8010006f40).stop 2022-07-20T03:16:38.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 shutdown_connections 2022-07-20T03:16:38.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8000053a40 0x7f8000055f00 unknown :-1 s=CLOSED pgs=609 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f80200b1bb0 0x7f8020151bf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8020062140 0x7f8020151700 unknown :-1 s=CLOSED pgs=1264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 --2- 172.21.15.61:0/2094709508 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f802005f9e0 0x7f80200ac170 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 >> 172.21.15.61:0/2094709508 conn(0x7f80200b2ca0 msgr2=0x7f802013d860 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:38.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 shutdown_connections 2022-07-20T03:16:38.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.732+0000 7f802686c700 1 -- 172.21.15.61:0/2094709508 wait complete. 2022-07-20T03:16:38.755 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:38.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.852+0000 7fe3e4c1b700 1 Processor -- start 2022-07-20T03:16:38.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.852+0000 7fe3e4c1b700 1 -- start start 2022-07-20T03:16:38.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe3e00b1e00 0x7fe3e014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 0x7fe3e0146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe3e005d480 con 0x7fe3e005c3f0 2022-07-20T03:16:38.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe3e00b21e0 con 0x7fe3e00b1e00 2022-07-20T03:16:38.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e005cf40 con 0x7fe3e00acef0 2022-07-20T03:16:38.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 0x7fe3e0146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3de59c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe3e00b1e00 0x7fe3e014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ddd9b700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fe3e005c3f0 0x7fe3e00ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:55426/0 (socket says 172.21.15.61:55426) 2022-07-20T03:16:38.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 0x7fe3e0146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:37000/0 (socket says 172.21.15.61:37000) 2022-07-20T03:16:38.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ddd9b700 1 -- 172.21.15.61:0/2061892047 learned_addr learned my addr 172.21.15.61:0/2061892047 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:38.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 -- 172.21.15.61:0/2061892047 >> v2:172.21.15.117:3300/0 conn(0x7fe3e00b1e00 msgr2=0x7fe3e014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:38.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 --2- 172.21.15.61:0/2061892047 >> v2:172.21.15.117:3300/0 conn(0x7fe3e00b1e00 0x7fe3e014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 -- 172.21.15.61:0/2061892047 >> v1:172.21.15.117:6789/0 conn(0x7fe3e005c3f0 legacy=0x7fe3e00ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3dd59a700 1 -- 172.21.15.61:0/2061892047 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 486182688 0 0) 0x7fe3e005d480 con 0x7fe3e005c3f0 2022-07-20T03:16:38.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 -- 172.21.15.61:0/2061892047 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3e005d1e0 con 0x7fe3e00acef0 2022-07-20T03:16:38.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3ded9d700 1 --2- 172.21.15.61:0/2061892047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 0x7fe3e0146410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe3d8002f70 tx=0x7fe3d8006df0).ready entity=mon.2 client_cookie=2488f6352594f318 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3dd59a700 1 -- 172.21.15.61:0/2061892047 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3d800d570 con 0x7fe3e00acef0 2022-07-20T03:16:38.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3dd59a700 1 -- 172.21.15.61:0/2061892047 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d8000e90 con 0x7fe3e00acef0 2022-07-20T03:16:38.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3dd59a700 1 -- 172.21.15.61:0/2061892047 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3d800d8c0 con 0x7fe3e00acef0 2022-07-20T03:16:38.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/2061892047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 msgr2=0x7fe3e0146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/2061892047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 0x7fe3e0146410 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fe3d8002f70 tx=0x7fe3d8006df0).stop 2022-07-20T03:16:38.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/2061892047 shutdown_connections 2022-07-20T03:16:38.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/2061892047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e00acef0 0x7fe3e0146410 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/2061892047 >> v2:172.21.15.117:3300/0 conn(0x7fe3e00b1e00 0x7fe3e014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/2061892047 >> 172.21.15.61:0/2061892047 conn(0x7fe3e00611c0 msgr2=0x7fe3e0062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/2061892047 shutdown_connections 2022-07-20T03:16:38.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.856+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/2061892047 wait complete. 2022-07-20T03:16:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 Processor -- start 2022-07-20T03:16:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 -- start start 2022-07-20T03:16:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 0x7fe3e00aeaa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e00b1e00 0x7fe3e00a0670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e0149f00 0x7fe3e01492d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e009e240 con 0x7fe3e00acef0 2022-07-20T03:16:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e00a0b60 con 0x7fe3e00b1e00 2022-07-20T03:16:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3e4c1b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe3e00a1d40 con 0x7fe3e0149f00 2022-07-20T03:16:38.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3de59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 0x7fe3e00aeaa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3de59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 0x7fe3e00aeaa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50330/0 (socket says 172.21.15.61:50330) 2022-07-20T03:16:38.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3de59c700 1 -- 172.21.15.61:0/4233076102 learned_addr learned my addr 172.21.15.61:0/4233076102 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3ddd9b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e00b1e00 0x7fe3e00a0670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.860+0000 7fe3ded9d700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e0149f00 0x7fe3e01492d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3de59c700 1 -- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e0149f00 msgr2=0x7fe3e01492d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3de59c700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e0149f00 0x7fe3e01492d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3de59c700 1 -- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e00b1e00 msgr2=0x7fe3e00a0670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3de59c700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e00b1e00 0x7fe3e00a0670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3de59c700 1 -- 172.21.15.61:0/4233076102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe3d800b180 con 0x7fe3e00acef0 2022-07-20T03:16:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3ddd9b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e00b1e00 0x7fe3e00a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3de59c700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 0x7fe3e00aeaa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7fe3e00a0ca0 tx=0x7fe3d4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3d4010780 con 0x7fe3e00acef0 2022-07-20T03:16:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe3d40108e0 con 0x7fe3e00acef0 2022-07-20T03:16:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe3e009f270 con 0x7fe3e00acef0 2022-07-20T03:16:38.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe3d4010c40 con 0x7fe3e00acef0 2022-07-20T03:16:38.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe3e01497c0 con 0x7fe3e00acef0 2022-07-20T03:16:38.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe3e0149a70 con 0x7fe3e00acef0 2022-07-20T03:16:38.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe3d401c460 con 0x7fe3e00acef0 2022-07-20T03:16:38.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3cf7fe700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3bc053ad0 0x7fe3bc055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:38.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe3d406c360 con 0x7fe3e00acef0 2022-07-20T03:16:38.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.864+0000 7fe3ddd9b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3bc053ad0 0x7fe3bc055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:38.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.868+0000 7fe3ddd9b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3bc053ad0 0x7fe3bc055f90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fe3d000d000 tx=0x7fe3d0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:38.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.868+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe3d40430d0 con 0x7fe3e00acef0 2022-07-20T03:16:38.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.996+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe3e00b04b0 con 0x7fe3e00acef0 2022-07-20T03:16:39.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:38.996+0000 7fe3cf7fe700 1 -- 172.21.15.61:0/4233076102 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v129) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fe3d40466f0 con 0x7fe3e00acef0 2022-07-20T03:16:39.002 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:39.002 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":129,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:38.464143+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:16:36.305597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T03:16:37.477547+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:39.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3bc053ad0 msgr2=0x7fe3bc055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3bc053ad0 0x7fe3bc055f90 secure :-1 s=READY pgs=610 cs=0 l=1 rev1=1 rx=0x7fe3d000d000 tx=0x7fe3d0009040).stop 2022-07-20T03:16:39.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 msgr2=0x7fe3e00aeaa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 0x7fe3e00aeaa0 secure :-1 s=READY pgs=1265 cs=0 l=1 rev1=1 rx=0x7fe3e00a0ca0 tx=0x7fe3d4006df0).stop 2022-07-20T03:16:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 shutdown_connections 2022-07-20T03:16:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe3bc053ad0 0x7fe3bc055f90 unknown :-1 s=CLOSED pgs=610 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe3e00b1e00 0x7fe3e00a0670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe3e0149f00 0x7fe3e01492d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 --2- 172.21.15.61:0/4233076102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe3e00acef0 0x7fe3e00aeaa0 unknown :-1 s=CLOSED pgs=1265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 >> 172.21.15.61:0/4233076102 conn(0x7fe3e005c3f0 msgr2=0x7fe3e00b4690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:39.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 shutdown_connections 2022-07-20T03:16:39.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.000+0000 7fe3e4c1b700 1 -- 172.21.15.61:0/4233076102 wait complete. 2022-07-20T03:16:39.017 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:16:39.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.120+0000 7f1367a24700 1 Processor -- start 2022-07-20T03:16:39.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1367a24700 1 -- start start 2022-07-20T03:16:39.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1367a24700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f136005bd90 0x7f136014d700 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1367a24700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13601461b0 con 0x7f1360063820 2022-07-20T03:16:39.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1367a24700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13600b1970 con 0x7f13600b1590 2022-07-20T03:16:39.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1367a24700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1360061800 con 0x7f136005bd90 2022-07-20T03:16:39.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1364fbf700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1360063820 0x7f13600600b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46936/0 (socket says 172.21.15.61:46936) 2022-07-20T03:16:39.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1364fbf700 1 -- 172.21.15.61:0/973499015 learned_addr learned my addr 172.21.15.61:0/973499015 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:39.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1365fc1700 1 --2- 172.21.15.61:0/973499015 >> v2:172.21.15.61:3300/0 conn(0x7f136005bd90 0x7f136014d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1353fff700 1 -- 172.21.15.61:0/973499015 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4294644591 0 0) 0x7f13601461b0 con 0x7f1360063820 2022-07-20T03:16:39.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1353fff700 1 -- 172.21.15.61:0/973499015 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1344003260 con 0x7f1360063820 2022-07-20T03:16:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1365fc1700 1 -- 172.21.15.61:0/973499015 >> v1:172.21.15.117:6789/0 conn(0x7f13600b1590 legacy=0x7f1360146b20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1353fff700 1 -- 172.21.15.61:0/973499015 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092939358 0 0) 0x7f13600b1970 con 0x7f13600b1590 2022-07-20T03:16:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1365fc1700 1 -- 172.21.15.61:0/973499015 >> v1:172.21.15.61:6789/0 conn(0x7f1360063820 legacy=0x7f13600600b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1365fc1700 1 -- 172.21.15.61:0/973499015 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13601463e0 con 0x7f136005bd90 2022-07-20T03:16:39.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1365fc1700 1 --2- 172.21.15.61:0/973499015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f136005bd90 0x7f136014d700 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f13580032f0 tx=0x7f13580091d0).ready entity=mon.0 client_cookie=824b6522be8e34aa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1353fff700 1 -- 172.21.15.61:0/973499015 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f135800b6c0 con 0x7f136005bd90 2022-07-20T03:16:39.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1353fff700 1 -- 172.21.15.61:0/973499015 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f135800b820 con 0x7f136005bd90 2022-07-20T03:16:39.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1353fff700 1 -- 172.21.15.61:0/973499015 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f135800b990 con 0x7f136005bd90 2022-07-20T03:16:39.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.124+0000 7f1367a24700 1 -- 172.21.15.61:0/973499015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f136005bd90 msgr2=0x7f136014d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 --2- 172.21.15.61:0/973499015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f136005bd90 0x7f136014d700 secure :-1 s=READY pgs=1267 cs=0 l=1 rev1=1 rx=0x7f13580032f0 tx=0x7f13580091d0).stop 2022-07-20T03:16:39.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 -- 172.21.15.61:0/973499015 shutdown_connections 2022-07-20T03:16:39.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 --2- 172.21.15.61:0/973499015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f136005bd90 0x7f136014d700 unknown :-1 s=CLOSED pgs=1267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 -- 172.21.15.61:0/973499015 >> 172.21.15.61:0/973499015 conn(0x7f13600b2340 msgr2=0x7f13600b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:39.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 -- 172.21.15.61:0/973499015 shutdown_connections 2022-07-20T03:16:39.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 -- 172.21.15.61:0/973499015 wait complete. 2022-07-20T03:16:39.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 Processor -- start 2022-07-20T03:16:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.128+0000 7f1367a24700 1 -- start start 2022-07-20T03:16:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f136005bd90 0x7f13600664e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1360063820 0x7f136013fdc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 0x7f136008f990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360064870 con 0x7f13600b1590 2022-07-20T03:16:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1360064ce0 con 0x7f136005bd90 2022-07-20T03:16:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1360064ea0 con 0x7f1360063820 2022-07-20T03:16:39.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 0x7f136008f990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 0x7f136008f990 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50348/0 (socket says 172.21.15.61:50348) 2022-07-20T03:16:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f13657c0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f136005bd90 0x7f13600664e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 -- 172.21.15.61:0/1184428832 learned_addr learned my addr 172.21.15.61:0/1184428832 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:39.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1364fbf700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1360063820 0x7f136013fdc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 -- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1360063820 msgr2=0x7f136013fdc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1360063820 0x7f136013fdc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 -- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f136005bd90 msgr2=0x7f13600664e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f136005bd90 0x7f13600664e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 -- 172.21.15.61:0/1184428832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1358009470 con 0x7f13600b1590 2022-07-20T03:16:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f13657c0700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f136005bd90 0x7f13600664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:39.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1365fc1700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 0x7f136008f990 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7f136008f830 tx=0x7f1358016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f135800b6c0 con 0x7f13600b1590 2022-07-20T03:16:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1358021630 con 0x7f13600b1590 2022-07-20T03:16:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f135800bee0 con 0x7f13600b1590 2022-07-20T03:16:39.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1360145ca0 con 0x7f13600b1590 2022-07-20T03:16:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.132+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1360090020 con 0x7f13600b1590 2022-07-20T03:16:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.136+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1360146680 con 0x7f13600b1590 2022-07-20T03:16:39.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.136+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1358027020 con 0x7f13600b1590 2022-07-20T03:16:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.136+0000 7f13527fc700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1348053920 0x7f1348055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.136+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(129..129 src has 1..129) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f135806d290 con 0x7f13600b1590 2022-07-20T03:16:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.136+0000 7f13657c0700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1348053920 0x7f1348055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.136+0000 7f13657c0700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1348053920 0x7f1348055de0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f135400b000 tx=0x7f1354009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.140+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f13580441f0 con 0x7f13600b1590 2022-07-20T03:16:39.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.300+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f13600574e0 con 0x7f13600b1590 2022-07-20T03:16:39.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.484+0000 7f13527fc700 1 -- 172.21.15.61:0/1184428832 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v72) v1 ==== 118+0+0 (secure 0 0 0) 0x7f13580150b0 con 0x7f13600b1590 2022-07-20T03:16:39.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1348053920 msgr2=0x7f1348055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1348053920 0x7f1348055de0 secure :-1 s=READY pgs=611 cs=0 l=1 rev1=1 rx=0x7f135400b000 tx=0x7f1354009040).stop 2022-07-20T03:16:39.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 msgr2=0x7f136008f990 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 0x7f136008f990 secure :-1 s=READY pgs=1268 cs=0 l=1 rev1=1 rx=0x7f136008f830 tx=0x7f1358016000).stop 2022-07-20T03:16:39.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 shutdown_connections 2022-07-20T03:16:39.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1348053920 0x7f1348055de0 unknown :-1 s=CLOSED pgs=611 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13600b1590 0x7f136008f990 unknown :-1 s=CLOSED pgs=1268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1360063820 0x7f136013fdc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 --2- 172.21.15.61:0/1184428832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f136005bd90 0x7f13600664e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 >> 172.21.15.61:0/1184428832 conn(0x7f13600b2340 msgr2=0x7f13600b3ab0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:39.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 shutdown_connections 2022-07-20T03:16:39.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.488+0000 7f1367a24700 1 -- 172.21.15.61:0/1184428832 wait complete. 2022-07-20T03:16:39.507 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:39.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9052fa6700 1 Processor -- start 2022-07-20T03:16:39.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9052fa6700 1 -- start start 2022-07-20T03:16:39.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9052fa6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f904c05c3f0 0x7f904c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9052fa6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c05d480 con 0x7f904c0b1e00 2022-07-20T03:16:39.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9052fa6700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f904c0b21e0 con 0x7f904c0acef0 2022-07-20T03:16:39.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9052fa6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f904c05cf40 con 0x7f904c05c3f0 2022-07-20T03:16:39.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9050d42700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f904c0b1e00 0x7f904c0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46952/0 (socket says 172.21.15.61:46952) 2022-07-20T03:16:39.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f9050d42700 1 -- 172.21.15.61:0/2110928570 learned_addr learned my addr 172.21.15.61:0/2110928570 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:39.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904bfff700 1 --2- 172.21.15.61:0/2110928570 >> v2:172.21.15.117:3300/0 conn(0x7f904c05c3f0 0x7f904c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904b7fe700 1 -- 172.21.15.61:0/2110928570 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2992209851 0 0) 0x7f904c05d480 con 0x7f904c0b1e00 2022-07-20T03:16:39.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904b7fe700 1 -- 172.21.15.61:0/2110928570 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9038003260 con 0x7f904c0b1e00 2022-07-20T03:16:39.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904bfff700 1 -- 172.21.15.61:0/2110928570 >> v1:172.21.15.117:6789/0 conn(0x7f904c0acef0 legacy=0x7f904c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904bfff700 1 -- 172.21.15.61:0/2110928570 >> v1:172.21.15.61:6789/0 conn(0x7f904c0b1e00 legacy=0x7f904c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904bfff700 1 -- 172.21.15.61:0/2110928570 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f904c05d1e0 con 0x7f904c05c3f0 2022-07-20T03:16:39.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.612+0000 7f904bfff700 1 --2- 172.21.15.61:0/2110928570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c05c3f0 0x7f904c14a6d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f903c002f70 tx=0x7f903c006df0).ready entity=mon.1 client_cookie=6d82086e31e8725e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f904b7fe700 1 -- 172.21.15.61:0/2110928570 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f903c00d610 con 0x7f904c05c3f0 2022-07-20T03:16:39.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f904b7fe700 1 -- 172.21.15.61:0/2110928570 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c000e90 con 0x7f904c05c3f0 2022-07-20T03:16:39.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f904b7fe700 1 -- 172.21.15.61:0/2110928570 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f903c00d960 con 0x7f904c05c3f0 2022-07-20T03:16:39.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- 172.21.15.61:0/2110928570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c05c3f0 msgr2=0x7f904c14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2110928570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c05c3f0 0x7f904c14a6d0 secure :-1 s=READY pgs=578 cs=0 l=1 rev1=1 rx=0x7f903c002f70 tx=0x7f903c006df0).stop 2022-07-20T03:16:39.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- 172.21.15.61:0/2110928570 shutdown_connections 2022-07-20T03:16:39.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2110928570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c05c3f0 0x7f904c14a6d0 unknown :-1 s=CLOSED pgs=578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- 172.21.15.61:0/2110928570 >> 172.21.15.61:0/2110928570 conn(0x7f904c0611c0 msgr2=0x7f904c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:39.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- 172.21.15.61:0/2110928570 shutdown_connections 2022-07-20T03:16:39.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- 172.21.15.61:0/2110928570 wait complete. 2022-07-20T03:16:39.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 Processor -- start 2022-07-20T03:16:39.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- start start 2022-07-20T03:16:39.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f904c0acef0 0x7f904c0b3200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 0x7f904c0b42c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 0x7f904c0a1480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c067a00 con 0x7f904c0b1e00 2022-07-20T03:16:39.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f904c063c00 con 0x7f904c0b47b0 2022-07-20T03:16:39.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9052fa6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f904c063ea0 con 0x7f904c0acef0 2022-07-20T03:16:39.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f904bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 0x7f904c0b42c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9051543700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 0x7f904c0a1480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9051543700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 0x7f904c0a1480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:60040/0 (socket says 172.21.15.61:60040) 2022-07-20T03:16:39.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f904bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 0x7f904c0b42c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50362/0 (socket says 172.21.15.61:50362) 2022-07-20T03:16:39.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9051543700 1 -- 172.21.15.61:0/2096617277 learned_addr learned my addr 172.21.15.61:0/2096617277 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:39.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.616+0000 7f9050d42700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f904c0acef0 0x7f904c0b3200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f904bfff700 1 -- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f904c0acef0 msgr2=0x7f904c0b3200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f904bfff700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f904c0acef0 0x7f904c0b3200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f904bfff700 1 -- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 msgr2=0x7f904c0a1480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f904bfff700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 0x7f904c0a1480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f904bfff700 1 -- 172.21.15.61:0/2096617277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f903c00b180 con 0x7f904c0b1e00 2022-07-20T03:16:39.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9051543700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 0x7f904c0a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:39.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f904bfff700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 0x7f904c0b42c0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f903c002f70 tx=0x7f903c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f903c016070 con 0x7f904c0b1e00 2022-07-20T03:16:39.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f903c00d290 con 0x7f904c0b1e00 2022-07-20T03:16:39.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f904c064140 con 0x7f904c0b1e00 2022-07-20T03:16:39.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f903c013de0 con 0x7f904c0b1e00 2022-07-20T03:16:39.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f904c0b5980 con 0x7f904c0b1e00 2022-07-20T03:16:39.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f904c0b04b0 con 0x7f904c0b1e00 2022-07-20T03:16:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f903c00f030 con 0x7f904c0b1e00 2022-07-20T03:16:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9049ffb700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f902c0539b0 0x7f902c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f903c06cce0 con 0x7f904c0b1e00 2022-07-20T03:16:39.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.620+0000 7f9050d42700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f902c0539b0 0x7f902c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.624+0000 7f9050d42700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f902c0539b0 0x7f902c055e70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f904000d000 tx=0x7f9040009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.624+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f903c043c40 con 0x7f904c0b1e00 2022-07-20T03:16:39.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.784+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f904c0574e0 con 0x7f904c0b1e00 2022-07-20T03:16:39.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.784+0000 7f9049ffb700 1 -- 172.21.15.61:0/2096617277 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f903c047260 con 0x7f904c0b1e00 2022-07-20T03:16:39.795 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:39.796 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:16:39.468063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1435464119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1435464119},{"type":"v1","addr":"172.21.15.61:6833","nonce":1435464119}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:16:39.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f902c0539b0 msgr2=0x7f902c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f902c0539b0 0x7f902c055e70 secure :-1 s=READY pgs=612 cs=0 l=1 rev1=1 rx=0x7f904000d000 tx=0x7f9040009040).stop 2022-07-20T03:16:39.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 msgr2=0x7f904c0b42c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 0x7f904c0b42c0 secure :-1 s=READY pgs=1270 cs=0 l=1 rev1=1 rx=0x7f903c002f70 tx=0x7f903c014000).stop 2022-07-20T03:16:39.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 shutdown_connections 2022-07-20T03:16:39.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f902c0539b0 0x7f902c055e70 unknown :-1 s=CLOSED pgs=612 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f904c0b47b0 0x7f904c0a1480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f904c0b1e00 0x7f904c0b42c0 unknown :-1 s=CLOSED pgs=1270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 --2- 172.21.15.61:0/2096617277 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f904c0acef0 0x7f904c0b3200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 >> 172.21.15.61:0/2096617277 conn(0x7f904c05c3f0 msgr2=0x7f904c069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:39.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 shutdown_connections 2022-07-20T03:16:39.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.788+0000 7f9052fa6700 1 -- 172.21.15.61:0/2096617277 wait complete. 2022-07-20T03:16:39.806 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-20T03:16:39.808 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:39.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.908+0000 7fb4f0b29700 1 Processor -- start 2022-07-20T03:16:39.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.908+0000 7fb4f0b29700 1 -- start start 2022-07-20T03:16:39.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb4ec0b5650 0x7fb4ec0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 0x7fb4ec147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4ec147010 con 0x7fb4ec0af8e0 2022-07-20T03:16:39.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4ec1472b0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec146d40 con 0x7fb4ec0adff0 2022-07-20T03:16:39.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 0x7fb4ec147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eb7fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb4ec0b5650 0x7fb4ec0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 0x7fb4ec147750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:37038/0 (socket says 172.21.15.61:37038) 2022-07-20T03:16:39.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 -- 172.21.15.61:0/823127315 learned_addr learned my addr 172.21.15.61:0/823127315 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:39.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4ea7fc700 1 -- 172.21.15.61:0/823127315 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4115257440 0 0) 0x7fb4ec147010 con 0x7fb4ec0af8e0 2022-07-20T03:16:39.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 -- 172.21.15.61:0/823127315 >> v2:172.21.15.117:3300/0 conn(0x7fb4ec0b5650 msgr2=0x7fb4ec0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:39.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 --2- 172.21.15.61:0/823127315 >> v2:172.21.15.117:3300/0 conn(0x7fb4ec0b5650 0x7fb4ec0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 -- 172.21.15.61:0/823127315 >> v1:172.21.15.117:6789/0 conn(0x7fb4ec0af8e0 legacy=0x7fb4ec05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 -- 172.21.15.61:0/823127315 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4ec147c40 con 0x7fb4ec0adff0 2022-07-20T03:16:39.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4eaffd700 1 --2- 172.21.15.61:0/823127315 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 0x7fb4ec147750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb4d4002f70 tx=0x7fb4d4006e50).ready entity=mon.2 client_cookie=15f650b564808d25 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4ea7fc700 1 -- 172.21.15.61:0/823127315 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4d400d5c0 con 0x7fb4ec0adff0 2022-07-20T03:16:39.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4ea7fc700 1 -- 172.21.15.61:0/823127315 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4d4000e90 con 0x7fb4ec0adff0 2022-07-20T03:16:39.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4ea7fc700 1 -- 172.21.15.61:0/823127315 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4d400d910 con 0x7fb4ec0adff0 2022-07-20T03:16:39.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- 172.21.15.61:0/823127315 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 msgr2=0x7fb4ec147750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/823127315 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 0x7fb4ec147750 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fb4d4002f70 tx=0x7fb4d4006e50).stop 2022-07-20T03:16:39.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- 172.21.15.61:0/823127315 shutdown_connections 2022-07-20T03:16:39.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/823127315 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0adff0 0x7fb4ec147750 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/823127315 >> v2:172.21.15.117:3300/0 conn(0x7fb4ec0b5650 0x7fb4ec0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- 172.21.15.61:0/823127315 >> 172.21.15.61:0/823127315 conn(0x7fb4ec0634e0 msgr2=0x7fb4ec0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:39.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- 172.21.15.61:0/823127315 shutdown_connections 2022-07-20T03:16:39.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.912+0000 7fb4f0b29700 1 -- 172.21.15.61:0/823127315 wait complete. 2022-07-20T03:16:39.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 Processor -- start 2022-07-20T03:16:39.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 -- start start 2022-07-20T03:16:39.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4ec0adff0 0x7fb4ec10fd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0af8e0 0x7fb4ec1483e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 0x7fb4ec144ff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec10eec0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec10f160 con 0x7fb4ec0adff0 2022-07-20T03:16:39.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.916+0000 7fb4f0b29700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4ec10f400 con 0x7fb4ec0af8e0 2022-07-20T03:16:39.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 0x7fb4ec144ff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 0x7fb4ec144ff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50376/0 (socket says 172.21.15.61:50376) 2022-07-20T03:16:39.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 -- 172.21.15.61:0/2700803661 learned_addr learned my addr 172.21.15.61:0/2700803661 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:39.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4eb7fe700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4ec0adff0 0x7fb4ec10fd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4eaffd700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0af8e0 0x7fb4ec1483e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 -- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0af8e0 msgr2=0x7fb4ec1483e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0af8e0 0x7fb4ec1483e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 -- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4ec0adff0 msgr2=0x7fb4ec10fd50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:39.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4ec0adff0 0x7fb4ec10fd50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:39.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 -- 172.21.15.61:0/2700803661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4d400b1e0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4ebfff700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 0x7fb4ec144ff0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7fb4e4000f40 tx=0x7fb4e4007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4e400f470 con 0x7fb4ec0b5650 2022-07-20T03:16:39.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4e400f5d0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4ec1488d0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4e400f930 con 0x7fb4ec0b5650 2022-07-20T03:16:39.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4ec0654f0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4ec05c4a0 con 0x7fb4ec0b5650 2022-07-20T03:16:39.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb4e4010070 con 0x7fb4ec0b5650 2022-07-20T03:16:39.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4e8ff9700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4c4053920 0x7fb4c4055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:39.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4e406b150 con 0x7fb4ec0b5650 2022-07-20T03:16:39.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.920+0000 7fb4eb7fe700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4c4053920 0x7fb4c4055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:39.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.924+0000 7fb4eb7fe700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4c4053920 0x7fb4c4055de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fb4dc00b000 tx=0x7fb4dc006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:39.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:39.928+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4e4042050 con 0x7fb4ec0b5650 2022-07-20T03:16:40.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.052+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4ec0574e0 con 0x7fb4ec0b5650 2022-07-20T03:16:40.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.052+0000 7fb4e8ff9700 1 -- 172.21.15.61:0/2700803661 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fb4e4045670 con 0x7fb4ec0b5650 2022-07-20T03:16:40.057 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:40.058 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":130,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:39.466964+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:16:36.305597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T03:16:37.477547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:40.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4c4053920 msgr2=0x7fb4c4055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4c4053920 0x7fb4c4055de0 secure :-1 s=READY pgs=613 cs=0 l=1 rev1=1 rx=0x7fb4dc00b000 tx=0x7fb4dc006df0).stop 2022-07-20T03:16:40.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 msgr2=0x7fb4ec144ff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 0x7fb4ec144ff0 secure :-1 s=READY pgs=1271 cs=0 l=1 rev1=1 rx=0x7fb4e4000f40 tx=0x7fb4e4007d60).stop 2022-07-20T03:16:40.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 shutdown_connections 2022-07-20T03:16:40.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4c4053920 0x7fb4c4055de0 unknown :-1 s=CLOSED pgs=613 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4ec0b5650 0x7fb4ec144ff0 unknown :-1 s=CLOSED pgs=1271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4ec0af8e0 0x7fb4ec1483e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 --2- 172.21.15.61:0/2700803661 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4ec0adff0 0x7fb4ec10fd50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 >> 172.21.15.61:0/2700803661 conn(0x7fb4ec0634e0 msgr2=0x7fb4ec0b3ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:40.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 shutdown_connections 2022-07-20T03:16:40.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.056+0000 7fb4f0b29700 1 -- 172.21.15.61:0/2700803661 wait complete. 2022-07-20T03:16:40.080 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:16:40.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 Processor -- start 2022-07-20T03:16:40.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- start start 2022-07-20T03:16:40.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f37a00b5830 0x7f37a0142c30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37a0143120 0x7f37a00af2c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37a0147120 con 0x7f37a005cf40 2022-07-20T03:16:40.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a00b0130 con 0x7f37a0143120 2022-07-20T03:16:40.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37a01462d0 con 0x7f37a00b5830 2022-07-20T03:16:40.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37a0143120 0x7f37a00af2c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37a0143120 0x7f37a00af2c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50384/0 (socket says 172.21.15.61:50384) 2022-07-20T03:16:40.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 -- 172.21.15.61:0/1048204846 learned_addr learned my addr 172.21.15.61:0/1048204846 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:40.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37977fe700 1 --2- 172.21.15.61:0/1048204846 >> v2:172.21.15.117:3300/0 conn(0x7f37a00b5830 0x7f37a0142c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 -- 172.21.15.61:0/1048204846 >> v2:172.21.15.117:3300/0 conn(0x7f37a00b5830 msgr2=0x7f37a0142c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:40.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 --2- 172.21.15.61:0/1048204846 >> v2:172.21.15.117:3300/0 conn(0x7f37a00b5830 0x7f37a0142c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 -- 172.21.15.61:0/1048204846 >> v1:172.21.15.117:6789/0 conn(0x7f37a005cf40 legacy=0x7f37a0146a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 -- 172.21.15.61:0/1048204846 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37a0146570 con 0x7f37a0143120 2022-07-20T03:16:40.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37977fe700 1 --2- 172.21.15.61:0/1048204846 >> v2:172.21.15.117:3300/0 conn(0x7f37a00b5830 0x7f37a0142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:40.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a4adf700 1 --2- 172.21.15.61:0/1048204846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a0143120 0x7f37a00af2c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f37a0067d40 tx=0x7f3798009290).ready entity=mon.0 client_cookie=50bd168e9934ca1b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f379f7fe700 1 -- 172.21.15.61:0/1048204846 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379800b860 con 0x7f37a0143120 2022-07-20T03:16:40.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f379f7fe700 1 -- 172.21.15.61:0/1048204846 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f379800b9c0 con 0x7f37a0143120 2022-07-20T03:16:40.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f379f7fe700 1 -- 172.21.15.61:0/1048204846 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f379800bb30 con 0x7f37a0143120 2022-07-20T03:16:40.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- 172.21.15.61:0/1048204846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a0143120 msgr2=0x7f37a00af2c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 --2- 172.21.15.61:0/1048204846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a0143120 0x7f37a00af2c0 secure :-1 s=READY pgs=1272 cs=0 l=1 rev1=1 rx=0x7f37a0067d40 tx=0x7f3798009290).stop 2022-07-20T03:16:40.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- 172.21.15.61:0/1048204846 shutdown_connections 2022-07-20T03:16:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 --2- 172.21.15.61:0/1048204846 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a0143120 0x7f37a00af2c0 unknown :-1 s=CLOSED pgs=1272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 --2- 172.21.15.61:0/1048204846 >> v2:172.21.15.117:3300/0 conn(0x7f37a00b5830 0x7f37a0142c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.196+0000 7f37a6542700 1 -- 172.21.15.61:0/1048204846 >> 172.21.15.61:0/1048204846 conn(0x7f37a00b69e0 msgr2=0x7f37a0066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:40.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- 172.21.15.61:0/1048204846 shutdown_connections 2022-07-20T03:16:40.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- 172.21.15.61:0/1048204846 wait complete. 2022-07-20T03:16:40.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 Processor -- start 2022-07-20T03:16:40.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- start start 2022-07-20T03:16:40.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 0x7f37a0148cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37a00b5830 0x7f37a0149fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37a0143120 0x7f37a01483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a013d6e0 con 0x7f37a005cf40 2022-07-20T03:16:40.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37a013d910 con 0x7f37a0143120 2022-07-20T03:16:40.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37a013dbb0 con 0x7f37a00b5830 2022-07-20T03:16:40.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 0x7f37a0148cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37977fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37a00b5830 0x7f37a0149fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 0x7f37a0148cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50388/0 (socket says 172.21.15.61:50388) 2022-07-20T03:16:40.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 -- 172.21.15.61:0/1835225037 learned_addr learned my addr 172.21.15.61:0/1835225037 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:40.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a4adf700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37a0143120 0x7f37a01483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 -- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37a00b5830 msgr2=0x7f37a0149fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37a00b5830 0x7f37a0149fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 -- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37a0143120 msgr2=0x7f37a01483b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37a0143120 0x7f37a01483b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 -- 172.21.15.61:0/1835225037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3798009530 con 0x7f37a005cf40 2022-07-20T03:16:40.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379ffff700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 0x7f37a0148cc0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f37a00ae7c0 tx=0x7f3790007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3790012400 con 0x7f37a005cf40 2022-07-20T03:16:40.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3790012560 con 0x7f37a005cf40 2022-07-20T03:16:40.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37a01488a0 con 0x7f37a005cf40 2022-07-20T03:16:40.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37900128c0 con 0x7f37a005cf40 2022-07-20T03:16:40.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.200+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37a013e3c0 con 0x7f37a005cf40 2022-07-20T03:16:40.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.204+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37a013dd70 con 0x7f37a005cf40 2022-07-20T03:16:40.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.204+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3790019030 con 0x7f37a005cf40 2022-07-20T03:16:40.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.204+0000 7f379dffb700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3784053920 0x7f3784055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.204+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f379006b2a0 con 0x7f37a005cf40 2022-07-20T03:16:40.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.204+0000 7f37977fe700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3784053920 0x7f3784055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.204+0000 7f37977fe700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3784053920 0x7f3784055de0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f378800c000 tx=0x7f3788008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.212+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f37900420f0 con 0x7f37a005cf40 2022-07-20T03:16:40.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.376+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f37a00574e0 con 0x7f37a005cf40 2022-07-20T03:16:40.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.384+0000 7f379dffb700 1 -- 172.21.15.61:0/1835225037 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v41) v1 ==== 184+0+0 (secure 0 0 0) 0x7f3790045710 con 0x7f37a005cf40 2022-07-20T03:16:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3784053920 msgr2=0x7f3784055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3784053920 0x7f3784055de0 secure :-1 s=READY pgs=614 cs=0 l=1 rev1=1 rx=0x7f378800c000 tx=0x7f3788008040).stop 2022-07-20T03:16:40.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 msgr2=0x7f37a0148cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 0x7f37a0148cc0 secure :-1 s=READY pgs=1273 cs=0 l=1 rev1=1 rx=0x7f37a00ae7c0 tx=0x7f3790007c50).stop 2022-07-20T03:16:40.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 shutdown_connections 2022-07-20T03:16:40.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3784053920 0x7f3784055de0 unknown :-1 s=CLOSED pgs=614 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37a0143120 0x7f37a01483b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37a00b5830 0x7f37a0149fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 --2- 172.21.15.61:0/1835225037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37a005cf40 0x7f37a0148cc0 unknown :-1 s=CLOSED pgs=1273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 >> 172.21.15.61:0/1835225037 conn(0x7f37a00b69e0 msgr2=0x7f37a0144130 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:40.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 shutdown_connections 2022-07-20T03:16:40.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.388+0000 7f37a6542700 1 -- 172.21.15.61:0/1835225037 wait complete. 2022-07-20T03:16:40.400 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:16:40.406 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:16:40.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.512+0000 7f403f3f7700 1 Processor -- start 2022-07-20T03:16:40.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.512+0000 7f403f3f7700 1 -- start start 2022-07-20T03:16:40.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403f3f7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f40380b5970 0x7f40380ae100 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403f3f7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f40380b0b80 con 0x7f4038143a10 2022-07-20T03:16:40.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403f3f7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f403805d4d0 con 0x7f40380af8c0 2022-07-20T03:16:40.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403f3f7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f40380b5d50 con 0x7f40380b5970 2022-07-20T03:16:40.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403ebf6700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4038143a10 0x7f4038143e50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:46992/0 (socket says 172.21.15.61:46992) 2022-07-20T03:16:40.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403ebf6700 1 -- 172.21.15.61:0/2557130569 learned_addr learned my addr 172.21.15.61:0/2557130569 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:40.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403e3f5700 1 --2- 172.21.15.61:0/2557130569 >> v2:172.21.15.117:3300/0 conn(0x7f40380b5970 0x7f40380ae100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4131251811 0 0) 0x7f40380b0b80 con 0x7f4038143a10 2022-07-20T03:16:40.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4020003260 con 0x7f4038143a10 2022-07-20T03:16:40.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 610141983 0 0) 0x7f403805d4d0 con 0x7f40380af8c0 2022-07-20T03:16:40.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f40380b0b80 con 0x7f40380af8c0 2022-07-20T03:16:40.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4030448320 0 0) 0x7f4020003260 con 0x7f4038143a10 2022-07-20T03:16:40.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403e3f5700 1 -- 172.21.15.61:0/2557130569 >> v1:172.21.15.117:6789/0 conn(0x7f40380af8c0 legacy=0x7f40380ae5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403e3f5700 1 -- 172.21.15.61:0/2557130569 >> v1:172.21.15.61:6789/0 conn(0x7f4038143a10 legacy=0x7f4038143e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403e3f5700 1 -- 172.21.15.61:0/2557130569 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40380b0db0 con 0x7f40380b5970 2022-07-20T03:16:40.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403e3f5700 1 --2- 172.21.15.61:0/2557130569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f40380ae100 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f4030001490 tx=0x7f4034006df0).ready entity=mon.1 client_cookie=4d960808337bc07e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f40300019e0 con 0x7f40380b5970 2022-07-20T03:16:40.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4034000e90 con 0x7f40380b5970 2022-07-20T03:16:40.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.516+0000 7f403d3f3700 1 -- 172.21.15.61:0/2557130569 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4034012460 con 0x7f40380b5970 2022-07-20T03:16:40.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- 172.21.15.61:0/2557130569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 msgr2=0x7f40380ae100 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 --2- 172.21.15.61:0/2557130569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f40380ae100 secure :-1 s=READY pgs=582 cs=0 l=1 rev1=1 rx=0x7f4030001490 tx=0x7f4034006df0).stop 2022-07-20T03:16:40.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- 172.21.15.61:0/2557130569 shutdown_connections 2022-07-20T03:16:40.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 --2- 172.21.15.61:0/2557130569 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f40380ae100 unknown :-1 s=CLOSED pgs=582 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- 172.21.15.61:0/2557130569 >> 172.21.15.61:0/2557130569 conn(0x7f4038063490 msgr2=0x7f40380b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:40.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- 172.21.15.61:0/2557130569 shutdown_connections 2022-07-20T03:16:40.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- 172.21.15.61:0/2557130569 wait complete. 2022-07-20T03:16:40.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 Processor -- start 2022-07-20T03:16:40.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- start start 2022-07-20T03:16:40.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 0x7f40381422a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f403813bf40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4038143a10 0x7f4038156050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4038065660 con 0x7f40380af8c0 2022-07-20T03:16:40.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f403810e490 con 0x7f40380b5970 2022-07-20T03:16:40.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f403810e730 con 0x7f4038143a10 2022-07-20T03:16:40.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 0x7f40381422a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 0x7f40381422a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50400/0 (socket says 172.21.15.61:50400) 2022-07-20T03:16:40.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 -- 172.21.15.61:0/418279121 learned_addr learned my addr 172.21.15.61:0/418279121 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:40.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403dbf4700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f403813bf40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403ebf6700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4038143a10 0x7f4038156050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 -- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4038143a10 msgr2=0x7f4038156050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4038143a10 0x7f4038156050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 -- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 msgr2=0x7f403813bf40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f403813bf40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 -- 172.21.15.61:0/418279121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f403400b180 con 0x7f40380af8c0 2022-07-20T03:16:40.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403e3f5700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 0x7f40381422a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f4030001490 tx=0x7f4034014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f403400d9a0 con 0x7f40380af8c0 2022-07-20T03:16:40.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f403810e9d0 con 0x7f40380af8c0 2022-07-20T03:16:40.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f403400db00 con 0x7f40380af8c0 2022-07-20T03:16:40.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.520+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f403400de60 con 0x7f40380af8c0 2022-07-20T03:16:40.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f403810fa00 con 0x7f40380af8c0 2022-07-20T03:16:40.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f403810eb60 con 0x7f40380af8c0 2022-07-20T03:16:40.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f403401c070 con 0x7f40380af8c0 2022-07-20T03:16:40.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f402b7fe700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4018053ad0 0x7f4018055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f403406c970 con 0x7f40380af8c0 2022-07-20T03:16:40.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f403dbf4700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4018053ad0 0x7f4018055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.524+0000 7f403dbf4700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4018053ad0 0x7f4018055f90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f402c00d000 tx=0x7f402c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.532+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4034043830 con 0x7f40380af8c0 2022-07-20T03:16:40.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.703+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f40380acae0 con 0x7f40380af8c0 2022-07-20T03:16:40.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f402b7fe700 1 -- 172.21.15.61:0/418279121 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v42) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4034023070 con 0x7f40380af8c0 2022-07-20T03:16:40.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4018053ad0 msgr2=0x7f4018055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4018053ad0 0x7f4018055f90 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f402c00d000 tx=0x7f402c007a20).stop 2022-07-20T03:16:40.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 msgr2=0x7f40381422a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 0x7f40381422a0 secure :-1 s=READY pgs=1275 cs=0 l=1 rev1=1 rx=0x7f4030001490 tx=0x7f4034014000).stop 2022-07-20T03:16:40.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 shutdown_connections 2022-07-20T03:16:40.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4018053ad0 0x7f4018055f90 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f40380b5970 0x7f403813bf40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4038143a10 0x7f4038156050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 --2- 172.21.15.61:0/418279121 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f40380af8c0 0x7f40381422a0 unknown :-1 s=CLOSED pgs=1275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 >> 172.21.15.61:0/418279121 conn(0x7f4038063490 msgr2=0x7f403814d810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:40.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 shutdown_connections 2022-07-20T03:16:40.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.711+0000 7f403f3f7700 1 -- 172.21.15.61:0/418279121 wait complete. 2022-07-20T03:16:40.726 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:16:40.741 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:16:40.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 Processor -- start 2022-07-20T03:16:40.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 -- start start 2022-07-20T03:16:40.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b3650 0x7fc33c0b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 0x7fc33c0b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc33c0b3e40 con 0x7fc33c0b9ef0 2022-07-20T03:16:40.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc33c0ba2d0 con 0x7fc33c0b2290 2022-07-20T03:16:40.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc3412a2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c0b4070 con 0x7fc33c0b3650 2022-07-20T03:16:40.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc33b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 0x7fc33c0b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc33b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 0x7fc33c0b2740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50412/0 (socket says 172.21.15.61:50412) 2022-07-20T03:16:40.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc33b7fe700 1 -- 172.21.15.61:0/3268124039 learned_addr learned my addr 172.21.15.61:0/3268124039 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:40.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.859+0000 7fc33a7fc700 1 --2- 172.21.15.61:0/3268124039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b3650 0x7fc33c0b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc339ffb700 1 -- 172.21.15.61:0/3268124039 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2435723086 0 0) 0x7fc33c0b3e40 con 0x7fc33c0b9ef0 2022-07-20T03:16:40.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc33a7fc700 1 -- 172.21.15.61:0/3268124039 >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 msgr2=0x7fc33c0b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:40.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc33a7fc700 1 --2- 172.21.15.61:0/3268124039 >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 0x7fc33c0b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc33a7fc700 1 -- 172.21.15.61:0/3268124039 >> v1:172.21.15.117:6789/0 conn(0x7fc33c0b9ef0 legacy=0x7fc33c0b48b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc33a7fc700 1 -- 172.21.15.61:0/3268124039 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc33c0b1df0 con 0x7fc33c0b3650 2022-07-20T03:16:40.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc33b7fe700 1 --2- 172.21.15.61:0/3268124039 >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 0x7fc33c0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:40.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc33a7fc700 1 --2- 172.21.15.61:0/3268124039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b3650 0x7fc33c0b0750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc32c002f70 tx=0x7fc32c006df0).ready entity=mon.2 client_cookie=77f53382263e6e84 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc339ffb700 1 -- 172.21.15.61:0/3268124039 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc32c00d570 con 0x7fc33c0b3650 2022-07-20T03:16:40.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc339ffb700 1 -- 172.21.15.61:0/3268124039 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c000e90 con 0x7fc33c0b3650 2022-07-20T03:16:40.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc339ffb700 1 -- 172.21.15.61:0/3268124039 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc32c00d8c0 con 0x7fc33c0b3650 2022-07-20T03:16:40.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 -- 172.21.15.61:0/3268124039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b3650 msgr2=0x7fc33c0b0750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 --2- 172.21.15.61:0/3268124039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b3650 0x7fc33c0b0750 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fc32c002f70 tx=0x7fc32c006df0).stop 2022-07-20T03:16:40.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 -- 172.21.15.61:0/3268124039 shutdown_connections 2022-07-20T03:16:40.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 --2- 172.21.15.61:0/3268124039 >> v2:172.21.15.61:3300/0 conn(0x7fc33c0b2290 0x7fc33c0b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 --2- 172.21.15.61:0/3268124039 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b3650 0x7fc33c0b0750 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 -- 172.21.15.61:0/3268124039 >> 172.21.15.61:0/3268124039 conn(0x7fc33c067980 msgr2=0x7fc33c0b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:40.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 -- 172.21.15.61:0/3268124039 shutdown_connections 2022-07-20T03:16:40.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.863+0000 7fc3412a2700 1 -- 172.21.15.61:0/3268124039 wait complete. 2022-07-20T03:16:40.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 Processor -- start 2022-07-20T03:16:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 -- start start 2022-07-20T03:16:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b2290 0x7fc33c0ace20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 0x7fc33c0ad310 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc33c0b9ef0 0x7fc33c151e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c1461c0 con 0x7fc33c0b3650 2022-07-20T03:16:40.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c156190 con 0x7fc33c0b9ef0 2022-07-20T03:16:40.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc3412a2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc33c13fd50 con 0x7fc33c0b2290 2022-07-20T03:16:40.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 0x7fc33c0ad310 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 0x7fc33c0ad310 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50416/0 (socket says 172.21.15.61:50416) 2022-07-20T03:16:40.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 -- 172.21.15.61:0/847535509 learned_addr learned my addr 172.21.15.61:0/847535509 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:40.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33affd700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b2290 0x7fc33c0ace20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33b7fe700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc33c0b9ef0 0x7fc33c151e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 -- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b2290 msgr2=0x7fc33c0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b2290 0x7fc33c0ace20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 -- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc33c0b9ef0 msgr2=0x7fc33c151e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:40.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc33c0b9ef0 0x7fc33c151e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:40.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.867+0000 7fc33a7fc700 1 -- 172.21.15.61:0/847535509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc334009530 con 0x7fc33c0b3650 2022-07-20T03:16:40.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc33a7fc700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 0x7fc33c0ad310 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7fc32c002f70 tx=0x7fc32c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc32c016070 con 0x7fc33c0b3650 2022-07-20T03:16:40.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc32c0138d0 con 0x7fc33c0b3650 2022-07-20T03:16:40.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc32c00b180 con 0x7fc33c0b3650 2022-07-20T03:16:40.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc32c013c30 con 0x7fc33c0b3650 2022-07-20T03:16:40.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc33c155e40 con 0x7fc33c0b3650 2022-07-20T03:16:40.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc32c00f030 con 0x7fc33c0b3650 2022-07-20T03:16:40.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc32bfff700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc318053a60 0x7fc318055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:40.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc33c156390 con 0x7fc33c0b3650 2022-07-20T03:16:40.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc32c06ca00 con 0x7fc33c0b3650 2022-07-20T03:16:40.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc33affd700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc318053a60 0x7fc318055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:40.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.871+0000 7fc33affd700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc318053a60 0x7fc318055f20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fc33000d000 tx=0x7fc330007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:40.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:40.879+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc32c023080 con 0x7fc33c0b3650 2022-07-20T03:16:41.048 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.043+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fc33c13ff10 con 0x7fc33c0b3650 2022-07-20T03:16:41.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.051+0000 7fc32bfff700 1 -- 172.21.15.61:0/847535509 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v43) v1 ==== 184+0+0 (secure 0 0 0) 0x7fc32c043960 con 0x7fc33c0b3650 2022-07-20T03:16:41.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.051+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc318053a60 msgr2=0x7fc318055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.051+0000 7fc3412a2700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc318053a60 0x7fc318055f20 secure :-1 s=READY pgs=616 cs=0 l=1 rev1=1 rx=0x7fc33000d000 tx=0x7fc330007b90).stop 2022-07-20T03:16:41.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.051+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 msgr2=0x7fc33c0ad310 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.051+0000 7fc3412a2700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 0x7fc33c0ad310 secure :-1 s=READY pgs=1276 cs=0 l=1 rev1=1 rx=0x7fc32c002f70 tx=0x7fc32c014000).stop 2022-07-20T03:16:41.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 shutdown_connections 2022-07-20T03:16:41.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc318053a60 0x7fc318055f20 unknown :-1 s=CLOSED pgs=616 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc33c0b9ef0 0x7fc33c151e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc33c0b3650 0x7fc33c0ad310 unknown :-1 s=CLOSED pgs=1276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 --2- 172.21.15.61:0/847535509 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc33c0b2290 0x7fc33c0ace20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 >> 172.21.15.61:0/847535509 conn(0x7fc33c067980 msgr2=0x7fc33c0b5230 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:41.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 shutdown_connections 2022-07-20T03:16:41.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.055+0000 7fc3412a2700 1 -- 172.21.15.61:0/847535509 wait complete. 2022-07-20T03:16:41.069 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:16:41.073 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:41.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.171+0000 7fa18359e700 1 Processor -- start 2022-07-20T03:16:41.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.171+0000 7fa18359e700 1 -- start start 2022-07-20T03:16:41.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b1200 0x7fa1840b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa18405bd90 0x7fa184142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1840aea30 con 0x7fa1840b5650 2022-07-20T03:16:41.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa184142970 con 0x7fa18405bd90 2022-07-20T03:16:41.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840aec60 con 0x7fa1840b1200 2022-07-20T03:16:41.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18259c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa1840b5650 0x7fa18405c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47020/0 (socket says 172.21.15.61:47020) 2022-07-20T03:16:41.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18259c700 1 -- 172.21.15.61:0/2005249912 learned_addr learned my addr 172.21.15.61:0/2005249912 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:41.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa182d9d700 1 --2- 172.21.15.61:0/2005249912 >> v2:172.21.15.61:3300/0 conn(0x7fa18405bd90 0x7fa184142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18159a700 1 -- 172.21.15.61:0/2005249912 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955746550 0 0) 0x7fa1840aea30 con 0x7fa1840b5650 2022-07-20T03:16:41.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18159a700 1 -- 172.21.15.61:0/2005249912 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa164003260 con 0x7fa1840b5650 2022-07-20T03:16:41.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa181d9b700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b1200 0x7fa1840b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.200 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa182d9d700 1 -- 172.21.15.61:0/2005249912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b1200 msgr2=0x7fa1840b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:41.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18159a700 1 -- 172.21.15.61:0/2005249912 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 671201440 0 0) 0x7fa164003260 con 0x7fa1840b5650 2022-07-20T03:16:41.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa182d9d700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b1200 0x7fa1840b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa182d9d700 1 -- 172.21.15.61:0/2005249912 >> v1:172.21.15.61:6789/0 conn(0x7fa1840b5650 legacy=0x7fa18405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa182d9d700 1 -- 172.21.15.61:0/2005249912 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1840af180 con 0x7fa18405bd90 2022-07-20T03:16:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa181d9b700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b1200 0x7fa1840b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:41.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa182d9d700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa18405bd90 0x7fa184142d60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7fa17c001b80 tx=0x7fa174006df0).ready entity=mon.0 client_cookie=bfcab6062bc0c34d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:41.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18159a700 1 -- 172.21.15.61:0/2005249912 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa17c0019e0 con 0x7fa18405bd90 2022-07-20T03:16:41.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18159a700 1 -- 172.21.15.61:0/2005249912 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa174000e90 con 0x7fa18405bd90 2022-07-20T03:16:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18159a700 1 -- 172.21.15.61:0/2005249912 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa17400d4a0 con 0x7fa18405bd90 2022-07-20T03:16:41.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- 172.21.15.61:0/2005249912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa18405bd90 msgr2=0x7fa184142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa18405bd90 0x7fa184142d60 secure :-1 s=READY pgs=1278 cs=0 l=1 rev1=1 rx=0x7fa17c001b80 tx=0x7fa174006df0).stop 2022-07-20T03:16:41.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- 172.21.15.61:0/2005249912 shutdown_connections 2022-07-20T03:16:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa18405bd90 0x7fa184142d60 unknown :-1 s=CLOSED pgs=1278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 --2- 172.21.15.61:0/2005249912 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b1200 0x7fa1840b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- 172.21.15.61:0/2005249912 >> 172.21.15.61:0/2005249912 conn(0x7fa1840634e0 msgr2=0x7fa1840b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- 172.21.15.61:0/2005249912 shutdown_connections 2022-07-20T03:16:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- 172.21.15.61:0/2005249912 wait complete. 2022-07-20T03:16:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 Processor -- start 2022-07-20T03:16:41.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.175+0000 7fa18359e700 1 -- start start 2022-07-20T03:16:41.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa18405bd90 0x7fa1840b00b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 0x7fa184065880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b5650 0x7fa184151c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840654f0 con 0x7fa1840b1200 2022-07-20T03:16:41.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa184155390 con 0x7fa18405bd90 2022-07-20T03:16:41.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1840651e0 con 0x7fa1840b5650 2022-07-20T03:16:41.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 0x7fa184065880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 0x7fa184065880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50430/0 (socket says 172.21.15.61:50430) 2022-07-20T03:16:41.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 -- 172.21.15.61:0/2849949995 learned_addr learned my addr 172.21.15.61:0/2849949995 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:41.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa182d9d700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b5650 0x7fa184151c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18259c700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa18405bd90 0x7fa1840b00b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 -- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b5650 msgr2=0x7fa184151c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b5650 0x7fa184151c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 -- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa18405bd90 msgr2=0x7fa1840b00b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa18405bd90 0x7fa1840b00b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 -- 172.21.15.61:0/2849949995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa17400b180 con 0x7fa1840b1200 2022-07-20T03:16:41.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa181d9b700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 0x7fa184065880 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7fa16c0068e0 tx=0x7fa16c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:41.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16c011780 con 0x7fa1840b1200 2022-07-20T03:16:41.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa16c0118e0 con 0x7fa1840b1200 2022-07-20T03:16:41.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa184152130 con 0x7fa1840b1200 2022-07-20T03:16:41.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa16c011c40 con 0x7fa1840b1200 2022-07-20T03:16:41.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa18413ed90 con 0x7fa1840b1200 2022-07-20T03:16:41.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.179+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1840574e0 con 0x7fa1840b1200 2022-07-20T03:16:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.183+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa16c012420 con 0x7fa1840b1200 2022-07-20T03:16:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.183+0000 7fa17b7fe700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa168053a40 0x7fa168055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.183+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa16c06bd80 con 0x7fa1840b1200 2022-07-20T03:16:41.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.183+0000 7fa18259c700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa168053a40 0x7fa168055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.183+0000 7fa18259c700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa168053a40 0x7fa168055f00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fa17c00d000 tx=0x7fa17c000f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:41.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.187+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa16c042c80 con 0x7fa1840b1200 2022-07-20T03:16:41.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.351+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa184067ea0 con 0x7fa1840b1200 2022-07-20T03:16:41.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.351+0000 7fa17b7fe700 1 -- 172.21.15.61:0/2849949995 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fa16c06d090 con 0x7fa1840b1200 2022-07-20T03:16:41.359 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:41.360 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:16:39.468063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1435464119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1435464119},{"type":"v1","addr":"172.21.15.61:6833","nonce":1435464119}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:16:41.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa168053a40 msgr2=0x7fa168055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa168053a40 0x7fa168055f00 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7fa17c00d000 tx=0x7fa17c000f40).stop 2022-07-20T03:16:41.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 msgr2=0x7fa184065880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 0x7fa184065880 secure :-1 s=READY pgs=1279 cs=0 l=1 rev1=1 rx=0x7fa16c0068e0 tx=0x7fa16c008040).stop 2022-07-20T03:16:41.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 shutdown_connections 2022-07-20T03:16:41.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa168053a40 0x7fa168055f00 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1840b5650 0x7fa184151c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1840b1200 0x7fa184065880 unknown :-1 s=CLOSED pgs=1279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 --2- 172.21.15.61:0/2849949995 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa18405bd90 0x7fa1840b00b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 >> 172.21.15.61:0/2849949995 conn(0x7fa1840634e0 msgr2=0x7fa1840aded0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:41.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 shutdown_connections 2022-07-20T03:16:41.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.355+0000 7fa18359e700 1 -- 172.21.15.61:0/2849949995 wait complete. 2022-07-20T03:16:41.365 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-20T03:16:41.573 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:16:38.464416+0000', 'modified': '2022-07-20T03:16:39.468063+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1435464119', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1435464119}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1435464119}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:41.574 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:41.576 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:16:41.700 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8091,"want_state":"up:active","state":"up:active","rank_uptime":3.2191776129999998,"mdsmap_epoch":72,"osdmap_epoch":130,"osdmap_epoch_barrier":130,"uptime":8.1873312170000005} 2022-07-20T03:16:41.701 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8091, "mdsmap_epoch": 72, "osdmap_epoch": 130, "osdmap_epoch_barrier": 130, "rank_uptime": 3.219177613, "state": "up:active", "uptime": 8.187331217, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:41.707 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:41.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 Processor -- start 2022-07-20T03:16:41.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 -- start start 2022-07-20T03:16:41.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c05c3f0 0x7fb21c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb21c0acef0 0x7fb21c146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb21c05d480 con 0x7fb21c0b1e00 2022-07-20T03:16:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb21c0b21e0 con 0x7fb21c0acef0 2022-07-20T03:16:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb220cea700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c05cf40 con 0x7fb21c05c3f0 2022-07-20T03:16:41.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21a59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb21c0b1e00 0x7fb21c0ac7e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47034/0 (socket says 172.21.15.61:47034) 2022-07-20T03:16:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21a59c700 1 -- 172.21.15.61:0/2638368214 learned_addr learned my addr 172.21.15.61:0/2638368214 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb21c0acef0 0x7fb21c146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb219d9b700 1 --2- 172.21.15.61:0/2638368214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c05c3f0 0x7fb21c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21959a700 1 -- 172.21.15.61:0/2638368214 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3602751966 0 0) 0x7fb21c05d480 con 0x7fb21c0b1e00 2022-07-20T03:16:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21959a700 1 -- 172.21.15.61:0/2638368214 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb1fc003260 con 0x7fb21c0b1e00 2022-07-20T03:16:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21ad9d700 1 -- 172.21.15.61:0/2638368214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c05c3f0 msgr2=0x7fb21c14a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21ad9d700 1 --2- 172.21.15.61:0/2638368214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c05c3f0 0x7fb21c14a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21ad9d700 1 -- 172.21.15.61:0/2638368214 >> v1:172.21.15.61:6789/0 conn(0x7fb21c0b1e00 legacy=0x7fb21c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21ad9d700 1 -- 172.21.15.61:0/2638368214 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb21c05d1e0 con 0x7fb21c0acef0 2022-07-20T03:16:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21ad9d700 1 --2- 172.21.15.61:0/2638368214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c146410 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7fb20c002f70 tx=0x7fb20c006df0).ready entity=mon.0 client_cookie=618fb9cabfa0005a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21959a700 1 -- 172.21.15.61:0/2638368214 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20c00d4a0 con 0x7fb21c0acef0 2022-07-20T03:16:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21959a700 1 -- 172.21.15.61:0/2638368214 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb20c000e90 con 0x7fb21c0acef0 2022-07-20T03:16:41.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.811+0000 7fb21959a700 1 -- 172.21.15.61:0/2638368214 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20c00d7f0 con 0x7fb21c0acef0 2022-07-20T03:16:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- 172.21.15.61:0/2638368214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 msgr2=0x7fb21c146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 --2- 172.21.15.61:0/2638368214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c146410 secure :-1 s=READY pgs=1281 cs=0 l=1 rev1=1 rx=0x7fb20c002f70 tx=0x7fb20c006df0).stop 2022-07-20T03:16:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- 172.21.15.61:0/2638368214 shutdown_connections 2022-07-20T03:16:41.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 --2- 172.21.15.61:0/2638368214 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c146410 unknown :-1 s=CLOSED pgs=1281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 --2- 172.21.15.61:0/2638368214 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c05c3f0 0x7fb21c14a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- 172.21.15.61:0/2638368214 >> 172.21.15.61:0/2638368214 conn(0x7fb21c0611c0 msgr2=0x7fb21c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- 172.21.15.61:0/2638368214 shutdown_connections 2022-07-20T03:16:41.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- 172.21.15.61:0/2638368214 wait complete. 2022-07-20T03:16:41.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 Processor -- start 2022-07-20T03:16:41.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- start start 2022-07-20T03:16:41.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c1474b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb21c0b1e00 0x7fb21c09ebf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c0a0510 0x7fb21c0a09c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c09f800 con 0x7fb21c0acef0 2022-07-20T03:16:41.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c0a0eb0 con 0x7fb21c0b1e00 2022-07-20T03:16:41.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb220cea700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb21c149c80 con 0x7fb21c0a0510 2022-07-20T03:16:41.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb21a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c1474b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb21a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c1474b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50440/0 (socket says 172.21.15.61:50440) 2022-07-20T03:16:41.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb21a59c700 1 -- 172.21.15.61:0/3978323319 learned_addr learned my addr 172.21.15.61:0/3978323319 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:41.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb21ad9d700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c0a0510 0x7fb21c0a09c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.815+0000 7fb219d9b700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb21c0b1e00 0x7fb21c09ebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb21a59c700 1 -- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c0a0510 msgr2=0x7fb21c0a09c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb21a59c700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c0a0510 0x7fb21c0a09c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb21a59c700 1 -- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb21c0b1e00 msgr2=0x7fb21c09ebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb21a59c700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb21c0b1e00 0x7fb21c09ebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb21a59c700 1 -- 172.21.15.61:0/3978323319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb20c00b180 con 0x7fb21c0acef0 2022-07-20T03:16:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb21a59c700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c1474b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7fb204001110 tx=0x7fb204007ad0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb20400ce80 con 0x7fb21c0acef0 2022-07-20T03:16:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2040070d0 con 0x7fb21c0acef0 2022-07-20T03:16:41.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb21c09f0e0 con 0x7fb21c0acef0 2022-07-20T03:16:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb204007430 con 0x7fb21c0acef0 2022-07-20T03:16:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb21c09e4b0 con 0x7fb21c0acef0 2022-07-20T03:16:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb21c149aa0 con 0x7fb21c0acef0 2022-07-20T03:16:41.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.819+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb204010070 con 0x7fb21c0acef0 2022-07-20T03:16:41.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.823+0000 7fb2137fe700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb200053a60 0x7fb200055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:41.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.823+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb20406b320 con 0x7fb21c0acef0 2022-07-20T03:16:41.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.823+0000 7fb219d9b700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb200053a60 0x7fb200055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.823+0000 7fb219d9b700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb200053a60 0x7fb200055f20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fb21400b000 tx=0x7fb2140091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:41.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.827+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb204068e90 con 0x7fb21c0acef0 2022-07-20T03:16:41.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.955+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb21c14aae0 con 0x7fb21c0acef0 2022-07-20T03:16:41.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb2137fe700 1 -- 172.21.15.61:0/3978323319 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fb204047020 con 0x7fb21c0acef0 2022-07-20T03:16:41.963 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:41.964 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":130,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:39.466964+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:16:36.305597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T03:16:37.477547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:41.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb200053a60 msgr2=0x7fb200055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb200053a60 0x7fb200055f20 secure :-1 s=READY pgs=618 cs=0 l=1 rev1=1 rx=0x7fb21400b000 tx=0x7fb2140091d0).stop 2022-07-20T03:16:41.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 msgr2=0x7fb21c1474b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:41.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c1474b0 secure :-1 s=READY pgs=1282 cs=0 l=1 rev1=1 rx=0x7fb204001110 tx=0x7fb204007ad0).stop 2022-07-20T03:16:41.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 shutdown_connections 2022-07-20T03:16:41.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb200053a60 0x7fb200055f20 unknown :-1 s=CLOSED pgs=618 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb21c0b1e00 0x7fb21c09ebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb21c0a0510 0x7fb21c0a09c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 --2- 172.21.15.61:0/3978323319 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb21c0acef0 0x7fb21c1474b0 unknown :-1 s=CLOSED pgs=1282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:41.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 >> 172.21.15.61:0/3978323319 conn(0x7fb21c05c3f0 msgr2=0x7fb21c069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:41.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 shutdown_connections 2022-07-20T03:16:41.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:41.959+0000 7fb220cea700 1 -- 172.21.15.61:0/3978323319 wait complete. 2022-07-20T03:16:41.984 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:16:36.305597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:16:37.477547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:16:41.985 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:16:42.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.091+0000 7f8b47723700 1 Processor -- start 2022-07-20T03:16:42.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.091+0000 7f8b47723700 1 -- start start 2022-07-20T03:16:42.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8b400ab180 0x7f8b40061ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 0x7f8b40142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8b4005f840 con 0x7f8b400b34d0 2022-07-20T03:16:42.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8b40142b90 con 0x7f8b400ab180 2022-07-20T03:16:42.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b4005fa70 con 0x7f8b40142720 2022-07-20T03:16:42.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 0x7f8b40142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 0x7f8b40142fd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:37122/0 (socket says 172.21.15.61:37122) 2022-07-20T03:16:42.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 -- 172.21.15.61:0/1295511846 learned_addr learned my addr 172.21.15.61:0/1295511846 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b44cbe700 1 --2- 172.21.15.61:0/1295511846 >> v2:172.21.15.117:3300/0 conn(0x7f8b400ab180 0x7f8b40061ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b454bf700 1 --1- 172.21.15.61:0/1295511846 >> v1:172.21.15.117:6789/0 conn(0x7f8b400b34d0 0x7f8b40142010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:55550/0 (socket says 172.21.15.61:55550) 2022-07-20T03:16:42.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 -- 172.21.15.61:0/1295511846 >> v2:172.21.15.117:3300/0 conn(0x7f8b400ab180 msgr2=0x7f8b40061ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:42.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 --2- 172.21.15.61:0/1295511846 >> v2:172.21.15.117:3300/0 conn(0x7f8b400ab180 0x7f8b40061ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 -- 172.21.15.61:0/1295511846 >> v1:172.21.15.117:6789/0 conn(0x7f8b400b34d0 legacy=0x7f8b40142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 -- 172.21.15.61:0/1295511846 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b4005fd10 con 0x7f8b40142720 2022-07-20T03:16:42.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b45cc0700 1 --2- 172.21.15.61:0/1295511846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 0x7f8b40142fd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8b40060870 tx=0x7f8b38006eb0).ready entity=mon.2 client_cookie=9fca06a211cd81ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b3ffff700 1 -- 172.21.15.61:0/1295511846 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3800d5c0 con 0x7f8b40142720 2022-07-20T03:16:42.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b3ffff700 1 -- 172.21.15.61:0/1295511846 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38000e90 con 0x7f8b40142720 2022-07-20T03:16:42.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b3ffff700 1 -- 172.21.15.61:0/1295511846 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b3800d950 con 0x7f8b40142720 2022-07-20T03:16:42.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- 172.21.15.61:0/1295511846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 msgr2=0x7f8b40142fd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 --2- 172.21.15.61:0/1295511846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 0x7f8b40142fd0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f8b40060870 tx=0x7f8b38006eb0).stop 2022-07-20T03:16:42.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- 172.21.15.61:0/1295511846 shutdown_connections 2022-07-20T03:16:42.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 --2- 172.21.15.61:0/1295511846 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b40142720 0x7f8b40142fd0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 --2- 172.21.15.61:0/1295511846 >> v2:172.21.15.117:3300/0 conn(0x7f8b400ab180 0x7f8b40061ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- 172.21.15.61:0/1295511846 >> 172.21.15.61:0/1295511846 conn(0x7f8b400b6c50 msgr2=0x7f8b400b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- 172.21.15.61:0/1295511846 shutdown_connections 2022-07-20T03:16:42.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.095+0000 7f8b47723700 1 -- 172.21.15.61:0/1295511846 wait complete. 2022-07-20T03:16:42.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 Processor -- start 2022-07-20T03:16:42.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 -- start start 2022-07-20T03:16:42.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b400ab180 0x7f8b400b22d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b400b34d0 0x7f8b40132f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 0x7f8b40133410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b40155e60 con 0x7f8b40142720 2022-07-20T03:16:42.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8b401455e0 con 0x7f8b400ab180 2022-07-20T03:16:42.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8b400ad740 con 0x7f8b400b34d0 2022-07-20T03:16:42.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 0x7f8b40133410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 0x7f8b40133410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50458/0 (socket says 172.21.15.61:50458) 2022-07-20T03:16:42.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 -- 172.21.15.61:0/581281603 learned_addr learned my addr 172.21.15.61:0/581281603 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b454bf700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b400ab180 0x7f8b400b22d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b44cbe700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b400b34d0 0x7f8b40132f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 -- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b400b34d0 msgr2=0x7f8b40132f20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b400b34d0 0x7f8b40132f20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 -- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b400ab180 msgr2=0x7f8b400b22d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b400ab180 0x7f8b400b22d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 -- 172.21.15.61:0/581281603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8b3800b1e0 con 0x7f8b40142720 2022-07-20T03:16:42.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b45cc0700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 0x7f8b40133410 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f8b40060870 tx=0x7f8b38015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b38021070 con 0x7f8b40142720 2022-07-20T03:16:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8b38006b40 con 0x7f8b40142720 2022-07-20T03:16:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8b40145fe0 con 0x7f8b40142720 2022-07-20T03:16:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8b38011e20 con 0x7f8b40142720 2022-07-20T03:16:42.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.099+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8b4009c030 con 0x7f8b40142720 2022-07-20T03:16:42.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.103+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8b3800f070 con 0x7f8b40142720 2022-07-20T03:16:42.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.103+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8b4005ffb0 con 0x7f8b40142720 2022-07-20T03:16:42.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.103+0000 7f8b3e7fc700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b20053a40 0x7f8b20055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.103+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8b3806d2a0 con 0x7f8b40142720 2022-07-20T03:16:42.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.103+0000 7f8b454bf700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b20053a40 0x7f8b20055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.103+0000 7f8b454bf700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b20053a40 0x7f8b20055f00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f8b3400d000 tx=0x7f8b34007ec0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.111+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8b38044200 con 0x7f8b40142720 2022-07-20T03:16:42.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.231+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8b400574e0 con 0x7f8b40142720 2022-07-20T03:16:42.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.231+0000 7f8b3e7fc700 1 -- 172.21.15.61:0/581281603 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v130) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8b38047820 con 0x7f8b40142720 2022-07-20T03:16:42.240 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:16:42.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b20053a40 msgr2=0x7f8b20055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b20053a40 0x7f8b20055f00 secure :-1 s=READY pgs=619 cs=0 l=1 rev1=1 rx=0x7f8b3400d000 tx=0x7f8b34007ec0).stop 2022-07-20T03:16:42.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 msgr2=0x7f8b40133410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 0x7f8b40133410 secure :-1 s=READY pgs=1283 cs=0 l=1 rev1=1 rx=0x7f8b40060870 tx=0x7f8b38015000).stop 2022-07-20T03:16:42.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 shutdown_connections 2022-07-20T03:16:42.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8b20053a40 0x7f8b20055f00 unknown :-1 s=CLOSED pgs=619 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8b40142720 0x7f8b40133410 unknown :-1 s=CLOSED pgs=1283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8b400b34d0 0x7f8b40132f20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 --2- 172.21.15.61:0/581281603 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8b400ab180 0x7f8b400b22d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 >> 172.21.15.61:0/581281603 conn(0x7f8b400b6c50 msgr2=0x7f8b40061020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 shutdown_connections 2022-07-20T03:16:42.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.235+0000 7f8b47723700 1 -- 172.21.15.61:0/581281603 wait complete. 2022-07-20T03:16:42.252 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:16:42.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.355+0000 7f3a0dadf700 1 Processor -- start 2022-07-20T03:16:42.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.355+0000 7f3a0dadf700 1 -- start start 2022-07-20T03:16:42.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b6880 0x7f3a08143590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3a08145f90 0x7f3a080b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3a080b6c60 con 0x7f3a080b4220 2022-07-20T03:16:42.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3a081429c0 con 0x7f3a08145f90 2022-07-20T03:16:42.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08142bb0 con 0x7f3a080b6880 2022-07-20T03:16:42.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0cadd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b6880 0x7f3a08143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3a08145f90 0x7f3a080b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3a08145f90 0x7f3a080b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:60142/0 (socket says 172.21.15.61:60142) 2022-07-20T03:16:42.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 -- 172.21.15.61:0/1119310618 learned_addr learned my addr 172.21.15.61:0/1119310618 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 -- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b6880 msgr2=0x7f3a08143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:42.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 --2- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b6880 0x7f3a08143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 -- 172.21.15.61:0/1119310618 >> v1:172.21.15.117:6789/0 conn(0x7f3a080b4220 legacy=0x7f3a0813fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 -- 172.21.15.61:0/1119310618 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3a08142e50 con 0x7f3a08145f90 2022-07-20T03:16:42.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a07fff700 1 --2- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a080b3d30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f39f8002f70 tx=0x7f39f8006eb0).ready entity=mon.1 client_cookie=6607bc42f3946545 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a077fe700 1 -- 172.21.15.61:0/1119310618 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39f800d570 con 0x7f3a08145f90 2022-07-20T03:16:42.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a077fe700 1 -- 172.21.15.61:0/1119310618 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f8000e90 con 0x7f3a08145f90 2022-07-20T03:16:42.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a077fe700 1 -- 172.21.15.61:0/1119310618 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39f800d900 con 0x7f3a08145f90 2022-07-20T03:16:42.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 -- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 msgr2=0x7f3a080b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.359+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a080b3d30 secure :-1 s=READY pgs=586 cs=0 l=1 rev1=1 rx=0x7f39f8002f70 tx=0x7f39f8006eb0).stop 2022-07-20T03:16:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- 172.21.15.61:0/1119310618 shutdown_connections 2022-07-20T03:16:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a080b3d30 unknown :-1 s=CLOSED pgs=586 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/1119310618 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b6880 0x7f3a08143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- 172.21.15.61:0/1119310618 >> 172.21.15.61:0/1119310618 conn(0x7f3a080b9e60 msgr2=0x7f3a080b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- 172.21.15.61:0/1119310618 shutdown_connections 2022-07-20T03:16:42.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- 172.21.15.61:0/1119310618 wait complete. 2022-07-20T03:16:42.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 Processor -- start 2022-07-20T03:16:42.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- start start 2022-07-20T03:16:42.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b4220 0x7f3a0813c650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a080b6880 0x7f3a0813cb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a08138b40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08152e50 con 0x7f3a080b6880 2022-07-20T03:16:42.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3a081530f0 con 0x7f3a08145f90 2022-07-20T03:16:42.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0dadf700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3a08153390 con 0x7f3a080b4220 2022-07-20T03:16:42.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a080b6880 0x7f3a0813cb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0cadd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b4220 0x7f3a0813c650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a080b6880 0x7f3a0813cb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50470/0 (socket says 172.21.15.61:50470) 2022-07-20T03:16:42.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a07fff700 1 -- 172.21.15.61:0/3375280046 learned_addr learned my addr 172.21.15.61:0/3375280046 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.363+0000 7f3a0d2de700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a08138b40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0cadd700 1 -- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 msgr2=0x7f3a08138b40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0cadd700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a08138b40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0cadd700 1 -- 172.21.15.61:0/3375280046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a080b6880 msgr2=0x7f3a0813cb40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0cadd700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a080b6880 0x7f3a0813cb40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0cadd700 1 -- 172.21.15.61:0/3375280046 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39f800b1e0 con 0x7f3a080b4220 2022-07-20T03:16:42.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0cadd700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b4220 0x7f3a0813c650 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f39f000c000 tx=0x7f39f0006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39f0010780 con 0x7f3a080b4220 2022-07-20T03:16:42.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f39f00108e0 con 0x7f3a080b4220 2022-07-20T03:16:42.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3a080ad170 con 0x7f3a080b4220 2022-07-20T03:16:42.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f39f0010c40 con 0x7f3a080b4220 2022-07-20T03:16:42.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3a081391d0 con 0x7f3a080b4220 2022-07-20T03:16:42.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3a081430f0 con 0x7f3a080b4220 2022-07-20T03:16:42.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f39f001b410 con 0x7f3a080b4220 2022-07-20T03:16:42.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a05ffb700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f39ec053a40 0x7f39ec055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f39f006c0d0 con 0x7f3a080b4220 2022-07-20T03:16:42.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.367+0000 7f3a07fff700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f39ec053a40 0x7f39ec055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.371+0000 7f3a07fff700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f39ec053a40 0x7f39ec055f00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f39f801c000 tx=0x7f39f8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.379+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f39f0042e40 con 0x7f3a080b4220 2022-07-20T03:16:42.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.511+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f3a080574e0 con 0x7f3a080b4220 2022-07-20T03:16:42.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.511+0000 7f3a05ffb700 1 -- 172.21.15.61:0/3375280046 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v130) v1 ==== 92+0+10 (secure 0 0 0) 0x7f39f0046460 con 0x7f3a080b4220 2022-07-20T03:16:42.516 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:42.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f39ec053a40 msgr2=0x7f39ec055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f39ec053a40 0x7f39ec055f00 secure :-1 s=READY pgs=620 cs=0 l=1 rev1=1 rx=0x7f39f801c000 tx=0x7f39f8015000).stop 2022-07-20T03:16:42.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b4220 msgr2=0x7f3a0813c650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b4220 0x7f3a0813c650 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f39f000c000 tx=0x7f39f0006df0).stop 2022-07-20T03:16:42.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 shutdown_connections 2022-07-20T03:16:42.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f39ec053a40 0x7f39ec055f00 unknown :-1 s=CLOSED pgs=620 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3a08145f90 0x7f3a08138b40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3a080b6880 0x7f3a0813cb40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 --2- 172.21.15.61:0/3375280046 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3a080b4220 0x7f3a0813c650 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 >> 172.21.15.61:0/3375280046 conn(0x7f3a080b9e60 msgr2=0x7f3a0805c7d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 shutdown_connections 2022-07-20T03:16:42.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.515+0000 7f3a0dadf700 1 -- 172.21.15.61:0/3375280046 wait complete. 2022-07-20T03:16:42.536 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:16:42.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 Processor -- start 2022-07-20T03:16:42.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 -- start start 2022-07-20T03:16:42.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9f8c09df40 0x7f9f8c145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c060ee0 0x7f9f8c09cba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9f8c0607e0 con 0x7f9f8c0b4d60 2022-07-20T03:16:42.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9f8c060d20 con 0x7f9f8c09df40 2022-07-20T03:16:42.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f91a72700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c060a80 con 0x7f9f8c060ee0 2022-07-20T03:16:42.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f83fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9f8c0b4d60 0x7f9f8c145b70 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47076/0 (socket says 172.21.15.61:47076) 2022-07-20T03:16:42.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f83fff700 1 -- 172.21.15.61:0/861558469 learned_addr learned my addr 172.21.15.61:0/861558469 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:42.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8b7fe700 1 --2- 172.21.15.61:0/861558469 >> v2:172.21.15.117:3300/0 conn(0x7f9f8c09df40 0x7f9f8c145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8bfff700 1 --2- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c060ee0 0x7f9f8c09cba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8affd700 1 -- 172.21.15.61:0/861558469 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2732554657 0 0) 0x7f9f8c0607e0 con 0x7f9f8c0b4d60 2022-07-20T03:16:42.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8affd700 1 -- 172.21.15.61:0/861558469 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9f6c003260 con 0x7f9f8c0b4d60 2022-07-20T03:16:42.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c060ee0 msgr2=0x7f9f8c09cba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:42.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8b7fe700 1 --2- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c060ee0 0x7f9f8c09cba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/861558469 >> v1:172.21.15.61:6789/0 conn(0x7f9f8c0b4d60 legacy=0x7f9f8c145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/861558469 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f8c09e320 con 0x7f9f8c09df40 2022-07-20T03:16:42.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8b7fe700 1 --2- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c09df40 0x7f9f8c145680 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f9f7c002f70 tx=0x7f9f7c006d90).ready entity=mon.1 client_cookie=f5c1af6026be9ff8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8affd700 1 -- 172.21.15.61:0/861558469 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7c00e590 con 0x7f9f8c09df40 2022-07-20T03:16:42.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8affd700 1 -- 172.21.15.61:0/861558469 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c000e90 con 0x7f9f8c09df40 2022-07-20T03:16:42.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.655+0000 7f9f8affd700 1 -- 172.21.15.61:0/861558469 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7c00e8e0 con 0x7f9f8c09df40 2022-07-20T03:16:42.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 -- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c09df40 msgr2=0x7f9f8c145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 --2- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c09df40 0x7f9f8c145680 secure :-1 s=READY pgs=587 cs=0 l=1 rev1=1 rx=0x7f9f7c002f70 tx=0x7f9f7c006d90).stop 2022-07-20T03:16:42.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 -- 172.21.15.61:0/861558469 shutdown_connections 2022-07-20T03:16:42.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 --2- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c060ee0 0x7f9f8c09cba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 --2- 172.21.15.61:0/861558469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c09df40 0x7f9f8c145680 unknown :-1 s=CLOSED pgs=587 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 -- 172.21.15.61:0/861558469 >> 172.21.15.61:0/861558469 conn(0x7f9f8c0b5f30 msgr2=0x7f9f8c0b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 -- 172.21.15.61:0/861558469 shutdown_connections 2022-07-20T03:16:42.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 -- 172.21.15.61:0/861558469 wait complete. 2022-07-20T03:16:42.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 Processor -- start 2022-07-20T03:16:42.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.659+0000 7f9f91a72700 1 -- start start 2022-07-20T03:16:42.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 0x7f9f8c145790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c09df40 0x7f9f8c090d10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c0b4d60 0x7f9f8c091200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b2cf0 con 0x7f9f8c060ee0 2022-07-20T03:16:42.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b1440 con 0x7f9f8c0b4d60 2022-07-20T03:16:42.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8c0b1630 con 0x7f9f8c09df40 2022-07-20T03:16:42.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 0x7f9f8c145790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 0x7f9f8c145790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50482/0 (socket says 172.21.15.61:50482) 2022-07-20T03:16:42.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/3823762006 learned_addr learned my addr 172.21.15.61:0/3823762006 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f83fff700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c09df40 0x7f9f8c090d10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8bfff700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c0b4d60 0x7f9f8c091200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c09df40 msgr2=0x7f9f8c090d10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c09df40 0x7f9f8c090d10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c0b4d60 msgr2=0x7f9f8c091200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c0b4d60 0x7f9f8c091200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 -- 172.21.15.61:0/3823762006 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7c00b120 con 0x7f9f8c060ee0 2022-07-20T03:16:42.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f8b7fe700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 0x7f9f8c145790 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f9f7c002f70 tx=0x7f9f7c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7c0116c0 con 0x7f9f8c060ee0 2022-07-20T03:16:42.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9f7c011820 con 0x7f9f8c060ee0 2022-07-20T03:16:42.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8c0b18d0 con 0x7f9f8c060ee0 2022-07-20T03:16:42.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9f7c011b80 con 0x7f9f8c060ee0 2022-07-20T03:16:42.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.663+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8c155620 con 0x7f9f8c060ee0 2022-07-20T03:16:42.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.667+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9f8c0574e0 con 0x7f9f8c060ee0 2022-07-20T03:16:42.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.667+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9f7c00f070 con 0x7f9f8c060ee0 2022-07-20T03:16:42.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.667+0000 7f9f897fa700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f700539b0 0x7f9f70055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.667+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9f7c06d190 con 0x7f9f8c060ee0 2022-07-20T03:16:42.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.667+0000 7f9f83fff700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f700539b0 0x7f9f70055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.667+0000 7f9f83fff700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f700539b0 0x7f9f70055e70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f9f7400d000 tx=0x7f9f74009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.675+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9f7c043fe0 con 0x7f9f8c060ee0 2022-07-20T03:16:42.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f9f8c061810 con 0x7f9f8c060ee0 2022-07-20T03:16:42.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f897fa700 1 -- 172.21.15.61:0/3823762006 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+10 (secure 0 0 0) 0x7f9f7c047600 con 0x7f9f8c060ee0 2022-07-20T03:16:42.808 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:42.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f700539b0 msgr2=0x7f9f70055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f700539b0 0x7f9f70055e70 secure :-1 s=READY pgs=621 cs=0 l=1 rev1=1 rx=0x7f9f7400d000 tx=0x7f9f74009040).stop 2022-07-20T03:16:42.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 msgr2=0x7f9f8c145790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 0x7f9f8c145790 secure :-1 s=READY pgs=1285 cs=0 l=1 rev1=1 rx=0x7f9f7c002f70 tx=0x7f9f7c014000).stop 2022-07-20T03:16:42.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 shutdown_connections 2022-07-20T03:16:42.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9f700539b0 0x7f9f70055e70 unknown :-1 s=CLOSED pgs=621 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9f8c0b4d60 0x7f9f8c091200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9f8c09df40 0x7f9f8c090d10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 --2- 172.21.15.61:0/3823762006 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9f8c060ee0 0x7f9f8c145790 unknown :-1 s=CLOSED pgs=1285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 >> 172.21.15.61:0/3823762006 conn(0x7f9f8c0b5f30 msgr2=0x7f9f8c05bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 shutdown_connections 2022-07-20T03:16:42.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.803+0000 7f9f91a72700 1 -- 172.21.15.61:0/3823762006 wait complete. 2022-07-20T03:16:42.821 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:16:42.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 Processor -- start 2022-07-20T03:16:42.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 -- start start 2022-07-20T03:16:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fae3c0b2ec0 0x7fae3c0b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c1461c0 0x7fae3c146630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fae3c0b3f10 con 0x7fae3c05d160 2022-07-20T03:16:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fae3c0b4ea0 con 0x7fae3c0b2ec0 2022-07-20T03:16:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0b3790 con 0x7fae3c1461c0 2022-07-20T03:16:42.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fae3c0b2ec0 0x7fae3c0b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fae3c0b2ec0 0x7fae3c0b32a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50490/0 (socket says 172.21.15.61:50490) 2022-07-20T03:16:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 -- 172.21.15.61:0/3450313246 learned_addr learned my addr 172.21.15.61:0/3450313246 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3b7fe700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c1461c0 0x7fae3c146630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 -- 172.21.15.61:0/3450313246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c1461c0 msgr2=0x7fae3c146630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c1461c0 0x7fae3c146630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 -- 172.21.15.61:0/3450313246 >> v1:172.21.15.117:6789/0 conn(0x7fae3c05d160 legacy=0x7fae3c142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 -- 172.21.15.61:0/3450313246 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae3c0b2620 con 0x7fae3c0b2ec0 2022-07-20T03:16:42.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3b7fe700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c1461c0 0x7fae3c146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:42.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3affd700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c0b32a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7fae2c002f70 tx=0x7fae2c006eb0).ready entity=mon.0 client_cookie=3d8ea1de48106842 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3a7fc700 1 -- 172.21.15.61:0/3450313246 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae2c00d5c0 con 0x7fae3c0b2ec0 2022-07-20T03:16:42.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3a7fc700 1 -- 172.21.15.61:0/3450313246 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae2c000e90 con 0x7fae3c0b2ec0 2022-07-20T03:16:42.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.935+0000 7fae3a7fc700 1 -- 172.21.15.61:0/3450313246 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae2c00d950 con 0x7fae3c0b2ec0 2022-07-20T03:16:42.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- 172.21.15.61:0/3450313246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c0b2ec0 msgr2=0x7fae3c0b32a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c0b32a0 secure :-1 s=READY pgs=1286 cs=0 l=1 rev1=1 rx=0x7fae2c002f70 tx=0x7fae2c006eb0).stop 2022-07-20T03:16:42.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- 172.21.15.61:0/3450313246 shutdown_connections 2022-07-20T03:16:42.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c1461c0 0x7fae3c146630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3450313246 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c0b32a0 unknown :-1 s=CLOSED pgs=1286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- 172.21.15.61:0/3450313246 >> 172.21.15.61:0/3450313246 conn(0x7fae3c066e20 msgr2=0x7fae3c067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:42.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- 172.21.15.61:0/3450313246 shutdown_connections 2022-07-20T03:16:42.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- 172.21.15.61:0/3450313246 wait complete. 2022-07-20T03:16:42.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 Processor -- start 2022-07-20T03:16:42.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- start start 2022-07-20T03:16:42.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c05d160 0x7fae3c0ace10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c145390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 0x7fae3c155c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0af4b0 con 0x7fae3c1461c0 2022-07-20T03:16:42.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0af970 con 0x7fae3c0b2ec0 2022-07-20T03:16:42.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.939+0000 7fae3bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fae3c0af6d0 con 0x7fae3c05d160 2022-07-20T03:16:42.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 0x7fae3c155c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 0x7fae3c155c30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50498/0 (socket says 172.21.15.61:50498) 2022-07-20T03:16:42.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 -- 172.21.15.61:0/3448777992 learned_addr learned my addr 172.21.15.61:0/3448777992 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:42.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3affd700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c05d160 0x7fae3c0ace10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae33fff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c145390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 -- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c05d160 msgr2=0x7fae3c0ace10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c05d160 0x7fae3c0ace10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 -- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae3c0b2ec0 msgr2=0x7fae3c145390 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:42.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c145390 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:42.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 -- 172.21.15.61:0/3448777992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae2c00b1e0 con 0x7fae3c1461c0 2022-07-20T03:16:42.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3b7fe700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 0x7fae3c155c30 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fae3c0af5f0 tx=0x7fae34008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3400f710 con 0x7fae3c1461c0 2022-07-20T03:16:42.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae3400f870 con 0x7fae3c1461c0 2022-07-20T03:16:42.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae3400fbd0 con 0x7fae3c1461c0 2022-07-20T03:16:42.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae3c0b8340 con 0x7fae3c1461c0 2022-07-20T03:16:42.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae3c0adf90 con 0x7fae3c1461c0 2022-07-20T03:16:42.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae3c0b84d0 con 0x7fae3c1461c0 2022-07-20T03:16:42.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.943+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fae3401a030 con 0x7fae3c1461c0 2022-07-20T03:16:42.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.947+0000 7fae38ff9700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae20053ad0 0x7fae20055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:42.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.947+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fae3406bdf0 con 0x7fae3c1461c0 2022-07-20T03:16:42.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.947+0000 7fae3affd700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae20053ad0 0x7fae20055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:42.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.947+0000 7fae3affd700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae20053ad0 0x7fae20055f90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fae2c01c000 tx=0x7fae2c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:42.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:42.955+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fae34042d50 con 0x7fae3c1461c0 2022-07-20T03:16:43.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.079+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fae3c0b28c0 con 0x7fae3c1461c0 2022-07-20T03:16:43.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae38ff9700 1 -- 172.21.15.61:0/3448777992 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+10 (secure 0 0 0) 0x7fae34046370 con 0x7fae3c1461c0 2022-07-20T03:16:43.086 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:43.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae20053ad0 msgr2=0x7fae20055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae20053ad0 0x7fae20055f90 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7fae2c01c000 tx=0x7fae2c015000).stop 2022-07-20T03:16:43.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 msgr2=0x7fae3c155c30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 0x7fae3c155c30 secure :-1 s=READY pgs=1287 cs=0 l=1 rev1=1 rx=0x7fae3c0af5f0 tx=0x7fae34008040).stop 2022-07-20T03:16:43.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 shutdown_connections 2022-07-20T03:16:43.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae20053ad0 0x7fae20055f90 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fae3c0b2ec0 0x7fae3c145390 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fae3c1461c0 0x7fae3c155c30 unknown :-1 s=CLOSED pgs=1287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 --2- 172.21.15.61:0/3448777992 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fae3c05d160 0x7fae3c0ace10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 >> 172.21.15.61:0/3448777992 conn(0x7fae3c066e20 msgr2=0x7fae3c0b5ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:43.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 shutdown_connections 2022-07-20T03:16:43.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.083+0000 7fae3bfff700 1 -- 172.21.15.61:0/3448777992 wait complete. 2022-07-20T03:16:43.100 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:16:43.101 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:43.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 Processor -- start 2022-07-20T03:16:43.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 -- start start 2022-07-20T03:16:43.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b61d0 0x7fcf080ae960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:43.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcf080b1000 0x7fcf0814a880 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:43.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf080b1580 con 0x7fcf080b01a0 2022-07-20T03:16:43.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf080b65b0 con 0x7fcf080b1000 2022-07-20T03:16:43.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.203+0000 7fcf0e680700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf08060a10 con 0x7fcf080b61d0 2022-07-20T03:16:43.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0ce7d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcf080b01a0 0x7fcf080b18e0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47104/0 (socket says 172.21.15.61:47104) 2022-07-20T03:16:43.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0ce7d700 1 -- 172.21.15.61:0/1321519890 learned_addr learned my addr 172.21.15.61:0/1321519890 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:43.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0de7f700 1 --2- 172.21.15.61:0/1321519890 >> v2:172.21.15.117:3300/0 conn(0x7fcf080b1000 0x7fcf0814a880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:43.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0d67e700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b61d0 0x7fcf080ae960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:43.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcefffff700 1 -- 172.21.15.61:0/1321519890 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 259930064 0 0) 0x7fcf080b1580 con 0x7fcf080b01a0 2022-07-20T03:16:43.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcefffff700 1 -- 172.21.15.61:0/1321519890 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fceec003260 con 0x7fcf080b01a0 2022-07-20T03:16:43.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0de7f700 1 -- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b61d0 msgr2=0x7fcf080ae960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:43.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0de7f700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b61d0 0x7fcf080ae960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0de7f700 1 -- 172.21.15.61:0/1321519890 >> v1:172.21.15.61:6789/0 conn(0x7fcf080b01a0 legacy=0x7fcf080b18e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0de7f700 1 -- 172.21.15.61:0/1321519890 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf08060cb0 con 0x7fcf080b1000 2022-07-20T03:16:43.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0d67e700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b61d0 0x7fcf080ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:43.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0de7f700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf0814a880 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fcf04002f70 tx=0x7fcf04006df0).ready entity=mon.1 client_cookie=3b989d6bd2ac85a6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:43.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcefffff700 1 -- 172.21.15.61:0/1321519890 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf0400d4a0 con 0x7fcf080b1000 2022-07-20T03:16:43.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcefffff700 1 -- 172.21.15.61:0/1321519890 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf04000e90 con 0x7fcf080b1000 2022-07-20T03:16:43.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcefffff700 1 -- 172.21.15.61:0/1321519890 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf0400d7f0 con 0x7fcf080b1000 2022-07-20T03:16:43.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 -- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 msgr2=0x7fcf0814a880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf0814a880 secure :-1 s=READY pgs=589 cs=0 l=1 rev1=1 rx=0x7fcf04002f70 tx=0x7fcf04006df0).stop 2022-07-20T03:16:43.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 -- 172.21.15.61:0/1321519890 shutdown_connections 2022-07-20T03:16:43.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf0814a880 unknown :-1 s=CLOSED pgs=589 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 --2- 172.21.15.61:0/1321519890 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b61d0 0x7fcf080ae960 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 -- 172.21.15.61:0/1321519890 >> 172.21.15.61:0/1321519890 conn(0x7fcf08063cf0 msgr2=0x7fcf080b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:43.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 -- 172.21.15.61:0/1321519890 shutdown_connections 2022-07-20T03:16:43.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 -- 172.21.15.61:0/1321519890 wait complete. 2022-07-20T03:16:43.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 Processor -- start 2022-07-20T03:16:43.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.207+0000 7fcf0e680700 1 -- start start 2022-07-20T03:16:43.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b01a0 0x7fcf08112600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:43.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf080ae2b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:43.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 0x7fcf08142a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:43.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0810ea50 con 0x7fcf080b61d0 2022-07-20T03:16:43.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080673c0 con 0x7fcf080b1000 2022-07-20T03:16:43.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf080ae7a0 con 0x7fcf080b01a0 2022-07-20T03:16:43.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 0x7fcf08142a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:43.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 0x7fcf08142a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50512/0 (socket says 172.21.15.61:50512) 2022-07-20T03:16:43.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 -- 172.21.15.61:0/3730099511 learned_addr learned my addr 172.21.15.61:0/3730099511 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:43.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0d67e700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b01a0 0x7fcf08112600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:43.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0ce7d700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf080ae2b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:43.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 -- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b01a0 msgr2=0x7fcf08112600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b01a0 0x7fcf08112600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 -- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 msgr2=0x7fcf080ae2b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf080ae2b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 -- 172.21.15.61:0/3730099511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf00009470 con 0x7fcf080b61d0 2022-07-20T03:16:43.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0de7f700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 0x7fcf08142a30 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7fcf04002f70 tx=0x7fcf04014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:43.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf04012930 con 0x7fcf080b61d0 2022-07-20T03:16:43.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf04012a90 con 0x7fcf080b61d0 2022-07-20T03:16:43.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0400b180 con 0x7fcf080b61d0 2022-07-20T03:16:43.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf04012df0 con 0x7fcf080b61d0 2022-07-20T03:16:43.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf080667c0 con 0x7fcf080b61d0 2022-07-20T03:16:43.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.211+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf080afa70 con 0x7fcf080b61d0 2022-07-20T03:16:43.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.215+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcf0401c070 con 0x7fcf080b61d0 2022-07-20T03:16:43.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.215+0000 7fcefe7fc700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef0053af0 0x7fcef0055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:43.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.215+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcf0406ca40 con 0x7fcf080b61d0 2022-07-20T03:16:43.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.215+0000 7fcf0d67e700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef0053af0 0x7fcef0055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:43.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.215+0000 7fcf0d67e700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef0053af0 0x7fcef0055fb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fcf0000b000 tx=0x7fcf000091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:43.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.223+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcf040439a0 con 0x7fcf080b61d0 2022-07-20T03:16:43.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.387+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fcf080574e0 con 0x7fcf080b61d0 2022-07-20T03:16:43.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.387+0000 7fcefe7fc700 1 -- 172.21.15.61:0/3730099511 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fcf04019070 con 0x7fcf080b61d0 2022-07-20T03:16:43.394 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:43.395 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:16:39.468063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1435464119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1435464119},{"type":"v1","addr":"172.21.15.61:6833","nonce":1435464119}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:16:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef0053af0 msgr2=0x7fcef0055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef0053af0 0x7fcef0055fb0 secure :-1 s=READY pgs=623 cs=0 l=1 rev1=1 rx=0x7fcf0000b000 tx=0x7fcf000091d0).stop 2022-07-20T03:16:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 msgr2=0x7fcf08142a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 0x7fcf08142a30 secure :-1 s=READY pgs=1289 cs=0 l=1 rev1=1 rx=0x7fcf04002f70 tx=0x7fcf04014000).stop 2022-07-20T03:16:43.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 shutdown_connections 2022-07-20T03:16:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef0053af0 0x7fcef0055fb0 unknown :-1 s=CLOSED pgs=623 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf080b1000 0x7fcf080ae2b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf080b61d0 0x7fcf08142a30 unknown :-1 s=CLOSED pgs=1289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 --2- 172.21.15.61:0/3730099511 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf080b01a0 0x7fcf08112600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 >> 172.21.15.61:0/3730099511 conn(0x7fcf08063cf0 msgr2=0x7fcf080b49e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:43.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 shutdown_connections 2022-07-20T03:16:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:43.391+0000 7fcf0e680700 1 -- 172.21.15.61:0/3730099511 wait complete. 2022-07-20T03:16:43.402 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-20T03:16:43.612 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:16:38.464416+0000', 'modified': '2022-07-20T03:16:39.468063+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1435464119', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1435464119}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1435464119}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:43.613 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:43.614 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:16:43.734 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8091,"want_state":"up:active","state":"up:active","rank_uptime":5.257589533,"mdsmap_epoch":72,"osdmap_epoch":130,"osdmap_epoch_barrier":130,"uptime":10.225743095} 2022-07-20T03:16:43.735 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8091, "mdsmap_epoch": 72, "osdmap_epoch": 130, "osdmap_epoch_barrier": 130, "rank_uptime": 5.257589533, "state": "up:active", "uptime": 10.225743095, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:43.735 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:16:43.736 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:16:43.736 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:16:43.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:43.737 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:43.746 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:16:43.763 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:43.763 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:43.821 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:43.822 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:43.823 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:43.823 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:16:43.828 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:16:43.829 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:43.831 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:16:43.849 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:43.849 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:43.863 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:43.864 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:43.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:43.865 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:16:43.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:16:43.896+0000 7f8645279080 -1 init, newargv = 0x5587ec39b340 newargc=9ceph-fuse[23445]: starting ceph client 2022-07-20T03:16:43.901 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr: 2022-07-20T03:16:43.910 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23445]: starting fuse 2022-07-20T03:16:45.011 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:16:45.031 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:45.031 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:45.041 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:45.042 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:45.043 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:45.044 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:16:45.050 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:16:45.051 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:16:45.052 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:45.053 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:16:45.053 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:16:45.054 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:16:45.055 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:16:45.055 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:16:45.056 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:45.056 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:16:45.057 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:16:45.057 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:16:45.058 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:45.058 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:16:45.059 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:16:45.059 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:16:45.060 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:16:45.060 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:45.061 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:16:45.061 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:16:45.062 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:16:45.062 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:16:45.063 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:16:45.063 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:45.064 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:16:45.064 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:16:45.140 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23445.asok 2022-07-20T03:16:45.146 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23445.asok 2022-07-20T03:16:45.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:45.147 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23445.asok status 2022-07-20T03:16:45.233 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:16:45.234 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:16:45.234 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:16:45.234 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:16:45.234 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:16:45.235 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:16:45.235 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:16:45.235 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23445", 2022-07-20T03:16:45.235 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:16:45.236 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:45.236 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:16:45.236 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:16:45.237 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 7028, 2022-07-20T03:16:45.237 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:16:45.237 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:16:45.237 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:16:45.238 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 7028 2022-07-20T03:16:45.238 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:45.238 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:16:45.239 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:16:45.239 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:16:45.239 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 4026656449 2022-07-20T03:16:45.239 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:16:45.240 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:45.240 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:16:45.240 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:16:45.240 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:16:45.240 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 4026656449 2022-07-20T03:16:45.241 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:16:45.241 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.7028 172.21.15.124:0/4026656449", 2022-07-20T03:16:45.241 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/4026656449", 2022-07-20T03:16:45.242 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:16:45.242 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 72, 2022-07-20T03:16:45.242 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 130, 2022-07-20T03:16:45.243 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:16:45.243 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:16:45.243 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:16:45.243 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:16:45.245 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:45.245 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:45.246 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:45.253 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:16:45.255 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:45.255 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:45.256 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:45.318 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.427+0000 7f2a43c95700 1 Processor -- start 2022-07-20T03:16:45.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.427+0000 7f2a43c95700 1 -- start start 2022-07-20T03:16:45.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0b21f0 0x7f2a3c0b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3c060ec0 con 0x7f2a3c1438b0 2022-07-20T03:16:45.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2a3c13fb20 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c143d40 con 0x7f2a3c0b21f0 2022-07-20T03:16:45.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a42232700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2a3c1438b0 0x7f2a3c13ef90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47120/0 (socket says 172.21.15.61:47120) 2022-07-20T03:16:45.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a42232700 1 -- 172.21.15.61:0/3291618980 learned_addr learned my addr 172.21.15.61:0/3291618980 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:45.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a41a31700 1 --2- 172.21.15.61:0/3291618980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0b21f0 0x7f2a3c0b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a40a2f700 1 -- 172.21.15.61:0/3291618980 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2241439970 0 0) 0x7f2a3c060ec0 con 0x7f2a3c1438b0 2022-07-20T03:16:45.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a40a2f700 1 -- 172.21.15.61:0/3291618980 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2a28003260 con 0x7f2a3c1438b0 2022-07-20T03:16:45.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a41a31700 1 -- 172.21.15.61:0/3291618980 >> v1:172.21.15.117:6789/0 conn(0x7f2a3c0ac8a0 legacy=0x7f2a3c140260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a41a31700 1 -- 172.21.15.61:0/3291618980 >> v1:172.21.15.61:6789/0 conn(0x7f2a3c1438b0 legacy=0x7f2a3c13ef90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a41a31700 1 -- 172.21.15.61:0/3291618980 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a3c13fdc0 con 0x7f2a3c0b21f0 2022-07-20T03:16:45.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a41a31700 1 --2- 172.21.15.61:0/3291618980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0b21f0 0x7f2a3c0b25d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2a3c0625f0 tx=0x7f2a340091d0).ready entity=mon.2 client_cookie=db5afda6f81d38c4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a40a2f700 1 -- 172.21.15.61:0/3291618980 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a3400f8a0 con 0x7f2a3c0b21f0 2022-07-20T03:16:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a40a2f700 1 -- 172.21.15.61:0/3291618980 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3400fa00 con 0x7f2a3c0b21f0 2022-07-20T03:16:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a40a2f700 1 -- 172.21.15.61:0/3291618980 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a3400fd60 con 0x7f2a3c0b21f0 2022-07-20T03:16:45.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- 172.21.15.61:0/3291618980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0b21f0 msgr2=0x7f2a3c0b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 --2- 172.21.15.61:0/3291618980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0b21f0 0x7f2a3c0b25d0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f2a3c0625f0 tx=0x7f2a340091d0).stop 2022-07-20T03:16:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- 172.21.15.61:0/3291618980 shutdown_connections 2022-07-20T03:16:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 --2- 172.21.15.61:0/3291618980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0b21f0 0x7f2a3c0b25d0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- 172.21.15.61:0/3291618980 >> 172.21.15.61:0/3291618980 conn(0x7f2a3c0b36b0 msgr2=0x7f2a3c0b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- 172.21.15.61:0/3291618980 shutdown_connections 2022-07-20T03:16:45.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.431+0000 7f2a43c95700 1 -- 172.21.15.61:0/3291618980 wait complete. 2022-07-20T03:16:45.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.435+0000 7f2a43c95700 1 Processor -- start 2022-07-20T03:16:45.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.435+0000 7f2a43c95700 1 -- start start 2022-07-20T03:16:45.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.435+0000 7f2a43c95700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 0x7f2a3c0994a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.435+0000 7f2a43c95700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0b21f0 0x7f2a3c14dbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.435+0000 7f2a43c95700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a3c1438b0 0x7f2a3c14e0d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.435+0000 7f2a43c95700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0ae930 con 0x7f2a3c1438b0 2022-07-20T03:16:45.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a43c95700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0aebd0 con 0x7f2a3c0b21f0 2022-07-20T03:16:45.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a43c95700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2a3c0aee70 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a42232700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a3c1438b0 0x7f2a3c14e0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 0x7f2a3c0994a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 0x7f2a3c0994a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:37196/0 (socket says 172.21.15.61:37196) 2022-07-20T03:16:45.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 -- 172.21.15.61:0/1319689023 learned_addr learned my addr 172.21.15.61:0/1319689023 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:45.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41230700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0b21f0 0x7f2a3c14dbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 -- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0b21f0 msgr2=0x7f2a3c14dbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0b21f0 0x7f2a3c14dbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 -- 172.21.15.61:0/1319689023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a3c1438b0 msgr2=0x7f2a3c14e0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a3c1438b0 0x7f2a3c14e0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 -- 172.21.15.61:0/1319689023 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2a34009470 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a41a31700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 0x7f2a3c0994a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f2a3c0625f0 tx=0x7f2a34018000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a34023630 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2a3401e040 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2a3c0ae500 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2a34023980 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2a3c0af4b0 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2a3c0574e0 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.439+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2a3402a070 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.443+0000 7f2a3affd700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2a1c053920 0x7f2a1c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.443+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f2a3406efe0 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.443+0000 7f2a41230700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2a1c053920 0x7f2a1c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.443+0000 7f2a41230700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2a1c053920 0x7f2a1c055de0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f2a3000d000 tx=0x7f2a30000bc0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.451+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2a34045e30 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.579+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f2a3c05c400 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.591 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.579+0000 7f2a3affd700 1 -- 172.21.15.61:0/1319689023 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f2a34049450 con 0x7f2a3c0ac8a0 2022-07-20T03:16:45.592 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:45.592 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":130,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:39.466964+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:16:36.305597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T03:16:37.477547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:45.593 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2a1c053920 msgr2=0x7f2a1c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2a1c053920 0x7f2a1c055de0 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f2a3000d000 tx=0x7f2a30000bc0).stop 2022-07-20T03:16:45.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 msgr2=0x7f2a3c0994a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.594 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 0x7f2a3c0994a0 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f2a3c0625f0 tx=0x7f2a34018000).stop 2022-07-20T03:16:45.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 shutdown_connections 2022-07-20T03:16:45.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2a1c053920 0x7f2a1c055de0 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2a3c0b21f0 0x7f2a3c14dbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2a3c1438b0 0x7f2a3c14e0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 --2- 172.21.15.61:0/1319689023 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2a3c0ac8a0 0x7f2a3c0994a0 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 >> 172.21.15.61:0/1319689023 conn(0x7f2a3c0b36b0 msgr2=0x7f2a3c143190 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:45.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 shutdown_connections 2022-07-20T03:16:45.598 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.583+0000 7f2a43c95700 1 -- 172.21.15.61:0/1319689023 wait complete. 2022-07-20T03:16:45.604 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:16:36.305597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:16:37.477547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:16:45.606 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:16:45.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 Processor -- start 2022-07-20T03:16:45.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 -- start start 2022-07-20T03:16:45.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08280b3ed0 0x7f0828143530 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f082813fcc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f08280b6680 con 0x7f08280b62a0 2022-07-20T03:16:45.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08280b4af0 con 0x7f08280b3ed0 2022-07-20T03:16:45.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082fda7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08281465c0 con 0x7f0828143a20 2022-07-20T03:16:45.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082d342700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08280b3ed0 0x7f0828143530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082d342700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08280b3ed0 0x7f0828143530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50534/0 (socket says 172.21.15.61:50534) 2022-07-20T03:16:45.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082d342700 1 -- 172.21.15.61:0/106720286 learned_addr learned my addr 172.21.15.61:0/106720286 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:45.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.707+0000 7f082e344700 1 --2- 172.21.15.61:0/106720286 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f082813fcc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082cb41700 1 -- 172.21.15.61:0/106720286 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1535256012 0 0) 0x7f08280b6680 con 0x7f08280b62a0 2022-07-20T03:16:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082d342700 1 -- 172.21.15.61:0/106720286 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 msgr2=0x7f082813fcc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082d342700 1 --2- 172.21.15.61:0/106720286 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f082813fcc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082d342700 1 -- 172.21.15.61:0/106720286 >> v1:172.21.15.61:6789/0 conn(0x7f08280b62a0 legacy=0x7f08280b43e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082d342700 1 -- 172.21.15.61:0/106720286 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0828142df0 con 0x7f08280b3ed0 2022-07-20T03:16:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082d342700 1 --2- 172.21.15.61:0/106720286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f0828143530 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f0824002f70 tx=0x7f0824006eb0).ready entity=mon.0 client_cookie=d166155fc475b508 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082cb41700 1 -- 172.21.15.61:0/106720286 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082400d570 con 0x7f08280b3ed0 2022-07-20T03:16:45.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082cb41700 1 -- 172.21.15.61:0/106720286 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0824000e90 con 0x7f08280b3ed0 2022-07-20T03:16:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082cb41700 1 -- 172.21.15.61:0/106720286 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f082400d8c0 con 0x7f08280b3ed0 2022-07-20T03:16:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 -- 172.21.15.61:0/106720286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 msgr2=0x7f0828143530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 --2- 172.21.15.61:0/106720286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f0828143530 secure :-1 s=READY pgs=1293 cs=0 l=1 rev1=1 rx=0x7f0824002f70 tx=0x7f0824006eb0).stop 2022-07-20T03:16:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 -- 172.21.15.61:0/106720286 shutdown_connections 2022-07-20T03:16:45.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 --2- 172.21.15.61:0/106720286 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f082813fcc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 --2- 172.21.15.61:0/106720286 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f0828143530 unknown :-1 s=CLOSED pgs=1293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 -- 172.21.15.61:0/106720286 >> 172.21.15.61:0/106720286 conn(0x7f0828066f00 msgr2=0x7f08280672e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:45.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 -- 172.21.15.61:0/106720286 shutdown_connections 2022-07-20T03:16:45.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 -- 172.21.15.61:0/106720286 wait complete. 2022-07-20T03:16:45.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 Processor -- start 2022-07-20T03:16:45.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.711+0000 7f082fda7700 1 -- start start 2022-07-20T03:16:45.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f082813d150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08280b62a0 0x7f0828062b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f0828063020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0828156ae0 con 0x7f08280b3ed0 2022-07-20T03:16:45.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0828156cc0 con 0x7f08280b62a0 2022-07-20T03:16:45.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0828156f60 con 0x7f0828143a20 2022-07-20T03:16:45.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f082813d150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f082813d150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50538/0 (socket says 172.21.15.61:50538) 2022-07-20T03:16:45.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 -- 172.21.15.61:0/3241200479 learned_addr learned my addr 172.21.15.61:0/3241200479 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:45.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082d342700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08280b62a0 0x7f0828062b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082e344700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f0828063020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 -- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 msgr2=0x7f0828063020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f0828063020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 -- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08280b62a0 msgr2=0x7f0828062b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08280b62a0 0x7f0828062b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 -- 172.21.15.61:0/3241200479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f082400b1e0 con 0x7f08280b3ed0 2022-07-20T03:16:45.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082db43700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f082813d150 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f081c000ed0 tx=0x7f081c007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081c012400 con 0x7f08280b3ed0 2022-07-20T03:16:45.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f081c012560 con 0x7f08280b3ed0 2022-07-20T03:16:45.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08280af980 con 0x7f08280b3ed0 2022-07-20T03:16:45.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f081c0128c0 con 0x7f08280b3ed0 2022-07-20T03:16:45.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.715+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08280ade00 con 0x7f08280b3ed0 2022-07-20T03:16:45.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.719+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08280ad410 con 0x7f08280b3ed0 2022-07-20T03:16:45.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.719+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f081c019030 con 0x7f08280b3ed0 2022-07-20T03:16:45.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.719+0000 7f081affd700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f08080539d0 0x7f0808055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.719+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f081c06b0a0 con 0x7f08280b3ed0 2022-07-20T03:16:45.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.719+0000 7f082d342700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f08080539d0 0x7f0808055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.719+0000 7f082d342700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f08080539d0 0x7f0808055e90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f082401c000 tx=0x7f0824015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.723+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08280ad410 con 0x7f08280b3ed0 2022-07-20T03:16:45.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.847+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f08280ad410 con 0x7f08280b3ed0 2022-07-20T03:16:45.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.847+0000 7f081affd700 1 -- 172.21.15.61:0/3241200479 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v130) v1 ==== 110+0+10 (secure 0 0 0) 0x7f08280ad410 con 0x7f08280b3ed0 2022-07-20T03:16:45.854 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:16:45.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f08080539d0 msgr2=0x7f0808055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f08080539d0 0x7f0808055e90 secure :-1 s=READY pgs=625 cs=0 l=1 rev1=1 rx=0x7f082401c000 tx=0x7f0824015000).stop 2022-07-20T03:16:45.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 msgr2=0x7f082813d150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f082813d150 secure :-1 s=READY pgs=1294 cs=0 l=1 rev1=1 rx=0x7f081c000ed0 tx=0x7f081c007ac0).stop 2022-07-20T03:16:45.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 shutdown_connections 2022-07-20T03:16:45.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f08080539d0 0x7f0808055e90 unknown :-1 s=CLOSED pgs=625 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08280b62a0 0x7f0828062b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0828143a20 0x7f0828063020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 --2- 172.21.15.61:0/3241200479 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08280b3ed0 0x7f082813d150 unknown :-1 s=CLOSED pgs=1294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 >> 172.21.15.61:0/3241200479 conn(0x7f0828066f00 msgr2=0x7f08280b4c30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:45.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 shutdown_connections 2022-07-20T03:16:45.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.851+0000 7f082fda7700 1 -- 172.21.15.61:0/3241200479 wait complete. 2022-07-20T03:16:45.868 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:16:45.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.975+0000 7fb13de98700 1 Processor -- start 2022-07-20T03:16:45.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.975+0000 7fb13de98700 1 -- start start 2022-07-20T03:16:45.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb1380b58f0 0x7fb1380aa9d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b4b90 0x7fb138143f60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:45.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb1380b4150 con 0x7fb1380b01f0 2022-07-20T03:16:45.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb1380b4fd0 con 0x7fb1380b58f0 2022-07-20T03:16:45.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb1380b43f0 con 0x7fb1380b4b90 2022-07-20T03:16:45.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13d697700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb1380b01f0 0x7fb1380abfb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47148/0 (socket says 172.21.15.61:47148) 2022-07-20T03:16:45.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13d697700 1 -- 172.21.15.61:0/3791524978 learned_addr learned my addr 172.21.15.61:0/3791524978 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:45.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13ce96700 1 --2- 172.21.15.61:0/3791524978 >> v2:172.21.15.117:3300/0 conn(0x7fb1380b58f0 0x7fb1380aa9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb12ffff700 1 --2- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b4b90 0x7fb138143f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:45.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb12f7fe700 1 -- 172.21.15.61:0/3791524978 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 771712035 0 0) 0x7fb1380b4150 con 0x7fb1380b01f0 2022-07-20T03:16:45.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb12f7fe700 1 -- 172.21.15.61:0/3791524978 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb124003260 con 0x7fb1380b01f0 2022-07-20T03:16:45.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13ce96700 1 -- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b4b90 msgr2=0x7fb138143f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:45.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13ce96700 1 --2- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b4b90 0x7fb138143f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13ce96700 1 -- 172.21.15.61:0/3791524978 >> v1:172.21.15.61:6789/0 conn(0x7fb1380b01f0 legacy=0x7fb1380abfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13ce96700 1 -- 172.21.15.61:0/3791524978 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1380b4690 con 0x7fb1380b58f0 2022-07-20T03:16:45.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13ce96700 1 --2- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b58f0 0x7fb1380aa9d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fb134002f70 tx=0x7fb134006df0).ready entity=mon.1 client_cookie=e06078aeb9a7e63b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:45.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb12f7fe700 1 -- 172.21.15.61:0/3791524978 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb13400d4a0 con 0x7fb1380b58f0 2022-07-20T03:16:45.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb12f7fe700 1 -- 172.21.15.61:0/3791524978 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb134000e90 con 0x7fb1380b58f0 2022-07-20T03:16:45.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb12f7fe700 1 -- 172.21.15.61:0/3791524978 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb13400d7f0 con 0x7fb1380b58f0 2022-07-20T03:16:45.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b58f0 msgr2=0x7fb1380aa9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:45.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 --2- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b58f0 0x7fb1380aa9d0 secure :-1 s=READY pgs=592 cs=0 l=1 rev1=1 rx=0x7fb134002f70 tx=0x7fb134006df0).stop 2022-07-20T03:16:45.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- 172.21.15.61:0/3791524978 shutdown_connections 2022-07-20T03:16:45.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 --2- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b4b90 0x7fb138143f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 --2- 172.21.15.61:0/3791524978 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b58f0 0x7fb1380aa9d0 unknown :-1 s=CLOSED pgs=592 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:45.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- 172.21.15.61:0/3791524978 >> 172.21.15.61:0/3791524978 conn(0x7fb1380632b0 msgr2=0x7fb138063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- 172.21.15.61:0/3791524978 shutdown_connections 2022-07-20T03:16:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.979+0000 7fb13de98700 1 -- 172.21.15.61:0/3791524978 wait complete. 2022-07-20T03:16:45.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 Processor -- start 2022-07-20T03:16:45.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 -- start start 2022-07-20T03:16:46.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b01f0 0x7fb1381532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 0x7fb138065190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b58f0 0x7fb138065680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb138091930 con 0x7fb1380b4b90 2022-07-20T03:16:46.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb13808f760 con 0x7fb1380b01f0 2022-07-20T03:16:46.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb138090b60 con 0x7fb1380b58f0 2022-07-20T03:16:46.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 0x7fb138065190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13ce96700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b01f0 0x7fb1381532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 0x7fb138065190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50554/0 (socket says 172.21.15.61:50554) 2022-07-20T03:16:46.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 -- 172.21.15.61:0/1218404257 learned_addr learned my addr 172.21.15.61:0/1218404257 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13d697700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b58f0 0x7fb138065680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 -- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b58f0 msgr2=0x7fb138065680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b58f0 0x7fb138065680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 -- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b01f0 msgr2=0x7fb1381532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b01f0 0x7fb1381532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 -- 172.21.15.61:0/1218404257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb13400b180 con 0x7fb1380b4b90 2022-07-20T03:16:46.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12ffff700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 0x7fb138065190 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fb12800c000 tx=0x7fb128006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb128010780 con 0x7fb1380b4b90 2022-07-20T03:16:46.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb1280108e0 con 0x7fb1380b4b90 2022-07-20T03:16:46.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1381537c0 con 0x7fb1380b4b90 2022-07-20T03:16:46.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb128010c40 con 0x7fb1380b4b90 2022-07-20T03:16:46.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.983+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb138065b70 con 0x7fb1380b4b90 2022-07-20T03:16:46.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.987+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb138067040 con 0x7fb1380b4b90 2022-07-20T03:16:46.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.987+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb12801c460 con 0x7fb1380b4b90 2022-07-20T03:16:46.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.987+0000 7fb12dffb700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb1180539d0 0x7fb118055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.987+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb12806c270 con 0x7fb1380b4b90 2022-07-20T03:16:46.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.987+0000 7fb13ce96700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb1180539d0 0x7fb118055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.987+0000 7fb13ce96700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb1180539d0 0x7fb118055e90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7fb13401a000 tx=0x7fb134014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:45.995+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb1280430d0 con 0x7fb1380b4b90 2022-07-20T03:16:46.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.123+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fb1380574e0 con 0x7fb1380b4b90 2022-07-20T03:16:46.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb12dffb700 1 -- 172.21.15.61:0/1218404257 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v130) v1 ==== 92+0+10 (secure 0 0 0) 0x7fb1280466f0 con 0x7fb1380b4b90 2022-07-20T03:16:46.130 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:46.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb1180539d0 msgr2=0x7fb118055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb1180539d0 0x7fb118055e90 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7fb13401a000 tx=0x7fb134014000).stop 2022-07-20T03:16:46.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 msgr2=0x7fb138065190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 0x7fb138065190 secure :-1 s=READY pgs=1296 cs=0 l=1 rev1=1 rx=0x7fb12800c000 tx=0x7fb128006df0).stop 2022-07-20T03:16:46.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 shutdown_connections 2022-07-20T03:16:46.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb1180539d0 0x7fb118055e90 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb1380b58f0 0x7fb138065680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb1380b4b90 0x7fb138065190 unknown :-1 s=CLOSED pgs=1296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 --2- 172.21.15.61:0/1218404257 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb1380b01f0 0x7fb1381532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 >> 172.21.15.61:0/1218404257 conn(0x7fb1380632b0 msgr2=0x7fb138142c40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 shutdown_connections 2022-07-20T03:16:46.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.127+0000 7fb13de98700 1 -- 172.21.15.61:0/1218404257 wait complete. 2022-07-20T03:16:46.144 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:16:46.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 Processor -- start 2022-07-20T03:16:46.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- start start 2022-07-20T03:16:46.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44880b5b50 0x7f44880accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b0480 0x7f448814a670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f448814ab60 0x7f4488144320 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44880ac5e0 con 0x7f44880b5b50 2022-07-20T03:16:46.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f44880b5f30 con 0x7f448814ab60 2022-07-20T03:16:46.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f448805cf70 con 0x7f44880b0480 2022-07-20T03:16:46.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44880b5b50 0x7f44880accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f44880b5b50 0x7f44880accb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50560/0 (socket says 172.21.15.61:50560) 2022-07-20T03:16:46.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 -- 172.21.15.61:0/1355151828 learned_addr learned my addr 172.21.15.61:0/1355151828 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f44877fe700 1 --2- 172.21.15.61:0/1355151828 >> v2:172.21.15.117:3300/0 conn(0x7f448814ab60 0x7f4488144320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f447e7fc700 1 --2- 172.21.15.61:0/1355151828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b0480 0x7f448814a670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 -- 172.21.15.61:0/1355151828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b0480 msgr2=0x7f448814a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:46.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 --2- 172.21.15.61:0/1355151828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b0480 0x7f448814a670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 -- 172.21.15.61:0/1355151828 >> v2:172.21.15.117:3300/0 conn(0x7f448814ab60 msgr2=0x7f4488144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:46.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 --2- 172.21.15.61:0/1355151828 >> v2:172.21.15.117:3300/0 conn(0x7f448814ab60 0x7f4488144320 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 -- 172.21.15.61:0/1355151828 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44880ac810 con 0x7f44880b5b50 2022-07-20T03:16:46.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f44877fe700 1 --2- 172.21.15.61:0/1355151828 >> v2:172.21.15.117:3300/0 conn(0x7f448814ab60 0x7f4488144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:46.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f4486ffd700 1 --2- 172.21.15.61:0/1355151828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44880b5b50 0x7f44880accb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f4478002f70 tx=0x7f4478006eb0).ready entity=mon.0 client_cookie=e9fc2c11ad7573b4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f44867fc700 1 -- 172.21.15.61:0/1355151828 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f447800d570 con 0x7f44880b5b50 2022-07-20T03:16:46.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f44867fc700 1 -- 172.21.15.61:0/1355151828 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4478000e90 con 0x7f44880b5b50 2022-07-20T03:16:46.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f44867fc700 1 -- 172.21.15.61:0/1355151828 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f447800d900 con 0x7f44880b5b50 2022-07-20T03:16:46.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- 172.21.15.61:0/1355151828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44880b5b50 msgr2=0x7f44880accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- 172.21.15.61:0/1355151828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44880b5b50 0x7f44880accb0 secure :-1 s=READY pgs=1297 cs=0 l=1 rev1=1 rx=0x7f4478002f70 tx=0x7f4478006eb0).stop 2022-07-20T03:16:46.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- 172.21.15.61:0/1355151828 shutdown_connections 2022-07-20T03:16:46.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- 172.21.15.61:0/1355151828 >> v2:172.21.15.117:3300/0 conn(0x7f448814ab60 0x7f4488144320 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- 172.21.15.61:0/1355151828 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b0480 0x7f448814a670 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 --2- 172.21.15.61:0/1355151828 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f44880b5b50 0x7f44880accb0 unknown :-1 s=CLOSED pgs=1297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- 172.21.15.61:0/1355151828 >> 172.21.15.61:0/1355151828 conn(0x7f44880635e0 msgr2=0x7f44880b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- 172.21.15.61:0/1355151828 shutdown_connections 2022-07-20T03:16:46.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.247+0000 7f448d323700 1 -- 172.21.15.61:0/1355151828 wait complete. 2022-07-20T03:16:46.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 Processor -- start 2022-07-20T03:16:46.269 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 -- start start 2022-07-20T03:16:46.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44880b0480 0x7f44880aec60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b5b50 0x7f4488156650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 0x7f448815a190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.270 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488063a90 con 0x7f448814ab60 2022-07-20T03:16:46.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4488156b40 con 0x7f44880b0480 2022-07-20T03:16:46.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.251+0000 7f448d323700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4488159d30 con 0x7f44880b5b50 2022-07-20T03:16:46.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 0x7f448815a190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 0x7f448815a190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50570/0 (socket says 172.21.15.61:50570) 2022-07-20T03:16:46.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 -- 172.21.15.61:0/87303250 learned_addr learned my addr 172.21.15.61:0/87303250 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4486ffd700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44880b0480 0x7f44880aec60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f447e7fc700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b5b50 0x7f4488156650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.272 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 -- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b5b50 msgr2=0x7f4488156650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b5b50 0x7f4488156650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 -- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44880b0480 msgr2=0x7f44880aec60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44880b0480 0x7f44880aec60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 -- 172.21.15.61:0/87303250 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f447800b1e0 con 0x7f448814ab60 2022-07-20T03:16:46.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f44877fe700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 0x7f448815a190 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f448000a000 tx=0x7f4480008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f448000f710 con 0x7f448814ab60 2022-07-20T03:16:46.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f448000f870 con 0x7f448814ab60 2022-07-20T03:16:46.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f448000fbd0 con 0x7f448814ab60 2022-07-20T03:16:46.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4488063f60 con 0x7f448814ab60 2022-07-20T03:16:46.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f448814a2c0 con 0x7f448814ab60 2022-07-20T03:16:46.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f448001a030 con 0x7f448814ab60 2022-07-20T03:16:46.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f448805d210 con 0x7f448814ab60 2022-07-20T03:16:46.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4484ff9700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f446c053920 0x7f446c055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f448006bd40 con 0x7f448814ab60 2022-07-20T03:16:46.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.255+0000 7f4486ffd700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f446c053920 0x7f446c055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.259+0000 7f4486ffd700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f446c053920 0x7f446c055de0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f447801c000 tx=0x7f4478015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.263+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4480042b30 con 0x7f448814ab60 2022-07-20T03:16:46.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.387+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f4488159ec0 con 0x7f448814ab60 2022-07-20T03:16:46.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.387+0000 7f4484ff9700 1 -- 172.21.15.61:0/87303250 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+10 (secure 0 0 0) 0x7f4480046150 con 0x7f448814ab60 2022-07-20T03:16:46.395 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:46.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f446c053920 msgr2=0x7f446c055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f446c053920 0x7f446c055de0 secure :-1 s=READY pgs=627 cs=0 l=1 rev1=1 rx=0x7f447801c000 tx=0x7f4478015000).stop 2022-07-20T03:16:46.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 msgr2=0x7f448815a190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 0x7f448815a190 secure :-1 s=READY pgs=1298 cs=0 l=1 rev1=1 rx=0x7f448000a000 tx=0x7f4480008040).stop 2022-07-20T03:16:46.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 shutdown_connections 2022-07-20T03:16:46.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f446c053920 0x7f446c055de0 unknown :-1 s=CLOSED pgs=627 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f448814ab60 0x7f448815a190 unknown :-1 s=CLOSED pgs=1298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f44880b5b50 0x7f4488156650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 --2- 172.21.15.61:0/87303250 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f44880b0480 0x7f44880aec60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 >> 172.21.15.61:0/87303250 conn(0x7f44880635e0 msgr2=0x7f448805bee0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 shutdown_connections 2022-07-20T03:16:46.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.391+0000 7f448d323700 1 -- 172.21.15.61:0/87303250 wait complete. 2022-07-20T03:16:46.409 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:16:46.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.515+0000 7f431c696700 1 Processor -- start 2022-07-20T03:16:46.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.515+0000 7f431c696700 1 -- start start 2022-07-20T03:16:46.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f431409b6d0 0x7f431409bb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4314147030 con 0x7f431405d0e0 2022-07-20T03:16:46.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f431405d4c0 con 0x7f43140b5960 2022-07-20T03:16:46.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f43141461e0 con 0x7f431409b6d0 2022-07-20T03:16:46.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f431409b6d0 0x7f431409bb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f431409b6d0 0x7f431409bb00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50578/0 (socket says 172.21.15.61:50578) 2022-07-20T03:16:46.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 -- 172.21.15.61:0/1022218362 learned_addr learned my addr 172.21.15.61:0/1022218362 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 -- 172.21.15.61:0/1022218362 >> v1:172.21.15.117:6789/0 conn(0x7f43140b5960 legacy=0x7f4314142b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 -- 172.21.15.61:0/1022218362 >> v1:172.21.15.61:6789/0 conn(0x7f431405d0e0 legacy=0x7f4314146920 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 -- 172.21.15.61:0/1022218362 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4314146480 con 0x7f431409b6d0 2022-07-20T03:16:46.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431ac33700 1 --2- 172.21.15.61:0/1022218362 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431409b6d0 0x7f431409bb00 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f430c0032f0 tx=0x7f430c009290).ready entity=mon.0 client_cookie=d7277fec67e46d0c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f4319430700 1 -- 172.21.15.61:0/1022218362 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00b8a0 con 0x7f431409b6d0 2022-07-20T03:16:46.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f4319430700 1 -- 172.21.15.61:0/1022218362 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f430c00ba00 con 0x7f431409b6d0 2022-07-20T03:16:46.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f4319430700 1 -- 172.21.15.61:0/1022218362 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f430c00bb70 con 0x7f431409b6d0 2022-07-20T03:16:46.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- 172.21.15.61:0/1022218362 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431409b6d0 msgr2=0x7f431409bb00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 --2- 172.21.15.61:0/1022218362 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431409b6d0 0x7f431409bb00 secure :-1 s=READY pgs=1300 cs=0 l=1 rev1=1 rx=0x7f430c0032f0 tx=0x7f430c009290).stop 2022-07-20T03:16:46.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- 172.21.15.61:0/1022218362 shutdown_connections 2022-07-20T03:16:46.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 --2- 172.21.15.61:0/1022218362 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431409b6d0 0x7f431409bb00 unknown :-1 s=CLOSED pgs=1300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- 172.21.15.61:0/1022218362 >> 172.21.15.61:0/1022218362 conn(0x7f43140b6a20 msgr2=0x7f4314066f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- 172.21.15.61:0/1022218362 shutdown_connections 2022-07-20T03:16:46.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- 172.21.15.61:0/1022218362 wait complete. 2022-07-20T03:16:46.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 Processor -- start 2022-07-20T03:16:46.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.519+0000 7f431c696700 1 -- start start 2022-07-20T03:16:46.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 0x7f43140b0ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431409b6d0 0x7f431413f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43140b5960 0x7f431413fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f431408f7e0 con 0x7f431405d0e0 2022-07-20T03:16:46.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43141558b0 con 0x7f431409b6d0 2022-07-20T03:16:46.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4314155b50 con 0x7f43140b5960 2022-07-20T03:16:46.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 0x7f43140b0ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 0x7f43140b0ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50580/0 (socket says 172.21.15.61:50580) 2022-07-20T03:16:46.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 -- 172.21.15.61:0/4283505102 learned_addr learned my addr 172.21.15.61:0/4283505102 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431ac33700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43140b5960 0x7f431413fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f4319c31700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431409b6d0 0x7f431413f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 -- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43140b5960 msgr2=0x7f431413fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43140b5960 0x7f431413fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 -- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431409b6d0 msgr2=0x7f431413f6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431409b6d0 0x7f431413f6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 -- 172.21.15.61:0/4283505102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f430c009530 con 0x7f431405d0e0 2022-07-20T03:16:46.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f4319c31700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431409b6d0 0x7f431413f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:46.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431a432700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 0x7f43140b0ef0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f43140b7080 tx=0x7f4310007c50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f431000ce30 con 0x7f431405d0e0 2022-07-20T03:16:46.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f431000f470 con 0x7f431405d0e0 2022-07-20T03:16:46.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4314155df0 con 0x7f431405d0e0 2022-07-20T03:16:46.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f431000f7d0 con 0x7f431405d0e0 2022-07-20T03:16:46.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4314140130 con 0x7f431405d0e0 2022-07-20T03:16:46.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.523+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4314143280 con 0x7f431405d0e0 2022-07-20T03:16:46.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.527+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4310019030 con 0x7f431405d0e0 2022-07-20T03:16:46.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.527+0000 7f43037fe700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f42f4053a40 0x7f42f4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.527+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f431006b400 con 0x7f431405d0e0 2022-07-20T03:16:46.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.527+0000 7f4319c31700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f42f4053a40 0x7f42f4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.527+0000 7f4319c31700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f42f4053a40 0x7f42f4055f00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f430400b000 tx=0x7f4304009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.535+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f43100421f0 con 0x7f431405d0e0 2022-07-20T03:16:46.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f43140574e0 con 0x7f431405d0e0 2022-07-20T03:16:46.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f43037fe700 1 -- 172.21.15.61:0/4283505102 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+10 (secure 0 0 0) 0x7f4310045810 con 0x7f431405d0e0 2022-07-20T03:16:46.672 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:46.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f42f4053a40 msgr2=0x7f42f4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f42f4053a40 0x7f42f4055f00 secure :-1 s=READY pgs=628 cs=0 l=1 rev1=1 rx=0x7f430400b000 tx=0x7f4304009040).stop 2022-07-20T03:16:46.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 msgr2=0x7f43140b0ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 0x7f43140b0ef0 secure :-1 s=READY pgs=1301 cs=0 l=1 rev1=1 rx=0x7f43140b7080 tx=0x7f4310007c50).stop 2022-07-20T03:16:46.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 shutdown_connections 2022-07-20T03:16:46.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f42f4053a40 0x7f42f4055f00 unknown :-1 s=CLOSED pgs=628 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f431409b6d0 0x7f431413f6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f43140b5960 0x7f431413fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 --2- 172.21.15.61:0/4283505102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f431405d0e0 0x7f43140b0ef0 unknown :-1 s=CLOSED pgs=1301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 >> 172.21.15.61:0/4283505102 conn(0x7f43140b6a20 msgr2=0x7f4314142750 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 shutdown_connections 2022-07-20T03:16:46.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.667+0000 7f431c696700 1 -- 172.21.15.61:0/4283505102 wait complete. 2022-07-20T03:16:46.687 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:16:46.688 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:46.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 Processor -- start 2022-07-20T03:16:46.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 -- start start 2022-07-20T03:16:46.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f70900afa30 0x7f70900ae0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900ae5c0 0x7f7090143200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f70900aff40 con 0x7f70900b1e00 2022-07-20T03:16:46.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f70900b0bc0 con 0x7f70900afa30 2022-07-20T03:16:46.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f7094a61700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70900b21e0 con 0x7f70900ae5c0 2022-07-20T03:16:46.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f70900afa30 0x7f70900ae0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f70900afa30 0x7f70900ae0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50588/0 (socket says 172.21.15.61:50588) 2022-07-20T03:16:46.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 -- 172.21.15.61:0/2221079307 learned_addr learned my addr 172.21.15.61:0/2221079307 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708ed9d700 1 --2- 172.21.15.61:0/2221079307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900ae5c0 0x7f7090143200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 -- 172.21.15.61:0/2221079307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900ae5c0 msgr2=0x7f7090143200 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:46.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 --2- 172.21.15.61:0/2221079307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900ae5c0 0x7f7090143200 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 -- 172.21.15.61:0/2221079307 >> v1:172.21.15.117:6789/0 conn(0x7f70900b1e00 legacy=0x7f70900ad9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 -- 172.21.15.61:0/2221079307 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f70900af510 con 0x7f70900afa30 2022-07-20T03:16:46.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708dd9b700 1 --2- 172.21.15.61:0/2221079307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900afa30 0x7f70900ae0d0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f70880032f0 tx=0x7f7088009290).ready entity=mon.0 client_cookie=bd3274a70249fc88 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708d59a700 1 -- 172.21.15.61:0/2221079307 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708800b8a0 con 0x7f70900afa30 2022-07-20T03:16:46.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708d59a700 1 -- 172.21.15.61:0/2221079307 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f708800ba00 con 0x7f70900afa30 2022-07-20T03:16:46.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.795+0000 7f708d59a700 1 -- 172.21.15.61:0/2221079307 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708800bb70 con 0x7f70900afa30 2022-07-20T03:16:46.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- 172.21.15.61:0/2221079307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900afa30 msgr2=0x7f70900ae0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 --2- 172.21.15.61:0/2221079307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900afa30 0x7f70900ae0d0 secure :-1 s=READY pgs=1302 cs=0 l=1 rev1=1 rx=0x7f70880032f0 tx=0x7f7088009290).stop 2022-07-20T03:16:46.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- 172.21.15.61:0/2221079307 shutdown_connections 2022-07-20T03:16:46.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 --2- 172.21.15.61:0/2221079307 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900ae5c0 0x7f7090143200 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 --2- 172.21.15.61:0/2221079307 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900afa30 0x7f70900ae0d0 unknown :-1 s=CLOSED pgs=1302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- 172.21.15.61:0/2221079307 >> 172.21.15.61:0/2221079307 conn(0x7f70900611c0 msgr2=0x7f7090062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- 172.21.15.61:0/2221079307 shutdown_connections 2022-07-20T03:16:46.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- 172.21.15.61:0/2221079307 wait complete. 2022-07-20T03:16:46.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 Processor -- start 2022-07-20T03:16:46.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- start start 2022-07-20T03:16:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70900afa30 0x7f70900a56a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 0x7f70901432f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900a2b70 0x7f70900aad60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70900a4240 con 0x7f70900b1e00 2022-07-20T03:16:46.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70900a4f60 con 0x7f70900afa30 2022-07-20T03:16:46.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f7094a61700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f70900a5200 con 0x7f70900a2b70 2022-07-20T03:16:46.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f708dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 0x7f70901432f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f708dd9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 0x7f70901432f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50594/0 (socket says 172.21.15.61:50594) 2022-07-20T03:16:46.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f708e59c700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70900afa30 0x7f70900a56a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f708dd9b700 1 -- 172.21.15.61:0/3930760299 learned_addr learned my addr 172.21.15.61:0/3930760299 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:46.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.799+0000 7f708ed9d700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900a2b70 0x7f70900aad60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f708dd9b700 1 -- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900a2b70 msgr2=0x7f70900aad60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f708dd9b700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900a2b70 0x7f70900aad60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f708dd9b700 1 -- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70900afa30 msgr2=0x7f70900a56a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f708dd9b700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70900afa30 0x7f70900a56a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f708dd9b700 1 -- 172.21.15.61:0/3930760299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7088009530 con 0x7f70900b1e00 2022-07-20T03:16:46.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f708dd9b700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 0x7f70901432f0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f70880032f0 tx=0x7f7088016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7088021480 con 0x7f70900b1e00 2022-07-20T03:16:46.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f70880215e0 con 0x7f70900b1e00 2022-07-20T03:16:46.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f70900a4cb0 con 0x7f70900b1e00 2022-07-20T03:16:46.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7088021940 con 0x7f70900b1e00 2022-07-20T03:16:46.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f709005c040 con 0x7f70900b1e00 2022-07-20T03:16:46.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f708800f070 con 0x7f70900b1e00 2022-07-20T03:16:46.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70900b0170 con 0x7f70900b1e00 2022-07-20T03:16:46.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f70877fe700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7074053a60 0x7f7074055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:46.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.803+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f708806d320 con 0x7f70900b1e00 2022-07-20T03:16:46.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.807+0000 7f708e59c700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7074053a60 0x7f7074055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:46.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.807+0000 7f708e59c700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7074053a60 0x7f7074055f20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f707800d000 tx=0x7f7078007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:46.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.811+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7088044170 con 0x7f70900b1e00 2022-07-20T03:16:46.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.971+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f70900574e0 con 0x7f70900b1e00 2022-07-20T03:16:46.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.971+0000 7f70877fe700 1 -- 172.21.15.61:0/3930760299 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2142 (secure 0 0 0) 0x7f7088015020 con 0x7f70900b1e00 2022-07-20T03:16:46.980 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:46.980 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:16:39.468063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1435464119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1435464119},{"type":"v1","addr":"172.21.15.61:6833","nonce":1435464119}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:16:46.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7074053a60 msgr2=0x7f7074055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7074053a60 0x7f7074055f20 secure :-1 s=READY pgs=629 cs=0 l=1 rev1=1 rx=0x7f707800d000 tx=0x7f7078007b90).stop 2022-07-20T03:16:46.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 msgr2=0x7f70901432f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:46.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 0x7f70901432f0 secure :-1 s=READY pgs=1303 cs=0 l=1 rev1=1 rx=0x7f70880032f0 tx=0x7f7088016000).stop 2022-07-20T03:16:46.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 shutdown_connections 2022-07-20T03:16:46.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7074053a60 0x7f7074055f20 unknown :-1 s=CLOSED pgs=629 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f70900a2b70 0x7f70900aad60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f70900b1e00 0x7f70901432f0 unknown :-1 s=CLOSED pgs=1303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 --2- 172.21.15.61:0/3930760299 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f70900afa30 0x7f70900a56a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:46.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 >> 172.21.15.61:0/3930760299 conn(0x7f70900ae5c0 msgr2=0x7f70900b2ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:46.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 shutdown_connections 2022-07-20T03:16:46.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:46.975+0000 7f7094a61700 1 -- 172.21.15.61:0/3930760299 wait complete. 2022-07-20T03:16:46.989 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-20T03:16:47.201 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:16:38.464416+0000', 'modified': '2022-07-20T03:16:39.468063+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1435464119', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1435464119}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1435464119}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:47.202 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:47.203 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:16:47.323 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8091,"want_state":"up:active","state":"up:active","rank_uptime":8.8467163939999995,"mdsmap_epoch":72,"osdmap_epoch":130,"osdmap_epoch_barrier":130,"uptime":13.814879433} 2022-07-20T03:16:47.324 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8091, "mdsmap_epoch": 72, "osdmap_epoch": 130, "osdmap_epoch_barrier": 130, "rank_uptime": 8.846716394, "state": "up:active", "uptime": 13.814879433, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:47.326 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:16:47.326 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:16:47.327 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:16:47.328 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.329 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.338 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:47.399 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:47.399 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:47.408 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:47.409 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:47.410 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.411 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:47.416 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:16:47.417 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.419 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:47.434 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:47.435 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:47.495 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:47.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:16:47.490+0000 7f6686433080 -1 init, newargv = 0x5616dc9ce1a0 newargc=9 2022-07-20T03:16:47.496 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22852]: starting ceph client 2022-07-20T03:16:47.497 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:47.498 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.498 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:47.504 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22852]: starting fuse 2022-07-20T03:16:47.510 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:16:47.511 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:16:47.511 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:16:47.512 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:16:47.513 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:16:47.513 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:16:47.513 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:16:47.514 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:47.514 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:16:47.515 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:16:47.515 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:16:47.516 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:47.516 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:16:47.516 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:16:47.517 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:16:47.517 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:16:47.518 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:47.518 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:16:47.519 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:16:47.519 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:16:47.519 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:16:47.520 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:16:47.520 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:47.521 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:16:47.521 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:16:47.600 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22852.asok 2022-07-20T03:16:47.601 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22852.asok 2022-07-20T03:16:47.602 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.603 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22852.asok status 2022-07-20T03:16:47.693 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:16:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:16:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:16:47.693 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:16:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:16:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:16:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:16:47.694 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22852", 2022-07-20T03:16:47.695 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:16:47.695 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:47.695 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:16:47.695 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:16:47.696 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 8191, 2022-07-20T03:16:47.696 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:16:47.696 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:16:47.696 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:16:47.696 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 8191 2022-07-20T03:16:47.697 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:47.697 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:16:47.697 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:16:47.697 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:16:47.698 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1571317921 2022-07-20T03:16:47.698 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:16:47.698 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:47.698 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:16:47.698 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:16:47.699 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:16:47.699 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1571317921 2022-07-20T03:16:47.699 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:47.699 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.8191 172.21.15.163:0/1571317921", 2022-07-20T03:16:47.700 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/1571317921", 2022-07-20T03:16:47.700 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:16:47.701 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 72, 2022-07-20T03:16:47.701 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 130, 2022-07-20T03:16:47.702 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:16:47.702 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:16:47.703 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:16:47.703 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:16:47.706 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.707 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:47.708 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.716 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:47.718 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:47.719 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.719 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.781 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.782 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:47.783 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.791 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:47.793 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:47.793 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.794 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:47.794 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.841 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:47.844 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:47.846 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:16:47.846 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:16:47.847 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.863 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22852]: fuse finished with error 0 and tester_r 0 2022-07-20T03:16:47.865 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:47.866 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:47.867 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:47.921 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:16:47.922 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:16:47.923 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:16:54.024 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:54.025 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:54.035 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:16:54.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:16:54.041 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:16:54.045 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:16:54.049 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:16:54.074 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2022-07-20T03:16:54.075 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2022-07-20T03:16:54.075 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:16:54.076 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2022-07-20T03:16:54.076 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.077 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2022-07-20T03:16:54.077 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.078 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2022-07-20T03:16:54.078 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.078 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2022-07-20T03:16:54.079 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2022-07-20T03:16:54.079 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.079 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2022-07-20T03:16:54.079 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2022-07-20T03:16:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:16:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2022-07-20T03:16:54.080 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:16:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:16:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.081 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2022-07-20T03:16:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2022-07-20T03:16:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2022-07-20T03:16:54.082 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:16:54.083 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:16:54.083 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2022-07-20T03:16:54.083 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:16:54.083 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2022-07-20T03:16:54.084 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:16:54.084 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:16:54.084 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.085 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:16:54.085 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.085 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2022-07-20T03:16:54.086 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.086 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2022-07-20T03:16:54.086 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.086 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2022-07-20T03:16:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2022-07-20T03:16:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2022-07-20T03:16:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:16:54.087 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:16:54.088 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.088 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2022-07-20T03:16:54.088 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2022-07-20T03:16:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2022-07-20T03:16:54.089 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2022-07-20T03:16:54.090 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.090 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2022-07-20T03:16:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2022-07-20T03:16:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2022-07-20T03:16:54.091 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2022-07-20T03:16:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2022-07-20T03:16:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2022-07-20T03:16:54.092 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2022-07-20T03:16:54.093 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.093 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:16:54.093 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.093 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2022-07-20T03:16:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2022-07-20T03:16:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.094 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:16:54.095 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2022-07-20T03:16:54.095 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2022-07-20T03:16:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2022-07-20T03:16:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:16:54.096 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 20 2022-07-20T03:16:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = filestore 2022-07-20T03:16:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-20T03:16:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: osd sloppy crc = True 2022-07-20T03:16:54.097 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.098 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2022-07-20T03:16:54.098 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:16:54.098 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2022-07-20T03:16:54.098 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-20T03:16:54.099 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-20T03:16:54.099 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.099 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.100 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2022-07-20T03:16:54.100 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.100 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2022-07-20T03:16:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:16:54.101 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-20T03:16:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2022-07-20T03:16:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-20T03:16:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2022-07-20T03:16:54.102 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2022-07-20T03:16:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2022-07-20T03:16:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2022-07-20T03:16:54.103 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:16:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:16:54.104 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:16:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:16:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:16:54.105 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:16:54.106 INFO:teuthology.orchestra.run.smithi061.stdout: mon op complaint time = 120 2022-07-20T03:16:54.106 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.106 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2022-07-20T03:16:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2022-07-20T03:16:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2022-07-20T03:16:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2022-07-20T03:16:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:16:54.107 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:16:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: client mount timeout = 600 2022-07-20T03:16:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 20 2022-07-20T03:16:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:16:54.108 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.109 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.109 INFO:teuthology.orchestra.run.smithi061.stdout: client debug getattr caps = true 2022-07-20T03:16:54.109 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2022-07-20T03:16:54.109 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.b] 2022-07-20T03:16:54.109 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.c] 2022-07-20T03:16:54.110 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-20T03:16:54.110 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-20T03:16:54.110 INFO:teuthology.orchestra.run.smithi061.stdout:[mds] 2022-07-20T03:16:54.110 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 20 2022-07-20T03:16:54.111 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:16:54.111 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal frag = True 2022-07-20T03:16:54.111 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal fragment size max = 10000 2022-07-20T03:16:54.111 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal merge size = 5 2022-07-20T03:16:54.112 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split bits = 3 2022-07-20T03:16:54.112 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split size = 100 2022-07-20T03:16:54.112 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug frag = True 2022-07-20T03:16:54.112 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug scatterstat = True 2022-07-20T03:16:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: mds op complaint time = 180 2022-07-20T03:16:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: mds verify scatter = True 2022-07-20T03:16:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-20T03:16:54.113 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.114 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.119 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2022-07-20T03:16:54.120 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2022-07-20T03:16:54.121 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:16:54.121 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2022-07-20T03:16:54.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.122 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2022-07-20T03:16:54.122 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.123 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2022-07-20T03:16:54.123 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.123 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2022-07-20T03:16:54.123 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2022-07-20T03:16:54.124 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.124 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2022-07-20T03:16:54.124 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2022-07-20T03:16:54.124 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.125 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:16:54.125 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2022-07-20T03:16:54.125 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.125 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:16:54.126 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:16:54.126 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.126 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2022-07-20T03:16:54.126 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.127 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2022-07-20T03:16:54.127 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2022-07-20T03:16:54.127 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:16:54.127 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:16:54.128 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2022-07-20T03:16:54.128 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:16:54.129 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2022-07-20T03:16:54.129 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:16:54.129 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:16:54.130 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.130 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:16:54.131 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.131 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2022-07-20T03:16:54.131 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.132 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2022-07-20T03:16:54.132 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.133 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2022-07-20T03:16:54.133 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2022-07-20T03:16:54.134 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2022-07-20T03:16:54.134 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:16:54.135 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:16:54.135 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.136 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2022-07-20T03:16:54.136 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2022-07-20T03:16:54.137 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.137 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2022-07-20T03:16:54.137 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2022-07-20T03:16:54.138 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.138 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2022-07-20T03:16:54.139 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2022-07-20T03:16:54.139 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.140 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2022-07-20T03:16:54.140 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2022-07-20T03:16:54.141 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2022-07-20T03:16:54.142 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.142 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2022-07-20T03:16:54.143 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2022-07-20T03:16:54.143 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.144 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:16:54.144 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.145 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2022-07-20T03:16:54.145 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.145 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2022-07-20T03:16:54.146 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.146 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:16:54.146 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2022-07-20T03:16:54.146 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.147 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2022-07-20T03:16:54.147 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2022-07-20T03:16:54.147 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:16:54.147 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2022-07-20T03:16:54.148 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = filestore 2022-07-20T03:16:54.148 INFO:teuthology.orchestra.run.smithi179.stdout: osd op complaint time = 180 2022-07-20T03:16:54.148 INFO:teuthology.orchestra.run.smithi179.stdout: osd sloppy crc = True 2022-07-20T03:16:54.148 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.148 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2022-07-20T03:16:54.149 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:16:54.149 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2022-07-20T03:16:54.149 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2022-07-20T03:16:54.149 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2022-07-20T03:16:54.150 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.150 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.150 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2022-07-20T03:16:54.150 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.151 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2022-07-20T03:16:54.151 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:16:54.151 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2022-07-20T03:16:54.151 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2022-07-20T03:16:54.151 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2022-07-20T03:16:54.152 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2022-07-20T03:16:54.152 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2022-07-20T03:16:54.152 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.152 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2022-07-20T03:16:54.153 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2022-07-20T03:16:54.153 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.153 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.153 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:16:54.154 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:16:54.154 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:16:54.154 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.154 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:16:54.155 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:16:54.155 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:16:54.155 INFO:teuthology.orchestra.run.smithi179.stdout: mon op complaint time = 120 2022-07-20T03:16:54.155 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:16:54.156 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2022-07-20T03:16:54.156 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2022-07-20T03:16:54.156 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2022-07-20T03:16:54.156 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2022-07-20T03:16:54.157 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:16:54.157 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:16:54.157 INFO:teuthology.orchestra.run.smithi179.stdout: client mount timeout = 600 2022-07-20T03:16:54.157 INFO:teuthology.orchestra.run.smithi179.stdout: debug client = 20 2022-07-20T03:16:54.158 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:16:54.158 INFO:teuthology.orchestra.run.smithi179.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.158 INFO:teuthology.orchestra.run.smithi179.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.158 INFO:teuthology.orchestra.run.smithi179.stdout: client debug getattr caps = true 2022-07-20T03:16:54.159 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2022-07-20T03:16:54.159 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2022-07-20T03:16:54.159 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2022-07-20T03:16:54.159 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.a] 2022-07-20T03:16:54.160 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.b] 2022-07-20T03:16:54.160 INFO:teuthology.orchestra.run.smithi179.stdout:[mds] 2022-07-20T03:16:54.160 INFO:teuthology.orchestra.run.smithi179.stdout: debug mds = 20 2022-07-20T03:16:54.160 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:16:54.161 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal frag = True 2022-07-20T03:16:54.161 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal fragment size max = 10000 2022-07-20T03:16:54.161 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal merge size = 5 2022-07-20T03:16:54.161 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal split bits = 3 2022-07-20T03:16:54.162 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal split size = 100 2022-07-20T03:16:54.162 INFO:teuthology.orchestra.run.smithi179.stdout: mds debug frag = True 2022-07-20T03:16:54.162 INFO:teuthology.orchestra.run.smithi179.stdout: mds debug scatterstat = True 2022-07-20T03:16:54.162 INFO:teuthology.orchestra.run.smithi179.stdout: mds op complaint time = 180 2022-07-20T03:16:54.162 INFO:teuthology.orchestra.run.smithi179.stdout: mds verify scatter = True 2022-07-20T03:16:54.163 INFO:teuthology.orchestra.run.smithi179.stdout: osd op complaint time = 180 2022-07-20T03:16:54.163 INFO:teuthology.orchestra.run.smithi179.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.163 INFO:teuthology.orchestra.run.smithi179.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.163 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2022-07-20T03:16:54.164 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2022-07-20T03:16:54.164 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:16:54.164 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2022-07-20T03:16:54.164 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.165 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2022-07-20T03:16:54.165 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.165 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2022-07-20T03:16:54.165 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.166 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2022-07-20T03:16:54.166 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2022-07-20T03:16:54.166 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.166 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2022-07-20T03:16:54.167 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2022-07-20T03:16:54.167 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.167 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:16:54.167 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2022-07-20T03:16:54.168 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.168 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:16:54.168 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:16:54.168 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.169 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2022-07-20T03:16:54.169 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.169 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2022-07-20T03:16:54.169 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2022-07-20T03:16:54.170 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:16:54.170 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:16:54.170 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2022-07-20T03:16:54.170 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:16:54.171 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2022-07-20T03:16:54.171 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:16:54.171 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:16:54.171 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.171 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:16:54.172 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.172 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2022-07-20T03:16:54.172 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.172 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2022-07-20T03:16:54.173 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.173 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2022-07-20T03:16:54.173 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2022-07-20T03:16:54.173 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2022-07-20T03:16:54.174 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:16:54.174 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:16:54.174 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.174 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2022-07-20T03:16:54.174 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2022-07-20T03:16:54.175 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.175 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2022-07-20T03:16:54.175 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2022-07-20T03:16:54.175 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.176 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2022-07-20T03:16:54.176 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2022-07-20T03:16:54.176 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.176 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2022-07-20T03:16:54.177 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2022-07-20T03:16:54.177 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2022-07-20T03:16:54.177 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.177 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2022-07-20T03:16:54.178 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2022-07-20T03:16:54.178 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.178 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:16:54.178 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.178 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2022-07-20T03:16:54.179 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.179 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2022-07-20T03:16:54.179 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.180 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:16:54.180 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2022-07-20T03:16:54.180 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.180 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2022-07-20T03:16:54.181 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2022-07-20T03:16:54.181 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:16:54.181 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2022-07-20T03:16:54.181 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = filestore 2022-07-20T03:16:54.181 INFO:teuthology.orchestra.run.smithi163.stdout: osd op complaint time = 180 2022-07-20T03:16:54.182 INFO:teuthology.orchestra.run.smithi163.stdout: osd sloppy crc = True 2022-07-20T03:16:54.182 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.182 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2022-07-20T03:16:54.182 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:16:54.183 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2022-07-20T03:16:54.183 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2022-07-20T03:16:54.183 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2022-07-20T03:16:54.183 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.184 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.184 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2022-07-20T03:16:54.184 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.184 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2022-07-20T03:16:54.185 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:16:54.185 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2022-07-20T03:16:54.185 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2022-07-20T03:16:54.185 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2022-07-20T03:16:54.186 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2022-07-20T03:16:54.186 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2022-07-20T03:16:54.186 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.186 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2022-07-20T03:16:54.187 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2022-07-20T03:16:54.187 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.187 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.187 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:16:54.187 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:16:54.188 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:16:54.188 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.188 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:16:54.188 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:16:54.189 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:16:54.189 INFO:teuthology.orchestra.run.smithi163.stdout: mon op complaint time = 120 2022-07-20T03:16:54.189 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:16:54.189 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2022-07-20T03:16:54.190 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2022-07-20T03:16:54.190 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2022-07-20T03:16:54.190 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2022-07-20T03:16:54.190 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:16:54.190 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:16:54.191 INFO:teuthology.orchestra.run.smithi163.stdout: client mount timeout = 600 2022-07-20T03:16:54.191 INFO:teuthology.orchestra.run.smithi163.stdout: debug client = 20 2022-07-20T03:16:54.191 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:16:54.191 INFO:teuthology.orchestra.run.smithi163.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.192 INFO:teuthology.orchestra.run.smithi163.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.192 INFO:teuthology.orchestra.run.smithi163.stdout: client debug getattr caps = true 2022-07-20T03:16:54.192 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2022-07-20T03:16:54.192 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2022-07-20T03:16:54.193 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2022-07-20T03:16:54.193 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.a] 2022-07-20T03:16:54.193 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.b] 2022-07-20T03:16:54.193 INFO:teuthology.orchestra.run.smithi163.stdout:[mds] 2022-07-20T03:16:54.193 INFO:teuthology.orchestra.run.smithi163.stdout: debug mds = 20 2022-07-20T03:16:54.194 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:16:54.194 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal frag = True 2022-07-20T03:16:54.194 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal fragment size max = 10000 2022-07-20T03:16:54.194 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal merge size = 5 2022-07-20T03:16:54.195 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal split bits = 3 2022-07-20T03:16:54.195 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal split size = 100 2022-07-20T03:16:54.195 INFO:teuthology.orchestra.run.smithi163.stdout: mds debug frag = True 2022-07-20T03:16:54.195 INFO:teuthology.orchestra.run.smithi163.stdout: mds debug scatterstat = True 2022-07-20T03:16:54.196 INFO:teuthology.orchestra.run.smithi163.stdout: mds op complaint time = 180 2022-07-20T03:16:54.196 INFO:teuthology.orchestra.run.smithi163.stdout: mds verify scatter = True 2022-07-20T03:16:54.196 INFO:teuthology.orchestra.run.smithi163.stdout: osd op complaint time = 180 2022-07-20T03:16:54.196 INFO:teuthology.orchestra.run.smithi163.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.197 INFO:teuthology.orchestra.run.smithi163.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.197 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2022-07-20T03:16:54.197 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2022-07-20T03:16:54.197 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:16:54.198 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2022-07-20T03:16:54.198 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.198 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2022-07-20T03:16:54.198 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.199 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2022-07-20T03:16:54.199 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.199 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2022-07-20T03:16:54.199 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2022-07-20T03:16:54.199 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.200 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2022-07-20T03:16:54.200 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2022-07-20T03:16:54.200 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.200 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:16:54.201 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2022-07-20T03:16:54.201 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.201 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:16:54.201 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:16:54.202 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.202 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2022-07-20T03:16:54.202 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.202 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2022-07-20T03:16:54.203 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2022-07-20T03:16:54.203 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:16:54.203 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:16:54.203 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2022-07-20T03:16:54.203 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:16:54.204 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2022-07-20T03:16:54.204 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:16:54.204 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:16:54.204 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.205 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:16:54.205 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.205 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2022-07-20T03:16:54.205 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.206 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2022-07-20T03:16:54.206 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.206 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2022-07-20T03:16:54.206 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2022-07-20T03:16:54.207 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2022-07-20T03:16:54.207 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:16:54.207 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:16:54.207 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.208 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2022-07-20T03:16:54.208 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2022-07-20T03:16:54.208 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.208 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2022-07-20T03:16:54.208 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2022-07-20T03:16:54.209 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.209 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2022-07-20T03:16:54.209 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2022-07-20T03:16:54.209 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.210 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2022-07-20T03:16:54.210 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2022-07-20T03:16:54.210 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2022-07-20T03:16:54.210 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.211 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2022-07-20T03:16:54.211 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2022-07-20T03:16:54.211 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.211 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:16:54.211 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.212 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2022-07-20T03:16:54.212 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.212 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2022-07-20T03:16:54.212 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.213 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:16:54.213 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2022-07-20T03:16:54.213 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.213 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2022-07-20T03:16:54.214 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2022-07-20T03:16:54.214 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:16:54.214 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2022-07-20T03:16:54.214 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = filestore 2022-07-20T03:16:54.214 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2022-07-20T03:16:54.215 INFO:teuthology.orchestra.run.smithi124.stdout: osd sloppy crc = True 2022-07-20T03:16:54.215 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.215 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2022-07-20T03:16:54.215 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:16:54.216 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2022-07-20T03:16:54.216 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2022-07-20T03:16:54.216 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2022-07-20T03:16:54.216 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.217 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.217 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2022-07-20T03:16:54.217 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.217 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2022-07-20T03:16:54.218 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:16:54.218 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2022-07-20T03:16:54.218 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2022-07-20T03:16:54.218 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2022-07-20T03:16:54.218 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2022-07-20T03:16:54.219 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2022-07-20T03:16:54.219 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.219 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2022-07-20T03:16:54.219 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2022-07-20T03:16:54.220 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.220 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.220 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:16:54.220 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:16:54.221 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:16:54.221 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.221 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:16:54.221 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:16:54.221 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:16:54.222 INFO:teuthology.orchestra.run.smithi124.stdout: mon op complaint time = 120 2022-07-20T03:16:54.222 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:16:54.222 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2022-07-20T03:16:54.222 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2022-07-20T03:16:54.223 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2022-07-20T03:16:54.223 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2022-07-20T03:16:54.223 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:16:54.223 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:16:54.223 INFO:teuthology.orchestra.run.smithi124.stdout: client mount timeout = 600 2022-07-20T03:16:54.224 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2022-07-20T03:16:54.224 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:16:54.224 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.224 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.225 INFO:teuthology.orchestra.run.smithi124.stdout: client debug getattr caps = true 2022-07-20T03:16:54.225 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2022-07-20T03:16:54.225 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.b] 2022-07-20T03:16:54.225 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.c] 2022-07-20T03:16:54.226 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2022-07-20T03:16:54.226 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2022-07-20T03:16:54.226 INFO:teuthology.orchestra.run.smithi124.stdout:[mds] 2022-07-20T03:16:54.226 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds = 20 2022-07-20T03:16:54.226 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:16:54.227 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal frag = True 2022-07-20T03:16:54.227 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal fragment size max = 10000 2022-07-20T03:16:54.227 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal merge size = 5 2022-07-20T03:16:54.227 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal split bits = 3 2022-07-20T03:16:54.228 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal split size = 100 2022-07-20T03:16:54.228 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug frag = True 2022-07-20T03:16:54.228 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug scatterstat = True 2022-07-20T03:16:54.228 INFO:teuthology.orchestra.run.smithi124.stdout: mds op complaint time = 180 2022-07-20T03:16:54.228 INFO:teuthology.orchestra.run.smithi124.stdout: mds verify scatter = True 2022-07-20T03:16:54.229 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2022-07-20T03:16:54.229 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.229 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.230 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2022-07-20T03:16:54.230 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2022-07-20T03:16:54.230 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:16:54.230 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2022-07-20T03:16:54.231 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.231 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2022-07-20T03:16:54.231 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.231 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2022-07-20T03:16:54.231 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.232 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2022-07-20T03:16:54.232 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2022-07-20T03:16:54.232 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.232 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2022-07-20T03:16:54.233 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2022-07-20T03:16:54.233 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.233 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:16:54.233 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2022-07-20T03:16:54.234 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.234 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:16:54.234 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:16:54.234 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.235 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2022-07-20T03:16:54.235 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.235 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2022-07-20T03:16:54.235 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2022-07-20T03:16:54.235 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:16:54.236 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:16:54.236 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2022-07-20T03:16:54.236 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:16:54.236 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2022-07-20T03:16:54.237 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:16:54.237 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:16:54.237 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.237 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:16:54.238 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.238 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2022-07-20T03:16:54.238 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.238 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2022-07-20T03:16:54.239 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.239 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2022-07-20T03:16:54.239 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2022-07-20T03:16:54.239 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2022-07-20T03:16:54.239 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:16:54.240 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:16:54.240 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.240 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2022-07-20T03:16:54.240 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2022-07-20T03:16:54.241 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.241 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2022-07-20T03:16:54.241 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2022-07-20T03:16:54.241 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.241 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2022-07-20T03:16:54.242 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2022-07-20T03:16:54.242 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.242 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2022-07-20T03:16:54.242 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2022-07-20T03:16:54.243 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2022-07-20T03:16:54.243 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.243 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2022-07-20T03:16:54.243 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2022-07-20T03:16:54.244 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.244 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:16:54.244 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.244 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2022-07-20T03:16:54.244 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.245 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2022-07-20T03:16:54.245 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.245 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:16:54.245 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2022-07-20T03:16:54.246 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.246 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2022-07-20T03:16:54.246 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2022-07-20T03:16:54.246 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:16:54.247 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2022-07-20T03:16:54.247 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = filestore 2022-07-20T03:16:54.247 INFO:teuthology.orchestra.run.smithi117.stdout: osd op complaint time = 180 2022-07-20T03:16:54.247 INFO:teuthology.orchestra.run.smithi117.stdout: osd sloppy crc = True 2022-07-20T03:16:54.248 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.248 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2022-07-20T03:16:54.248 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:16:54.248 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2022-07-20T03:16:54.248 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2022-07-20T03:16:54.249 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2022-07-20T03:16:54.249 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.249 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.249 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2022-07-20T03:16:54.250 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.250 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2022-07-20T03:16:54.250 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:16:54.250 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2022-07-20T03:16:54.251 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2022-07-20T03:16:54.251 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2022-07-20T03:16:54.251 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2022-07-20T03:16:54.251 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2022-07-20T03:16:54.251 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:16:54.252 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2022-07-20T03:16:54.252 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2022-07-20T03:16:54.252 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:16:54.252 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.253 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:16:54.253 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:16:54.253 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:16:54.253 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.253 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:16:54.254 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:16:54.254 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:16:54.254 INFO:teuthology.orchestra.run.smithi117.stdout: mon op complaint time = 120 2022-07-20T03:16:54.254 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:16:54.255 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2022-07-20T03:16:54.255 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2022-07-20T03:16:54.255 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2022-07-20T03:16:54.255 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2022-07-20T03:16:54.256 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:16:54.256 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:16:54.256 INFO:teuthology.orchestra.run.smithi117.stdout: client mount timeout = 600 2022-07-20T03:16:54.256 INFO:teuthology.orchestra.run.smithi117.stdout: debug client = 20 2022-07-20T03:16:54.257 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:16:54.257 INFO:teuthology.orchestra.run.smithi117.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.257 INFO:teuthology.orchestra.run.smithi117.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.257 INFO:teuthology.orchestra.run.smithi117.stdout: client debug getattr caps = true 2022-07-20T03:16:54.258 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2022-07-20T03:16:54.258 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2022-07-20T03:16:54.258 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2022-07-20T03:16:54.258 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a] 2022-07-20T03:16:54.258 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b] 2022-07-20T03:16:54.259 INFO:teuthology.orchestra.run.smithi117.stdout:[mds] 2022-07-20T03:16:54.259 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds = 20 2022-07-20T03:16:54.259 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:16:54.259 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal frag = True 2022-07-20T03:16:54.260 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal fragment size max = 10000 2022-07-20T03:16:54.260 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal merge size = 5 2022-07-20T03:16:54.260 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal split bits = 3 2022-07-20T03:16:54.260 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal split size = 100 2022-07-20T03:16:54.261 INFO:teuthology.orchestra.run.smithi117.stdout: mds debug frag = True 2022-07-20T03:16:54.261 INFO:teuthology.orchestra.run.smithi117.stdout: mds debug scatterstat = True 2022-07-20T03:16:54.261 INFO:teuthology.orchestra.run.smithi117.stdout: mds op complaint time = 180 2022-07-20T03:16:54.261 INFO:teuthology.orchestra.run.smithi117.stdout: mds verify scatter = True 2022-07-20T03:16:54.262 INFO:teuthology.orchestra.run.smithi117.stdout: osd op complaint time = 180 2022-07-20T03:16:54.262 INFO:teuthology.orchestra.run.smithi117.stdout: rados mon op timeout = 15m 2022-07-20T03:16:54.262 INFO:teuthology.orchestra.run.smithi117.stdout: rados osd op timeout = 15m 2022-07-20T03:16:54.263 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:16:54.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 Processor -- start 2022-07-20T03:16:54.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 -- start start 2022-07-20T03:16:54.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b2840 0x7fd5680ae720 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd568141d80 0x7fd5681421d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5680b2c20 con 0x7fd568143030 2022-07-20T03:16:54.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5680aed60 con 0x7fd568141d80 2022-07-20T03:16:54.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56f2b3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd568067820 con 0x7fd5680b2840 2022-07-20T03:16:54.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56dab0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd568143030 0x7fd568143450 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47200/0 (socket says 172.21.15.61:47200) 2022-07-20T03:16:54.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56dab0700 1 -- 172.21.15.61:0/3464982494 learned_addr learned my addr 172.21.15.61:0/3464982494 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:54.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56eab2700 1 --2- 172.21.15.61:0/3464982494 >> v2:172.21.15.117:3300/0 conn(0x7fd568141d80 0x7fd5681421d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56e2b1700 1 --2- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b2840 0x7fd5680ae720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56d2af700 1 -- 172.21.15.61:0/3464982494 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1413691136 0 0) 0x7fd5680b2c20 con 0x7fd568143030 2022-07-20T03:16:54.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56d2af700 1 -- 172.21.15.61:0/3464982494 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd550003260 con 0x7fd568143030 2022-07-20T03:16:54.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56eab2700 1 -- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b2840 msgr2=0x7fd5680ae720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:54.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56eab2700 1 --2- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b2840 0x7fd5680ae720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56eab2700 1 -- 172.21.15.61:0/3464982494 >> v1:172.21.15.61:6789/0 conn(0x7fd568143030 legacy=0x7fd568143450 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56eab2700 1 -- 172.21.15.61:0/3464982494 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd568067ac0 con 0x7fd568141d80 2022-07-20T03:16:54.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.371+0000 7fd56eab2700 1 --2- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd568141d80 0x7fd5681421d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fd5600032f0 tx=0x7fd5600091d0).ready entity=mon.1 client_cookie=8301ebc7da379017 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56d2af700 1 -- 172.21.15.61:0/3464982494 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd56000b600 con 0x7fd568141d80 2022-07-20T03:16:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56d2af700 1 -- 172.21.15.61:0/3464982494 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd56000b760 con 0x7fd568141d80 2022-07-20T03:16:54.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56d2af700 1 -- 172.21.15.61:0/3464982494 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd56000b8d0 con 0x7fd568141d80 2022-07-20T03:16:54.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd568141d80 msgr2=0x7fd5681421d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd568141d80 0x7fd5681421d0 secure :-1 s=READY pgs=596 cs=0 l=1 rev1=1 rx=0x7fd5600032f0 tx=0x7fd5600091d0).stop 2022-07-20T03:16:54.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- 172.21.15.61:0/3464982494 shutdown_connections 2022-07-20T03:16:54.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd568141d80 0x7fd5681421d0 unknown :-1 s=CLOSED pgs=596 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/3464982494 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b2840 0x7fd5680ae720 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- 172.21.15.61:0/3464982494 >> 172.21.15.61:0/3464982494 conn(0x7fd5680632c0 msgr2=0x7fd5680636a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- 172.21.15.61:0/3464982494 shutdown_connections 2022-07-20T03:16:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- 172.21.15.61:0/3464982494 wait complete. 2022-07-20T03:16:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 Processor -- start 2022-07-20T03:16:54.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- start start 2022-07-20T03:16:54.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 0x7fd5680b3c40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd568141d80 0x7fd568145850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd568143030 0x7fd5681559a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568145d40 con 0x7fd568143030 2022-07-20T03:16:54.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568146d60 con 0x7fd5680b2840 2022-07-20T03:16:54.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.375+0000 7fd56f2b3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd568147000 con 0x7fd568141d80 2022-07-20T03:16:54.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd568143030 0x7fd5681559a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56e2b1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 0x7fd5680b3c40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56dab0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd568141d80 0x7fd568145850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56e2b1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 0x7fd5680b3c40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:60288/0 (socket says 172.21.15.61:60288) 2022-07-20T03:16:54.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56e2b1700 1 -- 172.21.15.61:0/478817059 learned_addr learned my addr 172.21.15.61:0/478817059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:54.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 -- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd568141d80 msgr2=0x7fd568145850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd568141d80 0x7fd568145850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 -- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 msgr2=0x7fd5680b3c40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 0x7fd5680b3c40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 -- 172.21.15.61:0/478817059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd560009470 con 0x7fd568143030 2022-07-20T03:16:54.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56dab0700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd568141d80 0x7fd568145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:54.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56e2b1700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 0x7fd5680b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56eab2700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd568143030 0x7fd5681559a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7fd5600032f0 tx=0x7fd560016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd56000b600 con 0x7fd568143030 2022-07-20T03:16:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd560011b60 con 0x7fd568143030 2022-07-20T03:16:54.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd568146a40 con 0x7fd568143030 2022-07-20T03:16:54.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd56001f3f0 con 0x7fd568143030 2022-07-20T03:16:54.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd568146590 con 0x7fd568143030 2022-07-20T03:16:54.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5681471c0 con 0x7fd568143030 2022-07-20T03:16:54.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.379+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd560026030 con 0x7fd568143030 2022-07-20T03:16:54.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.383+0000 7fd5577fe700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd548053920 0x7fd548055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.383+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd56006c770 con 0x7fd568143030 2022-07-20T03:16:54.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.383+0000 7fd56e2b1700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd548053920 0x7fd548055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.383+0000 7fd56e2b1700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd548053920 0x7fd548055de0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fd55800b000 tx=0x7fd558006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.387+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd5600436d0 con 0x7fd568143030 2022-07-20T03:16:54.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.515+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd568067d60 con 0x7fd568143030 2022-07-20T03:16:54.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.515+0000 7fd5577fe700 1 -- 172.21.15.61:0/478817059 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v130) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fd560015090 con 0x7fd568143030 2022-07-20T03:16:54.522 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:54.522 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":130,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:16:39.466964+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":25,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:16:36.305597+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"129","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":26,"pool_name":"cephfs_data","create_time":"2022-07-20T03:16:37.477547+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"130","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:16:54.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd548053920 msgr2=0x7fd548055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd548053920 0x7fd548055de0 secure :-1 s=READY pgs=630 cs=0 l=1 rev1=1 rx=0x7fd55800b000 tx=0x7fd558006df0).stop 2022-07-20T03:16:54.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd568143030 msgr2=0x7fd5681559a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd568143030 0x7fd5681559a0 secure :-1 s=READY pgs=1306 cs=0 l=1 rev1=1 rx=0x7fd5600032f0 tx=0x7fd560016000).stop 2022-07-20T03:16:54.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 shutdown_connections 2022-07-20T03:16:54.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd548053920 0x7fd548055de0 unknown :-1 s=CLOSED pgs=630 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd568143030 0x7fd5681559a0 unknown :-1 s=CLOSED pgs=1306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd568141d80 0x7fd568145850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 --2- 172.21.15.61:0/478817059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680b2840 0x7fd5680b3c40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 >> 172.21.15.61:0/478817059 conn(0x7fd5680632c0 msgr2=0x7fd5680636a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:54.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 shutdown_connections 2022-07-20T03:16:54.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.519+0000 7fd56f2b3700 1 -- 172.21.15.61:0/478817059 wait complete. 2022-07-20T03:16:54.535 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 25, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:16:36.305597+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '129', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 26, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:16:37.477547+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '130', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:16:54.536 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:16:54.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.643+0000 7f997c02d700 1 Processor -- start 2022-07-20T03:16:54.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.643+0000 7f997c02d700 1 -- start start 2022-07-20T03:16:54.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.643+0000 7f997c02d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99740b9860 0x7f99741437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f99741462a0 0x7f997413fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f99740b9c40 con 0x7f99740ad270 2022-07-20T03:16:54.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9974143cb0 con 0x7f99740b9860 2022-07-20T03:16:54.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9974142de0 con 0x7f99741462a0 2022-07-20T03:16:54.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99740b9860 0x7f99741437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99740b9860 0x7f99741437c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50618/0 (socket says 172.21.15.61:50618) 2022-07-20T03:16:54.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 -- 172.21.15.61:0/129361378 learned_addr learned my addr 172.21.15.61:0/129361378 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:54.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997a5ca700 1 --2- 172.21.15.61:0/129361378 >> v2:172.21.15.117:3300/0 conn(0x7f99741462a0 0x7f997413fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9978dc7700 1 -- 172.21.15.61:0/129361378 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3605139550 0 0) 0x7f99740b9c40 con 0x7f99740ad270 2022-07-20T03:16:54.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 -- 172.21.15.61:0/129361378 >> v2:172.21.15.117:3300/0 conn(0x7f99741462a0 msgr2=0x7f997413fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:54.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 --2- 172.21.15.61:0/129361378 >> v2:172.21.15.117:3300/0 conn(0x7f99741462a0 0x7f997413fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 -- 172.21.15.61:0/129361378 >> v1:172.21.15.117:6789/0 conn(0x7f99740ad270 legacy=0x7f99740b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 -- 172.21.15.61:0/129361378 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9974143080 con 0x7f99740b9860 2022-07-20T03:16:54.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9979dc9700 1 --2- 172.21.15.61:0/129361378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740b9860 0x7f99741437c0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f9974142d00 tx=0x7f9970006eb0).ready entity=mon.0 client_cookie=c707d8108bd842ea server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9978dc7700 1 -- 172.21.15.61:0/129361378 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f997000d610 con 0x7f99740b9860 2022-07-20T03:16:54.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9978dc7700 1 -- 172.21.15.61:0/129361378 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9970000e90 con 0x7f99740b9860 2022-07-20T03:16:54.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f9978dc7700 1 -- 172.21.15.61:0/129361378 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f997000d960 con 0x7f99740b9860 2022-07-20T03:16:54.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- 172.21.15.61:0/129361378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740b9860 msgr2=0x7f99741437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 --2- 172.21.15.61:0/129361378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740b9860 0x7f99741437c0 secure :-1 s=READY pgs=1307 cs=0 l=1 rev1=1 rx=0x7f9974142d00 tx=0x7f9970006eb0).stop 2022-07-20T03:16:54.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- 172.21.15.61:0/129361378 shutdown_connections 2022-07-20T03:16:54.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 --2- 172.21.15.61:0/129361378 >> v2:172.21.15.117:3300/0 conn(0x7f99741462a0 0x7f997413fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 --2- 172.21.15.61:0/129361378 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740b9860 0x7f99741437c0 unknown :-1 s=CLOSED pgs=1307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- 172.21.15.61:0/129361378 >> 172.21.15.61:0/129361378 conn(0x7f9974066e20 msgr2=0x7f9974067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:54.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- 172.21.15.61:0/129361378 shutdown_connections 2022-07-20T03:16:54.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.647+0000 7f997c02d700 1 -- 172.21.15.61:0/129361378 wait complete. 2022-07-20T03:16:54.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 Processor -- start 2022-07-20T03:16:54.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 -- start start 2022-07-20T03:16:54.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 0x7f99740b3830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99740b9860 0x7f9974112d60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99741462a0 0x7f997413e230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9974113ea0 con 0x7f99740ad270 2022-07-20T03:16:54.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99741154a0 con 0x7f99741462a0 2022-07-20T03:16:54.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99740b2bf0 con 0x7f99740b9860 2022-07-20T03:16:54.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 0x7f99740b3830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 0x7f99740b3830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50622/0 (socket says 172.21.15.61:50622) 2022-07-20T03:16:54.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 -- 172.21.15.61:0/2836944397 learned_addr learned my addr 172.21.15.61:0/2836944397 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:54.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997a5ca700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99741462a0 0x7f997413e230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f99795c8700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99740b9860 0x7f9974112d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 -- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99740b9860 msgr2=0x7f9974112d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99740b9860 0x7f9974112d60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 -- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99741462a0 msgr2=0x7f997413e230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99741462a0 0x7f997413e230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 -- 172.21.15.61:0/2836944397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f997000b1e0 con 0x7f99740ad270 2022-07-20T03:16:54.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9979dc9700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 0x7f99740b3830 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7f99740b2d80 tx=0x7f9970015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9970002f50 con 0x7f99740ad270 2022-07-20T03:16:54.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f997000de10 con 0x7f99740ad270 2022-07-20T03:16:54.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99740b2950 con 0x7f99740ad270 2022-07-20T03:16:54.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9970011d30 con 0x7f99740ad270 2022-07-20T03:16:54.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.651+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9974113250 con 0x7f99740ad270 2022-07-20T03:16:54.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.655+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9974143320 con 0x7f99740ad270 2022-07-20T03:16:54.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.655+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9970025030 con 0x7f99740ad270 2022-07-20T03:16:54.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.655+0000 7f9962ffd700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9954053a60 0x7f9954055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.655+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f997006d140 con 0x7f99740ad270 2022-07-20T03:16:54.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.655+0000 7f99795c8700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9954053a60 0x7f9954055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.655+0000 7f99795c8700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9954053a60 0x7f9954055f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f996400d000 tx=0x7f99640079b0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.667+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9970043f90 con 0x7f99740ad270 2022-07-20T03:16:54.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.791+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f99740b7190 con 0x7f99740ad270 2022-07-20T03:16:54.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.791+0000 7f9962ffd700 1 -- 172.21.15.61:0/2836944397 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v130) v1 ==== 110+0+10 (secure 0 0 0) 0x7f9970023020 con 0x7f99740ad270 2022-07-20T03:16:54.797 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:16:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9954053a60 msgr2=0x7f9954055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9954053a60 0x7f9954055f20 secure :-1 s=READY pgs=631 cs=0 l=1 rev1=1 rx=0x7f996400d000 tx=0x7f99640079b0).stop 2022-07-20T03:16:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 msgr2=0x7f99740b3830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 0x7f99740b3830 secure :-1 s=READY pgs=1308 cs=0 l=1 rev1=1 rx=0x7f99740b2d80 tx=0x7f9970015000).stop 2022-07-20T03:16:54.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 shutdown_connections 2022-07-20T03:16:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9954053a60 0x7f9954055f20 unknown :-1 s=CLOSED pgs=631 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f99741462a0 0x7f997413e230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f99740b9860 0x7f9974112d60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 --2- 172.21.15.61:0/2836944397 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99740ad270 0x7f99740b3830 unknown :-1 s=CLOSED pgs=1308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 >> 172.21.15.61:0/2836944397 conn(0x7f9974066e20 msgr2=0x7f9974145ae0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:54.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 shutdown_connections 2022-07-20T03:16:54.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.795+0000 7f997c02d700 1 -- 172.21.15.61:0/2836944397 wait complete. 2022-07-20T03:16:54.815 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:16:54.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.919+0000 7f20daf3e700 1 Processor -- start 2022-07-20T03:16:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.919+0000 7f20daf3e700 1 -- start start 2022-07-20T03:16:54.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f20d40b5ed0 0x7f20d4060f90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d41422b0 0x7f20d41426e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f20d41464b0 con 0x7f20d40b62b0 2022-07-20T03:16:54.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f20d40b4140 con 0x7f20d40b5ed0 2022-07-20T03:16:54.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4144170 con 0x7f20d41422b0 2022-07-20T03:16:54.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d973b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f20d40b62b0 0x7f20d4146b80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47230/0 (socket says 172.21.15.61:47230) 2022-07-20T03:16:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d973b700 1 -- 172.21.15.61:0/416219224 learned_addr learned my addr 172.21.15.61:0/416219224 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d9f3c700 1 --2- 172.21.15.61:0/416219224 >> v2:172.21.15.117:3300/0 conn(0x7f20d40b5ed0 0x7f20d4060f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20da73d700 1 --2- 172.21.15.61:0/416219224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d41422b0 0x7f20d41426e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d8f3a700 1 -- 172.21.15.61:0/416219224 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1092829027 0 0) 0x7f20d41464b0 con 0x7f20d40b62b0 2022-07-20T03:16:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d8f3a700 1 -- 172.21.15.61:0/416219224 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f20bc003260 con 0x7f20d40b62b0 2022-07-20T03:16:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20da73d700 1 -- 172.21.15.61:0/416219224 >> v2:172.21.15.117:3300/0 conn(0x7f20d40b5ed0 msgr2=0x7f20d4060f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20da73d700 1 --2- 172.21.15.61:0/416219224 >> v2:172.21.15.117:3300/0 conn(0x7f20d40b5ed0 0x7f20d4060f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20da73d700 1 -- 172.21.15.61:0/416219224 >> v1:172.21.15.61:6789/0 conn(0x7f20d40b62b0 legacy=0x7f20d4146b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20da73d700 1 -- 172.21.15.61:0/416219224 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d40aff60 con 0x7f20d41422b0 2022-07-20T03:16:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d9f3c700 1 --2- 172.21.15.61:0/416219224 >> v2:172.21.15.117:3300/0 conn(0x7f20d40b5ed0 0x7f20d4060f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:54.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20da73d700 1 --2- 172.21.15.61:0/416219224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d41422b0 0x7f20d41426e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f20cc0032f0 tx=0x7f20cc0091d0).ready entity=mon.2 client_cookie=a6a9153bb2c04dd7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d8f3a700 1 -- 172.21.15.61:0/416219224 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20cc00b600 con 0x7f20d41422b0 2022-07-20T03:16:54.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d8f3a700 1 -- 172.21.15.61:0/416219224 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20cc00b760 con 0x7f20d41422b0 2022-07-20T03:16:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20d8f3a700 1 -- 172.21.15.61:0/416219224 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20cc00b8d0 con 0x7f20d41422b0 2022-07-20T03:16:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- 172.21.15.61:0/416219224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d41422b0 msgr2=0x7f20d41426e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 --2- 172.21.15.61:0/416219224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d41422b0 0x7f20d41426e0 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f20cc0032f0 tx=0x7f20cc0091d0).stop 2022-07-20T03:16:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- 172.21.15.61:0/416219224 shutdown_connections 2022-07-20T03:16:54.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 --2- 172.21.15.61:0/416219224 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d41422b0 0x7f20d41426e0 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 --2- 172.21.15.61:0/416219224 >> v2:172.21.15.117:3300/0 conn(0x7f20d40b5ed0 0x7f20d4060f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- 172.21.15.61:0/416219224 >> 172.21.15.61:0/416219224 conn(0x7f20d405be30 msgr2=0x7f20d405c210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- 172.21.15.61:0/416219224 shutdown_connections 2022-07-20T03:16:54.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.923+0000 7f20daf3e700 1 -- 172.21.15.61:0/416219224 wait complete. 2022-07-20T03:16:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 Processor -- start 2022-07-20T03:16:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 -- start start 2022-07-20T03:16:54.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d40b5ed0 0x7f20d4145e80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 0x7f20d41520d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d41422b0 0x7f20d409c4e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4144fa0 con 0x7f20d40b62b0 2022-07-20T03:16:54.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f20d409e260 con 0x7f20d41422b0 2022-07-20T03:16:54.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f20d4144d00 con 0x7f20d40b5ed0 2022-07-20T03:16:54.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d973b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 0x7f20d41520d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d973b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 0x7f20d41520d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50636/0 (socket says 172.21.15.61:50636) 2022-07-20T03:16:54.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d40b5ed0 0x7f20d4145e80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d973b700 1 -- 172.21.15.61:0/2263101619 learned_addr learned my addr 172.21.15.61:0/2263101619 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:54.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20da73d700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d41422b0 0x7f20d409c4e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 -- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d41422b0 msgr2=0x7f20d409c4e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d41422b0 0x7f20d409c4e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 -- 172.21.15.61:0/2263101619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 msgr2=0x7f20d41520d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:54.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 0x7f20d41520d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:54.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 -- 172.21.15.61:0/2263101619 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f20d000b180 con 0x7f20d40b5ed0 2022-07-20T03:16:54.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d973b700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 0x7f20d41520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:54.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20d9f3c700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d40b5ed0 0x7f20d4145e80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f20d000c000 tx=0x7f20d0006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d0010780 con 0x7f20d40b5ed0 2022-07-20T03:16:54.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f20d00108e0 con 0x7f20d40b5ed0 2022-07-20T03:16:54.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f20d0010c40 con 0x7f20d40b5ed0 2022-07-20T03:16:54.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f20cc009470 con 0x7f20d40b5ed0 2022-07-20T03:16:54.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.927+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f20d41525c0 con 0x7f20d40b5ed0 2022-07-20T03:16:54.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.931+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f20d40b01c0 con 0x7f20d40b5ed0 2022-07-20T03:16:54.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.931+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f20d001c460 con 0x7f20d40b5ed0 2022-07-20T03:16:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.931+0000 7f20c2ffd700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20b40539d0 0x7f20b4055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.931+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f20d006c3a0 con 0x7f20d40b5ed0 2022-07-20T03:16:54.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.931+0000 7f20d973b700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20b40539d0 0x7f20b4055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:54.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.931+0000 7f20d973b700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20b40539d0 0x7f20b4055e90 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f20c400d000 tx=0x7f20c4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:54.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:54.943+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f20d0043200 con 0x7f20d40b5ed0 2022-07-20T03:16:55.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.075+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f20d40574e0 con 0x7f20d40b5ed0 2022-07-20T03:16:55.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.075+0000 7f20c2ffd700 1 -- 172.21.15.61:0/2263101619 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v130) v1 ==== 92+0+10 (secure 0 0 0) 0x7f20d0046820 con 0x7f20d40b5ed0 2022-07-20T03:16:55.082 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:55.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20b40539d0 msgr2=0x7f20b4055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20b40539d0 0x7f20b4055e90 secure :-1 s=READY pgs=632 cs=0 l=1 rev1=1 rx=0x7f20c400d000 tx=0x7f20c4009040).stop 2022-07-20T03:16:55.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d40b5ed0 msgr2=0x7f20d4145e80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d40b5ed0 0x7f20d4145e80 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f20d000c000 tx=0x7f20d0006df0).stop 2022-07-20T03:16:55.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 shutdown_connections 2022-07-20T03:16:55.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f20b40539d0 0x7f20b4055e90 unknown :-1 s=CLOSED pgs=632 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f20d41422b0 0x7f20d409c4e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f20d40b62b0 0x7f20d41520d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 --2- 172.21.15.61:0/2263101619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f20d40b5ed0 0x7f20d4145e80 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 >> 172.21.15.61:0/2263101619 conn(0x7f20d405be30 msgr2=0x7f20d405c210 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 shutdown_connections 2022-07-20T03:16:55.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.079+0000 7f20daf3e700 1 -- 172.21.15.61:0/2263101619 wait complete. 2022-07-20T03:16:55.095 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:16:55.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.203+0000 7ff959d7b700 1 Processor -- start 2022-07-20T03:16:55.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.203+0000 7ff959d7b700 1 -- start start 2022-07-20T03:16:55.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff9540ad270 0x7ff9541437c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff9541462a0 0x7ff95413fdb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff9540b9c40 con 0x7ff9540b9860 2022-07-20T03:16:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff954143cb0 con 0x7ff9540ad270 2022-07-20T03:16:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff954142de0 con 0x7ff9541462a0 2022-07-20T03:16:55.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff952ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff9540ad270 0x7ff9541437c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff9537fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff9540b9860 0x7ff9540b4d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47242/0 (socket says 172.21.15.61:47242) 2022-07-20T03:16:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff9537fe700 1 -- 172.21.15.61:0/3791777896 learned_addr learned my addr 172.21.15.61:0/3791777896 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff953fff700 1 --2- 172.21.15.61:0/3791777896 >> v2:172.21.15.117:3300/0 conn(0x7ff9541462a0 0x7ff95413fdb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff952ffd700 1 -- 172.21.15.61:0/3791777896 >> v2:172.21.15.117:3300/0 conn(0x7ff9541462a0 msgr2=0x7ff95413fdb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:55.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff952ffd700 1 --2- 172.21.15.61:0/3791777896 >> v2:172.21.15.117:3300/0 conn(0x7ff9541462a0 0x7ff95413fdb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff952ffd700 1 -- 172.21.15.61:0/3791777896 >> v1:172.21.15.61:6789/0 conn(0x7ff9540b9860 legacy=0x7ff9540b4d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff9527fc700 1 -- 172.21.15.61:0/3791777896 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 862209429 0 0) 0x7ff9540b9c40 con 0x7ff9540b9860 2022-07-20T03:16:55.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff952ffd700 1 -- 172.21.15.61:0/3791777896 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff954143080 con 0x7ff9540ad270 2022-07-20T03:16:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff952ffd700 1 --2- 172.21.15.61:0/3791777896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540ad270 0x7ff9541437c0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7ff954142d00 tx=0x7ff944006df0).ready entity=mon.0 client_cookie=ffa0033945eb3377 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff9527fc700 1 -- 172.21.15.61:0/3791777896 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff94400d5c0 con 0x7ff9540ad270 2022-07-20T03:16:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff9527fc700 1 -- 172.21.15.61:0/3791777896 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff944000e90 con 0x7ff9540ad270 2022-07-20T03:16:55.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff9527fc700 1 -- 172.21.15.61:0/3791777896 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff94400d910 con 0x7ff9540ad270 2022-07-20T03:16:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- 172.21.15.61:0/3791777896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540ad270 msgr2=0x7ff9541437c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 --2- 172.21.15.61:0/3791777896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540ad270 0x7ff9541437c0 secure :-1 s=READY pgs=1311 cs=0 l=1 rev1=1 rx=0x7ff954142d00 tx=0x7ff944006df0).stop 2022-07-20T03:16:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- 172.21.15.61:0/3791777896 shutdown_connections 2022-07-20T03:16:55.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 --2- 172.21.15.61:0/3791777896 >> v2:172.21.15.117:3300/0 conn(0x7ff9541462a0 0x7ff95413fdb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 --2- 172.21.15.61:0/3791777896 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540ad270 0x7ff9541437c0 unknown :-1 s=CLOSED pgs=1311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- 172.21.15.61:0/3791777896 >> 172.21.15.61:0/3791777896 conn(0x7ff954066e20 msgr2=0x7ff954067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- 172.21.15.61:0/3791777896 shutdown_connections 2022-07-20T03:16:55.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.207+0000 7ff959d7b700 1 -- 172.21.15.61:0/3791777896 wait complete. 2022-07-20T03:16:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 Processor -- start 2022-07-20T03:16:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 -- start start 2022-07-20T03:16:55.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9540ad270 0x7ff9540af100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 0x7ff9540b0d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff9541462a0 0x7ff9540b1220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9540ae5e0 con 0x7ff9540b9860 2022-07-20T03:16:55.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff9540b34f0 con 0x7ff9540ad270 2022-07-20T03:16:55.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff959d7b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff9540b0340 con 0x7ff9541462a0 2022-07-20T03:16:55.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff952ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 0x7ff9540b0d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff952ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 0x7ff9540b0d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50652/0 (socket says 172.21.15.61:50652) 2022-07-20T03:16:55.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff952ffd700 1 -- 172.21.15.61:0/2727969091 learned_addr learned my addr 172.21.15.61:0/2727969091 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:55.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff9537fe700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9540ad270 0x7ff9540af100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.211+0000 7ff953fff700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff9541462a0 0x7ff9540b1220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff952ffd700 1 -- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff9541462a0 msgr2=0x7ff9540b1220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff952ffd700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff9541462a0 0x7ff9540b1220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff952ffd700 1 -- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9540ad270 msgr2=0x7ff9540af100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff952ffd700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9540ad270 0x7ff9540af100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff952ffd700 1 -- 172.21.15.61:0/2727969091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff94400b180 con 0x7ff9540b9860 2022-07-20T03:16:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff953fff700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff9541462a0 0x7ff9540b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff9537fe700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9540ad270 0x7ff9540af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:55.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff952ffd700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 0x7ff9540b0d30 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7ff954142d00 tx=0x7ff944014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff944013de0 con 0x7ff9540b9860 2022-07-20T03:16:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff944002f70 con 0x7ff9540b9860 2022-07-20T03:16:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff9540b3720 con 0x7ff9540b9860 2022-07-20T03:16:55.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff94401e5f0 con 0x7ff9540b9860 2022-07-20T03:16:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff9540b05e0 con 0x7ff9540b9860 2022-07-20T03:16:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.215+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff954143320 con 0x7ff9540b9860 2022-07-20T03:16:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.219+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff944025030 con 0x7ff9540b9860 2022-07-20T03:16:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.219+0000 7ff950ff9700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff938053ad0 0x7ff938055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.219+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff94406cf20 con 0x7ff9540b9860 2022-07-20T03:16:55.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.219+0000 7ff9537fe700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff938053ad0 0x7ff938055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.219+0000 7ff9537fe700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff938053ad0 0x7ff938055f90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7ff93c00d000 tx=0x7ff93c007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.227+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff944043d70 con 0x7ff9540b9860 2022-07-20T03:16:55.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.351+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7ff9540b7190 con 0x7ff9540b9860 2022-07-20T03:16:55.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff950ff9700 1 -- 172.21.15.61:0/2727969091 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v130) v1 ==== 104+0+10 (secure 0 0 0) 0x7ff944023020 con 0x7ff9540b9860 2022-07-20T03:16:55.359 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff938053ad0 msgr2=0x7ff938055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff938053ad0 0x7ff938055f90 secure :-1 s=READY pgs=633 cs=0 l=1 rev1=1 rx=0x7ff93c00d000 tx=0x7ff93c007a00).stop 2022-07-20T03:16:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 msgr2=0x7ff9540b0d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 0x7ff9540b0d30 secure :-1 s=READY pgs=1312 cs=0 l=1 rev1=1 rx=0x7ff954142d00 tx=0x7ff944014000).stop 2022-07-20T03:16:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 shutdown_connections 2022-07-20T03:16:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff938053ad0 0x7ff938055f90 unknown :-1 s=CLOSED pgs=633 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff9541462a0 0x7ff9540b1220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff9540b9860 0x7ff9540b0d30 unknown :-1 s=CLOSED pgs=1312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 --2- 172.21.15.61:0/2727969091 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff9540ad270 0x7ff9540af100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 >> 172.21.15.61:0/2727969091 conn(0x7ff954066e20 msgr2=0x7ff954067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 shutdown_connections 2022-07-20T03:16:55.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.355+0000 7ff959d7b700 1 -- 172.21.15.61:0/2727969091 wait complete. 2022-07-20T03:16:55.371 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:16:55.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d776d700 1 Processor -- start 2022-07-20T03:16:55.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d776d700 1 -- start start 2022-07-20T03:16:55.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d776d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00b9820 0x7fd4d0144200 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d776d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d00b9c00 con 0x7fd4d00ae5f0 2022-07-20T03:16:55.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d776d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd4d0143320 con 0x7fd4d0141cb0 2022-07-20T03:16:55.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d776d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b4400 con 0x7fd4d00b9820 2022-07-20T03:16:55.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d4d08700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd4d00ae5f0 0x7fd4d01446f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47256/0 (socket says 172.21.15.61:47256) 2022-07-20T03:16:55.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.471+0000 7fd4d4d08700 1 -- 172.21.15.61:0/1212593667 learned_addr learned my addr 172.21.15.61:0/1212593667 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:55.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d5509700 1 --2- 172.21.15.61:0/1212593667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00b9820 0x7fd4d0144200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1466478432 0 0) 0x7fd4d00b9c00 con 0x7fd4d00ae5f0 2022-07-20T03:16:55.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4b4003260 con 0x7fd4d00ae5f0 2022-07-20T03:16:55.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 520488808 0 0) 0x7fd4d0143320 con 0x7fd4d0141cb0 2022-07-20T03:16:55.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd4d00b9c00 con 0x7fd4d0141cb0 2022-07-20T03:16:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2939555119 0 0) 0x7fd4b4003260 con 0x7fd4d00ae5f0 2022-07-20T03:16:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d5509700 1 -- 172.21.15.61:0/1212593667 >> v1:172.21.15.117:6789/0 conn(0x7fd4d0141cb0 legacy=0x7fd4d0142120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d5509700 1 -- 172.21.15.61:0/1212593667 >> v1:172.21.15.61:6789/0 conn(0x7fd4d00ae5f0 legacy=0x7fd4d01446f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d5509700 1 -- 172.21.15.61:0/1212593667 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4d01435c0 con 0x7fd4d00b9820 2022-07-20T03:16:55.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d5509700 1 --2- 172.21.15.61:0/1212593667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00b9820 0x7fd4d0144200 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd4cc001110 tx=0x7fd4c0006df0).ready entity=mon.2 client_cookie=fc915cde648e8906 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4cc001660 con 0x7fd4d00b9820 2022-07-20T03:16:55.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4c0000e90 con 0x7fd4d00b9820 2022-07-20T03:16:55.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4c7fff700 1 -- 172.21.15.61:0/1212593667 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4c0012460 con 0x7fd4d00b9820 2022-07-20T03:16:55.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 -- 172.21.15.61:0/1212593667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00b9820 msgr2=0x7fd4d0144200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 --2- 172.21.15.61:0/1212593667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00b9820 0x7fd4d0144200 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7fd4cc001110 tx=0x7fd4c0006df0).stop 2022-07-20T03:16:55.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 -- 172.21.15.61:0/1212593667 shutdown_connections 2022-07-20T03:16:55.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 --2- 172.21.15.61:0/1212593667 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00b9820 0x7fd4d0144200 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 -- 172.21.15.61:0/1212593667 >> 172.21.15.61:0/1212593667 conn(0x7fd4d0069d80 msgr2=0x7fd4d0066e20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 -- 172.21.15.61:0/1212593667 shutdown_connections 2022-07-20T03:16:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 -- 172.21.15.61:0/1212593667 wait complete. 2022-07-20T03:16:55.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 Processor -- start 2022-07-20T03:16:55.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 -- start start 2022-07-20T03:16:55.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.475+0000 7fd4d776d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00ae5f0 0x7fd4d00b26c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 0x7fd4d0145880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 0x7fd4d00aff00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0112fe0 con 0x7fd4d0141cb0 2022-07-20T03:16:55.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d0114470 con 0x7fd4d0112760 2022-07-20T03:16:55.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd4d00b2e80 con 0x7fd4d00ae5f0 2022-07-20T03:16:55.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 0x7fd4d0145880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d5d0a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 0x7fd4d00aff00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d5d0a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 0x7fd4d00aff00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:60348/0 (socket says 172.21.15.61:60348) 2022-07-20T03:16:55.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d5509700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00ae5f0 0x7fd4d00b26c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d5d0a700 1 -- 172.21.15.61:0/4008744929 learned_addr learned my addr 172.21.15.61:0/4008744929 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:55.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 0x7fd4d0145880 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50666/0 (socket says 172.21.15.61:50666) 2022-07-20T03:16:55.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 -- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00ae5f0 msgr2=0x7fd4d00b26c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00ae5f0 0x7fd4d00b26c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 -- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 msgr2=0x7fd4d00aff00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 0x7fd4d00aff00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 -- 172.21.15.61:0/4008744929 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4c000b180 con 0x7fd4d0141cb0 2022-07-20T03:16:55.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d5d0a700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 0x7fd4d00aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:55.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d4d08700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 0x7fd4d0145880 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fd4cc000fe0 tx=0x7fd4cc0016b0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4cc007170 con 0x7fd4d0141cb0 2022-07-20T03:16:55.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd4cc0072d0 con 0x7fd4d0141cb0 2022-07-20T03:16:55.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd4cc004710 con 0x7fd4d0141cb0 2022-07-20T03:16:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4d00b3120 con 0x7fd4d0141cb0 2022-07-20T03:16:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4d01148d0 con 0x7fd4d0141cb0 2022-07-20T03:16:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.479+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4d0143860 con 0x7fd4d0141cb0 2022-07-20T03:16:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.483+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd4cc01b070 con 0x7fd4d0141cb0 2022-07-20T03:16:55.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.483+0000 7fd4c67fc700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4b80539b0 0x7fd4b8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.483+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd4cc06b480 con 0x7fd4d0141cb0 2022-07-20T03:16:55.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.483+0000 7fd4d5509700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4b80539b0 0x7fd4b8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.483+0000 7fd4d5509700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4b80539b0 0x7fd4b8055e70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7fd4c001a000 tx=0x7fd4c0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.487+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd4cc0422d0 con 0x7fd4d0141cb0 2022-07-20T03:16:55.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.611+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7fd4d00b7170 con 0x7fd4d0141cb0 2022-07-20T03:16:55.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4c67fc700 1 -- 172.21.15.61:0/4008744929 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v130) v1 ==== 100+0+10 (secure 0 0 0) 0x7fd4cc0458f0 con 0x7fd4d0141cb0 2022-07-20T03:16:55.620 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:16:55.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4b80539b0 msgr2=0x7fd4b8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4b80539b0 0x7fd4b8055e70 secure :-1 s=READY pgs=634 cs=0 l=1 rev1=1 rx=0x7fd4c001a000 tx=0x7fd4c0014000).stop 2022-07-20T03:16:55.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 msgr2=0x7fd4d0145880 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 0x7fd4d0145880 secure :-1 s=READY pgs=1314 cs=0 l=1 rev1=1 rx=0x7fd4cc000fe0 tx=0x7fd4cc0016b0).stop 2022-07-20T03:16:55.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 shutdown_connections 2022-07-20T03:16:55.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd4b80539b0 0x7fd4b8055e70 unknown :-1 s=CLOSED pgs=634 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd4d0112760 0x7fd4d00aff00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd4d0141cb0 0x7fd4d0145880 unknown :-1 s=CLOSED pgs=1314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 --2- 172.21.15.61:0/4008744929 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd4d00ae5f0 0x7fd4d00b26c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 >> 172.21.15.61:0/4008744929 conn(0x7fd4d0069d80 msgr2=0x7fd4d00b67b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 shutdown_connections 2022-07-20T03:16:55.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.615+0000 7fd4d776d700 1 -- 172.21.15.61:0/4008744929 wait complete. 2022-07-20T03:16:55.633 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:16:55.634 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:16:55.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.731+0000 7fc83dc0c700 1 Processor -- start 2022-07-20T03:16:55.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.731+0000 7fc83dc0c700 1 -- start start 2022-07-20T03:16:55.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc83805c880 0x7fc8380ae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8380af150 0x7fc838147170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc8380adc90 con 0x7fc8380afd00 2022-07-20T03:16:55.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc83805cc60 con 0x7fc8380af150 2022-07-20T03:16:55.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc8380adec0 con 0x7fc83805c880 2022-07-20T03:16:55.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8380af150 0x7fc838147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc8380af150 0x7fc838147170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50672/0 (socket says 172.21.15.61:50672) 2022-07-20T03:16:55.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 -- 172.21.15.61:0/1225750485 learned_addr learned my addr 172.21.15.61:0/1225750485 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:55.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc8377fe700 1 --2- 172.21.15.61:0/1225750485 >> v2:172.21.15.117:3300/0 conn(0x7fc83805c880 0x7fc8380ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 -- 172.21.15.61:0/1225750485 >> v2:172.21.15.117:3300/0 conn(0x7fc83805c880 msgr2=0x7fc8380ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:55.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 --2- 172.21.15.61:0/1225750485 >> v2:172.21.15.117:3300/0 conn(0x7fc83805c880 0x7fc8380ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 -- 172.21.15.61:0/1225750485 >> v1:172.21.15.117:6789/0 conn(0x7fc8380afd00 legacy=0x7fc838147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 -- 172.21.15.61:0/1225750485 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc8380aeaf0 con 0x7fc8380af150 2022-07-20T03:16:55.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc836ffd700 1 --2- 172.21.15.61:0/1225750485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380af150 0x7fc838147170 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7fc8380ad630 tx=0x7fc828006eb0).ready entity=mon.0 client_cookie=d26da638efc49fd0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc8367fc700 1 -- 172.21.15.61:0/1225750485 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82800d610 con 0x7fc8380af150 2022-07-20T03:16:55.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc8367fc700 1 -- 172.21.15.61:0/1225750485 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc828000e90 con 0x7fc8380af150 2022-07-20T03:16:55.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc8367fc700 1 -- 172.21.15.61:0/1225750485 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82800d960 con 0x7fc8380af150 2022-07-20T03:16:55.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- 172.21.15.61:0/1225750485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380af150 msgr2=0x7fc838147170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/1225750485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380af150 0x7fc838147170 secure :-1 s=READY pgs=1315 cs=0 l=1 rev1=1 rx=0x7fc8380ad630 tx=0x7fc828006eb0).stop 2022-07-20T03:16:55.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- 172.21.15.61:0/1225750485 shutdown_connections 2022-07-20T03:16:55.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/1225750485 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc8380af150 0x7fc838147170 unknown :-1 s=CLOSED pgs=1315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/1225750485 >> v2:172.21.15.117:3300/0 conn(0x7fc83805c880 0x7fc8380ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- 172.21.15.61:0/1225750485 >> 172.21.15.61:0/1225750485 conn(0x7fc838063460 msgr2=0x7fc83805be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- 172.21.15.61:0/1225750485 shutdown_connections 2022-07-20T03:16:55.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.735+0000 7fc83dc0c700 1 -- 172.21.15.61:0/1225750485 wait complete. 2022-07-20T03:16:55.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 Processor -- start 2022-07-20T03:16:55.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 -- start start 2022-07-20T03:16:55.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 0x7fc8381566d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8380af150 0x7fc8380b2f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8380afd00 0x7fc838064860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838145810 con 0x7fc83805c880 2022-07-20T03:16:55.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838142690 con 0x7fc8380af150 2022-07-20T03:16:55.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc83dc0c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc838156cc0 con 0x7fc8380afd00 2022-07-20T03:16:55.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc8377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 0x7fc8381566d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc8377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 0x7fc8381566d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50678/0 (socket says 172.21.15.61:50678) 2022-07-20T03:16:55.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc8377fe700 1 -- 172.21.15.61:0/4042267047 learned_addr learned my addr 172.21.15.61:0/4042267047 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:55.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc836ffd700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8380af150 0x7fc8380b2f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.739+0000 7fc837fff700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8380afd00 0x7fc838064860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc8377fe700 1 -- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8380afd00 msgr2=0x7fc838064860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc8377fe700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8380afd00 0x7fc838064860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc8377fe700 1 -- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8380af150 msgr2=0x7fc8380b2f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc8377fe700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8380af150 0x7fc8380b2f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc8377fe700 1 -- 172.21.15.61:0/4042267047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc82800b1e0 con 0x7fc83805c880 2022-07-20T03:16:55.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc837fff700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8380afd00 0x7fc838064860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:16:55.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc8377fe700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 0x7fc8381566d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7fc82c00a000 tx=0x7fc82c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82c0106e0 con 0x7fc83805c880 2022-07-20T03:16:55.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc82c010840 con 0x7fc83805c880 2022-07-20T03:16:55.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc838156f60 con 0x7fc83805c880 2022-07-20T03:16:55.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc82c010ba0 con 0x7fc83805c880 2022-07-20T03:16:55.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc8381576a0 con 0x7fc83805c880 2022-07-20T03:16:55.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc82c01a030 con 0x7fc83805c880 2022-07-20T03:16:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc834ff9700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc818053940 0x7fc818055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc82c06baf0 con 0x7fc83805c880 2022-07-20T03:16:55.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc836ffd700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc818053940 0x7fc818055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc8380624c0 con 0x7fc83805c880 2022-07-20T03:16:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.743+0000 7fc836ffd700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc818053940 0x7fc818055e00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7fc82801c000 tx=0x7fc828015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:55.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.751+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc82c042950 con 0x7fc83805c880 2022-07-20T03:16:55.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.911+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fc8380af7f0 con 0x7fc83805c880 2022-07-20T03:16:55.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc834ff9700 1 -- 172.21.15.61:0/4042267047 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2142 (secure 0 0 0) 0x7fc82c045f70 con 0x7fc83805c880 2022-07-20T03:16:55.918 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:55.919 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:16:39.468063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1435464119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1435464119},{"type":"v1","addr":"172.21.15.61:6833","nonce":1435464119}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:16:55.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc818053940 msgr2=0x7fc818055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc818053940 0x7fc818055e00 secure :-1 s=READY pgs=635 cs=0 l=1 rev1=1 rx=0x7fc82801c000 tx=0x7fc828015000).stop 2022-07-20T03:16:55.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 msgr2=0x7fc8381566d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:55.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 0x7fc8381566d0 secure :-1 s=READY pgs=1316 cs=0 l=1 rev1=1 rx=0x7fc82c00a000 tx=0x7fc82c008040).stop 2022-07-20T03:16:55.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 shutdown_connections 2022-07-20T03:16:55.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc818053940 0x7fc818055e00 unknown :-1 s=CLOSED pgs=635 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc8380af150 0x7fc8380b2f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc8380afd00 0x7fc838064860 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 --2- 172.21.15.61:0/4042267047 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc83805c880 0x7fc8381566d0 unknown :-1 s=CLOSED pgs=1316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:55.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 >> 172.21.15.61:0/4042267047 conn(0x7fc838063460 msgr2=0x7fc83805be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:55.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 shutdown_connections 2022-07-20T03:16:55.923 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:55.915+0000 7fc83dc0c700 1 -- 172.21.15.61:0/4042267047 wait complete. 2022-07-20T03:16:55.923 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-20T03:16:56.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 72, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:16:38.464416+0000', 'modified': '2022-07-20T03:16:39.468063+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8091}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8091': {'gid': 8091, 'name': 'a', 'rank': 0, 'incarnation': 71, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/1435464119', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 1435464119}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 1435464119}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [26], 'metadata_pool': 25, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:16:56.133 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:16:56.134 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:16:56.253 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8091,"want_state":"up:active","state":"up:active","rank_uptime":17.776792457999999,"mdsmap_epoch":72,"osdmap_epoch":130,"osdmap_epoch_barrier":130,"uptime":22.744956599999998} 2022-07-20T03:16:56.254 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8091, "mdsmap_epoch": 72, "osdmap_epoch": 130, "osdmap_epoch_barrier": 130, "rank_uptime": 17.776792458, "state": "up:active", "uptime": 22.7449566, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:16:56.255 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:16:56.255 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:16:56.256 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:16:56.256 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:56.257 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:56.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:56.283 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:56.283 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:56.296 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:56.297 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:56.298 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:56.298 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:56.348 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:16:56.348 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:56.351 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:56.369 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:56.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:56.380 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:56.381 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:56.382 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:56.383 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:56.429 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:16:56.422+0000 7f2814cc0080 -1 init, newargv = 0x55c4a068d4d0 newargc=9 2022-07-20T03:16:56.430 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22931]: starting ceph client 2022-07-20T03:16:56.438 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22931]: starting fuse 2022-07-20T03:16:57.534 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:16:57.554 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:16:57.554 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:16:57.565 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:16:57.567 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:16:57.568 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:57.569 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:16:57.616 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:16:57.616 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:16:57.617 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:57.617 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:16:57.618 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:16:57.618 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:16:57.619 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:16:57.619 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:16:57.620 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:57.620 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:16:57.620 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:16:57.621 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:16:57.622 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:57.622 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:16:57.623 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:16:57.623 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:16:57.623 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:16:57.624 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:57.624 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:16:57.625 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:16:57.625 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:16:57.626 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:16:57.626 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:16:57.627 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:16:57.627 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:16:57.627 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:16:57.673 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.22931.asok 2022-07-20T03:16:57.678 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.22931.asok 2022-07-20T03:16:57.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:57.682 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.22931.asok status 2022-07-20T03:16:57.824 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:16:57.825 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:16:57.825 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:16:57.825 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:16:57.826 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:16:57.826 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:16:57.827 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:16:57.827 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "22931", 2022-07-20T03:16:57.827 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:16:57.827 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:57.828 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:16:57.828 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:16:57.828 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 8239, 2022-07-20T03:16:57.828 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:16:57.829 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:16:57.829 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:16:57.829 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 8239 2022-07-20T03:16:57.829 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:57.830 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:16:57.830 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:16:57.830 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:16:57.831 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2877867601 2022-07-20T03:16:57.831 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:16:57.831 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:57.831 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:16:57.832 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:16:57.832 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:16:57.832 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2877867601 2022-07-20T03:16:57.832 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:16:57.833 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.8239 172.21.15.163:0/2877867601", 2022-07-20T03:16:57.833 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/2877867601", 2022-07-20T03:16:57.833 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:16:57.833 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 72, 2022-07-20T03:16:57.834 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 130, 2022-07-20T03:16:57.834 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:16:57.834 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:16:57.834 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:16:57.835 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:16:57.836 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:57.837 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:57.837 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:57.845 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:57.847 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:57.848 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:57.848 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:57.867 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:57.869 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:57.869 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:57.880 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:16:57.881 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:57.882 DEBUG:teuthology.orchestra.run.smithi124:> sudo adjust-ulimits daemon-helper kill python3 -c ' 2022-07-20T03:16:57.883 DEBUG:teuthology.orchestra.run.smithi124:> import time 2022-07-20T03:16:57.884 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:16:57.885 DEBUG:teuthology.orchestra.run.smithi124:> with open("/home/ubuntu/cephtest/mnt.0/testfile", '"'"'w'"'"') as f: 2022-07-20T03:16:57.886 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content'"'"') 2022-07-20T03:16:57.886 DEBUG:teuthology.orchestra.run.smithi124:> f.flush() 2022-07-20T03:16:57.887 DEBUG:teuthology.orchestra.run.smithi124:> f.write('"'"'content2'"'"') 2022-07-20T03:16:57.888 DEBUG:teuthology.orchestra.run.smithi124:> while True: 2022-07-20T03:16:57.889 DEBUG:teuthology.orchestra.run.smithi124:> time.sleep(1) 2022-07-20T03:16:57.891 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:16:57.896 DEBUG:teuthology.orchestra.run.smithi124:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-20T03:16:57.917 INFO:teuthology.orchestra.run.smithi124.stderr:ls: cannot access '/home/ubuntu/cephtest/mnt.0/testfile': No such file or directory 2022-07-20T03:16:57.918 DEBUG:teuthology.orchestra.run:got remote process result: 2 2022-07-20T03:16:58.920 DEBUG:teuthology.orchestra.run.smithi124:> sudo ls /home/ubuntu/cephtest/mnt.0/testfile 2022-07-20T03:16:58.940 INFO:teuthology.orchestra.run.smithi124.stdout:/home/ubuntu/cephtest/mnt.0/testfile 2022-07-20T03:16:58.941 DEBUG:tasks.cephfs.mount:File testfile became visible from 0 after 1s 2022-07-20T03:16:58.942 DEBUG:teuthology.orchestra.run.smithi163:> sudo ls /home/ubuntu/cephtest/mnt.1/testfile 2022-07-20T03:16:58.979 INFO:teuthology.orchestra.run.smithi163.stdout:/home/ubuntu/cephtest/mnt.1/testfile 2022-07-20T03:16:58.981 DEBUG:tasks.cephfs.mount:File testfile became visible from 1 after 0s 2022-07-20T03:16:58.982 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T03:16:58.983 DEBUG:teuthology.orchestra.run.smithi163:> cd /home/ubuntu/cephtest/mnt.1 && sudo cat testfile 2022-07-20T03:16:59.024 INFO:teuthology.orchestra.run.smithi163.stdout:content 2022-07-20T03:16:59.041 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:16:59.042 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:59.064 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:59.087 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:59.109 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:59.131 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:16:59.152 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:16:59.152 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:59.153 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:59.154 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:59.163 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:16:59.165 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:16:59.165 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:16:59.166 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:16:59.166 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:59.182 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23445]: fuse finished with error 0 and tester_r 0 2022-07-20T03:16:59.184 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:59.185 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:59.186 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:59.237 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:16:59.239 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:16:59.318 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:59.319 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:16:59.336 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:59.337 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:59.337 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:59.347 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:16:59.348 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:16:59.348 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:16:59.349 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:16:59.350 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:59.404 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[22931]: fuse finished with error 0 and tester_r 0 2022-07-20T03:16:59.406 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:59.407 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:59.407 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:59.414 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:16:59.416 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:16:59.533 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:59.534 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:16:59.543 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:16:59.543 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:16:59.544 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:16:59.551 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:16:59.553 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:59.555 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:16:59.557 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:16:59.557 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:16:59.565 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:16:59.565 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:16:59.567 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:16:59.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 Processor -- start 2022-07-20T03:16:59.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 -- start start 2022-07-20T03:16:59.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcbac0afbd0 0x7fcbac14ac40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:59.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcbac0ae390 0x7fcbac0ac590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:59.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcbac0b5650 con 0x7fcbac0b5270 2022-07-20T03:16:59.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac0ad9b0 con 0x7fcbac0afbd0 2022-07-20T03:16:59.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbab59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcbac0adc50 con 0x7fcbac0ae390 2022-07-20T03:16:59.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbaa59c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcbac0b5270 0x7fcbac14a530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47284/0 (socket says 172.21.15.61:47284) 2022-07-20T03:16:59.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbaa59c700 1 -- 172.21.15.61:0/503987733 learned_addr learned my addr 172.21.15.61:0/503987733 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:16:59.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcba9d9b700 1 --2- 172.21.15.61:0/503987733 >> v2:172.21.15.61:3300/0 conn(0x7fcbac0afbd0 0x7fcbac14ac40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:59.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.675+0000 7fcbaad9d700 1 --2- 172.21.15.61:0/503987733 >> v2:172.21.15.117:3300/0 conn(0x7fcbac0ae390 0x7fcbac0ac590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:59.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba959a700 1 -- 172.21.15.61:0/503987733 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3755167379 0 0) 0x7fcbac0b5650 con 0x7fcbac0b5270 2022-07-20T03:16:59.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba959a700 1 -- 172.21.15.61:0/503987733 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcb90003260 con 0x7fcbac0b5270 2022-07-20T03:16:59.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba9d9b700 1 -- 172.21.15.61:0/503987733 >> v2:172.21.15.117:3300/0 conn(0x7fcbac0ae390 msgr2=0x7fcbac0ac590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:16:59.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba9d9b700 1 --2- 172.21.15.61:0/503987733 >> v2:172.21.15.117:3300/0 conn(0x7fcbac0ae390 0x7fcbac0ac590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:59.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba9d9b700 1 -- 172.21.15.61:0/503987733 >> v1:172.21.15.61:6789/0 conn(0x7fcbac0b5270 legacy=0x7fcbac14a530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:59.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba9d9b700 1 -- 172.21.15.61:0/503987733 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbac0adef0 con 0x7fcbac0afbd0 2022-07-20T03:16:59.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbaad9d700 1 --2- 172.21.15.61:0/503987733 >> v2:172.21.15.117:3300/0 conn(0x7fcbac0ae390 0x7fcbac0ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:16:59.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba9d9b700 1 --2- 172.21.15.61:0/503987733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0afbd0 0x7fcbac14ac40 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7fcba0002f70 tx=0x7fcba0006df0).ready entity=mon.0 client_cookie=73ce041865a7779d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:59.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba959a700 1 -- 172.21.15.61:0/503987733 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcba000d4f0 con 0x7fcbac0afbd0 2022-07-20T03:16:59.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba959a700 1 -- 172.21.15.61:0/503987733 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba0000e90 con 0x7fcbac0afbd0 2022-07-20T03:16:59.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcba959a700 1 -- 172.21.15.61:0/503987733 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcba000d840 con 0x7fcbac0afbd0 2022-07-20T03:16:59.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 -- 172.21.15.61:0/503987733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0afbd0 msgr2=0x7fcbac14ac40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:59.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 --2- 172.21.15.61:0/503987733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0afbd0 0x7fcbac14ac40 secure :-1 s=READY pgs=1318 cs=0 l=1 rev1=1 rx=0x7fcba0002f70 tx=0x7fcba0006df0).stop 2022-07-20T03:16:59.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 -- 172.21.15.61:0/503987733 shutdown_connections 2022-07-20T03:16:59.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 --2- 172.21.15.61:0/503987733 >> v2:172.21.15.117:3300/0 conn(0x7fcbac0ae390 0x7fcbac0ac590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:59.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 --2- 172.21.15.61:0/503987733 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0afbd0 0x7fcbac14ac40 unknown :-1 s=CLOSED pgs=1318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:59.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 -- 172.21.15.61:0/503987733 >> 172.21.15.61:0/503987733 conn(0x7fcbac062940 msgr2=0x7fcbac062d20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:16:59.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 -- 172.21.15.61:0/503987733 shutdown_connections 2022-07-20T03:16:59.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.679+0000 7fcbab59e700 1 -- 172.21.15.61:0/503987733 wait complete. 2022-07-20T03:16:59.698 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 Processor -- start 2022-07-20T03:16:59.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 -- start start 2022-07-20T03:16:59.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 0x7fcbac156cc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:59.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbac0afbd0 0x7fcbac0a69d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:59.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbac0b5270 0x7fcbac157580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:59.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac0662f0 con 0x7fcbac0ae390 2022-07-20T03:16:59.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac1571b0 con 0x7fcbac0b5270 2022-07-20T03:16:59.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbab59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcbac066520 con 0x7fcbac0afbd0 2022-07-20T03:16:59.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 0x7fcbac156cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:59.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 0x7fcbac156cc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50692/0 (socket says 172.21.15.61:50692) 2022-07-20T03:16:59.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 -- 172.21.15.61:0/3017486860 learned_addr learned my addr 172.21.15.61:0/3017486860 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:16:59.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaad9d700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbac0b5270 0x7fcbac157580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:59.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcba9d9b700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbac0afbd0 0x7fcbac0a69d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:59.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 -- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbac0afbd0 msgr2=0x7fcbac0a69d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:59.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbac0afbd0 0x7fcbac0a69d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:59.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 -- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbac0b5270 msgr2=0x7fcbac157580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:16:59.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbac0b5270 0x7fcbac157580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:16:59.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 -- 172.21.15.61:0/3017486860 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb9c00b180 con 0x7fcbac0ae390 2022-07-20T03:16:59.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcbaa59c700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 0x7fcbac156cc0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7fcba4001490 tx=0x7fcba4007e20).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:59.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcba4005ba0 con 0x7fcbac0ae390 2022-07-20T03:16:59.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.683+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcba4005d00 con 0x7fcbac0ae390 2022-07-20T03:16:59.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcba000b180 con 0x7fcbac0ae390 2022-07-20T03:16:59.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcba40072c0 con 0x7fcbac0ae390 2022-07-20T03:16:59.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbac0a86b0 con 0x7fcbac0ae390 2022-07-20T03:16:59.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbac066090 con 0x7fcbac0ae390 2022-07-20T03:16:59.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcba4010070 con 0x7fcbac0ae390 2022-07-20T03:16:59.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcb9b7fe700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb88053ad0 0x7fcb88055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:16:59.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcba406b8b0 con 0x7fcbac0ae390 2022-07-20T03:16:59.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcba9d9b700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb88053ad0 0x7fcb88055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:16:59.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.687+0000 7fcba9d9b700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb88053ad0 0x7fcb88055f90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fcba0018000 tx=0x7fcba0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:16:59.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.691+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcba4042810 con 0x7fcbac0ae390 2022-07-20T03:16:59.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.823+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fcbac145f90 con 0x7fcbac0ae390 2022-07-20T03:16:59.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcb9b7fe700 1 -- 172.21.15.61:0/3017486860 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v130) v1 ==== 81+0+33707 (secure 0 0 0) 0x7fcba4045e30 con 0x7fcbac0ae390 2022-07-20T03:16:59.832 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:16:59.833 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:16:59.833 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 130, 2022-07-20T03:16:59.833 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:16:59.833 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:16:59.834 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:16:39.466964+0000", 2022-07-20T03:16:59.834 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:16:59.835 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:16:59.835 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:16:59.835 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:16:59.836 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:16:59.836 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:16:59.836 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:16:59.837 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:16:59.837 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:16:59.837 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:59.837 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:16:59.838 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:16:59.838 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:16:59.839 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:16:59.840 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:16:59.841 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 26, 2022-07-20T03:16:59.841 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:16:59.841 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:16:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:16:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:16:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:16:59.842 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.843 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:16:59.843 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:16:59.843 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:16:59.843 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:59.844 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:59.845 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:59.845 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:59.845 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:59.845 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:16:59.846 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:16:59.846 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:16:59.846 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:16:59.846 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:16:59.847 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:59.847 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:59.847 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:59.847 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:59.848 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:59.848 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:59.848 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:59.848 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:16:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:59.849 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:59.850 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:59.851 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:59.851 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:59.851 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:59.851 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:59.852 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:59.852 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:59.852 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:59.852 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:59.853 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:59.854 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:59.854 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:59.855 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:59.855 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:59.855 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:59.855 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.856 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:59.856 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:59.856 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:59.856 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:59.857 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:59.858 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:59.858 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:59.858 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:16:59.858 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:16:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:16:59.859 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:16:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:16:59.860 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:16:59.861 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:59.861 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:59.861 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:59.861 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:59.862 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:59.862 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:59.862 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:59.863 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:59.863 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:59.863 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:59.863 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:59.864 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:59.865 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:16:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:59.866 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:59.867 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:59.867 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:59.867 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:59.867 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:59.868 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:59.869 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:59.870 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:59.870 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:59.870 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:59.870 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:59.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:59.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:59.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:59.871 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:59.872 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:59.873 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:59.874 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:59.875 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:59.875 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:59.875 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:59.875 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:16:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:16:59.876 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 25, 2022-07-20T03:16:59.877 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:16:59.878 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:16:36.305597+0000", 2022-07-20T03:16:59.878 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:59.878 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:59.878 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:59.879 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:59.879 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:59.879 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:59.880 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:59.880 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:59.880 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:59.881 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:59.881 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:59.881 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:59.881 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:59.882 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:59.882 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:59.882 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:59.882 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:59.883 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:59.883 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:59.883 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:59.883 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.883 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "129", 2022-07-20T03:16:59.884 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:59.884 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:59.884 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:59.884 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:59.885 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:59.885 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:59.885 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:59.885 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:59.886 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:59.886 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:59.886 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:59.886 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:59.887 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:59.887 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:59.887 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:59.887 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:59.888 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:59.888 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:59.888 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:59.888 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:59.889 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:59.889 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:59.889 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:59.889 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:59.890 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:59.890 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:59.890 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.891 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:59.891 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:59.891 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:59.891 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:59.892 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:59.892 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:59.892 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:59.892 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:59.892 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:59.893 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:59.893 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:59.893 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:16:59.893 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:16:59.894 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:16:59.894 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:16:59.894 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.894 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:59.895 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:16:59.895 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:16:59.895 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.895 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.895 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.896 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.896 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 26, 2022-07-20T03:16:59.896 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:16:59.897 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:16:37.477547+0000", 2022-07-20T03:16:59.897 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:16:59.897 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:16:59.897 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:16:59.898 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:16:59.898 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:16:59.898 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:16:59.898 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:16:59.899 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:16:59.899 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:16:59.899 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:16:59.900 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:16:59.900 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:16:59.900 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:16:59.900 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:16:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:16:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:16:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:16:59.901 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:16:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:16:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:16:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.902 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "130", 2022-07-20T03:16:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:16:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:16:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:16:59.903 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:16:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:16:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:16:59.904 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:16:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:16:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:16:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:16:59.905 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:16:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:16:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:16:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:16:59.906 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:16:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:16:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:16:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:16:59.907 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:16:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:16:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:16:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:16:59.908 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:16:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:16:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:16:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:16:59.909 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:16:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:16:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:16:59.910 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:16:59.911 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:16:59.911 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:16:59.911 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:16:59.912 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:16:59.912 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:16:59.912 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:16:59.912 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:16:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:16:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:16:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:16:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:16:59.913 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.914 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.914 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.914 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:16:59.914 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:16:59.915 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.915 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:16:59.915 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:16:59.915 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:59.916 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:59.916 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:59.916 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:59.916 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 127, 2022-07-20T03:16:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:59.917 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:59.918 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:59.918 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.918 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.918 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.919 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:16:59.919 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.919 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.919 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.920 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.920 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:16:59.920 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.920 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.921 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.921 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.921 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:59.921 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.922 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.922 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.922 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:16:59.923 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.923 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.923 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:16:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.924 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.925 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.925 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.925 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:59.925 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.926 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.926 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.926 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:16:59.926 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.927 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.927 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.927 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.927 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:16:59.928 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.928 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.928 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.929 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.929 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:59.929 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.929 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.930 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.930 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:16:59.930 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.930 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.931 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.931 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.931 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:16:59.931 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:16:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:16:59.932 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:16:59.933 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:16:59.933 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:16:59.933 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:59.933 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:59.934 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:59.934 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.934 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.934 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:16:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:16:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:59.935 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:59.936 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:59.937 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:16:59.937 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:59.937 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:59.937 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:59.938 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.938 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.938 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.938 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:16:59.939 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.939 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.939 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.939 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:16:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.940 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.941 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.941 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:59.941 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.941 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.942 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.942 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:16:59.942 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.942 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.943 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.943 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.943 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:16:59.943 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.943 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.944 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.944 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.944 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:59.944 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.945 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.945 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.945 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:16:59.945 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.946 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.946 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.946 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.946 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:16:59.946 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.947 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.947 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.947 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.947 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:59.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.948 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.948 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.948 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:16:59.949 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.949 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.949 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.949 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.949 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:16:59.950 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:16:59.950 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.950 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.950 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.951 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:16:59.951 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:16:59.951 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:16:59.951 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:16:59.952 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:59.952 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:59.952 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:59.952 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.952 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.953 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.953 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:16:59.953 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:16:59.953 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:59.954 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:59.954 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:59.954 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:59.955 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:59.955 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:59.955 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:59.955 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:16:59.956 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:59.956 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:59.956 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:59.956 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.956 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.957 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.957 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:16:59.957 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.957 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.958 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.958 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.958 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:16:59.958 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.959 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.959 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.959 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.959 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:59.959 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.960 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.960 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.960 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:16:59.960 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.961 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.961 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.961 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.962 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:16:59.962 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.962 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.962 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.963 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.963 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:59.963 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.963 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.963 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.964 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:16:59.964 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.964 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.964 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.965 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.965 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:16:59.965 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.965 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.966 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.966 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.966 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:59.966 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.966 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.967 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.967 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:16:59.967 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.967 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.968 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.968 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.968 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:16:59.968 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:16:59.969 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.969 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.969 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.969 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:16:59.970 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:16:59.970 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:16:59.970 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:16:59.970 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:59.971 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:59.971 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:59.971 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.971 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:16:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:16:59.972 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:59.973 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:59.974 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:59.974 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 127, 2022-07-20T03:16:59.974 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:59.974 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.975 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:16:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.976 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:16:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.977 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.978 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.978 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:59.978 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.978 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:16:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.979 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.980 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.980 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:16:59.980 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.980 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.981 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.981 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.981 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:16:59.981 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.982 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.982 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.982 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:16:59.982 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:16:59.983 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.984 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.984 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.984 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.984 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:16:59.985 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.985 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.985 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.985 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:16:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.986 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:16:59.987 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:16:59.987 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.987 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.987 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:16:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:16:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:16:59.988 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:16:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:16:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:16:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:16:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.989 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.990 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.990 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:16:59.990 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:16:59.990 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:16:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:16:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:16:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:16:59.991 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:16:59.992 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:16:59.992 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:16:59.992 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:16:59.992 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:16:59.993 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:16:59.993 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:16:59.993 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.993 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:16:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.994 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:16:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:59.995 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:16:59.996 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:16:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:16:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:16:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:59.997 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:16:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:16:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:16:59.998 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:16:59.999 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:16:59.999 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:16:59.999 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:16:59.999 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.000 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:17:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.001 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:17:00.002 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:00.002 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.002 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.002 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.003 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:00.003 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.003 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.003 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.004 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:17:00.004 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:00.004 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.004 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.004 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.005 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:17:00.005 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:00.005 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.005 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.006 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.006 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:17:00.006 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:17:00.006 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:17:00.007 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:17:00.007 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:00.007 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:00.007 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:00.007 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.008 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.008 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.008 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:00.008 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:17:00.009 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:00.009 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:00.009 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:00.009 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:00.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:00.010 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:00.010 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:00.010 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:17:00.010 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:00.011 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:00.011 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:00.011 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.011 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.012 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.012 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:17:00.012 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.012 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.013 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.013 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.013 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:17:00.013 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.013 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.014 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.014 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.014 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:00.014 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.015 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.015 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.015 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:17:00.015 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:17:00.016 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.017 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:00.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.018 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.018 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.018 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:17:00.019 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.019 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.019 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.019 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.020 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:17:00.020 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.021 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.021 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.021 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.021 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:00.022 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.022 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.022 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.022 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:17:00.023 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.023 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.023 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.023 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.024 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:17:00.024 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:00.024 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.025 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.025 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.025 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:17:00.025 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:17:00.026 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:17:00.026 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:17:00.026 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:00.027 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:00.027 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:00.027 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.027 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.027 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.028 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:00.028 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:17:00.028 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:00.028 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:00.029 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:00.029 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:00.029 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:00.029 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:00.030 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:00.030 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:00.030 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:00.030 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:00.031 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:00.031 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.032 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.032 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.033 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:17:00.033 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.033 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.034 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.034 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.034 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:17:00.034 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.035 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.035 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.035 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.035 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:00.036 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.036 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.036 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.037 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:17:00.037 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.037 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.037 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.038 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.038 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:17:00.038 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.038 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.039 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.039 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.039 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:00.039 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.040 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.040 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.040 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:17:00.040 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.041 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:17:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.042 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.043 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:00.043 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.043 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.043 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:17:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.044 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:17:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:00.045 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.046 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.046 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.046 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:17:00.046 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:17:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:17:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:17:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:00.047 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:00.048 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.048 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.048 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.048 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:17:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:00.049 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:00.050 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:00.050 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:00.050 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:00.050 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:00.050 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:00.051 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:00.051 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:00.051 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:00.051 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.052 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.052 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.052 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:17:00.052 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.053 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.053 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.053 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.053 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:17:00.053 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.054 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.054 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.054 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.054 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:00.055 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.055 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.055 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.055 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:17:00.056 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.056 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.056 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.056 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.056 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:17:00.057 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.057 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.057 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.057 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.058 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:00.058 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.058 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.058 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.059 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:17:00.059 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.059 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.059 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.060 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.060 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:17:00.060 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.060 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.060 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.061 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.061 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:00.061 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:00.061 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.062 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:00.062 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:17:00.062 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.062 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.063 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.063 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:00.063 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:17:00.063 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:00.063 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.064 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.064 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.064 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:17:00.064 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:17:00.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:17:00.065 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:17:00.065 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:00.065 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:00.066 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:00.066 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:00.066 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.066 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:17:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.067 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.068 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.068 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.068 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.068 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:17:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.069 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.070 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:17:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.071 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.072 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:00.072 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.072 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.073 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:17:00.074 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.074 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.074 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.074 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:00.075 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.075 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.075 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.075 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:17:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.076 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.077 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.077 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:00.077 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.078 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.078 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.078 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.078 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.079 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:17:00.079 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.079 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.079 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.080 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:00.080 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.080 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.080 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.081 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.081 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.081 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:17:00.081 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.082 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.082 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.082 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:00.082 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.083 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.083 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.083 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.083 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.084 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:17:00.084 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.084 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.084 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:00.085 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:00.085 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:00.085 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:00.085 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:00.086 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:00.086 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:00.086 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:17:00.086 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:00.087 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.087 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:00.087 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:17:00.087 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:17:00.088 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:17:00.088 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:17:00.088 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:17:00.088 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:17:00.089 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:17:00.089 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:17:00.089 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:17:00.090 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:17:00.090 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:17:00.090 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:17:00.090 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:17:00.091 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:00.091 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:00.091 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:17:00.091 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:17:00.091 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:17:00.092 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:17:00.092 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:17:00.092 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:00.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb88053ad0 msgr2=0x7fcb88055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb88053ad0 0x7fcb88055f90 secure :-1 s=READY pgs=636 cs=0 l=1 rev1=1 rx=0x7fcba0018000 tx=0x7fcba0014000).stop 2022-07-20T03:17:00.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 msgr2=0x7fcbac156cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 0x7fcbac156cc0 secure :-1 s=READY pgs=1319 cs=0 l=1 rev1=1 rx=0x7fcba4001490 tx=0x7fcba4007e20).stop 2022-07-20T03:17:00.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 shutdown_connections 2022-07-20T03:17:00.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcb88053ad0 0x7fcb88055f90 unknown :-1 s=CLOSED pgs=636 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.827+0000 7fcbab59e700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcbac0b5270 0x7fcbac157580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.831+0000 7fcbab59e700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcbac0afbd0 0x7fcbac0a69d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.831+0000 7fcbab59e700 1 --2- 172.21.15.61:0/3017486860 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcbac0ae390 0x7fcbac156cc0 unknown :-1 s=CLOSED pgs=1319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.831+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 >> 172.21.15.61:0/3017486860 conn(0x7fcbac062940 msgr2=0x7fcbac0b17e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:00.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.831+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 shutdown_connections 2022-07-20T03:17:00.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:16:59.831+0000 7fcbab59e700 1 -- 172.21.15.61:0/3017486860 wait complete. 2022-07-20T03:17:00.098 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:00.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 Processor -- start 2022-07-20T03:17:00.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- start start 2022-07-20T03:17:00.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c05c880 0x7efd1c0ae600 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efd1c0af150 0x7efd1c147170 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7efd1c0adc90 con 0x7efd1c0afd00 2022-07-20T03:17:00.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7efd1c05cc60 con 0x7efd1c0af150 2022-07-20T03:17:00.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0adec0 con 0x7efd1c05c880 2022-07-20T03:17:00.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efd1c0af150 0x7efd1c147170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7efd1c0af150 0x7efd1c147170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50700/0 (socket says 172.21.15.61:50700) 2022-07-20T03:17:00.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 -- 172.21.15.61:0/140038356 learned_addr learned my addr 172.21.15.61:0/140038356 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:00.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1b7fe700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c05c880 0x7efd1c0ae600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 -- 172.21.15.61:0/140038356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c05c880 msgr2=0x7efd1c0ae600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:00.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c05c880 0x7efd1c0ae600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 -- 172.21.15.61:0/140038356 >> v1:172.21.15.117:6789/0 conn(0x7efd1c0afd00 legacy=0x7efd1c147660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 -- 172.21.15.61:0/140038356 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd1c0aeaf0 con 0x7efd1c0af150 2022-07-20T03:17:00.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1b7fe700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c05c880 0x7efd1c0ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:00.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1affd700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c0af150 0x7efd1c147170 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7efd0c002f70 tx=0x7efd0c006eb0).ready entity=mon.0 client_cookie=172c6277e938381c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:00.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1a7fc700 1 -- 172.21.15.61:0/140038356 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd0c00d5c0 con 0x7efd1c0af150 2022-07-20T03:17:00.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1a7fc700 1 -- 172.21.15.61:0/140038356 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd0c000e90 con 0x7efd1c0af150 2022-07-20T03:17:00.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd1a7fc700 1 -- 172.21.15.61:0/140038356 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd0c00d950 con 0x7efd1c0af150 2022-07-20T03:17:00.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- 172.21.15.61:0/140038356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c0af150 msgr2=0x7efd1c147170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c0af150 0x7efd1c147170 secure :-1 s=READY pgs=1320 cs=0 l=1 rev1=1 rx=0x7efd0c002f70 tx=0x7efd0c006eb0).stop 2022-07-20T03:17:00.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- 172.21.15.61:0/140038356 shutdown_connections 2022-07-20T03:17:00.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c0af150 0x7efd1c147170 unknown :-1 s=CLOSED pgs=1320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 --2- 172.21.15.61:0/140038356 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c05c880 0x7efd1c0ae600 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- 172.21.15.61:0/140038356 >> 172.21.15.61:0/140038356 conn(0x7efd1c063460 msgr2=0x7efd1c05be30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:00.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- 172.21.15.61:0/140038356 shutdown_connections 2022-07-20T03:17:00.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.203+0000 7efd2222c700 1 -- 172.21.15.61:0/140038356 wait complete. 2022-07-20T03:17:00.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 Processor -- start 2022-07-20T03:17:00.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 -- start start 2022-07-20T03:17:00.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 0x7efd1c14aff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1c0af150 0x7efd1c159e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c0afd00 0x7efd1c15a380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0ab180 con 0x7efd1c05c880 2022-07-20T03:17:00.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c159a30 con 0x7efd1c0af150 2022-07-20T03:17:00.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd2222c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7efd1c0ab3b0 con 0x7efd1c0afd00 2022-07-20T03:17:00.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 0x7efd1c14aff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 0x7efd1c14aff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50706/0 (socket says 172.21.15.61:50706) 2022-07-20T03:17:00.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1c0af150 0x7efd1c159e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 -- 172.21.15.61:0/3374579070 learned_addr learned my addr 172.21.15.61:0/3374579070 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:00.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1bfff700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c0afd00 0x7efd1c15a380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 -- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c0afd00 msgr2=0x7efd1c15a380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c0afd00 0x7efd1c15a380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 -- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1c0af150 msgr2=0x7efd1c159e90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1c0af150 0x7efd1c159e90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.207+0000 7efd1b7fe700 1 -- 172.21.15.61:0/3374579070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd0c00b1e0 con 0x7efd1c05c880 2022-07-20T03:17:00.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd1b7fe700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 0x7efd1c14aff0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7efd10006940 tx=0x7efd10008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:00.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd10011780 con 0x7efd1c05c880 2022-07-20T03:17:00.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7efd100118e0 con 0x7efd1c05c880 2022-07-20T03:17:00.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7efd10011c40 con 0x7efd1c05c880 2022-07-20T03:17:00.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd1c15a870 con 0x7efd1c05c880 2022-07-20T03:17:00.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd1c063940 con 0x7efd1c05c880 2022-07-20T03:17:00.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd1c0ae160 con 0x7efd1c05c880 2022-07-20T03:17:00.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7efd1001c460 con 0x7efd1c05c880 2022-07-20T03:17:00.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd18ff9700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcfc053920 0x7efcfc055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7efd1006beb0 con 0x7efd1c05c880 2022-07-20T03:17:00.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.211+0000 7efd1affd700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcfc053920 0x7efcfc055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.215+0000 7efd1affd700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcfc053920 0x7efcfc055de0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7efd0c01c000 tx=0x7efd0c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:00.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.219+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7efd10042db0 con 0x7efd1c05c880 2022-07-20T03:17:00.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7efd1c0af7f0 con 0x7efd1c05c880 2022-07-20T03:17:00.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd18ff9700 1 -- 172.21.15.61:0/3374579070 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 72 v72) v1 ==== 94+0+2142 (secure 0 0 0) 0x7efd100463d0 con 0x7efd1c05c880 2022-07-20T03:17:00.388 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:00.388 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":72,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":72,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:16:39.468063+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8091},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8091":{"gid":8091,"name":"a","rank":0,"incarnation":71,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/1435464119","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1435464119},{"type":"v1","addr":"172.21.15.61:6833","nonce":1435464119}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:17:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcfc053920 msgr2=0x7efcfc055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcfc053920 0x7efcfc055de0 secure :-1 s=READY pgs=637 cs=0 l=1 rev1=1 rx=0x7efd0c01c000 tx=0x7efd0c015000).stop 2022-07-20T03:17:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 msgr2=0x7efd1c14aff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 0x7efd1c14aff0 secure :-1 s=READY pgs=1321 cs=0 l=1 rev1=1 rx=0x7efd10006940 tx=0x7efd10008040).stop 2022-07-20T03:17:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 shutdown_connections 2022-07-20T03:17:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7efcfc053920 0x7efcfc055de0 unknown :-1 s=CLOSED pgs=637 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7efd1c0af150 0x7efd1c159e90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7efd1c0afd00 0x7efd1c15a380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 --2- 172.21.15.61:0/3374579070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7efd1c05c880 0x7efd1c14aff0 unknown :-1 s=CLOSED pgs=1321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 >> 172.21.15.61:0/3374579070 conn(0x7efd1c063460 msgr2=0x7efd1c1466e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.383+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 shutdown_connections 2022-07-20T03:17:00.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.387+0000 7efd2222c700 1 -- 172.21.15.61:0/3374579070 wait complete. 2022-07-20T03:17:00.395 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 72 2022-07-20T03:17:00.403 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:17:00.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 Processor -- start 2022-07-20T03:17:00.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 -- start start 2022-07-20T03:17:00.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4200b3650 0x7ff4200adf80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4200b1510 0x7ff42013efc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff420144150 con 0x7ff42013f4b0 2022-07-20T03:17:00.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff4200b3a30 con 0x7ff4200b1510 2022-07-20T03:17:00.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff426166700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200ab240 con 0x7ff4200b3650 2022-07-20T03:17:00.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425965700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff42013f4b0 0x7ff4200b1c30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47316/0 (socket says 172.21.15.61:47316) 2022-07-20T03:17:00.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425965700 1 -- 172.21.15.61:0/83808248 learned_addr learned my addr 172.21.15.61:0/83808248 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:00.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425164700 1 --2- 172.21.15.61:0/83808248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4200b3650 0x7ff4200adf80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff424963700 1 --2- 172.21.15.61:0/83808248 >> v2:172.21.15.117:3300/0 conn(0x7ff4200b1510 0x7ff42013efc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff417fff700 1 -- 172.21.15.61:0/83808248 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1935923709 0 0) 0x7ff420144150 con 0x7ff42013f4b0 2022-07-20T03:17:00.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff417fff700 1 -- 172.21.15.61:0/83808248 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff40c003260 con 0x7ff42013f4b0 2022-07-20T03:17:00.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425164700 1 -- 172.21.15.61:0/83808248 >> v2:172.21.15.117:3300/0 conn(0x7ff4200b1510 msgr2=0x7ff42013efc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425164700 1 --2- 172.21.15.61:0/83808248 >> v2:172.21.15.117:3300/0 conn(0x7ff4200b1510 0x7ff42013efc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425164700 1 -- 172.21.15.61:0/83808248 >> v1:172.21.15.61:6789/0 conn(0x7ff42013f4b0 legacy=0x7ff4200b1c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425164700 1 -- 172.21.15.61:0/83808248 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4200ab4e0 con 0x7ff4200b3650 2022-07-20T03:17:00.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff424963700 1 --2- 172.21.15.61:0/83808248 >> v2:172.21.15.117:3300/0 conn(0x7ff4200b1510 0x7ff42013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff425164700 1 --2- 172.21.15.61:0/83808248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4200b3650 0x7ff4200adf80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff41c002f70 tx=0x7ff41c006df0).ready entity=mon.2 client_cookie=c8bf91fd0f18088a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff417fff700 1 -- 172.21.15.61:0/83808248 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff41c00d4a0 con 0x7ff4200b3650 2022-07-20T03:17:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff417fff700 1 -- 172.21.15.61:0/83808248 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c000e90 con 0x7ff4200b3650 2022-07-20T03:17:00.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.503+0000 7ff417fff700 1 -- 172.21.15.61:0/83808248 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff41c00d7f0 con 0x7ff4200b3650 2022-07-20T03:17:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 -- 172.21.15.61:0/83808248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4200b3650 msgr2=0x7ff4200adf80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 --2- 172.21.15.61:0/83808248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4200b3650 0x7ff4200adf80 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7ff41c002f70 tx=0x7ff41c006df0).stop 2022-07-20T03:17:00.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 -- 172.21.15.61:0/83808248 shutdown_connections 2022-07-20T03:17:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 --2- 172.21.15.61:0/83808248 >> v2:172.21.15.117:3300/0 conn(0x7ff4200b1510 0x7ff42013efc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 --2- 172.21.15.61:0/83808248 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4200b3650 0x7ff4200adf80 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 -- 172.21.15.61:0/83808248 >> 172.21.15.61:0/83808248 conn(0x7ff4200b6c30 msgr2=0x7ff4200b2c00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:00.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 -- 172.21.15.61:0/83808248 shutdown_connections 2022-07-20T03:17:00.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 -- 172.21.15.61:0/83808248 wait complete. 2022-07-20T03:17:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 Processor -- start 2022-07-20T03:17:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.507+0000 7ff426166700 1 -- start start 2022-07-20T03:17:00.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 0x7ff4200b56b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4200b3650 0x7ff42009bc70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff42013f4b0 0x7ff42009b3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff42014e100 con 0x7ff4200b1510 2022-07-20T03:17:00.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200b4e80 con 0x7ff4200b3650 2022-07-20T03:17:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4200b5120 con 0x7ff42013f4b0 2022-07-20T03:17:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 0x7ff4200b56b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 0x7ff4200b56b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50722/0 (socket says 172.21.15.61:50722) 2022-07-20T03:17:00.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 -- 172.21.15.61:0/3171910510 learned_addr learned my addr 172.21.15.61:0/3171910510 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff424963700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4200b3650 0x7ff42009bc70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425965700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff42013f4b0 0x7ff42009b3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 -- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff42013f4b0 msgr2=0x7ff42009b3c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff42013f4b0 0x7ff42009b3c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 -- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4200b3650 msgr2=0x7ff42009bc70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4200b3650 0x7ff42009bc70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:00.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 -- 172.21.15.61:0/3171910510 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff410009470 con 0x7ff4200b1510 2022-07-20T03:17:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425965700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff42013f4b0 0x7ff42009b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff425164700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 0x7ff4200b56b0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7ff41c002f70 tx=0x7ff41c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff41c0139c0 con 0x7ff4200b1510 2022-07-20T03:17:00.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff41c013b20 con 0x7ff4200b1510 2022-07-20T03:17:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff41c00b180 con 0x7ff4200b1510 2022-07-20T03:17:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff41c013e80 con 0x7ff4200b1510 2022-07-20T03:17:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.511+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff42009c1e0 con 0x7ff4200b1510 2022-07-20T03:17:00.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.515+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff42009ba50 con 0x7ff4200b1510 2022-07-20T03:17:00.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.515+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff41c025030 con 0x7ff4200b1510 2022-07-20T03:17:00.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.515+0000 7ff4167fc700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff400053a40 0x7ff400055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:00.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.515+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(130..130 src has 1..130) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff41c06cbd0 con 0x7ff4200b1510 2022-07-20T03:17:00.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.515+0000 7ff424963700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff400053a40 0x7ff400055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:00.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.515+0000 7ff424963700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff400053a40 0x7ff400055f00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7ff41000d000 tx=0x7ff4100091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:00.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.523+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff41c043b30 con 0x7ff4200b1510 2022-07-20T03:17:00.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:00.679+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7ff4200ab780 con 0x7ff4200b1510 2022-07-20T03:17:01.287 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:17:01.283+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:17:01.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.291+0000 7ff4167fc700 1 -- 172.21.15.61:0/3171910510 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v73) v1 ==== 161+0+0 (secure 0 0 0) 0x7ff41c023070 con 0x7ff4200b1510 2022-07-20T03:17:01.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff400053a40 msgr2=0x7ff400055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff400053a40 0x7ff400055f00 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7ff41000d000 tx=0x7ff4100091d0).stop 2022-07-20T03:17:01.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 msgr2=0x7ff4200b56b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 0x7ff4200b56b0 secure :-1 s=READY pgs=1323 cs=0 l=1 rev1=1 rx=0x7ff41c002f70 tx=0x7ff41c014000).stop 2022-07-20T03:17:01.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 shutdown_connections 2022-07-20T03:17:01.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff400053a40 0x7ff400055f00 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4200b3650 0x7ff42009bc70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff42013f4b0 0x7ff42009b3c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 --2- 172.21.15.61:0/3171910510 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4200b1510 0x7ff4200b56b0 unknown :-1 s=CLOSED pgs=1323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 >> 172.21.15.61:0/3171910510 conn(0x7ff4200b6c30 msgr2=0x7ff4200adcf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:01.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 shutdown_connections 2022-07-20T03:17:01.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.295+0000 7ff426166700 1 -- 172.21.15.61:0/3171910510 wait complete. 2022-07-20T03:17:01.303 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:17:01.317 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:01.327 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:17:01.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.407+0000 7f9968de7700 1 Processor -- start 2022-07-20T03:17:01.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.407+0000 7f9968de7700 1 -- start start 2022-07-20T03:17:01.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9968de7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99640b27a0 0x7f99640af9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9968de7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996405c290 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9968de7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f996405c780 0x7f9964142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9968de7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99640ae8b0 con 0x7f99640b27a0 2022-07-20T03:17:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9968de7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f99640aeb50 con 0x7f996405c780 2022-07-20T03:17:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9968de7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f99640aedf0 con 0x7f996405be30 2022-07-20T03:17:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99640b27a0 0x7f99640af9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f99640b27a0 0x7f99640af9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50740/0 (socket says 172.21.15.61:50740) 2022-07-20T03:17:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 -- 172.21.15.61:0/3192050711 learned_addr learned my addr 172.21.15.61:0/3192050711 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9963fff700 1 --2- 172.21.15.61:0/3192050711 >> v2:172.21.15.117:3300/0 conn(0x7f996405c780 0x7f9964142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9962ffd700 1 --2- 172.21.15.61:0/3192050711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996405c290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 -- 172.21.15.61:0/3192050711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 msgr2=0x7f996405c290 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 --2- 172.21.15.61:0/3192050711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996405c290 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 -- 172.21.15.61:0/3192050711 >> v2:172.21.15.117:3300/0 conn(0x7f996405c780 msgr2=0x7f9964142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:01.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 --2- 172.21.15.61:0/3192050711 >> v2:172.21.15.117:3300/0 conn(0x7f996405c780 0x7f9964142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 -- 172.21.15.61:0/3192050711 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99640b2b80 con 0x7f99640b27a0 2022-07-20T03:17:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f9963fff700 1 --2- 172.21.15.61:0/3192050711 >> v2:172.21.15.117:3300/0 conn(0x7f996405c780 0x7f9964142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:01.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99637fe700 1 --2- 172.21.15.61:0/3192050711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f99640af9c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f9954002f70 tx=0x7f9954006f40).ready entity=mon.0 client_cookie=6553f4abc07a2c41 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99627fc700 1 -- 172.21.15.61:0/3192050711 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995400d9a0 con 0x7f99640b27a0 2022-07-20T03:17:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99627fc700 1 -- 172.21.15.61:0/3192050711 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9954000e90 con 0x7f99640b27a0 2022-07-20T03:17:01.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.411+0000 7f99627fc700 1 -- 172.21.15.61:0/3192050711 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995400dd30 con 0x7f99640b27a0 2022-07-20T03:17:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 -- 172.21.15.61:0/3192050711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 msgr2=0x7f99640af9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 --2- 172.21.15.61:0/3192050711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f99640af9c0 secure :-1 s=READY pgs=1325 cs=0 l=1 rev1=1 rx=0x7f9954002f70 tx=0x7f9954006f40).stop 2022-07-20T03:17:01.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 -- 172.21.15.61:0/3192050711 shutdown_connections 2022-07-20T03:17:01.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 --2- 172.21.15.61:0/3192050711 >> v2:172.21.15.117:3300/0 conn(0x7f996405c780 0x7f9964142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 --2- 172.21.15.61:0/3192050711 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996405c290 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 --2- 172.21.15.61:0/3192050711 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f99640af9c0 unknown :-1 s=CLOSED pgs=1325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 -- 172.21.15.61:0/3192050711 >> 172.21.15.61:0/3192050711 conn(0x7f99640632c0 msgr2=0x7f99640636a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 -- 172.21.15.61:0/3192050711 shutdown_connections 2022-07-20T03:17:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 -- 172.21.15.61:0/3192050711 wait complete. 2022-07-20T03:17:01.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 Processor -- start 2022-07-20T03:17:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.415+0000 7f9968de7700 1 -- start start 2022-07-20T03:17:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996413fbe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f996405c780 0x7f996413dde0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f996413e2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964066870 con 0x7f99640b27a0 2022-07-20T03:17:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9964066b10 con 0x7f996405c780 2022-07-20T03:17:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f996413e7c0 con 0x7f996405be30 2022-07-20T03:17:01.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f996413e2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9962ffd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f996405c780 0x7f996413dde0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f996413e2d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50750/0 (socket says 172.21.15.61:50750) 2022-07-20T03:17:01.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 -- 172.21.15.61:0/3562251549 learned_addr learned my addr 172.21.15.61:0/3562251549 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f99637fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996413fbe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 -- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 msgr2=0x7f996413fbe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996413fbe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 -- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f996405c780 msgr2=0x7f996413dde0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f996405c780 0x7f996413dde0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 -- 172.21.15.61:0/3562251549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f995c009470 con 0x7f99640b27a0 2022-07-20T03:17:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9962ffd700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f996405c780 0x7f996413dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:01.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f99637fe700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996413fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9963fff700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f996413e2d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f995c00c000 tx=0x7f995c0091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995c00c920 con 0x7f99640b27a0 2022-07-20T03:17:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f995c00ca80 con 0x7f99640b27a0 2022-07-20T03:17:01.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f995c00cde0 con 0x7f99640b27a0 2022-07-20T03:17:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f995400b460 con 0x7f99640b27a0 2022-07-20T03:17:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.419+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9964067470 con 0x7f99640b27a0 2022-07-20T03:17:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.423+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99640666d0 con 0x7f99640b27a0 2022-07-20T03:17:01.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.423+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f995c022070 con 0x7f99640b27a0 2022-07-20T03:17:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.423+0000 7f9960ff9700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9944053a40 0x7f9944055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.423+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f995c06c880 con 0x7f99640b27a0 2022-07-20T03:17:01.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.423+0000 7f99637fe700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9944053a40 0x7f9944055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.423+0000 7f99637fe700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9944053a40 0x7f9944055f00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f995401b000 tx=0x7f9954006f40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:01.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.427+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f99640666d0 con 0x7f99640b27a0 2022-07-20T03:17:01.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.591+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f99640666d0 con 0x7f99640b27a0 2022-07-20T03:17:01.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.591+0000 7f9960ff9700 1 -- 172.21.15.61:0/3562251549 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 73 v73) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f995c016710 con 0x7f99640b27a0 2022-07-20T03:17:01.597 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:01.597 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":73,"default_fscid":13,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70}],"filesystems":[{"mdsmap":{"epoch":73,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:16:38.464416+0000","modified":"2022-07-20T03:17:01.287434+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":131,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[26],"metadata_pool":25,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":13}]} 2022-07-20T03:17:01.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.591+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9944053a40 msgr2=0x7f9944055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.591+0000 7f9968de7700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9944053a40 0x7f9944055f00 secure :-1 s=READY pgs=639 cs=0 l=1 rev1=1 rx=0x7f995401b000 tx=0x7f9954006f40).stop 2022-07-20T03:17:01.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 msgr2=0x7f996413e2d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f996413e2d0 secure :-1 s=READY pgs=1326 cs=0 l=1 rev1=1 rx=0x7f995c00c000 tx=0x7f995c0091d0).stop 2022-07-20T03:17:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 shutdown_connections 2022-07-20T03:17:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9944053a40 0x7f9944055f00 unknown :-1 s=CLOSED pgs=639 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f996405c780 0x7f996413dde0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f99640b27a0 0x7f996413e2d0 unknown :-1 s=CLOSED pgs=1326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 --2- 172.21.15.61:0/3562251549 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f996405be30 0x7f996413fbe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 >> 172.21.15.61:0/3562251549 conn(0x7f99640632c0 msgr2=0x7f99640b5080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:01.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 shutdown_connections 2022-07-20T03:17:01.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.595+0000 7f9968de7700 1 -- 172.21.15.61:0/3562251549 wait complete. 2022-07-20T03:17:01.603 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 73 2022-07-20T03:17:01.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:17:01.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 Processor -- start 2022-07-20T03:17:01.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 -- start start 2022-07-20T03:17:01.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe6f80654f0 0x7fe6f814a500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6f80b2b00 0x7fe6f8146430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe6f814a9f0 con 0x7fe6f80b1040 2022-07-20T03:17:01.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f80b1420 con 0x7fe6f80b2b00 2022-07-20T03:17:01.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe6f80b2fb0 con 0x7fe6f80654f0 2022-07-20T03:17:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6f80b2b00 0x7fe6f8146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fcf47700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe6f80b1040 0x7fe6f8064de0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47356/0 (socket says 172.21.15.61:47356) 2022-07-20T03:17:01.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe6f80b2b00 0x7fe6f8146430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50756/0 (socket says 172.21.15.61:50756) 2022-07-20T03:17:01.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fcf47700 1 -- 172.21.15.61:0/3951422292 learned_addr learned my addr 172.21.15.61:0/3951422292 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:01.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6f7fff700 1 --2- 172.21.15.61:0/3951422292 >> v2:172.21.15.117:3300/0 conn(0x7fe6f80654f0 0x7fe6f814a500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 -- 172.21.15.61:0/3951422292 >> v2:172.21.15.117:3300/0 conn(0x7fe6f80654f0 msgr2=0x7fe6f814a500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:01.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 --2- 172.21.15.61:0/3951422292 >> v2:172.21.15.117:3300/0 conn(0x7fe6f80654f0 0x7fe6f814a500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6f77fe700 1 -- 172.21.15.61:0/3951422292 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1825569042 0 0) 0x7fe6e0001110 con 0x7fe6f80b1040 2022-07-20T03:17:01.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 -- 172.21.15.61:0/3951422292 >> v1:172.21.15.61:6789/0 conn(0x7fe6f80b1040 legacy=0x7fe6f8064de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 -- 172.21.15.61:0/3951422292 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f80b3250 con 0x7fe6f80b2b00 2022-07-20T03:17:01.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fd748700 1 --2- 172.21.15.61:0/3951422292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8146430 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fe6f00032f0 tx=0x7fe6f00091d0).ready entity=mon.0 client_cookie=287769ab4d97b93c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6f77fe700 1 -- 172.21.15.61:0/3951422292 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f000b680 con 0x7fe6f80b2b00 2022-07-20T03:17:01.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6f77fe700 1 -- 172.21.15.61:0/3951422292 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6f000b7e0 con 0x7fe6f80b2b00 2022-07-20T03:17:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6f77fe700 1 -- 172.21.15.61:0/3951422292 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6f000b950 con 0x7fe6f80b2b00 2022-07-20T03:17:01.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 -- 172.21.15.61:0/3951422292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80b2b00 msgr2=0x7fe6f8146430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.723+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/3951422292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8146430 secure :-1 s=READY pgs=1328 cs=0 l=1 rev1=1 rx=0x7fe6f00032f0 tx=0x7fe6f00091d0).stop 2022-07-20T03:17:01.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- 172.21.15.61:0/3951422292 shutdown_connections 2022-07-20T03:17:01.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/3951422292 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8146430 unknown :-1 s=CLOSED pgs=1328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/3951422292 >> v2:172.21.15.117:3300/0 conn(0x7fe6f80654f0 0x7fe6f814a500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- 172.21.15.61:0/3951422292 >> 172.21.15.61:0/3951422292 conn(0x7fe6f8061d70 msgr2=0x7fe6f8063a50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:01.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- 172.21.15.61:0/3951422292 shutdown_connections 2022-07-20T03:17:01.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- 172.21.15.61:0/3951422292 wait complete. 2022-07-20T03:17:01.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 Processor -- start 2022-07-20T03:17:01.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- start start 2022-07-20T03:17:01.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 0x7fe6f8159f40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f80b1040 0x7fe6f814a2d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8110ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f80b0340 con 0x7fe6f80654f0 2022-07-20T03:17:01.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f810f4c0 con 0x7fe6f80b2b00 2022-07-20T03:17:01.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fdf49700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe6f810f760 con 0x7fe6f80b1040 2022-07-20T03:17:01.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 0x7fe6f8159f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 0x7fe6f8159f40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50760/0 (socket says 172.21.15.61:50760) 2022-07-20T03:17:01.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 -- 172.21.15.61:0/2116584616 learned_addr learned my addr 172.21.15.61:0/2116584616 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:01.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fd748700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8110ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6f7fff700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f80b1040 0x7fe6f814a2d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 -- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f80b1040 msgr2=0x7fe6f814a2d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f80b1040 0x7fe6f814a2d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 -- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f80b2b00 msgr2=0x7fe6f8110ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:01.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8110ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:01.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 -- 172.21.15.61:0/2116584616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6f0009470 con 0x7fe6f80654f0 2022-07-20T03:17:01.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.727+0000 7fe6fcf47700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 0x7fe6f8159f40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fe6e0000bc0 tx=0x7fe6e00078e0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:01.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6e00070d0 con 0x7fe6f80654f0 2022-07-20T03:17:01.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe6f810fa00 con 0x7fe6f80654f0 2022-07-20T03:17:01.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe6f813d4c0 con 0x7fe6f80654f0 2022-07-20T03:17:01.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe6e0007230 con 0x7fe6f80654f0 2022-07-20T03:17:01.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe6e0007590 con 0x7fe6f80654f0 2022-07-20T03:17:01.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe6d0003410 con 0x7fe6f80654f0 2022-07-20T03:17:01.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe6e0010070 con 0x7fe6f80654f0 2022-07-20T03:17:01.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f5ffb700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6dc053a40 0x7fe6dc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:01.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fe6e006b460 con 0x7fe6f80654f0 2022-07-20T03:17:01.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.731+0000 7fe6f7fff700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6dc053a40 0x7fe6dc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:01.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.735+0000 7fe6f7fff700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6dc053a40 0x7fe6dc055f00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7fe6e800b000 tx=0x7fe6e8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:01.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.739+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe6e00422d0 con 0x7fe6f80654f0 2022-07-20T03:17:01.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:01.907+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fe6d0003c40 con 0x7fe6f80654f0 2022-07-20T03:17:02.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.307+0000 7fe6f5ffb700 1 -- 172.21.15.61:0/2116584616 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v74) v1 ==== 104+0+0 (secure 0 0 0) 0x7fe6e00458f0 con 0x7fe6f80654f0 2022-07-20T03:17:02.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6dc053a40 msgr2=0x7fe6dc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:02.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6dc053a40 0x7fe6dc055f00 secure :-1 s=READY pgs=640 cs=0 l=1 rev1=1 rx=0x7fe6e800b000 tx=0x7fe6e8008040).stop 2022-07-20T03:17:02.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 msgr2=0x7fe6f8159f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:02.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 0x7fe6f8159f40 secure :-1 s=READY pgs=1329 cs=0 l=1 rev1=1 rx=0x7fe6e0000bc0 tx=0x7fe6e00078e0).stop 2022-07-20T03:17:02.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 shutdown_connections 2022-07-20T03:17:02.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe6dc053a40 0x7fe6dc055f00 unknown :-1 s=CLOSED pgs=640 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe6f80b2b00 0x7fe6f8110ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe6f80b1040 0x7fe6f814a2d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 --2- 172.21.15.61:0/2116584616 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe6f80654f0 0x7fe6f8159f40 unknown :-1 s=CLOSED pgs=1329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 >> 172.21.15.61:0/2116584616 conn(0x7fe6f8061d70 msgr2=0x7fe6f8146df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:02.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 shutdown_connections 2022-07-20T03:17:02.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.311+0000 7fe6fdf49700 1 -- 172.21.15.61:0/2116584616 wait complete. 2022-07-20T03:17:02.333 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:17:02.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 Processor -- start 2022-07-20T03:17:02.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 -- start start 2022-07-20T03:17:02.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d900b9fc0 0x7f9d900b0030 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b5340 0x7f9d90142c00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d900b4b20 0x7f9d900b0af0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d900ba3a0 con 0x7f9d900b9fc0 2022-07-20T03:17:02.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d90146e20 con 0x7f9d900b4b20 2022-07-20T03:17:02.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d95400700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d901470c0 con 0x7f9d900b5340 2022-07-20T03:17:02.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d900b9fc0 0x7f9d900b0030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d900b9fc0 0x7f9d900b0030 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50770/0 (socket says 172.21.15.61:50770) 2022-07-20T03:17:02.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 -- 172.21.15.61:0/1982940118 learned_addr learned my addr 172.21.15.61:0/1982940118 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:02.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8f7fe700 1 --2- 172.21.15.61:0/1982940118 >> v2:172.21.15.117:3300/0 conn(0x7f9d900b4b20 0x7f9d900b0af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8e7fc700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b5340 0x7f9d90142c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 -- 172.21.15.61:0/1982940118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b5340 msgr2=0x7f9d90142c00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:02.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b5340 0x7f9d90142c00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 -- 172.21.15.61:0/1982940118 >> v2:172.21.15.117:3300/0 conn(0x7f9d900b4b20 msgr2=0x7f9d900b0af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:02.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 --2- 172.21.15.61:0/1982940118 >> v2:172.21.15.117:3300/0 conn(0x7f9d900b4b20 0x7f9d900b0af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8effd700 1 -- 172.21.15.61:0/1982940118 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d90066780 con 0x7f9d900b9fc0 2022-07-20T03:17:02.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.451+0000 7f9d8e7fc700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b5340 0x7f9d90142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:02.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d8effd700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b9fc0 0x7f9d900b0030 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f9d84002f70 tx=0x7f9d84006eb0).ready entity=mon.0 client_cookie=5c01b26776c9ed64 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:02.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d8dffb700 1 -- 172.21.15.61:0/1982940118 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d8400d570 con 0x7f9d900b9fc0 2022-07-20T03:17:02.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d8dffb700 1 -- 172.21.15.61:0/1982940118 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d84000e90 con 0x7f9d900b9fc0 2022-07-20T03:17:02.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d8dffb700 1 -- 172.21.15.61:0/1982940118 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d8400d900 con 0x7f9d900b9fc0 2022-07-20T03:17:02.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 -- 172.21.15.61:0/1982940118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b9fc0 msgr2=0x7f9d900b0030 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:02.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b9fc0 0x7f9d900b0030 secure :-1 s=READY pgs=1330 cs=0 l=1 rev1=1 rx=0x7f9d84002f70 tx=0x7f9d84006eb0).stop 2022-07-20T03:17:02.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 -- 172.21.15.61:0/1982940118 shutdown_connections 2022-07-20T03:17:02.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 --2- 172.21.15.61:0/1982940118 >> v2:172.21.15.117:3300/0 conn(0x7f9d900b4b20 0x7f9d900b0af0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b5340 0x7f9d90142c00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 --2- 172.21.15.61:0/1982940118 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b9fc0 0x7f9d900b0030 unknown :-1 s=CLOSED pgs=1330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 -- 172.21.15.61:0/1982940118 >> 172.21.15.61:0/1982940118 conn(0x7f9d900678e0 msgr2=0x7f9d900b9570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:02.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 -- 172.21.15.61:0/1982940118 shutdown_connections 2022-07-20T03:17:02.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.455+0000 7f9d95400700 1 -- 172.21.15.61:0/1982940118 wait complete. 2022-07-20T03:17:02.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 Processor -- start 2022-07-20T03:17:02.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 -- start start 2022-07-20T03:17:02.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d900b4b20 0x7f9d900ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 0x7f9d90114ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b9fc0 0x7f9d901153e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d9013f8f0 con 0x7f9d900b5340 2022-07-20T03:17:02.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d901141b0 con 0x7f9d900b4b20 2022-07-20T03:17:02.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d90152910 con 0x7f9d900b9fc0 2022-07-20T03:17:02.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 0x7f9d90114ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 0x7f9d90114ef0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50778/0 (socket says 172.21.15.61:50778) 2022-07-20T03:17:02.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 -- 172.21.15.61:0/3099224059 learned_addr learned my addr 172.21.15.61:0/3099224059 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:02.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8effd700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d900b4b20 0x7f9d900ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8f7fe700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b9fc0 0x7f9d901153e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 -- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b9fc0 msgr2=0x7f9d901153e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:02.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b9fc0 0x7f9d901153e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 -- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d900b4b20 msgr2=0x7f9d900ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:02.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d900b4b20 0x7f9d900ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:02.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 -- 172.21.15.61:0/3099224059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d8400b1e0 con 0x7f9d900b5340 2022-07-20T03:17:02.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8f7fe700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b9fc0 0x7f9d901153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:17:02.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d8e7fc700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 0x7f9d90114ef0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f9d80006940 tx=0x7f9d80008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:02.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d80011780 con 0x7f9d900b5340 2022-07-20T03:17:02.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d800118e0 con 0x7f9d900b5340 2022-07-20T03:17:02.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d90115990 con 0x7f9d900b5340 2022-07-20T03:17:02.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d80011c40 con 0x7f9d900b5340 2022-07-20T03:17:02.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.459+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d90152e20 con 0x7f9d900b5340 2022-07-20T03:17:02.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.463+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d900b0fe0 con 0x7f9d900b5340 2022-07-20T03:17:02.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.463+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d80012420 con 0x7f9d900b5340 2022-07-20T03:17:02.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.463+0000 7f9d7ffff700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d6c053a40 0x7f9d6c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:02.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.463+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(131..131 src has 1..131) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9d8006bd70 con 0x7f9d900b5340 2022-07-20T03:17:02.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.463+0000 7f9d8effd700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d6c053a40 0x7f9d6c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:02.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.463+0000 7f9d8effd700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d6c053a40 0x7f9d6c055f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f9d8401a000 tx=0x7f9d84015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:02.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.471+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d80042ba0 con 0x7f9d900b5340 2022-07-20T03:17:02.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:02.595+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f9d90152ad0 con 0x7f9d900b5340 2022-07-20T03:17:03.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.319+0000 7f9d7ffff700 1 -- 172.21.15.61:0/3099224059 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v132) v1 ==== 185+0+0 (secure 0 0 0) 0x7f9d800461c0 con 0x7f9d900b5340 2022-07-20T03:17:03.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d6c053a40 msgr2=0x7f9d6c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:03.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d6c053a40 0x7f9d6c055f00 secure :-1 s=READY pgs=642 cs=0 l=1 rev1=1 rx=0x7f9d8401a000 tx=0x7f9d84015000).stop 2022-07-20T03:17:03.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 msgr2=0x7f9d90114ef0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:03.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 0x7f9d90114ef0 secure :-1 s=READY pgs=1331 cs=0 l=1 rev1=1 rx=0x7f9d80006940 tx=0x7f9d80008040).stop 2022-07-20T03:17:03.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 shutdown_connections 2022-07-20T03:17:03.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d6c053a40 0x7f9d6c055f00 unknown :-1 s=CLOSED pgs=642 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d900b9fc0 0x7f9d901153e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d900b5340 0x7f9d90114ef0 unknown :-1 s=CLOSED pgs=1331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 --2- 172.21.15.61:0/3099224059 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d900b4b20 0x7f9d900ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 >> 172.21.15.61:0/3099224059 conn(0x7f9d900678e0 msgr2=0x7f9d9013fc40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:03.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 shutdown_connections 2022-07-20T03:17:03.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.323+0000 7f9d95400700 1 -- 172.21.15.61:0/3099224059 wait complete. 2022-07-20T03:17:03.337 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:17:03.343 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:17:03.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.455+0000 7f709f511700 1 Processor -- start 2022-07-20T03:17:03.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.455+0000 7f709f511700 1 -- start start 2022-07-20T03:17:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f709805c170 0x7f7098142d50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7098146ed0 0x7f70980affc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7098146390 con 0x7f7098060520 2022-07-20T03:17:03.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f709805c550 con 0x7f7098146ed0 2022-07-20T03:17:03.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7098143240 con 0x7f709805c170 2022-07-20T03:17:03.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7098146ed0 0x7f70980affc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:03.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7098146ed0 0x7f70980affc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50784/0 (socket says 172.21.15.61:50784) 2022-07-20T03:17:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 -- 172.21.15.61:0/2876544683 learned_addr learned my addr 172.21.15.61:0/2876544683 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709d2ad700 1 --2- 172.21.15.61:0/2876544683 >> v2:172.21.15.117:3300/0 conn(0x7f709805c170 0x7f7098142d50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 -- 172.21.15.61:0/2876544683 >> v2:172.21.15.117:3300/0 conn(0x7f709805c170 msgr2=0x7f7098142d50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:03.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 --2- 172.21.15.61:0/2876544683 >> v2:172.21.15.117:3300/0 conn(0x7f709805c170 0x7f7098142d50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 -- 172.21.15.61:0/2876544683 >> v1:172.21.15.117:6789/0 conn(0x7f7098060520 legacy=0x7f70981467c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 -- 172.21.15.61:0/2876544683 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f709805fe20 con 0x7f7098146ed0 2022-07-20T03:17:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709daae700 1 --2- 172.21.15.61:0/2876544683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098146ed0 0x7f70980affc0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f70900032f0 tx=0x7f7090009290).ready entity=mon.0 client_cookie=dc003ebdb830cf14 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:03.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f7087fff700 1 -- 172.21.15.61:0/2876544683 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f709000b870 con 0x7f7098146ed0 2022-07-20T03:17:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f7087fff700 1 -- 172.21.15.61:0/2876544683 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f709000b9d0 con 0x7f7098146ed0 2022-07-20T03:17:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f7087fff700 1 -- 172.21.15.61:0/2876544683 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f709000bb40 con 0x7f7098146ed0 2022-07-20T03:17:03.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- 172.21.15.61:0/2876544683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098146ed0 msgr2=0x7f70980affc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:03.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 --2- 172.21.15.61:0/2876544683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098146ed0 0x7f70980affc0 secure :-1 s=READY pgs=1332 cs=0 l=1 rev1=1 rx=0x7f70900032f0 tx=0x7f7090009290).stop 2022-07-20T03:17:03.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- 172.21.15.61:0/2876544683 shutdown_connections 2022-07-20T03:17:03.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 --2- 172.21.15.61:0/2876544683 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098146ed0 0x7f70980affc0 unknown :-1 s=CLOSED pgs=1332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 --2- 172.21.15.61:0/2876544683 >> v2:172.21.15.117:3300/0 conn(0x7f709805c170 0x7f7098142d50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- 172.21.15.61:0/2876544683 >> 172.21.15.61:0/2876544683 conn(0x7f7098063b60 msgr2=0x7f7098063f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- 172.21.15.61:0/2876544683 shutdown_connections 2022-07-20T03:17:03.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.459+0000 7f709f511700 1 -- 172.21.15.61:0/2876544683 wait complete. 2022-07-20T03:17:03.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 Processor -- start 2022-07-20T03:17:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 -- start start 2022-07-20T03:17:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f709805c170 0x7f7098115a40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:03.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 0x7f7098115f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:03.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7098146ed0 0x7f70981526a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:03.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7098114000 con 0x7f7098060520 2022-07-20T03:17:03.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f70981183b0 con 0x7f709805c170 2022-07-20T03:17:03.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7098118650 con 0x7f7098146ed0 2022-07-20T03:17:03.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 0x7f7098115f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:03.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 0x7f7098115f30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50792/0 (socket says 172.21.15.61:50792) 2022-07-20T03:17:03.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 -- 172.21.15.61:0/3591762801 learned_addr learned my addr 172.21.15.61:0/3591762801 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:03.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709d2ad700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f709805c170 0x7f7098115a40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:03.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709daae700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7098146ed0 0x7f70981526a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:03.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 -- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7098146ed0 msgr2=0x7f70981526a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7098146ed0 0x7f70981526a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 -- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f709805c170 msgr2=0x7f7098115a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:03.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f709805c170 0x7f7098115a40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 -- 172.21.15.61:0/3591762801 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7090009530 con 0x7f7098060520 2022-07-20T03:17:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709d2ad700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f709805c170 0x7f7098115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709caac700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 0x7f7098115f30 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7f709805f9d0 tx=0x7f7088007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:03.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f708800ce30 con 0x7f7098060520 2022-07-20T03:17:03.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7088007240 con 0x7f7098060520 2022-07-20T03:17:03.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7098114270 con 0x7f7098060520 2022-07-20T03:17:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7088006700 con 0x7f7098060520 2022-07-20T03:17:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.463+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7098115210 con 0x7f7098060520 2022-07-20T03:17:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.467+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f70981154f0 con 0x7f7098060520 2022-07-20T03:17:03.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.467+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7088010030 con 0x7f7098060520 2022-07-20T03:17:03.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.467+0000 7f70867fc700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70780539b0 0x7f7078055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:03.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.467+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(132..132 src has 1..132) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f708806b990 con 0x7f7098060520 2022-07-20T03:17:03.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.467+0000 7f709d2ad700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70780539b0 0x7f7078055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:03.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.467+0000 7f709d2ad700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70780539b0 0x7f7078055e70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f709400c000 tx=0x7f7094008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:03.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.479+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7088042a50 con 0x7f7098060520 2022-07-20T03:17:03.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:03.607+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f7098114400 con 0x7f7098060520 2022-07-20T03:17:04.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.331+0000 7f70867fc700 1 -- 172.21.15.61:0/3591762801 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v133) v1 ==== 173+0+0 (secure 0 0 0) 0x7f7088046070 con 0x7f7098060520 2022-07-20T03:17:04.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70780539b0 msgr2=0x7f7078055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70780539b0 0x7f7078055e70 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7f709400c000 tx=0x7f7094008040).stop 2022-07-20T03:17:04.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 msgr2=0x7f7098115f30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 0x7f7098115f30 secure :-1 s=READY pgs=1333 cs=0 l=1 rev1=1 rx=0x7f709805f9d0 tx=0x7f7088007c10).stop 2022-07-20T03:17:04.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 shutdown_connections 2022-07-20T03:17:04.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f70780539b0 0x7f7078055e70 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7098146ed0 0x7f70981526a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7098060520 0x7f7098115f30 unknown :-1 s=CLOSED pgs=1333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 --2- 172.21.15.61:0/3591762801 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f709805c170 0x7f7098115a40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.335+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 >> 172.21.15.61:0/3591762801 conn(0x7f7098063b60 msgr2=0x7f7098065930 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:04.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.339+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 shutdown_connections 2022-07-20T03:17:04.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.339+0000 7f709f511700 1 -- 172.21.15.61:0/3591762801 wait complete. 2022-07-20T03:17:04.348 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:17:04.359 DEBUG:teuthology.orchestra.run.smithi061:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:17:04.365 DEBUG:teuthology.orchestra.run.smithi117:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:17:04.368 DEBUG:teuthology.orchestra.run.smithi124:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:17:04.373 DEBUG:teuthology.orchestra.run.smithi163:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:17:04.378 DEBUG:teuthology.orchestra.run.smithi179:> sudo mkdir -p /etc/ceph && sudo chmod 0755 /etc/ceph && sudo tee /etc/ceph/ceph.conf && sudo chmod 0644 /etc/ceph/ceph.conf > /dev/null 2022-07-20T03:17:04.395 INFO:teuthology.orchestra.run.smithi061.stdout:[global] 2022-07-20T03:17:04.396 INFO:teuthology.orchestra.run.smithi061.stdout: chdir = "" 2022-07-20T03:17:04.396 INFO:teuthology.orchestra.run.smithi061.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:17:04.396 INFO:teuthology.orchestra.run.smithi061.stdout: auth supported = cephx 2022-07-20T03:17:04.396 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.396 INFO:teuthology.orchestra.run.smithi061.stdout: filestore xattr use omap = true 2022-07-20T03:17:04.397 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.397 INFO:teuthology.orchestra.run.smithi061.stdout: mon clock drift allowed = 1.000 2022-07-20T03:17:04.397 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.397 INFO:teuthology.orchestra.run.smithi061.stdout: osd crush chooseleaf type = 0 2022-07-20T03:17:04.398 INFO:teuthology.orchestra.run.smithi061.stdout: auth debug = true 2022-07-20T03:17:04.398 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.398 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on old message = true 2022-07-20T03:17:04.399 INFO:teuthology.orchestra.run.smithi061.stdout: ms die on bug = true 2022-07-20T03:17:04.399 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.399 INFO:teuthology.orchestra.run.smithi061.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:17:04.400 INFO:teuthology.orchestra.run.smithi061.stdout: mon pg warn max object skew = 0 2022-07-20T03:17:04.400 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.401 INFO:teuthology.orchestra.run.smithi061.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:17:04.401 INFO:teuthology.orchestra.run.smithi061.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:17:04.401 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.402 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default size = 2 2022-07-20T03:17:04.403 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.403 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow primary affinity = true 2022-07-20T03:17:04.404 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd allow pg remap = true 2022-07-20T03:17:04.404 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:17:04.404 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:17:04.405 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on no sortbitwise = false 2022-07-20T03:17:04.405 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:17:04.406 INFO:teuthology.orchestra.run.smithi061.stdout: mon warn on too few osds = false 2022-07-20T03:17:04.406 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:17:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:17:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:17:04.407 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.408 INFO:teuthology.orchestra.run.smithi061.stdout: osd default data pool replay window = 5 2022-07-20T03:17:04.408 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.408 INFO:teuthology.orchestra.run.smithi061.stdout: mon allow pool delete = true 2022-07-20T03:17:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: mon cluster log file level = debug 2022-07-20T03:17:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: debug asserts on shutdown = true 2022-07-20T03:17:04.409 INFO:teuthology.orchestra.run.smithi061.stdout: mon health detail to clog = false 2022-07-20T03:17:04.410 INFO:teuthology.orchestra.run.smithi061.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:17:04.410 INFO:teuthology.orchestra.run.smithi061.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:17:04.410 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.411 INFO:teuthology.orchestra.run.smithi061.stdout:[osd] 2022-07-20T03:17:04.411 INFO:teuthology.orchestra.run.smithi061.stdout: osd journal size = 100 2022-07-20T03:17:04.411 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub load threshold = 5.0 2022-07-20T03:17:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: osd scrub max interval = 600 2022-07-20T03:17:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.412 INFO:teuthology.orchestra.run.smithi061.stdout: osd recover clone overlap = true 2022-07-20T03:17:04.413 INFO:teuthology.orchestra.run.smithi061.stdout: osd recovery max chunk = 1048576 2022-07-20T03:17:04.413 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.413 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug shutdown = true 2022-07-20T03:17:04.413 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug op order = true 2022-07-20T03:17:04.414 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug verify stray on activate = true 2022-07-20T03:17:04.414 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.414 INFO:teuthology.orchestra.run.smithi061.stdout: osd open classes on start = true 2022-07-20T03:17:04.414 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug pg log writeout = true 2022-07-20T03:17:04.415 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.415 INFO:teuthology.orchestra.run.smithi061.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:17:04.415 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: osd map max advance = 10 2022-07-20T03:17:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: journal zero on create = true 2022-07-20T03:17:04.416 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:17:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: filestore apply finisher threads = 3 2022-07-20T03:17:04.417 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.418 INFO:teuthology.orchestra.run.smithi061.stdout: bdev debug aio = true 2022-07-20T03:17:04.418 INFO:teuthology.orchestra.run.smithi061.stdout: osd debug misdirected ops = true 2022-07-20T03:17:04.418 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:17:04.418 INFO:teuthology.orchestra.run.smithi061.stdout: debug osd = 20 2022-07-20T03:17:04.419 INFO:teuthology.orchestra.run.smithi061.stdout: osd objectstore = filestore 2022-07-20T03:17:04.419 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-20T03:17:04.419 INFO:teuthology.orchestra.run.smithi061.stdout: osd sloppy crc = True 2022-07-20T03:17:04.420 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.420 INFO:teuthology.orchestra.run.smithi061.stdout:[mgr] 2022-07-20T03:17:04.420 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:17:04.421 INFO:teuthology.orchestra.run.smithi061.stdout: debug mgr = 20 2022-07-20T03:17:04.421 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-20T03:17:04.421 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-20T03:17:04.421 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.422 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.422 INFO:teuthology.orchestra.run.smithi061.stdout: mgr/telemetry/nag = false 2022-07-20T03:17:04.422 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.423 INFO:teuthology.orchestra.run.smithi061.stdout:[mon] 2022-07-20T03:17:04.423 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:17:04.423 INFO:teuthology.orchestra.run.smithi061.stdout: debug mon = 20 2022-07-20T03:17:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: debug paxos = 20 2022-07-20T03:17:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: debug auth = 20 2022-07-20T03:17:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: mon data avail warn = 5 2022-07-20T03:17:04.424 INFO:teuthology.orchestra.run.smithi061.stdout: mon mgr mkfs grace = 240 2022-07-20T03:17:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd reporter subtree level = osd 2022-07-20T03:17:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: mon osd prime pg temp = true 2022-07-20T03:17:04.425 INFO:teuthology.orchestra.run.smithi061.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.426 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.426 INFO:teuthology.orchestra.run.smithi061.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:17:04.426 INFO:teuthology.orchestra.run.smithi061.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:17:04.427 INFO:teuthology.orchestra.run.smithi061.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:17:04.427 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.427 INFO:teuthology.orchestra.run.smithi061.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:17:04.428 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:17:04.428 INFO:teuthology.orchestra.run.smithi061.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:17:04.428 INFO:teuthology.orchestra.run.smithi061.stdout: mon op complaint time = 120 2022-07-20T03:17:04.428 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:04.429 INFO:teuthology.orchestra.run.smithi061.stdout:[client] 2022-07-20T03:17:04.429 INFO:teuthology.orchestra.run.smithi061.stdout: rgw cache enabled = true 2022-07-20T03:17:04.429 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable ops log = true 2022-07-20T03:17:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: rgw enable usage log = true 2022-07-20T03:17:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:17:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:17:04.430 INFO:teuthology.orchestra.run.smithi061.stdout: client mount timeout = 600 2022-07-20T03:17:04.431 INFO:teuthology.orchestra.run.smithi061.stdout: debug client = 20 2022-07-20T03:17:04.431 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:17:04.431 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.431 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.431 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.a] 2022-07-20T03:17:04.432 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.b] 2022-07-20T03:17:04.432 INFO:teuthology.orchestra.run.smithi061.stdout:[mon.c] 2022-07-20T03:17:04.432 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.a] 2022-07-20T03:17:04.432 INFO:teuthology.orchestra.run.smithi061.stdout:[mds.b] 2022-07-20T03:17:04.433 INFO:teuthology.orchestra.run.smithi061.stdout:[mds] 2022-07-20T03:17:04.433 INFO:teuthology.orchestra.run.smithi061.stdout: debug mds = 20 2022-07-20T03:17:04.433 INFO:teuthology.orchestra.run.smithi061.stdout: debug ms = 1 2022-07-20T03:17:04.434 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal frag = True 2022-07-20T03:17:04.434 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal fragment size max = 10000 2022-07-20T03:17:04.434 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal merge size = 5 2022-07-20T03:17:04.435 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split bits = 3 2022-07-20T03:17:04.435 INFO:teuthology.orchestra.run.smithi061.stdout: mds bal split size = 100 2022-07-20T03:17:04.435 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug frag = True 2022-07-20T03:17:04.435 INFO:teuthology.orchestra.run.smithi061.stdout: mds debug scatterstat = True 2022-07-20T03:17:04.436 INFO:teuthology.orchestra.run.smithi061.stdout: mds op complaint time = 180 2022-07-20T03:17:04.436 INFO:teuthology.orchestra.run.smithi061.stdout: mds verify scatter = True 2022-07-20T03:17:04.436 INFO:teuthology.orchestra.run.smithi061.stdout: osd op complaint time = 180 2022-07-20T03:17:04.436 INFO:teuthology.orchestra.run.smithi061.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.437 INFO:teuthology.orchestra.run.smithi061.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.441 INFO:teuthology.orchestra.run.smithi179.stdout:[global] 2022-07-20T03:17:04.441 INFO:teuthology.orchestra.run.smithi179.stdout: chdir = "" 2022-07-20T03:17:04.441 INFO:teuthology.orchestra.run.smithi179.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:17:04.442 INFO:teuthology.orchestra.run.smithi179.stdout: auth supported = cephx 2022-07-20T03:17:04.442 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.442 INFO:teuthology.orchestra.run.smithi179.stdout: filestore xattr use omap = true 2022-07-20T03:17:04.442 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.443 INFO:teuthology.orchestra.run.smithi179.stdout: mon clock drift allowed = 1.000 2022-07-20T03:17:04.443 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.443 INFO:teuthology.orchestra.run.smithi179.stdout: osd crush chooseleaf type = 0 2022-07-20T03:17:04.443 INFO:teuthology.orchestra.run.smithi179.stdout: auth debug = true 2022-07-20T03:17:04.444 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.444 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on old message = true 2022-07-20T03:17:04.444 INFO:teuthology.orchestra.run.smithi179.stdout: ms die on bug = true 2022-07-20T03:17:04.444 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.445 INFO:teuthology.orchestra.run.smithi179.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:17:04.445 INFO:teuthology.orchestra.run.smithi179.stdout: mon pg warn max object skew = 0 2022-07-20T03:17:04.445 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.445 INFO:teuthology.orchestra.run.smithi179.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:17:04.446 INFO:teuthology.orchestra.run.smithi179.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:17:04.446 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.446 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default size = 2 2022-07-20T03:17:04.446 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.447 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow primary affinity = true 2022-07-20T03:17:04.447 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd allow pg remap = true 2022-07-20T03:17:04.447 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:17:04.447 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:17:04.447 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on no sortbitwise = false 2022-07-20T03:17:04.448 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:17:04.448 INFO:teuthology.orchestra.run.smithi179.stdout: mon warn on too few osds = false 2022-07-20T03:17:04.448 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:17:04.449 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:17:04.449 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.449 INFO:teuthology.orchestra.run.smithi179.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:17:04.450 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.450 INFO:teuthology.orchestra.run.smithi179.stdout: osd default data pool replay window = 5 2022-07-20T03:17:04.450 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.450 INFO:teuthology.orchestra.run.smithi179.stdout: mon allow pool delete = true 2022-07-20T03:17:04.451 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.451 INFO:teuthology.orchestra.run.smithi179.stdout: mon cluster log file level = debug 2022-07-20T03:17:04.452 INFO:teuthology.orchestra.run.smithi179.stdout: debug asserts on shutdown = true 2022-07-20T03:17:04.452 INFO:teuthology.orchestra.run.smithi179.stdout: mon health detail to clog = false 2022-07-20T03:17:04.452 INFO:teuthology.orchestra.run.smithi179.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:17:04.453 INFO:teuthology.orchestra.run.smithi179.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:17:04.453 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.453 INFO:teuthology.orchestra.run.smithi179.stdout:[osd] 2022-07-20T03:17:04.454 INFO:teuthology.orchestra.run.smithi179.stdout: osd journal size = 100 2022-07-20T03:17:04.454 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.454 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub load threshold = 5.0 2022-07-20T03:17:04.455 INFO:teuthology.orchestra.run.smithi179.stdout: osd scrub max interval = 600 2022-07-20T03:17:04.455 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.455 INFO:teuthology.orchestra.run.smithi179.stdout: osd recover clone overlap = true 2022-07-20T03:17:04.455 INFO:teuthology.orchestra.run.smithi179.stdout: osd recovery max chunk = 1048576 2022-07-20T03:17:04.456 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.456 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug shutdown = true 2022-07-20T03:17:04.456 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug op order = true 2022-07-20T03:17:04.457 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug verify stray on activate = true 2022-07-20T03:17:04.457 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.457 INFO:teuthology.orchestra.run.smithi179.stdout: osd open classes on start = true 2022-07-20T03:17:04.457 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug pg log writeout = true 2022-07-20T03:17:04.458 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.458 INFO:teuthology.orchestra.run.smithi179.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:17:04.458 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.458 INFO:teuthology.orchestra.run.smithi179.stdout: osd map max advance = 10 2022-07-20T03:17:04.459 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.459 INFO:teuthology.orchestra.run.smithi179.stdout: journal zero on create = true 2022-07-20T03:17:04.459 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.459 INFO:teuthology.orchestra.run.smithi179.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:17:04.460 INFO:teuthology.orchestra.run.smithi179.stdout: filestore apply finisher threads = 3 2022-07-20T03:17:04.460 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.460 INFO:teuthology.orchestra.run.smithi179.stdout: bdev debug aio = true 2022-07-20T03:17:04.460 INFO:teuthology.orchestra.run.smithi179.stdout: osd debug misdirected ops = true 2022-07-20T03:17:04.461 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:17:04.461 INFO:teuthology.orchestra.run.smithi179.stdout: debug osd = 20 2022-07-20T03:17:04.461 INFO:teuthology.orchestra.run.smithi179.stdout: osd objectstore = filestore 2022-07-20T03:17:04.461 INFO:teuthology.orchestra.run.smithi179.stdout: osd op complaint time = 180 2022-07-20T03:17:04.462 INFO:teuthology.orchestra.run.smithi179.stdout: osd sloppy crc = True 2022-07-20T03:17:04.462 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.462 INFO:teuthology.orchestra.run.smithi179.stdout:[mgr] 2022-07-20T03:17:04.462 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:17:04.463 INFO:teuthology.orchestra.run.smithi179.stdout: debug mgr = 20 2022-07-20T03:17:04.463 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2022-07-20T03:17:04.463 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2022-07-20T03:17:04.463 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.464 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.464 INFO:teuthology.orchestra.run.smithi179.stdout: mgr/telemetry/nag = false 2022-07-20T03:17:04.464 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.464 INFO:teuthology.orchestra.run.smithi179.stdout:[mon] 2022-07-20T03:17:04.465 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:17:04.465 INFO:teuthology.orchestra.run.smithi179.stdout: debug mon = 20 2022-07-20T03:17:04.465 INFO:teuthology.orchestra.run.smithi179.stdout: debug paxos = 20 2022-07-20T03:17:04.465 INFO:teuthology.orchestra.run.smithi179.stdout: debug auth = 20 2022-07-20T03:17:04.466 INFO:teuthology.orchestra.run.smithi179.stdout: mon data avail warn = 5 2022-07-20T03:17:04.466 INFO:teuthology.orchestra.run.smithi179.stdout: mon mgr mkfs grace = 240 2022-07-20T03:17:04.466 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.466 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd reporter subtree level = osd 2022-07-20T03:17:04.467 INFO:teuthology.orchestra.run.smithi179.stdout: mon osd prime pg temp = true 2022-07-20T03:17:04.467 INFO:teuthology.orchestra.run.smithi179.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.467 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.467 INFO:teuthology.orchestra.run.smithi179.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:17:04.468 INFO:teuthology.orchestra.run.smithi179.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:17:04.468 INFO:teuthology.orchestra.run.smithi179.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:17:04.468 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.468 INFO:teuthology.orchestra.run.smithi179.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:17:04.469 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:17:04.469 INFO:teuthology.orchestra.run.smithi179.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:17:04.469 INFO:teuthology.orchestra.run.smithi179.stdout: mon op complaint time = 120 2022-07-20T03:17:04.469 INFO:teuthology.orchestra.run.smithi179.stdout: 2022-07-20T03:17:04.470 INFO:teuthology.orchestra.run.smithi179.stdout:[client] 2022-07-20T03:17:04.470 INFO:teuthology.orchestra.run.smithi179.stdout: rgw cache enabled = true 2022-07-20T03:17:04.470 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable ops log = true 2022-07-20T03:17:04.470 INFO:teuthology.orchestra.run.smithi179.stdout: rgw enable usage log = true 2022-07-20T03:17:04.470 INFO:teuthology.orchestra.run.smithi179.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:17:04.471 INFO:teuthology.orchestra.run.smithi179.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:17:04.471 INFO:teuthology.orchestra.run.smithi179.stdout: client mount timeout = 600 2022-07-20T03:17:04.471 INFO:teuthology.orchestra.run.smithi179.stdout: debug client = 20 2022-07-20T03:17:04.471 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:17:04.472 INFO:teuthology.orchestra.run.smithi179.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.472 INFO:teuthology.orchestra.run.smithi179.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.a] 2022-07-20T03:17:04.472 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.b] 2022-07-20T03:17:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:[mon.c] 2022-07-20T03:17:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.a] 2022-07-20T03:17:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:[mds.b] 2022-07-20T03:17:04.473 INFO:teuthology.orchestra.run.smithi179.stdout:[mds] 2022-07-20T03:17:04.474 INFO:teuthology.orchestra.run.smithi179.stdout: debug mds = 20 2022-07-20T03:17:04.474 INFO:teuthology.orchestra.run.smithi179.stdout: debug ms = 1 2022-07-20T03:17:04.474 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal frag = True 2022-07-20T03:17:04.474 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal fragment size max = 10000 2022-07-20T03:17:04.475 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal merge size = 5 2022-07-20T03:17:04.475 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal split bits = 3 2022-07-20T03:17:04.475 INFO:teuthology.orchestra.run.smithi179.stdout: mds bal split size = 100 2022-07-20T03:17:04.476 INFO:teuthology.orchestra.run.smithi179.stdout: mds debug frag = True 2022-07-20T03:17:04.476 INFO:teuthology.orchestra.run.smithi179.stdout: mds debug scatterstat = True 2022-07-20T03:17:04.476 INFO:teuthology.orchestra.run.smithi179.stdout: mds op complaint time = 180 2022-07-20T03:17:04.476 INFO:teuthology.orchestra.run.smithi179.stdout: mds verify scatter = True 2022-07-20T03:17:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: osd op complaint time = 180 2022-07-20T03:17:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.477 INFO:teuthology.orchestra.run.smithi179.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.477 INFO:teuthology.orchestra.run.smithi163.stdout:[global] 2022-07-20T03:17:04.478 INFO:teuthology.orchestra.run.smithi163.stdout: chdir = "" 2022-07-20T03:17:04.478 INFO:teuthology.orchestra.run.smithi163.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:17:04.478 INFO:teuthology.orchestra.run.smithi163.stdout: auth supported = cephx 2022-07-20T03:17:04.478 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.479 INFO:teuthology.orchestra.run.smithi163.stdout: filestore xattr use omap = true 2022-07-20T03:17:04.479 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.479 INFO:teuthology.orchestra.run.smithi163.stdout: mon clock drift allowed = 1.000 2022-07-20T03:17:04.479 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.480 INFO:teuthology.orchestra.run.smithi163.stdout: osd crush chooseleaf type = 0 2022-07-20T03:17:04.480 INFO:teuthology.orchestra.run.smithi163.stdout: auth debug = true 2022-07-20T03:17:04.480 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.480 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on old message = true 2022-07-20T03:17:04.481 INFO:teuthology.orchestra.run.smithi163.stdout: ms die on bug = true 2022-07-20T03:17:04.481 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.481 INFO:teuthology.orchestra.run.smithi163.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:17:04.481 INFO:teuthology.orchestra.run.smithi163.stdout: mon pg warn max object skew = 0 2022-07-20T03:17:04.482 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.482 INFO:teuthology.orchestra.run.smithi163.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:17:04.482 INFO:teuthology.orchestra.run.smithi163.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:17:04.482 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.483 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default size = 2 2022-07-20T03:17:04.483 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.483 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow primary affinity = true 2022-07-20T03:17:04.483 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd allow pg remap = true 2022-07-20T03:17:04.484 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:17:04.484 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:17:04.484 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on no sortbitwise = false 2022-07-20T03:17:04.484 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:17:04.485 INFO:teuthology.orchestra.run.smithi163.stdout: mon warn on too few osds = false 2022-07-20T03:17:04.485 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:17:04.485 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:17:04.485 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.486 INFO:teuthology.orchestra.run.smithi163.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:17:04.486 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.486 INFO:teuthology.orchestra.run.smithi163.stdout: osd default data pool replay window = 5 2022-07-20T03:17:04.486 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.487 INFO:teuthology.orchestra.run.smithi163.stdout: mon allow pool delete = true 2022-07-20T03:17:04.487 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.487 INFO:teuthology.orchestra.run.smithi163.stdout: mon cluster log file level = debug 2022-07-20T03:17:04.487 INFO:teuthology.orchestra.run.smithi163.stdout: debug asserts on shutdown = true 2022-07-20T03:17:04.488 INFO:teuthology.orchestra.run.smithi163.stdout: mon health detail to clog = false 2022-07-20T03:17:04.488 INFO:teuthology.orchestra.run.smithi163.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:17:04.488 INFO:teuthology.orchestra.run.smithi163.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:17:04.488 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.489 INFO:teuthology.orchestra.run.smithi163.stdout:[osd] 2022-07-20T03:17:04.489 INFO:teuthology.orchestra.run.smithi163.stdout: osd journal size = 100 2022-07-20T03:17:04.489 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.489 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub load threshold = 5.0 2022-07-20T03:17:04.490 INFO:teuthology.orchestra.run.smithi163.stdout: osd scrub max interval = 600 2022-07-20T03:17:04.490 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.490 INFO:teuthology.orchestra.run.smithi163.stdout: osd recover clone overlap = true 2022-07-20T03:17:04.490 INFO:teuthology.orchestra.run.smithi163.stdout: osd recovery max chunk = 1048576 2022-07-20T03:17:04.491 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.491 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug shutdown = true 2022-07-20T03:17:04.491 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug op order = true 2022-07-20T03:17:04.492 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug verify stray on activate = true 2022-07-20T03:17:04.492 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.492 INFO:teuthology.orchestra.run.smithi163.stdout: osd open classes on start = true 2022-07-20T03:17:04.492 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug pg log writeout = true 2022-07-20T03:17:04.493 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.493 INFO:teuthology.orchestra.run.smithi163.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:17:04.493 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.493 INFO:teuthology.orchestra.run.smithi163.stdout: osd map max advance = 10 2022-07-20T03:17:04.494 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.494 INFO:teuthology.orchestra.run.smithi163.stdout: journal zero on create = true 2022-07-20T03:17:04.494 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.494 INFO:teuthology.orchestra.run.smithi163.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:17:04.495 INFO:teuthology.orchestra.run.smithi163.stdout: filestore apply finisher threads = 3 2022-07-20T03:17:04.495 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.495 INFO:teuthology.orchestra.run.smithi163.stdout: bdev debug aio = true 2022-07-20T03:17:04.495 INFO:teuthology.orchestra.run.smithi163.stdout: osd debug misdirected ops = true 2022-07-20T03:17:04.496 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:17:04.496 INFO:teuthology.orchestra.run.smithi163.stdout: debug osd = 20 2022-07-20T03:17:04.496 INFO:teuthology.orchestra.run.smithi163.stdout: osd objectstore = filestore 2022-07-20T03:17:04.496 INFO:teuthology.orchestra.run.smithi163.stdout: osd op complaint time = 180 2022-07-20T03:17:04.497 INFO:teuthology.orchestra.run.smithi163.stdout: osd sloppy crc = True 2022-07-20T03:17:04.497 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.497 INFO:teuthology.orchestra.run.smithi163.stdout:[mgr] 2022-07-20T03:17:04.497 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:17:04.498 INFO:teuthology.orchestra.run.smithi163.stdout: debug mgr = 20 2022-07-20T03:17:04.498 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2022-07-20T03:17:04.498 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2022-07-20T03:17:04.498 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.499 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.499 INFO:teuthology.orchestra.run.smithi163.stdout: mgr/telemetry/nag = false 2022-07-20T03:17:04.499 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.499 INFO:teuthology.orchestra.run.smithi163.stdout:[mon] 2022-07-20T03:17:04.500 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:17:04.500 INFO:teuthology.orchestra.run.smithi163.stdout: debug mon = 20 2022-07-20T03:17:04.500 INFO:teuthology.orchestra.run.smithi163.stdout: debug paxos = 20 2022-07-20T03:17:04.500 INFO:teuthology.orchestra.run.smithi163.stdout: debug auth = 20 2022-07-20T03:17:04.501 INFO:teuthology.orchestra.run.smithi163.stdout: mon data avail warn = 5 2022-07-20T03:17:04.501 INFO:teuthology.orchestra.run.smithi163.stdout: mon mgr mkfs grace = 240 2022-07-20T03:17:04.501 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.502 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd reporter subtree level = osd 2022-07-20T03:17:04.502 INFO:teuthology.orchestra.run.smithi163.stdout: mon osd prime pg temp = true 2022-07-20T03:17:04.502 INFO:teuthology.orchestra.run.smithi163.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.502 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.503 INFO:teuthology.orchestra.run.smithi163.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:17:04.503 INFO:teuthology.orchestra.run.smithi163.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:17:04.503 INFO:teuthology.orchestra.run.smithi163.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:17:04.503 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.504 INFO:teuthology.orchestra.run.smithi163.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:17:04.504 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:17:04.504 INFO:teuthology.orchestra.run.smithi163.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:17:04.505 INFO:teuthology.orchestra.run.smithi163.stdout: mon op complaint time = 120 2022-07-20T03:17:04.505 INFO:teuthology.orchestra.run.smithi163.stdout: 2022-07-20T03:17:04.505 INFO:teuthology.orchestra.run.smithi163.stdout:[client] 2022-07-20T03:17:04.505 INFO:teuthology.orchestra.run.smithi163.stdout: rgw cache enabled = true 2022-07-20T03:17:04.506 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable ops log = true 2022-07-20T03:17:04.506 INFO:teuthology.orchestra.run.smithi163.stdout: rgw enable usage log = true 2022-07-20T03:17:04.506 INFO:teuthology.orchestra.run.smithi163.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:17:04.507 INFO:teuthology.orchestra.run.smithi163.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:17:04.507 INFO:teuthology.orchestra.run.smithi163.stdout: client mount timeout = 600 2022-07-20T03:17:04.507 INFO:teuthology.orchestra.run.smithi163.stdout: debug client = 20 2022-07-20T03:17:04.507 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:17:04.508 INFO:teuthology.orchestra.run.smithi163.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.508 INFO:teuthology.orchestra.run.smithi163.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.508 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.a] 2022-07-20T03:17:04.508 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.b] 2022-07-20T03:17:04.509 INFO:teuthology.orchestra.run.smithi163.stdout:[mon.c] 2022-07-20T03:17:04.509 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.a] 2022-07-20T03:17:04.509 INFO:teuthology.orchestra.run.smithi163.stdout:[mds.b] 2022-07-20T03:17:04.509 INFO:teuthology.orchestra.run.smithi163.stdout:[mds] 2022-07-20T03:17:04.510 INFO:teuthology.orchestra.run.smithi163.stdout: debug mds = 20 2022-07-20T03:17:04.510 INFO:teuthology.orchestra.run.smithi163.stdout: debug ms = 1 2022-07-20T03:17:04.510 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal frag = True 2022-07-20T03:17:04.510 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal fragment size max = 10000 2022-07-20T03:17:04.511 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal merge size = 5 2022-07-20T03:17:04.511 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal split bits = 3 2022-07-20T03:17:04.511 INFO:teuthology.orchestra.run.smithi163.stdout: mds bal split size = 100 2022-07-20T03:17:04.511 INFO:teuthology.orchestra.run.smithi163.stdout: mds debug frag = True 2022-07-20T03:17:04.512 INFO:teuthology.orchestra.run.smithi163.stdout: mds debug scatterstat = True 2022-07-20T03:17:04.512 INFO:teuthology.orchestra.run.smithi163.stdout: mds op complaint time = 180 2022-07-20T03:17:04.512 INFO:teuthology.orchestra.run.smithi163.stdout: mds verify scatter = True 2022-07-20T03:17:04.512 INFO:teuthology.orchestra.run.smithi163.stdout: osd op complaint time = 180 2022-07-20T03:17:04.513 INFO:teuthology.orchestra.run.smithi163.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.513 INFO:teuthology.orchestra.run.smithi163.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.513 INFO:teuthology.orchestra.run.smithi124.stdout:[global] 2022-07-20T03:17:04.514 INFO:teuthology.orchestra.run.smithi124.stdout: chdir = "" 2022-07-20T03:17:04.514 INFO:teuthology.orchestra.run.smithi124.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:17:04.514 INFO:teuthology.orchestra.run.smithi124.stdout: auth supported = cephx 2022-07-20T03:17:04.514 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.515 INFO:teuthology.orchestra.run.smithi124.stdout: filestore xattr use omap = true 2022-07-20T03:17:04.515 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.515 INFO:teuthology.orchestra.run.smithi124.stdout: mon clock drift allowed = 1.000 2022-07-20T03:17:04.515 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.516 INFO:teuthology.orchestra.run.smithi124.stdout: osd crush chooseleaf type = 0 2022-07-20T03:17:04.516 INFO:teuthology.orchestra.run.smithi124.stdout: auth debug = true 2022-07-20T03:17:04.516 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.516 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on old message = true 2022-07-20T03:17:04.517 INFO:teuthology.orchestra.run.smithi124.stdout: ms die on bug = true 2022-07-20T03:17:04.517 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.517 INFO:teuthology.orchestra.run.smithi124.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:17:04.517 INFO:teuthology.orchestra.run.smithi124.stdout: mon pg warn max object skew = 0 2022-07-20T03:17:04.518 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.518 INFO:teuthology.orchestra.run.smithi124.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:17:04.518 INFO:teuthology.orchestra.run.smithi124.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:17:04.518 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.519 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default size = 2 2022-07-20T03:17:04.519 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.519 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow primary affinity = true 2022-07-20T03:17:04.519 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd allow pg remap = true 2022-07-20T03:17:04.520 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:17:04.520 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:17:04.520 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on no sortbitwise = false 2022-07-20T03:17:04.520 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:17:04.521 INFO:teuthology.orchestra.run.smithi124.stdout: mon warn on too few osds = false 2022-07-20T03:17:04.521 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:17:04.521 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:17:04.521 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.522 INFO:teuthology.orchestra.run.smithi124.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:17:04.522 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.522 INFO:teuthology.orchestra.run.smithi124.stdout: osd default data pool replay window = 5 2022-07-20T03:17:04.522 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.523 INFO:teuthology.orchestra.run.smithi124.stdout: mon allow pool delete = true 2022-07-20T03:17:04.523 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.523 INFO:teuthology.orchestra.run.smithi124.stdout: mon cluster log file level = debug 2022-07-20T03:17:04.523 INFO:teuthology.orchestra.run.smithi124.stdout: debug asserts on shutdown = true 2022-07-20T03:17:04.524 INFO:teuthology.orchestra.run.smithi124.stdout: mon health detail to clog = false 2022-07-20T03:17:04.524 INFO:teuthology.orchestra.run.smithi124.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:17:04.524 INFO:teuthology.orchestra.run.smithi124.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:17:04.524 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.524 INFO:teuthology.orchestra.run.smithi124.stdout:[osd] 2022-07-20T03:17:04.525 INFO:teuthology.orchestra.run.smithi124.stdout: osd journal size = 100 2022-07-20T03:17:04.525 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.525 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub load threshold = 5.0 2022-07-20T03:17:04.525 INFO:teuthology.orchestra.run.smithi124.stdout: osd scrub max interval = 600 2022-07-20T03:17:04.526 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.526 INFO:teuthology.orchestra.run.smithi124.stdout: osd recover clone overlap = true 2022-07-20T03:17:04.526 INFO:teuthology.orchestra.run.smithi124.stdout: osd recovery max chunk = 1048576 2022-07-20T03:17:04.526 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.527 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug shutdown = true 2022-07-20T03:17:04.527 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug op order = true 2022-07-20T03:17:04.527 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug verify stray on activate = true 2022-07-20T03:17:04.527 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.528 INFO:teuthology.orchestra.run.smithi124.stdout: osd open classes on start = true 2022-07-20T03:17:04.528 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug pg log writeout = true 2022-07-20T03:17:04.528 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.528 INFO:teuthology.orchestra.run.smithi124.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:17:04.529 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.529 INFO:teuthology.orchestra.run.smithi124.stdout: osd map max advance = 10 2022-07-20T03:17:04.529 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.529 INFO:teuthology.orchestra.run.smithi124.stdout: journal zero on create = true 2022-07-20T03:17:04.530 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.530 INFO:teuthology.orchestra.run.smithi124.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:17:04.530 INFO:teuthology.orchestra.run.smithi124.stdout: filestore apply finisher threads = 3 2022-07-20T03:17:04.530 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.531 INFO:teuthology.orchestra.run.smithi124.stdout: bdev debug aio = true 2022-07-20T03:17:04.531 INFO:teuthology.orchestra.run.smithi124.stdout: osd debug misdirected ops = true 2022-07-20T03:17:04.531 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:17:04.531 INFO:teuthology.orchestra.run.smithi124.stdout: debug osd = 20 2022-07-20T03:17:04.532 INFO:teuthology.orchestra.run.smithi124.stdout: osd objectstore = filestore 2022-07-20T03:17:04.532 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2022-07-20T03:17:04.532 INFO:teuthology.orchestra.run.smithi124.stdout: osd sloppy crc = True 2022-07-20T03:17:04.532 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.533 INFO:teuthology.orchestra.run.smithi124.stdout:[mgr] 2022-07-20T03:17:04.533 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:17:04.533 INFO:teuthology.orchestra.run.smithi124.stdout: debug mgr = 20 2022-07-20T03:17:04.533 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2022-07-20T03:17:04.534 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2022-07-20T03:17:04.534 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.534 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.534 INFO:teuthology.orchestra.run.smithi124.stdout: mgr/telemetry/nag = false 2022-07-20T03:17:04.535 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.535 INFO:teuthology.orchestra.run.smithi124.stdout:[mon] 2022-07-20T03:17:04.535 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:17:04.535 INFO:teuthology.orchestra.run.smithi124.stdout: debug mon = 20 2022-07-20T03:17:04.536 INFO:teuthology.orchestra.run.smithi124.stdout: debug paxos = 20 2022-07-20T03:17:04.536 INFO:teuthology.orchestra.run.smithi124.stdout: debug auth = 20 2022-07-20T03:17:04.536 INFO:teuthology.orchestra.run.smithi124.stdout: mon data avail warn = 5 2022-07-20T03:17:04.536 INFO:teuthology.orchestra.run.smithi124.stdout: mon mgr mkfs grace = 240 2022-07-20T03:17:04.536 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.537 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd reporter subtree level = osd 2022-07-20T03:17:04.537 INFO:teuthology.orchestra.run.smithi124.stdout: mon osd prime pg temp = true 2022-07-20T03:17:04.537 INFO:teuthology.orchestra.run.smithi124.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.537 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.538 INFO:teuthology.orchestra.run.smithi124.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:17:04.538 INFO:teuthology.orchestra.run.smithi124.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:17:04.538 INFO:teuthology.orchestra.run.smithi124.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:17:04.538 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.539 INFO:teuthology.orchestra.run.smithi124.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:17:04.539 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:17:04.539 INFO:teuthology.orchestra.run.smithi124.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:17:04.539 INFO:teuthology.orchestra.run.smithi124.stdout: mon op complaint time = 120 2022-07-20T03:17:04.540 INFO:teuthology.orchestra.run.smithi124.stdout: 2022-07-20T03:17:04.540 INFO:teuthology.orchestra.run.smithi124.stdout:[client] 2022-07-20T03:17:04.540 INFO:teuthology.orchestra.run.smithi124.stdout: rgw cache enabled = true 2022-07-20T03:17:04.540 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable ops log = true 2022-07-20T03:17:04.541 INFO:teuthology.orchestra.run.smithi124.stdout: rgw enable usage log = true 2022-07-20T03:17:04.541 INFO:teuthology.orchestra.run.smithi124.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:17:04.541 INFO:teuthology.orchestra.run.smithi124.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:17:04.541 INFO:teuthology.orchestra.run.smithi124.stdout: client mount timeout = 600 2022-07-20T03:17:04.542 INFO:teuthology.orchestra.run.smithi124.stdout: debug client = 20 2022-07-20T03:17:04.542 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:17:04.542 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.542 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.543 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.a] 2022-07-20T03:17:04.543 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.b] 2022-07-20T03:17:04.543 INFO:teuthology.orchestra.run.smithi124.stdout:[mon.c] 2022-07-20T03:17:04.543 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.a] 2022-07-20T03:17:04.544 INFO:teuthology.orchestra.run.smithi124.stdout:[mds.b] 2022-07-20T03:17:04.544 INFO:teuthology.orchestra.run.smithi124.stdout:[mds] 2022-07-20T03:17:04.544 INFO:teuthology.orchestra.run.smithi124.stdout: debug mds = 20 2022-07-20T03:17:04.544 INFO:teuthology.orchestra.run.smithi124.stdout: debug ms = 1 2022-07-20T03:17:04.545 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal frag = True 2022-07-20T03:17:04.545 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal fragment size max = 10000 2022-07-20T03:17:04.545 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal merge size = 5 2022-07-20T03:17:04.545 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal split bits = 3 2022-07-20T03:17:04.545 INFO:teuthology.orchestra.run.smithi124.stdout: mds bal split size = 100 2022-07-20T03:17:04.546 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug frag = True 2022-07-20T03:17:04.546 INFO:teuthology.orchestra.run.smithi124.stdout: mds debug scatterstat = True 2022-07-20T03:17:04.546 INFO:teuthology.orchestra.run.smithi124.stdout: mds op complaint time = 180 2022-07-20T03:17:04.546 INFO:teuthology.orchestra.run.smithi124.stdout: mds verify scatter = True 2022-07-20T03:17:04.547 INFO:teuthology.orchestra.run.smithi124.stdout: osd op complaint time = 180 2022-07-20T03:17:04.547 INFO:teuthology.orchestra.run.smithi124.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.547 INFO:teuthology.orchestra.run.smithi124.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.547 INFO:teuthology.orchestra.run.smithi117.stdout:[global] 2022-07-20T03:17:04.548 INFO:teuthology.orchestra.run.smithi117.stdout: chdir = "" 2022-07-20T03:17:04.548 INFO:teuthology.orchestra.run.smithi117.stdout: pid file = /var/run/ceph/$cluster-$name.pid 2022-07-20T03:17:04.548 INFO:teuthology.orchestra.run.smithi117.stdout: auth supported = cephx 2022-07-20T03:17:04.549 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.549 INFO:teuthology.orchestra.run.smithi117.stdout: filestore xattr use omap = true 2022-07-20T03:17:04.549 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.549 INFO:teuthology.orchestra.run.smithi117.stdout: mon clock drift allowed = 1.000 2022-07-20T03:17:04.550 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.550 INFO:teuthology.orchestra.run.smithi117.stdout: osd crush chooseleaf type = 0 2022-07-20T03:17:04.550 INFO:teuthology.orchestra.run.smithi117.stdout: auth debug = true 2022-07-20T03:17:04.551 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.551 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on old message = true 2022-07-20T03:17:04.551 INFO:teuthology.orchestra.run.smithi117.stdout: ms die on bug = true 2022-07-20T03:17:04.551 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.551 INFO:teuthology.orchestra.run.smithi117.stdout: mon max pg per osd = 10000 # >= luminous 2022-07-20T03:17:04.552 INFO:teuthology.orchestra.run.smithi117.stdout: mon pg warn max object skew = 0 2022-07-20T03:17:04.552 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.552 INFO:teuthology.orchestra.run.smithi117.stdout: # disable pg_autoscaler by default for new pools 2022-07-20T03:17:04.552 INFO:teuthology.orchestra.run.smithi117.stdout: osd_pool_default_pg_autoscale_mode = off 2022-07-20T03:17:04.553 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.553 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default size = 2 2022-07-20T03:17:04.553 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.554 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow primary affinity = true 2022-07-20T03:17:04.554 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd allow pg remap = true 2022-07-20T03:17:04.554 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on legacy crush tunables = false 2022-07-20T03:17:04.555 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on crush straw calc version zero = false 2022-07-20T03:17:04.555 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on no sortbitwise = false 2022-07-20T03:17:04.555 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on osd down out interval zero = false 2022-07-20T03:17:04.555 INFO:teuthology.orchestra.run.smithi117.stdout: mon warn on too few osds = false 2022-07-20T03:17:04.556 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_pg_num_not_power_of_two = false 2022-07-20T03:17:04.556 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_pool_no_redundancy = false 2022-07-20T03:17:04.556 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.556 INFO:teuthology.orchestra.run.smithi117.stdout: osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd 2022-07-20T03:17:04.557 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.557 INFO:teuthology.orchestra.run.smithi117.stdout: osd default data pool replay window = 5 2022-07-20T03:17:04.557 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.558 INFO:teuthology.orchestra.run.smithi117.stdout: mon allow pool delete = true 2022-07-20T03:17:04.558 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.558 INFO:teuthology.orchestra.run.smithi117.stdout: mon cluster log file level = debug 2022-07-20T03:17:04.558 INFO:teuthology.orchestra.run.smithi117.stdout: debug asserts on shutdown = true 2022-07-20T03:17:04.559 INFO:teuthology.orchestra.run.smithi117.stdout: mon health detail to clog = false 2022-07-20T03:17:04.559 INFO:teuthology.orchestra.run.smithi117.stdout: mon host = "172.21.15.61,172.21.15.117,[v2:172.21.15.117:3301,v1:172.21.15.117:6790]" 2022-07-20T03:17:04.559 INFO:teuthology.orchestra.run.smithi117.stdout: fsid = 69b2e2c1-da2e-4470-a6a9-37f8d410d6ed 2022-07-20T03:17:04.559 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.560 INFO:teuthology.orchestra.run.smithi117.stdout:[osd] 2022-07-20T03:17:04.560 INFO:teuthology.orchestra.run.smithi117.stdout: osd journal size = 100 2022-07-20T03:17:04.560 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.560 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub load threshold = 5.0 2022-07-20T03:17:04.561 INFO:teuthology.orchestra.run.smithi117.stdout: osd scrub max interval = 600 2022-07-20T03:17:04.561 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.561 INFO:teuthology.orchestra.run.smithi117.stdout: osd recover clone overlap = true 2022-07-20T03:17:04.561 INFO:teuthology.orchestra.run.smithi117.stdout: osd recovery max chunk = 1048576 2022-07-20T03:17:04.562 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.562 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug shutdown = true 2022-07-20T03:17:04.562 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug op order = true 2022-07-20T03:17:04.562 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug verify stray on activate = true 2022-07-20T03:17:04.563 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.563 INFO:teuthology.orchestra.run.smithi117.stdout: osd open classes on start = true 2022-07-20T03:17:04.563 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug pg log writeout = true 2022-07-20T03:17:04.563 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.564 INFO:teuthology.orchestra.run.smithi117.stdout: osd deep scrub update digest min age = 30 2022-07-20T03:17:04.564 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.564 INFO:teuthology.orchestra.run.smithi117.stdout: osd map max advance = 10 2022-07-20T03:17:04.564 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.565 INFO:teuthology.orchestra.run.smithi117.stdout: journal zero on create = true 2022-07-20T03:17:04.565 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.565 INFO:teuthology.orchestra.run.smithi117.stdout: filestore ondisk finisher threads = 3 2022-07-20T03:17:04.565 INFO:teuthology.orchestra.run.smithi117.stdout: filestore apply finisher threads = 3 2022-07-20T03:17:04.566 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.566 INFO:teuthology.orchestra.run.smithi117.stdout: bdev debug aio = true 2022-07-20T03:17:04.566 INFO:teuthology.orchestra.run.smithi117.stdout: osd debug misdirected ops = true 2022-07-20T03:17:04.567 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:17:04.567 INFO:teuthology.orchestra.run.smithi117.stdout: debug osd = 20 2022-07-20T03:17:04.567 INFO:teuthology.orchestra.run.smithi117.stdout: osd objectstore = filestore 2022-07-20T03:17:04.567 INFO:teuthology.orchestra.run.smithi117.stdout: osd op complaint time = 180 2022-07-20T03:17:04.568 INFO:teuthology.orchestra.run.smithi117.stdout: osd sloppy crc = True 2022-07-20T03:17:04.568 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.568 INFO:teuthology.orchestra.run.smithi117.stdout:[mgr] 2022-07-20T03:17:04.568 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:17:04.569 INFO:teuthology.orchestra.run.smithi117.stdout: debug mgr = 20 2022-07-20T03:17:04.569 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2022-07-20T03:17:04.569 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2022-07-20T03:17:04.569 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.570 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.570 INFO:teuthology.orchestra.run.smithi117.stdout: mgr/telemetry/nag = false 2022-07-20T03:17:04.570 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.571 INFO:teuthology.orchestra.run.smithi117.stdout:[mon] 2022-07-20T03:17:04.571 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:17:04.571 INFO:teuthology.orchestra.run.smithi117.stdout: debug mon = 20 2022-07-20T03:17:04.571 INFO:teuthology.orchestra.run.smithi117.stdout: debug paxos = 20 2022-07-20T03:17:04.572 INFO:teuthology.orchestra.run.smithi117.stdout: debug auth = 20 2022-07-20T03:17:04.572 INFO:teuthology.orchestra.run.smithi117.stdout: mon data avail warn = 5 2022-07-20T03:17:04.572 INFO:teuthology.orchestra.run.smithi117.stdout: mon mgr mkfs grace = 240 2022-07-20T03:17:04.572 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min pgs per osd = 4 2022-07-20T03:17:04.573 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd reporter subtree level = osd 2022-07-20T03:17:04.573 INFO:teuthology.orchestra.run.smithi117.stdout: mon osd prime pg temp = true 2022-07-20T03:17:04.573 INFO:teuthology.orchestra.run.smithi117.stdout: mon reweight min bytes per osd = 10 2022-07-20T03:17:04.574 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.574 INFO:teuthology.orchestra.run.smithi117.stdout: # rotate auth tickets quickly to exercise renewal paths 2022-07-20T03:17:04.574 INFO:teuthology.orchestra.run.smithi117.stdout: auth mon ticket ttl = 660 # 11m 2022-07-20T03:17:04.574 INFO:teuthology.orchestra.run.smithi117.stdout: auth service ticket ttl = 240 # 4m 2022-07-20T03:17:04.575 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.575 INFO:teuthology.orchestra.run.smithi117.stdout: # don't complain about insecure global_id in the test suite 2022-07-20T03:17:04.575 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim = false 2022-07-20T03:17:04.576 INFO:teuthology.orchestra.run.smithi117.stdout: mon_warn_on_insecure_global_id_reclaim_allowed = false 2022-07-20T03:17:04.576 INFO:teuthology.orchestra.run.smithi117.stdout: mon op complaint time = 120 2022-07-20T03:17:04.576 INFO:teuthology.orchestra.run.smithi117.stdout: 2022-07-20T03:17:04.577 INFO:teuthology.orchestra.run.smithi117.stdout:[client] 2022-07-20T03:17:04.577 INFO:teuthology.orchestra.run.smithi117.stdout: rgw cache enabled = true 2022-07-20T03:17:04.577 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable ops log = true 2022-07-20T03:17:04.577 INFO:teuthology.orchestra.run.smithi117.stdout: rgw enable usage log = true 2022-07-20T03:17:04.578 INFO:teuthology.orchestra.run.smithi117.stdout: log file = /var/log/ceph/$cluster-$name.$pid.log 2022-07-20T03:17:04.578 INFO:teuthology.orchestra.run.smithi117.stdout: admin socket = /var/run/ceph/$cluster-$name.$pid.asok 2022-07-20T03:17:04.578 INFO:teuthology.orchestra.run.smithi117.stdout: client mount timeout = 600 2022-07-20T03:17:04.579 INFO:teuthology.orchestra.run.smithi117.stdout: debug client = 20 2022-07-20T03:17:04.579 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:17:04.579 INFO:teuthology.orchestra.run.smithi117.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.579 INFO:teuthology.orchestra.run.smithi117.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.580 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.a] 2022-07-20T03:17:04.580 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.b] 2022-07-20T03:17:04.580 INFO:teuthology.orchestra.run.smithi117.stdout:[mon.c] 2022-07-20T03:17:04.581 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.a] 2022-07-20T03:17:04.581 INFO:teuthology.orchestra.run.smithi117.stdout:[mds.b] 2022-07-20T03:17:04.581 INFO:teuthology.orchestra.run.smithi117.stdout:[mds] 2022-07-20T03:17:04.581 INFO:teuthology.orchestra.run.smithi117.stdout: debug mds = 20 2022-07-20T03:17:04.582 INFO:teuthology.orchestra.run.smithi117.stdout: debug ms = 1 2022-07-20T03:17:04.582 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal frag = True 2022-07-20T03:17:04.582 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal fragment size max = 10000 2022-07-20T03:17:04.582 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal merge size = 5 2022-07-20T03:17:04.583 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal split bits = 3 2022-07-20T03:17:04.583 INFO:teuthology.orchestra.run.smithi117.stdout: mds bal split size = 100 2022-07-20T03:17:04.583 INFO:teuthology.orchestra.run.smithi117.stdout: mds debug frag = True 2022-07-20T03:17:04.584 INFO:teuthology.orchestra.run.smithi117.stdout: mds debug scatterstat = True 2022-07-20T03:17:04.584 INFO:teuthology.orchestra.run.smithi117.stdout: mds op complaint time = 180 2022-07-20T03:17:04.584 INFO:teuthology.orchestra.run.smithi117.stdout: mds verify scatter = True 2022-07-20T03:17:04.584 INFO:teuthology.orchestra.run.smithi117.stdout: osd op complaint time = 180 2022-07-20T03:17:04.585 INFO:teuthology.orchestra.run.smithi117.stdout: rados mon op timeout = 15m 2022-07-20T03:17:04.585 INFO:teuthology.orchestra.run.smithi117.stdout: rados osd op timeout = 15m 2022-07-20T03:17:04.585 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:17:04.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 Processor -- start 2022-07-20T03:17:04.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 -- start start 2022-07-20T03:17:04.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff7340b9ea0 0x7ff7340b4300 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff7340b00f0 0x7ff734143490 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff734142dc0 con 0x7ff7340b3df0 2022-07-20T03:17:04.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff7340ba280 con 0x7ff7340b00f0 2022-07-20T03:17:04.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff73ada6700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff734146280 con 0x7ff7340b9ea0 2022-07-20T03:17:04.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff739343700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff7340b3df0 0x7ff73405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47398/0 (socket says 172.21.15.61:47398) 2022-07-20T03:17:04.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff733fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff7340b00f0 0x7ff734143490 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff739343700 1 -- 172.21.15.61:0/802592132 learned_addr learned my addr 172.21.15.61:0/802592132 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:04.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff738b42700 1 --2- 172.21.15.61:0/802592132 >> v2:172.21.15.117:3300/0 conn(0x7ff7340b9ea0 0x7ff7340b4300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff7337fe700 1 -- 172.21.15.61:0/802592132 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3072986617 0 0) 0x7ff734142dc0 con 0x7ff7340b3df0 2022-07-20T03:17:04.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff7337fe700 1 -- 172.21.15.61:0/802592132 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff718003260 con 0x7ff7340b3df0 2022-07-20T03:17:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff733fff700 1 -- 172.21.15.61:0/802592132 >> v2:172.21.15.117:3300/0 conn(0x7ff7340b9ea0 msgr2=0x7ff7340b4300 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff733fff700 1 --2- 172.21.15.61:0/802592132 >> v2:172.21.15.117:3300/0 conn(0x7ff7340b9ea0 0x7ff7340b4300 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff733fff700 1 -- 172.21.15.61:0/802592132 >> v1:172.21.15.61:6789/0 conn(0x7ff7340b3df0 legacy=0x7ff73405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff733fff700 1 -- 172.21.15.61:0/802592132 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff734142ff0 con 0x7ff7340b00f0 2022-07-20T03:17:04.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff733fff700 1 --2- 172.21.15.61:0/802592132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b00f0 0x7ff734143490 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7ff71c002f70 tx=0x7ff71c006df0).ready entity=mon.0 client_cookie=20192123a33962fc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:04.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff7337fe700 1 -- 172.21.15.61:0/802592132 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c00d4f0 con 0x7ff7340b00f0 2022-07-20T03:17:04.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff7337fe700 1 -- 172.21.15.61:0/802592132 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c000e90 con 0x7ff7340b00f0 2022-07-20T03:17:04.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.695+0000 7ff7337fe700 1 -- 172.21.15.61:0/802592132 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c00d840 con 0x7ff7340b00f0 2022-07-20T03:17:04.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- 172.21.15.61:0/802592132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b00f0 msgr2=0x7ff734143490 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 --2- 172.21.15.61:0/802592132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b00f0 0x7ff734143490 secure :-1 s=READY pgs=1335 cs=0 l=1 rev1=1 rx=0x7ff71c002f70 tx=0x7ff71c006df0).stop 2022-07-20T03:17:04.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- 172.21.15.61:0/802592132 shutdown_connections 2022-07-20T03:17:04.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 --2- 172.21.15.61:0/802592132 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b00f0 0x7ff734143490 unknown :-1 s=CLOSED pgs=1335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 --2- 172.21.15.61:0/802592132 >> v2:172.21.15.117:3300/0 conn(0x7ff7340b9ea0 0x7ff7340b4300 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- 172.21.15.61:0/802592132 >> 172.21.15.61:0/802592132 conn(0x7ff734067960 msgr2=0x7ff7340b9450 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- 172.21.15.61:0/802592132 shutdown_connections 2022-07-20T03:17:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- 172.21.15.61:0/802592132 wait complete. 2022-07-20T03:17:04.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 Processor -- start 2022-07-20T03:17:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- start start 2022-07-20T03:17:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7340b00f0 0x7ff734145210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 0x7ff7340af520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7340b9ea0 0x7ff73413ee20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734156720 con 0x7ff7340b3df0 2022-07-20T03:17:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff734156480 con 0x7ff7340b9ea0 2022-07-20T03:17:04.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff73ada6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff734156160 con 0x7ff7340b00f0 2022-07-20T03:17:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff733fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 0x7ff7340af520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff733fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 0x7ff7340af520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50806/0 (socket says 172.21.15.61:50806) 2022-07-20T03:17:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff733fff700 1 -- 172.21.15.61:0/3726857619 learned_addr learned my addr 172.21.15.61:0/3726857619 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:04.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff739343700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7340b9ea0 0x7ff73413ee20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff738b42700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7340b00f0 0x7ff734145210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.699+0000 7ff733fff700 1 -- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7340b00f0 msgr2=0x7ff734145210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff733fff700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7340b00f0 0x7ff734145210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff733fff700 1 -- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7340b9ea0 msgr2=0x7ff73413ee20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff733fff700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7340b9ea0 0x7ff73413ee20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff733fff700 1 -- 172.21.15.61:0/3726857619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff71c00b180 con 0x7ff7340b3df0 2022-07-20T03:17:04.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff738b42700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7340b00f0 0x7ff734145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:17:04.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff739343700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7340b9ea0 0x7ff73413ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:04.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff733fff700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 0x7ff7340af520 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7ff71c002f70 tx=0x7ff71c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c016070 con 0x7ff7340b3df0 2022-07-20T03:17:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff71c0139c0 con 0x7ff7340b3df0 2022-07-20T03:17:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff734145fa0 con 0x7ff7340b3df0 2022-07-20T03:17:04.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff71c013d20 con 0x7ff7340b3df0 2022-07-20T03:17:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff734145c40 con 0x7ff7340b3df0 2022-07-20T03:17:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff71c00f030 con 0x7ff7340b3df0 2022-07-20T03:17:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff734112980 con 0x7ff7340b3df0 2022-07-20T03:17:04.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff731ffb700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff70c053ad0 0x7ff70c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7ff71c06c890 con 0x7ff7340b3df0 2022-07-20T03:17:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.703+0000 7ff738b42700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff70c053ad0 0x7ff70c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.707+0000 7ff738b42700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff70c053ad0 0x7ff70c055f90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7ff72400b000 tx=0x7ff724008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:04.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.711+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff734112980 con 0x7ff7340b3df0 2022-07-20T03:17:04.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.839+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7ff734112980 con 0x7ff7340b3df0 2022-07-20T03:17:04.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.847+0000 7ff731ffb700 1 -- 172.21.15.61:0/3726857619 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v21) v1 ==== 104+0+0 (secure 0 0 0) 0x7ff734112980 con 0x7ff7340b3df0 2022-07-20T03:17:04.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.847+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff70c053ad0 msgr2=0x7ff70c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.847+0000 7ff73ada6700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff70c053ad0 0x7ff70c055f90 secure :-1 s=READY pgs=644 cs=0 l=1 rev1=1 rx=0x7ff72400b000 tx=0x7ff724008040).stop 2022-07-20T03:17:04.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.847+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 msgr2=0x7ff7340af520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.847+0000 7ff73ada6700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 0x7ff7340af520 secure :-1 s=READY pgs=1336 cs=0 l=1 rev1=1 rx=0x7ff71c002f70 tx=0x7ff71c014000).stop 2022-07-20T03:17:04.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 shutdown_connections 2022-07-20T03:17:04.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff70c053ad0 0x7ff70c055f90 unknown :-1 s=CLOSED pgs=644 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff7340b9ea0 0x7ff73413ee20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff7340b3df0 0x7ff7340af520 unknown :-1 s=CLOSED pgs=1336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 --2- 172.21.15.61:0/3726857619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff7340b00f0 0x7ff734145210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 >> 172.21.15.61:0/3726857619 conn(0x7ff734067960 msgr2=0x7ff7340b86c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 shutdown_connections 2022-07-20T03:17:04.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.851+0000 7ff73ada6700 1 -- 172.21.15.61:0/3726857619 wait complete. 2022-07-20T03:17:04.870 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps' 2022-07-20T03:17:04.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 Processor -- start 2022-07-20T03:17:04.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 -- start start 2022-07-20T03:17:04.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f08bc0b6880 0x7f08bc143590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08bc145f90 0x7f08bc0b3d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc13fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:04.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc1429c0 con 0x7f08bc145f90 2022-07-20T03:17:04.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f08bc0b6c60 con 0x7f08bc0b6880 2022-07-20T03:17:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1f57700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc142bb0 con 0x7f08bc0b4220 2022-07-20T03:17:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08bc145f90 0x7f08bc0b3d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c0f55700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f08bc0b6880 0x7f08bc143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08c1756700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc13fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:04.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f08bc145f90 0x7f08bc0b3d30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50816/0 (socket says 172.21.15.61:50816) 2022-07-20T03:17:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 -- 172.21.15.61:0/1956656837 learned_addr learned my addr 172.21.15.61:0/1956656837 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:04.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 -- 172.21.15.61:0/1956656837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 msgr2=0x7f08bc13fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 --2- 172.21.15.61:0/1956656837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc13fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 -- 172.21.15.61:0/1956656837 >> v2:172.21.15.117:3300/0 conn(0x7f08bc0b6880 msgr2=0x7f08bc143590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:04.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 --2- 172.21.15.61:0/1956656837 >> v2:172.21.15.117:3300/0 conn(0x7f08bc0b6880 0x7f08bc143590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 -- 172.21.15.61:0/1956656837 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08bc142e50 con 0x7f08bc145f90 2022-07-20T03:17:04.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bbfff700 1 --2- 172.21.15.61:0/1956656837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc145f90 0x7f08bc0b3d30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac006eb0).ready entity=mon.0 client_cookie=ced882f9159eaba3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:04.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bb7fe700 1 -- 172.21.15.61:0/1956656837 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac00d570 con 0x7f08bc145f90 2022-07-20T03:17:04.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bb7fe700 1 -- 172.21.15.61:0/1956656837 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac000e90 con 0x7f08bc145f90 2022-07-20T03:17:04.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.967+0000 7f08bb7fe700 1 -- 172.21.15.61:0/1956656837 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac00d900 con 0x7f08bc145f90 2022-07-20T03:17:04.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 -- 172.21.15.61:0/1956656837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc145f90 msgr2=0x7f08bc0b3d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:04.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 --2- 172.21.15.61:0/1956656837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc145f90 0x7f08bc0b3d30 secure :-1 s=READY pgs=1337 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac006eb0).stop 2022-07-20T03:17:04.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 -- 172.21.15.61:0/1956656837 shutdown_connections 2022-07-20T03:17:04.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 --2- 172.21.15.61:0/1956656837 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc13fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 --2- 172.21.15.61:0/1956656837 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc145f90 0x7f08bc0b3d30 unknown :-1 s=CLOSED pgs=1337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 --2- 172.21.15.61:0/1956656837 >> v2:172.21.15.117:3300/0 conn(0x7f08bc0b6880 0x7f08bc143590 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:04.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 -- 172.21.15.61:0/1956656837 >> 172.21.15.61:0/1956656837 conn(0x7f08bc0b9e60 msgr2=0x7f08bc0b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:04.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 -- 172.21.15.61:0/1956656837 shutdown_connections 2022-07-20T03:17:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 -- 172.21.15.61:0/1956656837 wait complete. 2022-07-20T03:17:04.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 Processor -- start 2022-07-20T03:17:04.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.971+0000 7f08c1f57700 1 -- start start 2022-07-20T03:17:05.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc145160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 0x7f08bc1553a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08bc145f90 0x7f08bc138d70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc155890 con 0x7f08bc0b6880 2022-07-20T03:17:05.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc0ae960 con 0x7f08bc145f90 2022-07-20T03:17:05.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f08bc155ab0 con 0x7f08bc0b4220 2022-07-20T03:17:05.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 0x7f08bc1553a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 0x7f08bc1553a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50822/0 (socket says 172.21.15.61:50822) 2022-07-20T03:17:05.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 -- 172.21.15.61:0/4275412940 learned_addr learned my addr 172.21.15.61:0/4275412940 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:05.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1756700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08bc145f90 0x7f08bc138d70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c0f55700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc145160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 -- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 msgr2=0x7f08bc145160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc145160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 -- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08bc145f90 msgr2=0x7f08bc138d70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08bc145f90 0x7f08bc138d70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 -- 172.21.15.61:0/4275412940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08ac00b1e0 con 0x7f08bc0b6880 2022-07-20T03:17:05.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1756700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08bc145f90 0x7f08bc138d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:05.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08bbfff700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 0x7f08bc1553a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:05.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac0116e0 con 0x7f08bc0b6880 2022-07-20T03:17:05.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f08ac011840 con 0x7f08bc0b6880 2022-07-20T03:17:05.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08bc155d50 con 0x7f08bc0b6880 2022-07-20T03:17:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f08ac011ba0 con 0x7f08bc0b6880 2022-07-20T03:17:05.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08bc0ad950 con 0x7f08bc0b6880 2022-07-20T03:17:05.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08bc1430f0 con 0x7f08bc0b6880 2022-07-20T03:17:05.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.975+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f08ac025030 con 0x7f08bc0b6880 2022-07-20T03:17:05.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.979+0000 7f08b9ffb700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0894053a60 0x7f0894055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.979+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f08ac06ccf0 con 0x7f08bc0b6880 2022-07-20T03:17:05.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.979+0000 7f08c0f55700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0894053a60 0x7f0894055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.979+0000 7f08c0f55700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0894053a60 0x7f0894055f20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f08a400b000 tx=0x7f08a4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:05.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:04.983+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f08ac043f30 con 0x7f08bc0b6880 2022-07-20T03:17:05.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.151+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]} v 0) v1 -- 0x7f08bc0574e0 con 0x7f08bc0b6880 2022-07-20T03:17:05.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.335+0000 7f08b9ffb700 1 -- 172.21.15.61:0/4275412940 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_getattr_caps"]}]=0 v465) v1 ==== 128+0+0 (secure 0 0 0) 0x7f08ac023020 con 0x7f08bc0b6880 2022-07-20T03:17:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0894053a60 msgr2=0x7f0894055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0894053a60 0x7f0894055f20 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f08a400b000 tx=0x7f08a4006df0).stop 2022-07-20T03:17:05.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 msgr2=0x7f08bc1553a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 0x7f08bc1553a0 secure :-1 s=READY pgs=1338 cs=0 l=1 rev1=1 rx=0x7f08ac002f70 tx=0x7f08ac015000).stop 2022-07-20T03:17:05.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 shutdown_connections 2022-07-20T03:17:05.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0894053a60 0x7f0894055f20 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f08bc145f90 0x7f08bc138d70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f08bc0b6880 0x7f08bc1553a0 unknown :-1 s=CLOSED pgs=1338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 --2- 172.21.15.61:0/4275412940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f08bc0b4220 0x7f08bc145160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 >> 172.21.15.61:0/4275412940 conn(0x7f08bc0b9e60 msgr2=0x7f08bc141d90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:05.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 shutdown_connections 2022-07-20T03:17:05.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.339+0000 7f08c1f57700 1 -- 172.21.15.61:0/4275412940 wait complete. 2022-07-20T03:17:05.364 INFO:tasks.cephfs_test_runner:test_getattr_caps (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:17:05.365 INFO:tasks.cephfs_test_runner:Starting test: test_root_rctime (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:17:05.365 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-20T03:17:05.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.475+0000 7f68cf59e700 1 Processor -- start 2022-07-20T03:17:05.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.475+0000 7f68cf59e700 1 -- start start 2022-07-20T03:17:05.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68d00b5650 0x7f68d00b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d00b1200 0x7f68d005c4f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f68d00aea30 con 0x7f68d005bd90 2022-07-20T03:17:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f68d0142970 con 0x7f68d00b5650 2022-07-20T03:17:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d00aec60 con 0x7f68d00b1200 2022-07-20T03:17:05.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ce59c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f68d00b5650 0x7f68d00b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ced9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f68d005bd90 0x7f68d0142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47426/0 (socket says 172.21.15.61:47426) 2022-07-20T03:17:05.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ced9d700 1 -- 172.21.15.61:0/1595351933 learned_addr learned my addr 172.21.15.61:0/1595351933 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:05.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68c7fff700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d00b1200 0x7f68d005c4f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ce59c700 1 -- 172.21.15.61:0/1595351933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d00b1200 msgr2=0x7f68d005c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ce59c700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d00b1200 0x7f68d005c4f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ce59c700 1 -- 172.21.15.61:0/1595351933 >> v1:172.21.15.61:6789/0 conn(0x7f68d005bd90 legacy=0x7f68d0142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cdd9b700 1 -- 172.21.15.61:0/1595351933 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 988433404 0 0) 0x7f68d00aea30 con 0x7f68d005bd90 2022-07-20T03:17:05.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ce59c700 1 -- 172.21.15.61:0/1595351933 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d00af180 con 0x7f68d00b5650 2022-07-20T03:17:05.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68c7fff700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d00b1200 0x7f68d005c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:05.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68ce59c700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d00b5a30 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f68c0002f70 tx=0x7f68c0006df0).ready entity=mon.0 client_cookie=64d92f1ec9e5ea01 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:05.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cdd9b700 1 -- 172.21.15.61:0/1595351933 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c000d570 con 0x7f68d00b5650 2022-07-20T03:17:05.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cdd9b700 1 -- 172.21.15.61:0/1595351933 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c0000e90 con 0x7f68d00b5650 2022-07-20T03:17:05.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cdd9b700 1 -- 172.21.15.61:0/1595351933 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c000d8c0 con 0x7f68d00b5650 2022-07-20T03:17:05.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- 172.21.15.61:0/1595351933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 msgr2=0x7f68d00b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d00b5a30 secure :-1 s=READY pgs=1340 cs=0 l=1 rev1=1 rx=0x7f68c0002f70 tx=0x7f68c0006df0).stop 2022-07-20T03:17:05.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- 172.21.15.61:0/1595351933 shutdown_connections 2022-07-20T03:17:05.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d00b1200 0x7f68d005c4f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 --2- 172.21.15.61:0/1595351933 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d00b5a30 unknown :-1 s=CLOSED pgs=1340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- 172.21.15.61:0/1595351933 >> 172.21.15.61:0/1595351933 conn(0x7f68d00634e0 msgr2=0x7f68d00b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:05.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- 172.21.15.61:0/1595351933 shutdown_connections 2022-07-20T03:17:05.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.479+0000 7f68cf59e700 1 -- 172.21.15.61:0/1595351933 wait complete. 2022-07-20T03:17:05.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 Processor -- start 2022-07-20T03:17:05.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 -- start start 2022-07-20T03:17:05.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d005bd90 0x7f68d013dd10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68d00b1200 0x7f68d013e600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d0066e60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d0064840 con 0x7f68d00b5650 2022-07-20T03:17:05.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68d013eaf0 con 0x7f68d00b1200 2022-07-20T03:17:05.497 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.483+0000 7f68cf59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f68d00b47b0 con 0x7f68d005bd90 2022-07-20T03:17:05.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d0066e60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d0066e60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50836/0 (socket says 172.21.15.61:50836) 2022-07-20T03:17:05.498 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 -- 172.21.15.61:0/3850182293 learned_addr learned my addr 172.21.15.61:0/3850182293 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:05.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68c7fff700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68d00b1200 0x7f68d013e600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ce59c700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d005bd90 0x7f68d013dd10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 -- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d005bd90 msgr2=0x7f68d013dd10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d005bd90 0x7f68d013dd10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 -- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68d00b1200 msgr2=0x7f68d013e600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:05.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68d00b1200 0x7f68d013e600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 -- 172.21.15.61:0/3850182293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68c000b180 con 0x7f68d00b5650 2022-07-20T03:17:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68ced9d700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d0066e60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f68c8001250 tx=0x7f68c8007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c8005ba0 con 0x7f68d00b5650 2022-07-20T03:17:05.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f68c8005d00 con 0x7f68d00b5650 2022-07-20T03:17:05.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68d013ed70 con 0x7f68d00b5650 2022-07-20T03:17:05.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f68c8004a90 con 0x7f68d00b5650 2022-07-20T03:17:05.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68d00b5110 con 0x7f68d00b5650 2022-07-20T03:17:05.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68d0064c30 con 0x7f68d00b5650 2022-07-20T03:17:05.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f68c801b070 con 0x7f68d00b5650 2022-07-20T03:17:05.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.487+0000 7f68c77fe700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68b4053ad0 0x7f68b4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:05.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.491+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f68c806b4f0 con 0x7f68d00b5650 2022-07-20T03:17:05.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.491+0000 7f68ce59c700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68b4053ad0 0x7f68b4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:05.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.491+0000 7f68ce59c700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68b4053ad0 0x7f68b4055f90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f68c001a000 tx=0x7f68c0015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:05.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.491+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f68d0064c30 con 0x7f68d00b5650 2022-07-20T03:17:05.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:05.655+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7f68d0064c30 con 0x7f68d00b5650 2022-07-20T03:17:06.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.339+0000 7f68c77fe700 1 -- 172.21.15.61:0/3850182293 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v466) v1 ==== 130+0+0 (secure 0 0 0) 0x7f68d0064c30 con 0x7f68d00b5650 2022-07-20T03:17:06.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68b4053ad0 msgr2=0x7f68b4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68b4053ad0 0x7f68b4055f90 secure :-1 s=READY pgs=646 cs=0 l=1 rev1=1 rx=0x7f68c001a000 tx=0x7f68c0015000).stop 2022-07-20T03:17:06.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 msgr2=0x7f68d0066e60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d0066e60 secure :-1 s=READY pgs=1341 cs=0 l=1 rev1=1 rx=0x7f68c8001250 tx=0x7f68c8007d80).stop 2022-07-20T03:17:06.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 shutdown_connections 2022-07-20T03:17:06.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f68b4053ad0 0x7f68b4055f90 unknown :-1 s=CLOSED pgs=646 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f68d00b1200 0x7f68d013e600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f68d00b5650 0x7f68d0066e60 unknown :-1 s=CLOSED pgs=1341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 --2- 172.21.15.61:0/3850182293 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f68d005bd90 0x7f68d013dd10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 >> 172.21.15.61:0/3850182293 conn(0x7f68d00634e0 msgr2=0x7f68d0140540 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:06.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 shutdown_connections 2022-07-20T03:17:06.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.343+0000 7f68cf59e700 1 -- 172.21.15.61:0/3850182293 wait complete. 2022-07-20T03:17:06.364 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:17:06.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.479+0000 7f7ec8edb700 1 Processor -- start 2022-07-20T03:17:06.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.479+0000 7f7ec8edb700 1 -- start start 2022-07-20T03:17:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ec41462a0 0x7f7ec4146680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7ec40988b0 0x7f7ec4146b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7ec40afdb0 con 0x7f7ec41422c0 2022-07-20T03:17:06.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec40b0050 con 0x7f7ec41462a0 2022-07-20T03:17:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7ec405fcd0 con 0x7f7ec40988b0 2022-07-20T03:17:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ec41462a0 0x7f7ec4146680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f7ec41462a0 0x7f7ec4146680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50840/0 (socket says 172.21.15.61:50840) 2022-07-20T03:17:06.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 -- 172.21.15.61:0/3732519716 learned_addr learned my addr 172.21.15.61:0/3732519716 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec1d9b700 1 --2- 172.21.15.61:0/3732519716 >> v2:172.21.15.117:3300/0 conn(0x7f7ec40988b0 0x7f7ec4146b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 -- 172.21.15.61:0/3732519716 >> v2:172.21.15.117:3300/0 conn(0x7f7ec40988b0 msgr2=0x7f7ec4146b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:06.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 --2- 172.21.15.61:0/3732519716 >> v2:172.21.15.117:3300/0 conn(0x7f7ec40988b0 0x7f7ec4146b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 -- 172.21.15.61:0/3732519716 >> v1:172.21.15.117:6789/0 conn(0x7f7ec41422c0 legacy=0x7f7ec4060170 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 -- 172.21.15.61:0/3732519716 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7ec40afae0 con 0x7f7ec41462a0 2022-07-20T03:17:06.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec1d9b700 1 --2- 172.21.15.61:0/3732519716 >> v2:172.21.15.117:3300/0 conn(0x7f7ec40988b0 0x7f7ec4146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:06.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec259c700 1 --2- 172.21.15.61:0/3732519716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41462a0 0x7f7ec4146680 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f7eb8002f70 tx=0x7f7eb8006eb0).ready entity=mon.0 client_cookie=e13d9b2da23adcd4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:06.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec159a700 1 -- 172.21.15.61:0/3732519716 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb800d610 con 0x7f7ec41462a0 2022-07-20T03:17:06.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec159a700 1 -- 172.21.15.61:0/3732519716 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb8000e90 con 0x7f7ec41462a0 2022-07-20T03:17:06.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec159a700 1 -- 172.21.15.61:0/3732519716 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb800d9a0 con 0x7f7ec41462a0 2022-07-20T03:17:06.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- 172.21.15.61:0/3732519716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41462a0 msgr2=0x7f7ec4146680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/3732519716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41462a0 0x7f7ec4146680 secure :-1 s=READY pgs=1342 cs=0 l=1 rev1=1 rx=0x7f7eb8002f70 tx=0x7f7eb8006eb0).stop 2022-07-20T03:17:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- 172.21.15.61:0/3732519716 shutdown_connections 2022-07-20T03:17:06.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/3732519716 >> v2:172.21.15.117:3300/0 conn(0x7f7ec40988b0 0x7f7ec4146b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/3732519716 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41462a0 0x7f7ec4146680 unknown :-1 s=CLOSED pgs=1342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- 172.21.15.61:0/3732519716 >> 172.21.15.61:0/3732519716 conn(0x7f7ec40b5f00 msgr2=0x7f7ec40b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:06.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- 172.21.15.61:0/3732519716 shutdown_connections 2022-07-20T03:17:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.483+0000 7f7ec8edb700 1 -- 172.21.15.61:0/3732519716 wait complete. 2022-07-20T03:17:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 Processor -- start 2022-07-20T03:17:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 -- start start 2022-07-20T03:17:06.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ec40988b0 0x7f7ec4152c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 0x7f7ec40b8130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ec41462a0 0x7f7ec4155640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec4152240 con 0x7f7ec41422c0 2022-07-20T03:17:06.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec41524e0 con 0x7f7ec41462a0 2022-07-20T03:17:06.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec8edb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7ec4152780 con 0x7f7ec40988b0 2022-07-20T03:17:06.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 0x7f7ec40b8130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec2d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ec41462a0 0x7f7ec4155640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 0x7f7ec40b8130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50850/0 (socket says 172.21.15.61:50850) 2022-07-20T03:17:06.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 -- 172.21.15.61:0/4188460216 learned_addr learned my addr 172.21.15.61:0/4188460216 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:06.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec259c700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ec40988b0 0x7f7ec4152c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 -- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ec40988b0 msgr2=0x7f7ec4152c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ec40988b0 0x7f7ec4152c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 -- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ec41462a0 msgr2=0x7f7ec4155640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ec41462a0 0x7f7ec4155640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 -- 172.21.15.61:0/4188460216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7eb800b1e0 con 0x7f7ec41422c0 2022-07-20T03:17:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.487+0000 7f7ec1d9b700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 0x7f7ec40b8130 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f7eb4006940 tx=0x7f7eb4008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:06.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb4011780 con 0x7f7ec41422c0 2022-07-20T03:17:06.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7eb40118e0 con 0x7f7ec41422c0 2022-07-20T03:17:06.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7ec40b8620 con 0x7f7ec41422c0 2022-07-20T03:17:06.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7eb4011c40 con 0x7f7ec41422c0 2022-07-20T03:17:06.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7ec40b09f0 con 0x7f7ec41422c0 2022-07-20T03:17:06.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7ec4152940 con 0x7f7ec41422c0 2022-07-20T03:17:06.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7eb4012420 con 0x7f7ec41422c0 2022-07-20T03:17:06.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7eb37fe700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ea0053ad0 0x7f7ea0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7eb406b9b0 con 0x7f7ec41422c0 2022-07-20T03:17:06.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.491+0000 7f7ec259c700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ea0053ad0 0x7f7ea0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.495+0000 7f7ec259c700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ea0053ad0 0x7f7ea0055f90 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f7eb801c000 tx=0x7f7eb8015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:06.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.499+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7eb4042b10 con 0x7f7ec41422c0 2022-07-20T03:17:06.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.623+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7f7ec4067ea0 con 0x7f7ec41422c0 2022-07-20T03:17:06.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7eb37fe700 1 -- 172.21.15.61:0/4188460216 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v22)=0 v22) v1 ==== 122+0+0 (secure 0 0 0) 0x7f7eb400f240 con 0x7f7ec41422c0 2022-07-20T03:17:06.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ea0053ad0 msgr2=0x7f7ea0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ea0053ad0 0x7f7ea0055f90 secure :-1 s=READY pgs=647 cs=0 l=1 rev1=1 rx=0x7f7eb801c000 tx=0x7f7eb8015000).stop 2022-07-20T03:17:06.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 msgr2=0x7f7ec40b8130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 0x7f7ec40b8130 secure :-1 s=READY pgs=1343 cs=0 l=1 rev1=1 rx=0x7f7eb4006940 tx=0x7f7eb4008040).stop 2022-07-20T03:17:06.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 shutdown_connections 2022-07-20T03:17:06.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7ea0053ad0 0x7f7ea0055f90 unknown :-1 s=CLOSED pgs=647 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7ec41462a0 0x7f7ec4155640 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7ec41422c0 0x7f7ec40b8130 unknown :-1 s=CLOSED pgs=1343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 --2- 172.21.15.61:0/4188460216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7ec40988b0 0x7f7ec4152c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 >> 172.21.15.61:0/4188460216 conn(0x7f7ec40b5f00 msgr2=0x7f7ec405cec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:06.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 shutdown_connections 2022-07-20T03:17:06.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.631+0000 7f7ec8edb700 1 -- 172.21.15.61:0/4188460216 wait complete. 2022-07-20T03:17:06.649 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:17:06.651 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:06.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:06.692 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:06.718 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:06.744 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:06.767 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:17:06.767 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:06.768 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:06.769 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:06.777 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:17:06.779 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:06.780 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:17:06.780 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:06.781 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:06.781 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:06.801 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:17:06.802 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:06.803 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:06.804 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:06.805 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:06.806 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:17:06.814 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:17:06.815 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:06.817 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:17:06.818 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:17:06.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.927+0000 7f0c4fe99700 1 Processor -- start 2022-07-20T03:17:06.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.927+0000 7f0c4fe99700 1 -- start start 2022-07-20T03:17:06.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0c480b1e00 0x7f0c4814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c4805d480 con 0x7f0c480acef0 2022-07-20T03:17:06.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c480b21e0 con 0x7f0c4805c3f0 2022-07-20T03:17:06.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c4805cf40 con 0x7f0c480b1e00 2022-07-20T03:17:06.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4e436700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0c480acef0 0x7f0c48146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47454/0 (socket says 172.21.15.61:47454) 2022-07-20T03:17:06.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4e436700 1 -- 172.21.15.61:0/3385451992 learned_addr learned my addr 172.21.15.61:0/3385451992 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:06.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4dc35700 1 --2- 172.21.15.61:0/3385451992 >> v2:172.21.15.117:3300/0 conn(0x7f0c480b1e00 0x7f0c4814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4cc33700 1 -- 172.21.15.61:0/3385451992 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3617437870 0 0) 0x7f0c4805d480 con 0x7f0c480acef0 2022-07-20T03:17:06.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4cc33700 1 -- 172.21.15.61:0/3385451992 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0c30003260 con 0x7f0c480acef0 2022-07-20T03:17:06.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4dc35700 1 -- 172.21.15.61:0/3385451992 >> v1:172.21.15.117:6789/0 conn(0x7f0c4805c3f0 legacy=0x7f0c480ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4dc35700 1 -- 172.21.15.61:0/3385451992 >> v1:172.21.15.61:6789/0 conn(0x7f0c480acef0 legacy=0x7f0c48146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4dc35700 1 -- 172.21.15.61:0/3385451992 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c4805d1e0 con 0x7f0c480b1e00 2022-07-20T03:17:06.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4dc35700 1 --2- 172.21.15.61:0/3385451992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480b1e00 0x7f0c4814a6d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f0c400032f0 tx=0x7f0c400091d0).ready entity=mon.1 client_cookie=922b3447dc40564a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:06.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4cc33700 1 -- 172.21.15.61:0/3385451992 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c4000b6c0 con 0x7f0c480b1e00 2022-07-20T03:17:06.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4cc33700 1 -- 172.21.15.61:0/3385451992 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c4000b820 con 0x7f0c480b1e00 2022-07-20T03:17:06.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4cc33700 1 -- 172.21.15.61:0/3385451992 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c4000b990 con 0x7f0c480b1e00 2022-07-20T03:17:06.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- 172.21.15.61:0/3385451992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480b1e00 msgr2=0x7f0c4814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/3385451992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480b1e00 0x7f0c4814a6d0 secure :-1 s=READY pgs=605 cs=0 l=1 rev1=1 rx=0x7f0c400032f0 tx=0x7f0c400091d0).stop 2022-07-20T03:17:06.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- 172.21.15.61:0/3385451992 shutdown_connections 2022-07-20T03:17:06.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/3385451992 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480b1e00 0x7f0c4814a6d0 unknown :-1 s=CLOSED pgs=605 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- 172.21.15.61:0/3385451992 >> 172.21.15.61:0/3385451992 conn(0x7f0c480611c0 msgr2=0x7f0c48062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:06.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- 172.21.15.61:0/3385451992 shutdown_connections 2022-07-20T03:17:06.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.931+0000 7f0c4fe99700 1 -- 172.21.15.61:0/3385451992 wait complete. 2022-07-20T03:17:06.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 Processor -- start 2022-07-20T03:17:06.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 -- start start 2022-07-20T03:17:06.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 0x7f0c48149670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c480b1e00 0x7f0c480b34c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c48143540 0x7f0c4813d240 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c481488c0 con 0x7f0c48143540 2022-07-20T03:17:06.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48066d40 con 0x7f0c480acef0 2022-07-20T03:17:06.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c48149c10 con 0x7f0c480b1e00 2022-07-20T03:17:06.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4dc35700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 0x7f0c48149670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c48143540 0x7f0c4813d240 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4dc35700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 0x7f0c48149670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:60540/0 (socket says 172.21.15.61:60540) 2022-07-20T03:17:06.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4dc35700 1 -- 172.21.15.61:0/82240600 learned_addr learned my addr 172.21.15.61:0/82240600 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:06.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4d434700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c480b1e00 0x7f0c480b34c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 -- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c480b1e00 msgr2=0x7f0c480b34c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c480b1e00 0x7f0c480b34c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 -- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 msgr2=0x7f0c48149670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:06.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 0x7f0c48149670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:06.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 -- 172.21.15.61:0/82240600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c40009470 con 0x7f0c48143540 2022-07-20T03:17:06.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4dc35700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 0x7f0c48149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:06.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4e436700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c48143540 0x7f0c4813d240 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f0c44001110 tx=0x7f0c440079c0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:06.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c4400cdb0 con 0x7f0c48143540 2022-07-20T03:17:06.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c440070d0 con 0x7f0c48143540 2022-07-20T03:17:06.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c48066fa0 con 0x7f0c48143540 2022-07-20T03:17:06.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c44007430 con 0x7f0c48143540 2022-07-20T03:17:06.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.935+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c4813d730 con 0x7f0c48143540 2022-07-20T03:17:06.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.939+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c48067160 con 0x7f0c48143540 2022-07-20T03:17:06.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.939+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0c44010070 con 0x7f0c48143540 2022-07-20T03:17:06.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.939+0000 7f0c36ffd700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c280539b0 0x7f0c28055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:06.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.939+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0c4406b100 con 0x7f0c48143540 2022-07-20T03:17:06.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.939+0000 7f0c4dc35700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c280539b0 0x7f0c28055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:06.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.943+0000 7f0c4dc35700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c280539b0 0x7f0c28055e70 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f0c4001b000 tx=0x7f0c40016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:06.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:06.947+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c44042340 con 0x7f0c48143540 2022-07-20T03:17:06.963 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:06.966 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:06.968 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:06.971 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:06.974 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:07.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.075+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f0c480574e0 con 0x7f0c48143540 2022-07-20T03:17:07.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c36ffd700 1 -- 172.21.15.61:0/82240600 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v133) v1 ==== 81+0+28885 (secure 0 0 0) 0x7f0c44045960 con 0x7f0c48143540 2022-07-20T03:17:07.083 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:07.084 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:17:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 133, 2022-07-20T03:17:07.084 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:17:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:17:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:17:04.320719+0000", 2022-07-20T03:17:07.085 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:07.086 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:17:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:17:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:17:07.087 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:17:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:17:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:17:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:17:07.088 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:17:07.089 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:17:07.090 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:17:07.090 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:17:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:17:07.091 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 26, 2022-07-20T03:17:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:17:07.092 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:17:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:17:07.093 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:17:07.094 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:17:07.094 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.095 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:17:07.095 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:17:07.096 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:17:07.096 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:07.097 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:07.098 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:07.099 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:07.100 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:17:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:17:07.101 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:17:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:17:07.102 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:17:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:07.103 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:07.104 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:07.105 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:07.106 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:07.109 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.110 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:17:07.111 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:07.112 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:07.113 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:07.114 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:07.115 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:07.116 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:07.117 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:07.118 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:07.119 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:07.120 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:07.121 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:07.122 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:07.123 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:07.124 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:07.125 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.125 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:07.126 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:07.127 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:17:07.128 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:17:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.129 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:07.139 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:17:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.140 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:17:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:17:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:17:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:07.141 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:07.142 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:07.142 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:07.142 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:07.142 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:07.143 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:07.144 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:07.144 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:07.144 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:07.144 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:07.145 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:17:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:07.146 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:07.147 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:07.148 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:07.149 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:07.150 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:07.151 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:07.152 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:07.153 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:07.154 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:07.154 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:07.154 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:07.154 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.155 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:07.155 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:07.155 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:07.156 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:07.157 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:17:07.158 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:17:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.160 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.161 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:07.161 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:17:07.162 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.162 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:07.162 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:17:07.163 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.163 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.164 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.164 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.166 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.166 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 127, 2022-07-20T03:17:07.167 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.167 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.168 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.169 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:17:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.171 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.172 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:17:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.173 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.174 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.175 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.176 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:17:07.177 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.178 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.179 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:17:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.180 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.181 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.182 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.183 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:17:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.184 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:17:07.186 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.187 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.188 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.189 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.190 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.190 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:17:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.191 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.192 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:17:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:07.193 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.194 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:17:07.195 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:17:07.196 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:17:07.196 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:17:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.197 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.198 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.198 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.199 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.199 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:07.200 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:17:07.201 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.201 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.202 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.202 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.203 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.204 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.204 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:07.205 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.205 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.206 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.206 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.207 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.207 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:17:07.208 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.209 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.210 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:17:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.211 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.212 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.213 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.213 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.214 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.214 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:17:07.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.216 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.217 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:17:07.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.218 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.218 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.219 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.220 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.220 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.221 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.221 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:17:07.222 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.222 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.223 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.224 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:17:07.224 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.225 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.225 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.226 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.226 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.227 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.228 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:17:07.228 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.229 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.230 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.230 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:17:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:07.231 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.232 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:17:07.233 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:17:07.234 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:17:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:17:07.235 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.236 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.238 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:17:07.239 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.240 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.241 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.242 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.243 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:07.243 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.244 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.244 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.245 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:17:07.246 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.247 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.248 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:17:07.249 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.249 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.250 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.251 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.252 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:17:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.254 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:17:07.255 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.256 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.257 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.258 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:17:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.259 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.260 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:17:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.261 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.262 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.263 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:17:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.264 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:17:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:07.265 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.266 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:17:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:17:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:17:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:17:07.267 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.268 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:07.269 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:17:07.270 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.270 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.271 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 127, 2022-07-20T03:17:07.272 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.273 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:17:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:17:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.275 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.276 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:17:07.277 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:17:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.281 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.282 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.282 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.282 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:17:07.283 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:17:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.284 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.285 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.286 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:17:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.287 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.288 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.288 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:17:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:07.289 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.290 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.290 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.290 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:17:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:17:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:17:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:17:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.291 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.292 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.292 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.292 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.293 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:07.293 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:17:07.293 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.294 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.294 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.294 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.296 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.296 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:07.296 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.297 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.297 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.297 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.298 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.298 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:17:07.298 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.298 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.299 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.299 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.300 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:17:07.300 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.300 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.301 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.301 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.302 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.303 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:17:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.304 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:17:07.305 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.306 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.308 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:17:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.309 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:17:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.310 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.311 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:17:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.312 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:17:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:07.313 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:17:07.314 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:17:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:17:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:17:07.315 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.316 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:17:07.317 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.318 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 120, 2022-07-20T03:17:07.319 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.320 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:17:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.321 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.322 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:17:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.323 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.324 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.324 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.324 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.324 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:17:07.325 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.325 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.326 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.326 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:17:07.326 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.326 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.327 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.327 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.327 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.327 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.328 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.328 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:17:07.328 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.329 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:17:07.330 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.330 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.330 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.331 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.331 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.331 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.331 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.332 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:17:07.332 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.332 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.333 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:17:07.333 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:07.333 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.334 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.334 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.334 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:17:07.334 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:17:07.335 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:17:07.335 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:17:07.335 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.335 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.336 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.336 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.336 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.336 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.337 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:07.337 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:17:07.337 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.337 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.338 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.338 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.338 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.338 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.339 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.339 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:07.339 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.339 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.340 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.340 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.341 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.341 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:17:07.342 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.342 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.343 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.344 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:17:07.344 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.345 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.345 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.345 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.346 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.346 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.347 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.347 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.348 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:17:07.348 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.349 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.350 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:17:07.350 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.351 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.351 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.352 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.352 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.353 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.353 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.354 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:17:07.354 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.355 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.356 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:17:07.357 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.357 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.358 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.358 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.359 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.360 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.360 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:17:07.361 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.362 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.362 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:17:07.363 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:07.363 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.364 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.364 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.365 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:17:07.365 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:17:07.365 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:17:07.366 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:17:07.366 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.367 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.367 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.368 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.368 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.369 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:07.369 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:17:07.370 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:07.370 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:07.371 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:07.371 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:07.372 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:07.372 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:07.372 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:07.373 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 128, 2022-07-20T03:17:07.373 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:07.374 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:07.374 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:07.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.375 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.376 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.376 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:17:07.376 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.377 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.378 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.378 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:17:07.379 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.379 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.380 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.380 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.380 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:07.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.381 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.382 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.382 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:17:07.383 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.383 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.383 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.384 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.384 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:17:07.385 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.385 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.386 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.386 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.386 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:07.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.387 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.388 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:17:07.389 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.390 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.390 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:17:07.390 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.390 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.391 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.391 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.391 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:07.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:07.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.392 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:07.392 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:17:07.392 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.393 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.393 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:07.393 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:17:07.393 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:07.394 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.394 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.394 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:17:07.395 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:17:07.395 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:17:07.395 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:17:07.395 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:07.396 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:07.396 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:07.396 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:07.396 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.396 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:07.397 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:17:07.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.397 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:07.397 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.398 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.398 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.398 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.398 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.399 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:17:07.399 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.399 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.399 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:07.400 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.400 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.400 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.400 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.401 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.401 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:17:07.401 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.401 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.402 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:07.402 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.402 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.402 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.403 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.403 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.403 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:17:07.403 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.404 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.404 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.404 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:07.404 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.405 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.405 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.405 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.405 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.405 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:17:07.406 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.406 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.406 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.406 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:07.407 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.407 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.407 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.407 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.408 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.408 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:17:07.408 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.408 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.408 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.409 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:07.409 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.409 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.409 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.410 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.410 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.410 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:17:07.410 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.411 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.411 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.411 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:07.411 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.411 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.412 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.412 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.412 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.412 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:17:07.413 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:07.413 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:07.413 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:07.414 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:07.414 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:07.414 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:07.414 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:07.415 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:17:07.415 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:07.415 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.415 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:07.416 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:17:07.416 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:17:07.416 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:17:07.416 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:17:07.416 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:17:07.417 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/1435464119": "2022-07-21T03:17:00.683791+0000", 2022-07-20T03:17:07.417 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/1435464119": "2022-07-21T03:17:00.683791+0000" 2022-07-20T03:17:07.417 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.417 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:17:07.418 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:17:07.418 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:17:07.418 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:17:07.418 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:17:07.419 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:17:07.419 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:17:07.419 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:17:07.419 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:07.419 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:07.420 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:17:07.420 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:17:07.420 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:17:07.420 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:17:07.421 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:17:07.421 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:07.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c280539b0 msgr2=0x7f0c28055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c280539b0 0x7f0c28055e70 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f0c4001b000 tx=0x7f0c40016000).stop 2022-07-20T03:17:07.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c48143540 msgr2=0x7f0c4813d240 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c48143540 0x7f0c4813d240 secure :-1 s=READY pgs=1345 cs=0 l=1 rev1=1 rx=0x7f0c44001110 tx=0x7f0c440079c0).stop 2022-07-20T03:17:07.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 shutdown_connections 2022-07-20T03:17:07.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c280539b0 0x7f0c28055e70 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c48143540 0x7f0c4813d240 unknown :-1 s=CLOSED pgs=1345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c480b1e00 0x7f0c480b34c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 --2- 172.21.15.61:0/82240600 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c480acef0 0x7f0c48149670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 >> 172.21.15.61:0/82240600 conn(0x7f0c4805c3f0 msgr2=0x7f0c48069d80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:07.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 shutdown_connections 2022-07-20T03:17:07.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.079+0000 7f0c4fe99700 1 -- 172.21.15.61:0/82240600 wait complete. 2022-07-20T03:17:07.427 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:07.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 Processor -- start 2022-07-20T03:17:07.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 -- start start 2022-07-20T03:17:07.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a00b1ef0 0x7f77a0062040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77a0098ae0 0x7f77a00ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f77a00b22d0 con 0x7f77a0061050 2022-07-20T03:17:07.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f77a00abe50 con 0x7f77a0098ae0 2022-07-20T03:17:07.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a65e5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a00ac0f0 con 0x7f77a00b1ef0 2022-07-20T03:17:07.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a4de2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77a0098ae0 0x7f77a00ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a4de2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f77a0098ae0 0x7f77a00ac830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50868/0 (socket says 172.21.15.61:50868) 2022-07-20T03:17:07.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a4de2700 1 -- 172.21.15.61:0/1591512218 learned_addr learned my addr 172.21.15.61:0/1591512218 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:07.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.535+0000 7f77a55e3700 1 --2- 172.21.15.61:0/1591512218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a00b1ef0 0x7f77a0062040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a4de2700 1 -- 172.21.15.61:0/1591512218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a00b1ef0 msgr2=0x7f77a0062040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:07.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a4de2700 1 --2- 172.21.15.61:0/1591512218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a00b1ef0 0x7f77a0062040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a4de2700 1 -- 172.21.15.61:0/1591512218 >> v1:172.21.15.117:6789/0 conn(0x7f77a0061050 legacy=0x7f77a00614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a4de2700 1 -- 172.21.15.61:0/1591512218 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f77a00ac390 con 0x7f77a0098ae0 2022-07-20T03:17:07.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f7797fff700 1 -- 172.21.15.61:0/1591512218 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3615650154 0 0) 0x7f77a00b22d0 con 0x7f77a0061050 2022-07-20T03:17:07.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a4de2700 1 --2- 172.21.15.61:0/1591512218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a0098ae0 0x7f77a00ac830 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f7790002f70 tx=0x7f7790006eb0).ready entity=mon.0 client_cookie=509837b6308395e0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f7797fff700 1 -- 172.21.15.61:0/1591512218 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779000d570 con 0x7f77a0098ae0 2022-07-20T03:17:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f7797fff700 1 -- 172.21.15.61:0/1591512218 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7790000e90 con 0x7f77a0098ae0 2022-07-20T03:17:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f7797fff700 1 -- 172.21.15.61:0/1591512218 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779000d8c0 con 0x7f77a0098ae0 2022-07-20T03:17:07.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 -- 172.21.15.61:0/1591512218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a0098ae0 msgr2=0x7f77a00ac830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 --2- 172.21.15.61:0/1591512218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a0098ae0 0x7f77a00ac830 secure :-1 s=READY pgs=1346 cs=0 l=1 rev1=1 rx=0x7f7790002f70 tx=0x7f7790006eb0).stop 2022-07-20T03:17:07.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 -- 172.21.15.61:0/1591512218 shutdown_connections 2022-07-20T03:17:07.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 --2- 172.21.15.61:0/1591512218 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a0098ae0 0x7f77a00ac830 unknown :-1 s=CLOSED pgs=1346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 --2- 172.21.15.61:0/1591512218 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a00b1ef0 0x7f77a0062040 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 -- 172.21.15.61:0/1591512218 >> 172.21.15.61:0/1591512218 conn(0x7f77a00b2ca0 msgr2=0x7f77a00b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:07.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 -- 172.21.15.61:0/1591512218 shutdown_connections 2022-07-20T03:17:07.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 -- 172.21.15.61:0/1591512218 wait complete. 2022-07-20T03:17:07.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 Processor -- start 2022-07-20T03:17:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.539+0000 7f77a65e5700 1 -- start start 2022-07-20T03:17:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a0061050 0x7f77a00b5050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a0098ae0 0x7f77a01499a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 0x7f77a00af4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a0156640 con 0x7f77a00b1ef0 2022-07-20T03:17:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f77a00ad750 con 0x7f77a0098ae0 2022-07-20T03:17:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f77a00ad9f0 con 0x7f77a0061050 2022-07-20T03:17:07.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 0x7f77a00af4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 0x7f77a00af4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50878/0 (socket says 172.21.15.61:50878) 2022-07-20T03:17:07.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a55e3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a0061050 0x7f77a00b5050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 -- 172.21.15.61:0/3657527029 learned_addr learned my addr 172.21.15.61:0/3657527029 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:07.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a4de2700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a0098ae0 0x7f77a01499a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 -- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a0061050 msgr2=0x7f77a00b5050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a0061050 0x7f77a00b5050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 -- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a0098ae0 msgr2=0x7f77a01499a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a0098ae0 0x7f77a01499a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 -- 172.21.15.61:0/3657527029 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f779000b1e0 con 0x7f77a00b1ef0 2022-07-20T03:17:07.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a5de4700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 0x7f77a00af4b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f7798001250 tx=0x7f7798007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:07.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779800f470 con 0x7f77a00b1ef0 2022-07-20T03:17:07.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f779800f5d0 con 0x7f77a00b1ef0 2022-07-20T03:17:07.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f77a00adc90 con 0x7f77a00b1ef0 2022-07-20T03:17:07.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f779800f930 con 0x7f77a00b1ef0 2022-07-20T03:17:07.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.543+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77a0149e90 con 0x7f77a00b1ef0 2022-07-20T03:17:07.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.547+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f77a00ade20 con 0x7f77a00b1ef0 2022-07-20T03:17:07.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.547+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7798019030 con 0x7f77a00b1ef0 2022-07-20T03:17:07.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.547+0000 7f77967fc700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7788053920 0x7f7788055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.547+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f779806b0e0 con 0x7f77a00b1ef0 2022-07-20T03:17:07.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.547+0000 7f77a55e3700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7788053920 0x7f7788055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.547+0000 7f77a55e3700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7788053920 0x7f7788055de0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f779c006000 tx=0x7f779c006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:07.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.551+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f77980423d0 con 0x7f77a00b1ef0 2022-07-20T03:17:07.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.711+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f77a00574e0 con 0x7f77a00b1ef0 2022-07-20T03:17:07.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77967fc700 1 -- 172.21.15.61:0/3657527029 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1209 (secure 0 0 0) 0x7f7798017070 con 0x7f77a00b1ef0 2022-07-20T03:17:07.720 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:07.720 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70},{"gid":7121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1206909947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1206909947},{"type":"v1","addr":"172.21.15.61:6833","nonce":1206909947}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":74}],"filesystems":[]} 2022-07-20T03:17:07.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7788053920 msgr2=0x7f7788055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7788053920 0x7f7788055de0 secure :-1 s=READY pgs=649 cs=0 l=1 rev1=1 rx=0x7f779c006000 tx=0x7f779c006d90).stop 2022-07-20T03:17:07.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 msgr2=0x7f77a00af4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 0x7f77a00af4b0 secure :-1 s=READY pgs=1347 cs=0 l=1 rev1=1 rx=0x7f7798001250 tx=0x7f7798007dc0).stop 2022-07-20T03:17:07.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 shutdown_connections 2022-07-20T03:17:07.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7788053920 0x7f7788055de0 unknown :-1 s=CLOSED pgs=649 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f77a0098ae0 0x7f77a01499a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f77a00b1ef0 0x7f77a00af4b0 unknown :-1 s=CLOSED pgs=1347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 --2- 172.21.15.61:0/3657527029 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f77a0061050 0x7f77a00b5050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 >> 172.21.15.61:0/3657527029 conn(0x7f77a00b2ca0 msgr2=0x7f77a01433c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:07.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 shutdown_connections 2022-07-20T03:17:07.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.715+0000 7f77a65e5700 1 -- 172.21.15.61:0/3657527029 wait complete. 2022-07-20T03:17:07.726 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 74 2022-07-20T03:17:07.732 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:07.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 Processor -- start 2022-07-20T03:17:07.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 -- start start 2022-07-20T03:17:07.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0b5650 0x7fa94c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa94c0af8e0 0x7fa94c05bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa94c147010 con 0x7fa94c0adff0 2022-07-20T03:17:07.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa94c1472b0 con 0x7fa94c0af8e0 2022-07-20T03:17:07.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c146d40 con 0x7fa94c0b5650 2022-07-20T03:17:07.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa94c0af8e0 0x7fa94c05bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa94c0af8e0 0x7fa94c05bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50884/0 (socket says 172.21.15.61:50884) 2022-07-20T03:17:07.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 -- 172.21.15.61:0/388149993 learned_addr learned my addr 172.21.15.61:0/388149993 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:07.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa949d9b700 1 --1- 172.21.15.61:0/388149993 >> v1:172.21.15.61:6789/0 conn(0x7fa94c0adff0 0x7fa94c147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47482/0 (socket says 172.21.15.61:47482) 2022-07-20T03:17:07.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94a59c700 1 --2- 172.21.15.61:0/388149993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0b5650 0x7fa94c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 -- 172.21.15.61:0/388149993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0b5650 msgr2=0x7fa94c0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:07.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 --2- 172.21.15.61:0/388149993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0b5650 0x7fa94c0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 -- 172.21.15.61:0/388149993 >> v1:172.21.15.61:6789/0 conn(0x7fa94c0adff0 legacy=0x7fa94c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 -- 172.21.15.61:0/388149993 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa94c0acae0 con 0x7fa94c0af8e0 2022-07-20T03:17:07.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94ad9d700 1 --2- 172.21.15.61:0/388149993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0af8e0 0x7fa94c05bd90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7fa93c002f70 tx=0x7fa93c006eb0).ready entity=mon.0 client_cookie=6b0596914bab6eef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:07.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94959a700 1 -- 172.21.15.61:0/388149993 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa93c00d5c0 con 0x7fa94c0af8e0 2022-07-20T03:17:07.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94959a700 1 -- 172.21.15.61:0/388149993 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c000e90 con 0x7fa94c0af8e0 2022-07-20T03:17:07.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.831+0000 7fa94959a700 1 -- 172.21.15.61:0/388149993 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa93c00d950 con 0x7fa94c0af8e0 2022-07-20T03:17:07.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- 172.21.15.61:0/388149993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0af8e0 msgr2=0x7fa94c05bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 --2- 172.21.15.61:0/388149993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0af8e0 0x7fa94c05bd90 secure :-1 s=READY pgs=1349 cs=0 l=1 rev1=1 rx=0x7fa93c002f70 tx=0x7fa93c006eb0).stop 2022-07-20T03:17:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- 172.21.15.61:0/388149993 shutdown_connections 2022-07-20T03:17:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 --2- 172.21.15.61:0/388149993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0af8e0 0x7fa94c05bd90 unknown :-1 s=CLOSED pgs=1349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 --2- 172.21.15.61:0/388149993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0b5650 0x7fa94c0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- 172.21.15.61:0/388149993 >> 172.21.15.61:0/388149993 conn(0x7fa94c0634e0 msgr2=0x7fa94c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:07.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- 172.21.15.61:0/388149993 shutdown_connections 2022-07-20T03:17:07.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- 172.21.15.61:0/388149993 wait complete. 2022-07-20T03:17:07.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 Processor -- start 2022-07-20T03:17:07.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- start start 2022-07-20T03:17:07.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa94c0adff0 0x7fa94c064d40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0af8e0 0x7fa94c065230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 0x7fa94c156670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c063990 con 0x7fa94c0b5650 2022-07-20T03:17:07.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c063bc0 con 0x7fa94c0adff0 2022-07-20T03:17:07.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa94c067420 con 0x7fa94c0af8e0 2022-07-20T03:17:07.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 0x7fa94c156670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94ad9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 0x7fa94c156670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50892/0 (socket says 172.21.15.61:50892) 2022-07-20T03:17:07.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94ad9d700 1 -- 172.21.15.61:0/2071702545 learned_addr learned my addr 172.21.15.61:0/2071702545 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:07.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa94a59c700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa94c0adff0 0x7fa94c064d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.835+0000 7fa949d9b700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0af8e0 0x7fa94c065230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94ad9d700 1 -- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0af8e0 msgr2=0x7fa94c065230 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94ad9d700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0af8e0 0x7fa94c065230 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94ad9d700 1 -- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa94c0adff0 msgr2=0x7fa94c064d40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:07.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94ad9d700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa94c0adff0 0x7fa94c064d40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:07.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94ad9d700 1 -- 172.21.15.61:0/2071702545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa93c00b1e0 con 0x7fa94c0b5650 2022-07-20T03:17:07.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94ad9d700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 0x7fa94c156670 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7fa93c002f70 tx=0x7fa93c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:07.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa93c006b40 con 0x7fa94c0b5650 2022-07-20T03:17:07.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa93c00d940 con 0x7fa94c0b5650 2022-07-20T03:17:07.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa94c0676c0 con 0x7fa94c0b5650 2022-07-20T03:17:07.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa93c011cf0 con 0x7fa94c0b5650 2022-07-20T03:17:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa94c111f10 con 0x7fa94c0b5650 2022-07-20T03:17:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa94c0574e0 con 0x7fa94c0b5650 2022-07-20T03:17:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa93c025030 con 0x7fa94c0b5650 2022-07-20T03:17:07.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa9437fe700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa930053a40 0x7fa930055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:07.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(133..133 src has 1..133) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fa93c06cda0 con 0x7fa94c0b5650 2022-07-20T03:17:07.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.839+0000 7fa94a59c700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa930053a40 0x7fa930055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:07.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.843+0000 7fa94a59c700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa930053a40 0x7fa930055f00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fa94400c000 tx=0x7fa944008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:07.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:07.847+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa93c043f40 con 0x7fa94c0b5650 2022-07-20T03:17:08.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.007+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa94c067ea0 con 0x7fa94c0b5650 2022-07-20T03:17:08.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.007+0000 7fa9437fe700 1 -- 172.21.15.61:0/2071702545 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 74 v74) v1 ==== 94+0+1209 (secure 0 0 0) 0x7fa93c023020 con 0x7fa94c0b5650 2022-07-20T03:17:08.013 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:08.014 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":74,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":6905,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/651854075","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":651854075},{"type":"v1","addr":"172.21.15.117:6835","nonce":651854075}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":70},{"gid":7121,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1206909947","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1206909947},{"type":"v1","addr":"172.21.15.61:6833","nonce":1206909947}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":74}],"filesystems":[]} 2022-07-20T03:17:08.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa930053a40 msgr2=0x7fa930055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:08.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa930053a40 0x7fa930055f00 secure :-1 s=READY pgs=650 cs=0 l=1 rev1=1 rx=0x7fa94400c000 tx=0x7fa944008040).stop 2022-07-20T03:17:08.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 msgr2=0x7fa94c156670 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:08.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 0x7fa94c156670 secure :-1 s=READY pgs=1350 cs=0 l=1 rev1=1 rx=0x7fa93c002f70 tx=0x7fa93c015000).stop 2022-07-20T03:17:08.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 shutdown_connections 2022-07-20T03:17:08.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa930053a40 0x7fa930055f00 unknown :-1 s=CLOSED pgs=650 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa94c0b5650 0x7fa94c156670 unknown :-1 s=CLOSED pgs=1350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa94c0af8e0 0x7fa94c065230 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 --2- 172.21.15.61:0/2071702545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa94c0adff0 0x7fa94c064d40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 >> 172.21.15.61:0/2071702545 conn(0x7fa94c0634e0 msgr2=0x7fa94c14a4f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:08.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 shutdown_connections 2022-07-20T03:17:08.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.011+0000 7fa94b59e700 1 -- 172.21.15.61:0/2071702545 wait complete. 2022-07-20T03:17:08.019 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 74 2022-07-20T03:17:08.026 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:17:08.027 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:17:08.027 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:17:08.027 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:17:08.028 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:17:08.028 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:17:08.028 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:17:08.029 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:17:08.042 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:17:08.042 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:17:08.044 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:17:08.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:17:08.087 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:17:08.088 DEBUG:teuthology.parallel:result is None 2022-07-20T03:17:08.090 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:17:08.091 DEBUG:teuthology.parallel:result is None 2022-07-20T03:17:08.091 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:17:08.163 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:17:08.165 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:17:08.198 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 Processor -- start 2022-07-20T03:17:08.199 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 -- start start 2022-07-20T03:17:08.201 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6c740b49f0 0x7f6c740b4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:08.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74146260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:08.202 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c740b17f0 con 0x7f6c740b6200 2022-07-20T03:17:08.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c740b4640 con 0x7f6c740b49f0 2022-07-20T03:17:08.203 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c79bb1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740b5190 con 0x7f6c740b08e0 2022-07-20T03:17:08.204 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c73fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6c740b49f0 0x7f6c740b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:08.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c78baf700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f6c740b6200 0x7f6c740b01d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:56010/0 (socket says 172.21.15.61:56010) 2022-07-20T03:17:08.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c793b0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74146260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:08.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c78baf700 1 -- 172.21.15.61:0/264601299 learned_addr learned my addr 172.21.15.61:0/264601299 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:08.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c793b0700 1 -- 172.21.15.61:0/264601299 >> v2:172.21.15.117:3300/0 conn(0x7f6c740b49f0 msgr2=0x7f6c740b4080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:08.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c793b0700 1 --2- 172.21.15.61:0/264601299 >> v2:172.21.15.117:3300/0 conn(0x7f6c740b49f0 0x7f6c740b4080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c793b0700 1 -- 172.21.15.61:0/264601299 >> v1:172.21.15.117:6789/0 conn(0x7f6c740b6200 legacy=0x7f6c740b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:08.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c793b0700 1 -- 172.21.15.61:0/264601299 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c740b65e0 con 0x7f6c740b08e0 2022-07-20T03:17:08.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.195+0000 7f6c793b0700 1 --2- 172.21.15.61:0/264601299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74146260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6c6c0032f0 tx=0x7f6c6c0091d0).ready entity=mon.2 client_cookie=36cf2b94db309397 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:08.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c737fe700 1 -- 172.21.15.61:0/264601299 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6c00b6c0 con 0x7f6c740b08e0 2022-07-20T03:17:08.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c737fe700 1 -- 172.21.15.61:0/264601299 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c00b820 con 0x7f6c740b08e0 2022-07-20T03:17:08.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c737fe700 1 -- 172.21.15.61:0/264601299 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6c00b990 con 0x7f6c740b08e0 2022-07-20T03:17:08.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- 172.21.15.61:0/264601299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 msgr2=0x7f6c74146260 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:08.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/264601299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74146260 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f6c6c0032f0 tx=0x7f6c6c0091d0).stop 2022-07-20T03:17:08.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- 172.21.15.61:0/264601299 shutdown_connections 2022-07-20T03:17:08.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/264601299 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74146260 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/264601299 >> v2:172.21.15.117:3300/0 conn(0x7f6c740b49f0 0x7f6c740b4080 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- 172.21.15.61:0/264601299 >> 172.21.15.61:0/264601299 conn(0x7f6c74063d10 msgr2=0x7f6c740b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:08.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- 172.21.15.61:0/264601299 shutdown_connections 2022-07-20T03:17:08.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- 172.21.15.61:0/264601299 wait complete. 2022-07-20T03:17:08.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 Processor -- start 2022-07-20T03:17:08.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- start start 2022-07-20T03:17:08.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74064510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:08.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c740b49f0 0x7f6c740adb00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:08.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 0x7f6c740adff0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:08.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740ad3a0 con 0x7f6c740b6200 2022-07-20T03:17:08.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740acd40 con 0x7f6c740b49f0 2022-07-20T03:17:08.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.199+0000 7f6c79bb1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c740afab0 con 0x7f6c740b08e0 2022-07-20T03:17:08.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 0x7f6c740adff0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:08.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c73fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c740b49f0 0x7f6c740adb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:08.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c78baf700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74064510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:08.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 0x7f6c740adff0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50918/0 (socket says 172.21.15.61:50918) 2022-07-20T03:17:08.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 -- 172.21.15.61:0/1938500987 learned_addr learned my addr 172.21.15.61:0/1938500987 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:08.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 -- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 msgr2=0x7f6c74064510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:08.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74064510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 -- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c740b49f0 msgr2=0x7f6c740adb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:08.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c740b49f0 0x7f6c740adb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:08.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 -- 172.21.15.61:0/1938500987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c6c009470 con 0x7f6c740b6200 2022-07-20T03:17:08.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c78baf700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:08.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c793b0700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 0x7f6c740adff0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f6c6c0032f0 tx=0x7f6c6c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:08.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6c011e20 con 0x7f6c740b6200 2022-07-20T03:17:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c6c021440 con 0x7f6c740b6200 2022-07-20T03:17:08.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c740ae4e0 con 0x7f6c740b6200 2022-07-20T03:17:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c6c0217a0 con 0x7f6c740b6200 2022-07-20T03:17:08.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c74147c00 con 0x7f6c740b6200 2022-07-20T03:17:08.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c740ad510 con 0x7f6c740b6200 2022-07-20T03:17:08.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.203+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6c6c027020 con 0x7f6c740b6200 2022-07-20T03:17:08.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.207+0000 7f6c71ffb700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c54053a40 0x7f6c54055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:08.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.207+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(135..135 src has 1..135) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f6c6c06d080 con 0x7f6c740b6200 2022-07-20T03:17:08.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.207+0000 7f6c78baf700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c54053a40 0x7f6c54055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:08.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.207+0000 7f6c78baf700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c54053a40 0x7f6c54055f00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f6c6800d000 tx=0x7f6c68007be0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:08.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.211+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f6c740ad510 con 0x7f6c740b6200 2022-07-20T03:17:08.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a is failed for ~0s 2022-07-20T03:17:08.246 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b is failed for ~0s 2022-07-20T03:17:08.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:08.347+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f6c740ad720 con 0x7f6c740b6200 2022-07-20T03:17:09.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.203+0000 7f6c71ffb700 1 -- 172.21.15.61:0/1938500987 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v136) v1 ==== 97+0+0 (secure 0 0 0) 0x7f6c6c015020 con 0x7f6c740b6200 2022-07-20T03:17:09.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c54053a40 msgr2=0x7f6c54055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c54053a40 0x7f6c54055f00 secure :-1 s=READY pgs=651 cs=0 l=1 rev1=1 rx=0x7f6c6800d000 tx=0x7f6c68007be0).stop 2022-07-20T03:17:09.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 msgr2=0x7f6c740adff0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 0x7f6c740adff0 secure :-1 s=READY pgs=1354 cs=0 l=1 rev1=1 rx=0x7f6c6c0032f0 tx=0x7f6c6c016000).stop 2022-07-20T03:17:09.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 shutdown_connections 2022-07-20T03:17:09.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6c54053a40 0x7f6c54055f00 unknown :-1 s=CLOSED pgs=651 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c740b49f0 0x7f6c740adb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c740b6200 0x7f6c740adff0 unknown :-1 s=CLOSED pgs=1354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 --2- 172.21.15.61:0/1938500987 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c740b08e0 0x7f6c74064510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 >> 172.21.15.61:0/1938500987 conn(0x7f6c74063d10 msgr2=0x7f6c740b1f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:09.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 shutdown_connections 2022-07-20T03:17:09.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.207+0000 7f6c79bb1700 1 -- 172.21.15.61:0/1938500987 wait complete. 2022-07-20T03:17:09.218 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:17:09.228 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:17:09.229 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:17:09.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f119359e700 1 Processor -- start 2022-07-20T03:17:09.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f119359e700 1 -- start start 2022-07-20T03:17:09.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f119359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f11940af8e0 0x7f11940b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f119359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1194147010 con 0x7f11940adff0 2022-07-20T03:17:09.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f119359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f11941472b0 con 0x7f11940b5650 2022-07-20T03:17:09.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f119359e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1194146d40 con 0x7f11940af8e0 2022-07-20T03:17:09.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f1189d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f11940adff0 0x7f119405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47522/0 (socket says 172.21.15.61:47522) 2022-07-20T03:17:09.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f1189d9b700 1 -- 172.21.15.61:0/2483868360 learned_addr learned my addr 172.21.15.61:0/2483868360 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:09.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f1192d9d700 1 --2- 172.21.15.61:0/2483868360 >> v2:172.21.15.117:3300/0 conn(0x7f11940af8e0 0x7f11940b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941750343 0 0) 0x7f1194147010 con 0x7f11940adff0 2022-07-20T03:17:09.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.331+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1174003260 con 0x7f11940adff0 2022-07-20T03:17:09.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 991441476 0 0) 0x7f1174003260 con 0x7f11940adff0 2022-07-20T03:17:09.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 >> v2:172.21.15.117:3300/0 conn(0x7f11940af8e0 msgr2=0x7f11940b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:09.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 --2- 172.21.15.61:0/2483868360 >> v2:172.21.15.117:3300/0 conn(0x7f11940af8e0 0x7f11940b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 >> v1:172.21.15.117:6789/0 conn(0x7f11940b5650 legacy=0x7f1194147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119405c4a0 con 0x7f11940adff0 2022-07-20T03:17:09.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f117c0017e0 con 0x7f11940adff0 2022-07-20T03:17:09.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 >> v1:172.21.15.61:6789/0 conn(0x7f11940adff0 legacy=0x7f119405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1174003750 0x7f1174003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1174004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1174004e70 0x7f117400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194146d40 con 0x7f1174004500 2022-07-20T03:17:09.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f117400bdd0 con 0x7f1174003750 2022-07-20T03:17:09.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f117400bf60 con 0x7f1174004e70 2022-07-20T03:17:09.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1174004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1192d9d700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1174004e70 0x7f117400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119259c700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1174003750 0x7f1174003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 -- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1174004e70 msgr2=0x7f117400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:09.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1174004e70 0x7f117400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 -- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1174003750 msgr2=0x7f1174003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:09.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1174003750 0x7f1174003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 -- 172.21.15.61:0/2483868360 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1174003260 con 0x7f1174004500 2022-07-20T03:17:09.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1192d9d700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1174004e70 0x7f117400b750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:09.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1192d9d700 1 -- 172.21.15.61:0/2483868360 reap_dead start 2022-07-20T03:17:09.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1189d9b700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1174004980 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f117c0014f0 tx=0x7f117c007590).ready entity=mon.0 client_cookie=bda7ff2c4a265818 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f117c0070d0 con 0x7f1174004500 2022-07-20T03:17:09.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f117c007230 con 0x7f1174004500 2022-07-20T03:17:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f1191d9b700 1 -- 172.21.15.61:0/2483868360 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f117c0073a0 con 0x7f1174004500 2022-07-20T03:17:09.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 -- 172.21.15.61:0/2483868360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 msgr2=0x7f1174004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1174004980 secure :-1 s=READY pgs=1356 cs=0 l=1 rev1=1 rx=0x7f117c0014f0 tx=0x7f117c007590).stop 2022-07-20T03:17:09.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 -- 172.21.15.61:0/2483868360 shutdown_connections 2022-07-20T03:17:09.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 --2- 172.21.15.61:0/2483868360 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1174004980 unknown :-1 s=CLOSED pgs=1356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 -- 172.21.15.61:0/2483868360 >> 172.21.15.61:0/2483868360 conn(0x7f11940634e0 msgr2=0x7f11940b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:09.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 -- 172.21.15.61:0/2483868360 shutdown_connections 2022-07-20T03:17:09.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.335+0000 7f119359e700 1 -- 172.21.15.61:0/2483868360 wait complete. 2022-07-20T03:17:09.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 Processor -- start 2022-07-20T03:17:09.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 -- start start 2022-07-20T03:17:09.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1194159af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11940642a0 0x7f1194063990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1194110240 0x7f1194144100 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194159fe0 con 0x7f1174004500 2022-07-20T03:17:09.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1194111e30 con 0x7f1194110240 2022-07-20T03:17:09.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f119415a220 con 0x7f11940642a0 2022-07-20T03:17:09.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1194159af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f1192d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1194110240 0x7f1194144100 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f1189d9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11940642a0 0x7f1194063990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1194159af0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50936/0 (socket says 172.21.15.61:50936) 2022-07-20T03:17:09.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 -- 172.21.15.61:0/1963265743 learned_addr learned my addr 172.21.15.61:0/1963265743 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:09.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 -- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11940642a0 msgr2=0x7f1194063990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11940642a0 0x7f1194063990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 -- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1194110240 msgr2=0x7f1194144100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1194110240 0x7f1194144100 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 -- 172.21.15.61:0/1963265743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f119415a4c0 con 0x7f1174004500 2022-07-20T03:17:09.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119259c700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1194159af0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f118400b000 tx=0x7f1184009040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1184014620 con 0x7f1174004500 2022-07-20T03:17:09.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1184014780 con 0x7f1174004500 2022-07-20T03:17:09.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1194063e80 con 0x7f1174004500 2022-07-20T03:17:09.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1184014ae0 con 0x7f1174004500 2022-07-20T03:17:09.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.339+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1194065e20 con 0x7f1174004500 2022-07-20T03:17:09.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.343+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1194064070 con 0x7f1174004500 2022-07-20T03:17:09.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.343+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1184014c40 con 0x7f1174004500 2022-07-20T03:17:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.343+0000 7f118bfff700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1178053a40 0x7f1178055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.343+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f118406bae0 con 0x7f1174004500 2022-07-20T03:17:09.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.343+0000 7f1189d9b700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1178053a40 0x7f1178055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.343+0000 7f1189d9b700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1178053a40 0x7f1178055f00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f117c019000 tx=0x7f117c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.347+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1184042c00 con 0x7f1174004500 2022-07-20T03:17:09.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.511+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f11940574e0 con 0x7f1174004500 2022-07-20T03:17:09.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.511+0000 7f118bfff700 1 -- 172.21.15.61:0/1963265743 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v43) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f1184046220 con 0x7f1174004500 2022-07-20T03:17:09.517 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:09.518 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:17:09.518 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:17:09.518 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:17:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:17:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.519 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.520 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:17:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:17:09.521 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.522 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:09.522 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.522 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.522 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.523 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:17:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:17:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.524 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.525 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:17:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:17:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.526 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.527 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:17:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:17:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.528 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.529 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.530 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.530 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:17:09.530 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:17:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.531 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:17:09.532 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:17:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.533 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.534 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.534 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.534 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.534 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:17:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:17:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.535 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.536 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.536 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.536 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.536 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:17:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:17:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.537 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.538 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:17:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:17:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.539 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.540 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:17:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:17:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.541 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:09.542 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:17:09.542 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:17:09.542 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.542 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.543 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.543 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:17:09.543 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:17:09.543 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:17:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:17:09.544 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:17:09.545 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:17:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:17:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:17:09.546 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.547 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.547 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.547 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:17:09.547 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:17:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:17:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:09.548 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:09.549 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.549 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.549 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.549 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.550 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:17:09.550 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:17:09.550 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:17:09.550 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:17:09.551 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.551 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.551 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:09.551 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:09.552 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:17:09.552 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:17:09.552 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:09.552 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:17:09.553 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:17:09.553 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:09.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.553 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:09.554 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:09.554 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:09.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1178053a40 msgr2=0x7f1178055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.556 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1178053a40 0x7f1178055f00 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f117c019000 tx=0x7f117c015000).stop 2022-07-20T03:17:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 msgr2=0x7f1194159af0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1194159af0 secure :-1 s=READY pgs=1357 cs=0 l=1 rev1=1 rx=0x7f118400b000 tx=0x7f1184009040).stop 2022-07-20T03:17:09.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 shutdown_connections 2022-07-20T03:17:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1178053a40 0x7f1178055f00 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1194110240 0x7f1194144100 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f11940642a0 0x7f1194063990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 --2- 172.21.15.61:0/1963265743 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1174004500 0x7f1194159af0 unknown :-1 s=CLOSED pgs=1357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 >> 172.21.15.61:0/1963265743 conn(0x7f11940634e0 msgr2=0x7f11940ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 shutdown_connections 2022-07-20T03:17:09.559 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.515+0000 7f119359e700 1 -- 172.21.15.61:0/1963265743 wait complete. 2022-07-20T03:17:09.560 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:09.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 Processor -- start 2022-07-20T03:17:09.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 -- start start 2022-07-20T03:17:09.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29200b61d0 0x7f2920143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29201434f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f29200afac0 con 0x7f29201439e0 2022-07-20T03:17:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f29200add80 con 0x7f29200b61d0 2022-07-20T03:17:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2926f27700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f29200b65b0 con 0x7f29200b18d0 2022-07-20T03:17:09.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29200b61d0 0x7f2920143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f29200b61d0 0x7f2920143000 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50944/0 (socket says 172.21.15.61:50944) 2022-07-20T03:17:09.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 -- 172.21.15.61:0/2446612329 learned_addr learned my addr 172.21.15.61:0/2446612329 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:09.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925724700 1 --2- 172.21.15.61:0/2446612329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29201434f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 -- 172.21.15.61:0/2446612329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 msgr2=0x7f29201434f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:09.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 --2- 172.21.15.61:0/2446612329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29201434f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 -- 172.21.15.61:0/2446612329 >> v1:172.21.15.117:6789/0 conn(0x7f29201439e0 legacy=0x7f292013ec50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 -- 172.21.15.61:0/2446612329 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f292005c3e0 con 0x7f29200b61d0 2022-07-20T03:17:09.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.663+0000 7f2925f25700 1 --2- 172.21.15.61:0/2446612329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f2920143000 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f291c002f70 tx=0x7f291c006eb0).ready entity=mon.0 client_cookie=cb5e2545764cb7af server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2924f23700 1 -- 172.21.15.61:0/2446612329 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f291c00d570 con 0x7f29200b61d0 2022-07-20T03:17:09.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2924f23700 1 -- 172.21.15.61:0/2446612329 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f291c000e90 con 0x7f29200b61d0 2022-07-20T03:17:09.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2924f23700 1 -- 172.21.15.61:0/2446612329 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f291c00d900 con 0x7f29200b61d0 2022-07-20T03:17:09.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 -- 172.21.15.61:0/2446612329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 msgr2=0x7f2920143000 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 --2- 172.21.15.61:0/2446612329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f2920143000 secure :-1 s=READY pgs=1358 cs=0 l=1 rev1=1 rx=0x7f291c002f70 tx=0x7f291c006eb0).stop 2022-07-20T03:17:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 -- 172.21.15.61:0/2446612329 shutdown_connections 2022-07-20T03:17:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 --2- 172.21.15.61:0/2446612329 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29201434f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 --2- 172.21.15.61:0/2446612329 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f2920143000 unknown :-1 s=CLOSED pgs=1358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 -- 172.21.15.61:0/2446612329 >> 172.21.15.61:0/2446612329 conn(0x7f2920063cf0 msgr2=0x7f29200b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 -- 172.21.15.61:0/2446612329 shutdown_connections 2022-07-20T03:17:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 -- 172.21.15.61:0/2446612329 wait complete. 2022-07-20T03:17:09.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 Processor -- start 2022-07-20T03:17:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.667+0000 7f2926f27700 1 -- start start 2022-07-20T03:17:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29200b4f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f29201398e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29201439e0 0x7f2920135fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2920064a40 con 0x7f29200b61d0 2022-07-20T03:17:09.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29200647e0 con 0x7f29201439e0 2022-07-20T03:17:09.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f292013cad0 con 0x7f29200b18d0 2022-07-20T03:17:09.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f29201398e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f29201398e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50950/0 (socket says 172.21.15.61:50950) 2022-07-20T03:17:09.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 -- 172.21.15.61:0/1779677879 learned_addr learned my addr 172.21.15.61:0/1779677879 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:09.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926726700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29201439e0 0x7f2920135fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925f25700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29200b4f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 -- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 msgr2=0x7f29200b4f10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29200b4f10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 -- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29201439e0 msgr2=0x7f2920135fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29201439e0 0x7f2920135fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 -- 172.21.15.61:0/1779677879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f291c00b1e0 con 0x7f29200b61d0 2022-07-20T03:17:09.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2925724700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f29201398e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f291000c000 tx=0x7f2910006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2910010780 con 0x7f29200b61d0 2022-07-20T03:17:09.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f29100108e0 con 0x7f29200b61d0 2022-07-20T03:17:09.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f292013ff10 con 0x7f29200b61d0 2022-07-20T03:17:09.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2910010c40 con 0x7f29200b61d0 2022-07-20T03:17:09.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29200643b0 con 0x7f29200b61d0 2022-07-20T03:17:09.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.671+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29200679b0 con 0x7f29200b61d0 2022-07-20T03:17:09.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.675+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f291001c460 con 0x7f29200b61d0 2022-07-20T03:17:09.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.675+0000 7f290effd700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2900053a40 0x7f2900055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.675+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f291006c060 con 0x7f29200b61d0 2022-07-20T03:17:09.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.675+0000 7f2925f25700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2900053a40 0x7f2900055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.675+0000 7f2925f25700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2900053a40 0x7f2900055f00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f291c01c000 tx=0x7f291c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.679+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2910043180 con 0x7f29200b61d0 2022-07-20T03:17:09.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.807+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f29200574e0 con 0x7f29200b61d0 2022-07-20T03:17:09.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.807+0000 7f290effd700 1 -- 172.21.15.61:0/1779677879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v136) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f29100467a0 con 0x7f29200b61d0 2022-07-20T03:17:09.814 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:09.815 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":136,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:09.198289+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":26,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":120,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":128,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:09.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2900053a40 msgr2=0x7f2900055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2900053a40 0x7f2900055f00 secure :-1 s=READY pgs=655 cs=0 l=1 rev1=1 rx=0x7f291c01c000 tx=0x7f291c015000).stop 2022-07-20T03:17:09.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 msgr2=0x7f29201398e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f29201398e0 secure :-1 s=READY pgs=1359 cs=0 l=1 rev1=1 rx=0x7f291000c000 tx=0x7f2910006df0).stop 2022-07-20T03:17:09.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 shutdown_connections 2022-07-20T03:17:09.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2900053a40 0x7f2900055f00 unknown :-1 s=CLOSED pgs=655 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f29201439e0 0x7f2920135fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f29200b61d0 0x7f29201398e0 unknown :-1 s=CLOSED pgs=1359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 --2- 172.21.15.61:0/1779677879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f29200b18d0 0x7f29200b4f10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 >> 172.21.15.61:0/1779677879 conn(0x7f2920063cf0 msgr2=0x7f29200aef30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:09.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 shutdown_connections 2022-07-20T03:17:09.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.811+0000 7f2926f27700 1 -- 172.21.15.61:0/1779677879 wait complete. 2022-07-20T03:17:09.834 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:17:09.835 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:17:09.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.939+0000 7f8d2b59e700 1 Processor -- start 2022-07-20T03:17:09.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.939+0000 7f8d2b59e700 1 -- start start 2022-07-20T03:17:09.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.939+0000 7f8d2b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c0b1de0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.939+0000 7f8d2b59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8d2c143870 0x7f8d2c13fb50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.939+0000 7f8d2b59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8d2c098ab0 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.939+0000 7f8d2b59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8d2c0ac480 con 0x7f8d2c143870 2022-07-20T03:17:09.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c0ac720 con 0x7f8d2c13f5c0 2022-07-20T03:17:09.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8d2c143870 0x7f8d2c13fb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8d2c143870 0x7f8d2c13fb50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50962/0 (socket says 172.21.15.61:50962) 2022-07-20T03:17:09.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 -- 172.21.15.61:0/3881575669 learned_addr learned my addr 172.21.15.61:0/3881575669 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:09.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d29d9b700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c0b1de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 -- 172.21.15.61:0/3881575669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 msgr2=0x7f8d2c0b1de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:09.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c0b1de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 -- 172.21.15.61:0/3881575669 >> v1:172.21.15.117:6789/0 conn(0x7f8d2c0b1a00 legacy=0x7f8d2c13eeb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 -- 172.21.15.61:0/3881575669 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2c143d20 con 0x7f8d2c143870 2022-07-20T03:17:09.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d29d9b700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c0b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:09.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c143870 0x7f8d2c13fb50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f8d240032f0 tx=0x7f8d2400b530).ready entity=mon.0 client_cookie=b779cd485d4439d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2959a700 1 -- 172.21.15.61:0/3881575669 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d24010660 con 0x7f8d2c143870 2022-07-20T03:17:09.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2959a700 1 -- 172.21.15.61:0/3881575669 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d240107c0 con 0x7f8d2c143870 2022-07-20T03:17:09.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2959a700 1 -- 172.21.15.61:0/3881575669 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d24010b20 con 0x7f8d2c143870 2022-07-20T03:17:09.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- 172.21.15.61:0/3881575669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c143870 msgr2=0x7f8d2c13fb50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c143870 0x7f8d2c13fb50 secure :-1 s=READY pgs=1360 cs=0 l=1 rev1=1 rx=0x7f8d240032f0 tx=0x7f8d2400b530).stop 2022-07-20T03:17:09.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- 172.21.15.61:0/3881575669 shutdown_connections 2022-07-20T03:17:09.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c143870 0x7f8d2c13fb50 unknown :-1 s=CLOSED pgs=1360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/3881575669 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c0b1de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- 172.21.15.61:0/3881575669 >> 172.21.15.61:0/3881575669 conn(0x7f8d2c0b2af0 msgr2=0x7f8d2c0b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- 172.21.15.61:0/3881575669 shutdown_connections 2022-07-20T03:17:09.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- 172.21.15.61:0/3881575669 wait complete. 2022-07-20T03:17:09.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 Processor -- start 2022-07-20T03:17:09.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- start start 2022-07-20T03:17:09.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 0x7f8d2c096fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c09ab10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d2c143870 0x7f8d2c0ad850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c13dfd0 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c094dc0 con 0x7f8d2c143870 2022-07-20T03:17:09.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2b59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8d2c094ff0 con 0x7f8d2c13f5c0 2022-07-20T03:17:09.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 0x7f8d2c096fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2a59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 0x7f8d2c096fc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50964/0 (socket says 172.21.15.61:50964) 2022-07-20T03:17:09.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2a59c700 1 -- 172.21.15.61:0/4244985386 learned_addr learned my addr 172.21.15.61:0/4244985386 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:09.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d29d9b700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c09ab10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.943+0000 7f8d2ad9d700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d2c143870 0x7f8d2c0ad850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2a59c700 1 -- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 msgr2=0x7f8d2c09ab10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2a59c700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c09ab10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2a59c700 1 -- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d2c143870 msgr2=0x7f8d2c0ad850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2a59c700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d2c143870 0x7f8d2c0ad850 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2a59c700 1 -- 172.21.15.61:0/4244985386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d2400b7d0 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2a59c700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 0x7f8d2c096fc0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f8d2c0b36b0 tx=0x7f8d1c007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d1c00ce30 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d2c095290 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8d1c007240 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d2c099b90 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8d1c006700 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8d1c010030 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d2c0ad560 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d237fe700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d10053920 0x7f8d10055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:09.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f8d1c06b6e0 con 0x7f8d2c0b1a00 2022-07-20T03:17:09.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.947+0000 7f8d29d9b700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d10053920 0x7f8d10055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:09.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.951+0000 7f8d29d9b700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d10053920 0x7f8d10055de0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f8d1400c000 tx=0x7f8d14008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:09.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:09.955+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8d1c0428d0 con 0x7f8d2c0b1a00 2022-07-20T03:17:10.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.083+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f8d2c098ab0 con 0x7f8d2c0b1a00 2022-07-20T03:17:10.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.083+0000 7f8d237fe700 1 -- 172.21.15.61:0/4244985386 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v136) v1 ==== 110+0+10 (secure 0 0 0) 0x7f8d2c098ab0 con 0x7f8d2c0b1a00 2022-07-20T03:17:10.090 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:17:10.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d10053920 msgr2=0x7f8d10055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d10053920 0x7f8d10055de0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7f8d1400c000 tx=0x7f8d14008040).stop 2022-07-20T03:17:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 msgr2=0x7f8d2c096fc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 0x7f8d2c096fc0 secure :-1 s=READY pgs=1361 cs=0 l=1 rev1=1 rx=0x7f8d2c0b36b0 tx=0x7f8d1c007c10).stop 2022-07-20T03:17:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 shutdown_connections 2022-07-20T03:17:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8d10053920 0x7f8d10055de0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8d2c143870 0x7f8d2c0ad850 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8d2c13f5c0 0x7f8d2c09ab10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 --2- 172.21.15.61:0/4244985386 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8d2c0b1a00 0x7f8d2c096fc0 unknown :-1 s=CLOSED pgs=1361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 >> 172.21.15.61:0/4244985386 conn(0x7f8d2c0b2af0 msgr2=0x7f8d2c060770 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:10.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 shutdown_connections 2022-07-20T03:17:10.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.087+0000 7f8d2b59e700 1 -- 172.21.15.61:0/4244985386 wait complete. 2022-07-20T03:17:10.103 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:17:10.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 Processor -- start 2022-07-20T03:17:10.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 -- start start 2022-07-20T03:17:10.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4bc00967e0 0x7f4bc0140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4bc00ac680 0x7f4bc00b25f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4bc013fe80 con 0x7f4bc00b2210 2022-07-20T03:17:10.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc0140120 con 0x7f4bc00ac680 2022-07-20T03:17:10.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4bc01403c0 con 0x7f4bc00967e0 2022-07-20T03:17:10.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4bc00ac680 0x7f4bc00b25f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbf7fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4bc00b2210 0x7f4bc00ab0b0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47570/0 (socket says 172.21.15.61:47570) 2022-07-20T03:17:10.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbf7fe700 1 -- 172.21.15.61:0/1314487781 learned_addr learned my addr 172.21.15.61:0/1314487781 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:10.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1314487781 >> v2:172.21.15.117:3300/0 conn(0x7f4bc00967e0 0x7f4bc0140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbffff700 1 -- 172.21.15.61:0/1314487781 >> v2:172.21.15.117:3300/0 conn(0x7f4bc00967e0 msgr2=0x7f4bc0140860 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:10.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbffff700 1 --2- 172.21.15.61:0/1314487781 >> v2:172.21.15.117:3300/0 conn(0x7f4bc00967e0 0x7f4bc0140860 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbffff700 1 -- 172.21.15.61:0/1314487781 >> v1:172.21.15.61:6789/0 conn(0x7f4bc00b2210 legacy=0x7f4bc00ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbffff700 1 -- 172.21.15.61:0/1314487781 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bc00ab7c0 con 0x7f4bc00ac680 2022-07-20T03:17:10.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/1314487781 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1423395196 0 0) 0x7f4bc013fe80 con 0x7f4bc00b2210 2022-07-20T03:17:10.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbffff700 1 --2- 172.21.15.61:0/1314487781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc00b25f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f4bb80032f0 tx=0x7f4bb80091d0).ready entity=mon.0 client_cookie=24307287409cd871 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/1314487781 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb800b680 con 0x7f4bc00ac680 2022-07-20T03:17:10.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/1314487781 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb800b7e0 con 0x7f4bc00ac680 2022-07-20T03:17:10.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bbe7fc700 1 -- 172.21.15.61:0/1314487781 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb800b950 con 0x7f4bc00ac680 2022-07-20T03:17:10.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1314487781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 msgr2=0x7f4bc00b25f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1314487781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc00b25f0 secure :-1 s=READY pgs=1363 cs=0 l=1 rev1=1 rx=0x7f4bb80032f0 tx=0x7f4bb80091d0).stop 2022-07-20T03:17:10.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1314487781 shutdown_connections 2022-07-20T03:17:10.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1314487781 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc00b25f0 unknown :-1 s=CLOSED pgs=1363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.211+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1314487781 >> v2:172.21.15.117:3300/0 conn(0x7f4bc00967e0 0x7f4bc0140860 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1314487781 >> 172.21.15.61:0/1314487781 conn(0x7f4bc00b36c0 msgr2=0x7f4bc00b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:10.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1314487781 shutdown_connections 2022-07-20T03:17:10.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1314487781 wait complete. 2022-07-20T03:17:10.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 Processor -- start 2022-07-20T03:17:10.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- start start 2022-07-20T03:17:10.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bc00967e0 0x7f4bc0151df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc005c6b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bc00b2210 0x7f4bc013af70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc0093fe0 con 0x7f4bc00ac680 2022-07-20T03:17:10.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc005fc30 con 0x7f4bc00b2210 2022-07-20T03:17:10.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bc5c23700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4bc00b4950 con 0x7f4bc00967e0 2022-07-20T03:17:10.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc005c6b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc005c6b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50980/0 (socket says 172.21.15.61:50980) 2022-07-20T03:17:10.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1563123409 learned_addr learned my addr 172.21.15.61:0/1563123409 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:10.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbffff700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bc00b2210 0x7f4bc013af70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbf7fe700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bc00967e0 0x7f4bc0151df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bc00967e0 msgr2=0x7f4bc0151df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bc00967e0 0x7f4bc0151df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bc00b2210 msgr2=0x7f4bc013af70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bc00b2210 0x7f4bc013af70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 -- 172.21.15.61:0/1563123409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4bb8009470 con 0x7f4bc00ac680 2022-07-20T03:17:10.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.215+0000 7f4bbeffd700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc005c6b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f4bb400c000 tx=0x7f4bb4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb4010590 con 0x7f4bc00ac680 2022-07-20T03:17:10.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4bb40106f0 con 0x7f4bc00ac680 2022-07-20T03:17:10.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4bc005cba0 con 0x7f4bc00ac680 2022-07-20T03:17:10.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4bb4010a50 con 0x7f4bc00ac680 2022-07-20T03:17:10.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4bc0093ce0 con 0x7f4bc00ac680 2022-07-20T03:17:10.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4bb4010bb0 con 0x7f4bc00ac680 2022-07-20T03:17:10.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4bc00941e0 con 0x7f4bc00ac680 2022-07-20T03:17:10.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbcff9700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4ba00539b0 0x7f4ba0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4bb406ba90 con 0x7f4bc00ac680 2022-07-20T03:17:10.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbf7fe700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4ba00539b0 0x7f4ba0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.219+0000 7f4bbf7fe700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4ba00539b0 0x7f4ba0055e70 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f4bb000d000 tx=0x7f4bb0007c00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.227+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4bb4042d20 con 0x7f4bc00ac680 2022-07-20T03:17:10.354 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f4bc00574e0 con 0x7f4bc00ac680 2022-07-20T03:17:10.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bbcff9700 1 -- 172.21.15.61:0/1563123409 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v136) v1 ==== 92+0+10 (secure 0 0 0) 0x7f4bb4046340 con 0x7f4bc00ac680 2022-07-20T03:17:10.356 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:10.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4ba00539b0 msgr2=0x7f4ba0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4ba00539b0 0x7f4ba0055e70 secure :-1 s=READY pgs=657 cs=0 l=1 rev1=1 rx=0x7f4bb000d000 tx=0x7f4bb0007c00).stop 2022-07-20T03:17:10.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 msgr2=0x7f4bc005c6b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc005c6b0 secure :-1 s=READY pgs=1364 cs=0 l=1 rev1=1 rx=0x7f4bb400c000 tx=0x7f4bb4006df0).stop 2022-07-20T03:17:10.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 shutdown_connections 2022-07-20T03:17:10.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4ba00539b0 0x7f4ba0055e70 unknown :-1 s=CLOSED pgs=657 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4bc00b2210 0x7f4bc013af70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4bc00ac680 0x7f4bc005c6b0 unknown :-1 s=CLOSED pgs=1364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 --2- 172.21.15.61:0/1563123409 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4bc00967e0 0x7f4bc0151df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 >> 172.21.15.61:0/1563123409 conn(0x7f4bc00b36c0 msgr2=0x7f4bc0098380 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:10.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 shutdown_connections 2022-07-20T03:17:10.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.351+0000 7f4bc5c23700 1 -- 172.21.15.61:0/1563123409 wait complete. 2022-07-20T03:17:10.370 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:17:10.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 Processor -- start 2022-07-20T03:17:10.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 -- start start 2022-07-20T03:17:10.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac0b0950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1fac1424e0 0x7f1fac0ad350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1fac1433d0 con 0x7f1fac0aea10 2022-07-20T03:17:10.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1fac143b10 con 0x7f1fac1424e0 2022-07-20T03:17:10.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fb094a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac143670 con 0x7f1fac0b0570 2022-07-20T03:17:10.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.467+0000 7f1fabfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1fac1424e0 0x7f1fac0ad350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fabfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1fac1424e0 0x7f1fac0ad350 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50988/0 (socket says 172.21.15.61:50988) 2022-07-20T03:17:10.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fab7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac0b0950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fabfff700 1 -- 172.21.15.61:0/1079764956 learned_addr learned my addr 172.21.15.61:0/1079764956 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:10.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fab7fe700 1 -- 172.21.15.61:0/1079764956 >> v2:172.21.15.61:3300/0 conn(0x7f1fac1424e0 msgr2=0x7f1fac0ad350 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:10.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fab7fe700 1 --2- 172.21.15.61:0/1079764956 >> v2:172.21.15.61:3300/0 conn(0x7f1fac1424e0 0x7f1fac0ad350 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fab7fe700 1 -- 172.21.15.61:0/1079764956 >> v1:172.21.15.117:6789/0 conn(0x7f1fac0aea10 legacy=0x7f1fac141dd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1faa7fc700 1 -- 172.21.15.61:0/1079764956 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 30828081 0 0) 0x7f1fac1433d0 con 0x7f1fac0aea10 2022-07-20T03:17:10.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fab7fe700 1 -- 172.21.15.61:0/1079764956 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fac0b1010 con 0x7f1fac0b0570 2022-07-20T03:17:10.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fab7fe700 1 --2- 172.21.15.61:0/1079764956 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac0b0950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1fa0002f70 tx=0x7f1fa0006df0).ready entity=mon.2 client_cookie=942f6a7a9933da74 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1faa7fc700 1 -- 172.21.15.61:0/1079764956 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fa000d570 con 0x7f1fac0b0570 2022-07-20T03:17:10.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1faa7fc700 1 -- 172.21.15.61:0/1079764956 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa0000e90 con 0x7f1fac0b0570 2022-07-20T03:17:10.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1faa7fc700 1 -- 172.21.15.61:0/1079764956 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fa000d8c0 con 0x7f1fac0b0570 2022-07-20T03:17:10.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 -- 172.21.15.61:0/1079764956 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 msgr2=0x7f1fac0b0950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 --2- 172.21.15.61:0/1079764956 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac0b0950 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f1fa0002f70 tx=0x7f1fa0006df0).stop 2022-07-20T03:17:10.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 -- 172.21.15.61:0/1079764956 shutdown_connections 2022-07-20T03:17:10.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 --2- 172.21.15.61:0/1079764956 >> v2:172.21.15.61:3300/0 conn(0x7f1fac1424e0 0x7f1fac0ad350 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 --2- 172.21.15.61:0/1079764956 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac0b0950 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 -- 172.21.15.61:0/1079764956 >> 172.21.15.61:0/1079764956 conn(0x7f1fac0b9360 msgr2=0x7f1fac0b9740 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:10.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 -- 172.21.15.61:0/1079764956 shutdown_connections 2022-07-20T03:17:10.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 -- 172.21.15.61:0/1079764956 wait complete. 2022-07-20T03:17:10.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 Processor -- start 2022-07-20T03:17:10.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.471+0000 7f1fb094a700 1 -- start start 2022-07-20T03:17:10.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 0x7f1fac146830 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac13e750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fac1424e0 0x7f1fac0b1750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac1560e0 con 0x7f1fac0aea10 2022-07-20T03:17:10.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac0b1310 con 0x7f1fac1424e0 2022-07-20T03:17:10.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1fac0b3ce0 con 0x7f1fac0b0570 2022-07-20T03:17:10.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 0x7f1fac146830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 0x7f1fac146830 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:50990/0 (socket says 172.21.15.61:50990) 2022-07-20T03:17:10.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 -- 172.21.15.61:0/4213524476 learned_addr learned my addr 172.21.15.61:0/4213524476 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:10.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1faaffd700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac13e750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fabfff700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fac1424e0 0x7f1fac0b1750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 -- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 msgr2=0x7f1fac13e750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac13e750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 -- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fac1424e0 msgr2=0x7f1fac0b1750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fac1424e0 0x7f1fac0b1750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 -- 172.21.15.61:0/4213524476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fa000b180 con 0x7f1fac0aea10 2022-07-20T03:17:10.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fab7fe700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 0x7f1fac146830 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f1fa0002f70 tx=0x7f1fa0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fa0016070 con 0x7f1fac0aea10 2022-07-20T03:17:10.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1fa0013a70 con 0x7f1fac0aea10 2022-07-20T03:17:10.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fac13f080 con 0x7f1fac0aea10 2022-07-20T03:17:10.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1fa0013dd0 con 0x7f1fac0aea10 2022-07-20T03:17:10.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.475+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fac1455e0 con 0x7f1fac0aea10 2022-07-20T03:17:10.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.479+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1fa000f030 con 0x7f1fac0aea10 2022-07-20T03:17:10.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.479+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fac0b3b30 con 0x7f1fac0aea10 2022-07-20T03:17:10.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.479+0000 7f1fa8ff9700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1f8c053a40 0x7f1f8c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.479+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f1fa006c9b0 con 0x7f1fac0aea10 2022-07-20T03:17:10.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.479+0000 7f1faaffd700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1f8c053a40 0x7f1f8c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.479+0000 7f1faaffd700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1f8c053a40 0x7f1f8c055f00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f1f9c00b000 tx=0x7f1f9c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.483+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1fa0043ab0 con 0x7f1fac0aea10 2022-07-20T03:17:10.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.615+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1fac0b1510 con 0x7f1fac0aea10 2022-07-20T03:17:10.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.615+0000 7f1fa8ff9700 1 -- 172.21.15.61:0/4213524476 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v136) v1 ==== 84+0+143 (secure 0 0 0) 0x7f1fa0023070 con 0x7f1fac0aea10 2022-07-20T03:17:10.621 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:10.621 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:17:10.621 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:10.622 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:17:10.622 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:17:10.622 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:10.623 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:10.623 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:17:10.623 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:17:10.623 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:10.624 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:17:10.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1f8c053a40 msgr2=0x7f1f8c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1f8c053a40 0x7f1f8c055f00 secure :-1 s=READY pgs=658 cs=0 l=1 rev1=1 rx=0x7f1f9c00b000 tx=0x7f1f9c009040).stop 2022-07-20T03:17:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 msgr2=0x7f1fac146830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 0x7f1fac146830 secure :-1 s=READY pgs=1365 cs=0 l=1 rev1=1 rx=0x7f1fa0002f70 tx=0x7f1fa0014000).stop 2022-07-20T03:17:10.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 shutdown_connections 2022-07-20T03:17:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1f8c053a40 0x7f1f8c055f00 unknown :-1 s=CLOSED pgs=658 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1fac1424e0 0x7f1fac0b1750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1fac0b0570 0x7f1fac13e750 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 --2- 172.21.15.61:0/4213524476 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1fac0aea10 0x7f1fac146830 unknown :-1 s=CLOSED pgs=1365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 >> 172.21.15.61:0/4213524476 conn(0x7f1fac0b9360 msgr2=0x7f1fac141020 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:10.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 shutdown_connections 2022-07-20T03:17:10.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.619+0000 7f1fb094a700 1 -- 172.21.15.61:0/4213524476 wait complete. 2022-07-20T03:17:10.635 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:17:10.636 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:17:10.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f059a81e700 1 Processor -- start 2022-07-20T03:17:10.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f059a81e700 1 -- start start 2022-07-20T03:17:10.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f059a81e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f05940ab4a0 0x7f0594061810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f059a81e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05940ac340 con 0x7f05940b34d0 2022-07-20T03:17:10.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f059a81e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f05940ac5e0 con 0x7f0594145f60 2022-07-20T03:17:10.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f059a81e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f05940acd20 con 0x7f05940ab4a0 2022-07-20T03:17:10.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f0593fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f05940b34d0 0x7f0594061100 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47596/0 (socket says 172.21.15.61:47596) 2022-07-20T03:17:10.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f0593fff700 1 -- 172.21.15.61:0/907180545 learned_addr learned my addr 172.21.15.61:0/907180545 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:10.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 --2- 172.21.15.61:0/907180545 >> v2:172.21.15.117:3300/0 conn(0x7f05940ab4a0 0x7f0594061810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4001185747 0 0) 0x7f05940ac340 con 0x7f05940b34d0 2022-07-20T03:17:10.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f0574003260 con 0x7f05940b34d0 2022-07-20T03:17:10.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1286595447 0 0) 0x7f0574003260 con 0x7f05940b34d0 2022-07-20T03:17:10.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 >> v2:172.21.15.117:3300/0 conn(0x7f05940ab4a0 msgr2=0x7f0594061810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:10.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 --2- 172.21.15.61:0/907180545 >> v2:172.21.15.117:3300/0 conn(0x7f05940ab4a0 0x7f0594061810 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 >> v1:172.21.15.117:6789/0 conn(0x7f0594145f60 legacy=0x7f05941463f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f05940ac880 con 0x7f05940b34d0 2022-07-20T03:17:10.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f0584001660 con 0x7f05940b34d0 2022-07-20T03:17:10.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 >> v1:172.21.15.61:6789/0 conn(0x7f05940b34d0 legacy=0x7f0594061100 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0574003750 0x7f0574003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0574004500 0x7f0574004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004e70 0x7f057400b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f05940acd20 con 0x7f0574004500 2022-07-20T03:17:10.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f057400bdd0 con 0x7f0574003750 2022-07-20T03:17:10.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f057400bf90 con 0x7f0574004e70 2022-07-20T03:17:10.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0574004500 0x7f0574004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f0593fff700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0574003750 0x7f0574003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f0598dbb700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004e70 0x7f057400b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 -- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004e70 msgr2=0x7f057400b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004e70 0x7f057400b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 -- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0574003750 msgr2=0x7f0574003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:10.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0574003750 0x7f0574003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f0598dbb700 1 -- 172.21.15.61:0/907180545 reap_dead start 2022-07-20T03:17:10.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 -- 172.21.15.61:0/907180545 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0574003260 con 0x7f0574004500 2022-07-20T03:17:10.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f058bfff700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0574004500 0x7f0574004980 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f057c00c000 tx=0x7f057c006df0).ready entity=mon.0 client_cookie=9934cad9cd1299b6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057c00e830 con 0x7f0574004500 2022-07-20T03:17:10.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c00e990 con 0x7f0574004500 2022-07-20T03:17:10.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.747+0000 7f05937fe700 1 -- 172.21.15.61:0/907180545 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057c00ecf0 con 0x7f0574004500 2022-07-20T03:17:10.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- 172.21.15.61:0/907180545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0574004500 msgr2=0x7f0574004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0574004500 0x7f0574004980 secure :-1 s=READY pgs=1367 cs=0 l=1 rev1=1 rx=0x7f057c00c000 tx=0x7f057c006df0).stop 2022-07-20T03:17:10.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- 172.21.15.61:0/907180545 shutdown_connections 2022-07-20T03:17:10.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 --2- 172.21.15.61:0/907180545 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0574004500 0x7f0574004980 unknown :-1 s=CLOSED pgs=1367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- 172.21.15.61:0/907180545 >> 172.21.15.61:0/907180545 conn(0x7f05940b6c50 msgr2=0x7f05940b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:10.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- 172.21.15.61:0/907180545 shutdown_connections 2022-07-20T03:17:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- 172.21.15.61:0/907180545 wait complete. 2022-07-20T03:17:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 Processor -- start 2022-07-20T03:17:10.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- start start 2022-07-20T03:17:10.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004500 0x7f0594156440 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 0x7f0594140780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f059415a620 0x7f0594140c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0594156930 con 0x7f059414a050 2022-07-20T03:17:10.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0594145320 con 0x7f059415a620 2022-07-20T03:17:10.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f059a81e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f05940af040 con 0x7f0574004500 2022-07-20T03:17:10.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.751+0000 7f058bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 0x7f0594140780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 0x7f0594140780 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51012/0 (socket says 172.21.15.61:51012) 2022-07-20T03:17:10.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 -- 172.21.15.61:0/2562684455 learned_addr learned my addr 172.21.15.61:0/2562684455 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:10.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0593fff700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004500 0x7f0594156440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0598dbb700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f059415a620 0x7f0594140c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 -- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004500 msgr2=0x7f0594156440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004500 0x7f0594156440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 -- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f059415a620 msgr2=0x7f0594140c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f059415a620 0x7f0594140c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 -- 172.21.15.61:0/2562684455 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0594145550 con 0x7f059414a050 2022-07-20T03:17:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0598dbb700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f059415a620 0x7f0594140c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:10.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f058bfff700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 0x7f0594140780 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f0594141160 tx=0x7f057c019000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057c01cd40 con 0x7f059414a050 2022-07-20T03:17:10.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f057c000e90 con 0x7f059414a050 2022-07-20T03:17:10.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0594141500 con 0x7f059414a050 2022-07-20T03:17:10.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f057c01cea0 con 0x7f059414a050 2022-07-20T03:17:10.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0594099440 con 0x7f059414a050 2022-07-20T03:17:10.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f05940574e0 con 0x7f059414a050 2022-07-20T03:17:10.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.755+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f057c02a020 con 0x7f059414a050 2022-07-20T03:17:10.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.759+0000 7f0591ffb700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05780539d0 0x7f0578055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:10.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.759+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(136..136 src has 1..136) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f057c06d080 con 0x7f059414a050 2022-07-20T03:17:10.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.759+0000 7f0593fff700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05780539d0 0x7f0578055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:10.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.759+0000 7f0593fff700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05780539d0 0x7f0578055e90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f0584005000 tx=0x7f05940ac770).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:10.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.763+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f057c047d40 con 0x7f059414a050 2022-07-20T03:17:10.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:10.891+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7f059405db10 con 0x7f059414a050 2022-07-20T03:17:11.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.663+0000 7f0591ffb700 1 -- 172.21.15.61:0/2562684455 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v137) v1 ==== 133+0+0 (secure 0 0 0) 0x7f057c047930 con 0x7f059414a050 2022-07-20T03:17:11.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.667+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05780539d0 msgr2=0x7f0578055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05780539d0 0x7f0578055e90 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7f0584005000 tx=0x7f05940ac770).stop 2022-07-20T03:17:11.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 msgr2=0x7f0594140780 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 0x7f0594140780 secure :-1 s=READY pgs=1368 cs=0 l=1 rev1=1 rx=0x7f0594141160 tx=0x7f057c019000).stop 2022-07-20T03:17:11.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 shutdown_connections 2022-07-20T03:17:11.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f05780539d0 0x7f0578055e90 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f059415a620 0x7f0594140c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f059414a050 0x7f0594140780 unknown :-1 s=CLOSED pgs=1368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 --2- 172.21.15.61:0/2562684455 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0574004500 0x7f0594156440 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 >> 172.21.15.61:0/2562684455 conn(0x7f05940b6c50 msgr2=0x7f059405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:11.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 shutdown_connections 2022-07-20T03:17:11.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.671+0000 7f059a81e700 1 -- 172.21.15.61:0/2562684455 wait complete. 2022-07-20T03:17:11.682 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:17:11.688 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:17:11.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c5a7fc700 1 Processor -- start 2022-07-20T03:17:11.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c5a7fc700 1 -- start start 2022-07-20T03:17:11.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c5a7fc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f2c541416e0 0x7f2c5405fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:11.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c5a7fc700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c54060fa0 con 0x7f2c54141280 2022-07-20T03:17:11.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c5a7fc700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f2c540b02e0 con 0x7f2c54140490 2022-07-20T03:17:11.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c5a7fc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2c5405d4b0 con 0x7f2c541416e0 2022-07-20T03:17:11.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c537fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f2c54141280 0x7f2c5405cda0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47618/0 (socket says 172.21.15.61:47618) 2022-07-20T03:17:11.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c537fe700 1 -- 172.21.15.61:0/3591267863 learned_addr learned my addr 172.21.15.61:0/3591267863 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:11.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c58d99700 1 --2- 172.21.15.61:0/3591267863 >> v2:172.21.15.117:3300/0 conn(0x7f2c541416e0 0x7f2c5405fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:11.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1333575197 0 0) 0x7f2c54060fa0 con 0x7f2c54141280 2022-07-20T03:17:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c34003260 con 0x7f2c54141280 2022-07-20T03:17:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3218347239 0 0) 0x7f2c540b02e0 con 0x7f2c54140490 2022-07-20T03:17:11.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f2c54060fa0 con 0x7f2c54140490 2022-07-20T03:17:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c58d99700 1 -- 172.21.15.61:0/3591267863 >> v1:172.21.15.117:6789/0 conn(0x7f2c54140490 legacy=0x7f2c540afa40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c58d99700 1 -- 172.21.15.61:0/3591267863 >> v1:172.21.15.61:6789/0 conn(0x7f2c54141280 legacy=0x7f2c5405cda0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c58d99700 1 -- 172.21.15.61:0/3591267863 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c54140b40 con 0x7f2c541416e0 2022-07-20T03:17:11.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c58d99700 1 --2- 172.21.15.61:0/3591267863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c541416e0 0x7f2c5405fa50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f2c4c0032f0 tx=0x7f2c4c0091d0).ready entity=mon.1 client_cookie=49821479d015699e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:11.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c4c00b600 con 0x7f2c541416e0 2022-07-20T03:17:11.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c00b760 con 0x7f2c541416e0 2022-07-20T03:17:11.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.795+0000 7f2c52ffd700 1 -- 172.21.15.61:0/3591267863 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c4c00b8d0 con 0x7f2c541416e0 2022-07-20T03:17:11.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/3591267863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c541416e0 msgr2=0x7f2c5405fa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/3591267863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c541416e0 0x7f2c5405fa50 secure :-1 s=READY pgs=615 cs=0 l=1 rev1=1 rx=0x7f2c4c0032f0 tx=0x7f2c4c0091d0).stop 2022-07-20T03:17:11.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/3591267863 shutdown_connections 2022-07-20T03:17:11.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/3591267863 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c541416e0 0x7f2c5405fa50 unknown :-1 s=CLOSED pgs=615 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/3591267863 >> 172.21.15.61:0/3591267863 conn(0x7f2c540b5b70 msgr2=0x7f2c540b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:11.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/3591267863 shutdown_connections 2022-07-20T03:17:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/3591267863 wait complete. 2022-07-20T03:17:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 Processor -- start 2022-07-20T03:17:11.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- start start 2022-07-20T03:17:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c54140490 0x7f2c540ab460 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:11.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c54141280 0x7f2c540ae040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:11.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 0x7f2c540ae530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:11.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540aa9a0 con 0x7f2c541416e0 2022-07-20T03:17:11.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540adb00 con 0x7f2c54141280 2022-07-20T03:17:11.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c5a7fc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2c540adda0 con 0x7f2c54140490 2022-07-20T03:17:11.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c58d99700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 0x7f2c540ae530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c58d99700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 0x7f2c540ae530 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51028/0 (socket says 172.21.15.61:51028) 2022-07-20T03:17:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c58d99700 1 -- 172.21.15.61:0/4113454922 learned_addr learned my addr 172.21.15.61:0/4113454922 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:11.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c537fe700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c54141280 0x7f2c540ae040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.799+0000 7f2c53fff700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c54140490 0x7f2c540ab460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:11.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c58d99700 1 -- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c54140490 msgr2=0x7f2c540ab460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c58d99700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c54140490 0x7f2c540ab460 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c58d99700 1 -- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c54141280 msgr2=0x7f2c540ae040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:11.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c58d99700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c54141280 0x7f2c540ae040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:11.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c58d99700 1 -- 172.21.15.61:0/4113454922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2c4c009470 con 0x7f2c541416e0 2022-07-20T03:17:11.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c58d99700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 0x7f2c540ae530 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7f2c4c0032f0 tx=0x7f2c4c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:11.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c4c011b60 con 0x7f2c541416e0 2022-07-20T03:17:11.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2c4c011cc0 con 0x7f2c541416e0 2022-07-20T03:17:11.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2c540aa1e0 con 0x7f2c541416e0 2022-07-20T03:17:11.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2c4c01f5e0 con 0x7f2c541416e0 2022-07-20T03:17:11.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2c540ab950 con 0x7f2c541416e0 2022-07-20T03:17:11.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2c540aab10 con 0x7f2c541416e0 2022-07-20T03:17:11.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.803+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2c4c026030 con 0x7f2c541416e0 2022-07-20T03:17:11.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.807+0000 7f2c517fa700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c380539d0 0x7f2c38055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:11.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.807+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(137..137 src has 1..137) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f2c4c06c500 con 0x7f2c541416e0 2022-07-20T03:17:11.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.807+0000 7f2c53fff700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c380539d0 0x7f2c38055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:11.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.807+0000 7f2c53fff700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c380539d0 0x7f2c38055e90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f2c3c00d000 tx=0x7f2c3c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:11.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.811+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2c540aab10 con 0x7f2c541416e0 2022-07-20T03:17:11.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:11.939+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f2c540aaae0 con 0x7f2c541416e0 2022-07-20T03:17:12.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.675+0000 7f2c517fa700 1 -- 172.21.15.61:0/4113454922 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v138) v1 ==== 125+0+0 (secure 0 0 0) 0x7f2c4c0240a0 con 0x7f2c541416e0 2022-07-20T03:17:12.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c380539d0 msgr2=0x7f2c38055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:12.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c380539d0 0x7f2c38055e90 secure :-1 s=READY pgs=660 cs=0 l=1 rev1=1 rx=0x7f2c3c00d000 tx=0x7f2c3c007a20).stop 2022-07-20T03:17:12.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 msgr2=0x7f2c540ae530 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:12.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 0x7f2c540ae530 secure :-1 s=READY pgs=1370 cs=0 l=1 rev1=1 rx=0x7f2c4c0032f0 tx=0x7f2c4c016000).stop 2022-07-20T03:17:12.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 shutdown_connections 2022-07-20T03:17:12.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2c380539d0 0x7f2c38055e90 unknown :-1 s=CLOSED pgs=660 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f2c54141280 0x7f2c540ae040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2c541416e0 0x7f2c540ae530 unknown :-1 s=CLOSED pgs=1370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 --2- 172.21.15.61:0/4113454922 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2c54140490 0x7f2c540ab460 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 >> 172.21.15.61:0/4113454922 conn(0x7f2c540b5b70 msgr2=0x7f2c5413e040 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:12.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 shutdown_connections 2022-07-20T03:17:12.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.683+0000 7f2c5a7fc700 1 -- 172.21.15.61:0/4113454922 wait complete. 2022-07-20T03:17:12.695 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:17:12.702 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:17:12.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.811+0000 7ff45dc4d700 1 Processor -- start 2022-07-20T03:17:12.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.811+0000 7ff45dc4d700 1 -- start start 2022-07-20T03:17:12.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.811+0000 7ff45dc4d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4580b6690 0x7ff4580b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:12.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.811+0000 7ff45dc4d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4580b59a0 0x7ff45813c970 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:12.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.811+0000 7ff45dc4d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff4580acae0 con 0x7ff45813ce60 2022-07-20T03:17:12.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff458147100 con 0x7ff4580b6690 2022-07-20T03:17:12.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff4580b5dc0 con 0x7ff4580b59a0 2022-07-20T03:17:12.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff456ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4580b59a0 0x7ff45813c970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:12.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4580b6690 0x7ff4580b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:12.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff4580b6690 0x7ff4580b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:60712/0 (socket says 172.21.15.61:60712) 2022-07-20T03:17:12.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff457fff700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7ff45813ce60 0x7ff4580ae670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:56138/0 (socket says 172.21.15.61:56138) 2022-07-20T03:17:12.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff456ffd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4580b59a0 0x7ff45813c970 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:37704/0 (socket says 172.21.15.61:37704) 2022-07-20T03:17:12.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 -- 172.21.15.61:0/422711034 learned_addr learned my addr 172.21.15.61:0/422711034 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:12.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 -- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4580b59a0 msgr2=0x7ff45813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:12.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 --2- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4580b59a0 0x7ff45813c970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 -- 172.21.15.61:0/422711034 >> v1:172.21.15.117:6789/0 conn(0x7ff45813ce60 legacy=0x7ff4580ae670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:12.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 -- 172.21.15.61:0/422711034 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4580acd10 con 0x7ff4580b6690 2022-07-20T03:17:12.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4577fe700 1 --2- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b6690 0x7ff4580b6a70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff4580add60 tx=0x7ff448006eb0).ready entity=mon.1 client_cookie=207e145768db0ac6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:12.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4567fc700 1 -- 172.21.15.61:0/422711034 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44800d5c0 con 0x7ff4580b6690 2022-07-20T03:17:12.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4567fc700 1 -- 172.21.15.61:0/422711034 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff448000e90 con 0x7ff4580b6690 2022-07-20T03:17:12.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff4567fc700 1 -- 172.21.15.61:0/422711034 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff44800d950 con 0x7ff4580b6690 2022-07-20T03:17:12.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b6690 msgr2=0x7ff4580b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:12.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b6690 0x7ff4580b6a70 secure :-1 s=READY pgs=617 cs=0 l=1 rev1=1 rx=0x7ff4580add60 tx=0x7ff448006eb0).stop 2022-07-20T03:17:12.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- 172.21.15.61:0/422711034 shutdown_connections 2022-07-20T03:17:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff4580b59a0 0x7ff45813c970 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/422711034 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b6690 0x7ff4580b6a70 unknown :-1 s=CLOSED pgs=617 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- 172.21.15.61:0/422711034 >> 172.21.15.61:0/422711034 conn(0x7ff4580b9f70 msgr2=0x7ff4580b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:12.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- 172.21.15.61:0/422711034 shutdown_connections 2022-07-20T03:17:12.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.815+0000 7ff45dc4d700 1 -- 172.21.15.61:0/422711034 wait complete. 2022-07-20T03:17:12.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 Processor -- start 2022-07-20T03:17:12.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 -- start start 2022-07-20T03:17:12.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b59a0 0x7ff4580b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:12.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 0x7ff458142010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:12.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff45813ce60 0x7ff4581529a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:12.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4581427c0 con 0x7ff4580b6690 2022-07-20T03:17:12.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff458156120 con 0x7ff4580b59a0 2022-07-20T03:17:12.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff45dc4d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff458142500 con 0x7ff45813ce60 2022-07-20T03:17:12.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 0x7ff458142010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:12.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 0x7ff458142010 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51040/0 (socket says 172.21.15.61:51040) 2022-07-20T03:17:12.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 -- 172.21.15.61:0/1137640203 learned_addr learned my addr 172.21.15.61:0/1137640203 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:12.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff4577fe700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b59a0 0x7ff4580b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:12.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff457fff700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff45813ce60 0x7ff4581529a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:12.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 -- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff45813ce60 msgr2=0x7ff4581529a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:12.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff45813ce60 0x7ff4581529a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 -- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b59a0 msgr2=0x7ff4580b2760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:12.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b59a0 0x7ff4580b2760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:12.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff456ffd700 1 -- 172.21.15.61:0/1137640203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff44800b1e0 con 0x7ff4580b6690 2022-07-20T03:17:12.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff4577fe700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b59a0 0x7ff4580b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:12.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.819+0000 7ff457fff700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff45813ce60 0x7ff4581529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:12.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff456ffd700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 0x7ff458142010 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7ff44000c000 tx=0x7ff440006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:12.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff440010780 con 0x7ff4580b6690 2022-07-20T03:17:12.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff4400108e0 con 0x7ff4580b6690 2022-07-20T03:17:12.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff458152e90 con 0x7ff4580b6690 2022-07-20T03:17:12.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff440010c40 con 0x7ff4580b6690 2022-07-20T03:17:12.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4581129a0 con 0x7ff4580b6690 2022-07-20T03:17:12.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4581429c0 con 0x7ff4580b6690 2022-07-20T03:17:12.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.823+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff44001c460 con 0x7ff4580b6690 2022-07-20T03:17:12.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.827+0000 7ff454ff9700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff43c053a60 0x7ff43c055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:12.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.827+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(138..138 src has 1..138) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ff44006bfd0 con 0x7ff4580b6690 2022-07-20T03:17:12.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.827+0000 7ff4577fe700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff43c053a60 0x7ff43c055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:12.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.827+0000 7ff4577fe700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff43c053a60 0x7ff43c055f20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7ff44801c000 tx=0x7ff448015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:12.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.835+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff440042f40 con 0x7ff4580b6690 2022-07-20T03:17:13.001 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:12.995+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ff4581129a0 con 0x7ff4580b6690 2022-07-20T03:17:13.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.007+0000 7ff454ff9700 1 -- 172.21.15.61:0/1137640203 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 27 and data pool 28 v76) v1 ==== 189+0+0 (secure 0 0 0) 0x7ff4581129a0 con 0x7ff4580b6690 2022-07-20T03:17:13.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff43c053a60 msgr2=0x7ff43c055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff43c053a60 0x7ff43c055f20 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7ff44801c000 tx=0x7ff448015000).stop 2022-07-20T03:17:13.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 msgr2=0x7ff458142010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 0x7ff458142010 secure :-1 s=READY pgs=1371 cs=0 l=1 rev1=1 rx=0x7ff44000c000 tx=0x7ff440006df0).stop 2022-07-20T03:17:13.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 shutdown_connections 2022-07-20T03:17:13.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff43c053a60 0x7ff43c055f20 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff45813ce60 0x7ff4581529a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff4580b6690 0x7ff458142010 unknown :-1 s=CLOSED pgs=1371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 --2- 172.21.15.61:0/1137640203 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff4580b59a0 0x7ff4580b2760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 >> 172.21.15.61:0/1137640203 conn(0x7ff4580b9f70 msgr2=0x7ff4580b5f60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:13.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 shutdown_connections 2022-07-20T03:17:13.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.011+0000 7ff45dc4d700 1 -- 172.21.15.61:0/1137640203 wait complete. 2022-07-20T03:17:13.019 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 27 and data pool 28 2022-07-20T03:17:13.031 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:13.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 Processor -- start 2022-07-20T03:17:13.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 -- start start 2022-07-20T03:17:13.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fbfe00b1620 0x7fbfe005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbfe0143840 0x7fbfe0141b80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbfe0142ed0 con 0x7fbfe005fdb0 2022-07-20T03:17:13.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00ab0e0 con 0x7fbfe00b1620 2022-07-20T03:17:13.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe84b0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbfe00b1a00 con 0x7fbfe0143840 2022-07-20T03:17:13.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe5a4b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbfe005fdb0 0x7fbfe00aa9d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47656/0 (socket says 172.21.15.61:47656) 2022-07-20T03:17:13.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe5a4b700 1 -- 172.21.15.61:0/2447566964 learned_addr learned my addr 172.21.15.61:0/2447566964 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:13.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe624c700 1 --2- 172.21.15.61:0/2447566964 >> v2:172.21.15.61:3300/0 conn(0x7fbfe00b1620 0x7fbfe005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe6a4d700 1 --2- 172.21.15.61:0/2447566964 >> v2:172.21.15.117:3300/0 conn(0x7fbfe0143840 0x7fbfe0141b80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe524a700 1 -- 172.21.15.61:0/2447566964 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 423486416 0 0) 0x7fbfe0142ed0 con 0x7fbfe005fdb0 2022-07-20T03:17:13.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe524a700 1 -- 172.21.15.61:0/2447566964 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbfc8003260 con 0x7fbfe005fdb0 2022-07-20T03:17:13.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe624c700 1 -- 172.21.15.61:0/2447566964 >> v2:172.21.15.117:3300/0 conn(0x7fbfe0143840 msgr2=0x7fbfe0141b80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:13.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe624c700 1 --2- 172.21.15.61:0/2447566964 >> v2:172.21.15.117:3300/0 conn(0x7fbfe0143840 0x7fbfe0141b80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe624c700 1 -- 172.21.15.61:0/2447566964 >> v1:172.21.15.61:6789/0 conn(0x7fbfe005fdb0 legacy=0x7fbfe00aa9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.123+0000 7fbfe624c700 1 -- 172.21.15.61:0/2447566964 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfe0142c30 con 0x7fbfe00b1620 2022-07-20T03:17:13.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe624c700 1 --2- 172.21.15.61:0/2447566964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe005bd90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7fbfe00604e0 tx=0x7fbfdc006df0).ready entity=mon.0 client_cookie=6abbe2b07f518e3f server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe524a700 1 -- 172.21.15.61:0/2447566964 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfdc00d4f0 con 0x7fbfe00b1620 2022-07-20T03:17:13.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe524a700 1 -- 172.21.15.61:0/2447566964 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfdc000e90 con 0x7fbfe00b1620 2022-07-20T03:17:13.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe524a700 1 -- 172.21.15.61:0/2447566964 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfdc00d840 con 0x7fbfe00b1620 2022-07-20T03:17:13.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 -- 172.21.15.61:0/2447566964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 msgr2=0x7fbfe005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/2447566964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe005bd90 secure :-1 s=READY pgs=1373 cs=0 l=1 rev1=1 rx=0x7fbfe00604e0 tx=0x7fbfdc006df0).stop 2022-07-20T03:17:13.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 -- 172.21.15.61:0/2447566964 shutdown_connections 2022-07-20T03:17:13.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/2447566964 >> v2:172.21.15.117:3300/0 conn(0x7fbfe0143840 0x7fbfe0141b80 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/2447566964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe005bd90 unknown :-1 s=CLOSED pgs=1373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 -- 172.21.15.61:0/2447566964 >> 172.21.15.61:0/2447566964 conn(0x7fbfe00b23d0 msgr2=0x7fbfe00b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:13.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 -- 172.21.15.61:0/2447566964 shutdown_connections 2022-07-20T03:17:13.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 -- 172.21.15.61:0/2447566964 wait complete. 2022-07-20T03:17:13.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 Processor -- start 2022-07-20T03:17:13.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.127+0000 7fbfe84b0700 1 -- start start 2022-07-20T03:17:13.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe005fdb0 0x7fbfe013cc80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe00adb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe0143840 0x7fbfe00ae030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0153160 con 0x7fbfe00b1620 2022-07-20T03:17:13.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0152c70 con 0x7fbfe005fdb0 2022-07-20T03:17:13.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbfe0144720 con 0x7fbfe0143840 2022-07-20T03:17:13.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe00adb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe00adb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51062/0 (socket says 172.21.15.61:51062) 2022-07-20T03:17:13.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 -- 172.21.15.61:0/4066276096 learned_addr learned my addr 172.21.15.61:0/4066276096 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:13.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe624c700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe005fdb0 0x7fbfe013cc80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe6a4d700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe0143840 0x7fbfe00ae030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 -- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe0143840 msgr2=0x7fbfe00ae030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe0143840 0x7fbfe00ae030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 -- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe005fdb0 msgr2=0x7fbfe013cc80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe005fdb0 0x7fbfe013cc80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 -- 172.21.15.61:0/4066276096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbfdc00b180 con 0x7fbfe00b1620 2022-07-20T03:17:13.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe5a4b700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe00adb40 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7fbfd0001110 tx=0x7fbfd0007aa0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd00070d0 con 0x7fbfe00b1620 2022-07-20T03:17:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbfd0007230 con 0x7fbfe00b1620 2022-07-20T03:17:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbfe0144bc0 con 0x7fbfe00b1620 2022-07-20T03:17:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbfd0005610 con 0x7fbfe00b1620 2022-07-20T03:17:13.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbfe0152850 con 0x7fbfe00b1620 2022-07-20T03:17:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.131+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbfe0152ad0 con 0x7fbfe00b1620 2022-07-20T03:17:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.135+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbfd001a060 con 0x7fbfe00b1620 2022-07-20T03:17:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.135+0000 7fbfcf7fe700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc0053a40 0x7fbfc0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.135+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbfd006b190 con 0x7fbfe00b1620 2022-07-20T03:17:13.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.135+0000 7fbfe624c700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc0053a40 0x7fbfc0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.135+0000 7fbfe624c700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc0053a40 0x7fbfc0055f00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fbfdc01a000 tx=0x7fbfdc014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.143+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbfe0152ad0 con 0x7fbfe00b1620 2022-07-20T03:17:13.271 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.267+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbfe0152ad0 con 0x7fbfe00b1620 2022-07-20T03:17:13.273 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.267+0000 7fbfcf7fe700 1 -- 172.21.15.61:0/4066276096 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fbfe0152ad0 con 0x7fbfe00b1620 2022-07-20T03:17:13.273 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:13.274 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":139,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:13.001605+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:10.948609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:12.001181+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:13.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc0053a40 msgr2=0x7fbfc0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc0053a40 0x7fbfc0055f00 secure :-1 s=READY pgs=662 cs=0 l=1 rev1=1 rx=0x7fbfdc01a000 tx=0x7fbfdc014000).stop 2022-07-20T03:17:13.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 msgr2=0x7fbfe00adb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe00adb40 secure :-1 s=READY pgs=1374 cs=0 l=1 rev1=1 rx=0x7fbfd0001110 tx=0x7fbfd0007aa0).stop 2022-07-20T03:17:13.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 shutdown_connections 2022-07-20T03:17:13.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbfc0053a40 0x7fbfc0055f00 unknown :-1 s=CLOSED pgs=662 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbfe0143840 0x7fbfe00ae030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbfe00b1620 0x7fbfe00adb40 unknown :-1 s=CLOSED pgs=1374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 --2- 172.21.15.61:0/4066276096 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbfe005fdb0 0x7fbfe013cc80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 >> 172.21.15.61:0/4066276096 conn(0x7fbfe00b23d0 msgr2=0x7fbfe00b36c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:13.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 shutdown_connections 2022-07-20T03:17:13.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.271+0000 7fbfe84b0700 1 -- 172.21.15.61:0/4066276096 wait complete. 2022-07-20T03:17:13.287 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:13.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 Processor -- start 2022-07-20T03:17:13.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- start start 2022-07-20T03:17:13.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2380b1e00 0x7ff23814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff23805c3f0 0x7ff2380ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff23805d480 con 0x7ff23805bd90 2022-07-20T03:17:13.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff2380b21e0 con 0x7ff2380b1e00 2022-07-20T03:17:13.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff23805cf40 con 0x7ff23805c3f0 2022-07-20T03:17:13.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2380b1e00 0x7ff23814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff2380b1e00 0x7ff23814a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51070/0 (socket says 172.21.15.61:51070) 2022-07-20T03:17:13.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 -- 172.21.15.61:0/1757118407 learned_addr learned my addr 172.21.15.61:0/1757118407 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:13.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23db4b700 1 --2- 172.21.15.61:0/1757118407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff23805c3f0 0x7ff2380ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 -- 172.21.15.61:0/1757118407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff23805c3f0 msgr2=0x7ff2380ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:13.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 --2- 172.21.15.61:0/1757118407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff23805c3f0 0x7ff2380ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 -- 172.21.15.61:0/1757118407 >> v1:172.21.15.61:6789/0 conn(0x7ff23805bd90 legacy=0x7ff238146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 -- 172.21.15.61:0/1757118407 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff23805d1e0 con 0x7ff2380b1e00 2022-07-20T03:17:13.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23e34c700 1 --2- 172.21.15.61:0/1757118407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff23814a6d0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7ff228002f70 tx=0x7ff228006eb0).ready entity=mon.0 client_cookie=6ed9ef58107af163 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23d34a700 1 -- 172.21.15.61:0/1757118407 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22800d5c0 con 0x7ff2380b1e00 2022-07-20T03:17:13.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23d34a700 1 -- 172.21.15.61:0/1757118407 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff228000e90 con 0x7ff2380b1e00 2022-07-20T03:17:13.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff23d34a700 1 -- 172.21.15.61:0/1757118407 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff22800d950 con 0x7ff2380b1e00 2022-07-20T03:17:13.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- 172.21.15.61:0/1757118407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 msgr2=0x7ff23814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 --2- 172.21.15.61:0/1757118407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff23814a6d0 secure :-1 s=READY pgs=1376 cs=0 l=1 rev1=1 rx=0x7ff228002f70 tx=0x7ff228006eb0).stop 2022-07-20T03:17:13.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- 172.21.15.61:0/1757118407 shutdown_connections 2022-07-20T03:17:13.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 --2- 172.21.15.61:0/1757118407 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff23805c3f0 0x7ff2380ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 --2- 172.21.15.61:0/1757118407 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff23814a6d0 unknown :-1 s=CLOSED pgs=1376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- 172.21.15.61:0/1757118407 >> 172.21.15.61:0/1757118407 conn(0x7ff2380611c0 msgr2=0x7ff238062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:13.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- 172.21.15.61:0/1757118407 shutdown_connections 2022-07-20T03:17:13.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.387+0000 7ff2405b0700 1 -- 172.21.15.61:0/1757118407 wait complete. 2022-07-20T03:17:13.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 Processor -- start 2022-07-20T03:17:13.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 -- start start 2022-07-20T03:17:13.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff23805c3f0 0x7ff2380b3a90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff238156a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2380a0530 0x7ff238156f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380afb30 con 0x7ff2380b1e00 2022-07-20T03:17:13.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380a1750 con 0x7ff23805c3f0 2022-07-20T03:17:13.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff2380a19f0 con 0x7ff2380a0530 2022-07-20T03:17:13.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff238156a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff238156a10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51076/0 (socket says 172.21.15.61:51076) 2022-07-20T03:17:13.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 -- 172.21.15.61:0/3746070454 learned_addr learned my addr 172.21.15.61:0/3746070454 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:13.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23eb4d700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2380a0530 0x7ff238156f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23e34c700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff23805c3f0 0x7ff2380b3a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 -- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2380a0530 msgr2=0x7ff238156f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2380a0530 0x7ff238156f00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 -- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff23805c3f0 msgr2=0x7ff2380b3a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff23805c3f0 0x7ff2380b3a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 -- 172.21.15.61:0/3746070454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff22800b1e0 con 0x7ff2380b1e00 2022-07-20T03:17:13.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23e34c700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff23805c3f0 0x7ff2380b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:13.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23eb4d700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2380a0530 0x7ff238156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:13.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff23db4b700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff238156a10 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7ff23400a000 tx=0x7ff234008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff23400c630 con 0x7ff2380b1e00 2022-07-20T03:17:13.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff23400c790 con 0x7ff2380b1e00 2022-07-20T03:17:13.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2380a1c90 con 0x7ff2380b1e00 2022-07-20T03:17:13.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff23400caf0 con 0x7ff2380b1e00 2022-07-20T03:17:13.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.391+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff238156540 con 0x7ff2380b1e00 2022-07-20T03:17:13.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.395+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff23400cc50 con 0x7ff2380b1e00 2022-07-20T03:17:13.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.395+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2380a1e20 con 0x7ff2380b1e00 2022-07-20T03:17:13.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.395+0000 7ff2277fe700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff218053a40 0x7ff218055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.395+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff23406c0c0 con 0x7ff2380b1e00 2022-07-20T03:17:13.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.395+0000 7ff23e34c700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff218053a40 0x7ff218055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.395+0000 7ff23e34c700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff218053a40 0x7ff218055f00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7ff22801a000 tx=0x7ff228015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.403+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff234043020 con 0x7ff2380b1e00 2022-07-20T03:17:13.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.523+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7ff2380574e0 con 0x7ff2380b1e00 2022-07-20T03:17:13.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.523+0000 7ff2277fe700 1 -- 172.21.15.61:0/3746070454 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v139) v1 ==== 74+0+16113 (secure 0 0 0) 0x7ff234046640 con 0x7ff2380b1e00 2022-07-20T03:17:13.530 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:13.530 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":139,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:13.001605+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:10.948609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:12.001181+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:13.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff218053a40 msgr2=0x7ff218055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff218053a40 0x7ff218055f00 secure :-1 s=READY pgs=663 cs=0 l=1 rev1=1 rx=0x7ff22801a000 tx=0x7ff228015000).stop 2022-07-20T03:17:13.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 msgr2=0x7ff238156a10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff238156a10 secure :-1 s=READY pgs=1377 cs=0 l=1 rev1=1 rx=0x7ff23400a000 tx=0x7ff234008040).stop 2022-07-20T03:17:13.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 shutdown_connections 2022-07-20T03:17:13.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff218053a40 0x7ff218055f00 unknown :-1 s=CLOSED pgs=663 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff2380a0530 0x7ff238156f00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff2380b1e00 0x7ff238156a10 unknown :-1 s=CLOSED pgs=1377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 --2- 172.21.15.61:0/3746070454 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff23805c3f0 0x7ff2380b3a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 >> 172.21.15.61:0/3746070454 conn(0x7ff23805bd90 msgr2=0x7ff2380aeba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:13.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 shutdown_connections 2022-07-20T03:17:13.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.527+0000 7ff2405b0700 1 -- 172.21.15.61:0/3746070454 wait complete. 2022-07-20T03:17:13.546 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:17:13.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cb469700 1 Processor -- start 2022-07-20T03:17:13.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cb469700 1 -- start start 2022-07-20T03:17:13.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cb469700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f59c414b590 0x7f59c414d980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cb469700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c40ba180 con 0x7f59c40b3cf0 2022-07-20T03:17:13.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cb469700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f59c41461b0 con 0x7f59c40b9da0 2022-07-20T03:17:13.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cb469700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f59c40ad5b0 con 0x7f59c414b590 2022-07-20T03:17:13.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59c9c66700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f59c40b3cf0 0x7f59c4142a50 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47684/0 (socket says 172.21.15.61:47684) 2022-07-20T03:17:13.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59cac68700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f59c414b590 0x7f59c414d980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.655+0000 7f59c9c66700 1 -- 172.21.15.61:0/2311161273 learned_addr learned my addr 172.21.15.61:0/2311161273 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:13.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1475653829 0 0) 0x7f59c40ba180 con 0x7f59c40b3cf0 2022-07-20T03:17:13.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59ac003260 con 0x7f59c40b3cf0 2022-07-20T03:17:13.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4147921731 0 0) 0x7f59c41461b0 con 0x7f59c40b9da0 2022-07-20T03:17:13.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f59c40ba180 con 0x7f59c40b9da0 2022-07-20T03:17:13.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cac68700 1 -- 172.21.15.61:0/2311161273 >> v1:172.21.15.117:6789/0 conn(0x7f59c40b9da0 legacy=0x7f59c41468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cac68700 1 -- 172.21.15.61:0/2311161273 >> v1:172.21.15.61:6789/0 conn(0x7f59c40b3cf0 legacy=0x7f59c4142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cac68700 1 -- 172.21.15.61:0/2311161273 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c4146450 con 0x7f59c414b590 2022-07-20T03:17:13.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cac68700 1 --2- 172.21.15.61:0/2311161273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c414b590 0x7f59c414d980 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f59c40ad3d0 tx=0x7f59bc0091d0).ready entity=mon.0 client_cookie=37370e8f225e1e45 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59bc00b600 con 0x7f59c414b590 2022-07-20T03:17:13.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59bc00b760 con 0x7f59c414b590 2022-07-20T03:17:13.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59c9465700 1 -- 172.21.15.61:0/2311161273 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59bc00b8d0 con 0x7f59c414b590 2022-07-20T03:17:13.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 -- 172.21.15.61:0/2311161273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c414b590 msgr2=0x7f59c414d980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 --2- 172.21.15.61:0/2311161273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c414b590 0x7f59c414d980 secure :-1 s=READY pgs=1379 cs=0 l=1 rev1=1 rx=0x7f59c40ad3d0 tx=0x7f59bc0091d0).stop 2022-07-20T03:17:13.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 -- 172.21.15.61:0/2311161273 shutdown_connections 2022-07-20T03:17:13.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 --2- 172.21.15.61:0/2311161273 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c414b590 0x7f59c414d980 unknown :-1 s=CLOSED pgs=1379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 -- 172.21.15.61:0/2311161273 >> 172.21.15.61:0/2311161273 conn(0x7f59c40678c0 msgr2=0x7f59c40b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:13.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 -- 172.21.15.61:0/2311161273 shutdown_connections 2022-07-20T03:17:13.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 -- 172.21.15.61:0/2311161273 wait complete. 2022-07-20T03:17:13.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 Processor -- start 2022-07-20T03:17:13.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.659+0000 7f59cb469700 1 -- start start 2022-07-20T03:17:13.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 0x7f59c4145660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f59c40b9da0 0x7f59c4152040 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59c414b590 0x7f59c413d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4148b40 con 0x7f59c40b3cf0 2022-07-20T03:17:13.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59c4152530 con 0x7f59c414b590 2022-07-20T03:17:13.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f59c41488a0 con 0x7f59c40b9da0 2022-07-20T03:17:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 0x7f59c4145660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 0x7f59c4145660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51090/0 (socket says 172.21.15.61:51090) 2022-07-20T03:17:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 -- 172.21.15.61:0/2215989519 learned_addr learned my addr 172.21.15.61:0/2215989519 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:13.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cac68700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59c414b590 0x7f59c413d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59c9c66700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f59c40b9da0 0x7f59c4152040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 -- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f59c40b9da0 msgr2=0x7f59c4152040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f59c40b9da0 0x7f59c4152040 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 -- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59c414b590 msgr2=0x7f59c413d0c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:13.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59c414b590 0x7f59c413d0c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:13.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 -- 172.21.15.61:0/2215989519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59bc009470 con 0x7f59c40b3cf0 2022-07-20T03:17:13.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59ca467700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 0x7f59c4145660 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f59b8001110 tx=0x7f59b8007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b8012400 con 0x7f59c40b3cf0 2022-07-20T03:17:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f59b8012560 con 0x7f59c40b3cf0 2022-07-20T03:17:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59c41556f0 con 0x7f59c40b3cf0 2022-07-20T03:17:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f59b80128c0 con 0x7f59c40b3cf0 2022-07-20T03:17:13.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59c414b0e0 con 0x7f59c40b3cf0 2022-07-20T03:17:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f59b8019030 con 0x7f59c40b3cf0 2022-07-20T03:17:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59c41486f0 con 0x7f59c40b3cf0 2022-07-20T03:17:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.663+0000 7f59b77fe700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f59a4053ad0 0x7f59a4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:13.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.667+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(139..139 src has 1..139) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f59b806b2f0 con 0x7f59c40b3cf0 2022-07-20T03:17:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.667+0000 7f59c9c66700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f59a4053ad0 0x7f59a4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.667+0000 7f59c9c66700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f59a4053ad0 0x7f59a4055f90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f59c000d000 tx=0x7f59c0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:13.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.671+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f59b80421b0 con 0x7f59c40b3cf0 2022-07-20T03:17:13.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:13.835+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7f59c40574e0 con 0x7f59c40b3cf0 2022-07-20T03:17:14.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.015+0000 7f59b77fe700 1 -- 172.21.15.61:0/2215989519 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v77) v1 ==== 118+0+0 (secure 0 0 0) 0x7f59b80457d0 con 0x7f59c40b3cf0 2022-07-20T03:17:14.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f59a4053ad0 msgr2=0x7f59a4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f59a4053ad0 0x7f59a4055f90 secure :-1 s=READY pgs=664 cs=0 l=1 rev1=1 rx=0x7f59c000d000 tx=0x7f59c0009040).stop 2022-07-20T03:17:14.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 msgr2=0x7f59c4145660 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 0x7f59c4145660 secure :-1 s=READY pgs=1380 cs=0 l=1 rev1=1 rx=0x7f59b8001110 tx=0x7f59b8007ae0).stop 2022-07-20T03:17:14.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 shutdown_connections 2022-07-20T03:17:14.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f59a4053ad0 0x7f59a4055f90 unknown :-1 s=CLOSED pgs=664 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f59c414b590 0x7f59c413d0c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f59c40b9da0 0x7f59c4152040 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 --2- 172.21.15.61:0/2215989519 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f59c40b3cf0 0x7f59c4145660 unknown :-1 s=CLOSED pgs=1380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 >> 172.21.15.61:0/2215989519 conn(0x7f59c40678c0 msgr2=0x7f59c405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 shutdown_connections 2022-07-20T03:17:14.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.019+0000 7f59cb469700 1 -- 172.21.15.61:0/2215989519 wait complete. 2022-07-20T03:17:14.041 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:14.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 Processor -- start 2022-07-20T03:17:14.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- start start 2022-07-20T03:17:14.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f25380b1620 0x7f2538142ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2538142fd0 0x7f253813e730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f253805bd90 0x7f253813ec20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2538060b50 con 0x7f2538142fd0 2022-07-20T03:17:14.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25381426b0 con 0x7f25380b1620 2022-07-20T03:17:14.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f25380b1a00 con 0x7f253805bd90 2022-07-20T03:17:14.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2538142fd0 0x7f253813e730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f2538142fd0 0x7f253813e730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51100/0 (socket says 172.21.15.61:51100) 2022-07-20T03:17:14.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 -- 172.21.15.61:0/2911190135 learned_addr learned my addr 172.21.15.61:0/2911190135 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:14.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253de58700 1 --2- 172.21.15.61:0/2911190135 >> v2:172.21.15.117:3300/0 conn(0x7f25380b1620 0x7f2538142ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253e659700 1 --2- 172.21.15.61:0/2911190135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f253805bd90 0x7f253813ec20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 -- 172.21.15.61:0/2911190135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f253805bd90 msgr2=0x7f253813ec20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:14.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 --2- 172.21.15.61:0/2911190135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f253805bd90 0x7f253813ec20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 -- 172.21.15.61:0/2911190135 >> v2:172.21.15.117:3300/0 conn(0x7f25380b1620 msgr2=0x7f2538142ae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:14.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 --2- 172.21.15.61:0/2911190135 >> v2:172.21.15.117:3300/0 conn(0x7f25380b1620 0x7f2538142ae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 -- 172.21.15.61:0/2911190135 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f253805f840 con 0x7f2538142fd0 2022-07-20T03:17:14.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253d657700 1 --2- 172.21.15.61:0/2911190135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2538142fd0 0x7f253813e730 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f2538060c90 tx=0x7f2528006eb0).ready entity=mon.0 client_cookie=14d3b39a5a1a512d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253ce56700 1 -- 172.21.15.61:0/2911190135 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252800e610 con 0x7f2538142fd0 2022-07-20T03:17:14.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253ce56700 1 -- 172.21.15.61:0/2911190135 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f2528000e90 con 0x7f2538142fd0 2022-07-20T03:17:14.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f253ce56700 1 -- 172.21.15.61:0/2911190135 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f252800e9a0 con 0x7f2538142fd0 2022-07-20T03:17:14.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- 172.21.15.61:0/2911190135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2538142fd0 msgr2=0x7f253813e730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- 172.21.15.61:0/2911190135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2538142fd0 0x7f253813e730 secure :-1 s=READY pgs=1381 cs=0 l=1 rev1=1 rx=0x7f2538060c90 tx=0x7f2528006eb0).stop 2022-07-20T03:17:14.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- 172.21.15.61:0/2911190135 shutdown_connections 2022-07-20T03:17:14.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- 172.21.15.61:0/2911190135 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f253805bd90 0x7f253813ec20 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- 172.21.15.61:0/2911190135 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f2538142fd0 0x7f253813e730 unknown :-1 s=CLOSED pgs=1381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 --2- 172.21.15.61:0/2911190135 >> v2:172.21.15.117:3300/0 conn(0x7f25380b1620 0x7f2538142ae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- 172.21.15.61:0/2911190135 >> 172.21.15.61:0/2911190135 conn(0x7f25380b23d0 msgr2=0x7f25380b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- 172.21.15.61:0/2911190135 shutdown_connections 2022-07-20T03:17:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.147+0000 7f25400bc700 1 -- 172.21.15.61:0/2911190135 wait complete. 2022-07-20T03:17:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 Processor -- start 2022-07-20T03:17:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 -- start start 2022-07-20T03:17:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253805bd90 0x7f253812e790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 0x7f253812c6c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2538142fd0 0x7f253812cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25380ad9a0 con 0x7f25380b1620 2022-07-20T03:17:14.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2538063420 con 0x7f253805bd90 2022-07-20T03:17:14.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f2538064fb0 con 0x7f2538142fd0 2022-07-20T03:17:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 0x7f253812c6c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 0x7f253812c6c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51104/0 (socket says 172.21.15.61:51104) 2022-07-20T03:17:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 -- 172.21.15.61:0/1371039521 learned_addr learned my addr 172.21.15.61:0/1371039521 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:14.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253e659700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2538142fd0 0x7f253812cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253de58700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253805bd90 0x7f253812e790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 -- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2538142fd0 msgr2=0x7f253812cbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2538142fd0 0x7f253812cbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 -- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253805bd90 msgr2=0x7f253812e790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253805bd90 0x7f253812e790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 -- 172.21.15.61:0/1371039521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f252800b1e0 con 0x7f25380b1620 2022-07-20T03:17:14.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253e659700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2538142fd0 0x7f253812cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:14.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f253d657700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 0x7f253812c6c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f2538060c90 tx=0x7f2528015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2528021070 con 0x7f25380b1620 2022-07-20T03:17:14.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25280116e0 con 0x7f25380b1620 2022-07-20T03:17:14.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2538064d10 con 0x7f25380b1620 2022-07-20T03:17:14.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2528011a40 con 0x7f25380b1620 2022-07-20T03:17:14.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.151+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f253812ec80 con 0x7f25380b1620 2022-07-20T03:17:14.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.155+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f253805fae0 con 0x7f25380b1620 2022-07-20T03:17:14.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.155+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f252800f070 con 0x7f25380b1620 2022-07-20T03:17:14.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.155+0000 7f2526ffd700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2518053920 0x7f2518055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.155+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f252806d200 con 0x7f25380b1620 2022-07-20T03:17:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.155+0000 7f253de58700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2518053920 0x7f2518055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.155+0000 7f253de58700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2518053920 0x7f2518055de0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f253400b000 tx=0x7f2534006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.163+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f2528044170 con 0x7f25380b1620 2022-07-20T03:17:14.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f25380574e0 con 0x7f25380b1620 2022-07-20T03:17:14.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f2526ffd700 1 -- 172.21.15.61:0/1371039521 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f2528047790 con 0x7f25380b1620 2022-07-20T03:17:14.337 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:14.337 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:13.001769+0000","modified":"2022-07-20T03:17:14.005556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8466":{"gid":8466,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3390219350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3390219350},{"type":"v1","addr":"172.21.15.61:6833","nonce":3390219350}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T03:17:14.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2518053920 msgr2=0x7f2518055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2518053920 0x7f2518055de0 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f253400b000 tx=0x7f2534006df0).stop 2022-07-20T03:17:14.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 msgr2=0x7f253812c6c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 0x7f253812c6c0 secure :-1 s=READY pgs=1382 cs=0 l=1 rev1=1 rx=0x7f2538060c90 tx=0x7f2528015000).stop 2022-07-20T03:17:14.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 shutdown_connections 2022-07-20T03:17:14.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f2518053920 0x7f2518055de0 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f2538142fd0 0x7f253812cbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25380b1620 0x7f253812c6c0 unknown :-1 s=CLOSED pgs=1382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 --2- 172.21.15.61:0/1371039521 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f253805bd90 0x7f253812e790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 >> 172.21.15.61:0/1371039521 conn(0x7f25380b23d0 msgr2=0x7f25380b42c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 shutdown_connections 2022-07-20T03:17:14.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.331+0000 7f25400bc700 1 -- 172.21.15.61:0/1371039521 wait complete. 2022-07-20T03:17:14.345 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-20T03:17:14.354 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:14.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbddbfff700 1 Processor -- start 2022-07-20T03:17:14.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbddbfff700 1 -- start start 2022-07-20T03:17:14.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbddbfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fbddc066160 0x7fbddc13df20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbddbfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc0b7f50 con 0x7fbddc0b31e0 2022-07-20T03:17:14.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbddbfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fbddc0b2800 con 0x7fbddc0b7b70 2022-07-20T03:17:14.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbddbfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fbddc0b2aa0 con 0x7fbddc066160 2022-07-20T03:17:14.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbdda7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fbddc0b31e0 0x7fbddc143d90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47710/0 (socket says 172.21.15.61:47710) 2022-07-20T03:17:14.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.455+0000 7fbdda7fc700 1 -- 172.21.15.61:0/199200235 learned_addr learned my addr 172.21.15.61:0/199200235 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:14.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 --2- 172.21.15.61:0/199200235 >> v2:172.21.15.117:3300/0 conn(0x7fbddc066160 0x7fbddc13df20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 174320390 0 0) 0x7fbddc0b7f50 con 0x7fbddc0b31e0 2022-07-20T03:17:14.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbdc0003260 con 0x7fbddc0b31e0 2022-07-20T03:17:14.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1710941813 0 0) 0x7fbddc0b2800 con 0x7fbddc0b7b70 2022-07-20T03:17:14.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fbddc0b7f50 con 0x7fbddc0b7b70 2022-07-20T03:17:14.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 50435546 0 0) 0x7fbdc0003260 con 0x7fbddc0b31e0 2022-07-20T03:17:14.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 >> v2:172.21.15.117:3300/0 conn(0x7fbddc066160 msgr2=0x7fbddc13df20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:14.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 --2- 172.21.15.61:0/199200235 >> v2:172.21.15.117:3300/0 conn(0x7fbddc066160 0x7fbddc13df20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 >> v1:172.21.15.117:6789/0 conn(0x7fbddc0b7b70 legacy=0x7fbddc13d810 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc0b2d40 con 0x7fbddc0b31e0 2022-07-20T03:17:14.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7fbdc4001660 con 0x7fbddc0b31e0 2022-07-20T03:17:14.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 >> v1:172.21.15.61:6789/0 conn(0x7fbddc0b31e0 legacy=0x7fbddc143d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdc00038e0 0x7fbdc0003ce0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0004500 0x7fbdc0004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbdc000fb60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0b2aa0 con 0x7fbdc0004e70 2022-07-20T03:17:14.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc0010100 con 0x7fbdc0004500 2022-07-20T03:17:14.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbdc00102c0 con 0x7fbdc00038e0 2022-07-20T03:17:14.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbdc000fb60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddaffd700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdc00038e0 0x7fbdc0003ce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdda7fc700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0004500 0x7fbdc0004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 -- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdc00038e0 msgr2=0x7fbdc0003ce0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:14.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbdc00038e0 0x7fbdc0003ce0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 -- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0004500 msgr2=0x7fbdc0004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0004500 0x7fbdc0004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 -- 172.21.15.61:0/199200235 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdc0003260 con 0x7fbdc0004e70 2022-07-20T03:17:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdda7fc700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbdc0004500 0x7fbdc0004980 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:14.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 -- 172.21.15.61:0/199200235 reap_dead start 2022-07-20T03:17:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddb7fe700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbdc000fb60 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7fbdd400c000 tx=0x7fbdd40091d0).ready entity=mon.0 client_cookie=593b9017c5f55111 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdd400e6e0 con 0x7fbdc0004e70 2022-07-20T03:17:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdd400e840 con 0x7fbdc0004e70 2022-07-20T03:17:14.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbdd9ffb700 1 -- 172.21.15.61:0/199200235 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdd400eba0 con 0x7fbdc0004e70 2022-07-20T03:17:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 -- 172.21.15.61:0/199200235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 msgr2=0x7fbdc000fb60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbdc000fb60 secure :-1 s=READY pgs=1384 cs=0 l=1 rev1=1 rx=0x7fbdd400c000 tx=0x7fbdd40091d0).stop 2022-07-20T03:17:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 -- 172.21.15.61:0/199200235 shutdown_connections 2022-07-20T03:17:14.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 --2- 172.21.15.61:0/199200235 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbdc000fb60 unknown :-1 s=CLOSED pgs=1384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 -- 172.21.15.61:0/199200235 >> 172.21.15.61:0/199200235 conn(0x7fbddc0656b0 msgr2=0x7fbddc0b7120 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 -- 172.21.15.61:0/199200235 shutdown_connections 2022-07-20T03:17:14.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.459+0000 7fbddbfff700 1 -- 172.21.15.61:0/199200235 wait complete. 2022-07-20T03:17:14.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 Processor -- start 2022-07-20T03:17:14.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 -- start start 2022-07-20T03:17:14.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbddc0b5290 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbddc0b0830 0x7fbddc13a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbddc144820 0x7fbddc13ae90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0b0210 con 0x7fbdc0004e70 2022-07-20T03:17:14.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc143110 con 0x7fbddc0b0830 2022-07-20T03:17:14.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fbddc0aff70 con 0x7fbddc144820 2022-07-20T03:17:14.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbddc0b5290 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbddc0b5290 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51124/0 (socket says 172.21.15.61:51124) 2022-07-20T03:17:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 -- 172.21.15.61:0/2035929258 learned_addr learned my addr 172.21.15.61:0/2035929258 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddb7fe700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbddc144820 0x7fbddc13ae90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbdda7fc700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbddc0b0830 0x7fbddc13a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 -- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbddc144820 msgr2=0x7fbddc13ae90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbddc144820 0x7fbddc13ae90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 -- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbddc0b0830 msgr2=0x7fbddc13a9a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbddc0b0830 0x7fbddc13a9a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 -- 172.21.15.61:0/2035929258 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbddc143c20 con 0x7fbdc0004e70 2022-07-20T03:17:14.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbdda7fc700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbddc0b0830 0x7fbddc13a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:14.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddaffd700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbddc0b5290 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7fbdcc00d000 tx=0x7fbdcc007a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdcc011780 con 0x7fbdc0004e70 2022-07-20T03:17:14.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fbdcc0118e0 con 0x7fbdc0004e70 2022-07-20T03:17:14.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbddc1440c0 con 0x7fbdc0004e70 2022-07-20T03:17:14.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fbdcc011c40 con 0x7fbdc0004e70 2022-07-20T03:17:14.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.463+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbddc151000 con 0x7fbdc0004e70 2022-07-20T03:17:14.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.467+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbddc143db0 con 0x7fbdc0004e70 2022-07-20T03:17:14.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.467+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fbdcc01b030 con 0x7fbdc0004e70 2022-07-20T03:17:14.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.467+0000 7fbdd3fff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053af0 0x7fbdb4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.467+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fbdcc06bf20 con 0x7fbdc0004e70 2022-07-20T03:17:14.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.467+0000 7fbdda7fc700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053af0 0x7fbdb4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.467+0000 7fbdda7fc700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053af0 0x7fbdb4055fb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7fbdc400d000 tx=0x7fbdc4009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.475+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fbdcc042e80 con 0x7fbdc0004e70 2022-07-20T03:17:14.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.599+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fbddc0b1ac0 con 0x7fbdc0004e70 2022-07-20T03:17:14.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbdd3fff700 1 -- 172.21.15.61:0/2035929258 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fbdcc019070 con 0x7fbdc0004e70 2022-07-20T03:17:14.611 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:14.611 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":140,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:14.005088+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:10.948609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:12.001181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:14.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053af0 msgr2=0x7fbdb4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053af0 0x7fbdb4055fb0 secure :-1 s=READY pgs=666 cs=0 l=1 rev1=1 rx=0x7fbdc400d000 tx=0x7fbdc4009040).stop 2022-07-20T03:17:14.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 msgr2=0x7fbddc0b5290 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbddc0b5290 secure :-1 s=READY pgs=1385 cs=0 l=1 rev1=1 rx=0x7fbdcc00d000 tx=0x7fbdcc007a00).stop 2022-07-20T03:17:14.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 shutdown_connections 2022-07-20T03:17:14.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fbdb4053af0 0x7fbdb4055fb0 unknown :-1 s=CLOSED pgs=666 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fbddc0b0830 0x7fbddc13a9a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fbddc144820 0x7fbddc13ae90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 --2- 172.21.15.61:0/2035929258 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fbdc0004e70 0x7fbddc0b5290 unknown :-1 s=CLOSED pgs=1385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 >> 172.21.15.61:0/2035929258 conn(0x7fbddc0656b0 msgr2=0x7fbddc05c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 shutdown_connections 2022-07-20T03:17:14.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.603+0000 7fbddbfff700 1 -- 172.21.15.61:0/2035929258 wait complete. 2022-07-20T03:17:14.624 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:17:14.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 Processor -- start 2022-07-20T03:17:14.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 -- start start 2022-07-20T03:17:14.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fcf1413f6c0 0x7fcf140b36e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fcf14060220 0x7fcf140618e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fcf140ac8a0 con 0x7fcf140b3300 2022-07-20T03:17:14.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf140acb40 con 0x7fcf14060220 2022-07-20T03:17:14.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf1a8f1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fcf140acde0 con 0x7fcf1413f6c0 2022-07-20T03:17:14.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf13fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fcf140b3300 0x7fcf1413efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47730/0 (socket says 172.21.15.61:47730) 2022-07-20T03:17:14.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf13fff700 1 -- 172.21.15.61:0/2043601543 learned_addr learned my addr 172.21.15.61:0/2043601543 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:14.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf18e8e700 1 --2- 172.21.15.61:0/2043601543 >> v2:172.21.15.61:3300/0 conn(0x7fcf14060220 0x7fcf140618e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf0b7fe700 1 --2- 172.21.15.61:0/2043601543 >> v2:172.21.15.117:3300/0 conn(0x7fcf1413f6c0 0x7fcf140b36e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf137fe700 1 -- 172.21.15.61:0/2043601543 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 332371163 0 0) 0x7fcf140ac8a0 con 0x7fcf140b3300 2022-07-20T03:17:14.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf137fe700 1 -- 172.21.15.61:0/2043601543 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fcef4003260 con 0x7fcf140b3300 2022-07-20T03:17:14.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf18e8e700 1 -- 172.21.15.61:0/2043601543 >> v2:172.21.15.117:3300/0 conn(0x7fcf1413f6c0 msgr2=0x7fcf140b36e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:14.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf18e8e700 1 --2- 172.21.15.61:0/2043601543 >> v2:172.21.15.117:3300/0 conn(0x7fcf1413f6c0 0x7fcf140b36e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf18e8e700 1 -- 172.21.15.61:0/2043601543 >> v1:172.21.15.61:6789/0 conn(0x7fcf140b3300 legacy=0x7fcf1413efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf18e8e700 1 -- 172.21.15.61:0/2043601543 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf140ac590 con 0x7fcf14060220 2022-07-20T03:17:14.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf18e8e700 1 --2- 172.21.15.61:0/2043601543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf14060220 0x7fcf140618e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7fcf0c0032f0 tx=0x7fcf0c0091d0).ready entity=mon.0 client_cookie=d6c3350c789627e5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.735+0000 7fcf137fe700 1 -- 172.21.15.61:0/2043601543 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf0c00b600 con 0x7fcf14060220 2022-07-20T03:17:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf137fe700 1 -- 172.21.15.61:0/2043601543 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcf0c00b760 con 0x7fcf14060220 2022-07-20T03:17:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf137fe700 1 -- 172.21.15.61:0/2043601543 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcf0c00b8d0 con 0x7fcf14060220 2022-07-20T03:17:14.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/2043601543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf14060220 msgr2=0x7fcf140618e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/2043601543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf14060220 0x7fcf140618e0 secure :-1 s=READY pgs=1387 cs=0 l=1 rev1=1 rx=0x7fcf0c0032f0 tx=0x7fcf0c0091d0).stop 2022-07-20T03:17:14.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/2043601543 shutdown_connections 2022-07-20T03:17:14.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/2043601543 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf14060220 0x7fcf140618e0 unknown :-1 s=CLOSED pgs=1387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/2043601543 >> v2:172.21.15.117:3300/0 conn(0x7fcf1413f6c0 0x7fcf140b36e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/2043601543 >> 172.21.15.61:0/2043601543 conn(0x7fcf140b6c20 msgr2=0x7fcf140b2bf0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/2043601543 shutdown_connections 2022-07-20T03:17:14.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/2043601543 wait complete. 2022-07-20T03:17:14.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 Processor -- start 2022-07-20T03:17:14.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- start start 2022-07-20T03:17:14.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf14060220 0x7fcf1413cb80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 0x7fcf140ad960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf1413f6c0 0x7fcf140ade50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1409b1f0 con 0x7fcf140b3300 2022-07-20T03:17:14.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1409af50 con 0x7fcf1413f6c0 2022-07-20T03:17:14.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf1a8f1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fcf1409bb20 con 0x7fcf14060220 2022-07-20T03:17:14.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 0x7fcf140ad960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 0x7fcf140ad960 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51140/0 (socket says 172.21.15.61:51140) 2022-07-20T03:17:14.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.739+0000 7fcf0b7fe700 1 -- 172.21.15.61:0/150251025 learned_addr learned my addr 172.21.15.61:0/150251025 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:14.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf18e8e700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf1413f6c0 0x7fcf140ade50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf13fff700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf14060220 0x7fcf1413cb80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf0b7fe700 1 -- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf14060220 msgr2=0x7fcf1413cb80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf0b7fe700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf14060220 0x7fcf1413cb80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf0b7fe700 1 -- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf1413f6c0 msgr2=0x7fcf140ade50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf0b7fe700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf1413f6c0 0x7fcf140ade50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf0b7fe700 1 -- 172.21.15.61:0/150251025 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf0c009470 con 0x7fcf140b3300 2022-07-20T03:17:14.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf0b7fe700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 0x7fcf140ad960 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7fcefc00c000 tx=0x7fcefc006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcefc011750 con 0x7fcf140b3300 2022-07-20T03:17:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcefc0118b0 con 0x7fcf140b3300 2022-07-20T03:17:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf1409b690 con 0x7fcf140b3300 2022-07-20T03:17:14.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcefc011c10 con 0x7fcf140b3300 2022-07-20T03:17:14.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf140ad730 con 0x7fcf140b3300 2022-07-20T03:17:14.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf14067ec0 con 0x7fcf140b3300 2022-07-20T03:17:14.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.743+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcefc011d70 con 0x7fcf140b3300 2022-07-20T03:17:14.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.747+0000 7fcf11ffb700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef8053a60 0x7fcef8055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:14.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.747+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fcefc06c190 con 0x7fcf140b3300 2022-07-20T03:17:14.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.747+0000 7fcf13fff700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef8053a60 0x7fcef8055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:14.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.747+0000 7fcf13fff700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef8053a60 0x7fcef8055f20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fcf04011000 tx=0x7fcf0400d040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:14.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.751+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcefc0430f0 con 0x7fcf140b3300 2022-07-20T03:17:14.777 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.a has been restored 2022-07-20T03:17:14.778 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mds.b has been restored 2022-07-20T03:17:14.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.911+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fcf1409bce0 con 0x7fcf140b3300 2022-07-20T03:17:14.925 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.919+0000 7fcf11ffb700 1 -- 172.21.15.61:0/150251025 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v44) v1 ==== 184+0+0 (secure 0 0 0) 0x7fcefc046710 con 0x7fcf140b3300 2022-07-20T03:17:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef8053a60 msgr2=0x7fcef8055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef8053a60 0x7fcef8055f20 secure :-1 s=READY pgs=667 cs=0 l=1 rev1=1 rx=0x7fcf04011000 tx=0x7fcf0400d040).stop 2022-07-20T03:17:14.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 msgr2=0x7fcf140ad960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:14.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 0x7fcf140ad960 secure :-1 s=READY pgs=1388 cs=0 l=1 rev1=1 rx=0x7fcefc00c000 tx=0x7fcefc006df0).stop 2022-07-20T03:17:14.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 shutdown_connections 2022-07-20T03:17:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcef8053a60 0x7fcef8055f20 unknown :-1 s=CLOSED pgs=667 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fcf1413f6c0 0x7fcf140ade50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fcf140b3300 0x7fcf140ad960 unknown :-1 s=CLOSED pgs=1388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 --2- 172.21.15.61:0/150251025 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fcf14060220 0x7fcf1413cb80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:14.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 >> 172.21.15.61:0/150251025 conn(0x7fcf140b6c20 msgr2=0x7fcf1405c620 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:14.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 shutdown_connections 2022-07-20T03:17:14.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:14.923+0000 7fcf1a8f1700 1 -- 172.21.15.61:0/150251025 wait complete. 2022-07-20T03:17:14.933 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:17:14.940 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:17:15.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba21c8700 1 Processor -- start 2022-07-20T03:17:15.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba21c8700 1 -- start start 2022-07-20T03:17:15.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba21c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4b9c0b6800 0x7f4b9c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba21c8700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0ae280 con 0x7f4b9c0b3cb0 2022-07-20T03:17:15.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba21c8700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4b9c0b6be0 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba21c8700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4b9c0ae4b0 con 0x7f4b9c0b6800 2022-07-20T03:17:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba11c6700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4b9c0b6800 0x7f4b9c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba09c5700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4b9c0b3cb0 0x7f4b9c145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47746/0 (socket says 172.21.15.61:47746) 2022-07-20T03:17:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba09c5700 1 -- 172.21.15.61:0/1132148030 learned_addr learned my addr 172.21.15.61:0/1132148030 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:15.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1132148030 >> v1:172.21.15.117:6789/0 conn(0x7f4b9c0ad0e0 legacy=0x7f4b9c1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1132148030 >> v1:172.21.15.61:6789/0 conn(0x7f4b9c0b3cb0 legacy=0x7f4b9c145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.055+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1132148030 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9c0ae750 con 0x7f4b9c0b6800 2022-07-20T03:17:15.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba11c6700 1 --2- 172.21.15.61:0/1132148030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c0aebf0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7f4b98002f70 tx=0x7f4b98006df0).ready entity=mon.0 client_cookie=bd390a62e2d078e7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4b93fff700 1 -- 172.21.15.61:0/1132148030 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9800d610 con 0x7f4b9c0b6800 2022-07-20T03:17:15.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4b93fff700 1 -- 172.21.15.61:0/1132148030 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98000e90 con 0x7f4b9c0b6800 2022-07-20T03:17:15.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4b93fff700 1 -- 172.21.15.61:0/1132148030 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9800d960 con 0x7f4b9c0b6800 2022-07-20T03:17:15.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1132148030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0b6800 msgr2=0x7f4b9c0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1132148030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c0aebf0 secure :-1 s=READY pgs=1390 cs=0 l=1 rev1=1 rx=0x7f4b98002f70 tx=0x7f4b98006df0).stop 2022-07-20T03:17:15.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1132148030 shutdown_connections 2022-07-20T03:17:15.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1132148030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c0aebf0 unknown :-1 s=CLOSED pgs=1390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1132148030 >> 172.21.15.61:0/1132148030 conn(0x7f4b9c0b9dd0 msgr2=0x7f4b9c0b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1132148030 shutdown_connections 2022-07-20T03:17:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1132148030 wait complete. 2022-07-20T03:17:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 Processor -- start 2022-07-20T03:17:15.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.059+0000 7f4ba21c8700 1 -- start start 2022-07-20T03:17:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 0x7f4b9c112e30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b9c0b3cb0 0x7f4b9c113910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c13ee60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c0b1e50 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c115f10 con 0x7f4b9c0b6800 2022-07-20T03:17:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4b9c116100 con 0x7f4b9c0b3cb0 2022-07-20T03:17:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 0x7f4b9c112e30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 0x7f4b9c112e30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51152/0 (socket says 172.21.15.61:51152) 2022-07-20T03:17:15.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1388439055 learned_addr learned my addr 172.21.15.61:0/1388439055 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba19c7700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c13ee60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba09c5700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b9c0b3cb0 0x7f4b9c113910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b9c0b3cb0 msgr2=0x7f4b9c113910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b9c0b3cb0 0x7f4b9c113910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b9c0b6800 msgr2=0x7f4b9c13ee60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c13ee60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 -- 172.21.15.61:0/1388439055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b9800b180 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba09c5700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b9c0b3cb0 0x7f4b9c113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:17:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba19c7700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c13ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:15.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba11c6700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 0x7f4b9c112e30 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f4b98002f70 tx=0x7f4b98015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9800d990 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4b98011ce0 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b9c0b2960 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4b9801e650 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b9c113e00 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.063+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b9c1162c0 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.067+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4b98025030 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.067+0000 7f4b927fc700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b84053ad0 0x7f4b84055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.067+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4b9806d300 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.067+0000 7f4ba09c5700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b84053ad0 0x7f4b84055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.067+0000 7f4ba09c5700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b84053ad0 0x7f4b84055f90 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f4b8c008000 tx=0x7f4b8c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.071+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4b980442a0 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.235+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f4b9c0b5530 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.243+0000 7f4b927fc700 1 -- 172.21.15.61:0/1388439055 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v45) v1 ==== 184+0+0 (secure 0 0 0) 0x7f4b980230e0 con 0x7f4b9c0ad0e0 2022-07-20T03:17:15.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b84053ad0 msgr2=0x7f4b84055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b84053ad0 0x7f4b84055f90 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7f4b8c008000 tx=0x7f4b8c009040).stop 2022-07-20T03:17:15.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 msgr2=0x7f4b9c112e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 0x7f4b9c112e30 secure :-1 s=READY pgs=1391 cs=0 l=1 rev1=1 rx=0x7f4b98002f70 tx=0x7f4b98015000).stop 2022-07-20T03:17:15.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 shutdown_connections 2022-07-20T03:17:15.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4b84053ad0 0x7f4b84055f90 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4b9c0b6800 0x7f4b9c13ee60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4b9c0b3cb0 0x7f4b9c113910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 --2- 172.21.15.61:0/1388439055 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4b9c0ad0e0 0x7f4b9c112e30 unknown :-1 s=CLOSED pgs=1391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 >> 172.21.15.61:0/1388439055 conn(0x7f4b9c0b9dd0 msgr2=0x7f4b9c1431f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 shutdown_connections 2022-07-20T03:17:15.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.247+0000 7f4ba21c8700 1 -- 172.21.15.61:0/1388439055 wait complete. 2022-07-20T03:17:15.258 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:17:15.266 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:17:15.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 Processor -- start 2022-07-20T03:17:15.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 -- start start 2022-07-20T03:17:15.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e40af0d0 0x7fb3e40b3640 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb3e40b4b10 0x7fb3e4141440 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3e4141930 0x7fb3e40abe00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e4140920 con 0x7fb3e4141930 2022-07-20T03:17:15.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb3e40af4b0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e92dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e40a97a0 con 0x7fb3e40af0d0 2022-07-20T03:17:15.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3e4141930 0x7fb3e40abe00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3e4141930 0x7fb3e40abe00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51162/0 (socket says 172.21.15.61:51162) 2022-07-20T03:17:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e3fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e40af0d0 0x7fb3e40b3640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 -- 172.21.15.61:0/1639477236 learned_addr learned my addr 172.21.15.61:0/1639477236 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:15.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e37fe700 1 --2- 172.21.15.61:0/1639477236 >> v2:172.21.15.117:3300/0 conn(0x7fb3e40b4b10 0x7fb3e4141440 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 -- 172.21.15.61:0/1639477236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e40af0d0 msgr2=0x7fb3e40b3640 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:15.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 --2- 172.21.15.61:0/1639477236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e40af0d0 0x7fb3e40b3640 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 -- 172.21.15.61:0/1639477236 >> v2:172.21.15.117:3300/0 conn(0x7fb3e40b4b10 msgr2=0x7fb3e4141440 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:15.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.383+0000 7fb3e8adc700 1 --2- 172.21.15.61:0/1639477236 >> v2:172.21.15.117:3300/0 conn(0x7fb3e40b4b10 0x7fb3e4141440 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e8adc700 1 -- 172.21.15.61:0/1639477236 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3e40a9a40 con 0x7fb3e4141930 2022-07-20T03:17:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e8adc700 1 --2- 172.21.15.61:0/1639477236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e4141930 0x7fb3e40abe00 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fb3e40b1750 tx=0x7fb3dc006eb0).ready entity=mon.0 client_cookie=fdb2f27c67d3a3db server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e2ffd700 1 -- 172.21.15.61:0/1639477236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc00e610 con 0x7fb3e4141930 2022-07-20T03:17:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e2ffd700 1 -- 172.21.15.61:0/1639477236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3dc000e90 con 0x7fb3e4141930 2022-07-20T03:17:15.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e2ffd700 1 -- 172.21.15.61:0/1639477236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3dc00e9a0 con 0x7fb3e4141930 2022-07-20T03:17:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 -- 172.21.15.61:0/1639477236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e4141930 msgr2=0x7fb3e40abe00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/1639477236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e4141930 0x7fb3e40abe00 secure :-1 s=READY pgs=1392 cs=0 l=1 rev1=1 rx=0x7fb3e40b1750 tx=0x7fb3dc006eb0).stop 2022-07-20T03:17:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 -- 172.21.15.61:0/1639477236 shutdown_connections 2022-07-20T03:17:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/1639477236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e4141930 0x7fb3e40abe00 unknown :-1 s=CLOSED pgs=1392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/1639477236 >> v2:172.21.15.117:3300/0 conn(0x7fb3e40b4b10 0x7fb3e4141440 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/1639477236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e40af0d0 0x7fb3e40b3640 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 -- 172.21.15.61:0/1639477236 >> 172.21.15.61:0/1639477236 conn(0x7fb3e4063150 msgr2=0x7fb3e4063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:15.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 -- 172.21.15.61:0/1639477236 shutdown_connections 2022-07-20T03:17:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.387+0000 7fb3e92dd700 1 -- 172.21.15.61:0/1639477236 wait complete. 2022-07-20T03:17:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 Processor -- start 2022-07-20T03:17:15.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 -- start start 2022-07-20T03:17:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3e40af0d0 0x7fb3e4146700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 0x7fb3e414fa20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e4141930 0x7fb3e41534a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e414ff10 con 0x7fb3e40b4b10 2022-07-20T03:17:15.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e4153070 con 0x7fb3e40af0d0 2022-07-20T03:17:15.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3e409e160 con 0x7fb3e4141930 2022-07-20T03:17:15.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 0x7fb3e414fa20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 0x7fb3e414fa20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51168/0 (socket says 172.21.15.61:51168) 2022-07-20T03:17:15.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 -- 172.21.15.61:0/3351637901 learned_addr learned my addr 172.21.15.61:0/3351637901 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:15.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e8adc700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e4141930 0x7fb3e41534a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e3fff700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3e40af0d0 0x7fb3e4146700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 -- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e4141930 msgr2=0x7fb3e41534a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e4141930 0x7fb3e41534a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 -- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3e40af0d0 msgr2=0x7fb3e4146700 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3e40af0d0 0x7fb3e4146700 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 -- 172.21.15.61:0/3351637901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3dc00b1e0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e8adc700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e4141930 0x7fb3e41534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:15.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e37fe700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 0x7fb3e414fa20 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fb3d400c000 tx=0x7fb3d4006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3d4010780 con 0x7fb3e40b4b10 2022-07-20T03:17:15.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb3d40108e0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3e409e400 con 0x7fb3e40b4b10 2022-07-20T03:17:15.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb3d4010c40 con 0x7fb3e40b4b10 2022-07-20T03:17:15.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.391+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3e40645d0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.395+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3e40574e0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.395+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb3d401c460 con 0x7fb3e40b4b10 2022-07-20T03:17:15.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.395+0000 7fb3e17fa700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c8053ad0 0x7fb3c8055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.395+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb3d406c490 con 0x7fb3e40b4b10 2022-07-20T03:17:15.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.395+0000 7fb3e3fff700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c8053ad0 0x7fb3c8055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.395+0000 7fb3e3fff700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c8053ad0 0x7fb3c8055f90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fb3cc00b000 tx=0x7fb3cc008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.399+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb3d4043290 con 0x7fb3e40b4b10 2022-07-20T03:17:15.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.567+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7fb3e40699b0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.575+0000 7fb3e17fa700 1 -- 172.21.15.61:0/3351637901 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v46) v1 ==== 184+0+0 (secure 0 0 0) 0x7fb3d40468b0 con 0x7fb3e40b4b10 2022-07-20T03:17:15.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c8053ad0 msgr2=0x7fb3c8055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.585 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c8053ad0 0x7fb3c8055f90 secure :-1 s=READY pgs=669 cs=0 l=1 rev1=1 rx=0x7fb3cc00b000 tx=0x7fb3cc008040).stop 2022-07-20T03:17:15.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 msgr2=0x7fb3e414fa20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 0x7fb3e414fa20 secure :-1 s=READY pgs=1393 cs=0 l=1 rev1=1 rx=0x7fb3d400c000 tx=0x7fb3d4006df0).stop 2022-07-20T03:17:15.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 shutdown_connections 2022-07-20T03:17:15.587 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb3c8053ad0 0x7fb3c8055f90 unknown :-1 s=CLOSED pgs=669 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3e4141930 0x7fb3e41534a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3e40b4b10 0x7fb3e414fa20 unknown :-1 s=CLOSED pgs=1393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 --2- 172.21.15.61:0/3351637901 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3e40af0d0 0x7fb3e4146700 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 >> 172.21.15.61:0/3351637901 conn(0x7fb3e4063150 msgr2=0x7fb3e405f840 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:15.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 shutdown_connections 2022-07-20T03:17:15.590 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.579+0000 7fb3e92dd700 1 -- 172.21.15.61:0/3351637901 wait complete. 2022-07-20T03:17:15.591 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:17:15.596 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:15.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 Processor -- start 2022-07-20T03:17:15.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- start start 2022-07-20T03:17:15.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4ae80b1480 0x7f4ae813f900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 0x7f4ae80af660 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4ae8140ba0 con 0x7f4ae813fdf0 2022-07-20T03:17:15.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae80ae410 con 0x7f4ae805bd90 2022-07-20T03:17:15.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4ae8142fa0 con 0x7f4ae80b1480 2022-07-20T03:17:15.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aeceee700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 0x7f4ae80af660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aeceee700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 0x7f4ae80af660 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51178/0 (socket says 172.21.15.61:51178) 2022-07-20T03:17:15.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aeceee700 1 -- 172.21.15.61:0/463003983 learned_addr learned my addr 172.21.15.61:0/463003983 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:15.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae7fff700 1 --2- 172.21.15.61:0/463003983 >> v2:172.21.15.117:3300/0 conn(0x7f4ae80b1480 0x7f4ae813f900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae7fff700 1 -- 172.21.15.61:0/463003983 >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 msgr2=0x7f4ae80af660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:15.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae7fff700 1 --2- 172.21.15.61:0/463003983 >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 0x7f4ae80af660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae7fff700 1 -- 172.21.15.61:0/463003983 >> v1:172.21.15.117:6789/0 conn(0x7f4ae813fdf0 legacy=0x7f4ae81436e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae7fff700 1 -- 172.21.15.61:0/463003983 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ae8143240 con 0x7f4ae80b1480 2022-07-20T03:17:15.710 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aeceee700 1 --2- 172.21.15.61:0/463003983 >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 0x7f4ae80af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:15.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae7fff700 1 --2- 172.21.15.61:0/463003983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae80b1480 0x7f4ae813f900 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f4ae0002f70 tx=0x7f4ae0006df0).ready entity=mon.1 client_cookie=f58fb91187434194 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae6ffd700 1 -- 172.21.15.61:0/463003983 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ae000d570 con 0x7f4ae80b1480 2022-07-20T03:17:15.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae6ffd700 1 -- 172.21.15.61:0/463003983 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae0000e90 con 0x7f4ae80b1480 2022-07-20T03:17:15.711 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4ae6ffd700 1 -- 172.21.15.61:0/463003983 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ae000d8c0 con 0x7f4ae80b1480 2022-07-20T03:17:15.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- 172.21.15.61:0/463003983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae80b1480 msgr2=0x7f4ae813f900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 --2- 172.21.15.61:0/463003983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae80b1480 0x7f4ae813f900 secure :-1 s=READY pgs=622 cs=0 l=1 rev1=1 rx=0x7f4ae0002f70 tx=0x7f4ae0006df0).stop 2022-07-20T03:17:15.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- 172.21.15.61:0/463003983 shutdown_connections 2022-07-20T03:17:15.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 --2- 172.21.15.61:0/463003983 >> v2:172.21.15.61:3300/0 conn(0x7f4ae805bd90 0x7f4ae80af660 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 --2- 172.21.15.61:0/463003983 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae80b1480 0x7f4ae813f900 unknown :-1 s=CLOSED pgs=622 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- 172.21.15.61:0/463003983 >> 172.21.15.61:0/463003983 conn(0x7f4ae80b23d0 msgr2=0x7f4ae80b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:15.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- 172.21.15.61:0/463003983 shutdown_connections 2022-07-20T03:17:15.713 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.699+0000 7f4aee951700 1 -- 172.21.15.61:0/463003983 wait complete. 2022-07-20T03:17:15.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 Processor -- start 2022-07-20T03:17:15.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 -- start start 2022-07-20T03:17:15.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 0x7f4ae80b49d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ae80b1480 0x7f4ae81521a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae813fdf0 0x7f4ae8155de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae8155710 con 0x7f4ae805bd90 2022-07-20T03:17:15.718 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae80946d0 con 0x7f4ae813fdf0 2022-07-20T03:17:15.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4ae8155940 con 0x7f4ae80b1480 2022-07-20T03:17:15.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 0x7f4ae80b49d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 0x7f4ae80b49d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51180/0 (socket says 172.21.15.61:51180) 2022-07-20T03:17:15.719 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 -- 172.21.15.61:0/2617066940 learned_addr learned my addr 172.21.15.61:0/2617066940 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:15.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae77fe700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ae80b1480 0x7f4ae81521a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.720 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aeceee700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae813fdf0 0x7f4ae8155de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 -- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ae80b1480 msgr2=0x7f4ae81521a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ae80b1480 0x7f4ae81521a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 -- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae813fdf0 msgr2=0x7f4ae8155de0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae813fdf0 0x7f4ae8155de0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.722 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 -- 172.21.15.61:0/2617066940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ad800b1e0 con 0x7f4ae805bd90 2022-07-20T03:17:15.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae7fff700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 0x7f4ae80b49d0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f4ae0002f70 tx=0x7f4ae0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ae000dea0 con 0x7f4ae805bd90 2022-07-20T03:17:15.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4ae0013de0 con 0x7f4ae805bd90 2022-07-20T03:17:15.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ae000b180 con 0x7f4ae805bd90 2022-07-20T03:17:15.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4ae001e5f0 con 0x7f4ae805bd90 2022-07-20T03:17:15.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.703+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ae8152690 con 0x7f4ae805bd90 2022-07-20T03:17:15.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.707+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ae8094840 con 0x7f4ae805bd90 2022-07-20T03:17:15.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.707+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4ae0025030 con 0x7f4ae805bd90 2022-07-20T03:17:15.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.707+0000 7f4ae57fa700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4acc053920 0x7f4acc055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:15.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.707+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4ae006caa0 con 0x7f4ae805bd90 2022-07-20T03:17:15.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.707+0000 7f4ae77fe700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4acc053920 0x7f4acc055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:15.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.707+0000 7f4ae77fe700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4acc053920 0x7f4acc055de0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f4ad000d000 tx=0x7f4ad0007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:15.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.711+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4ae8094840 con 0x7f4ae805bd90 2022-07-20T03:17:15.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.875+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4ae8094840 con 0x7f4ae805bd90 2022-07-20T03:17:15.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.875+0000 7f4ae57fa700 1 -- 172.21.15.61:0/2617066940 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f4ae8094840 con 0x7f4ae805bd90 2022-07-20T03:17:15.882 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:15.882 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:13.001769+0000","modified":"2022-07-20T03:17:14.005556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8466":{"gid":8466,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3390219350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3390219350},{"type":"v1","addr":"172.21.15.61:6833","nonce":3390219350}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T03:17:15.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4acc053920 msgr2=0x7f4acc055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4acc053920 0x7f4acc055de0 secure :-1 s=READY pgs=670 cs=0 l=1 rev1=1 rx=0x7f4ad000d000 tx=0x7f4ad0007f10).stop 2022-07-20T03:17:15.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 msgr2=0x7f4ae80b49d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:15.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 0x7f4ae80b49d0 secure :-1 s=READY pgs=1394 cs=0 l=1 rev1=1 rx=0x7f4ae0002f70 tx=0x7f4ae0014000).stop 2022-07-20T03:17:15.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 shutdown_connections 2022-07-20T03:17:15.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4acc053920 0x7f4acc055de0 unknown :-1 s=CLOSED pgs=670 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4ae813fdf0 0x7f4ae8155de0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4ae80b1480 0x7f4ae81521a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 --2- 172.21.15.61:0/2617066940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4ae805bd90 0x7f4ae80b49d0 unknown :-1 s=CLOSED pgs=1394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:15.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 >> 172.21.15.61:0/2617066940 conn(0x7f4ae80b23d0 msgr2=0x7f4ae805fba0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:15.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 shutdown_connections 2022-07-20T03:17:15.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:15.879+0000 7f4aee951700 1 -- 172.21.15.61:0/2617066940 wait complete. 2022-07-20T03:17:15.888 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-20T03:17:16.097 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:17:13.001769+0000', 'modified': '2022-07-20T03:17:14.005556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8466': {'gid': 8466, 'name': 'a', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3390219350', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3390219350}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3390219350}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:17:16.098 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:17:16.099 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:17:16.220 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8466,"want_state":"up:active","state":"up:active","rank_uptime":3.2062655919999998,"mdsmap_epoch":77,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":8.0440171760000005} 2022-07-20T03:17:16.221 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8466, "mdsmap_epoch": 77, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 3.206265592, "state": "up:active", "uptime": 8.044017176, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:17:16.223 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:16.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 Processor -- start 2022-07-20T03:17:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 -- start start 2022-07-20T03:17:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a80b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd3a805bd90 0x7fd3a8142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd3a80aea30 con 0x7fd3a80b1200 2022-07-20T03:17:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd3a8142970 con 0x7fd3a805bd90 2022-07-20T03:17:16.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ad150700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a80aec60 con 0x7fd3a80b5650 2022-07-20T03:17:16.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd39ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd3a80b1200 0x7fd3a805c4f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47786/0 (socket says 172.21.15.61:47786) 2022-07-20T03:17:16.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd39ffff700 1 -- 172.21.15.61:0/4082128568 learned_addr learned my addr 172.21.15.61:0/4082128568 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:16.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3ac94f700 1 --2- 172.21.15.61:0/4082128568 >> v2:172.21.15.61:3300/0 conn(0x7fd3a805bd90 0x7fd3a8142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.323+0000 7fd3a7fff700 1 --2- 172.21.15.61:0/4082128568 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a80b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ac94f700 1 -- 172.21.15.61:0/4082128568 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 msgr2=0x7fd3a80b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:16.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ac94f700 1 --2- 172.21.15.61:0/4082128568 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a80b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ac94f700 1 -- 172.21.15.61:0/4082128568 >> v1:172.21.15.61:6789/0 conn(0x7fd3a80b1200 legacy=0x7fd3a805c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ac94f700 1 -- 172.21.15.61:0/4082128568 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a80af180 con 0x7fd3a805bd90 2022-07-20T03:17:16.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ac94f700 1 --2- 172.21.15.61:0/4082128568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8142d60 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fd3a00032f0 tx=0x7fd3a00091d0).ready entity=mon.0 client_cookie=170f34abf2599706 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3a77fe700 1 -- 172.21.15.61:0/4082128568 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3a0009ea0 con 0x7fd3a805bd90 2022-07-20T03:17:16.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3a77fe700 1 -- 172.21.15.61:0/4082128568 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3a000b8a0 con 0x7fd3a805bd90 2022-07-20T03:17:16.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3a77fe700 1 -- 172.21.15.61:0/4082128568 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd3a000ba10 con 0x7fd3a805bd90 2022-07-20T03:17:16.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- 172.21.15.61:0/4082128568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 msgr2=0x7fd3a8142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 --2- 172.21.15.61:0/4082128568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8142d60 secure :-1 s=READY pgs=1396 cs=0 l=1 rev1=1 rx=0x7fd3a00032f0 tx=0x7fd3a00091d0).stop 2022-07-20T03:17:16.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- 172.21.15.61:0/4082128568 shutdown_connections 2022-07-20T03:17:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 --2- 172.21.15.61:0/4082128568 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8142d60 unknown :-1 s=CLOSED pgs=1396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 --2- 172.21.15.61:0/4082128568 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a80b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- 172.21.15.61:0/4082128568 >> 172.21.15.61:0/4082128568 conn(0x7fd3a80634e0 msgr2=0x7fd3a80b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:16.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- 172.21.15.61:0/4082128568 shutdown_connections 2022-07-20T03:17:16.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- 172.21.15.61:0/4082128568 wait complete. 2022-07-20T03:17:16.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 Processor -- start 2022-07-20T03:17:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- start start 2022-07-20T03:17:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8141810 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3a80b1200 0x7fd3a8152080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a8152990 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a813b990 con 0x7fd3a805bd90 2022-07-20T03:17:16.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8152570 con 0x7fd3a80b1200 2022-07-20T03:17:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ad150700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a8143ee0 con 0x7fd3a80b5650 2022-07-20T03:17:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8141810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3a7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8141810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51194/0 (socket says 172.21.15.61:51194) 2022-07-20T03:17:16.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3a7fff700 1 -- 172.21.15.61:0/3909165487 learned_addr learned my addr 172.21.15.61:0/3909165487 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:16.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd3ac94f700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a8152990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.327+0000 7fd39ffff700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3a80b1200 0x7fd3a8152080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a7fff700 1 -- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 msgr2=0x7fd3a8152990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a7fff700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a8152990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.355 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a7fff700 1 -- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3a80b1200 msgr2=0x7fd3a8152080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a7fff700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3a80b1200 0x7fd3a8152080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.356 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a7fff700 1 -- 172.21.15.61:0/3909165487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3a0009470 con 0x7fd3a805bd90 2022-07-20T03:17:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3ac94f700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a8152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:16.357 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a7fff700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8141810 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7fd3980030b0 tx=0x7fd398008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.358 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd398011780 con 0x7fd3a805bd90 2022-07-20T03:17:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd3980118e0 con 0x7fd3a805bd90 2022-07-20T03:17:16.359 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a8063cd0 con 0x7fd3a805bd90 2022-07-20T03:17:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd398011c40 con 0x7fd3a805bd90 2022-07-20T03:17:16.360 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a8145510 con 0x7fd3a805bd90 2022-07-20T03:17:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a8063ac0 con 0x7fd3a805bd90 2022-07-20T03:17:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd398012420 con 0x7fd3a805bd90 2022-07-20T03:17:16.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a5ffb700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd38c053a40 0x7fd38c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd39806bc00 con 0x7fd3a805bd90 2022-07-20T03:17:16.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.331+0000 7fd39ffff700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd38c053a40 0x7fd38c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.335+0000 7fd39ffff700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd38c053a40 0x7fd38c055f00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fd39000d000 tx=0x7fd390009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.339+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd398071020 con 0x7fd3a805bd90 2022-07-20T03:17:16.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.463+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fd3a80574e0 con 0x7fd3a805bd90 2022-07-20T03:17:16.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3a5ffb700 1 -- 172.21.15.61:0/3909165487 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fd398042b00 con 0x7fd3a805bd90 2022-07-20T03:17:16.472 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:16.472 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":140,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:14.005088+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:10.948609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:12.001181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:16.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd38c053a40 msgr2=0x7fd38c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd38c053a40 0x7fd38c055f00 secure :-1 s=READY pgs=671 cs=0 l=1 rev1=1 rx=0x7fd39000d000 tx=0x7fd390009040).stop 2022-07-20T03:17:16.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 msgr2=0x7fd3a8141810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8141810 secure :-1 s=READY pgs=1397 cs=0 l=1 rev1=1 rx=0x7fd3980030b0 tx=0x7fd398008040).stop 2022-07-20T03:17:16.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 shutdown_connections 2022-07-20T03:17:16.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd38c053a40 0x7fd38c055f00 unknown :-1 s=CLOSED pgs=671 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd3a80b1200 0x7fd3a8152080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd3a80b5650 0x7fd3a8152990 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 --2- 172.21.15.61:0/3909165487 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd3a805bd90 0x7fd3a8141810 unknown :-1 s=CLOSED pgs=1397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 >> 172.21.15.61:0/3909165487 conn(0x7fd3a80634e0 msgr2=0x7fd3a80b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:16.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 shutdown_connections 2022-07-20T03:17:16.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.467+0000 7fd3ad150700 1 -- 172.21.15.61:0/3909165487 wait complete. 2022-07-20T03:17:16.486 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:17:10.948609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:17:12.001181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:17:16.487 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:17:16.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 Processor -- start 2022-07-20T03:17:16.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 -- start start 2022-07-20T03:17:16.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f16b80b3330 0x7f16b805cbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f16b80b3710 0x7f16b809aac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f16b8144e70 con 0x7f16b8144a20 2022-07-20T03:17:16.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b8144040 con 0x7f16b80b3330 2022-07-20T03:17:16.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16bdc57700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f16b81442e0 con 0x7f16b80b3710 2022-07-20T03:17:16.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f16b80b3330 0x7f16b805cbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b7fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f16b8144a20 0x7f16b80b1b30 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47800/0 (socket says 172.21.15.61:47800) 2022-07-20T03:17:16.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b7fff700 1 -- 172.21.15.61:0/1350365327 learned_addr learned my addr 172.21.15.61:0/1350365327 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:16.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f16b80b3330 0x7f16b805cbb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51202/0 (socket says 172.21.15.61:51202) 2022-07-20T03:17:16.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/1350365327 >> v2:172.21.15.117:3300/0 conn(0x7f16b80b3710 0x7f16b809aac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 -- 172.21.15.61:0/1350365327 >> v2:172.21.15.117:3300/0 conn(0x7f16b80b3710 msgr2=0x7f16b809aac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:16.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 --2- 172.21.15.61:0/1350365327 >> v2:172.21.15.117:3300/0 conn(0x7f16b80b3710 0x7f16b809aac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 -- 172.21.15.61:0/1350365327 >> v1:172.21.15.61:6789/0 conn(0x7f16b8144a20 legacy=0x7f16b80b1b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b67fc700 1 -- 172.21.15.61:0/1350365327 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3788242754 0 0) 0x7f16b8144e70 con 0x7f16b8144a20 2022-07-20T03:17:16.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 -- 172.21.15.61:0/1350365327 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b8144580 con 0x7f16b80b3330 2022-07-20T03:17:16.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/1350365327 >> v2:172.21.15.117:3300/0 conn(0x7f16b80b3710 0x7f16b809aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:16.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.591+0000 7f16b77fe700 1 --2- 172.21.15.61:0/1350365327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3330 0x7f16b805cbb0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f16b00032f0 tx=0x7f16b00091d0).ready entity=mon.0 client_cookie=c9351ff3d8b835c3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16b67fc700 1 -- 172.21.15.61:0/1350365327 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b0009ea0 con 0x7f16b80b3330 2022-07-20T03:17:16.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16b67fc700 1 -- 172.21.15.61:0/1350365327 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16b000b870 con 0x7f16b80b3330 2022-07-20T03:17:16.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16b67fc700 1 -- 172.21.15.61:0/1350365327 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16b000b9e0 con 0x7f16b80b3330 2022-07-20T03:17:16.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- 172.21.15.61:0/1350365327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3330 msgr2=0x7f16b805cbb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 --2- 172.21.15.61:0/1350365327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3330 0x7f16b805cbb0 secure :-1 s=READY pgs=1399 cs=0 l=1 rev1=1 rx=0x7f16b00032f0 tx=0x7f16b00091d0).stop 2022-07-20T03:17:16.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- 172.21.15.61:0/1350365327 shutdown_connections 2022-07-20T03:17:16.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 --2- 172.21.15.61:0/1350365327 >> v2:172.21.15.117:3300/0 conn(0x7f16b80b3710 0x7f16b809aac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 --2- 172.21.15.61:0/1350365327 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3330 0x7f16b805cbb0 unknown :-1 s=CLOSED pgs=1399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- 172.21.15.61:0/1350365327 >> 172.21.15.61:0/1350365327 conn(0x7f16b80b6c50 msgr2=0x7f16b80b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:16.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- 172.21.15.61:0/1350365327 shutdown_connections 2022-07-20T03:17:16.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- 172.21.15.61:0/1350365327 wait complete. 2022-07-20T03:17:16.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 Processor -- start 2022-07-20T03:17:16.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- start start 2022-07-20T03:17:16.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16b80b3330 0x7f16b80ab620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 0x7f16b813cfc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16b8144a20 0x7f16b813d4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8143a00 con 0x7f16b80b3710 2022-07-20T03:17:16.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f16b8143ca0 con 0x7f16b8144a20 2022-07-20T03:17:16.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.595+0000 7f16bdc57700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f16b80abb10 con 0x7f16b80b3330 2022-07-20T03:17:16.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 0x7f16b813cfc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 0x7f16b813cfc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51210/0 (socket says 172.21.15.61:51210) 2022-07-20T03:17:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3649861389 learned_addr learned my addr 172.21.15.61:0/3649861389 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b77fe700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16b80b3330 0x7f16b80ab620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b7fff700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16b8144a20 0x7f16b813d4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16b80b3330 msgr2=0x7f16b80ab620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16b80b3330 0x7f16b80ab620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16b8144a20 msgr2=0x7f16b813d4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16b8144a20 0x7f16b813d4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 -- 172.21.15.61:0/3649861389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f16b0009470 con 0x7f16b80b3710 2022-07-20T03:17:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b6ffd700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 0x7f16b813cfc0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f16a0006940 tx=0x7f16a0008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a0011780 con 0x7f16b80b3710 2022-07-20T03:17:16.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f16a00118e0 con 0x7f16b80b3710 2022-07-20T03:17:16.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f16b81436f0 con 0x7f16b80b3710 2022-07-20T03:17:16.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f16a0011c40 con 0x7f16b80b3710 2022-07-20T03:17:16.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f16b81432d0 con 0x7f16b80b3710 2022-07-20T03:17:16.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f16b81429c0 con 0x7f16b80b3710 2022-07-20T03:17:16.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f16a001c460 con 0x7f16b80b3710 2022-07-20T03:17:16.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b4ff9700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f169c0539b0 0x7f169c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f16a006c090 con 0x7f16b80b3710 2022-07-20T03:17:16.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.599+0000 7f16b77fe700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f169c0539b0 0x7f169c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.603+0000 7f16b77fe700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f169c0539b0 0x7f169c055e70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f16b001b000 tx=0x7f16b0016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.607+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f16b81429c0 con 0x7f16b80b3710 2022-07-20T03:17:16.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.731+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f16b8143ea0 con 0x7f16b80b3710 2022-07-20T03:17:16.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.731+0000 7f16b4ff9700 1 -- 172.21.15.61:0/3649861389 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7f16b8143ea0 con 0x7f16b80b3710 2022-07-20T03:17:16.739 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:17:16.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f169c0539b0 msgr2=0x7f169c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f169c0539b0 0x7f169c055e70 secure :-1 s=READY pgs=672 cs=0 l=1 rev1=1 rx=0x7f16b001b000 tx=0x7f16b0016000).stop 2022-07-20T03:17:16.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 msgr2=0x7f16b813cfc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 0x7f16b813cfc0 secure :-1 s=READY pgs=1400 cs=0 l=1 rev1=1 rx=0x7f16a0006940 tx=0x7f16a0008040).stop 2022-07-20T03:17:16.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 shutdown_connections 2022-07-20T03:17:16.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f169c0539b0 0x7f169c055e70 unknown :-1 s=CLOSED pgs=672 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f16b8144a20 0x7f16b813d4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f16b80b3710 0x7f16b813cfc0 unknown :-1 s=CLOSED pgs=1400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 --2- 172.21.15.61:0/3649861389 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f16b80b3330 0x7f16b80ab620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 >> 172.21.15.61:0/3649861389 conn(0x7f16b80b6c50 msgr2=0x7f16b81411a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:16.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 shutdown_connections 2022-07-20T03:17:16.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.735+0000 7f16bdc57700 1 -- 172.21.15.61:0/3649861389 wait complete. 2022-07-20T03:17:16.750 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:17:16.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93b1228700 1 Processor -- start 2022-07-20T03:17:16.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93b1228700 1 -- start start 2022-07-20T03:17:16.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93b1228700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f93ac0b2210 0x7f93ac140860 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93b1228700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac13fe80 con 0x7f93ac0ac680 2022-07-20T03:17:16.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93b1228700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f93ac140120 con 0x7f93ac0967e0 2022-07-20T03:17:16.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93b1228700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93ac1403c0 con 0x7f93ac0b2210 2022-07-20T03:17:16.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93ab59e700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f93ac0ac680 0x7f93ac13efb0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47818/0 (socket says 172.21.15.61:47818) 2022-07-20T03:17:16.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93ab59e700 1 -- 172.21.15.61:0/3997459845 learned_addr learned my addr 172.21.15.61:0/3997459845 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:16.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93aad9d700 1 --2- 172.21.15.61:0/3997459845 >> v2:172.21.15.117:3300/0 conn(0x7f93ac0b2210 0x7f93ac140860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3215833419 0 0) 0x7f93ac13fe80 con 0x7f93ac0ac680 2022-07-20T03:17:16.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9390003260 con 0x7f93ac0ac680 2022-07-20T03:17:16.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2680995576 0 0) 0x7f93ac140120 con 0x7f93ac0967e0 2022-07-20T03:17:16.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f93ac13fe80 con 0x7f93ac0967e0 2022-07-20T03:17:16.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93aad9d700 1 -- 172.21.15.61:0/3997459845 >> v1:172.21.15.117:6789/0 conn(0x7f93ac0967e0 legacy=0x7f93ac0ab0b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93aad9d700 1 -- 172.21.15.61:0/3997459845 >> v1:172.21.15.61:6789/0 conn(0x7f93ac0ac680 legacy=0x7f93ac13efb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.855+0000 7f93aad9d700 1 -- 172.21.15.61:0/3997459845 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93ac0b25f0 con 0x7f93ac0b2210 2022-07-20T03:17:16.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93aad9d700 1 --2- 172.21.15.61:0/3997459845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0b2210 0x7f93ac140860 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f939c002f70 tx=0x7f939c006df0).ready entity=mon.1 client_cookie=3abd4e67d9d8a19d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.865 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f939c00d540 con 0x7f93ac0b2210 2022-07-20T03:17:16.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f939c000e90 con 0x7f93ac0b2210 2022-07-20T03:17:16.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93a9d9b700 1 -- 172.21.15.61:0/3997459845 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f939c00d890 con 0x7f93ac0b2210 2022-07-20T03:17:16.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 -- 172.21.15.61:0/3997459845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0b2210 msgr2=0x7f93ac140860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 --2- 172.21.15.61:0/3997459845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0b2210 0x7f93ac140860 secure :-1 s=READY pgs=624 cs=0 l=1 rev1=1 rx=0x7f939c002f70 tx=0x7f939c006df0).stop 2022-07-20T03:17:16.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 -- 172.21.15.61:0/3997459845 shutdown_connections 2022-07-20T03:17:16.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 --2- 172.21.15.61:0/3997459845 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0b2210 0x7f93ac140860 unknown :-1 s=CLOSED pgs=624 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 -- 172.21.15.61:0/3997459845 >> 172.21.15.61:0/3997459845 conn(0x7f93ac0b36c0 msgr2=0x7f93ac0b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:16.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 -- 172.21.15.61:0/3997459845 shutdown_connections 2022-07-20T03:17:16.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.859+0000 7f93b1228700 1 -- 172.21.15.61:0/3997459845 wait complete. 2022-07-20T03:17:16.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 Processor -- start 2022-07-20T03:17:16.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 -- start start 2022-07-20T03:17:16.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 0x7f93ac137980 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0ac680 0x7f93ac099e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93ac0b2210 0x7f93ac09a300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac143ae0 con 0x7f93ac0b2210 2022-07-20T03:17:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac09bbc0 con 0x7f93ac0ac680 2022-07-20T03:17:16.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f93ac09b6c0 con 0x7f93ac0967e0 2022-07-20T03:17:16.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93aad9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 0x7f93ac137980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93ac0b2210 0x7f93ac09a300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93aa59c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0ac680 0x7f93ac099e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.874 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93aad9d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 0x7f93ac137980 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:37894/0 (socket says 172.21.15.61:37894) 2022-07-20T03:17:16.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93aad9d700 1 -- 172.21.15.61:0/1785147418 learned_addr learned my addr 172.21.15.61:0/1785147418 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:16.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 -- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 msgr2=0x7f93ac137980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 0x7f93ac137980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.875 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 -- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0ac680 msgr2=0x7f93ac099e10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:16.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0ac680 0x7f93ac099e10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:16.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 -- 172.21.15.61:0/1785147418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f939c00b180 con 0x7f93ac0b2210 2022-07-20T03:17:16.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93aad9d700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 0x7f93ac137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:16.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93ab59e700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93ac0b2210 0x7f93ac09a300 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f93a4001490 tx=0x7f93a4007d60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93a4005ba0 con 0x7f93ac0b2210 2022-07-20T03:17:16.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f93a4005d00 con 0x7f93ac0b2210 2022-07-20T03:17:16.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93ac09b920 con 0x7f93ac0b2210 2022-07-20T03:17:16.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.863+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f93a40072c0 con 0x7f93ac0b2210 2022-07-20T03:17:16.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93ac13f840 con 0x7f93ac0b2210 2022-07-20T03:17:16.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f93a4010070 con 0x7f93ac0b2210 2022-07-20T03:17:16.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f939bfff700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9388053a60 0x7f9388055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:16.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93ac0b2840 con 0x7f93ac0b2210 2022-07-20T03:17:16.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f93a406b6d0 con 0x7f93ac0b2210 2022-07-20T03:17:16.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f93aad9d700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9388053a60 0x7f9388055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:16.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.867+0000 7f93aad9d700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9388053a60 0x7f9388055f20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f939c01a000 tx=0x7f939c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:16.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:16.875+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f93a4042630 con 0x7f93ac0b2210 2022-07-20T03:17:17.006 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f93ac0574e0 con 0x7f93ac0b2210 2022-07-20T03:17:17.007 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f939bfff700 1 -- 172.21.15.61:0/1785147418 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7f93a404a070 con 0x7f93ac0b2210 2022-07-20T03:17:17.008 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:17.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9388053a60 msgr2=0x7f9388055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9388053a60 0x7f9388055f20 secure :-1 s=READY pgs=673 cs=0 l=1 rev1=1 rx=0x7f939c01a000 tx=0x7f939c014000).stop 2022-07-20T03:17:17.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93ac0b2210 msgr2=0x7f93ac09a300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93ac0b2210 0x7f93ac09a300 secure :-1 s=READY pgs=1402 cs=0 l=1 rev1=1 rx=0x7f93a4001490 tx=0x7f93a4007d60).stop 2022-07-20T03:17:17.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 shutdown_connections 2022-07-20T03:17:17.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9388053a60 0x7f9388055f20 unknown :-1 s=CLOSED pgs=673 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.011 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93ac0ac680 0x7f93ac099e10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93ac0b2210 0x7f93ac09a300 unknown :-1 s=CLOSED pgs=1402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 --2- 172.21.15.61:0/1785147418 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93ac0967e0 0x7f93ac137980 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 >> 172.21.15.61:0/1785147418 conn(0x7f93ac0b36c0 msgr2=0x7f93ac096c50 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.012 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 shutdown_connections 2022-07-20T03:17:17.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.003+0000 7f93b1228700 1 -- 172.21.15.61:0/1785147418 wait complete. 2022-07-20T03:17:17.021 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:17:17.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 Processor -- start 2022-07-20T03:17:17.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 -- start start 2022-07-20T03:17:17.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce24141610 0x7fce24060710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24060000 0x7fce24139ef0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fce24144e00 con 0x7fce240b1d40 2022-07-20T03:17:17.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce240b2120 con 0x7fce24141610 2022-07-20T03:17:17.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.127+0000 7fce2a0c2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce24144b60 con 0x7fce24060000 2022-07-20T03:17:17.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce288bf700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce24141610 0x7fce24060710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce290c0700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fce240b1d40 0x7fce24140f00 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:56332/0 (socket says 172.21.15.61:56332) 2022-07-20T03:17:17.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce290c0700 1 -- 172.21.15.61:0/1313594223 learned_addr learned my addr 172.21.15.61:0/1313594223 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:17.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce298c1700 1 --2- 172.21.15.61:0/1313594223 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24060000 0x7fce24139ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce23fff700 1 -- 172.21.15.61:0/1313594223 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3147468638 0 0) 0x7fce24144e00 con 0x7fce240b1d40 2022-07-20T03:17:17.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce288bf700 1 -- 172.21.15.61:0/1313594223 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24060000 msgr2=0x7fce24139ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:17.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce288bf700 1 --2- 172.21.15.61:0/1313594223 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24060000 0x7fce24139ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce288bf700 1 -- 172.21.15.61:0/1313594223 >> v1:172.21.15.117:6789/0 conn(0x7fce240b1d40 legacy=0x7fce24140f00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce288bf700 1 -- 172.21.15.61:0/1313594223 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce24140a60 con 0x7fce24141610 2022-07-20T03:17:17.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce288bf700 1 --2- 172.21.15.61:0/1313594223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24141610 0x7fce24060710 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7fce18002f70 tx=0x7fce18006df0).ready entity=mon.0 client_cookie=f461fbd8c836a48a server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce23fff700 1 -- 172.21.15.61:0/1313594223 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce1800d570 con 0x7fce24141610 2022-07-20T03:17:17.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce23fff700 1 -- 172.21.15.61:0/1313594223 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce18000e90 con 0x7fce24141610 2022-07-20T03:17:17.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce23fff700 1 -- 172.21.15.61:0/1313594223 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce1800d8c0 con 0x7fce24141610 2022-07-20T03:17:17.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 -- 172.21.15.61:0/1313594223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24141610 msgr2=0x7fce24060710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/1313594223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24141610 0x7fce24060710 secure :-1 s=READY pgs=1403 cs=0 l=1 rev1=1 rx=0x7fce18002f70 tx=0x7fce18006df0).stop 2022-07-20T03:17:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 -- 172.21.15.61:0/1313594223 shutdown_connections 2022-07-20T03:17:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/1313594223 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24060000 0x7fce24139ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/1313594223 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24141610 0x7fce24060710 unknown :-1 s=CLOSED pgs=1403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 -- 172.21.15.61:0/1313594223 >> 172.21.15.61:0/1313594223 conn(0x7fce240b2af0 msgr2=0x7fce240b2ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 -- 172.21.15.61:0/1313594223 shutdown_connections 2022-07-20T03:17:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.131+0000 7fce2a0c2700 1 -- 172.21.15.61:0/1313594223 wait complete. 2022-07-20T03:17:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 Processor -- start 2022-07-20T03:17:17.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 -- start start 2022-07-20T03:17:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 0x7fce24143210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce240b1d40 0x7fce240ac130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24141610 0x7fce240ac620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce240acb10 con 0x7fce24060000 2022-07-20T03:17:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce24155e20 con 0x7fce240b1d40 2022-07-20T03:17:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce240abc90 con 0x7fce24141610 2022-07-20T03:17:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 0x7fce24143210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 0x7fce24143210 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51236/0 (socket says 172.21.15.61:51236) 2022-07-20T03:17:17.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce298c1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24141610 0x7fce240ac620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 -- 172.21.15.61:0/3023346341 learned_addr learned my addr 172.21.15.61:0/3023346341 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:17.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce288bf700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce240b1d40 0x7fce240ac130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 -- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24141610 msgr2=0x7fce240ac620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24141610 0x7fce240ac620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 -- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce240b1d40 msgr2=0x7fce240ac130 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce240b1d40 0x7fce240ac130 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 -- 172.21.15.61:0/3023346341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce1800b180 con 0x7fce24060000 2022-07-20T03:17:17.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce298c1700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24141610 0x7fce240ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:17.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce288bf700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce240b1d40 0x7fce240ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:17.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce290c0700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 0x7fce24143210 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7fce140012a0 tx=0x7fce14007e30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce14005bf0 con 0x7fce24060000 2022-07-20T03:17:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce24155b30 con 0x7fce24060000 2022-07-20T03:17:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fce14005d50 con 0x7fce24060000 2022-07-20T03:17:17.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fce14004ae0 con 0x7fce24060000 2022-07-20T03:17:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.135+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce24143e20 con 0x7fce24060000 2022-07-20T03:17:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.139+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce240ab3f0 con 0x7fce24060000 2022-07-20T03:17:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.139+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fce1401b070 con 0x7fce24060000 2022-07-20T03:17:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.139+0000 7fce227fc700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce04053ad0 0x7fce04055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.139+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fce1406baf0 con 0x7fce24060000 2022-07-20T03:17:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.139+0000 7fce288bf700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce04053ad0 0x7fce04055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.139+0000 7fce288bf700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce04053ad0 0x7fce04055f90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7fce1801a000 tx=0x7fce18014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.147+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fce140429f0 con 0x7fce24060000 2022-07-20T03:17:17.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7fce24143e20 con 0x7fce24060000 2022-07-20T03:17:17.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce227fc700 1 -- 172.21.15.61:0/3023346341 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7fce24143e20 con 0x7fce24060000 2022-07-20T03:17:17.281 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:17.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce04053ad0 msgr2=0x7fce04055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce04053ad0 0x7fce04055f90 secure :-1 s=READY pgs=674 cs=0 l=1 rev1=1 rx=0x7fce1801a000 tx=0x7fce18014000).stop 2022-07-20T03:17:17.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 msgr2=0x7fce24143210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 0x7fce24143210 secure :-1 s=READY pgs=1404 cs=0 l=1 rev1=1 rx=0x7fce140012a0 tx=0x7fce14007e30).stop 2022-07-20T03:17:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 shutdown_connections 2022-07-20T03:17:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fce04053ad0 0x7fce04055f90 unknown :-1 s=CLOSED pgs=674 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce240b1d40 0x7fce240ac130 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce24141610 0x7fce240ac620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 --2- 172.21.15.61:0/3023346341 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce24060000 0x7fce24143210 unknown :-1 s=CLOSED pgs=1404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 >> 172.21.15.61:0/3023346341 conn(0x7fce240b2af0 msgr2=0x7fce240b50d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 shutdown_connections 2022-07-20T03:17:17.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.275+0000 7fce2a0c2700 1 -- 172.21.15.61:0/3023346341 wait complete. 2022-07-20T03:17:17.292 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:17:17.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 Processor -- start 2022-07-20T03:17:17.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 -- start start 2022-07-20T03:17:17.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d6405d160 0x7f3d640b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3d641461c0 0x7f3d64146650 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3d640b3f10 con 0x7f3d640b2ec0 2022-07-20T03:17:17.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3d640b4ea0 con 0x7f3d641461c0 2022-07-20T03:17:17.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69d13700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640b3790 con 0x7f3d6405d160 2022-07-20T03:17:17.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d68d11700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f3d640b2ec0 0x7f3d64142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47842/0 (socket says 172.21.15.61:47842) 2022-07-20T03:17:17.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d68d11700 1 -- 172.21.15.61:0/421220225 learned_addr learned my addr 172.21.15.61:0/421220225 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:17.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d69512700 1 --2- 172.21.15.61:0/421220225 >> v2:172.21.15.117:3300/0 conn(0x7f3d641461c0 0x7f3d64146650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d63fff700 1 --2- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d6405d160 0x7f3d640b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d637fe700 1 -- 172.21.15.61:0/421220225 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2064029304 0 0) 0x7f3d640b3f10 con 0x7f3d640b2ec0 2022-07-20T03:17:17.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.399+0000 7f3d637fe700 1 -- 172.21.15.61:0/421220225 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f3d44003260 con 0x7f3d640b2ec0 2022-07-20T03:17:17.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d637fe700 1 -- 172.21.15.61:0/421220225 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2153124110 0 0) 0x7f3d44003260 con 0x7f3d640b2ec0 2022-07-20T03:17:17.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69512700 1 -- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d6405d160 msgr2=0x7f3d640b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:17.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69512700 1 --2- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d6405d160 0x7f3d640b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69512700 1 -- 172.21.15.61:0/421220225 >> v1:172.21.15.61:6789/0 conn(0x7f3d640b2ec0 legacy=0x7f3d64142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69512700 1 -- 172.21.15.61:0/421220225 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d640b2620 con 0x7f3d641461c0 2022-07-20T03:17:17.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69512700 1 --2- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d64146650 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f3d54001800 tx=0x7f3d5c0091d0).ready entity=mon.1 client_cookie=dad03257c9f50660 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d637fe700 1 -- 172.21.15.61:0/421220225 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d54001660 con 0x7f3d641461c0 2022-07-20T03:17:17.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d637fe700 1 -- 172.21.15.61:0/421220225 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d5c00f510 con 0x7f3d641461c0 2022-07-20T03:17:17.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d637fe700 1 -- 172.21.15.61:0/421220225 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d5c00f680 con 0x7f3d641461c0 2022-07-20T03:17:17.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 -- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 msgr2=0x7f3d64146650 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 --2- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d64146650 secure :-1 s=READY pgs=626 cs=0 l=1 rev1=1 rx=0x7f3d54001800 tx=0x7f3d5c0091d0).stop 2022-07-20T03:17:17.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 -- 172.21.15.61:0/421220225 shutdown_connections 2022-07-20T03:17:17.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 --2- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d64146650 unknown :-1 s=CLOSED pgs=626 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 --2- 172.21.15.61:0/421220225 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d6405d160 0x7f3d640b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 -- 172.21.15.61:0/421220225 >> 172.21.15.61:0/421220225 conn(0x7f3d64066e20 msgr2=0x7f3d64067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 -- 172.21.15.61:0/421220225 shutdown_connections 2022-07-20T03:17:17.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 -- 172.21.15.61:0/421220225 wait complete. 2022-07-20T03:17:17.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 Processor -- start 2022-07-20T03:17:17.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.403+0000 7f3d69d13700 1 -- start start 2022-07-20T03:17:17.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 0x7f3d6413f130 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d640b2ec0 0x7f3d640af840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d640b7d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640b8830 con 0x7f3d6405d160 2022-07-20T03:17:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640af100 con 0x7f3d641461c0 2022-07-20T03:17:17.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3d640af3a0 con 0x7f3d640b2ec0 2022-07-20T03:17:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 0x7f3d6413f130 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 0x7f3d6413f130 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51250/0 (socket says 172.21.15.61:51250) 2022-07-20T03:17:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 -- 172.21.15.61:0/2823895368 learned_addr learned my addr 172.21.15.61:0/2823895368 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69512700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d640b7d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d63fff700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d640b2ec0 0x7f3d640af840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 -- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d640b2ec0 msgr2=0x7f3d640af840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d640b2ec0 0x7f3d640af840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 -- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 msgr2=0x7f3d640b7d20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d640b7d20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 -- 172.21.15.61:0/2823895368 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d5c009470 con 0x7f3d6405d160 2022-07-20T03:17:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69512700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d640b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d68d11700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 0x7f3d6413f130 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f3d540018c0 tx=0x7f3d540011a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d54007170 con 0x7f3d6405d160 2022-07-20T03:17:17.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3d540072d0 con 0x7f3d6405d160 2022-07-20T03:17:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d640b8210 con 0x7f3d6405d160 2022-07-20T03:17:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3d54004920 con 0x7f3d6405d160 2022-07-20T03:17:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d64155930 con 0x7f3d6405d160 2022-07-20T03:17:17.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.407+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d640b28c0 con 0x7f3d6405d160 2022-07-20T03:17:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.411+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3d54004a80 con 0x7f3d6405d160 2022-07-20T03:17:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.411+0000 7f3d61ffb700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d48053ad0 0x7f3d48055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.411+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3d5406bae0 con 0x7f3d6405d160 2022-07-20T03:17:17.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.411+0000 7f3d63fff700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d48053ad0 0x7f3d48055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.411+0000 7f3d63fff700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d48053ad0 0x7f3d48055f90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f3d4c00b000 tx=0x7f3d4c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.415+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3d54042a40 con 0x7f3d6405d160 2022-07-20T03:17:17.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.543+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f3d640574e0 con 0x7f3d6405d160 2022-07-20T03:17:17.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.543+0000 7f3d61ffb700 1 -- 172.21.15.61:0/2823895368 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7f3d54001b40 con 0x7f3d6405d160 2022-07-20T03:17:17.548 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:17.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.543+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d48053ad0 msgr2=0x7f3d48055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.543+0000 7f3d69d13700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d48053ad0 0x7f3d48055f90 secure :-1 s=READY pgs=675 cs=0 l=1 rev1=1 rx=0x7f3d4c00b000 tx=0x7f3d4c008040).stop 2022-07-20T03:17:17.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.543+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 msgr2=0x7f3d6413f130 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.543+0000 7f3d69d13700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 0x7f3d6413f130 secure :-1 s=READY pgs=1406 cs=0 l=1 rev1=1 rx=0x7f3d540018c0 tx=0x7f3d540011a0).stop 2022-07-20T03:17:17.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 shutdown_connections 2022-07-20T03:17:17.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3d48053ad0 0x7f3d48055f90 unknown :-1 s=CLOSED pgs=675 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3d641461c0 0x7f3d640b7d20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3d640b2ec0 0x7f3d640af840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 --2- 172.21.15.61:0/2823895368 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3d6405d160 0x7f3d6413f130 unknown :-1 s=CLOSED pgs=1406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 >> 172.21.15.61:0/2823895368 conn(0x7f3d64066e20 msgr2=0x7f3d6405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 shutdown_connections 2022-07-20T03:17:17.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.547+0000 7f3d69d13700 1 -- 172.21.15.61:0/2823895368 wait complete. 2022-07-20T03:17:17.562 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:17:17.563 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:17.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88fdd6700 1 Processor -- start 2022-07-20T03:17:17.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88fdd6700 1 -- start start 2022-07-20T03:17:17.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88fdd6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa88805c3f0 0x7fa88814a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88fdd6700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa88805d480 con 0x7fa8880acef0 2022-07-20T03:17:17.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88fdd6700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa8880b21e0 con 0x7fa8880b1e00 2022-07-20T03:17:17.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88fdd6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88805cf40 con 0x7fa88805c3f0 2022-07-20T03:17:17.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88e373700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa8880acef0 0x7fa888146410 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47860/0 (socket says 172.21.15.61:47860) 2022-07-20T03:17:17.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88e373700 1 -- 172.21.15.61:0/3024376162 learned_addr learned my addr 172.21.15.61:0/3024376162 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:17.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88d371700 1 --2- 172.21.15.61:0/3024376162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa88805c3f0 0x7fa88814a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2401694879 0 0) 0x7fa88805d480 con 0x7fa8880acef0 2022-07-20T03:17:17.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa874003260 con 0x7fa8880acef0 2022-07-20T03:17:17.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1637603637 0 0) 0x7fa8880b21e0 con 0x7fa8880b1e00 2022-07-20T03:17:17.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fa88805d480 con 0x7fa8880b1e00 2022-07-20T03:17:17.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2714282372 0 0) 0x7fa874003260 con 0x7fa8880acef0 2022-07-20T03:17:17.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88d371700 1 -- 172.21.15.61:0/3024376162 >> v1:172.21.15.117:6789/0 conn(0x7fa8880b1e00 legacy=0x7fa8880ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88d371700 1 -- 172.21.15.61:0/3024376162 >> v1:172.21.15.61:6789/0 conn(0x7fa8880acef0 legacy=0x7fa888146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88d371700 1 -- 172.21.15.61:0/3024376162 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa88805d1e0 con 0x7fa88805c3f0 2022-07-20T03:17:17.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.663+0000 7fa88d371700 1 --2- 172.21.15.61:0/3024376162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa88805c3f0 0x7fa88814a6d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa878001110 tx=0x7fa87c006df0).ready entity=mon.2 client_cookie=d93731c2086e1760 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa878001660 con 0x7fa88805c3f0 2022-07-20T03:17:17.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa87c000e90 con 0x7fa88805c3f0 2022-07-20T03:17:17.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88cb70700 1 -- 172.21.15.61:0/3024376162 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa87c012460 con 0x7fa88805c3f0 2022-07-20T03:17:17.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- 172.21.15.61:0/3024376162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa88805c3f0 msgr2=0x7fa88814a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/3024376162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa88805c3f0 0x7fa88814a6d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fa878001110 tx=0x7fa87c006df0).stop 2022-07-20T03:17:17.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- 172.21.15.61:0/3024376162 shutdown_connections 2022-07-20T03:17:17.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/3024376162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa88805c3f0 0x7fa88814a6d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- 172.21.15.61:0/3024376162 >> 172.21.15.61:0/3024376162 conn(0x7fa8880611c0 msgr2=0x7fa888062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- 172.21.15.61:0/3024376162 shutdown_connections 2022-07-20T03:17:17.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- 172.21.15.61:0/3024376162 wait complete. 2022-07-20T03:17:17.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 Processor -- start 2022-07-20T03:17:17.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- start start 2022-07-20T03:17:17.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8880acef0 0x7fa8880b50d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8880b1e00 0x7fa888145a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 0x7fa8881567c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880b4420 con 0x7fa888067550 2022-07-20T03:17:17.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa8880656e0 con 0x7fa8880acef0 2022-07-20T03:17:17.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88fdd6700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa88815a7b0 con 0x7fa8880b1e00 2022-07-20T03:17:17.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88e373700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 0x7fa8881567c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.667+0000 7fa88e373700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 0x7fa8881567c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51264/0 (socket says 172.21.15.61:51264) 2022-07-20T03:17:17.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 -- 172.21.15.61:0/2132254070 learned_addr learned my addr 172.21.15.61:0/2132254070 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:17.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88d371700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8880b1e00 0x7fa888145a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88db72700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8880acef0 0x7fa8880b50d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 -- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8880b1e00 msgr2=0x7fa888145a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8880b1e00 0x7fa888145a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 -- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8880acef0 msgr2=0x7fa8880b50d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8880acef0 0x7fa8880b50d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 -- 172.21.15.61:0/2132254070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa87c00b180 con 0x7fa888067550 2022-07-20T03:17:17.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88e373700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 0x7fa8881567c0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7fa88815a940 tx=0x7fa878001660).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa878007170 con 0x7fa888067550 2022-07-20T03:17:17.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa8780072d0 con 0x7fa888067550 2022-07-20T03:17:17.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa8880b5a70 con 0x7fa888067550 2022-07-20T03:17:17.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa878004710 con 0x7fa888067550 2022-07-20T03:17:17.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa88815aca0 con 0x7fa888067550 2022-07-20T03:17:17.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa8880574e0 con 0x7fa888067550 2022-07-20T03:17:17.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.671+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa87801b070 con 0x7fa888067550 2022-07-20T03:17:17.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.675+0000 7fa886ffd700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa868053ad0 0x7fa868055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:17.690 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.675+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa87806b400 con 0x7fa888067550 2022-07-20T03:17:17.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.675+0000 7fa88db72700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa868053ad0 0x7fa868055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:17.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.675+0000 7fa88db72700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa868053ad0 0x7fa868055f90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fa88000d000 tx=0x7fa880007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:17.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.679+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa878068e90 con 0x7fa888067550 2022-07-20T03:17:17.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.843+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fa88814d6f0 con 0x7fa888067550 2022-07-20T03:17:17.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa886ffd700 1 -- 172.21.15.61:0/2132254070 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2145 (secure 0 0 0) 0x7fa878047070 con 0x7fa888067550 2022-07-20T03:17:17.851 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:17.852 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:13.001769+0000","modified":"2022-07-20T03:17:14.005556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8466":{"gid":8466,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3390219350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3390219350},{"type":"v1","addr":"172.21.15.61:6833","nonce":3390219350}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T03:17:17.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa868053ad0 msgr2=0x7fa868055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa868053ad0 0x7fa868055f90 secure :-1 s=READY pgs=676 cs=0 l=1 rev1=1 rx=0x7fa88000d000 tx=0x7fa880007da0).stop 2022-07-20T03:17:17.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 msgr2=0x7fa8881567c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:17.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 0x7fa8881567c0 secure :-1 s=READY pgs=1408 cs=0 l=1 rev1=1 rx=0x7fa88815a940 tx=0x7fa878001660).stop 2022-07-20T03:17:17.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 shutdown_connections 2022-07-20T03:17:17.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa868053ad0 0x7fa868055f90 unknown :-1 s=CLOSED pgs=676 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa888067550 0x7fa8881567c0 unknown :-1 s=CLOSED pgs=1408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa8880b1e00 0x7fa888145a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 --2- 172.21.15.61:0/2132254070 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa8880acef0 0x7fa8880b50d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:17.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 >> 172.21.15.61:0/2132254070 conn(0x7fa88805c3f0 msgr2=0x7fa88814c890 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:17.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 shutdown_connections 2022-07-20T03:17:17.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:17.847+0000 7fa88fdd6700 1 -- 172.21.15.61:0/2132254070 wait complete. 2022-07-20T03:17:17.861 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-20T03:17:18.067 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:17:13.001769+0000', 'modified': '2022-07-20T03:17:14.005556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8466': {'gid': 8466, 'name': 'a', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3390219350', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3390219350}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3390219350}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:17:18.068 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:17:18.069 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:17:18.190 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8466,"want_state":"up:active","state":"up:active","rank_uptime":5.1754873750000003,"mdsmap_epoch":77,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":10.013228664} 2022-07-20T03:17:18.190 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8466, "mdsmap_epoch": 77, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 5.175487375, "state": "up:active", "uptime": 10.013228664, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:17:18.191 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:17:18.192 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:17:18.193 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:17:18.193 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:18.194 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:18.204 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:17:18.263 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:18.264 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:18.277 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:18.278 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:18.279 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:18.280 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:17:18.285 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:17:18.286 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:18.332 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:17:18.351 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:18.352 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:18.361 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:18.362 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:18.363 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:18.364 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:17:18.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:17:18.400+0000 7ff096fcf080 -1 init, newargv = 0x5637be969480 newargc=9ceph-fuse[23563]: starting ceph client 2022-07-20T03:17:18.405 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr: 2022-07-20T03:17:18.414 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23563]: starting fuse 2022-07-20T03:17:19.513 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:17:19.536 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:19.537 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:19.593 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:19.595 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:19.598 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:19.599 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:17:19.607 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:17:19.608 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:17:19.608 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:19.609 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:17:19.610 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:17:19.611 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:17:19.612 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:17:19.612 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:17:19.613 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:19.614 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:17:19.615 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:17:19.615 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:17:19.616 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:19.617 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:17:19.618 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:17:19.618 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:17:19.619 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:17:19.620 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:19.620 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:17:19.621 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:17:19.621 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:17:19.622 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:17:19.622 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:17:19.623 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:19.623 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:17:19.623 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:17:19.696 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23563.asok 2022-07-20T03:17:19.701 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23563.asok 2022-07-20T03:17:19.702 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:19.702 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23563.asok status 2022-07-20T03:17:19.792 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:17:19.792 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:17:19.793 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:17:19.794 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:17:19.794 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:17:19.794 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:17:19.795 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:17:19.795 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23563", 2022-07-20T03:17:19.796 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:17:19.796 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:19.797 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:17:19.797 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:17:19.798 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 8503, 2022-07-20T03:17:19.798 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:17:19.799 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:17:19.799 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:17:19.800 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 8503 2022-07-20T03:17:19.800 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:19.801 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:17:19.801 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:17:19.801 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:17:19.802 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1160213514 2022-07-20T03:17:19.802 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:17:19.803 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:19.803 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:17:19.804 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:17:19.804 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:17:19.804 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 1160213514 2022-07-20T03:17:19.805 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:19.805 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.8503 172.21.15.124:0/1160213514", 2022-07-20T03:17:19.805 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/1160213514", 2022-07-20T03:17:19.805 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:17:19.806 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 77, 2022-07-20T03:17:19.806 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 140, 2022-07-20T03:17:19.807 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:17:19.807 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:17:19.808 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:17:19.808 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:17:19.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:19.811 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:19.812 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:19.821 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:17:19.823 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:17:19.824 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:19.825 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:19.885 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:20.004 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 Processor -- start 2022-07-20T03:17:20.005 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 -- start start 2022-07-20T03:17:20.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da00b61d0 0x7f1da01412a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1da0060670 0x7f1da0143e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1da005fb00 0x7f1da0144360 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da00b0f10 con 0x7f1da0060670 2022-07-20T03:17:20.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1da00b65b0 con 0x7f1da005fb00 2022-07-20T03:17:20.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da0141790 con 0x7f1da00b61d0 2022-07-20T03:17:20.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9e7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1da0060670 0x7f1da0143e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9e7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1da0060670 0x7f1da0143e70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51274/0 (socket says 172.21.15.61:51274) 2022-07-20T03:17:20.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9e7fc700 1 -- 172.21.15.61:0/97545137 learned_addr learned my addr 172.21.15.61:0/97545137 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.019 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9effd700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da00b61d0 0x7f1da01412a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:19.999+0000 7f1d9f7fe700 1 --2- 172.21.15.61:0/97545137 >> v2:172.21.15.117:3300/0 conn(0x7f1da005fb00 0x7f1da0144360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9e7fc700 1 -- 172.21.15.61:0/97545137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da00b61d0 msgr2=0x7f1da01412a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:20.020 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9e7fc700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da00b61d0 0x7f1da01412a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9e7fc700 1 -- 172.21.15.61:0/97545137 >> v2:172.21.15.117:3300/0 conn(0x7f1da005fb00 msgr2=0x7f1da0144360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:20.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9e7fc700 1 --2- 172.21.15.61:0/97545137 >> v2:172.21.15.117:3300/0 conn(0x7f1da005fb00 0x7f1da0144360 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9e7fc700 1 -- 172.21.15.61:0/97545137 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1da0140b60 con 0x7f1da0060670 2022-07-20T03:17:20.021 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9effd700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da00b61d0 0x7f1da01412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:20.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9f7fe700 1 --2- 172.21.15.61:0/97545137 >> v2:172.21.15.117:3300/0 conn(0x7f1da005fb00 0x7f1da0144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:20.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9e7fc700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da0060670 0x7f1da0143e70 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f1d94002f70 tx=0x7f1d94006eb0).ready entity=mon.0 client_cookie=6215f7de1831ed07 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9dffb700 1 -- 172.21.15.61:0/97545137 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d9400d570 con 0x7f1da0060670 2022-07-20T03:17:20.022 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9dffb700 1 -- 172.21.15.61:0/97545137 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d94000e90 con 0x7f1da0060670 2022-07-20T03:17:20.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9dffb700 1 -- 172.21.15.61:0/97545137 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d9400d900 con 0x7f1da0060670 2022-07-20T03:17:20.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 -- 172.21.15.61:0/97545137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da0060670 msgr2=0x7f1da0143e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da0060670 0x7f1da0143e70 secure :-1 s=READY pgs=1409 cs=0 l=1 rev1=1 rx=0x7f1d94002f70 tx=0x7f1d94006eb0).stop 2022-07-20T03:17:20.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 -- 172.21.15.61:0/97545137 shutdown_connections 2022-07-20T03:17:20.023 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/97545137 >> v2:172.21.15.117:3300/0 conn(0x7f1da005fb00 0x7f1da0144360 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da0060670 0x7f1da0143e70 unknown :-1 s=CLOSED pgs=1409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/97545137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da00b61d0 0x7f1da01412a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 -- 172.21.15.61:0/97545137 >> 172.21.15.61:0/97545137 conn(0x7f1da0063cf0 msgr2=0x7f1da00b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 -- 172.21.15.61:0/97545137 shutdown_connections 2022-07-20T03:17:20.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 -- 172.21.15.61:0/97545137 wait complete. 2022-07-20T03:17:20.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 Processor -- start 2022-07-20T03:17:20.025 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.003+0000 7f1d9ffff700 1 -- start start 2022-07-20T03:17:20.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1da005fb00 0x7f1da0110a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da0060670 0x7f1da00651a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 0x7f1da0065690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da010f0e0 con 0x7f1da00b61d0 2022-07-20T03:17:20.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1da010eba0 con 0x7f1da005fb00 2022-07-20T03:17:20.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1da010ee40 con 0x7f1da0060670 2022-07-20T03:17:20.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 0x7f1da0065690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 0x7f1da0065690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51282/0 (socket says 172.21.15.61:51282) 2022-07-20T03:17:20.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 -- 172.21.15.61:0/295948433 learned_addr learned my addr 172.21.15.61:0/295948433 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9e7fc700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da0060670 0x7f1da00651a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.028 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9effd700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1da005fb00 0x7f1da0110a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 -- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da0060670 msgr2=0x7f1da00651a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da0060670 0x7f1da00651a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 -- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1da005fb00 msgr2=0x7f1da0110a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1da005fb00 0x7f1da0110a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 -- 172.21.15.61:0/295948433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d9400b1e0 con 0x7f1da00b61d0 2022-07-20T03:17:20.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9f7fe700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 0x7f1da0065690 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f1d98006940 tx=0x7f1d98008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d98011780 con 0x7f1da00b61d0 2022-07-20T03:17:20.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1d980118e0 con 0x7f1da00b61d0 2022-07-20T03:17:20.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1da0066f90 con 0x7f1da00b61d0 2022-07-20T03:17:20.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1d98011c40 con 0x7f1da00b61d0 2022-07-20T03:17:20.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.007+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1da0110f70 con 0x7f1da00b61d0 2022-07-20T03:17:20.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.011+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1d98012420 con 0x7f1da00b61d0 2022-07-20T03:17:20.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.011+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1da0110820 con 0x7f1da00b61d0 2022-07-20T03:17:20.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.011+0000 7f1d8ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d7c0539b0 0x7f1d7c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.011+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f1d9806bc40 con 0x7f1da00b61d0 2022-07-20T03:17:20.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.011+0000 7f1d9effd700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d7c0539b0 0x7f1d7c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.011+0000 7f1d9effd700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d7c0539b0 0x7f1d7c055e70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f1d9000c000 tx=0x7f1d90008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.033 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.015+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1d98042b40 con 0x7f1da00b61d0 2022-07-20T03:17:20.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.143+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f1da0140e00 con 0x7f1da00b61d0 2022-07-20T03:17:20.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.143+0000 7f1d8ffff700 1 -- 172.21.15.61:0/295948433 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v140) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f1d98046160 con 0x7f1da00b61d0 2022-07-20T03:17:20.150 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:20.150 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":140,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:14.005088+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":27,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:10.948609+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"139","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":28,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:12.001181+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"140","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:20.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d7c0539b0 msgr2=0x7f1d7c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d7c0539b0 0x7f1d7c055e70 secure :-1 s=READY pgs=677 cs=0 l=1 rev1=1 rx=0x7f1d9000c000 tx=0x7f1d90008040).stop 2022-07-20T03:17:20.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 msgr2=0x7f1da0065690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 0x7f1da0065690 secure :-1 s=READY pgs=1410 cs=0 l=1 rev1=1 rx=0x7f1d98006940 tx=0x7f1d98008040).stop 2022-07-20T03:17:20.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 shutdown_connections 2022-07-20T03:17:20.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1d7c0539b0 0x7f1d7c055e70 unknown :-1 s=CLOSED pgs=677 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1da00b61d0 0x7f1da0065690 unknown :-1 s=CLOSED pgs=1410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1da0060670 0x7f1da00651a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 --2- 172.21.15.61:0/295948433 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1da005fb00 0x7f1da0110a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 >> 172.21.15.61:0/295948433 conn(0x7f1da0063cf0 msgr2=0x7f1da013f6f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 shutdown_connections 2022-07-20T03:17:20.156 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.147+0000 7f1d9ffff700 1 -- 172.21.15.61:0/295948433 wait complete. 2022-07-20T03:17:20.163 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 27, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:17:10.948609+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '139', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 28, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:17:12.001181+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '140', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:17:20.164 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:17:20.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 Processor -- start 2022-07-20T03:17:20.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 -- start start 2022-07-20T03:17:20.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b9da0 0x7ff8a41468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8a41419a0 0x7ff8a4141e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff8a40ba180 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff8a40ad5b0 con 0x7ff8a41419a0 2022-07-20T03:17:20.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8aa73e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41461b0 con 0x7ff8a40b9da0 2022-07-20T03:17:20.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8a9f3d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8a41419a0 0x7ff8a4141e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8a9f3d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff8a41419a0 0x7ff8a4141e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51290/0 (socket says 172.21.15.61:51290) 2022-07-20T03:17:20.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8a9f3d700 1 -- 172.21.15.61:0/2183054980 learned_addr learned my addr 172.21.15.61:0/2183054980 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.271+0000 7ff8a973c700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b9da0 0x7ff8a41468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.283 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8a9f3d700 1 -- 172.21.15.61:0/2183054980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b9da0 msgr2=0x7ff8a41468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:20.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8a9f3d700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b9da0 0x7ff8a41468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8a9f3d700 1 -- 172.21.15.61:0/2183054980 >> v1:172.21.15.117:6789/0 conn(0x7ff8a40b3cf0 legacy=0x7ff8a4142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8a9f3d700 1 -- 172.21.15.61:0/2183054980 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff8a4146450 con 0x7ff8a41419a0 2022-07-20T03:17:20.284 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8a973c700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b9da0 0x7ff8a41468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:20.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8a9f3d700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a41419a0 0x7ff8a4141e50 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7ff89c0032f0 tx=0x7ff89c009290).ready entity=mon.0 client_cookie=3c9f80c3a5004624 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff893fff700 1 -- 172.21.15.61:0/2183054980 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff89c00b8a0 con 0x7ff8a41419a0 2022-07-20T03:17:20.285 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff893fff700 1 -- 172.21.15.61:0/2183054980 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff89c00ba00 con 0x7ff8a41419a0 2022-07-20T03:17:20.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff893fff700 1 -- 172.21.15.61:0/2183054980 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff89c00bb70 con 0x7ff8a41419a0 2022-07-20T03:17:20.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 -- 172.21.15.61:0/2183054980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a41419a0 msgr2=0x7ff8a4141e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.286 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a41419a0 0x7ff8a4141e50 secure :-1 s=READY pgs=1411 cs=0 l=1 rev1=1 rx=0x7ff89c0032f0 tx=0x7ff89c009290).stop 2022-07-20T03:17:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 -- 172.21.15.61:0/2183054980 shutdown_connections 2022-07-20T03:17:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a41419a0 0x7ff8a4141e50 unknown :-1 s=CLOSED pgs=1411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/2183054980 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b9da0 0x7ff8a41468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 -- 172.21.15.61:0/2183054980 >> 172.21.15.61:0/2183054980 conn(0x7ff8a40678c0 msgr2=0x7ff8a40b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 -- 172.21.15.61:0/2183054980 shutdown_connections 2022-07-20T03:17:20.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 -- 172.21.15.61:0/2183054980 wait complete. 2022-07-20T03:17:20.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 Processor -- start 2022-07-20T03:17:20.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.275+0000 7ff8aa73e700 1 -- start start 2022-07-20T03:17:20.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b3cf0 0x7ff8a40afb10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 0x7ff8a40b16b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a41419a0 0x7ff8a40b1ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40b0000 con 0x7ff8a40b9da0 2022-07-20T03:17:20.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a40b1110 con 0x7ff8a41419a0 2022-07-20T03:17:20.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a4113290 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a8f3b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 0x7ff8a40b16b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a8f3b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 0x7ff8a40b16b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51292/0 (socket says 172.21.15.61:51292) 2022-07-20T03:17:20.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a8f3b700 1 -- 172.21.15.61:0/45443288 learned_addr learned my addr 172.21.15.61:0/45443288 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b3cf0 0x7ff8a40afb10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a9f3d700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a41419a0 0x7ff8a40b1ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 -- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a41419a0 msgr2=0x7ff8a40b1ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a41419a0 0x7ff8a40b1ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 -- 172.21.15.61:0/45443288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 msgr2=0x7ff8a40b16b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 0x7ff8a40b16b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 -- 172.21.15.61:0/45443288 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff89c009530 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a8f3b700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 0x7ff8a40b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:20.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8a973c700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b3cf0 0x7ff8a40afb10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff8a000a000 tx=0x7ff8a0008040).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8a0011780 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff8a00118e0 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a4113530 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff8a0011c40 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a40b0220 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.298 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.279+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a40af910 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.283+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff8a001c460 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.283+0000 7ff8927fc700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff8840539b0 0x7ff884055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.283+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff8a006be50 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.283+0000 7ff8a8f3b700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff8840539b0 0x7ff884055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.283+0000 7ff8a8f3b700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff8840539b0 0x7ff884055e70 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7ff89400d000 tx=0x7ff894009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.291+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff8a0042d50 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.419+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7ff8a41130f0 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.419+0000 7ff8927fc700 1 -- 172.21.15.61:0/45443288 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v140) v1 ==== 110+0+10 (secure 0 0 0) 0x7ff8a41130f0 con 0x7ff8a40b3cf0 2022-07-20T03:17:20.426 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:17:20.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff8840539b0 msgr2=0x7ff884055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff8840539b0 0x7ff884055e70 secure :-1 s=READY pgs=678 cs=0 l=1 rev1=1 rx=0x7ff89400d000 tx=0x7ff894009040).stop 2022-07-20T03:17:20.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b3cf0 msgr2=0x7ff8a40afb10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b3cf0 0x7ff8a40afb10 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ff8a000a000 tx=0x7ff8a0008040).stop 2022-07-20T03:17:20.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 shutdown_connections 2022-07-20T03:17:20.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff8840539b0 0x7ff884055e70 unknown :-1 s=CLOSED pgs=678 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff8a41419a0 0x7ff8a40b1ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff8a40b9da0 0x7ff8a40b16b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 --2- 172.21.15.61:0/45443288 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff8a40b3cf0 0x7ff8a40afb10 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 >> 172.21.15.61:0/45443288 conn(0x7ff8a40678c0 msgr2=0x7ff8a4141db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 shutdown_connections 2022-07-20T03:17:20.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.423+0000 7ff8aa73e700 1 -- 172.21.15.61:0/45443288 wait complete. 2022-07-20T03:17:20.446 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:17:20.554 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7fe3c700 1 Processor -- start 2022-07-20T03:17:20.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7fe3c700 1 -- start start 2022-07-20T03:17:20.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7fe3c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9c78060170 0x7f9c780605e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.557 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7fe3c700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c780afdb0 con 0x7f9c781462a0 2022-07-20T03:17:20.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7fe3c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9c780b0050 con 0x7f9c780988b0 2022-07-20T03:17:20.558 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7fe3c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9c7805fcd0 con 0x7f9c78060170 2022-07-20T03:17:20.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7dbd8700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9c781462a0 0x7f9c78146680 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47898/0 (socket says 172.21.15.61:47898) 2022-07-20T03:17:20.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7dbd8700 1 -- 172.21.15.61:0/2820104906 learned_addr learned my addr 172.21.15.61:0/2820104906 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:20.562 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7e3d9700 1 --2- 172.21.15.61:0/2820104906 >> v2:172.21.15.61:3300/0 conn(0x7f9c78060170 0x7f9c780605e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7cbd6700 1 -- 172.21.15.61:0/2820104906 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2495256919 0 0) 0x7f9c780afdb0 con 0x7f9c781462a0 2022-07-20T03:17:20.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7cbd6700 1 -- 172.21.15.61:0/2820104906 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9c60003260 con 0x7f9c781462a0 2022-07-20T03:17:20.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7cbd6700 1 -- 172.21.15.61:0/2820104906 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1970266146 0 0) 0x7f9c780b0050 con 0x7f9c780988b0 2022-07-20T03:17:20.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7e3d9700 1 -- 172.21.15.61:0/2820104906 >> v1:172.21.15.117:6789/0 conn(0x7f9c780988b0 legacy=0x7f9c781422c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7e3d9700 1 -- 172.21.15.61:0/2820104906 >> v1:172.21.15.61:6789/0 conn(0x7f9c781462a0 legacy=0x7f9c78146680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7e3d9700 1 -- 172.21.15.61:0/2820104906 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c780afae0 con 0x7f9c78060170 2022-07-20T03:17:20.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7e3d9700 1 --2- 172.21.15.61:0/2820104906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c780605e0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f9c74001800 tx=0x7f9c700091d0).ready entity=mon.0 client_cookie=8afb8d8c66282a38 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7cbd6700 1 -- 172.21.15.61:0/2820104906 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c74001660 con 0x7f9c78060170 2022-07-20T03:17:20.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7cbd6700 1 -- 172.21.15.61:0/2820104906 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c7000b580 con 0x7f9c78060170 2022-07-20T03:17:20.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.551+0000 7f9c7cbd6700 1 -- 172.21.15.61:0/2820104906 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c7000b6f0 con 0x7f9c78060170 2022-07-20T03:17:20.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2820104906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 msgr2=0x7f9c780605e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2820104906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c780605e0 secure :-1 s=READY pgs=1413 cs=0 l=1 rev1=1 rx=0x7f9c74001800 tx=0x7f9c700091d0).stop 2022-07-20T03:17:20.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2820104906 shutdown_connections 2022-07-20T03:17:20.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2820104906 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c780605e0 unknown :-1 s=CLOSED pgs=1413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2820104906 >> 172.21.15.61:0/2820104906 conn(0x7f9c780b5f00 msgr2=0x7f9c780b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.572 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2820104906 shutdown_connections 2022-07-20T03:17:20.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2820104906 wait complete. 2022-07-20T03:17:20.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 Processor -- start 2022-07-20T03:17:20.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.555+0000 7f9c7fe3c700 1 -- start start 2022-07-20T03:17:20.573 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c7809eb20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c780988b0 0x7f9c78149f70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c781462a0 0x7f9c78155690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7809f3a0 con 0x7f9c78060170 2022-07-20T03:17:20.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7809ce70 con 0x7f9c780988b0 2022-07-20T03:17:20.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9c7809d110 con 0x7f9c781462a0 2022-07-20T03:17:20.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c7809eb20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c7809eb20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51306/0 (socket says 172.21.15.61:51306) 2022-07-20T03:17:20.575 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 -- 172.21.15.61:0/2361163869 learned_addr learned my addr 172.21.15.61:0/2361163869 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7e3d9700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c781462a0 0x7f9c78155690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7d3d7700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c780988b0 0x7f9c78149f70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 -- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c781462a0 msgr2=0x7f9c78155690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.576 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c781462a0 0x7f9c78155690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 -- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c780988b0 msgr2=0x7f9c78149f70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c780988b0 0x7f9c78149f70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 -- 172.21.15.61:0/2361163869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c70009470 con 0x7f9c78060170 2022-07-20T03:17:20.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7dbd8700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c7809eb20 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f9c740011d0 tx=0x7f9c74001a00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c7400ce10 con 0x7f9c78060170 2022-07-20T03:17:20.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c74007170 con 0x7f9c78060170 2022-07-20T03:17:20.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9c7809d3b0 con 0x7f9c78060170 2022-07-20T03:17:20.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9c74004730 con 0x7f9c78060170 2022-07-20T03:17:20.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.559+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9c7809ca90 con 0x7f9c78060170 2022-07-20T03:17:20.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.563+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9c78146d90 con 0x7f9c78060170 2022-07-20T03:17:20.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.563+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9c7401b070 con 0x7f9c78060170 2022-07-20T03:17:20.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.563+0000 7f9c66ffd700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c58053a40 0x7f9c58055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.563+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f9c7406b650 con 0x7f9c78060170 2022-07-20T03:17:20.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.563+0000 7f9c7d3d7700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c58053a40 0x7f9c58055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.563+0000 7f9c7d3d7700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c58053a40 0x7f9c58055f00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f9c6800d000 tx=0x7f9c68007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.571+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9c740424a0 con 0x7f9c78060170 2022-07-20T03:17:20.696 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.691+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7f9c780af8c0 con 0x7f9c78060170 2022-07-20T03:17:20.697 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.691+0000 7f9c66ffd700 1 -- 172.21.15.61:0/2361163869 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v140) v1 ==== 92+0+10 (secure 0 0 0) 0x7f9c74045ac0 con 0x7f9c78060170 2022-07-20T03:17:20.699 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:20.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c58053a40 msgr2=0x7f9c58055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c58053a40 0x7f9c58055f00 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f9c6800d000 tx=0x7f9c68007a00).stop 2022-07-20T03:17:20.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 msgr2=0x7f9c7809eb20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c7809eb20 secure :-1 s=READY pgs=1414 cs=0 l=1 rev1=1 rx=0x7f9c740011d0 tx=0x7f9c74001a00).stop 2022-07-20T03:17:20.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 shutdown_connections 2022-07-20T03:17:20.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9c58053a40 0x7f9c58055f00 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9c780988b0 0x7f9c78149f70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9c781462a0 0x7f9c78155690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 --2- 172.21.15.61:0/2361163869 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9c78060170 0x7f9c7809eb20 unknown :-1 s=CLOSED pgs=1414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 >> 172.21.15.61:0/2361163869 conn(0x7f9c780b5f00 msgr2=0x7f9c78144db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 shutdown_connections 2022-07-20T03:17:20.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.695+0000 7f9c7fe3c700 1 -- 172.21.15.61:0/2361163869 wait complete. 2022-07-20T03:17:20.713 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:17:20.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 Processor -- start 2022-07-20T03:17:20.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- start start 2022-07-20T03:17:20.833 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ab009df40 0x7f3ab0145680 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f3ab00b4d60 0x7f3ab0145b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f3ab00607e0 con 0x7f3ab0060ee0 2022-07-20T03:17:20.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab0060d20 con 0x7f3ab009df40 2022-07-20T03:17:20.834 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f3ab0060a80 con 0x7f3ab00b4d60 2022-07-20T03:17:20.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ab009df40 0x7f3ab0145680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f3ab009df40 0x7f3ab0145680 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51314/0 (socket says 172.21.15.61:51314) 2022-07-20T03:17:20.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 -- 172.21.15.61:0/543819405 learned_addr learned my addr 172.21.15.61:0/543819405 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aae7fc700 1 --2- 172.21.15.61:0/543819405 >> v2:172.21.15.117:3300/0 conn(0x7f3ab00b4d60 0x7f3ab0145b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 -- 172.21.15.61:0/543819405 >> v2:172.21.15.117:3300/0 conn(0x7f3ab00b4d60 msgr2=0x7f3ab0145b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:20.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 --2- 172.21.15.61:0/543819405 >> v2:172.21.15.117:3300/0 conn(0x7f3ab00b4d60 0x7f3ab0145b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 -- 172.21.15.61:0/543819405 >> v1:172.21.15.117:6789/0 conn(0x7f3ab0060ee0 legacy=0x7f3ab0146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 -- 172.21.15.61:0/543819405 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ab009e320 con 0x7f3ab009df40 2022-07-20T03:17:20.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aae7fc700 1 --2- 172.21.15.61:0/543819405 >> v2:172.21.15.117:3300/0 conn(0x7f3ab00b4d60 0x7f3ab0145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:20.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aaeffd700 1 --2- 172.21.15.61:0/543819405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab009df40 0x7f3ab0145680 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7f3aa0002f70 tx=0x7f3aa0006e50).ready entity=mon.0 client_cookie=f49c8b30d07e1c36 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aadffb700 1 -- 172.21.15.61:0/543819405 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aa000d610 con 0x7f3ab009df40 2022-07-20T03:17:20.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aadffb700 1 -- 172.21.15.61:0/543819405 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa0000e90 con 0x7f3ab009df40 2022-07-20T03:17:20.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3aadffb700 1 -- 172.21.15.61:0/543819405 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aa000d9a0 con 0x7f3ab009df40 2022-07-20T03:17:20.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- 172.21.15.61:0/543819405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab009df40 msgr2=0x7f3ab0145680 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 --2- 172.21.15.61:0/543819405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab009df40 0x7f3ab0145680 secure :-1 s=READY pgs=1415 cs=0 l=1 rev1=1 rx=0x7f3aa0002f70 tx=0x7f3aa0006e50).stop 2022-07-20T03:17:20.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- 172.21.15.61:0/543819405 shutdown_connections 2022-07-20T03:17:20.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 --2- 172.21.15.61:0/543819405 >> v2:172.21.15.117:3300/0 conn(0x7f3ab00b4d60 0x7f3ab0145b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 --2- 172.21.15.61:0/543819405 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab009df40 0x7f3ab0145680 unknown :-1 s=CLOSED pgs=1415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- 172.21.15.61:0/543819405 >> 172.21.15.61:0/543819405 conn(0x7f3ab00b5f30 msgr2=0x7f3ab00b6310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- 172.21.15.61:0/543819405 shutdown_connections 2022-07-20T03:17:20.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.823+0000 7f3ab5312700 1 -- 172.21.15.61:0/543819405 wait complete. 2022-07-20T03:17:20.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 Processor -- start 2022-07-20T03:17:20.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 -- start start 2022-07-20T03:17:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 0x7f3ab00b2f10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ab009df40 0x7f3ab00aeb50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ab00b4d60 0x7f3ab0152840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab009b9a0 con 0x7f3ab0060ee0 2022-07-20T03:17:20.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00b1460 con 0x7f3ab009df40 2022-07-20T03:17:20.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f3ab00b2950 con 0x7f3ab00b4d60 2022-07-20T03:17:20.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 0x7f3ab00b2f10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 0x7f3ab00b2f10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51322/0 (socket says 172.21.15.61:51322) 2022-07-20T03:17:20.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 -- 172.21.15.61:0/2046357910 learned_addr learned my addr 172.21.15.61:0/2046357910 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:20.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aae7fc700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ab009df40 0x7f3ab00aeb50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaf7fe700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ab00b4d60 0x7f3ab0152840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 -- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ab00b4d60 msgr2=0x7f3ab0152840 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ab00b4d60 0x7f3ab0152840 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 -- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ab009df40 msgr2=0x7f3ab00aeb50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ab009df40 0x7f3ab00aeb50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 -- 172.21.15.61:0/2046357910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3aa000b1e0 con 0x7f3ab0060ee0 2022-07-20T03:17:20.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3aaeffd700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 0x7f3ab00b2f10 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7f3aa0002f70 tx=0x7f3aa0015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aa0011860 con 0x7f3ab0060ee0 2022-07-20T03:17:20.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3aa00119c0 con 0x7f3ab0060ee0 2022-07-20T03:17:20.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3aa0011d20 con 0x7f3ab0060ee0 2022-07-20T03:17:20.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ab00b16c0 con 0x7f3ab0060ee0 2022-07-20T03:17:20.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.827+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ab0152290 con 0x7f3ab0060ee0 2022-07-20T03:17:20.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.831+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ab00574e0 con 0x7f3ab0060ee0 2022-07-20T03:17:20.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.831+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3aa001a040 con 0x7f3ab0060ee0 2022-07-20T03:17:20.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.831+0000 7f3a9ffff700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a8c053af0 0x7f3a8c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:20.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.831+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f3aa006ce20 con 0x7f3ab0060ee0 2022-07-20T03:17:20.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.831+0000 7f3aae7fc700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a8c053af0 0x7f3a8c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:20.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.831+0000 7f3aae7fc700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a8c053af0 0x7f3a8c055fb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f3aa400c000 tx=0x7f3aa4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:20.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.835+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3aa0043c70 con 0x7f3ab0060ee0 2022-07-20T03:17:20.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.963+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f3ab0061810 con 0x7f3ab0060ee0 2022-07-20T03:17:20.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3a9ffff700 1 -- 172.21.15.61:0/2046357910 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v140) v1 ==== 104+0+10 (secure 0 0 0) 0x7f3aa0047290 con 0x7f3ab0060ee0 2022-07-20T03:17:20.972 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:20.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a8c053af0 msgr2=0x7f3a8c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a8c053af0 0x7f3a8c055fb0 secure :-1 s=READY pgs=680 cs=0 l=1 rev1=1 rx=0x7f3aa400c000 tx=0x7f3aa4008040).stop 2022-07-20T03:17:20.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 msgr2=0x7f3ab00b2f10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:20.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 0x7f3ab00b2f10 secure :-1 s=READY pgs=1416 cs=0 l=1 rev1=1 rx=0x7f3aa0002f70 tx=0x7f3aa0015000).stop 2022-07-20T03:17:20.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 shutdown_connections 2022-07-20T03:17:20.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3a8c053af0 0x7f3a8c055fb0 unknown :-1 s=CLOSED pgs=680 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f3ab009df40 0x7f3ab00aeb50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f3ab00b4d60 0x7f3ab0152840 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 --2- 172.21.15.61:0/2046357910 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f3ab0060ee0 0x7f3ab00b2f10 unknown :-1 s=CLOSED pgs=1416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:20.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 >> 172.21.15.61:0/2046357910 conn(0x7f3ab00b5f30 msgr2=0x7f3ab005c030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:20.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 shutdown_connections 2022-07-20T03:17:20.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:20.967+0000 7f3ab5312700 1 -- 172.21.15.61:0/2046357910 wait complete. 2022-07-20T03:17:20.989 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:17:21.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 Processor -- start 2022-07-20T03:17:21.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 -- start start 2022-07-20T03:17:21.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8cd40b2ec0 0x7f8cd40b32a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8cd41461c0 0x7f8cd41465f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8cd40b3f10 con 0x7f8cd405d160 2022-07-20T03:17:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd40b4ea0 con 0x7f8cd41461c0 2022-07-20T03:17:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8cd40b3790 con 0x7f8cd40b2ec0 2022-07-20T03:17:21.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8cd41461c0 0x7f8cd41465f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f8cd41461c0 0x7f8cd41465f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51328/0 (socket says 172.21.15.61:51328) 2022-07-20T03:17:21.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 -- 172.21.15.61:0/3910178871 learned_addr learned my addr 172.21.15.61:0/3910178871 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:21.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd97f2700 1 --2- 172.21.15.61:0/3910178871 >> v2:172.21.15.117:3300/0 conn(0x7f8cd40b2ec0 0x7f8cd40b32a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 -- 172.21.15.61:0/3910178871 >> v2:172.21.15.117:3300/0 conn(0x7f8cd40b2ec0 msgr2=0x7f8cd40b32a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:21.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 --2- 172.21.15.61:0/3910178871 >> v2:172.21.15.117:3300/0 conn(0x7f8cd40b2ec0 0x7f8cd40b32a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 -- 172.21.15.61:0/3910178871 >> v1:172.21.15.117:6789/0 conn(0x7f8cd405d160 legacy=0x7f8cd4142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cc3fff700 1 -- 172.21.15.61:0/3910178871 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 689074640 0 0) 0x7f8cd40b3f10 con 0x7f8cd405d160 2022-07-20T03:17:21.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 -- 172.21.15.61:0/3910178871 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cd40b2620 con 0x7f8cd41461c0 2022-07-20T03:17:21.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cd9ff3700 1 --2- 172.21.15.61:0/3910178871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd41461c0 0x7f8cd41465f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f8ccc0032f0 tx=0x7f8ccc009290).ready entity=mon.0 client_cookie=94827162adbc91d9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:21.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cc3fff700 1 -- 172.21.15.61:0/3910178871 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ccc00b870 con 0x7f8cd41461c0 2022-07-20T03:17:21.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cc3fff700 1 -- 172.21.15.61:0/3910178871 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8ccc00b9d0 con 0x7f8cd41461c0 2022-07-20T03:17:21.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cc3fff700 1 -- 172.21.15.61:0/3910178871 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8ccc00bb40 con 0x7f8cd41461c0 2022-07-20T03:17:21.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 -- 172.21.15.61:0/3910178871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd41461c0 msgr2=0x7f8cd41465f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/3910178871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd41461c0 0x7f8cd41465f0 secure :-1 s=READY pgs=1417 cs=0 l=1 rev1=1 rx=0x7f8ccc0032f0 tx=0x7f8ccc009290).stop 2022-07-20T03:17:21.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 -- 172.21.15.61:0/3910178871 shutdown_connections 2022-07-20T03:17:21.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/3910178871 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd41461c0 0x7f8cd41465f0 unknown :-1 s=CLOSED pgs=1417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.103+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/3910178871 >> v2:172.21.15.117:3300/0 conn(0x7f8cd40b2ec0 0x7f8cd40b32a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- 172.21.15.61:0/3910178871 >> 172.21.15.61:0/3910178871 conn(0x7f8cd4066e20 msgr2=0x7f8cd4067200 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:21.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- 172.21.15.61:0/3910178871 shutdown_connections 2022-07-20T03:17:21.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- 172.21.15.61:0/3910178871 wait complete. 2022-07-20T03:17:21.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 Processor -- start 2022-07-20T03:17:21.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- start start 2022-07-20T03:17:21.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 0x7f8cd40b84e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8cd40b2ec0 0x7f8cd413ff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8cd41461c0 0x7f8cd413a740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd4145840 con 0x7f8cd405d160 2022-07-20T03:17:21.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd40b7280 con 0x7f8cd41461c0 2022-07-20T03:17:21.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cda7f4700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8cd40b7520 con 0x7f8cd40b2ec0 2022-07-20T03:17:21.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 0x7f8cd40b84e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 0x7f8cd40b84e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51336/0 (socket says 172.21.15.61:51336) 2022-07-20T03:17:21.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 -- 172.21.15.61:0/704560263 learned_addr learned my addr 172.21.15.61:0/704560263 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:21.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd9ff3700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8cd41461c0 0x7f8cd413a740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd8ff1700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8cd40b2ec0 0x7f8cd413ff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 -- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8cd40b2ec0 msgr2=0x7f8cd413ff30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8cd40b2ec0 0x7f8cd413ff30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 -- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8cd41461c0 msgr2=0x7f8cd413a740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8cd41461c0 0x7f8cd413a740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd97f2700 1 -- 172.21.15.61:0/704560263 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ccc009530 con 0x7f8cd405d160 2022-07-20T03:17:21.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.107+0000 7f8cd9ff3700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8cd41461c0 0x7f8cd413a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cd97f2700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 0x7f8cd40b84e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f8cd000c000 tx=0x7f8cd0006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cd0010780 con 0x7f8cd405d160 2022-07-20T03:17:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8cd000b3b0 con 0x7f8cd405d160 2022-07-20T03:17:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cd4140420 con 0x7f8cd405d160 2022-07-20T03:17:21.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8cd0010ad0 con 0x7f8cd405d160 2022-07-20T03:17:21.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cd41559b0 con 0x7f8cd405d160 2022-07-20T03:17:21.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cd40b78a0 con 0x7f8cd405d160 2022-07-20T03:17:21.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8cd0010c30 con 0x7f8cd405d160 2022-07-20T03:17:21.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cc27fc700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8cb4053ad0 0x7f8cb4055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.111+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f8cd006c160 con 0x7f8cd405d160 2022-07-20T03:17:21.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.115+0000 7f8cd8ff1700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8cb4053ad0 0x7f8cb4055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.115+0000 7f8cd8ff1700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8cb4053ad0 0x7f8cb4055f90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f8cc400d000 tx=0x7f8cc4007a00).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:21.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.119+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8cd0043020 con 0x7f8cd405d160 2022-07-20T03:17:21.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f8cd40b28c0 con 0x7f8cd405d160 2022-07-20T03:17:21.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cc27fc700 1 -- 172.21.15.61:0/704560263 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v140) v1 ==== 100+0+10 (secure 0 0 0) 0x7f8cd0046640 con 0x7f8cd405d160 2022-07-20T03:17:21.253 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:21.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8cb4053ad0 msgr2=0x7f8cb4055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8cb4053ad0 0x7f8cb4055f90 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f8cc400d000 tx=0x7f8cc4007a00).stop 2022-07-20T03:17:21.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 msgr2=0x7f8cd40b84e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 0x7f8cd40b84e0 secure :-1 s=READY pgs=1418 cs=0 l=1 rev1=1 rx=0x7f8cd000c000 tx=0x7f8cd0006df0).stop 2022-07-20T03:17:21.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 shutdown_connections 2022-07-20T03:17:21.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8cb4053ad0 0x7f8cb4055f90 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8cd41461c0 0x7f8cd413a740 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8cd40b2ec0 0x7f8cd413ff30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 --2- 172.21.15.61:0/704560263 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8cd405d160 0x7f8cd40b84e0 unknown :-1 s=CLOSED pgs=1418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.247+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 >> 172.21.15.61:0/704560263 conn(0x7f8cd4066e20 msgr2=0x7f8cd4141510 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:21.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.251+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 shutdown_connections 2022-07-20T03:17:21.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.251+0000 7f8cda7f4700 1 -- 172.21.15.61:0/704560263 wait complete. 2022-07-20T03:17:21.266 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:17:21.267 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:21.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 Processor -- start 2022-07-20T03:17:21.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 -- start start 2022-07-20T03:17:21.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e80b1ef0 0x7f90e8062040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8098ae0 0x7f90e80ac830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f90e8061050 0x7f90e80614a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e80b22d0 con 0x7f90e80b1ef0 2022-07-20T03:17:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f90e80abe50 con 0x7f90e8061050 2022-07-20T03:17:21.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90ecbe2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80ac0f0 con 0x7f90e8098ae0 2022-07-20T03:17:21.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e80b1ef0 0x7f90e8062040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90e77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f90e80b1ef0 0x7f90e8062040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51344/0 (socket says 172.21.15.61:51344) 2022-07-20T03:17:21.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90e77fe700 1 -- 172.21.15.61:0/3876192236 learned_addr learned my addr 172.21.15.61:0/3876192236 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:21.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90dffff700 1 --2- 172.21.15.61:0/3876192236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8098ae0 0x7f90e80ac830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.371+0000 7f90e7fff700 1 --2- 172.21.15.61:0/3876192236 >> v2:172.21.15.117:3300/0 conn(0x7f90e8061050 0x7f90e80614a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e77fe700 1 -- 172.21.15.61:0/3876192236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8098ae0 msgr2=0x7f90e80ac830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e77fe700 1 --2- 172.21.15.61:0/3876192236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8098ae0 0x7f90e80ac830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e77fe700 1 -- 172.21.15.61:0/3876192236 >> v2:172.21.15.117:3300/0 conn(0x7f90e8061050 msgr2=0x7f90e80614a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:21.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e77fe700 1 --2- 172.21.15.61:0/3876192236 >> v2:172.21.15.117:3300/0 conn(0x7f90e8061050 0x7f90e80614a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e77fe700 1 -- 172.21.15.61:0/3876192236 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90e80ac390 con 0x7f90e80b1ef0 2022-07-20T03:17:21.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e77fe700 1 --2- 172.21.15.61:0/3876192236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e8062040 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7f90d8002f70 tx=0x7f90d8006e50).ready entity=mon.0 client_cookie=4b47105d1feba5b2 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:21.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e6ffd700 1 -- 172.21.15.61:0/3876192236 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d800d570 con 0x7f90e80b1ef0 2022-07-20T03:17:21.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e6ffd700 1 -- 172.21.15.61:0/3876192236 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90d8000e90 con 0x7f90e80b1ef0 2022-07-20T03:17:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90e6ffd700 1 -- 172.21.15.61:0/3876192236 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90d800d900 con 0x7f90e80b1ef0 2022-07-20T03:17:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 -- 172.21.15.61:0/3876192236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 msgr2=0x7f90e8062040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/3876192236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e8062040 secure :-1 s=READY pgs=1419 cs=0 l=1 rev1=1 rx=0x7f90d8002f70 tx=0x7f90d8006e50).stop 2022-07-20T03:17:21.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 -- 172.21.15.61:0/3876192236 shutdown_connections 2022-07-20T03:17:21.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/3876192236 >> v2:172.21.15.117:3300/0 conn(0x7f90e8061050 0x7f90e80614a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/3876192236 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8098ae0 0x7f90e80ac830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/3876192236 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e8062040 unknown :-1 s=CLOSED pgs=1419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 -- 172.21.15.61:0/3876192236 >> 172.21.15.61:0/3876192236 conn(0x7f90e80b2ca0 msgr2=0x7f90e80b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 -- 172.21.15.61:0/3876192236 shutdown_connections 2022-07-20T03:17:21.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.375+0000 7f90ecbe2700 1 -- 172.21.15.61:0/3876192236 wait complete. 2022-07-20T03:17:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 Processor -- start 2022-07-20T03:17:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 -- start start 2022-07-20T03:17:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8061050 0x7f90e80afd70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e8098ae0 0x7f90e814ab80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e814b070 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e815a640 con 0x7f90e80b1ef0 2022-07-20T03:17:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90e80b0260 con 0x7f90e8098ae0 2022-07-20T03:17:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f90e815a2a0 con 0x7f90e8061050 2022-07-20T03:17:21.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e814b070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e814b070 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51352/0 (socket says 172.21.15.61:51352) 2022-07-20T03:17:21.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 -- 172.21.15.61:0/2093918937 learned_addr learned my addr 172.21.15.61:0/2093918937 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:21.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8061050 0x7f90e80afd70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90dffff700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e8098ae0 0x7f90e814ab80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 -- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8061050 msgr2=0x7f90e80afd70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8061050 0x7f90e80afd70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 -- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e8098ae0 msgr2=0x7f90e814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e8098ae0 0x7f90e814ab80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 -- 172.21.15.61:0/2093918937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90d800b1e0 con 0x7f90e80b1ef0 2022-07-20T03:17:21.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e7fff700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e814b070 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f90e0006e00 tx=0x7f90e00091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:21.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e000c850 con 0x7f90e80b1ef0 2022-07-20T03:17:21.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f90e000c9b0 con 0x7f90e80b1ef0 2022-07-20T03:17:21.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90e809bec0 con 0x7f90e80b1ef0 2022-07-20T03:17:21.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f90e000cd10 con 0x7f90e80b1ef0 2022-07-20T03:17:21.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.379+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90e8148590 con 0x7f90e80b1ef0 2022-07-20T03:17:21.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.383+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90e809bc00 con 0x7f90e80b1ef0 2022-07-20T03:17:21.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.383+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f90e00123f0 con 0x7f90e80b1ef0 2022-07-20T03:17:21.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.383+0000 7f90e57fa700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90cc053940 0x7f90cc055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:21.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.383+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f90e006c170 con 0x7f90e80b1ef0 2022-07-20T03:17:21.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.383+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90cc053940 0x7f90cc055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:21.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.383+0000 7f90e77fe700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90cc053940 0x7f90cc055e00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f90d8018000 tx=0x7f90d8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:21.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.391+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f90e809bc00 con 0x7f90e80b1ef0 2022-07-20T03:17:21.560 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.555+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f90e809bc00 con 0x7f90e80b1ef0 2022-07-20T03:17:21.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.555+0000 7f90e57fa700 1 -- 172.21.15.61:0/2093918937 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f90e809bc00 con 0x7f90e80b1ef0 2022-07-20T03:17:21.561 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:21.561 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:13.001769+0000","modified":"2022-07-20T03:17:14.005556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8466":{"gid":8466,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3390219350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3390219350},{"type":"v1","addr":"172.21.15.61:6833","nonce":3390219350}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T03:17:21.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90cc053940 msgr2=0x7f90cc055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90cc053940 0x7f90cc055e00 secure :-1 s=READY pgs=682 cs=0 l=1 rev1=1 rx=0x7f90d8018000 tx=0x7f90d8014000).stop 2022-07-20T03:17:21.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 msgr2=0x7f90e814b070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:21.563 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e814b070 secure :-1 s=READY pgs=1420 cs=0 l=1 rev1=1 rx=0x7f90e0006e00 tx=0x7f90e00091d0).stop 2022-07-20T03:17:21.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 shutdown_connections 2022-07-20T03:17:21.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f90cc053940 0x7f90cc055e00 unknown :-1 s=CLOSED pgs=682 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f90e8098ae0 0x7f90e814ab80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f90e80b1ef0 0x7f90e814b070 unknown :-1 s=CLOSED pgs=1420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 --2- 172.21.15.61:0/2093918937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f90e8061050 0x7f90e80afd70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:21.567 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 >> 172.21.15.61:0/2093918937 conn(0x7f90e80b2ca0 msgr2=0x7f90e81470d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:21.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 shutdown_connections 2022-07-20T03:17:21.568 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:21.559+0000 7f90ecbe2700 1 -- 172.21.15.61:0/2093918937 wait complete. 2022-07-20T03:17:21.568 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-20T03:17:21.775 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 77, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:17:13.001769+0000', 'modified': '2022-07-20T03:17:14.005556+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8466}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8466': {'gid': 8466, 'name': 'a', 'rank': 0, 'incarnation': 76, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.61:6833/3390219350', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.61:6832', 'nonce': 3390219350}, {'type': 'v1', 'addr': '172.21.15.61:6833', 'nonce': 3390219350}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [28], 'metadata_pool': 27, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:17:21.777 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:17:21.778 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.a.asok --format=json status 2022-07-20T03:17:21.897 INFO:teuthology.orchestra.run.smithi061.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8466,"want_state":"up:active","state":"up:active","rank_uptime":8.8829039600000002,"mdsmap_epoch":77,"osdmap_epoch":140,"osdmap_epoch_barrier":139,"uptime":13.720665918} 2022-07-20T03:17:21.899 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8466, "mdsmap_epoch": 77, "osdmap_epoch": 140, "osdmap_epoch_barrier": 139, "rank_uptime": 8.88290396, "state": "up:active", "uptime": 13.720665918, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:17:21.900 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:17:21.900 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:17:21.901 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:17:21.901 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:21.902 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:21.919 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:17:21.978 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:21.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:21.998 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:22.000 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:22.001 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.001 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:17:22.048 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:17:22.050 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:17:22.077 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:22.078 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:22.141 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:17:22.134+0000 7fe638698080 -1 init, newargv = 0x5592154e1470 newargc=9 2022-07-20T03:17:22.142 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[23031]: starting ceph client 2022-07-20T03:17:22.144 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:22.144 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:22.146 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.147 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:17:22.148 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[23031]: starting fuse 2022-07-20T03:17:22.166 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:17:22.167 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:17:22.167 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.168 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:17:22.169 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:17:22.170 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:17:22.171 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:17:22.171 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:17:22.175 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:22.176 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:17:22.176 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:17:22.177 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:17:22.177 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:22.178 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:17:22.178 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:17:22.178 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:17:22.179 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:17:22.179 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:22.180 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:17:22.181 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:17:22.181 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:17:22.182 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:17:22.182 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:17:22.183 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:22.184 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:17:22.185 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:17:22.257 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.23031.asok 2022-07-20T03:17:22.262 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23031.asok 2022-07-20T03:17:22.264 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.265 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23031.asok status 2022-07-20T03:17:22.362 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:17:22.363 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:17:22.363 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:17:22.364 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:17:22.365 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:17:22.365 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:17:22.365 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:17:22.366 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "23031", 2022-07-20T03:17:22.366 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:17:22.367 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:22.367 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:17:22.368 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:17:22.368 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 7316, 2022-07-20T03:17:22.368 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:17:22.369 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:17:22.369 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:17:22.369 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 7316 2022-07-20T03:17:22.369 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:22.370 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:17:22.370 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:17:22.370 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:17:22.370 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2545732937 2022-07-20T03:17:22.373 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:17:22.373 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:22.373 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:17:22.374 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:17:22.374 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:17:22.374 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 2545732937 2022-07-20T03:17:22.375 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:22.375 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.7316 172.21.15.163:0/2545732937", 2022-07-20T03:17:22.376 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/2545732937", 2022-07-20T03:17:22.376 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:17:22.377 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 77, 2022-07-20T03:17:22.377 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 140, 2022-07-20T03:17:22.377 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:17:22.378 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:17:22.378 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:17:22.379 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:17:22.381 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.382 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:22.383 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.392 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:17:22.393 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:22.394 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.395 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.413 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T03:17:22.414 DEBUG:teuthology.orchestra.run.smithi124:> cd /home/ubuntu/cephtest/mnt.0 && sudo getfattr --only-values -n ceph.dir.rctime . 2022-07-20T03:17:22.450 INFO:teuthology.orchestra.run.smithi124.stdout:1658287042.411297835 2022-07-20T03:17:22.451 INFO:tasks.cephfs.test_misc:rctime = 1658287042.411297835 2022-07-20T03:17:22.452 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:17:22.452 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:22.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:22.503 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:22.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:22.549 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:22.572 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:17:22.573 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.574 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:22.575 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:22.587 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:17:22.587 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:17:22.588 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi124.front.sepia.ceph.com... 2022-07-20T03:17:22.589 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:17:22.590 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:22.604 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23563]: fuse finished with error 0 and tester_r 0 2022-07-20T03:17:22.609 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.610 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:22.611 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:22.659 INFO:teuthology.orchestra.run.smithi124.stdout:ext2/ext3 2022-07-20T03:17:22.660 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:17:22.777 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:17:22.778 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:22.786 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.786 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:22.787 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.795 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:17:22.797 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:22.798 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:17:22.799 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:17:22.799 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.857 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[23031]: fuse finished with error 0 and tester_r 0 2022-07-20T03:17:22.859 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.860 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:22.861 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.880 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:17:22.881 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:17:22.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:17:22.926 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:22.934 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:22.935 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:22.936 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:17:22.944 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:17:22.945 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:22.947 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:17:22.948 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:17:22.949 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:17:22.958 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:22.960 INFO:teuthology.orchestra.run.smithi179.stderr:rmdir: failed to remove '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:17:22.960 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:17:23.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.071+0000 7f7a8ffff700 1 Processor -- start 2022-07-20T03:17:23.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.071+0000 7f7a8ffff700 1 -- start start 2022-07-20T03:17:23.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.071+0000 7f7a8ffff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f7a90063d60 0x7f7a900b57e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.071+0000 7f7a8ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a90064140 con 0x7f7a900b1440 2022-07-20T03:17:23.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.071+0000 7f7a8ffff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7a900ad900 con 0x7f7a900af150 2022-07-20T03:17:23.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.071+0000 7f7a8ffff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7a90143800 con 0x7f7a90063d60 2022-07-20T03:17:23.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8e7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7a900b1440 0x7f7a900acae0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:47956/0 (socket says 172.21.15.61:47956) 2022-07-20T03:17:23.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8e7fc700 1 -- 172.21.15.61:0/2934654019 learned_addr learned my addr 172.21.15.61:0/2934654019 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:23.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8effd700 1 --2- 172.21.15.61:0/2934654019 >> v2:172.21.15.117:3300/0 conn(0x7f7a90063d60 0x7f7a900b57e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8dffb700 1 -- 172.21.15.61:0/2934654019 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2963498251 0 0) 0x7f7a90064140 con 0x7f7a900b1440 2022-07-20T03:17:23.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8dffb700 1 -- 172.21.15.61:0/2934654019 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7a70003260 con 0x7f7a900b1440 2022-07-20T03:17:23.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8effd700 1 -- 172.21.15.61:0/2934654019 >> v1:172.21.15.117:6789/0 conn(0x7f7a900af150 legacy=0x7f7a900ad1f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8effd700 1 -- 172.21.15.61:0/2934654019 >> v1:172.21.15.61:6789/0 conn(0x7f7a900b1440 legacy=0x7f7a900acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8effd700 1 -- 172.21.15.61:0/2934654019 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a90143aa0 con 0x7f7a90063d60 2022-07-20T03:17:23.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8effd700 1 --2- 172.21.15.61:0/2934654019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a90063d60 0x7f7a900b57e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f7a78002f70 tx=0x7f7a78006df0).ready entity=mon.1 client_cookie=9b18512546900ff9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8dffb700 1 -- 172.21.15.61:0/2934654019 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a7800d610 con 0x7f7a90063d60 2022-07-20T03:17:23.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8dffb700 1 -- 172.21.15.61:0/2934654019 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a78000e90 con 0x7f7a90063d60 2022-07-20T03:17:23.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8dffb700 1 -- 172.21.15.61:0/2934654019 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a7800d960 con 0x7f7a90063d60 2022-07-20T03:17:23.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2934654019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a90063d60 msgr2=0x7f7a900b57e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2934654019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a90063d60 0x7f7a900b57e0 secure :-1 s=READY pgs=638 cs=0 l=1 rev1=1 rx=0x7f7a78002f70 tx=0x7f7a78006df0).stop 2022-07-20T03:17:23.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2934654019 shutdown_connections 2022-07-20T03:17:23.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2934654019 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a90063d60 0x7f7a900b57e0 unknown :-1 s=CLOSED pgs=638 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2934654019 >> 172.21.15.61:0/2934654019 conn(0x7f7a900634e0 msgr2=0x7f7a900b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:23.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2934654019 shutdown_connections 2022-07-20T03:17:23.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.075+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2934654019 wait complete. 2022-07-20T03:17:23.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 Processor -- start 2022-07-20T03:17:23.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 -- start start 2022-07-20T03:17:23.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 0x7f7a90064f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a900af150 0x7f7a90066b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a900b1440 0x7f7a900b3b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a9012fe90 con 0x7f7a90063d60 2022-07-20T03:17:23.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a900673c0 con 0x7f7a900af150 2022-07-20T03:17:23.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7a90130fd0 con 0x7f7a900b1440 2022-07-20T03:17:23.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 0x7f7a90064f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 0x7f7a90064f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51364/0 (socket says 172.21.15.61:51364) 2022-07-20T03:17:23.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 -- 172.21.15.61:0/2198620030 learned_addr learned my addr 172.21.15.61:0/2198620030 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:23.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8e7fc700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a900af150 0x7f7a90066b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8f7fe700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a900b1440 0x7f7a900b3b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 -- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a900b1440 msgr2=0x7f7a900b3b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a900b1440 0x7f7a900b3b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 -- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a900af150 msgr2=0x7f7a90066b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a900af150 0x7f7a90066b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 -- 172.21.15.61:0/2198620030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a7800b180 con 0x7f7a90063d60 2022-07-20T03:17:23.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8effd700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 0x7f7a90064f00 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f7a78002f70 tx=0x7f7a78014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a78013de0 con 0x7f7a90063d60 2022-07-20T03:17:23.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7a7800d290 con 0x7f7a90063d60 2022-07-20T03:17:23.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7a7801e5f0 con 0x7f7a90063d60 2022-07-20T03:17:23.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a90067060 con 0x7f7a90063d60 2022-07-20T03:17:23.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.079+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a90131740 con 0x7f7a90063d60 2022-07-20T03:17:23.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.083+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a90067580 con 0x7f7a90063d60 2022-07-20T03:17:23.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.083+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7a7801c070 con 0x7f7a90063d60 2022-07-20T03:17:23.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.083+0000 7f7a87fff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a74053ad0 0x7f7a74055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.083+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f7a7806c8e0 con 0x7f7a90063d60 2022-07-20T03:17:23.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.083+0000 7f7a8e7fc700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a74053ad0 0x7f7a74055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.083+0000 7f7a8e7fc700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a74053ad0 0x7f7a74055f90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f7a8000d000 tx=0x7f7a80009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.091+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7a78043840 con 0x7f7a90063d60 2022-07-20T03:17:23.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.219+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f7a900671f0 con 0x7f7a90063d60 2022-07-20T03:17:23.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a87fff700 1 -- 172.21.15.61:0/2198620030 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v140) v1 ==== 81+0+33707 (secure 0 0 0) 0x7f7a78023020 con 0x7f7a90063d60 2022-07-20T03:17:23.228 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:23.228 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:17:23.229 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 140, 2022-07-20T03:17:23.229 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:17:23.230 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:17:23.230 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:17:14.005088+0000", 2022-07-20T03:17:23.231 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:23.231 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:23.232 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:17:23.232 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:17:23.233 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:17:23.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:17:23.233 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:17:23.234 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:17:23.234 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:17:23.235 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:23.235 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:17:23.236 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:17:23.236 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:17:23.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:17:23.237 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:17:23.237 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 28, 2022-07-20T03:17:23.238 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:17:23.238 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:17:23.239 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:17:23.239 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:17:23.240 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:17:23.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.240 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:17:23.241 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:17:23.241 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:17:23.241 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:23.242 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:23.242 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:23.243 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:23.243 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:23.243 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:23.243 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:23.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:23.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:17:23.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:17:23.244 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:17:23.245 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:17:23.245 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:17:23.245 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:23.245 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:23.246 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:23.246 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:23.246 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:23.246 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:23.247 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:23.247 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.247 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:17:23.247 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:23.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:23.248 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:23.248 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:23.248 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:23.248 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:23.249 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:23.249 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:23.249 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:23.249 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:23.250 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:23.250 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:23.250 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:23.250 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:23.251 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:23.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:23.251 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:23.251 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:23.251 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:23.252 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:23.252 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:23.252 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:23.252 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:23.253 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:23.253 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:23.253 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:23.253 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.253 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:23.254 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:23.254 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:23.254 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:23.254 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:23.255 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:23.255 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:23.255 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:23.255 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:23.256 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:23.256 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:23.256 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:17:23.256 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:17:23.256 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.257 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:23.257 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:17:23.257 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.257 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:17:23.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:17:23.258 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:17:23.259 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:23.259 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:23.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:23.259 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:23.259 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:23.260 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:23.260 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:23.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:23.260 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:23.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:23.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:23.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:23.261 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:23.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:23.262 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:23.262 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:23.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:23.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:23.263 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:23.264 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:23.264 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:17:23.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:23.265 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:23.266 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:23.267 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:23.267 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:23.268 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:23.269 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:23.269 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:23.270 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:23.270 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:23.271 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:23.271 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:23.272 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:23.272 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:23.273 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:23.273 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:23.274 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:23.274 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:23.275 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:23.275 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:23.276 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:23.276 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:23.277 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:23.277 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:23.277 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:23.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:23.278 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.278 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:23.278 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:23.279 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:23.279 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:23.279 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:23.279 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:23.280 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:23.280 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:23.280 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:23.281 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:23.281 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:23.281 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:17:23.282 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:23.282 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:17:23.282 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.283 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.283 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.283 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 27, 2022-07-20T03:17:23.284 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:17:23.284 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:17:10.948609+0000", 2022-07-20T03:17:23.284 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:23.284 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:23.285 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:23.285 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:23.285 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:23.286 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:23.286 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:23.287 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:23.287 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:23.287 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:23.288 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:23.288 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:23.289 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:23.289 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:23.290 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:23.290 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:23.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:23.291 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:23.292 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:23.293 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:23.293 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "139", 2022-07-20T03:17:23.294 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:23.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:23.295 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:23.296 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:23.296 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:23.297 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:23.297 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:23.298 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:23.298 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:23.299 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:23.299 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:23.299 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:23.300 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:23.300 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:23.301 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:23.301 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:23.302 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:23.302 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:23.303 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:23.303 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:23.304 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:23.304 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:23.305 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:23.305 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:23.306 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:23.306 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:23.307 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.307 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:23.308 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:23.308 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:23.309 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:23.309 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:23.310 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:23.310 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:23.311 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:23.311 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:23.312 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:23.312 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:23.313 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:17:23.313 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:17:23.314 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:17:23.314 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:17:23.315 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.315 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:23.316 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:17:23.316 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:17:23.317 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.317 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.318 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.318 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.318 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 28, 2022-07-20T03:17:23.319 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:17:23.319 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:17:12.001181+0000", 2022-07-20T03:17:23.319 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:23.319 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:23.320 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:23.320 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:23.320 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:23.320 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:23.321 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:23.321 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:23.321 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:23.321 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:23.322 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:23.322 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:23.322 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:23.322 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:23.322 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:23.323 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:23.323 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:23.323 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:23.323 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:23.324 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:23.324 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.324 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "140", 2022-07-20T03:17:23.324 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:23.324 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:23.325 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:23.325 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:23.325 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:23.325 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:23.326 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:23.326 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:23.326 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:23.326 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:23.326 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:23.327 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:23.327 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:23.327 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:23.327 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:23.328 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:23.328 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:23.328 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:23.328 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:23.328 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:23.329 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:23.329 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:23.329 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:23.329 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:23.330 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:23.330 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:23.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.330 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:23.331 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:23.331 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:23.331 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:23.331 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:23.331 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:23.332 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:23.332 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:23.332 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:23.333 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:23.333 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:23.333 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:17:23.333 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:23.334 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:17:23.334 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:17:23.334 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.334 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.335 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.335 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:23.335 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:17:23.335 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.335 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:23.336 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:17:23.336 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.336 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.336 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.337 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.337 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.337 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.337 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.337 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:23.338 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.338 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.338 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.339 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.339 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.339 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:17:23.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.340 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.340 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.340 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:17:23.340 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.341 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.341 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.341 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.341 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.342 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.342 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.342 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:17:23.342 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.343 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.343 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.343 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.343 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:17:23.344 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.344 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.344 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.344 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.345 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.345 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.345 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.345 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:17:23.346 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.346 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.346 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.346 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.347 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:17:23.347 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.347 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.347 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.348 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.348 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.348 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.348 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.349 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:17:23.349 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.349 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.350 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:17:23.350 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:23.350 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.350 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.351 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.351 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:17:23.351 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:17:23.352 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:17:23.352 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:17:23.352 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.352 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.353 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.353 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.353 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.354 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:23.354 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:17:23.354 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.354 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.355 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.355 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.355 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.355 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.355 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.356 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:23.356 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.356 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.357 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:17:23.358 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.358 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.359 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.359 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:17:23.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.359 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.360 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.360 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.360 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.361 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:17:23.361 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.361 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.361 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.362 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:17:23.362 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.362 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.363 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.363 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.363 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.363 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.364 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.364 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:17:23.364 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.365 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.365 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.365 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:17:23.366 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.366 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.366 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.366 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.367 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.367 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.367 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.367 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.368 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:17:23.368 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.368 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.369 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.369 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:17:23.369 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:23.370 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.370 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.370 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.370 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:17:23.370 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:17:23.371 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:17:23.371 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:17:23.371 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.371 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.372 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.372 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.372 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.373 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:23.373 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:17:23.374 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.374 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.374 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.374 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.375 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.376 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 137, 2022-07-20T03:17:23.376 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.376 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.377 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.378 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.378 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.379 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:17:23.379 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.380 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.380 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.381 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:17:23.382 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.382 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.383 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.383 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.384 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.384 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.385 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.385 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.385 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:17:23.386 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.386 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.387 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.387 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.388 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:17:23.388 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.389 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.390 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.390 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.391 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.391 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.391 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.392 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.392 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:17:23.393 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.393 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.394 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.394 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.395 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:17:23.395 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.396 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.396 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.396 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.396 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.397 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:17:23.397 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.398 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.398 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:17:23.400 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:23.400 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.401 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.402 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:17:23.402 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:17:23.402 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:17:23.403 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:17:23.403 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.404 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.404 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.404 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.405 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.405 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.405 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:23.406 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:17:23.406 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.406 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.407 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.407 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.407 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.407 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.408 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.408 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 127, 2022-07-20T03:17:23.408 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.408 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.409 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.409 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.409 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.409 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.409 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:17:23.410 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.410 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.410 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.411 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:17:23.411 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.411 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.411 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.411 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.412 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.412 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.412 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.412 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.413 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:17:23.413 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.413 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.413 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.413 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:17:23.414 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.414 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.414 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.415 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.415 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.415 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.416 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.416 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:17:23.416 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.416 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:17:23.417 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.417 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.418 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.418 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.418 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.418 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.419 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:17:23.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.419 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.420 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.420 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.420 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:17:23.420 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:23.421 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.421 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.421 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:17:23.421 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:17:23.422 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:17:23.422 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:17:23.422 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.422 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.423 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.423 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.423 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.423 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.424 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:23.424 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:17:23.424 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.424 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.425 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.425 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.425 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.425 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.425 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.426 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 137, 2022-07-20T03:17:23.426 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.426 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.426 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.427 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.427 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.427 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:17:23.428 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.428 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.428 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.428 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:17:23.429 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.429 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.429 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.429 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.430 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.430 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.430 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.430 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:17:23.431 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.431 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.431 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.431 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:17:23.432 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.432 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.432 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.433 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.433 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.433 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.434 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:17:23.434 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.434 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.434 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:17:23.435 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.435 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.436 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.436 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.436 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.437 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.437 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:17:23.437 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.438 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.438 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:17:23.438 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:23.439 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.439 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.439 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.439 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:17:23.439 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:17:23.440 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:17:23.440 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:17:23.440 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.440 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.441 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.441 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.441 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.441 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.442 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:23.442 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:17:23.442 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.442 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.443 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.443 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.443 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.443 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.443 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.444 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:23.444 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.444 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.444 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.445 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.445 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.445 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.445 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:17:23.446 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.446 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.446 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.446 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.447 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:17:23.447 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.447 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.447 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.448 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.448 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.448 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.448 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.448 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.449 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:17:23.449 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.449 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.449 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.450 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.450 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:17:23.450 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.450 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.450 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.451 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.451 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.451 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.452 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.452 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:17:23.452 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.453 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.453 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:17:23.453 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.453 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.454 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.454 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.455 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.455 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.455 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.455 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:17:23.456 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.456 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:17:23.457 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:23.457 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.457 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.458 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.458 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:17:23.458 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:17:23.458 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:17:23.459 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:17:23.459 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.459 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.459 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.459 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.460 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.460 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.460 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:23.460 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:17:23.461 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.461 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.461 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.461 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.462 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.462 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.462 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.462 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:23.462 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.463 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.463 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.463 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.463 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.464 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.464 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:17:23.464 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.464 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.464 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.465 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.465 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:17:23.465 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.466 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.466 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.466 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.466 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.467 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.467 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.467 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.467 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:17:23.468 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.468 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.468 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.468 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.469 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:17:23.469 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.469 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.469 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.470 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.470 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.470 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.470 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.470 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.471 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:17:23.471 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.471 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.471 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.472 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.472 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:17:23.472 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.472 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.473 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.473 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.473 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.473 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.474 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.474 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:17:23.474 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.475 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.475 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.475 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.475 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:17:23.475 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:23.476 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.476 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.476 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.476 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:17:23.477 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:17:23.477 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:17:23.477 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:17:23.477 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.478 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.478 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.478 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.478 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.478 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.479 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:23.479 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:17:23.479 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:23.479 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:23.480 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:23.480 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:23.480 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:23.480 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:23.480 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:23.481 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:23.481 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:23.481 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:23.481 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:23.482 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.482 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.482 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.482 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:17:23.482 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.483 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.483 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.483 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.483 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:17:23.484 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.484 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.484 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.484 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.484 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:23.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.485 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.485 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.485 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:17:23.486 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.486 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.486 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.486 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.487 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:17:23.487 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.487 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.487 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.488 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.488 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:23.488 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.488 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.489 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.489 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:17:23.489 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.489 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.490 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.490 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.490 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:17:23.490 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.491 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.491 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.491 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.492 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:23.492 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:23.492 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.492 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:23.493 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:17:23.493 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.493 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.493 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.494 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:23.494 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:17:23.494 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:23.494 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.495 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.495 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.495 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:17:23.495 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:17:23.496 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:17:23.496 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:17:23.496 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:23.496 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:23.496 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:23.497 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:23.497 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.497 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:23.497 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:17:23.497 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.498 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:23.498 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.498 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.498 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.499 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.499 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.499 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:17:23.499 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.500 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.500 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.500 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:23.500 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.500 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.501 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.501 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.501 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.501 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:17:23.502 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.502 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.502 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.502 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:23.503 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.503 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.503 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.503 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.503 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.504 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:17:23.504 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.504 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.504 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.505 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:23.505 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.505 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.505 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.505 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.506 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.506 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:17:23.506 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.506 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.507 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.507 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:23.507 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.507 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.508 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.508 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.508 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.508 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:17:23.509 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.509 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.509 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.509 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:23.510 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.510 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.510 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.510 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.510 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.511 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:17:23.511 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.511 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.511 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.512 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:23.512 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.512 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.512 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.512 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.513 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.513 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:17:23.513 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.513 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.514 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:23.514 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:23.514 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:23.514 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:23.515 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:23.515 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:23.515 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:23.515 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:17:23.516 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:23.516 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.516 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:23.516 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:17:23.516 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:17:23.517 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:17:23.517 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:17:23.517 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:17:23.517 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:17:23.518 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:17:23.518 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:17:23.518 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:17:23.518 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:17:23.518 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:17:23.519 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:17:23.519 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:17:23.519 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:23.519 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:23.520 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:17:23.520 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:17:23.520 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:17:23.520 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:17:23.521 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:17:23.521 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:23.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a74053ad0 msgr2=0x7f7a74055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a74053ad0 0x7f7a74055f90 secure :-1 s=READY pgs=683 cs=0 l=1 rev1=1 rx=0x7f7a8000d000 tx=0x7f7a80009040).stop 2022-07-20T03:17:23.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 msgr2=0x7f7a90064f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 0x7f7a90064f00 secure :-1 s=READY pgs=1423 cs=0 l=1 rev1=1 rx=0x7f7a78002f70 tx=0x7f7a78014000).stop 2022-07-20T03:17:23.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 shutdown_connections 2022-07-20T03:17:23.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7a74053ad0 0x7f7a74055f90 unknown :-1 s=CLOSED pgs=683 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7a900af150 0x7f7a90066b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7a900b1440 0x7f7a900b3b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 --2- 172.21.15.61:0/2198620030 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7a90063d60 0x7f7a90064f00 unknown :-1 s=CLOSED pgs=1423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 >> 172.21.15.61:0/2198620030 conn(0x7f7a900634e0 msgr2=0x7f7a90135e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:23.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 shutdown_connections 2022-07-20T03:17:23.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.223+0000 7f7a8ffff700 1 -- 172.21.15.61:0/2198620030 wait complete. 2022-07-20T03:17:23.526 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:23.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.631+0000 7f0d652f1700 1 Processor -- start 2022-07-20T03:17:23.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.631+0000 7f0d652f1700 1 -- start start 2022-07-20T03:17:23.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d600b9ef0 0x7f0d600b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0d6005d160 0x7f0d600b2250 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0d600b3e40 0x7f0d600b2740 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d600b2c30 con 0x7f0d6005d160 2022-07-20T03:17:23.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0d600ba2d0 con 0x7f0d600b3e40 2022-07-20T03:17:23.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600b4560 con 0x7f0d600b9ef0 2022-07-20T03:17:23.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0d6005d160 0x7f0d600b2250 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0d6005d160 0x7f0d600b2250 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51370/0 (socket says 172.21.15.61:51370) 2022-07-20T03:17:23.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 -- 172.21.15.61:0/459540229 learned_addr learned my addr 172.21.15.61:0/459540229 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:23.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5f7fe700 1 --2- 172.21.15.61:0/459540229 >> v2:172.21.15.117:3300/0 conn(0x7f0d600b3e40 0x7f0d600b2740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5effd700 1 --2- 172.21.15.61:0/459540229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d600b9ef0 0x7f0d600b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 -- 172.21.15.61:0/459540229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d600b9ef0 msgr2=0x7f0d600b4e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:23.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 --2- 172.21.15.61:0/459540229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d600b9ef0 0x7f0d600b4e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 -- 172.21.15.61:0/459540229 >> v2:172.21.15.117:3300/0 conn(0x7f0d600b3e40 msgr2=0x7f0d600b2740 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:23.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 --2- 172.21.15.61:0/459540229 >> v2:172.21.15.117:3300/0 conn(0x7f0d600b3e40 0x7f0d600b2740 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 -- 172.21.15.61:0/459540229 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d600b4800 con 0x7f0d6005d160 2022-07-20T03:17:23.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5e7fc700 1 --2- 172.21.15.61:0/459540229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6005d160 0x7f0d600b2250 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f0d500032f0 tx=0x7f0d50009290).ready entity=mon.0 client_cookie=20bf30904d1e5e83 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5dffb700 1 -- 172.21.15.61:0/459540229 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5000b680 con 0x7f0d6005d160 2022-07-20T03:17:23.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5dffb700 1 -- 172.21.15.61:0/459540229 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d5000b7e0 con 0x7f0d6005d160 2022-07-20T03:17:23.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d5dffb700 1 -- 172.21.15.61:0/459540229 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d5000b950 con 0x7f0d6005d160 2022-07-20T03:17:23.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- 172.21.15.61:0/459540229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6005d160 msgr2=0x7f0d600b2250 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- 172.21.15.61:0/459540229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6005d160 0x7f0d600b2250 secure :-1 s=READY pgs=1424 cs=0 l=1 rev1=1 rx=0x7f0d500032f0 tx=0x7f0d50009290).stop 2022-07-20T03:17:23.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- 172.21.15.61:0/459540229 shutdown_connections 2022-07-20T03:17:23.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- 172.21.15.61:0/459540229 >> v2:172.21.15.117:3300/0 conn(0x7f0d600b3e40 0x7f0d600b2740 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- 172.21.15.61:0/459540229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d6005d160 0x7f0d600b2250 unknown :-1 s=CLOSED pgs=1424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 --2- 172.21.15.61:0/459540229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d600b9ef0 0x7f0d600b4e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- 172.21.15.61:0/459540229 >> 172.21.15.61:0/459540229 conn(0x7f0d600679d0 msgr2=0x7f0d600b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:23.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- 172.21.15.61:0/459540229 shutdown_connections 2022-07-20T03:17:23.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.635+0000 7f0d652f1700 1 -- 172.21.15.61:0/459540229 wait complete. 2022-07-20T03:17:23.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 Processor -- start 2022-07-20T03:17:23.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 -- start start 2022-07-20T03:17:23.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 0x7f0d600adf90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 0x7f0d60144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d600b9ef0 0x7f0d6013d940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d600af1e0 con 0x7f0d600b3e40 2022-07-20T03:17:23.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d60145d00 con 0x7f0d600b9ef0 2022-07-20T03:17:23.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0d60145fa0 con 0x7f0d6005d160 2022-07-20T03:17:23.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 0x7f0d60144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 0x7f0d600adf90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5f7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d600b9ef0 0x7f0d6013d940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 0x7f0d600adf90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38048/0 (socket says 172.21.15.61:38048) 2022-07-20T03:17:23.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5e7fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 0x7f0d60144220 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51380/0 (socket says 172.21.15.61:51380) 2022-07-20T03:17:23.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 -- 172.21.15.61:0/2660885000 learned_addr learned my addr 172.21.15.61:0/2660885000 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:23.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 -- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d600b9ef0 msgr2=0x7f0d6013d940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d600b9ef0 0x7f0d6013d940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 -- 172.21.15.61:0/2660885000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 msgr2=0x7f0d60144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 0x7f0d60144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 -- 172.21.15.61:0/2660885000 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d50009530 con 0x7f0d6005d160 2022-07-20T03:17:23.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5e7fc700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 0x7f0d60144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:23.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d5effd700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 0x7f0d600adf90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0d5400c000 tx=0x7f0d54006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d54010780 con 0x7f0d6005d160 2022-07-20T03:17:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0d540108e0 con 0x7f0d6005d160 2022-07-20T03:17:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d6013de30 con 0x7f0d6005d160 2022-07-20T03:17:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0d54010c40 con 0x7f0d6005d160 2022-07-20T03:17:23.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.639+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d60144930 con 0x7f0d6005d160 2022-07-20T03:17:23.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.643+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0d5401c460 con 0x7f0d6005d160 2022-07-20T03:17:23.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.643+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d60145b30 con 0x7f0d6005d160 2022-07-20T03:17:23.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.643+0000 7f0d4ffff700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d3c053a40 0x7f0d3c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.643+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f0d5406c460 con 0x7f0d6005d160 2022-07-20T03:17:23.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.643+0000 7f0d5e7fc700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d3c053a40 0x7f0d3c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.643+0000 7f0d5e7fc700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d3c053a40 0x7f0d3c055f00 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f0d5001a000 tx=0x7f0d50017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.655+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0d540431d0 con 0x7f0d6005d160 2022-07-20T03:17:23.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.819+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f0d600574e0 con 0x7f0d6005d160 2022-07-20T03:17:23.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d4ffff700 1 -- 172.21.15.61:0/2660885000 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 77 v77) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f0d540467f0 con 0x7f0d6005d160 2022-07-20T03:17:23.826 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:23.826 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":77,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":77,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:13.001769+0000","modified":"2022-07-20T03:17:14.005556+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8466},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8466":{"gid":8466,"name":"a","rank":0,"incarnation":76,"state":"up:active","state_seq":3,"addr":"172.21.15.61:6833/3390219350","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3390219350},{"type":"v1","addr":"172.21.15.61:6833","nonce":3390219350}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T03:17:23.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d3c053a40 msgr2=0x7f0d3c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d3c053a40 0x7f0d3c055f00 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f0d5001a000 tx=0x7f0d50017000).stop 2022-07-20T03:17:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 msgr2=0x7f0d600adf90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 0x7f0d600adf90 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f0d5400c000 tx=0x7f0d54006df0).stop 2022-07-20T03:17:23.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 shutdown_connections 2022-07-20T03:17:23.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0d3c053a40 0x7f0d3c055f00 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0d600b9ef0 0x7f0d6013d940 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0d600b3e40 0x7f0d60144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 --2- 172.21.15.61:0/2660885000 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0d6005d160 0x7f0d600adf90 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.831 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 >> 172.21.15.61:0/2660885000 conn(0x7f0d600679d0 msgr2=0x7f0d600b5ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:23.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 shutdown_connections 2022-07-20T03:17:23.832 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.823+0000 7f0d652f1700 1 -- 172.21.15.61:0/2660885000 wait complete. 2022-07-20T03:17:23.833 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 77 2022-07-20T03:17:23.841 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:17:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.943+0000 7f515e45b700 1 Processor -- start 2022-07-20T03:17:23.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.943+0000 7f515e45b700 1 -- start start 2022-07-20T03:17:23.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51580b11b0 0x7f515805d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f5158060bb0 0x7f5158147760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5158147090 con 0x7f515805cda0 2022-07-20T03:17:23.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5158060160 con 0x7f51580b11b0 2022-07-20T03:17:23.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5158099080 con 0x7f5158060bb0 2022-07-20T03:17:23.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51580b11b0 0x7f515805d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f51580b11b0 0x7f515805d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51386/0 (socket says 172.21.15.61:51386) 2022-07-20T03:17:23.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 -- 172.21.15.61:0/234445412 learned_addr learned my addr 172.21.15.61:0/234445412 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:23.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515c9f8700 1 --2- 172.21.15.61:0/234445412 >> v2:172.21.15.117:3300/0 conn(0x7f5158060bb0 0x7f5158147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 -- 172.21.15.61:0/234445412 >> v2:172.21.15.117:3300/0 conn(0x7f5158060bb0 msgr2=0x7f5158147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 --2- 172.21.15.61:0/234445412 >> v2:172.21.15.117:3300/0 conn(0x7f5158060bb0 0x7f5158147760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 -- 172.21.15.61:0/234445412 >> v1:172.21.15.61:6789/0 conn(0x7f515805cda0 legacy=0x7f515805fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 -- 172.21.15.61:0/234445412 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f51581472c0 con 0x7f51580b11b0 2022-07-20T03:17:23.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515c9f8700 1 --2- 172.21.15.61:0/234445412 >> v2:172.21.15.117:3300/0 conn(0x7f5158060bb0 0x7f5158147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:23.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f51577fe700 1 --2- 172.21.15.61:0/234445412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f515805d180 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f5140002f70 tx=0x7f5140006eb0).ready entity=mon.0 client_cookie=9a94eaba632bb384 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f5156ffd700 1 -- 172.21.15.61:0/234445412 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f514000d610 con 0x7f51580b11b0 2022-07-20T03:17:23.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f5156ffd700 1 -- 172.21.15.61:0/234445412 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5140000e90 con 0x7f51580b11b0 2022-07-20T03:17:23.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f5156ffd700 1 -- 172.21.15.61:0/234445412 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f514000d9a0 con 0x7f51580b11b0 2022-07-20T03:17:23.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- 172.21.15.61:0/234445412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 msgr2=0x7f515805d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 --2- 172.21.15.61:0/234445412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f515805d180 secure :-1 s=READY pgs=1426 cs=0 l=1 rev1=1 rx=0x7f5140002f70 tx=0x7f5140006eb0).stop 2022-07-20T03:17:23.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- 172.21.15.61:0/234445412 shutdown_connections 2022-07-20T03:17:23.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 --2- 172.21.15.61:0/234445412 >> v2:172.21.15.117:3300/0 conn(0x7f5158060bb0 0x7f5158147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 --2- 172.21.15.61:0/234445412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f515805d180 unknown :-1 s=CLOSED pgs=1426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- 172.21.15.61:0/234445412 >> 172.21.15.61:0/234445412 conn(0x7f51580b5bb0 msgr2=0x7f51580b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:23.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- 172.21.15.61:0/234445412 shutdown_connections 2022-07-20T03:17:23.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.947+0000 7f515e45b700 1 -- 172.21.15.61:0/234445412 wait complete. 2022-07-20T03:17:23.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 Processor -- start 2022-07-20T03:17:23.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 -- start start 2022-07-20T03:17:23.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f515805cda0 0x7f51580b44d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5158060bb0 0x7f5158066fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f51580674d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f51581416e0 con 0x7f51580b11b0 2022-07-20T03:17:23.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5158141980 con 0x7f5158060bb0 2022-07-20T03:17:23.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5158141c20 con 0x7f515805cda0 2022-07-20T03:17:23.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f51580674d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f51580674d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51394/0 (socket says 172.21.15.61:51394) 2022-07-20T03:17:23.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 -- 172.21.15.61:0/3027229004 learned_addr learned my addr 172.21.15.61:0/3027229004 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:23.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f51577fe700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5158060bb0 0x7f5158066fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f5157fff700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f515805cda0 0x7f51580b44d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 -- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f515805cda0 msgr2=0x7f51580b44d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f515805cda0 0x7f51580b44d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 -- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5158060bb0 msgr2=0x7f5158066fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:23.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5158060bb0 0x7f5158066fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:23.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 -- 172.21.15.61:0/3027229004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f514000b1e0 con 0x7f51580b11b0 2022-07-20T03:17:23.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515c9f8700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f51580674d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f514800a000 tx=0x7f5148008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5148011780 con 0x7f51580b11b0 2022-07-20T03:17:23.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f51480118e0 con 0x7f51580b11b0 2022-07-20T03:17:23.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f51580679c0 con 0x7f51580b11b0 2022-07-20T03:17:23.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5148011c40 con 0x7f51580b11b0 2022-07-20T03:17:23.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.951+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5158159c10 con 0x7f51580b11b0 2022-07-20T03:17:23.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.955+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5158141de0 con 0x7f51580b11b0 2022-07-20T03:17:23.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.955+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f514801c460 con 0x7f51580b11b0 2022-07-20T03:17:23.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.955+0000 7f51557fa700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f513c053a40 0x7f513c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:23.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.955+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(140..140 src has 1..140) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f514806c040 con 0x7f51580b11b0 2022-07-20T03:17:23.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.955+0000 7f5157fff700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f513c053a40 0x7f513c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:23.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.955+0000 7f5157fff700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f513c053a40 0x7f513c055f00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f515000d000 tx=0x7f5150007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:23.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:23.959+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5148042f40 con 0x7f51580b11b0 2022-07-20T03:17:24.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.123+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f51580574e0 con 0x7f51580b11b0 2022-07-20T03:17:24.280 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:17:24.279+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:17:24.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.291+0000 7f51557fa700 1 -- 172.21.15.61:0/3027229004 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v78) v1 ==== 161+0+0 (secure 0 0 0) 0x7f5148046560 con 0x7f51580b11b0 2022-07-20T03:17:24.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f513c053a40 msgr2=0x7f513c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f513c053a40 0x7f513c055f00 secure :-1 s=READY pgs=685 cs=0 l=1 rev1=1 rx=0x7f515000d000 tx=0x7f5150007f10).stop 2022-07-20T03:17:24.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 msgr2=0x7f51580674d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f51580674d0 secure :-1 s=READY pgs=1427 cs=0 l=1 rev1=1 rx=0x7f514800a000 tx=0x7f5148008040).stop 2022-07-20T03:17:24.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 shutdown_connections 2022-07-20T03:17:24.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f513c053a40 0x7f513c055f00 unknown :-1 s=CLOSED pgs=685 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5158060bb0 0x7f5158066fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f51580b11b0 0x7f51580674d0 unknown :-1 s=CLOSED pgs=1427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 --2- 172.21.15.61:0/3027229004 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f515805cda0 0x7f51580b44d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 >> 172.21.15.61:0/3027229004 conn(0x7f51580b5bb0 msgr2=0x7f5158099280 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:24.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 shutdown_connections 2022-07-20T03:17:24.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.295+0000 7f515e45b700 1 -- 172.21.15.61:0/3027229004 wait complete. 2022-07-20T03:17:24.306 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:17:24.315 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:24.325 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:17:24.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 Processor -- start 2022-07-20T03:17:24.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- start start 2022-07-20T03:17:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab240b21f0 0x7fab240b25d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fab2413ffa0 0x7fab24143690 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fab240ab430 con 0x7fab240ac860 2022-07-20T03:17:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fab240ab6d0 con 0x7fab2413ffa0 2022-07-20T03:17:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab240ab970 con 0x7fab240b21f0 2022-07-20T03:17:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab22d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fab2413ffa0 0x7fab24143690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab21d9b700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7fab240ac860 0x7fab24142f80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:56512/0 (socket says 172.21.15.61:56512) 2022-07-20T03:17:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab21d9b700 1 -- 172.21.15.61:0/3470151894 learned_addr learned my addr 172.21.15.61:0/3470151894 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2259c700 1 --2- 172.21.15.61:0/3470151894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab240b21f0 0x7fab240b25d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2159a700 1 -- 172.21.15.61:0/3470151894 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 732305589 0 0) 0x7fab240ab430 con 0x7fab240ac860 2022-07-20T03:17:24.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2159a700 1 -- 172.21.15.61:0/3470151894 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fab04003260 con 0x7fab240ac860 2022-07-20T03:17:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2259c700 1 -- 172.21.15.61:0/3470151894 >> v2:172.21.15.61:3300/0 conn(0x7fab2413ffa0 msgr2=0x7fab24143690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2259c700 1 --2- 172.21.15.61:0/3470151894 >> v2:172.21.15.61:3300/0 conn(0x7fab2413ffa0 0x7fab24143690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2259c700 1 -- 172.21.15.61:0/3470151894 >> v1:172.21.15.117:6789/0 conn(0x7fab240ac860 legacy=0x7fab24142f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2259c700 1 -- 172.21.15.61:0/3470151894 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab240ab0b0 con 0x7fab240b21f0 2022-07-20T03:17:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2259c700 1 --2- 172.21.15.61:0/3470151894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab240b21f0 0x7fab240b25d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fab0c002f70 tx=0x7fab0c006df0).ready entity=mon.2 client_cookie=2726b7f8a15602ca server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2159a700 1 -- 172.21.15.61:0/3470151894 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab0c00d4a0 con 0x7fab240b21f0 2022-07-20T03:17:24.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2159a700 1 -- 172.21.15.61:0/3470151894 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0c000e90 con 0x7fab240b21f0 2022-07-20T03:17:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab2159a700 1 -- 172.21.15.61:0/3470151894 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab0c00d7f0 con 0x7fab240b21f0 2022-07-20T03:17:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- 172.21.15.61:0/3470151894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab240b21f0 msgr2=0x7fab240b25d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 --2- 172.21.15.61:0/3470151894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab240b21f0 0x7fab240b25d0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7fab0c002f70 tx=0x7fab0c006df0).stop 2022-07-20T03:17:24.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- 172.21.15.61:0/3470151894 shutdown_connections 2022-07-20T03:17:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 --2- 172.21.15.61:0/3470151894 >> v2:172.21.15.61:3300/0 conn(0x7fab2413ffa0 0x7fab24143690 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 --2- 172.21.15.61:0/3470151894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab240b21f0 0x7fab240b25d0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.415+0000 7fab28cb3700 1 -- 172.21.15.61:0/3470151894 >> 172.21.15.61:0/3470151894 conn(0x7fab240b3690 msgr2=0x7fab240b0ad0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:24.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 -- 172.21.15.61:0/3470151894 shutdown_connections 2022-07-20T03:17:24.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 -- 172.21.15.61:0/3470151894 wait complete. 2022-07-20T03:17:24.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 Processor -- start 2022-07-20T03:17:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 -- start start 2022-07-20T03:17:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 0x7fab2409b4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab240b21f0 0x7fab241518a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab2413ffa0 0x7fab24151d90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24143d30 con 0x7fab240ac860 2022-07-20T03:17:24.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab24142d10 con 0x7fab240b21f0 2022-07-20T03:17:24.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab28cb3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fab240adaf0 con 0x7fab2413ffa0 2022-07-20T03:17:24.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 0x7fab2409b4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 0x7fab2409b4b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51414/0 (socket says 172.21.15.61:51414) 2022-07-20T03:17:24.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 -- 172.21.15.61:0/4099875940 learned_addr learned my addr 172.21.15.61:0/4099875940 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:24.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab21d9b700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab240b21f0 0x7fab241518a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab22d9d700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab2413ffa0 0x7fab24151d90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 -- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab2413ffa0 msgr2=0x7fab24151d90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab2413ffa0 0x7fab24151d90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 -- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab240b21f0 msgr2=0x7fab241518a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab240b21f0 0x7fab241518a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 -- 172.21.15.61:0/4099875940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab0c00b180 con 0x7fab240ac860 2022-07-20T03:17:24.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab21d9b700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab240b21f0 0x7fab241518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:24.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab2259c700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 0x7fab2409b4b0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fab0c002f70 tx=0x7fab0c014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:24.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.419+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab0c013ac0 con 0x7fab240ac860 2022-07-20T03:17:24.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fab0c013c20 con 0x7fab240ac860 2022-07-20T03:17:24.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab240add90 con 0x7fab240ac860 2022-07-20T03:17:24.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fab0c01e400 con 0x7fab240ac860 2022-07-20T03:17:24.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab24151440 con 0x7fab240ac860 2022-07-20T03:17:24.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab240574e0 con 0x7fab240ac860 2022-07-20T03:17:24.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fab0c025030 con 0x7fab240ac860 2022-07-20T03:17:24.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab1b7fe700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab08053920 0x7fab08055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fab0c06cda0 con 0x7fab240ac860 2022-07-20T03:17:24.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.423+0000 7fab21d9b700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab08053920 0x7fab08055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.427+0000 7fab21d9b700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab08053920 0x7fab08055de0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fab1c00b000 tx=0x7fab1c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:24.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.431+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fab0c043ba0 con 0x7fab240ac860 2022-07-20T03:17:24.596 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.591+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7fab2405c400 con 0x7fab240ac860 2022-07-20T03:17:24.597 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.591+0000 7fab1b7fe700 1 -- 172.21.15.61:0/4099875940 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 78 v78) v1 ==== 94+0+1785 (secure 0 0 0) 0x7fab0c023070 con 0x7fab240ac860 2022-07-20T03:17:24.598 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:24.599 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":78,"default_fscid":14,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75}],"filesystems":[{"mdsmap":{"epoch":78,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:13.001769+0000","modified":"2022-07-20T03:17:24.280492+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":141,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[28],"metadata_pool":27,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":14}]} 2022-07-20T03:17:24.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.591+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab08053920 msgr2=0x7fab08055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.591+0000 7fab28cb3700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab08053920 0x7fab08055de0 secure :-1 s=READY pgs=686 cs=0 l=1 rev1=1 rx=0x7fab1c00b000 tx=0x7fab1c009040).stop 2022-07-20T03:17:24.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.591+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 msgr2=0x7fab2409b4b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.591+0000 7fab28cb3700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 0x7fab2409b4b0 secure :-1 s=READY pgs=1430 cs=0 l=1 rev1=1 rx=0x7fab0c002f70 tx=0x7fab0c014000).stop 2022-07-20T03:17:24.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 shutdown_connections 2022-07-20T03:17:24.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fab08053920 0x7fab08055de0 unknown :-1 s=CLOSED pgs=686 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fab240b21f0 0x7fab241518a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fab2413ffa0 0x7fab24151d90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 --2- 172.21.15.61:0/4099875940 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fab240ac860 0x7fab2409b4b0 unknown :-1 s=CLOSED pgs=1430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 >> 172.21.15.61:0/4099875940 conn(0x7fab240b3690 msgr2=0x7fab24096ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:24.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 shutdown_connections 2022-07-20T03:17:24.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.595+0000 7fab28cb3700 1 -- 172.21.15.61:0/4099875940 wait complete. 2022-07-20T03:17:24.608 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 78 2022-07-20T03:17:24.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:17:24.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 Processor -- start 2022-07-20T03:17:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 -- start start 2022-07-20T03:17:24.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c13fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fde1c0650c0 0x7fde1c1400a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fde1c0b5d50 con 0x7fde1c144a10 2022-07-20T03:17:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fde1c143d90 con 0x7fde1c0650c0 2022-07-20T03:17:24.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde20f65700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c144030 con 0x7fde1c0b5970 2022-07-20T03:17:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fde1c144a10 0x7fde1c0affd0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48024/0 (socket says 172.21.15.61:48024) 2022-07-20T03:17:24.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1bfff700 1 -- 172.21.15.61:0/321672033 learned_addr learned my addr 172.21.15.61:0/321672033 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:24.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1affd700 1 --2- 172.21.15.61:0/321672033 >> v2:172.21.15.117:3300/0 conn(0x7fde1c0650c0 0x7fde1c1400a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1b7fe700 1 --2- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c13fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1a7fc700 1 -- 172.21.15.61:0/321672033 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3058506156 0 0) 0x7fde1c0b5d50 con 0x7fde1c144a10 2022-07-20T03:17:24.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1a7fc700 1 -- 172.21.15.61:0/321672033 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fddfc003260 con 0x7fde1c144a10 2022-07-20T03:17:24.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1affd700 1 -- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 msgr2=0x7fde1c13fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:24.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1affd700 1 --2- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c13fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1affd700 1 -- 172.21.15.61:0/321672033 >> v1:172.21.15.61:6789/0 conn(0x7fde1c144a10 legacy=0x7fde1c0affd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1affd700 1 -- 172.21.15.61:0/321672033 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde1c1442d0 con 0x7fde1c0650c0 2022-07-20T03:17:24.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.723+0000 7fde1affd700 1 --2- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c0650c0 0x7fde1c1400a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7fde04002f70 tx=0x7fde04006df0).ready entity=mon.1 client_cookie=489af7a64986d90e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:24.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1a7fc700 1 -- 172.21.15.61:0/321672033 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde0400d4a0 con 0x7fde1c0650c0 2022-07-20T03:17:24.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1a7fc700 1 -- 172.21.15.61:0/321672033 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde04000e90 con 0x7fde1c0650c0 2022-07-20T03:17:24.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1a7fc700 1 -- 172.21.15.61:0/321672033 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde0400d7f0 con 0x7fde1c0650c0 2022-07-20T03:17:24.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c0650c0 msgr2=0x7fde1c1400a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 --2- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c0650c0 0x7fde1c1400a0 secure :-1 s=READY pgs=641 cs=0 l=1 rev1=1 rx=0x7fde04002f70 tx=0x7fde04006df0).stop 2022-07-20T03:17:24.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- 172.21.15.61:0/321672033 shutdown_connections 2022-07-20T03:17:24.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 --2- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c0650c0 0x7fde1c1400a0 unknown :-1 s=CLOSED pgs=641 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 --2- 172.21.15.61:0/321672033 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c13fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- 172.21.15.61:0/321672033 >> 172.21.15.61:0/321672033 conn(0x7fde1c063490 msgr2=0x7fde1c0b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:24.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- 172.21.15.61:0/321672033 shutdown_connections 2022-07-20T03:17:24.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- 172.21.15.61:0/321672033 wait complete. 2022-07-20T03:17:24.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 Processor -- start 2022-07-20T03:17:24.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- start start 2022-07-20T03:17:24.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 0x7fde1c0b4570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c0acae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c144a10 0x7fde1c13b9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0adab0 con 0x7fde1c0650c0 2022-07-20T03:17:24.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0add50 con 0x7fde1c144a10 2022-07-20T03:17:24.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde20f65700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fde1c0adff0 con 0x7fde1c0b5970 2022-07-20T03:17:24.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 0x7fde1c0b4570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 0x7fde1c0b4570 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51428/0 (socket says 172.21.15.61:51428) 2022-07-20T03:17:24.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 -- 172.21.15.61:0/1341931859 learned_addr learned my addr 172.21.15.61:0/1341931859 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:24.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1bfff700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c144a10 0x7fde1c13b9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1affd700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c0acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 -- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 msgr2=0x7fde1c0acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c0acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 -- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c144a10 msgr2=0x7fde1c13b9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c144a10 0x7fde1c13b9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:24.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 -- 172.21.15.61:0/1341931859 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0400b180 con 0x7fde1c0650c0 2022-07-20T03:17:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.727+0000 7fde1b7fe700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 0x7fde1c0b4570 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7fde0c00c000 tx=0x7fde0c006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde0c010780 con 0x7fde1c0650c0 2022-07-20T03:17:24.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fde0c0108e0 con 0x7fde1c0650c0 2022-07-20T03:17:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde1c0ae490 con 0x7fde1c0650c0 2022-07-20T03:17:24.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fde0c010c40 con 0x7fde1c0650c0 2022-07-20T03:17:24.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde1c0b4a60 con 0x7fde1c0650c0 2022-07-20T03:17:24.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde1c0574e0 con 0x7fde1c0650c0 2022-07-20T03:17:24.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fde0c01c460 con 0x7fde1c0650c0 2022-07-20T03:17:24.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde18ff9700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde000539b0 0x7fde00055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:24.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fde0c06c300 con 0x7fde1c0650c0 2022-07-20T03:17:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.731+0000 7fde1affd700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde000539b0 0x7fde00055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:24.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.735+0000 7fde1affd700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde000539b0 0x7fde00055e70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7fde0401a000 tx=0x7fde04014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:24.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.735+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fde0c043110 con 0x7fde1c0650c0 2022-07-20T03:17:24.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:24.899+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7fde1c0ae1b0 con 0x7fde1c0650c0 2022-07-20T03:17:25.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.291+0000 7fde18ff9700 1 -- 172.21.15.61:0/1341931859 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v79) v1 ==== 104+0+0 (secure 0 0 0) 0x7fde0c046730 con 0x7fde1c0650c0 2022-07-20T03:17:25.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde000539b0 msgr2=0x7fde00055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:25.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde000539b0 0x7fde00055e70 secure :-1 s=READY pgs=687 cs=0 l=1 rev1=1 rx=0x7fde0401a000 tx=0x7fde04014000).stop 2022-07-20T03:17:25.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 msgr2=0x7fde1c0b4570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:25.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 0x7fde1c0b4570 secure :-1 s=READY pgs=1432 cs=0 l=1 rev1=1 rx=0x7fde0c00c000 tx=0x7fde0c006df0).stop 2022-07-20T03:17:25.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 shutdown_connections 2022-07-20T03:17:25.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fde000539b0 0x7fde00055e70 unknown :-1 s=CLOSED pgs=687 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fde1c144a10 0x7fde1c13b9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fde1c0b5970 0x7fde1c0acae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 --2- 172.21.15.61:0/1341931859 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fde1c0650c0 0x7fde1c0b4570 unknown :-1 s=CLOSED pgs=1432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 >> 172.21.15.61:0/1341931859 conn(0x7fde1c063490 msgr2=0x7fde1c140400 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:25.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 shutdown_connections 2022-07-20T03:17:25.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.295+0000 7fde20f65700 1 -- 172.21.15.61:0/1341931859 wait complete. 2022-07-20T03:17:25.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:17:25.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.435+0000 7fd56e73c700 1 Processor -- start 2022-07-20T03:17:25.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.435+0000 7fd56e73c700 1 -- start start 2022-07-20T03:17:25.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd5680b6690 0x7fd5680b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:25.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680ae670 0x7fd5680aeb00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:25.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5680acae0 con 0x7fd5680b59a0 2022-07-20T03:17:25.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd568147100 con 0x7fd5680b6690 2022-07-20T03:17:25.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5680acd10 con 0x7fd5680ae670 2022-07-20T03:17:25.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd5680b6690 0x7fd5680b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:25.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd5680b6690 0x7fd5680b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51436/0 (socket says 172.21.15.61:51436) 2022-07-20T03:17:25.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 -- 172.21.15.61:0/307716204 learned_addr learned my addr 172.21.15.61:0/307716204 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:25.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56ccd9700 1 --2- 172.21.15.61:0/307716204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680ae670 0x7fd5680aeb00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:25.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 -- 172.21.15.61:0/307716204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680ae670 msgr2=0x7fd5680aeb00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:25.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 --2- 172.21.15.61:0/307716204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680ae670 0x7fd5680aeb00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 -- 172.21.15.61:0/307716204 >> v1:172.21.15.117:6789/0 conn(0x7fd5680b59a0 legacy=0x7fd56813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:25.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 -- 172.21.15.61:0/307716204 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd5680b5df0 con 0x7fd5680b6690 2022-07-20T03:17:25.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd567fff700 1 --2- 172.21.15.61:0/307716204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b6690 0x7fd5680b6a70 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7fd5580032f0 tx=0x7fd558009290).ready entity=mon.0 client_cookie=689fd69b09070d5e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:25.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd566ffd700 1 -- 172.21.15.61:0/307716204 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd55800b8a0 con 0x7fd5680b6690 2022-07-20T03:17:25.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd566ffd700 1 -- 172.21.15.61:0/307716204 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd55800ba00 con 0x7fd5680b6690 2022-07-20T03:17:25.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd566ffd700 1 -- 172.21.15.61:0/307716204 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd55800bb70 con 0x7fd5680b6690 2022-07-20T03:17:25.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- 172.21.15.61:0/307716204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b6690 msgr2=0x7fd5680b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:25.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 --2- 172.21.15.61:0/307716204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b6690 0x7fd5680b6a70 secure :-1 s=READY pgs=1433 cs=0 l=1 rev1=1 rx=0x7fd5580032f0 tx=0x7fd558009290).stop 2022-07-20T03:17:25.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- 172.21.15.61:0/307716204 shutdown_connections 2022-07-20T03:17:25.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 --2- 172.21.15.61:0/307716204 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680ae670 0x7fd5680aeb00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 --2- 172.21.15.61:0/307716204 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b6690 0x7fd5680b6a70 unknown :-1 s=CLOSED pgs=1433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- 172.21.15.61:0/307716204 >> 172.21.15.61:0/307716204 conn(0x7fd5680b9f70 msgr2=0x7fd5680b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:25.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- 172.21.15.61:0/307716204 shutdown_connections 2022-07-20T03:17:25.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.439+0000 7fd56e73c700 1 -- 172.21.15.61:0/307716204 wait complete. 2022-07-20T03:17:25.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 Processor -- start 2022-07-20T03:17:25.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 -- start start 2022-07-20T03:17:25.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680ae670 0x7fd56813ee00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:25.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 0x7fd56813f6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:25.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b6690 0x7fd568152590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:25.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd56813fef0 con 0x7fd5680b59a0 2022-07-20T03:17:25.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd568141de0 con 0x7fd5680ae670 2022-07-20T03:17:25.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd56813d920 con 0x7fd5680b6690 2022-07-20T03:17:25.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 0x7fd56813f6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:25.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd567fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680ae670 0x7fd56813ee00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:25.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 0x7fd56813f6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51444/0 (socket says 172.21.15.61:51444) 2022-07-20T03:17:25.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 -- 172.21.15.61:0/3533004424 learned_addr learned my addr 172.21.15.61:0/3533004424 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:25.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56ccd9700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b6690 0x7fd568152590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:25.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 -- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b6690 msgr2=0x7fd568152590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:25.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b6690 0x7fd568152590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 -- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680ae670 msgr2=0x7fd56813ee00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:25.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680ae670 0x7fd56813ee00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:25.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 -- 172.21.15.61:0/3533004424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd558009530 con 0x7fd5680b59a0 2022-07-20T03:17:25.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd567fff700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680ae670 0x7fd56813ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:25.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5677fe700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 0x7fd56813f6d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7fd568066820 tx=0x7fd550007c10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:25.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd550007030 con 0x7fd5680b59a0 2022-07-20T03:17:25.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd550007190 con 0x7fd5680b59a0 2022-07-20T03:17:25.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd568142080 con 0x7fd5680b59a0 2022-07-20T03:17:25.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd550006510 con 0x7fd5680b59a0 2022-07-20T03:17:25.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.443+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd56813e190 con 0x7fd5680b59a0 2022-07-20T03:17:25.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.447+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd56813dae0 con 0x7fd5680b59a0 2022-07-20T03:17:25.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.447+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd550010030 con 0x7fd5680b59a0 2022-07-20T03:17:25.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.447+0000 7fd5657fa700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd54c053af0 0x7fd54c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:25.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.447+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(141..141 src has 1..141) v4 ==== 6159+0+0 (secure 0 0 0) 0x7fd55006bb10 con 0x7fd5680b59a0 2022-07-20T03:17:25.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.447+0000 7fd567fff700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd54c053af0 0x7fd54c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:25.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.451+0000 7fd567fff700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd54c053af0 0x7fd54c055fb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7fd55801b000 tx=0x7fd558016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:25.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.455+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd550042980 con 0x7fd5680b59a0 2022-07-20T03:17:25.586 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:25.583+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7fd5680acfb0 con 0x7fd5680b59a0 2022-07-20T03:17:26.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd5657fa700 1 -- 172.21.15.61:0/3533004424 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v142) v1 ==== 185+0+0 (secure 0 0 0) 0x7fd550045fa0 con 0x7fd5680b59a0 2022-07-20T03:17:26.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd54c053af0 msgr2=0x7fd54c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:26.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd54c053af0 0x7fd54c055fb0 secure :-1 s=READY pgs=688 cs=0 l=1 rev1=1 rx=0x7fd55801b000 tx=0x7fd558016000).stop 2022-07-20T03:17:26.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 msgr2=0x7fd56813f6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:26.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 0x7fd56813f6d0 secure :-1 s=READY pgs=1434 cs=0 l=1 rev1=1 rx=0x7fd568066820 tx=0x7fd550007c10).stop 2022-07-20T03:17:26.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 shutdown_connections 2022-07-20T03:17:26.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd54c053af0 0x7fd54c055fb0 unknown :-1 s=CLOSED pgs=688 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5680b6690 0x7fd568152590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5680b59a0 0x7fd56813f6d0 unknown :-1 s=CLOSED pgs=1434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 --2- 172.21.15.61:0/3533004424 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5680ae670 0x7fd56813ee00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 >> 172.21.15.61:0/3533004424 conn(0x7fd5680b9f70 msgr2=0x7fd5680b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:26.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 shutdown_connections 2022-07-20T03:17:26.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.299+0000 7fd56e73c700 1 -- 172.21.15.61:0/3533004424 wait complete. 2022-07-20T03:17:26.310 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:17:26.320 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:17:26.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.427+0000 7ff6da933700 1 Processor -- start 2022-07-20T03:17:26.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.427+0000 7ff6da933700 1 -- start start 2022-07-20T03:17:26.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d4061410 0x7ff6d405bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:26.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff6d405c280 0x7ff6d405c710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:26.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ff6d409d220 con 0x7ff6d4060c00 2022-07-20T03:17:26.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff6d409e590 con 0x7ff6d405c280 2022-07-20T03:17:26.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4060fe0 con 0x7ff6d4061410 2022-07-20T03:17:26.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d3fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7ff6d4060c00 0x7ff6d409cb10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48048/0 (socket says 172.21.15.61:48048) 2022-07-20T03:17:26.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d3fff700 1 -- 172.21.15.61:0/2395790768 learned_addr learned my addr 172.21.15.61:0/2395790768 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:26.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d37fe700 1 --2- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d4061410 0x7ff6d405bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:26.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d8ed0700 1 --2- 172.21.15.61:0/2395790768 >> v2:172.21.15.117:3300/0 conn(0x7ff6d405c280 0x7ff6d405c710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:26.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d2ffd700 1 -- 172.21.15.61:0/2395790768 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3499137480 0 0) 0x7ff6d409d220 con 0x7ff6d4060c00 2022-07-20T03:17:26.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d2ffd700 1 -- 172.21.15.61:0/2395790768 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7ff6b4003260 con 0x7ff6d4060c00 2022-07-20T03:17:26.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d4061410 msgr2=0x7ff6d405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:26.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d8ed0700 1 --2- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d4061410 0x7ff6d405bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/2395790768 >> v1:172.21.15.61:6789/0 conn(0x7ff6d4060c00 legacy=0x7ff6d409cb10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:26.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/2395790768 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6d40b39c0 con 0x7ff6d405c280 2022-07-20T03:17:26.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d8ed0700 1 --2- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d405c280 0x7ff6d405c710 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7ff6cc0032f0 tx=0x7ff6cc0091d0).ready entity=mon.1 client_cookie=2ad2145161204348 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:26.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d2ffd700 1 -- 172.21.15.61:0/2395790768 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6cc00b600 con 0x7ff6d405c280 2022-07-20T03:17:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d2ffd700 1 -- 172.21.15.61:0/2395790768 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc00b760 con 0x7ff6d405c280 2022-07-20T03:17:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6d2ffd700 1 -- 172.21.15.61:0/2395790768 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6cc00b8d0 con 0x7ff6d405c280 2022-07-20T03:17:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 -- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d405c280 msgr2=0x7ff6d405c710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:26.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 --2- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d405c280 0x7ff6d405c710 secure :-1 s=READY pgs=643 cs=0 l=1 rev1=1 rx=0x7ff6cc0032f0 tx=0x7ff6cc0091d0).stop 2022-07-20T03:17:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 -- 172.21.15.61:0/2395790768 shutdown_connections 2022-07-20T03:17:26.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 --2- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d405c280 0x7ff6d405c710 unknown :-1 s=CLOSED pgs=643 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 --2- 172.21.15.61:0/2395790768 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d4061410 0x7ff6d405bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.431+0000 7ff6da933700 1 -- 172.21.15.61:0/2395790768 >> 172.21.15.61:0/2395790768 conn(0x7ff6d40b5f30 msgr2=0x7ff6d40b6340 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:26.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 -- 172.21.15.61:0/2395790768 shutdown_connections 2022-07-20T03:17:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 -- 172.21.15.61:0/2395790768 wait complete. 2022-07-20T03:17:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 Processor -- start 2022-07-20T03:17:26.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 -- start start 2022-07-20T03:17:26.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d405c280 0x7ff6d414a090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d4060c00 0x7ff6d4155710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 0x7ff6d41528c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:26.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d408f990 con 0x7ff6d4061410 2022-07-20T03:17:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4152420 con 0x7ff6d4060c00 2022-07-20T03:17:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6da933700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d4152140 con 0x7ff6d405c280 2022-07-20T03:17:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 0x7ff6d41528c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:26.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 0x7ff6d41528c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51460/0 (socket says 172.21.15.61:51460) 2022-07-20T03:17:26.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/1407279233 learned_addr learned my addr 172.21.15.61:0/1407279233 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:26.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d37fe700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d4060c00 0x7ff6d4155710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:26.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d3fff700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d405c280 0x7ff6d414a090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:26.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d405c280 msgr2=0x7ff6d414a090 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:26.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d405c280 0x7ff6d414a090 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d4060c00 msgr2=0x7ff6d4155710 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:26.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d4060c00 0x7ff6d4155710 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:26.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d8ed0700 1 -- 172.21.15.61:0/1407279233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc009470 con 0x7ff6d4061410 2022-07-20T03:17:26.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.435+0000 7ff6d3fff700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d405c280 0x7ff6d414a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6d8ed0700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 0x7ff6d41528c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7ff6cc0032f0 tx=0x7ff6cc016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6cc00b8d0 con 0x7ff6d4061410 2022-07-20T03:17:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff6cc021630 con 0x7ff6d4061410 2022-07-20T03:17:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d4155c00 con 0x7ff6d4061410 2022-07-20T03:17:26.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff6cc00b600 con 0x7ff6d4061410 2022-07-20T03:17:26.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d40b26e0 con 0x7ff6d4061410 2022-07-20T03:17:26.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d408f7b0 con 0x7ff6d4061410 2022-07-20T03:17:26.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.439+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff6cc027020 con 0x7ff6d4061410 2022-07-20T03:17:26.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.443+0000 7ff6d17fa700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff6b8053a40 0x7ff6b8055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.443+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(142..142 src has 1..142) v4 ==== 5721+0+0 (secure 0 0 0) 0x7ff6cc06d310 con 0x7ff6d4061410 2022-07-20T03:17:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.443+0000 7ff6d3fff700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff6b8053a40 0x7ff6b8055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.443+0000 7ff6d3fff700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff6b8053a40 0x7ff6b8055f00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7ff6bc00b000 tx=0x7ff6bc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:26.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.447+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff6cc0443d0 con 0x7ff6d4061410 2022-07-20T03:17:26.584 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:26.579+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7ff6d409dbc0 con 0x7ff6d4061410 2022-07-20T03:17:27.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.311+0000 7ff6d17fa700 1 -- 172.21.15.61:0/1407279233 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v143) v1 ==== 173+0+0 (secure 0 0 0) 0x7ff6cc0150d0 con 0x7ff6d4061410 2022-07-20T03:17:27.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.311+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff6b8053a40 msgr2=0x7ff6b8055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.311+0000 7ff6da933700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff6b8053a40 0x7ff6b8055f00 secure :-1 s=READY pgs=689 cs=0 l=1 rev1=1 rx=0x7ff6bc00b000 tx=0x7ff6bc009040).stop 2022-07-20T03:17:27.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.311+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 msgr2=0x7ff6d41528c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.311+0000 7ff6da933700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 0x7ff6d41528c0 secure :-1 s=READY pgs=1436 cs=0 l=1 rev1=1 rx=0x7ff6cc0032f0 tx=0x7ff6cc016000).stop 2022-07-20T03:17:27.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 shutdown_connections 2022-07-20T03:17:27.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff6b8053a40 0x7ff6b8055f00 unknown :-1 s=CLOSED pgs=689 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff6d4060c00 0x7ff6d4155710 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff6d4061410 0x7ff6d41528c0 unknown :-1 s=CLOSED pgs=1436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 --2- 172.21.15.61:0/1407279233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff6d405c280 0x7ff6d414a090 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 >> 172.21.15.61:0/1407279233 conn(0x7ff6d40b5f30 msgr2=0x7ff6d40b72c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:27.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 shutdown_connections 2022-07-20T03:17:27.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.315+0000 7ff6da933700 1 -- 172.21.15.61:0/1407279233 wait complete. 2022-07-20T03:17:27.323 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:17:27.331 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config rm mon mon_allow_pool_delete 2022-07-20T03:17:27.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 Processor -- start 2022-07-20T03:17:27.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 -- start start 2022-07-20T03:17:27.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f48840b5990 0x7f48840ae050 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4884146960 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4884142a90 con 0x7f48840af8e0 2022-07-20T03:17:27.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4884146e50 con 0x7f48840b5990 2022-07-20T03:17:27.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.439+0000 7f488359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b5d70 con 0x7f4884143600 2022-07-20T03:17:27.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f48840af8e0 0x7f4884142ef0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48062/0 (socket says 172.21.15.61:48062) 2022-07-20T03:17:27.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 -- 172.21.15.61:0/3949007588 learned_addr learned my addr 172.21.15.61:0/3949007588 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:27.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4882d9d700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4884146960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488259c700 1 --2- 172.21.15.61:0/3949007588 >> v2:172.21.15.117:3300/0 conn(0x7f48840b5990 0x7f48840ae050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3705957423 0 0) 0x7f4884142a90 con 0x7f48840af8e0 2022-07-20T03:17:27.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4864003260 con 0x7f48840af8e0 2022-07-20T03:17:27.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 3352005531 0 0) 0x7f4864003260 con 0x7f48840af8e0 2022-07-20T03:17:27.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 msgr2=0x7f4884146960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:27.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4884146960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 >> v2:172.21.15.117:3300/0 conn(0x7f48840b5990 msgr2=0x7f48840ae050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:27.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 --2- 172.21.15.61:0/3949007588 >> v2:172.21.15.117:3300/0 conn(0x7f48840b5990 0x7f48840ae050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48840afdf0 con 0x7f48840af8e0 2022-07-20T03:17:27.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f486c001a00 con 0x7f48840af8e0 2022-07-20T03:17:27.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 >> v1:172.21.15.61:6789/0 conn(0x7f48840af8e0 legacy=0x7f4884142ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864003750 0x7f4864003b70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4864004500 0x7f4864004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4864004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884146e50 con 0x7f4864004500 2022-07-20T03:17:27.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48840b5d70 con 0x7f4864003750 2022-07-20T03:17:27.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f486400b750 con 0x7f4884143600 2022-07-20T03:17:27.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4864004500 0x7f4864004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488259c700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864003750 0x7f4864003b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4882d9d700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4864004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 -- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 msgr2=0x7f4864004e70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:27.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4864004e70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 -- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864003750 msgr2=0x7f4864003b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:27.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864003750 0x7f4864003b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 -- 172.21.15.61:0/3949007588 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4864003260 con 0x7f4864004500 2022-07-20T03:17:27.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4882d9d700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884143600 0x7f4864004e70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:27.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4881d9b700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4864004500 0x7f4864004980 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f486c001710 tx=0x7f48840adec0).ready entity=mon.0 client_cookie=eca53a7af4b39d6e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:27.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c00cc40 con 0x7f4864004500 2022-07-20T03:17:27.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c00cda0 con 0x7f4864004500 2022-07-20T03:17:27.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488159a700 1 -- 172.21.15.61:0/3949007588 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c004b50 con 0x7f4864004500 2022-07-20T03:17:27.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488359e700 1 -- 172.21.15.61:0/3949007588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4864004500 msgr2=0x7f4864004980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f488359e700 1 --2- 172.21.15.61:0/3949007588 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4864004500 0x7f4864004980 secure :-1 s=READY pgs=1438 cs=0 l=1 rev1=1 rx=0x7f486c001710 tx=0x7f48840adec0).stop 2022-07-20T03:17:27.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.443+0000 7f4882d9d700 1 -- 172.21.15.61:0/3949007588 reap_dead start 2022-07-20T03:17:27.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.447+0000 7f488359e700 1 -- 172.21.15.61:0/3949007588 shutdown_connections 2022-07-20T03:17:27.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.447+0000 7f488359e700 1 -- 172.21.15.61:0/3949007588 >> 172.21.15.61:0/3949007588 conn(0x7f4884063480 msgr2=0x7f48840b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:27.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.447+0000 7f488359e700 1 -- 172.21.15.61:0/3949007588 shutdown_connections 2022-07-20T03:17:27.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.447+0000 7f488359e700 1 -- 172.21.15.61:0/3949007588 wait complete. 2022-07-20T03:17:27.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.447+0000 7f488359e700 1 Processor -- start 2022-07-20T03:17:27.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.447+0000 7f488359e700 1 -- start start 2022-07-20T03:17:27.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864004500 0x7f48840674f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 0x7f4884152230 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884152720 0x7f4884152ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48841156f0 con 0x7f4884112d40 2022-07-20T03:17:27.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4884115990 con 0x7f4864004500 2022-07-20T03:17:27.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4884115c30 con 0x7f4884152720 2022-07-20T03:17:27.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 0x7f4884152230 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488259c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864004500 0x7f48840674f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 0x7f4884152230 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51478/0 (socket says 172.21.15.61:51478) 2022-07-20T03:17:27.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 -- 172.21.15.61:0/2797948975 learned_addr learned my addr 172.21.15.61:0/2797948975 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:27.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4882d9d700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884152720 0x7f4884152ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 -- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884152720 msgr2=0x7f4884152ba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884152720 0x7f4884152ba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 -- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864004500 msgr2=0x7f48840674f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864004500 0x7f48840674f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 -- 172.21.15.61:0/2797948975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f48841153e0 con 0x7f4884112d40 2022-07-20T03:17:27.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488259c700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864004500 0x7f48840674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:27.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f4881d9b700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 0x7f4884152230 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f486c001710 tx=0x7f486c015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c00cc40 con 0x7f4884112d40 2022-07-20T03:17:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48841160d0 con 0x7f4884112d40 2022-07-20T03:17:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f486c007570 con 0x7f4884112d40 2022-07-20T03:17:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f486c0078d0 con 0x7f4884112d40 2022-07-20T03:17:27.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4884155780 con 0x7f4884112d40 2022-07-20T03:17:27.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.451+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48840b0090 con 0x7f4884112d40 2022-07-20T03:17:27.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.455+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f486c022470 con 0x7f4884112d40 2022-07-20T03:17:27.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.455+0000 7f487b7fe700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48680539d0 0x7f4868055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.455+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f486c01c040 con 0x7f4884112d40 2022-07-20T03:17:27.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.455+0000 7f488259c700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48680539d0 0x7f4868055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.455+0000 7f488259c700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48680539d0 0x7f4868055e90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f4874006000 tx=0x7f4874006d90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:27.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.459+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f486c044640 con 0x7f4884112d40 2022-07-20T03:17:27.595 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.591+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"} v 0) v1 -- 0x7f48840574e0 con 0x7f4884112d40 2022-07-20T03:17:27.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.595+0000 7f487b7fe700 1 -- 172.21.15.61:0/2797948975 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "config rm", "who": "mon", "name": "mon_allow_pool_delete"}]=0 v23) v1 ==== 104+0+0 (secure 0 0 0) 0x7f486c047c60 con 0x7f4884112d40 2022-07-20T03:17:27.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48680539d0 msgr2=0x7f4868055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48680539d0 0x7f4868055e90 secure :-1 s=READY pgs=690 cs=0 l=1 rev1=1 rx=0x7f4874006000 tx=0x7f4874006d90).stop 2022-07-20T03:17:27.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 msgr2=0x7f4884152230 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 0x7f4884152230 secure :-1 s=READY pgs=1439 cs=0 l=1 rev1=1 rx=0x7f486c001710 tx=0x7f486c015000).stop 2022-07-20T03:17:27.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 shutdown_connections 2022-07-20T03:17:27.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f48680539d0 0x7f4868055e90 unknown :-1 s=CLOSED pgs=690 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4884152720 0x7f4884152ba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4884112d40 0x7f4884152230 unknown :-1 s=CLOSED pgs=1439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 --2- 172.21.15.61:0/2797948975 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4864004500 0x7f48840674f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 >> 172.21.15.61:0/2797948975 conn(0x7f4884063480 msgr2=0x7f48840b3ff0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:27.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 shutdown_connections 2022-07-20T03:17:27.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.599+0000 7f488359e700 1 -- 172.21.15.61:0/2797948975 wait complete. 2022-07-20T03:17:27.615 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime' 2022-07-20T03:17:27.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 Processor -- start 2022-07-20T03:17:27.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 -- start start 2022-07-20T03:17:27.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0c640b2550 0x7f0c640977c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.721 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c64141330 0x7f0c64141790 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0c6405c4b0 0x7f0c6405c940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c640962e0 con 0x7f0c640b2550 2022-07-20T03:17:27.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c640b2930 con 0x7f0c6405c4b0 2022-07-20T03:17:27.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c68906700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c64097dc0 con 0x7f0c64141330 2022-07-20T03:17:27.725 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c6259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0c640b2550 0x7f0c640977c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c6259c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0c640b2550 0x7f0c640977c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51484/0 (socket says 172.21.15.61:51484) 2022-07-20T03:17:27.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c6259c700 1 -- 172.21.15.61:0/1818078657 learned_addr learned my addr 172.21.15.61:0/1818078657 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:27.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c62d9d700 1 --2- 172.21.15.61:0/1818078657 >> v2:172.21.15.117:3300/0 conn(0x7f0c6405c4b0 0x7f0c6405c940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.711+0000 7f0c61d9b700 1 --2- 172.21.15.61:0/1818078657 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c64141330 0x7f0c64141790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6259c700 1 -- 172.21.15.61:0/1818078657 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c64141330 msgr2=0x7f0c64141790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:27.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6259c700 1 --2- 172.21.15.61:0/1818078657 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c64141330 0x7f0c64141790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6259c700 1 -- 172.21.15.61:0/1818078657 >> v2:172.21.15.117:3300/0 conn(0x7f0c6405c4b0 msgr2=0x7f0c6405c940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6259c700 1 --2- 172.21.15.61:0/1818078657 >> v2:172.21.15.117:3300/0 conn(0x7f0c6405c4b0 0x7f0c6405c940 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6259c700 1 -- 172.21.15.61:0/1818078657 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c64098060 con 0x7f0c640b2550 2022-07-20T03:17:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c62d9d700 1 --2- 172.21.15.61:0/1818078657 >> v2:172.21.15.117:3300/0 conn(0x7f0c6405c4b0 0x7f0c6405c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:27.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6259c700 1 --2- 172.21.15.61:0/1818078657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c640b2550 0x7f0c640977c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f0c54002f70 tx=0x7f0c54006eb0).ready entity=mon.0 client_cookie=f674a625eec147ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:27.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6159a700 1 -- 172.21.15.61:0/1818078657 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c5400d570 con 0x7f0c640b2550 2022-07-20T03:17:27.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6159a700 1 -- 172.21.15.61:0/1818078657 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c54000e90 con 0x7f0c640b2550 2022-07-20T03:17:27.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c6159a700 1 -- 172.21.15.61:0/1818078657 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c5400d900 con 0x7f0c640b2550 2022-07-20T03:17:27.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 -- 172.21.15.61:0/1818078657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c640b2550 msgr2=0x7f0c640977c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 --2- 172.21.15.61:0/1818078657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c640b2550 0x7f0c640977c0 secure :-1 s=READY pgs=1440 cs=0 l=1 rev1=1 rx=0x7f0c54002f70 tx=0x7f0c54006eb0).stop 2022-07-20T03:17:27.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 -- 172.21.15.61:0/1818078657 shutdown_connections 2022-07-20T03:17:27.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 --2- 172.21.15.61:0/1818078657 >> v2:172.21.15.117:3300/0 conn(0x7f0c6405c4b0 0x7f0c6405c940 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 --2- 172.21.15.61:0/1818078657 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c64141330 0x7f0c64141790 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 --2- 172.21.15.61:0/1818078657 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c640b2550 0x7f0c640977c0 unknown :-1 s=CLOSED pgs=1440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 -- 172.21.15.61:0/1818078657 >> 172.21.15.61:0/1818078657 conn(0x7f0c640b36c0 msgr2=0x7f0c640b0af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:27.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 -- 172.21.15.61:0/1818078657 shutdown_connections 2022-07-20T03:17:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.715+0000 7f0c68906700 1 -- 172.21.15.61:0/1818078657 wait complete. 2022-07-20T03:17:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 Processor -- start 2022-07-20T03:17:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 -- start start 2022-07-20T03:17:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c6405c4b0 0x7f0c640ad590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c640b2550 0x7f0c641456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 0x7f0c64145bb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c640af710 con 0x7f0c64141330 2022-07-20T03:17:27.748 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c640af9b0 con 0x7f0c640b2550 2022-07-20T03:17:27.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c68906700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c640afc50 con 0x7f0c6405c4b0 2022-07-20T03:17:27.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 0x7f0c64145bb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 0x7f0c64145bb0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51494/0 (socket says 172.21.15.61:51494) 2022-07-20T03:17:27.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 -- 172.21.15.61:0/1857071645 learned_addr learned my addr 172.21.15.61:0/1857071645 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:27.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c61d9b700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c640b2550 0x7f0c641456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c6259c700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c6405c4b0 0x7f0c640ad590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 -- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c6405c4b0 msgr2=0x7f0c640ad590 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c6405c4b0 0x7f0c640ad590 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 -- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c640b2550 msgr2=0x7f0c641456c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c640b2550 0x7f0c641456c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:27.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 -- 172.21.15.61:0/1857071645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c5400b1e0 con 0x7f0c64141330 2022-07-20T03:17:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.719+0000 7f0c62d9d700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 0x7f0c64145bb0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f0c5c006cc0 tx=0x7f0c5c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c5c00f710 con 0x7f0c64141330 2022-07-20T03:17:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c5c00f870 con 0x7f0c64141330 2022-07-20T03:17:27.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c6413d720 con 0x7f0c64141330 2022-07-20T03:17:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c5c00fbd0 con 0x7f0c64141330 2022-07-20T03:17:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c6413f5d0 con 0x7f0c64141330 2022-07-20T03:17:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0c5c01a030 con 0x7f0c64141330 2022-07-20T03:17:27.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c640afe10 con 0x7f0c64141330 2022-07-20T03:17:27.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c537fe700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c400539b0 0x7f0c40055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:27.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0c5c06bbe0 con 0x7f0c64141330 2022-07-20T03:17:27.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c6259c700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c400539b0 0x7f0c40055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:27.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.723+0000 7f0c6259c700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c400539b0 0x7f0c40055e70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f0c5401c000 tx=0x7f0c54015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:27.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.727+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c5c042d10 con 0x7f0c64141330 2022-07-20T03:17:27.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:27.891+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]} v 0) v1 -- 0x7f0c64098300 con 0x7f0c64141330 2022-07-20T03:17:28.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.311+0000 7f0c537fe700 1 -- 172.21.15.61:0/1857071645 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Ended test tasks.cephfs.test_misc.TestMisc.test_root_rctime"]}]=0 v487) v1 ==== 127+0+0 (secure 0 0 0) 0x7f0c5c046330 con 0x7f0c64141330 2022-07-20T03:17:28.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c400539b0 msgr2=0x7f0c40055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c400539b0 0x7f0c40055e70 secure :-1 s=READY pgs=691 cs=0 l=1 rev1=1 rx=0x7f0c5401c000 tx=0x7f0c54015000).stop 2022-07-20T03:17:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 msgr2=0x7f0c64145bb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 0x7f0c64145bb0 secure :-1 s=READY pgs=1441 cs=0 l=1 rev1=1 rx=0x7f0c5c006cc0 tx=0x7f0c5c008040).stop 2022-07-20T03:17:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 shutdown_connections 2022-07-20T03:17:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c400539b0 0x7f0c40055e70 unknown :-1 s=CLOSED pgs=691 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c640b2550 0x7f0c641456c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c64141330 0x7f0c64145bb0 unknown :-1 s=CLOSED pgs=1441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 --2- 172.21.15.61:0/1857071645 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c6405c4b0 0x7f0c640ad590 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 >> 172.21.15.61:0/1857071645 conn(0x7f0c640b36c0 msgr2=0x7f0c64146e10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:28.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 shutdown_connections 2022-07-20T03:17:28.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.315+0000 7f0c68906700 1 -- 172.21.15.61:0/1857071645 wait complete. 2022-07-20T03:17:28.336 INFO:tasks.cephfs_test_runner:test_root_rctime (tasks.cephfs.test_misc.TestMisc) ... ok 2022-07-20T03:17:28.337 INFO:tasks.cephfs_test_runner:Starting test: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:17:28.338 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph log 'Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs' 2022-07-20T03:17:28.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.443+0000 7f74ba1f5700 1 Processor -- start 2022-07-20T03:17:28.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.443+0000 7f74ba1f5700 1 -- start start 2022-07-20T03:17:28.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b40b5690 0x7f74b4140730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:28.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b405cc80 con 0x7f74b4140c20 2022-07-20T03:17:28.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74b40af410 con 0x7f74b40af030 2022-07-20T03:17:28.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b4144050 con 0x7f74b40b5690 2022-07-20T03:17:28.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b89f2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b40b5690 0x7f74b4140730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:28.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b99f4700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f74b4140c20 0x7f74b41296f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48100/0 (socket says 172.21.15.61:48100) 2022-07-20T03:17:28.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b91f3700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7f74b40af030 0x7f74b405c570 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:56600/0 (socket says 172.21.15.61:56600) 2022-07-20T03:17:28.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b99f4700 1 -- 172.21.15.61:0/2713904353 learned_addr learned my addr 172.21.15.61:0/2713904353 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:28.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b89f2700 1 -- 172.21.15.61:0/2713904353 >> v1:172.21.15.117:6789/0 conn(0x7f74b40af030 legacy=0x7f74b405c570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74a3fff700 1 -- 172.21.15.61:0/2713904353 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1504817939 0 0) 0x7f74b40af410 con 0x7f74b40af030 2022-07-20T03:17:28.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b89f2700 1 -- 172.21.15.61:0/2713904353 >> v1:172.21.15.61:6789/0 conn(0x7f74b4140c20 legacy=0x7f74b41296f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b89f2700 1 -- 172.21.15.61:0/2713904353 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b405be30 con 0x7f74b40b5690 2022-07-20T03:17:28.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74b89f2700 1 --2- 172.21.15.61:0/2713904353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b40b5690 0x7f74b4140730 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f74ac001490 tx=0x7f74a4006df0).ready entity=mon.2 client_cookie=301f02bcc75e364b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:28.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74a3fff700 1 -- 172.21.15.61:0/2713904353 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a400d570 con 0x7f74b40b5690 2022-07-20T03:17:28.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74a3fff700 1 -- 172.21.15.61:0/2713904353 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74a4000e90 con 0x7f74b40b5690 2022-07-20T03:17:28.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74a3fff700 1 -- 172.21.15.61:0/2713904353 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74a400d8c0 con 0x7f74b40b5690 2022-07-20T03:17:28.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 -- 172.21.15.61:0/2713904353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b40b5690 msgr2=0x7f74b4140730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/2713904353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b40b5690 0x7f74b4140730 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f74ac001490 tx=0x7f74a4006df0).stop 2022-07-20T03:17:28.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 -- 172.21.15.61:0/2713904353 shutdown_connections 2022-07-20T03:17:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/2713904353 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b40b5690 0x7f74b4140730 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.447+0000 7f74ba1f5700 1 -- 172.21.15.61:0/2713904353 >> 172.21.15.61:0/2713904353 conn(0x7f74b40634c0 msgr2=0x7f74b40b4f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:28.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.451+0000 7f74ba1f5700 1 -- 172.21.15.61:0/2713904353 shutdown_connections 2022-07-20T03:17:28.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.451+0000 7f74ba1f5700 1 -- 172.21.15.61:0/2713904353 wait complete. 2022-07-20T03:17:28.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.451+0000 7f74ba1f5700 1 Processor -- start 2022-07-20T03:17:28.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.451+0000 7f74ba1f5700 1 -- start start 2022-07-20T03:17:28.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 0x7f74b412cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:28.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74b40b5690 0x7f74b413fc30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:28.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b4140c20 0x7f74b4140120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:28.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b4140610 con 0x7f74b40af030 2022-07-20T03:17:28.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74b4124490 con 0x7f74b40b5690 2022-07-20T03:17:28.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74b412d260 con 0x7f74b4140c20 2022-07-20T03:17:28.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 0x7f74b412cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:28.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 0x7f74b412cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51504/0 (socket says 172.21.15.61:51504) 2022-07-20T03:17:28.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 -- 172.21.15.61:0/51623449 learned_addr learned my addr 172.21.15.61:0/51623449 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b89f2700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74b40b5690 0x7f74b413fc30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b99f4700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b4140c20 0x7f74b4140120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:28.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 -- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b4140c20 msgr2=0x7f74b4140120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b4140c20 0x7f74b4140120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 -- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74b40b5690 msgr2=0x7f74b413fc30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74b40b5690 0x7f74b413fc30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 -- 172.21.15.61:0/51623449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74a400b180 con 0x7f74b40af030 2022-07-20T03:17:28.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b99f4700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b4140c20 0x7f74b4140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:28.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74b91f3700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 0x7f74b412cc10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f74b0000bc0 tx=0x7f74b0007a60).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:28.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b00070d0 con 0x7f74b40af030 2022-07-20T03:17:28.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0007230 con 0x7f74b40af030 2022-07-20T03:17:28.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74b412d500 con 0x7f74b40af030 2022-07-20T03:17:28.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b0005740 con 0x7f74b40af030 2022-07-20T03:17:28.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.455+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74b4153df0 con 0x7f74b40af030 2022-07-20T03:17:28.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.459+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74b405c0d0 con 0x7f74b40af030 2022-07-20T03:17:28.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.459+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f74b001a060 con 0x7f74b40af030 2022-07-20T03:17:28.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.459+0000 7f74a27fc700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7494053a40 0x7f7494055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:28.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.459+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f74b006aff0 con 0x7f74b40af030 2022-07-20T03:17:28.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.459+0000 7f74b89f2700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7494053a40 0x7f7494055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:28.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.459+0000 7f74b89f2700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7494053a40 0x7f7494055f00 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f74a401a000 tx=0x7f74a4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:28.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.463+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74b0042180 con 0x7f74b40af030 2022-07-20T03:17:28.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:28.631+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]} v 0) v1 -- 0x7f74b40574e0 con 0x7f74b40af030 2022-07-20T03:17:29.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.319+0000 7f74a27fc700 1 -- 172.21.15.61:0/51623449 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "log", "logtext": ["Starting test tasks.cephfs.test_misc.TestMisc.test_statfs_on_deleted_fs"]}]=0 v488) v1 ==== 139+0+0 (secure 0 0 0) 0x7f74b0018070 con 0x7f74b40af030 2022-07-20T03:17:29.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7494053a40 msgr2=0x7f7494055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7494053a40 0x7f7494055f00 secure :-1 s=READY pgs=692 cs=0 l=1 rev1=1 rx=0x7f74a401a000 tx=0x7f74a4014000).stop 2022-07-20T03:17:29.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 msgr2=0x7f74b412cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 0x7f74b412cc10 secure :-1 s=READY pgs=1443 cs=0 l=1 rev1=1 rx=0x7f74b0000bc0 tx=0x7f74b0007a60).stop 2022-07-20T03:17:29.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 shutdown_connections 2022-07-20T03:17:29.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7494053a40 0x7f7494055f00 unknown :-1 s=CLOSED pgs=692 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74b40b5690 0x7f74b413fc30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74b4140c20 0x7f74b4140120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 --2- 172.21.15.61:0/51623449 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74b40af030 0x7f74b412cc10 unknown :-1 s=CLOSED pgs=1443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 >> 172.21.15.61:0/51623449 conn(0x7f74b40634c0 msgr2=0x7f74b41294c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:29.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 shutdown_connections 2022-07-20T03:17:29.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.323+0000 7f74ba1f5700 1 -- 172.21.15.61:0/51623449 wait complete. 2022-07-20T03:17:29.342 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph config set mon mon_allow_pool_delete True 2022-07-20T03:17:29.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 Processor -- start 2022-07-20T03:17:29.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- start start 2022-07-20T03:17:29.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd2f005cf40 0x7fd2f00afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fd2f0147c90 0x7fd2f00b0210 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd2f01472b0 con 0x7fd2f00b1220 2022-07-20T03:17:29.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f0147550 con 0x7fd2f005cf40 2022-07-20T03:17:29.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd2f01477f0 con 0x7fd2f0147c90 2022-07-20T03:17:29.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f4906700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd2f005cf40 0x7fd2f00afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f5107700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd2f00b1220 0x7fd2f0099080 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48116/0 (socket says 172.21.15.61:48116) 2022-07-20T03:17:29.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f5107700 1 -- 172.21.15.61:0/677135968 learned_addr learned my addr 172.21.15.61:0/677135968 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:29.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2effff700 1 --2- 172.21.15.61:0/677135968 >> v2:172.21.15.117:3300/0 conn(0x7fd2f0147c90 0x7fd2f00b0210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2ef7fe700 1 -- 172.21.15.61:0/677135968 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840147308 0 0) 0x7fd2f01472b0 con 0x7fd2f00b1220 2022-07-20T03:17:29.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2ef7fe700 1 -- 172.21.15.61:0/677135968 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd2d4003260 con 0x7fd2f00b1220 2022-07-20T03:17:29.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f4906700 1 -- 172.21.15.61:0/677135968 >> v2:172.21.15.117:3300/0 conn(0x7fd2f0147c90 msgr2=0x7fd2f00b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:29.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f4906700 1 --2- 172.21.15.61:0/677135968 >> v2:172.21.15.117:3300/0 conn(0x7fd2f0147c90 0x7fd2f00b0210 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f4906700 1 -- 172.21.15.61:0/677135968 >> v1:172.21.15.61:6789/0 conn(0x7fd2f00b1220 legacy=0x7fd2f0099080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f4906700 1 -- 172.21.15.61:0/677135968 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2f0146fa0 con 0x7fd2f005cf40 2022-07-20T03:17:29.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f4906700 1 --2- 172.21.15.61:0/677135968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f00afd20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fd2dc002f70 tx=0x7fd2dc006df0).ready entity=mon.0 client_cookie=feb1d4549f12b6f6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:29.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2ef7fe700 1 -- 172.21.15.61:0/677135968 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2dc00d4f0 con 0x7fd2f005cf40 2022-07-20T03:17:29.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2ef7fe700 1 -- 172.21.15.61:0/677135968 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc000e90 con 0x7fd2f005cf40 2022-07-20T03:17:29.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2ef7fe700 1 -- 172.21.15.61:0/677135968 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2dc00d840 con 0x7fd2f005cf40 2022-07-20T03:17:29.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/677135968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 msgr2=0x7fd2f00afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/677135968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f00afd20 secure :-1 s=READY pgs=1445 cs=0 l=1 rev1=1 rx=0x7fd2dc002f70 tx=0x7fd2dc006df0).stop 2022-07-20T03:17:29.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/677135968 shutdown_connections 2022-07-20T03:17:29.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/677135968 >> v2:172.21.15.117:3300/0 conn(0x7fd2f0147c90 0x7fd2f00b0210 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/677135968 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f00afd20 unknown :-1 s=CLOSED pgs=1445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/677135968 >> 172.21.15.61:0/677135968 conn(0x7fd2f00b5b70 msgr2=0x7fd2f00b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:29.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/677135968 shutdown_connections 2022-07-20T03:17:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.451+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/677135968 wait complete. 2022-07-20T03:17:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 Processor -- start 2022-07-20T03:17:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 -- start start 2022-07-20T03:17:29.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f0149fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd2f00b1220 0x7fd2f00b3720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2f0147c90 0x7fd2f01432a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.480 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f01421b0 con 0x7fd2f005cf40 2022-07-20T03:17:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f0149fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0141940 con 0x7fd2f0147c90 2022-07-20T03:17:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f6b6a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd2f0141be0 con 0x7fd2f00b1220 2022-07-20T03:17:29.481 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f0149fe0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51520/0 (socket says 172.21.15.61:51520) 2022-07-20T03:17:29.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 -- 172.21.15.61:0/3156882162 learned_addr learned my addr 172.21.15.61:0/3156882162 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:29.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2effff700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd2f00b1220 0x7fd2f00b3720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f5107700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2f0147c90 0x7fd2f01432a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 -- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd2f00b1220 msgr2=0x7fd2f00b3720 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd2f00b1220 0x7fd2f00b3720 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 -- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2f0147c90 msgr2=0x7fd2f01432a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2f0147c90 0x7fd2f01432a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2f4906700 1 -- 172.21.15.61:0/3156882162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd2dc00b180 con 0x7fd2f005cf40 2022-07-20T03:17:29.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.455+0000 7fd2effff700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd2f00b1220 0x7fd2f00b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:29.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2f4906700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f0149fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7fd2dc002f70 tx=0x7fd2dc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:29.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2dc016070 con 0x7fd2f005cf40 2022-07-20T03:17:29.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd2dc013850 con 0x7fd2f005cf40 2022-07-20T03:17:29.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd2f00b43c0 con 0x7fd2f005cf40 2022-07-20T03:17:29.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd2dc013bb0 con 0x7fd2f005cf40 2022-07-20T03:17:29.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd2f01415c0 con 0x7fd2f005cf40 2022-07-20T03:17:29.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd2f0141da0 con 0x7fd2f005cf40 2022-07-20T03:17:29.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd2dc025030 con 0x7fd2f005cf40 2022-07-20T03:17:29.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.459+0000 7fd2edffb700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2d0053a40 0x7fd2d0055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.463+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd2dc06caa0 con 0x7fd2f005cf40 2022-07-20T03:17:29.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.463+0000 7fd2effff700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2d0053a40 0x7fd2d0055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.463+0000 7fd2effff700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2d0053a40 0x7fd2d0055f00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fd2e400b000 tx=0x7fd2e4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:29.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.467+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd2dc043bd0 con 0x7fd2f005cf40 2022-07-20T03:17:29.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.595+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_allow_pool_delete}] v 0) v1 -- 0x7fd2f005d320 con 0x7fd2f005cf40 2022-07-20T03:17:29.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.603+0000 7fd2edffb700 1 -- 172.21.15.61:0/3156882162 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mon_allow_pool_delete}]=0 v24)=0 v24) v1 ==== 122+0+0 (secure 0 0 0) 0x7fd2dc023070 con 0x7fd2f005cf40 2022-07-20T03:17:29.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2d0053a40 msgr2=0x7fd2d0055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2d0053a40 0x7fd2d0055f00 secure :-1 s=READY pgs=694 cs=0 l=1 rev1=1 rx=0x7fd2e400b000 tx=0x7fd2e4006df0).stop 2022-07-20T03:17:29.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 msgr2=0x7fd2f0149fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f0149fe0 secure :-1 s=READY pgs=1446 cs=0 l=1 rev1=1 rx=0x7fd2dc002f70 tx=0x7fd2dc014000).stop 2022-07-20T03:17:29.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 shutdown_connections 2022-07-20T03:17:29.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd2d0053a40 0x7fd2d0055f00 unknown :-1 s=CLOSED pgs=694 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd2f0147c90 0x7fd2f01432a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd2f00b1220 0x7fd2f00b3720 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 --2- 172.21.15.61:0/3156882162 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd2f005cf40 0x7fd2f0149fe0 unknown :-1 s=CLOSED pgs=1446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 >> 172.21.15.61:0/3156882162 conn(0x7fd2f00b5b70 msgr2=0x7fd2f0144f30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:29.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 shutdown_connections 2022-07-20T03:17:29.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.607+0000 7fd2f6b6a700 1 -- 172.21.15.61:0/3156882162 wait complete. 2022-07-20T03:17:29.624 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:17:29.625 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:29.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:29.671 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:29.694 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:29.720 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:17:29.742 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:17:29.743 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:29.743 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:29.744 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:29.752 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': No such file or directory 2022-07-20T03:17:29.753 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:29.755 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.0 2022-07-20T03:17:29.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:29.756 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:29.756 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:29.768 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:29.770 INFO:teuthology.orchestra.run.smithi163.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.1': No such file or directory 2022-07-20T03:17:29.771 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:29.772 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:29.773 DEBUG:teuthology.orchestra.run.smithi179:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:29.774 DEBUG:teuthology.orchestra.run.smithi179:> ' -- /home/ubuntu/cephtest/mnt.2) 2022-07-20T03:17:29.790 INFO:teuthology.orchestra.run.smithi179.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.2': No such file or directory 2022-07-20T03:17:29.791 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:17:29.792 INFO:tasks.cephfs.fuse_mount:mount point does not exist: /home/ubuntu/cephtest/mnt.2 2022-07-20T03:17:29.793 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:17:29.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 Processor -- start 2022-07-20T03:17:29.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- start start 2022-07-20T03:17:29.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f02400aea20 0x7f0240140a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02400675f0 0x7f0240140f80 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f02400aee00 con 0x7f0240141470 2022-07-20T03:17:29.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0240144470 con 0x7f02400675f0 2022-07-20T03:17:29.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0240144710 con 0x7f02400aea20 2022-07-20T03:17:29.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02400675f0 0x7f0240140f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f02400675f0 0x7f0240140f80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51530/0 (socket says 172.21.15.61:51530) 2022-07-20T03:17:29.913 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023ed9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f0240141470 0x7f024013a110 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48128/0 (socket says 172.21.15.61:48128) 2022-07-20T03:17:29.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023e59c700 1 --2- 172.21.15.61:0/3367231026 >> v2:172.21.15.117:3300/0 conn(0x7f02400aea20 0x7f0240140a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 -- 172.21.15.61:0/3367231026 learned_addr learned my addr 172.21.15.61:0/3367231026 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:29.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 -- 172.21.15.61:0/3367231026 >> v2:172.21.15.117:3300/0 conn(0x7f02400aea20 msgr2=0x7f0240140a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:29.914 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 --2- 172.21.15.61:0/3367231026 >> v2:172.21.15.117:3300/0 conn(0x7f02400aea20 0x7f0240140a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 -- 172.21.15.61:0/3367231026 >> v1:172.21.15.61:6789/0 conn(0x7f0240141470 legacy=0x7f024013a110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023d59a700 1 -- 172.21.15.61:0/3367231026 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3301835528 0 0) 0x7f02400aee00 con 0x7f0240141470 2022-07-20T03:17:29.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 -- 172.21.15.61:0/3367231026 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02401449b0 con 0x7f02400675f0 2022-07-20T03:17:29.915 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023dd9b700 1 --2- 172.21.15.61:0/3367231026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02400675f0 0x7f0240140f80 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f02400b1750 tx=0x7f0228006eb0).ready entity=mon.0 client_cookie=4f7657c487204468 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:29.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023d59a700 1 -- 172.21.15.61:0/3367231026 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f022800d610 con 0x7f02400675f0 2022-07-20T03:17:29.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023d59a700 1 -- 172.21.15.61:0/3367231026 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0228000e90 con 0x7f02400675f0 2022-07-20T03:17:29.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f023d59a700 1 -- 172.21.15.61:0/3367231026 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f022800d9a0 con 0x7f02400675f0 2022-07-20T03:17:29.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- 172.21.15.61:0/3367231026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02400675f0 msgr2=0x7f0240140f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 --2- 172.21.15.61:0/3367231026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02400675f0 0x7f0240140f80 secure :-1 s=READY pgs=1448 cs=0 l=1 rev1=1 rx=0x7f02400b1750 tx=0x7f0228006eb0).stop 2022-07-20T03:17:29.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- 172.21.15.61:0/3367231026 shutdown_connections 2022-07-20T03:17:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 --2- 172.21.15.61:0/3367231026 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f02400675f0 0x7f0240140f80 unknown :-1 s=CLOSED pgs=1448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 --2- 172.21.15.61:0/3367231026 >> v2:172.21.15.117:3300/0 conn(0x7f02400aea20 0x7f0240140a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- 172.21.15.61:0/3367231026 >> 172.21.15.61:0/3367231026 conn(0x7f02400611c0 msgr2=0x7f0240062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:29.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- 172.21.15.61:0/3367231026 shutdown_connections 2022-07-20T03:17:29.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.903+0000 7f0244e5a700 1 -- 172.21.15.61:0/3367231026 wait complete. 2022-07-20T03:17:29.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 Processor -- start 2022-07-20T03:17:29.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 -- start start 2022-07-20T03:17:29.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02400aea20 0x7f0240150ef0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0240141470 0x7f024013d6d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 0x7f024013dbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02401513e0 con 0x7f0240066c30 2022-07-20T03:17:29.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f024013a120 con 0x7f0240141470 2022-07-20T03:17:29.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.907+0000 7f0244e5a700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f024013a350 con 0x7f02400aea20 2022-07-20T03:17:29.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 0x7f024013dbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0240141470 0x7f024013d6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 0x7f024013dbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51538/0 (socket says 172.21.15.61:51538) 2022-07-20T03:17:29.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023e59c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02400aea20 0x7f0240150ef0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023ed9d700 1 -- 172.21.15.61:0/83808438 learned_addr learned my addr 172.21.15.61:0/83808438 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:29.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 -- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02400aea20 msgr2=0x7f0240150ef0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02400aea20 0x7f0240150ef0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 -- 172.21.15.61:0/83808438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 msgr2=0x7f024013dbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:29.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 0x7f024013dbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:29.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 -- 172.21.15.61:0/83808438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f022800b1e0 con 0x7f0240141470 2022-07-20T03:17:29.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023ed9d700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 0x7f024013dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:29.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f023dd9b700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0240141470 0x7f024013d6d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f02400b1750 tx=0x7f0228015000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:29.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f022800d290 con 0x7f0240141470 2022-07-20T03:17:29.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f02280068b0 con 0x7f0240141470 2022-07-20T03:17:29.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f024013a5f0 con 0x7f0240141470 2022-07-20T03:17:29.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0228011d70 con 0x7f0240141470 2022-07-20T03:17:29.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f024013e130 con 0x7f0240141470 2022-07-20T03:17:29.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.911+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0240144e50 con 0x7f0240141470 2022-07-20T03:17:29.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.915+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f022801a040 con 0x7f0240141470 2022-07-20T03:17:29.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.915+0000 7f02377fe700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f02240539b0 0x7f0224055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:29.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.915+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f022806d180 con 0x7f0240141470 2022-07-20T03:17:29.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.915+0000 7f023e59c700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f02240539b0 0x7f0224055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:29.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.915+0000 7f023e59c700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f02240539b0 0x7f0224055e70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f023000b000 tx=0x7f02300091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:29.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:29.923+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f02280442b0 con 0x7f0240141470 2022-07-20T03:17:30.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.051+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7f02400574e0 con 0x7f0240141470 2022-07-20T03:17:30.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.055+0000 7f02377fe700 1 -- 172.21.15.61:0/83808438 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v143) v1 ==== 81+0+28885 (secure 0 0 0) 0x7f0228002f50 con 0x7f0240141470 2022-07-20T03:17:30.059 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:30.060 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:17:30.060 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 143, 2022-07-20T03:17:30.060 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:17:30.060 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:17:30.061 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:17:27.298209+0000", 2022-07-20T03:17:30.061 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:30.061 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:30.061 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:17:30.062 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:17:30.062 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:17:30.062 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:17:30.062 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:17:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:17:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:17:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:30.063 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:17:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:17:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:17:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:17:30.064 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:17:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 28, 2022-07-20T03:17:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:17:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:17:30.065 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:17:30.066 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:17:30.066 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:17:30.066 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.066 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:17:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:17:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:17:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:30.067 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:30.068 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:30.068 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:30.068 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:30.069 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:30.069 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:30.070 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:30.070 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:17:30.070 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:17:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:17:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:17:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:17:30.071 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:30.072 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:30.072 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:30.072 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:30.073 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:17:30.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:30.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:30.074 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:30.075 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:30.076 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:30.077 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:30.077 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:30.077 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:30.078 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:30.078 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:30.079 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:30.079 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:30.080 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:30.081 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:30.081 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:30.082 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:30.083 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.083 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:30.084 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:30.084 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:30.085 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:30.085 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:30.086 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:30.086 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:30.087 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:30.087 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:30.088 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:30.088 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:30.089 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:17:30.089 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:17:30.090 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.090 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:30.091 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:17:30.091 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.092 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.092 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.093 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:17:30.093 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:17:30.094 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:17:30.094 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:30.095 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:30.095 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:30.096 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:30.097 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:30.097 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:30.098 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:30.098 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:30.099 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:30.100 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:30.100 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:30.101 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:30.101 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:30.102 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:30.104 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.105 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:17:30.105 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:30.106 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:30.106 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:30.107 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:30.107 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:30.108 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:30.109 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:30.109 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:30.110 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:30.110 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:30.111 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:30.111 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:30.112 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:30.112 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:30.113 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:30.113 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:30.114 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:30.114 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:30.115 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:30.116 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:30.116 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:30.117 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:30.117 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:30.118 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:30.118 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:30.119 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:30.119 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.120 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:30.120 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:30.121 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:30.121 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:30.122 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:30.122 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:30.123 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:30.123 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:30.124 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:30.124 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:30.125 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:30.125 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:17:30.126 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:30.126 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:17:30.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.127 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.128 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:30.128 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:17:30.129 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.129 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:30.129 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:17:30.130 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.130 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.130 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.131 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.132 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.133 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.134 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.134 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:17:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.135 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.136 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.136 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.137 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:17:30.140 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.141 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.141 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.142 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.143 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.143 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.144 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.144 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:17:30.145 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.145 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.146 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.147 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:17:30.147 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.148 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.149 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.149 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.150 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.150 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:17:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.151 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.152 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.152 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:17:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.153 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.154 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.154 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.155 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.155 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.156 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:17:30.157 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.157 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.158 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.158 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.159 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:17:30.159 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.160 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:17:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:17:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:17:30.161 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:17:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.162 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:30.163 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:17:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.164 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.165 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.166 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.167 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:17:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.168 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:17:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.169 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.170 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.171 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.171 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:17:30.172 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.172 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.173 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.173 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.173 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:17:30.174 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.174 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.175 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.175 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.176 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.176 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.177 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.177 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.178 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:17:30.178 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.179 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.179 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.180 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.180 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:17:30.181 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.181 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.182 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.182 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.183 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.183 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.185 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.185 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:17:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.187 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.188 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.188 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.189 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:17:30.189 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:30.190 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.190 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.191 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.191 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:17:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:17:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:17:30.192 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:17:30.193 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.193 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.193 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.194 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.194 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.194 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.195 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:30.195 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:17:30.196 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.196 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.196 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.197 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.197 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.198 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.198 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.199 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 137, 2022-07-20T03:17:30.199 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.200 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.200 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.200 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.200 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.200 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.201 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:17:30.201 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.201 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.201 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.202 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.202 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:17:30.202 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.202 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.203 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.203 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.203 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.203 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:17:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.204 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.205 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.205 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.205 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:17:30.205 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.206 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.206 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.206 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.206 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.207 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.207 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.207 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.208 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:17:30.208 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.209 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.210 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.210 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:17:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.211 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.212 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.212 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.213 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.214 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.214 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:17:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.215 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.216 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.216 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:17:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.217 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.218 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.218 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:17:30.219 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:17:30.220 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:17:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:17:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.221 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.222 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.222 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.223 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:30.224 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:17:30.224 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.225 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.226 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.226 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.227 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.227 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 127, 2022-07-20T03:17:30.228 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.229 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.229 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.231 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.231 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.231 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:17:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.232 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.233 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:17:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.234 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.235 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.235 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.235 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.236 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.236 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.236 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:17:30.236 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.237 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:17:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.238 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.239 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:17:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.240 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:17:30.241 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.242 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.242 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.243 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.243 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.243 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.244 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.244 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:17:30.245 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.245 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.246 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.246 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.246 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:17:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.247 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:17:30.248 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:17:30.249 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:17:30.249 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:17:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.250 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.251 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.251 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:30.252 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:17:30.252 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.253 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.253 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.254 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.254 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.255 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.255 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.256 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 137, 2022-07-20T03:17:30.256 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.257 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.257 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.258 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.258 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.259 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.259 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:17:30.260 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.260 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.261 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.261 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.261 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:17:30.262 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.262 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.262 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.262 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.263 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.263 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.263 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.263 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.264 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:17:30.264 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.265 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.265 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.266 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.266 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:17:30.267 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.267 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.267 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.268 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.268 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.269 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.269 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.270 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.270 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:17:30.271 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.271 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.271 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.272 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.272 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:17:30.273 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.273 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.274 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.275 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.275 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.276 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.276 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.276 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:17:30.277 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.277 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.278 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.278 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.279 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:17:30.279 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:30.279 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.280 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.280 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:17:30.281 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:17:30.281 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:17:30.281 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:17:30.282 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.282 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.283 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.283 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.284 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.284 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.284 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:30.285 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:17:30.285 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.286 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.286 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.287 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.287 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.287 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.288 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.288 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:30.289 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.289 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.289 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.290 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.290 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.290 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.291 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:17:30.291 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.291 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.292 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.292 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.292 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:17:30.293 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.293 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.293 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.294 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.294 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.294 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.295 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.295 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.295 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:17:30.296 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.296 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.296 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.297 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.297 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:17:30.297 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.298 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.298 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.299 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.299 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.299 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.300 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.301 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:17:30.301 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.302 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.302 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.303 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.303 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:17:30.303 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.304 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.304 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.305 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.306 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.306 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.307 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.307 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:17:30.307 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.308 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.309 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.309 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.310 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:17:30.310 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:30.311 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.311 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.312 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:17:30.312 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:17:30.313 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:17:30.313 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:17:30.314 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.315 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.316 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.316 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.317 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:30.318 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:17:30.318 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.319 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.319 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.320 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.320 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.320 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.321 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.321 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:30.322 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.322 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.322 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.323 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.323 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:17:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.324 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.325 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.326 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:17:30.326 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.327 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.327 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.327 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.328 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.330 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:17:30.330 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.331 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.331 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.331 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.332 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:17:30.332 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.333 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.333 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.334 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.334 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.335 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.335 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.336 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.336 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:17:30.336 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.337 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.337 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.338 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.338 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:17:30.339 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.339 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.339 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.339 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.340 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:17:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.341 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.342 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.342 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.342 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:17:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.343 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.344 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:17:30.344 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:17:30.345 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:17:30.345 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:17:30.346 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.346 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.347 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.347 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:30.348 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:17:30.349 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:30.349 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:30.350 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:30.350 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:30.351 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:30.351 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:30.352 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:30.352 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 138, 2022-07-20T03:17:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:30.353 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:30.354 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:30.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.355 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.355 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:17:30.356 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:17:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.359 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.360 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:30.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.361 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:17:30.362 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.363 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.364 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:17:30.364 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.365 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.366 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:17:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.367 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.368 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.368 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:17:30.368 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.369 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:30.370 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:30.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:17:30.372 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:30.373 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:17:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.374 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:17:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:17:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:17:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:17:30.375 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:30.376 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:17:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.377 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.378 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:17:30.379 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:30.380 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.381 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.381 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.382 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.382 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.383 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:17:30.383 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.384 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.384 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.385 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:30.385 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.386 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.386 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.387 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.388 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:17:30.388 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.389 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.390 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:30.390 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.391 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.392 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.393 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.393 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:17:30.394 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.394 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:30.395 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.396 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:17:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.398 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:17:30.399 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:30.400 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.401 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.402 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:17:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:30.403 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:30.404 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:17:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.405 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:17:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:17:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:17:30.406 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:17:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": { 2022-07-20T03:17:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6833/3390219350": "2022-07-21T03:17:24.127393+0000", 2022-07-20T03:17:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: "172.21.15.61:6832/3390219350": "2022-07-21T03:17:24.127393+0000" 2022-07-20T03:17:30.407 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:17:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:17:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:17:30.408 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:17:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:17:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:17:30.409 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:17:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:17:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:30.410 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:17:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:17:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:17:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:17:30.411 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:17:30.411 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:30.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f02240539b0 msgr2=0x7f0224055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f02240539b0 0x7f0224055e70 secure :-1 s=READY pgs=695 cs=0 l=1 rev1=1 rx=0x7f023000b000 tx=0x7f02300091d0).stop 2022-07-20T03:17:30.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0240141470 msgr2=0x7f024013d6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0240141470 0x7f024013d6d0 secure :-1 s=READY pgs=645 cs=0 l=1 rev1=1 rx=0x7f02400b1750 tx=0x7f0228015000).stop 2022-07-20T03:17:30.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 shutdown_connections 2022-07-20T03:17:30.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f02240539b0 0x7f0224055e70 unknown :-1 s=CLOSED pgs=695 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0240141470 0x7f024013d6d0 unknown :-1 s=CLOSED pgs=645 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0240066c30 0x7f024013dbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 --2- 172.21.15.61:0/83808438 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f02400aea20 0x7f0240150ef0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 >> 172.21.15.61:0/83808438 conn(0x7f02400675f0 msgr2=0x7f0240139570 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:30.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 shutdown_connections 2022-07-20T03:17:30.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.059+0000 7f0244e5a700 1 -- 172.21.15.61:0/83808438 wait complete. 2022-07-20T03:17:30.417 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:30.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 Processor -- start 2022-07-20T03:17:30.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 -- start start 2022-07-20T03:17:30.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f980b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 0x7f8f98147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f8f98147010 con 0x7f8f980af8e0 2022-07-20T03:17:30.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f8f981472b0 con 0x7f8f980adff0 2022-07-20T03:17:30.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9e399700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f98146d40 con 0x7f8f980b5650 2022-07-20T03:17:30.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f980b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9cb96700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 0x7f8f98147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f980b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38214/0 (socket says 172.21.15.61:38214) 2022-07-20T03:17:30.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9cb96700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 0x7f8f98147750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:32992/0 (socket says 172.21.15.61:32992) 2022-07-20T03:17:30.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 -- 172.21.15.61:0/2818000915 learned_addr learned my addr 172.21.15.61:0/2818000915 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:30.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 -- 172.21.15.61:0/2818000915 >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 msgr2=0x7f8f98147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:30.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 --2- 172.21.15.61:0/2818000915 >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 0x7f8f98147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 -- 172.21.15.61:0/2818000915 >> v1:172.21.15.117:6789/0 conn(0x7f8f980af8e0 legacy=0x7f8f9805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 -- 172.21.15.61:0/2818000915 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f98147c40 con 0x7f8f980b5650 2022-07-20T03:17:30.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9cb96700 1 --2- 172.21.15.61:0/2818000915 >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 0x7f8f98147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:30.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f9d397700 1 --2- 172.21.15.61:0/2818000915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f980b5a30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8f8c002f70 tx=0x7f8f8c006eb0).ready entity=mon.2 client_cookie=a0c11b3cc8c48712 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:30.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f97fff700 1 -- 172.21.15.61:0/2818000915 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f8c00d570 con 0x7f8f980b5650 2022-07-20T03:17:30.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f97fff700 1 -- 172.21.15.61:0/2818000915 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8c000e90 con 0x7f8f980b5650 2022-07-20T03:17:30.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.527+0000 7f8f97fff700 1 -- 172.21.15.61:0/2818000915 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f8c00d900 con 0x7f8f980b5650 2022-07-20T03:17:30.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- 172.21.15.61:0/2818000915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 msgr2=0x7f8f980b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 --2- 172.21.15.61:0/2818000915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f980b5a30 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f8f8c002f70 tx=0x7f8f8c006eb0).stop 2022-07-20T03:17:30.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- 172.21.15.61:0/2818000915 shutdown_connections 2022-07-20T03:17:30.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 --2- 172.21.15.61:0/2818000915 >> v2:172.21.15.117:3300/0 conn(0x7f8f980adff0 0x7f8f98147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 --2- 172.21.15.61:0/2818000915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f980b5a30 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- 172.21.15.61:0/2818000915 >> 172.21.15.61:0/2818000915 conn(0x7f8f980634e0 msgr2=0x7f8f980b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:30.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- 172.21.15.61:0/2818000915 shutdown_connections 2022-07-20T03:17:30.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- 172.21.15.61:0/2818000915 wait complete. 2022-07-20T03:17:30.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 Processor -- start 2022-07-20T03:17:30.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- start start 2022-07-20T03:17:30.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f980adff0 0x7f8f9813f770 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 0x7f8f98140170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f98140660 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f9813fc60 con 0x7f8f980af8e0 2022-07-20T03:17:30.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f980672c0 con 0x7f8f980adff0 2022-07-20T03:17:30.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9e399700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f8f98067510 con 0x7f8f980b5650 2022-07-20T03:17:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9cb96700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 0x7f8f98140170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9cb96700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 0x7f8f98140170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51550/0 (socket says 172.21.15.61:51550) 2022-07-20T03:17:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9d397700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f980adff0 0x7f8f9813f770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9cb96700 1 -- 172.21.15.61:0/1650169872 learned_addr learned my addr 172.21.15.61:0/1650169872 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.531+0000 7f8f9db98700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f98140660 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9cb96700 1 -- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 msgr2=0x7f8f98140660 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9cb96700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f98140660 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9cb96700 1 -- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f980adff0 msgr2=0x7f8f9813f770 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9cb96700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f980adff0 0x7f8f9813f770 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9cb96700 1 -- 172.21.15.61:0/1650169872 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f88009420 con 0x7f8f980af8e0 2022-07-20T03:17:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9d397700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f980adff0 0x7f8f9813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9cb96700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 0x7f8f98140170 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f8f8800c000 tx=0x7f8f880091a0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:30.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f8800c850 con 0x7f8f980af8e0 2022-07-20T03:17:30.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f8f8800c9b0 con 0x7f8f980af8e0 2022-07-20T03:17:30.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f8c00b1e0 con 0x7f8f980af8e0 2022-07-20T03:17:30.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f8f8800cd10 con 0x7f8f980af8e0 2022-07-20T03:17:30.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f98066400 con 0x7f8f980af8e0 2022-07-20T03:17:30.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f981427e0 con 0x7f8f980af8e0 2022-07-20T03:17:30.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f8f880123f0 con 0x7f8f980af8e0 2022-07-20T03:17:30.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f967fc700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f780539b0 0x7f8f78055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f8f8806bfd0 con 0x7f8f980af8e0 2022-07-20T03:17:30.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.535+0000 7f8f9d397700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f780539b0 0x7f8f78055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.539+0000 7f8f9d397700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f780539b0 0x7f8f78055e70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f8f8c01c000 tx=0x7f8f8c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:30.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.543+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f8f981427e0 con 0x7f8f980af8e0 2022-07-20T03:17:30.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.707+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f8f981427e0 con 0x7f8f980af8e0 2022-07-20T03:17:30.712 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.707+0000 7f8f967fc700 1 -- 172.21.15.61:0/1650169872 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f8f981427e0 con 0x7f8f980af8e0 2022-07-20T03:17:30.712 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:30.713 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8706,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3042270298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3042270298},{"type":"v1","addr":"172.21.15.61:6833","nonce":3042270298}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[]} 2022-07-20T03:17:30.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.707+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f780539b0 msgr2=0x7f8f78055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.707+0000 7f8f9e399700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f780539b0 0x7f8f78055e70 secure :-1 s=READY pgs=696 cs=0 l=1 rev1=1 rx=0x7f8f8c01c000 tx=0x7f8f8c015000).stop 2022-07-20T03:17:30.714 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.707+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 msgr2=0x7f8f98140170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.707+0000 7f8f9e399700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 0x7f8f98140170 secure :-1 s=READY pgs=1449 cs=0 l=1 rev1=1 rx=0x7f8f8800c000 tx=0x7f8f880091a0).stop 2022-07-20T03:17:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 shutdown_connections 2022-07-20T03:17:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f8f780539b0 0x7f8f78055e70 unknown :-1 s=CLOSED pgs=696 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.715 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f8f980b5650 0x7f8f98140660 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f8f980af8e0 0x7f8f98140170 unknown :-1 s=CLOSED pgs=1449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 --2- 172.21.15.61:0/1650169872 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f8f980adff0 0x7f8f9813f770 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 >> 172.21.15.61:0/1650169872 conn(0x7f8f980634e0 msgr2=0x7f8f980ae420 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:30.716 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 shutdown_connections 2022-07-20T03:17:30.717 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.711+0000 7f8f9e399700 1 -- 172.21.15.61:0/1650169872 wait complete. 2022-07-20T03:17:30.717 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 80 2022-07-20T03:17:30.727 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.823+0000 7f86c682d700 1 Processor -- start 2022-07-20T03:17:30.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.823+0000 7f86c682d700 1 -- start start 2022-07-20T03:17:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f86c00b1ef0 0x7f86c00605a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c0140940 con 0x7f86c00ab180 2022-07-20T03:17:30.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f86c00b22d0 con 0x7f86c013f980 2022-07-20T03:17:30.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f86c005fbc0 con 0x7f86c00b1ef0 2022-07-20T03:17:30.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c602c700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f86c00ab180 0x7f86c00ab5d0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48158/0 (socket says 172.21.15.61:48158) 2022-07-20T03:17:30.846 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c602c700 1 -- 172.21.15.61:0/305546549 learned_addr learned my addr 172.21.15.61:0/305546549 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:30.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c582b700 1 --2- 172.21.15.61:0/305546549 >> v2:172.21.15.117:3300/0 conn(0x7f86c00b1ef0 0x7f86c00605a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c582b700 1 -- 172.21.15.61:0/305546549 >> v1:172.21.15.117:6789/0 conn(0x7f86c013f980 legacy=0x7f86c00abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c582b700 1 -- 172.21.15.61:0/305546549 >> v1:172.21.15.61:6789/0 conn(0x7f86c00ab180 legacy=0x7f86c00ab5d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c582b700 1 -- 172.21.15.61:0/305546549 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86c005fe60 con 0x7f86c00b1ef0 2022-07-20T03:17:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c582b700 1 --2- 172.21.15.61:0/305546549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c00b1ef0 0x7f86c00605a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f86bc002f70 tx=0x7f86bc006df0).ready entity=mon.1 client_cookie=a5fc7c4f2e9b1349 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c4829700 1 -- 172.21.15.61:0/305546549 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86bc00d610 con 0x7f86c00b1ef0 2022-07-20T03:17:30.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c4829700 1 -- 172.21.15.61:0/305546549 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc000e90 con 0x7f86c00b1ef0 2022-07-20T03:17:30.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c4829700 1 -- 172.21.15.61:0/305546549 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86bc00d960 con 0x7f86c00b1ef0 2022-07-20T03:17:30.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- 172.21.15.61:0/305546549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c00b1ef0 msgr2=0x7f86c00605a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 --2- 172.21.15.61:0/305546549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c00b1ef0 0x7f86c00605a0 secure :-1 s=READY pgs=648 cs=0 l=1 rev1=1 rx=0x7f86bc002f70 tx=0x7f86bc006df0).stop 2022-07-20T03:17:30.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- 172.21.15.61:0/305546549 shutdown_connections 2022-07-20T03:17:30.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 --2- 172.21.15.61:0/305546549 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c00b1ef0 0x7f86c00605a0 unknown :-1 s=CLOSED pgs=648 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- 172.21.15.61:0/305546549 >> 172.21.15.61:0/305546549 conn(0x7f86c00b2ca0 msgr2=0x7f86c00b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:30.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- 172.21.15.61:0/305546549 shutdown_connections 2022-07-20T03:17:30.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.827+0000 7f86c682d700 1 -- 172.21.15.61:0/305546549 wait complete. 2022-07-20T03:17:30.853 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 Processor -- start 2022-07-20T03:17:30.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 -- start start 2022-07-20T03:17:30.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 0x7f86c00aff30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86c00b1ef0 0x7f86c0152650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c013f980 0x7f86c0139690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0155cd0 con 0x7f86c00ab180 2022-07-20T03:17:30.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86c009af50 con 0x7f86c013f980 2022-07-20T03:17:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f86c0152b40 con 0x7f86c00b1ef0 2022-07-20T03:17:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 0x7f86c00aff30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.856 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 0x7f86c00aff30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51562/0 (socket says 172.21.15.61:51562) 2022-07-20T03:17:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 -- 172.21.15.61:0/3468075663 learned_addr learned my addr 172.21.15.61:0/3468075663 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c502a700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86c00b1ef0 0x7f86c0152650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c602c700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c013f980 0x7f86c0139690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 -- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86c00b1ef0 msgr2=0x7f86c0152650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86c00b1ef0 0x7f86c0152650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 -- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c013f980 msgr2=0x7f86c0139690 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c013f980 0x7f86c0139690 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:30.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 -- 172.21.15.61:0/3468075663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86bc00b180 con 0x7f86c00ab180 2022-07-20T03:17:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c502a700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86c00b1ef0 0x7f86c0152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c602c700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c013f980 0x7f86c0139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c582b700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 0x7f86c00aff30 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f86c009b0e0 tx=0x7f86bc015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:30.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86bc00de70 con 0x7f86c00ab180 2022-07-20T03:17:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f86bc011c60 con 0x7f86c00ab180 2022-07-20T03:17:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86c009acb0 con 0x7f86c00ab180 2022-07-20T03:17:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f86bc01e460 con 0x7f86c00ab180 2022-07-20T03:17:30.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.831+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86c0099e70 con 0x7f86c00ab180 2022-07-20T03:17:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.835+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86c0060100 con 0x7f86c00ab180 2022-07-20T03:17:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.835+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f86bc025030 con 0x7f86c00ab180 2022-07-20T03:17:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.835+0000 7f86aeffd700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86a00539b0 0x7f86a0055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:30.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.835+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(143..143 src has 1..143) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f86bc06d060 con 0x7f86c00ab180 2022-07-20T03:17:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.835+0000 7f86c502a700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86a00539b0 0x7f86a0055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:30.862 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.835+0000 7f86c502a700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86a00539b0 0x7f86a0055e70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f86b000b000 tx=0x7f86b0009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:30.863 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:30.839+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f86bc044200 con 0x7f86c00ab180 2022-07-20T03:17:31.008 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.003+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f86c00574e0 con 0x7f86c00ab180 2022-07-20T03:17:31.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.003+0000 7f86aeffd700 1 -- 172.21.15.61:0/3468075663 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 80 v80) v1 ==== 94+0+1212 (secure 0 0 0) 0x7f86bc023020 con 0x7f86c00ab180 2022-07-20T03:17:31.010 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:31.011 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":80,"default_fscid":-1,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7175,"name":"b","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.117:6835/2359381918","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":2359381918},{"type":"v1","addr":"172.21.15.117:6835","nonce":2359381918}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":75},{"gid":8706,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":2,"addr":"172.21.15.61:6833/3042270298","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":3042270298},{"type":"v1","addr":"172.21.15.61:6833","nonce":3042270298}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":80}],"filesystems":[]} 2022-07-20T03:17:31.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86a00539b0 msgr2=0x7f86a0055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:31.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86a00539b0 0x7f86a0055e70 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f86b000b000 tx=0x7f86b0009040).stop 2022-07-20T03:17:31.013 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 msgr2=0x7f86c00aff30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:31.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 0x7f86c00aff30 secure :-1 s=READY pgs=1451 cs=0 l=1 rev1=1 rx=0x7f86c009b0e0 tx=0x7f86bc015000).stop 2022-07-20T03:17:31.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 shutdown_connections 2022-07-20T03:17:31.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f86a00539b0 0x7f86a0055e70 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f86c013f980 0x7f86c0139690 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f86c00b1ef0 0x7f86c0152650 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 --2- 172.21.15.61:0/3468075663 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f86c00ab180 0x7f86c00aff30 unknown :-1 s=CLOSED pgs=1451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 >> 172.21.15.61:0/3468075663 conn(0x7f86c00b2ca0 msgr2=0x7f86c013b870 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:31.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 shutdown_connections 2022-07-20T03:17:31.018 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.007+0000 7f86c682d700 1 -- 172.21.15.61:0/3468075663 wait complete. 2022-07-20T03:17:31.018 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 80 2022-07-20T03:17:31.022 INFO:tasks.ceph.mds.a:Restarting daemon 2022-07-20T03:17:31.023 INFO:tasks.ceph.mds.a:Stopping old one... 2022-07-20T03:17:31.024 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T03:17:31.025 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:17:31.026 INFO:tasks.ceph.mds.b:Restarting daemon 2022-07-20T03:17:31.026 INFO:tasks.ceph.mds.b:Stopping old one... 2022-07-20T03:17:31.027 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T03:17:31.027 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T03:17:31.037 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T03:17:31.038 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i a 2022-07-20T03:17:31.040 INFO:tasks.ceph.mds.a:Started 2022-07-20T03:17:31.041 DEBUG:teuthology.parallel:result is None 2022-07-20T03:17:31.043 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T03:17:31.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-mds -f --cluster ceph -i b 2022-07-20T03:17:31.086 INFO:tasks.ceph.mds.b:Started 2022-07-20T03:17:31.087 DEBUG:teuthology.parallel:result is None 2022-07-20T03:17:31.088 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd blacklist clear 2022-07-20T03:17:31.113 INFO:tasks.ceph.mds.a.smithi061.stdout:starting mds.a at 2022-07-20T03:17:31.164 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:17:31.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 Processor -- start 2022-07-20T03:17:31.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 -- start start 2022-07-20T03:17:31.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f46500b4f80 0x7f46500acae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 0x7f465014c670 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f46500b46e0 con 0x7f46500b5860 2022-07-20T03:17:31.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f46500ad990 con 0x7f46500b4f80 2022-07-20T03:17:31.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4656609700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f46500b4b10 con 0x7f4650146770 2022-07-20T03:17:31.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4655607700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f46500b5860 0x7f4650146060 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48180/0 (socket says 172.21.15.61:48180) 2022-07-20T03:17:31.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4655607700 1 -- 172.21.15.61:0/579962028 learned_addr learned my addr 172.21.15.61:0/579962028 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:31.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4654e06700 1 --2- 172.21.15.61:0/579962028 >> v2:172.21.15.117:3300/0 conn(0x7f46500b4f80 0x7f46500acae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4655e08700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 0x7f465014c670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3542866396 0 0) 0x7f46500b46e0 con 0x7f46500b5860 2022-07-20T03:17:31.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4638003260 con 0x7f46500b5860 2022-07-20T03:17:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 2051887390 0 0) 0x7f4638003260 con 0x7f46500b5860 2022-07-20T03:17:31.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 msgr2=0x7f465014c670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 0x7f465014c670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 >> v2:172.21.15.117:3300/0 conn(0x7f46500b4f80 msgr2=0x7f46500acae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:31.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 --2- 172.21.15.61:0/579962028 >> v2:172.21.15.117:3300/0 conn(0x7f46500b4f80 0x7f46500acae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f46500ad3d0 con 0x7f46500b5860 2022-07-20T03:17:31.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f4648001a00 con 0x7f46500b5860 2022-07-20T03:17:31.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 >> v1:172.21.15.61:6789/0 conn(0x7f46500b5860 legacy=0x7f4650146060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:31.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 0x7f4638003750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4638003c80 0x7f4638004500 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.226 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46380049f0 0x7f4638004e70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500ad990 con 0x7f46380049f0 2022-07-20T03:17:31.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500b4b10 con 0x7f4638003c80 2022-07-20T03:17:31.227 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f463800b750 con 0x7f4650146770 2022-07-20T03:17:31.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4655e08700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46380049f0 0x7f4638004e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4654e06700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4638003c80 0x7f4638004500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.191+0000 7f4655607700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 0x7f4638003750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 -- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 msgr2=0x7f4638003750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:31.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4650146770 0x7f4638003750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 -- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4638003c80 msgr2=0x7f4638004500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:31.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4638003c80 0x7f4638004500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 -- 172.21.15.61:0/579962028 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4638003260 con 0x7f46380049f0 2022-07-20T03:17:31.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4654e06700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4638003c80 0x7f4638004500 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:31.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46380049f0 0x7f4638004e70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f46380054d0 tx=0x7f464c006df0).ready entity=mon.0 client_cookie=6b0771783f38eab7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:31.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464c00e830 con 0x7f46380049f0 2022-07-20T03:17:31.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464c00be90 con 0x7f46380049f0 2022-07-20T03:17:31.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f463ffff700 1 -- 172.21.15.61:0/579962028 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464c00eb80 con 0x7f46380049f0 2022-07-20T03:17:31.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4656609700 1 -- 172.21.15.61:0/579962028 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46380049f0 msgr2=0x7f4638004e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:31.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4656609700 1 --2- 172.21.15.61:0/579962028 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46380049f0 0x7f4638004e70 secure :-1 s=READY pgs=1456 cs=0 l=1 rev1=1 rx=0x7f46380054d0 tx=0x7f464c006df0).stop 2022-07-20T03:17:31.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4655e08700 1 -- 172.21.15.61:0/579962028 reap_dead start 2022-07-20T03:17:31.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4656609700 1 -- 172.21.15.61:0/579962028 shutdown_connections 2022-07-20T03:17:31.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4656609700 1 -- 172.21.15.61:0/579962028 >> 172.21.15.61:0/579962028 conn(0x7f46500b5e10 msgr2=0x7f46500b6220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:31.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4656609700 1 -- 172.21.15.61:0/579962028 shutdown_connections 2022-07-20T03:17:31.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.195+0000 7f4656609700 1 -- 172.21.15.61:0/579962028 wait complete. 2022-07-20T03:17:31.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 Processor -- start 2022-07-20T03:17:31.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 -- start start 2022-07-20T03:17:31.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4650112700 0x7f4650152610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 0x7f46500b0430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46500b0920 0x7f46500af430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46500b0d80 con 0x7f46500b16c0 2022-07-20T03:17:31.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4650151c30 con 0x7f4650112700 2022-07-20T03:17:31.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4650151ed0 con 0x7f46500b0920 2022-07-20T03:17:31.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 0x7f46500b0430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 0x7f46500b0430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51596/0 (socket says 172.21.15.61:51596) 2022-07-20T03:17:31.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 -- 172.21.15.61:0/3297557881 learned_addr learned my addr 172.21.15.61:0/3297557881 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:31.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4655607700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4650112700 0x7f4650152610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4655e08700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46500b0920 0x7f46500af430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 -- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46500b0920 msgr2=0x7f46500af430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:31.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46500b0920 0x7f46500af430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 -- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4650112700 msgr2=0x7f4650152610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:31.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4650112700 0x7f4650152610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:31.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 -- 172.21.15.61:0/3297557881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4650152170 con 0x7f46500b16c0 2022-07-20T03:17:31.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4654e06700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 0x7f46500b0430 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f464000b000 tx=0x7f4640006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:31.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4640011e80 con 0x7f46500b16c0 2022-07-20T03:17:31.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f464000f940 con 0x7f46500b16c0 2022-07-20T03:17:31.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f464000fab0 con 0x7f46500b16c0 2022-07-20T03:17:31.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4650152c50 con 0x7f46500b16c0 2022-07-20T03:17:31.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.199+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f46500b2020 con 0x7f46500b16c0 2022-07-20T03:17:31.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.203+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4650152300 con 0x7f46500b16c0 2022-07-20T03:17:31.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.203+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4640024070 con 0x7f46500b16c0 2022-07-20T03:17:31.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.203+0000 7f463e7fc700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4630053a40 0x7f4630055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:31.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.203+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(144..144 src has 1..144) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f464006c440 con 0x7f46500b16c0 2022-07-20T03:17:31.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.203+0000 7f4655607700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4630053a40 0x7f4630055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:31.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.203+0000 7f4655607700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4630053a40 0x7f4630055f00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f4648005000 tx=0x7f4650066a60).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:31.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.207+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4640043680 con 0x7f46500b16c0 2022-07-20T03:17:31.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:31.347+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd blacklist clear"} v 0) v1 -- 0x7f46500af0f0 con 0x7f46500b16c0 2022-07-20T03:17:32.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.339+0000 7f463e7fc700 1 -- 172.21.15.61:0/3297557881 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd blacklist clear"}]=0 removed all blacklist entries v145) v1 ==== 97+0+0 (secure 0 0 0) 0x7f4640046ca0 con 0x7f46500b16c0 2022-07-20T03:17:32.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4630053a40 msgr2=0x7f4630055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4630053a40 0x7f4630055f00 secure :-1 s=READY pgs=698 cs=0 l=1 rev1=1 rx=0x7f4648005000 tx=0x7f4650066a60).stop 2022-07-20T03:17:32.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 msgr2=0x7f46500b0430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 0x7f46500b0430 secure :-1 s=READY pgs=1457 cs=0 l=1 rev1=1 rx=0x7f464000b000 tx=0x7f4640006df0).stop 2022-07-20T03:17:32.347 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 shutdown_connections 2022-07-20T03:17:32.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4630053a40 0x7f4630055f00 unknown :-1 s=CLOSED pgs=698 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f46500b0920 0x7f46500af430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f46500b16c0 0x7f46500b0430 unknown :-1 s=CLOSED pgs=1457 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 --2- 172.21.15.61:0/3297557881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4650112700 0x7f4650152610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 >> 172.21.15.61:0/3297557881 conn(0x7f46500b5e10 msgr2=0x7f4650141a20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:32.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 shutdown_connections 2022-07-20T03:17:32.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.343+0000 7f4656609700 1 -- 172.21.15.61:0/3297557881 wait complete. 2022-07-20T03:17:32.350 INFO:teuthology.orchestra.run.smithi061.stderr: removed all blacklist entries 2022-07-20T03:17:32.364 INFO:tasks.cephfs.cephfs_test_case:['0', '1', '2'] 2022-07-20T03:17:32.365 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth ls --format=json-pretty 2022-07-20T03:17:32.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 Processor -- start 2022-07-20T03:17:32.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 -- start start 2022-07-20T03:17:32.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18280b5650 0x7f18280b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f18280af8e0 0x7f182805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1828147010 con 0x7f18280adff0 2022-07-20T03:17:32.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18281472b0 con 0x7f18280b5650 2022-07-20T03:17:32.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182ee94700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1828146d40 con 0x7f18280af8e0 2022-07-20T03:17:32.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182de92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18280b5650 0x7f18280b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.511+0000 7f182de92700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18280b5650 0x7f18280b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51608/0 (socket says 172.21.15.61:51608) 2022-07-20T03:17:32.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182e693700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f18280af8e0 0x7f182805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182de92700 1 -- 172.21.15.61:0/4193430289 learned_addr learned my addr 172.21.15.61:0/4193430289 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:32.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182de92700 1 -- 172.21.15.61:0/4193430289 >> v2:172.21.15.117:3300/0 conn(0x7f18280af8e0 msgr2=0x7f182805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:32.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182de92700 1 --2- 172.21.15.61:0/4193430289 >> v2:172.21.15.117:3300/0 conn(0x7f18280af8e0 0x7f182805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182de92700 1 -- 172.21.15.61:0/4193430289 >> v1:172.21.15.117:6789/0 conn(0x7f18280adff0 legacy=0x7f1828147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182de92700 1 -- 172.21.15.61:0/4193430289 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18280acae0 con 0x7f18280b5650 2022-07-20T03:17:32.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182de92700 1 --2- 172.21.15.61:0/4193430289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280b5650 0x7f18280b5a30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7f181c0032f0 tx=0x7f181c009290).ready entity=mon.0 client_cookie=bd59a8979d33535b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:32.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ce90700 1 -- 172.21.15.61:0/4193430289 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c00b870 con 0x7f18280b5650 2022-07-20T03:17:32.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ce90700 1 -- 172.21.15.61:0/4193430289 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c00b9d0 con 0x7f18280b5650 2022-07-20T03:17:32.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ce90700 1 -- 172.21.15.61:0/4193430289 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c00bb40 con 0x7f18280b5650 2022-07-20T03:17:32.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 -- 172.21.15.61:0/4193430289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280b5650 msgr2=0x7f18280b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 --2- 172.21.15.61:0/4193430289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280b5650 0x7f18280b5a30 secure :-1 s=READY pgs=1458 cs=0 l=1 rev1=1 rx=0x7f181c0032f0 tx=0x7f181c009290).stop 2022-07-20T03:17:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 -- 172.21.15.61:0/4193430289 shutdown_connections 2022-07-20T03:17:32.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 --2- 172.21.15.61:0/4193430289 >> v2:172.21.15.117:3300/0 conn(0x7f18280af8e0 0x7f182805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 --2- 172.21.15.61:0/4193430289 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280b5650 0x7f18280b5a30 unknown :-1 s=CLOSED pgs=1458 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 -- 172.21.15.61:0/4193430289 >> 172.21.15.61:0/4193430289 conn(0x7f18280634e0 msgr2=0x7f18280b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 -- 172.21.15.61:0/4193430289 shutdown_connections 2022-07-20T03:17:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 -- 172.21.15.61:0/4193430289 wait complete. 2022-07-20T03:17:32.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 Processor -- start 2022-07-20T03:17:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.515+0000 7f182ee94700 1 -- start start 2022-07-20T03:17:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18280adff0 0x7f1828066800 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 0x7f18280672f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18280b5650 0x7f1828156510 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182815a470 con 0x7f18280af8e0 2022-07-20T03:17:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f182815a6a0 con 0x7f18280adff0 2022-07-20T03:17:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f182815a940 con 0x7f18280b5650 2022-07-20T03:17:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182d691700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 0x7f18280672f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182d691700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 0x7f18280672f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51612/0 (socket says 172.21.15.61:51612) 2022-07-20T03:17:32.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182d691700 1 -- 172.21.15.61:0/2155367592 learned_addr learned my addr 172.21.15.61:0/2155367592 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:32.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18280adff0 0x7f1828066800 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182e693700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18280b5650 0x7f1828156510 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 -- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18280b5650 msgr2=0x7f1828156510 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18280b5650 0x7f1828156510 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 -- 172.21.15.61:0/2155367592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 msgr2=0x7f18280672f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 0x7f18280672f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 -- 172.21.15.61:0/2155367592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f181c009530 con 0x7f18280adff0 2022-07-20T03:17:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182d691700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 0x7f18280672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182de92700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18280adff0 0x7f1828066800 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f181c0032f0 tx=0x7f181c016000).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c00b870 con 0x7f18280adff0 2022-07-20T03:17:32.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f181c021440 con 0x7f18280adff0 2022-07-20T03:17:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1828066e50 con 0x7f18280adff0 2022-07-20T03:17:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f181c0217a0 con 0x7f18280adff0 2022-07-20T03:17:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1828156a00 con 0x7f18280adff0 2022-07-20T03:17:32.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.519+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1828066fe0 con 0x7f18280adff0 2022-07-20T03:17:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.523+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f181c027020 con 0x7f18280adff0 2022-07-20T03:17:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.523+0000 7f1826ffd700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1808053af0 0x7f1808055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.523+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f181c06d020 con 0x7f18280adff0 2022-07-20T03:17:32.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.523+0000 7f182d691700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1808053af0 0x7f1808055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.523+0000 7f182d691700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1808053af0 0x7f1808055fb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f181800d000 tx=0x7f1818009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:32.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.531+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f181c044210 con 0x7f18280adff0 2022-07-20T03:17:32.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.699+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "auth ls", "format": "json-pretty"} v 0) v1 -- 0x7f18280574e0 con 0x7f18280adff0 2022-07-20T03:17:32.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.699+0000 7f1826ffd700 1 -- 172.21.15.61:0/2155367592 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "auth ls", "format": "json-pretty"}]=0 v46) v1 ==== 80+0+4242 (secure 0 0 0) 0x7f181c015070 con 0x7f18280adff0 2022-07-20T03:17:32.704 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:32.705 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:17:32.706 INFO:teuthology.orchestra.run.smithi061.stdout: "auth_dump": [ 2022-07-20T03:17:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.a", 2022-07-20T03:17:32.708 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi/GzkNhAA6PaL30sLG6fIFolxmnKajw==", 2022-07-20T03:17:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.709 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:32.710 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.710 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.711 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.711 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.712 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mds.b", 2022-07-20T03:17:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddi7EL8AhAAzISiGu+fDKAKIhPpPWOjvg==", 2022-07-20T03:17:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.713 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.714 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.715 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.715 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.715 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.0", 2022-07-20T03:17:32.715 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiy7C+DhAAw/6G2OTqtomqrV9bl/G7bQ==", 2022-07-20T03:17:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.716 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.717 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.1", 2022-07-20T03:17:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA8cddiaLgeKBAAedyhWuUfW5B9P9g9FRrnLw==", 2022-07-20T03:17:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.718 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.719 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.2", 2022-07-20T03:17:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiawgoBRAA9zFp2cLPHoEAXIrf2YrRwA==", 2022-07-20T03:17:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.720 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.721 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.722 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.3", 2022-07-20T03:17:32.722 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA9cddiY6HwFhAAJRAaCD5JRMpGg/de9+I9Lw==", 2022-07-20T03:17:32.722 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.722 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.723 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.4", 2022-07-20T03:17:32.724 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBAcddiCmGJMRAAXe6wns3a8Vky9liXFWrTbg==", 2022-07-20T03:17:32.725 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.726 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.727 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.5", 2022-07-20T03:17:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddiA94SEhAApTK2xUvPz9z/iBioEVNM9w==", 2022-07-20T03:17:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.728 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.729 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.6", 2022-07-20T03:17:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddims71IxAAl1y27XhaMrbxsvXDsgwjpQ==", 2022-07-20T03:17:32.730 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.731 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.732 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "osd.7", 2022-07-20T03:17:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQBBcddi3c+ZMxAAJ6cGZFI8oOctS2HZrMzTew==", 2022-07-20T03:17:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.733 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.734 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.0", 2022-07-20T03:17:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiw9u7CRAAtEsatwHVs3oD8FPlpstrRA==", 2022-07-20T03:17:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.735 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:17:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:17:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.736 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.1", 2022-07-20T03:17:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddimvyBDhAAWiS+jm+uDzH4obZ8NZ1aDg==", 2022-07-20T03:17:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.737 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:17:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:17:32.738 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.2", 2022-07-20T03:17:32.739 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA5cddiyrPtERAACajQyHDnVHO1e9kxyFKimg==", 2022-07-20T03:17:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow", 2022-07-20T03:17:32.740 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow r", 2022-07-20T03:17:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow rw pool=cephfs_data" 2022-07-20T03:17:32.741 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.742 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "client.admin", 2022-07-20T03:17:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddiW0PvARAAyUvAWU0Gb2zUc9CmBAPVBw==", 2022-07-20T03:17:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.743 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:17:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr": "allow *", 2022-07-20T03:17:32.744 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow *", 2022-07-20T03:17:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.745 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.746 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.746 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.746 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.x", 2022-07-20T03:17:32.746 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi4WFfMxAAmP+2yQHG4c/NiXuTYW1eFw==", 2022-07-20T03:17:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:17:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:17:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.747 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "entity": "mgr.y", 2022-07-20T03:17:32.748 INFO:teuthology.orchestra.run.smithi061.stdout: "key": "AQA4cddi2hfQLxAApq8Qvo+Wv3NWxegv5N2T+Q==", 2022-07-20T03:17:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: "caps": { 2022-07-20T03:17:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: "mds": "allow *", 2022-07-20T03:17:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: "mon": "allow profile mgr", 2022-07-20T03:17:32.749 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": "allow *" 2022-07-20T03:17:32.750 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.750 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:32.751 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:32.751 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:32.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1808053af0 msgr2=0x7f1808055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1808053af0 0x7f1808055fb0 secure :-1 s=READY pgs=700 cs=0 l=1 rev1=1 rx=0x7f181800d000 tx=0x7f1818009040).stop 2022-07-20T03:17:32.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18280adff0 msgr2=0x7f1828066800 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18280adff0 0x7f1828066800 secure :-1 s=READY pgs=652 cs=0 l=1 rev1=1 rx=0x7f181c0032f0 tx=0x7f181c016000).stop 2022-07-20T03:17:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 shutdown_connections 2022-07-20T03:17:32.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1808053af0 0x7f1808055fb0 unknown :-1 s=CLOSED pgs=700 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18280b5650 0x7f1828156510 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18280af8e0 0x7f18280672f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 --2- 172.21.15.61:0/2155367592 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18280adff0 0x7f1828066800 unknown :-1 s=CLOSED pgs=652 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 >> 172.21.15.61:0/2155367592 conn(0x7f18280634e0 msgr2=0x7f18280afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:32.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 shutdown_connections 2022-07-20T03:17:32.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.703+0000 7f182ee94700 1 -- 172.21.15.61:0/2155367592 wait complete. 2022-07-20T03:17:32.758 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:32.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0bfff700 1 Processor -- start 2022-07-20T03:17:32.864 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0bfff700 1 -- start start 2022-07-20T03:17:32.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0bfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9e0c0af8e0 0x7f9e0c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c147010 con 0x7f9e0c0adff0 2022-07-20T03:17:32.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9e0c1472b0 con 0x7f9e0c0b5650 2022-07-20T03:17:32.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0bfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9e0c146d40 con 0x7f9e0c0af8e0 2022-07-20T03:17:32.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0a7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f9e0c0adff0 0x7f9e0c05bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48218/0 (socket says 172.21.15.61:48218) 2022-07-20T03:17:32.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0a7fc700 1 -- 172.21.15.61:0/1235571497 learned_addr learned my addr 172.21.15.61:0/1235571497 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:32.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0b7fe700 1 --2- 172.21.15.61:0/1235571497 >> v2:172.21.15.117:3300/0 conn(0x7f9e0c0af8e0 0x7f9e0c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.868 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1697370543 0 0) 0x7f9e0c147010 con 0x7f9e0c0adff0 2022-07-20T03:17:32.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9df0003260 con 0x7f9e0c0adff0 2022-07-20T03:17:32.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2616511659 0 0) 0x7f9e0c1472b0 con 0x7f9e0c0b5650 2022-07-20T03:17:32.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9e0c147010 con 0x7f9e0c0b5650 2022-07-20T03:17:32.869 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/1235571497 >> v1:172.21.15.117:6789/0 conn(0x7f9e0c0b5650 legacy=0x7f9e0c147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/1235571497 >> v1:172.21.15.61:6789/0 conn(0x7f9e0c0adff0 legacy=0x7f9e0c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.859+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/1235571497 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e0c05c4a0 con 0x7f9e0c0af8e0 2022-07-20T03:17:32.870 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0b7fe700 1 --2- 172.21.15.61:0/1235571497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0af8e0 0x7f9e0c0b5a30 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f9dfc002f70 tx=0x7f9dfc006df0).ready entity=mon.1 client_cookie=153c763a93fa73d8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc00d540 con 0x7f9e0c0af8e0 2022-07-20T03:17:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc000e90 con 0x7f9e0c0af8e0 2022-07-20T03:17:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e09ffb700 1 -- 172.21.15.61:0/1235571497 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc00d890 con 0x7f9e0c0af8e0 2022-07-20T03:17:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1235571497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0af8e0 msgr2=0x7f9e0c0b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.871 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/1235571497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0af8e0 0x7f9e0c0b5a30 secure :-1 s=READY pgs=654 cs=0 l=1 rev1=1 rx=0x7f9dfc002f70 tx=0x7f9dfc006df0).stop 2022-07-20T03:17:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1235571497 shutdown_connections 2022-07-20T03:17:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/1235571497 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0af8e0 0x7f9e0c0b5a30 unknown :-1 s=CLOSED pgs=654 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1235571497 >> 172.21.15.61:0/1235571497 conn(0x7f9e0c0634e0 msgr2=0x7f9e0c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:32.872 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1235571497 shutdown_connections 2022-07-20T03:17:32.873 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- 172.21.15.61:0/1235571497 wait complete. 2022-07-20T03:17:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 Processor -- start 2022-07-20T03:17:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- start start 2022-07-20T03:17:32.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0adff0 0x7f9e0c156520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e0c0af8e0 0x7f9e0c14ce70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 0x7f9e0c146090 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c156a10 con 0x7f9e0c0b5650 2022-07-20T03:17:32.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c159c00 con 0x7f9e0c0adff0 2022-07-20T03:17:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9e0c1110d0 con 0x7f9e0c0af8e0 2022-07-20T03:17:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 0x7f9e0c146090 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 0x7f9e0c146090 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51628/0 (socket says 172.21.15.61:51628) 2022-07-20T03:17:32.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/3381156937 learned_addr learned my addr 172.21.15.61:0/3381156937 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.863+0000 7f9e0a7fc700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e0c0af8e0 0x7f9e0c14ce70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0affd700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0adff0 0x7f9e0c156520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e0c0af8e0 msgr2=0x7f9e0c14ce70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0b7fe700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e0c0af8e0 0x7f9e0c14ce70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0adff0 msgr2=0x7f9e0c156520 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0b7fe700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0adff0 0x7f9e0c156520 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0b7fe700 1 -- 172.21.15.61:0/3381156937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dfc00b180 con 0x7f9e0c0b5650 2022-07-20T03:17:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0b7fe700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 0x7f9e0c146090 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f9dfc002f70 tx=0x7f9dfc014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:32.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc016070 con 0x7f9e0c0b5650 2022-07-20T03:17:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9dfc0137d0 con 0x7f9e0c0b5650 2022-07-20T03:17:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9e0c110e30 con 0x7f9e0c0b5650 2022-07-20T03:17:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9dfc013b30 con 0x7f9e0c0b5650 2022-07-20T03:17:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9e0c14d360 con 0x7f9e0c0b5650 2022-07-20T03:17:32.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9e0c0574e0 con 0x7f9e0c0b5650 2022-07-20T03:17:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9dfc025030 con 0x7f9e0c0b5650 2022-07-20T03:17:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9dfbfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9de80539b0 0x7f9de8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:32.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f9dfc06c940 con 0x7f9e0c0b5650 2022-07-20T03:17:32.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.867+0000 7f9e0affd700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9de80539b0 0x7f9de8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:32.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.871+0000 7f9e0affd700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9de80539b0 0x7f9de8055e70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f9e0400d000 tx=0x7f9e04007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:32.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:32.875+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9dfc043c10 con 0x7f9e0c0b5650 2022-07-20T03:17:33.009 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.003+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9e0c0adc10 con 0x7f9e0c0b5650 2022-07-20T03:17:33.010 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.003+0000 7f9dfbfff700 1 -- 172.21.15.61:0/3381156937 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v145) v1 ==== 74+0+13239 (secure 0 0 0) 0x7f9dfc023020 con 0x7f9e0c0b5650 2022-07-20T03:17:33.011 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:33.012 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":145,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:32.334039+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":28,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":127,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":137,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":138,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:33.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9de80539b0 msgr2=0x7f9de8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9de80539b0 0x7f9de8055e70 secure :-1 s=READY pgs=701 cs=0 l=1 rev1=1 rx=0x7f9e0400d000 tx=0x7f9e04007da0).stop 2022-07-20T03:17:33.014 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 msgr2=0x7f9e0c146090 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 0x7f9e0c146090 secure :-1 s=READY pgs=1460 cs=0 l=1 rev1=1 rx=0x7f9dfc002f70 tx=0x7f9dfc014000).stop 2022-07-20T03:17:33.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 shutdown_connections 2022-07-20T03:17:33.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9de80539b0 0x7f9de8055e70 unknown :-1 s=CLOSED pgs=701 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.015 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9e0c0b5650 0x7f9e0c146090 unknown :-1 s=CLOSED pgs=1460 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9e0c0af8e0 0x7f9e0c14ce70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 --2- 172.21.15.61:0/3381156937 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9e0c0adff0 0x7f9e0c156520 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 >> 172.21.15.61:0/3381156937 conn(0x7f9e0c0634e0 msgr2=0x7f9e0c0b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.016 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 shutdown_connections 2022-07-20T03:17:33.017 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.007+0000 7f9e0bfff700 1 -- 172.21.15.61:0/3381156937 wait complete. 2022-07-20T03:17:33.029 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}] 2022-07-20T03:17:33.031 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:17:33.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 Processor -- start 2022-07-20T03:17:33.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- start start 2022-07-20T03:17:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f4c8c0b3eb0 0x7f4c8c140830 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4c8c0b1b10 0x7f4c8c144420 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c144910 0x7f4c8c13d630 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c140d20 con 0x7f4c8c0b1b10 2022-07-20T03:17:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4c8c0b4290 con 0x7f4c8c0b3eb0 2022-07-20T03:17:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c0b2350 con 0x7f4c8c144910 2022-07-20T03:17:33.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4c8c0b1b10 0x7f4c8c144420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4c8c0b1b10 0x7f4c8c144420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51634/0 (socket says 172.21.15.61:51634) 2022-07-20T03:17:33.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 -- 172.21.15.61:0/311513692 learned_addr learned my addr 172.21.15.61:0/311513692 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c90b75700 1 --2- 172.21.15.61:0/311513692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c144910 0x7f4c8c13d630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8bfff700 1 --2- 172.21.15.61:0/311513692 >> v2:172.21.15.117:3300/0 conn(0x7f4c8c0b3eb0 0x7f4c8c140830 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 -- 172.21.15.61:0/311513692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c144910 msgr2=0x7f4c8c13d630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 --2- 172.21.15.61:0/311513692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c144910 0x7f4c8c13d630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 -- 172.21.15.61:0/311513692 >> v2:172.21.15.117:3300/0 conn(0x7f4c8c0b3eb0 msgr2=0x7f4c8c140830 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:33.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 --2- 172.21.15.61:0/311513692 >> v2:172.21.15.117:3300/0 conn(0x7f4c8c0b3eb0 0x7f4c8c140830 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 -- 172.21.15.61:0/311513692 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c8c05d270 con 0x7f4c8c0b1b10 2022-07-20T03:17:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8b7fe700 1 --2- 172.21.15.61:0/311513692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c0b1b10 0x7f4c8c144420 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f4c7c002f70 tx=0x7f4c7c006eb0).ready entity=mon.0 client_cookie=30dfa21290950647 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8affd700 1 -- 172.21.15.61:0/311513692 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c7c00d570 con 0x7f4c8c0b1b10 2022-07-20T03:17:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8affd700 1 -- 172.21.15.61:0/311513692 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c7c000e90 con 0x7f4c8c0b1b10 2022-07-20T03:17:33.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c8affd700 1 -- 172.21.15.61:0/311513692 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c7c00d900 con 0x7f4c8c0b1b10 2022-07-20T03:17:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- 172.21.15.61:0/311513692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c0b1b10 msgr2=0x7f4c8c144420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- 172.21.15.61:0/311513692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c0b1b10 0x7f4c8c144420 secure :-1 s=READY pgs=1461 cs=0 l=1 rev1=1 rx=0x7f4c7c002f70 tx=0x7f4c7c006eb0).stop 2022-07-20T03:17:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- 172.21.15.61:0/311513692 shutdown_connections 2022-07-20T03:17:33.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- 172.21.15.61:0/311513692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c144910 0x7f4c8c13d630 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- 172.21.15.61:0/311513692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c0b1b10 0x7f4c8c144420 unknown :-1 s=CLOSED pgs=1461 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 --2- 172.21.15.61:0/311513692 >> v2:172.21.15.117:3300/0 conn(0x7f4c8c0b3eb0 0x7f4c8c140830 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.139+0000 7f4c91376700 1 -- 172.21.15.61:0/311513692 >> 172.21.15.61:0/311513692 conn(0x7f4c8c064c50 msgr2=0x7f4c8c065030 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 -- 172.21.15.61:0/311513692 shutdown_connections 2022-07-20T03:17:33.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 -- 172.21.15.61:0/311513692 wait complete. 2022-07-20T03:17:33.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 Processor -- start 2022-07-20T03:17:33.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 -- start start 2022-07-20T03:17:33.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c0b1b10 0x7f4c8c0b01a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c8c0b3eb0 0x7f4c8c05fbb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 0x7f4c8c13d160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c115910 con 0x7f4c8c144910 2022-07-20T03:17:33.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c115b90 con 0x7f4c8c0b3eb0 2022-07-20T03:17:33.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c91376700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4c8c115e30 con 0x7f4c8c0b1b10 2022-07-20T03:17:33.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c90b75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 0x7f4c8c13d160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c90b75700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 0x7f4c8c13d160 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51642/0 (socket says 172.21.15.61:51642) 2022-07-20T03:17:33.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c90b75700 1 -- 172.21.15.61:0/71618229 learned_addr learned my addr 172.21.15.61:0/71618229 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c8b7fe700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c8c0b3eb0 0x7f4c8c05fbb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.143+0000 7f4c8bfff700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c0b1b10 0x7f4c8c0b01a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c90b75700 1 -- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c0b1b10 msgr2=0x7f4c8c0b01a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c90b75700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c0b1b10 0x7f4c8c0b01a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c90b75700 1 -- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c8c0b3eb0 msgr2=0x7f4c8c05fbb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c90b75700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c8c0b3eb0 0x7f4c8c05fbb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c90b75700 1 -- 172.21.15.61:0/71618229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c7c00b1e0 con 0x7f4c8c144910 2022-07-20T03:17:33.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c8bfff700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c0b1b10 0x7f4c8c0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:33.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c90b75700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 0x7f4c8c13d160 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f4c84006e00 tx=0x7f4c840091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8400c670 con 0x7f4c8c144910 2022-07-20T03:17:33.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4c8400c7d0 con 0x7f4c8c144910 2022-07-20T03:17:33.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c8c1160d0 con 0x7f4c8c144910 2022-07-20T03:17:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4c8400cb30 con 0x7f4c8c144910 2022-07-20T03:17:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c8c116790 con 0x7f4c8c144910 2022-07-20T03:17:33.172 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c8c116260 con 0x7f4c8c144910 2022-07-20T03:17:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4c8400cc90 con 0x7f4c8c144910 2022-07-20T03:17:33.173 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.147+0000 7f4c897fa700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c6c0539b0 0x7f4c6c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.151+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f4c8406bf80 con 0x7f4c8c144910 2022-07-20T03:17:33.174 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.151+0000 7f4c8bfff700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c6c0539b0 0x7f4c6c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.175 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.151+0000 7f4c8bfff700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c6c0539b0 0x7f4c6c055e70 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f4c8000b000 tx=0x7f4c80008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.176 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.155+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4c84043100 con 0x7f4c8c144910 2022-07-20T03:17:33.287 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f4c8c143f80 con 0x7f4c8c144910 2022-07-20T03:17:33.288 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c897fa700 1 -- 172.21.15.61:0/71618229 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v145) v1 ==== 110+0+10 (secure 0 0 0) 0x7f4c84046720 con 0x7f4c8c144910 2022-07-20T03:17:33.288 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:17:33.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c6c0539b0 msgr2=0x7f4c6c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c6c0539b0 0x7f4c6c055e70 secure :-1 s=READY pgs=702 cs=0 l=1 rev1=1 rx=0x7f4c8000b000 tx=0x7f4c80008040).stop 2022-07-20T03:17:33.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 msgr2=0x7f4c8c13d160 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 0x7f4c8c13d160 secure :-1 s=READY pgs=1462 cs=0 l=1 rev1=1 rx=0x7f4c84006e00 tx=0x7f4c840091d0).stop 2022-07-20T03:17:33.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 shutdown_connections 2022-07-20T03:17:33.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4c6c0539b0 0x7f4c6c055e70 unknown :-1 s=CLOSED pgs=702 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4c8c0b3eb0 0x7f4c8c05fbb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4c8c144910 0x7f4c8c13d160 unknown :-1 s=CLOSED pgs=1462 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 --2- 172.21.15.61:0/71618229 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4c8c0b1b10 0x7f4c8c0b01a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 >> 172.21.15.61:0/71618229 conn(0x7f4c8c064c50 msgr2=0x7f4c8c0b78d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 shutdown_connections 2022-07-20T03:17:33.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.283+0000 7f4c91376700 1 -- 172.21.15.61:0/71618229 wait complete. 2022-07-20T03:17:33.301 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:17:33.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.411+0000 7fa4ddeb5700 1 Processor -- start 2022-07-20T03:17:33.416 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.411+0000 7fa4ddeb5700 1 -- start start 2022-07-20T03:17:33.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 0x7fa4d80af8b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa4d80b0890 0x7fa4d80ac1e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa4d80aabe0 con 0x7fa4d80b0350 2022-07-20T03:17:33.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa4d80b4ef0 con 0x7fa4d80b0890 2022-07-20T03:17:33.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d80aae10 con 0x7fa4d80b4b10 2022-07-20T03:17:33.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4dceb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 0x7fa4d80af8b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa4d80b0890 0x7fa4d80ac1e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4dceb3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 0x7fa4d80af8b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38318/0 (socket says 172.21.15.61:38318) 2022-07-20T03:17:33.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4dceb3700 1 -- 172.21.15.61:0/3704534643 learned_addr learned my addr 172.21.15.61:0/3704534643 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d7fff700 1 -- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 msgr2=0x7fa4d80af8b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:33.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d7fff700 1 --2- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 0x7fa4d80af8b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d7fff700 1 -- 172.21.15.61:0/3704534643 >> v1:172.21.15.117:6789/0 conn(0x7fa4d80b0350 legacy=0x7fa4d80b56e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d7fff700 1 -- 172.21.15.61:0/3704534643 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4d80ab0b0 con 0x7fa4d80b0890 2022-07-20T03:17:33.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4dceb3700 1 --2- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 0x7fa4d80af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:33.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d7fff700 1 --2- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b0890 0x7fa4d80ac1e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7fa4c4002f70 tx=0x7fa4c4006df0).ready entity=mon.1 client_cookie=b7fe001e297e6e3d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d77fe700 1 -- 172.21.15.61:0/3704534643 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c400d570 con 0x7fa4d80b0890 2022-07-20T03:17:33.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d77fe700 1 -- 172.21.15.61:0/3704534643 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4c4000e90 con 0x7fa4d80b0890 2022-07-20T03:17:33.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4d77fe700 1 -- 172.21.15.61:0/3704534643 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4c400d8c0 con 0x7fa4d80b0890 2022-07-20T03:17:33.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b0890 msgr2=0x7fa4d80ac1e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b0890 0x7fa4d80ac1e0 secure :-1 s=READY pgs=656 cs=0 l=1 rev1=1 rx=0x7fa4c4002f70 tx=0x7fa4c4006df0).stop 2022-07-20T03:17:33.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/3704534643 shutdown_connections 2022-07-20T03:17:33.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b0890 0x7fa4d80ac1e0 unknown :-1 s=CLOSED pgs=656 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/3704534643 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b4b10 0x7fa4d80af8b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/3704534643 >> 172.21.15.61:0/3704534643 conn(0x7fa4d80632b0 msgr2=0x7fa4d8063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/3704534643 shutdown_connections 2022-07-20T03:17:33.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.415+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/3704534643 wait complete. 2022-07-20T03:17:33.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.419+0000 7fa4ddeb5700 1 Processor -- start 2022-07-20T03:17:33.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.419+0000 7fa4ddeb5700 1 -- start start 2022-07-20T03:17:33.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.419+0000 7fa4ddeb5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 0x7fa4d814fae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b0890 0x7fa4d8145050 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b4b10 0x7fa4d8066420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8060250 con 0x7fa4d80b0350 2022-07-20T03:17:33.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d8139b60 con 0x7fa4d80b4b10 2022-07-20T03:17:33.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa4d81530e0 con 0x7fa4d80b0890 2022-07-20T03:17:33.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 0x7fa4d814fae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 0x7fa4d814fae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51652/0 (socket says 172.21.15.61:51652) 2022-07-20T03:17:33.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 -- 172.21.15.61:0/2614432788 learned_addr learned my addr 172.21.15.61:0/2614432788 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4d7fff700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b0890 0x7fa4d8145050 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dd6b4700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b4b10 0x7fa4d8066420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 -- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b0890 msgr2=0x7fa4d8145050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b0890 0x7fa4d8145050 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 -- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b4b10 msgr2=0x7fa4d8066420 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b4b10 0x7fa4d8066420 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 -- 172.21.15.61:0/2614432788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4cc00b1e0 con 0x7fa4d80b0350 2022-07-20T03:17:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4dceb3700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 0x7fa4d814fae0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7fa4d8145c10 tx=0x7fa4cc006eb0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4cc00b420 con 0x7fa4d80b0350 2022-07-20T03:17:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa4cc012600 con 0x7fa4d80b0350 2022-07-20T03:17:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa4c400b180 con 0x7fa4d80b0350 2022-07-20T03:17:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa4cc012960 con 0x7fa4d80b0350 2022-07-20T03:17:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa4d814ffd0 con 0x7fa4d80b0350 2022-07-20T03:17:33.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa4cc012ac0 con 0x7fa4d80b0350 2022-07-20T03:17:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.423+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa4d80b1800 con 0x7fa4d80b0350 2022-07-20T03:17:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.427+0000 7fa4d5ffb700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4b8053af0 0x7fa4b8055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.427+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fa4cc06bbd0 con 0x7fa4d80b0350 2022-07-20T03:17:33.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.427+0000 7fa4d7fff700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4b8053af0 0x7fa4b8055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.427+0000 7fa4d7fff700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4b8053af0 0x7fa4b8055fb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7fa4c401a000 tx=0x7fa4c4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.427+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa4d80b1800 con 0x7fa4d80b0350 2022-07-20T03:17:33.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.555+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fa4d80b1800 con 0x7fa4d80b0350 2022-07-20T03:17:33.561 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.555+0000 7fa4d5ffb700 1 -- 172.21.15.61:0/2614432788 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v145) v1 ==== 92+0+10 (secure 0 0 0) 0x7fa4d80b1800 con 0x7fa4d80b0350 2022-07-20T03:17:33.562 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:33.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4b8053af0 msgr2=0x7fa4b8055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4b8053af0 0x7fa4b8055fb0 secure :-1 s=READY pgs=703 cs=0 l=1 rev1=1 rx=0x7fa4c401a000 tx=0x7fa4c4014000).stop 2022-07-20T03:17:33.564 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 msgr2=0x7fa4d814fae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 0x7fa4d814fae0 secure :-1 s=READY pgs=1463 cs=0 l=1 rev1=1 rx=0x7fa4d8145c10 tx=0x7fa4cc006eb0).stop 2022-07-20T03:17:33.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 shutdown_connections 2022-07-20T03:17:33.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa4b8053af0 0x7fa4b8055fb0 unknown :-1 s=CLOSED pgs=703 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.565 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa4d80b4b10 0x7fa4d8066420 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa4d80b0890 0x7fa4d8145050 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 --2- 172.21.15.61:0/2614432788 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa4d80b0350 0x7fa4d814fae0 unknown :-1 s=CLOSED pgs=1463 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 >> 172.21.15.61:0/2614432788 conn(0x7fa4d80632b0 msgr2=0x7fa4d80b50c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 shutdown_connections 2022-07-20T03:17:33.566 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.559+0000 7fa4ddeb5700 1 -- 172.21.15.61:0/2614432788 wait complete. 2022-07-20T03:17:33.579 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph --format=json-pretty osd lspools 2022-07-20T03:17:33.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.675+0000 7f185ffff700 1 Processor -- start 2022-07-20T03:17:33.681 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.675+0000 7f185ffff700 1 -- start start 2022-07-20T03:17:33.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600adff0 0x7f18600b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18600af8e0 0x7f186005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1860147010 con 0x7f18600b5650 2022-07-20T03:17:33.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f18601472b0 con 0x7f18600af8e0 2022-07-20T03:17:33.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1860146d40 con 0x7f18600adff0 2022-07-20T03:17:33.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185f7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f18600af8e0 0x7f186005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185effd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f18600b5650 0x7f1860147750 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48258/0 (socket says 172.21.15.61:48258) 2022-07-20T03:17:33.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185effd700 1 -- 172.21.15.61:0/2347012137 learned_addr learned my addr 172.21.15.61:0/2347012137 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:33.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185e7fc700 1 --2- 172.21.15.61:0/2347012137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600adff0 0x7f18600b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185f7fe700 1 -- 172.21.15.61:0/2347012137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600adff0 msgr2=0x7f18600b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:33.687 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185f7fe700 1 --2- 172.21.15.61:0/2347012137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600adff0 0x7f18600b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185f7fe700 1 -- 172.21.15.61:0/2347012137 >> v1:172.21.15.61:6789/0 conn(0x7f18600b5650 legacy=0x7f1860147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185dffb700 1 -- 172.21.15.61:0/2347012137 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3271454925 0 0) 0x7f1860147010 con 0x7f18600b5650 2022-07-20T03:17:33.688 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185f7fe700 1 -- 172.21.15.61:0/2347012137 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f18600acae0 con 0x7f18600af8e0 2022-07-20T03:17:33.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185f7fe700 1 --2- 172.21.15.61:0/2347012137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600af8e0 0x7f186005bd90 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f1850002f70 tx=0x7f1850006df0).ready entity=mon.0 client_cookie=d40cc02ed20b0496 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.689 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185dffb700 1 -- 172.21.15.61:0/2347012137 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f185000d570 con 0x7f18600af8e0 2022-07-20T03:17:33.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185dffb700 1 -- 172.21.15.61:0/2347012137 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1850000e90 con 0x7f18600af8e0 2022-07-20T03:17:33.691 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185dffb700 1 -- 172.21.15.61:0/2347012137 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f185000d8c0 con 0x7f18600af8e0 2022-07-20T03:17:33.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- 172.21.15.61:0/2347012137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600af8e0 msgr2=0x7f186005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.692 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 --2- 172.21.15.61:0/2347012137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600af8e0 0x7f186005bd90 secure :-1 s=READY pgs=1465 cs=0 l=1 rev1=1 rx=0x7f1850002f70 tx=0x7f1850006df0).stop 2022-07-20T03:17:33.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- 172.21.15.61:0/2347012137 shutdown_connections 2022-07-20T03:17:33.693 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 --2- 172.21.15.61:0/2347012137 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600af8e0 0x7f186005bd90 unknown :-1 s=CLOSED pgs=1465 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 --2- 172.21.15.61:0/2347012137 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600adff0 0x7f18600b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- 172.21.15.61:0/2347012137 >> 172.21.15.61:0/2347012137 conn(0x7f18600634e0 msgr2=0x7f18600b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- 172.21.15.61:0/2347012137 shutdown_connections 2022-07-20T03:17:33.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.679+0000 7f185ffff700 1 -- 172.21.15.61:0/2347012137 wait complete. 2022-07-20T03:17:33.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 Processor -- start 2022-07-20T03:17:33.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 -- start start 2022-07-20T03:17:33.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 0x7f1860063cd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600af8e0 0x7f18600664b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18600b5650 0x7f1860064780 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860156140 con 0x7f18600adff0 2022-07-20T03:17:33.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1860065770 con 0x7f18600b5650 2022-07-20T03:17:33.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1860156d50 con 0x7f18600af8e0 2022-07-20T03:17:33.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 0x7f1860063cd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185e7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600af8e0 0x7f18600664b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 0x7f1860063cd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51666/0 (socket says 172.21.15.61:51666) 2022-07-20T03:17:33.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 -- 172.21.15.61:0/160308075 learned_addr learned my addr 172.21.15.61:0/160308075 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185f7fe700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18600b5650 0x7f1860064780 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 -- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600af8e0 msgr2=0x7f18600664b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600af8e0 0x7f18600664b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 -- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18600b5650 msgr2=0x7f1860064780 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18600b5650 0x7f1860064780 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 -- 172.21.15.61:0/160308075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f185000b180 con 0x7f18600adff0 2022-07-20T03:17:33.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185effd700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 0x7f1860063cd0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f1858001250 tx=0x7f18580013f0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1858005ba0 con 0x7f18600adff0 2022-07-20T03:17:33.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1858005d00 con 0x7f18600adff0 2022-07-20T03:17:33.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1860156ff0 con 0x7f18600adff0 2022-07-20T03:17:33.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.683+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f18580072c0 con 0x7f18600adff0 2022-07-20T03:17:33.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.687+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18600b38c0 con 0x7f18600adff0 2022-07-20T03:17:33.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.687+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f18600574e0 con 0x7f18600adff0 2022-07-20T03:17:33.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.687+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1858010070 con 0x7f18600adff0 2022-07-20T03:17:33.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.687+0000 7f1857fff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18440539b0 0x7f1844055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.687+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5253+0+0 (secure 0 0 0) 0x7f185806b640 con 0x7f18600adff0 2022-07-20T03:17:33.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.687+0000 7f185e7fc700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18440539b0 0x7f1844055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.691+0000 7f185e7fc700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18440539b0 0x7f1844055e70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f184800c000 tx=0x7f1848008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.695+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f18580427c0 con 0x7f18600adff0 2022-07-20T03:17:33.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.823+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd lspools", "format": "json-pretty"} v 0) v1 -- 0x7f1860067ea0 con 0x7f18600adff0 2022-07-20T03:17:33.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.823+0000 7f1857fff700 1 -- 172.21.15.61:0/160308075 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd lspools", "format": "json-pretty"}]=0 v145) v1 ==== 84+0+143 (secure 0 0 0) 0x7f1858045de0 con 0x7f18600adff0 2022-07-20T03:17:33.828 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:33.829 INFO:teuthology.orchestra.run.smithi061.stdout:[ 2022-07-20T03:17:33.829 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:33.829 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 1, 2022-07-20T03:17:33.830 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "device_health_metrics" 2022-07-20T03:17:33.830 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:33.831 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:33.831 INFO:teuthology.orchestra.run.smithi061.stdout: "poolnum": 2, 2022-07-20T03:17:33.832 INFO:teuthology.orchestra.run.smithi061.stdout: "poolname": "rbd" 2022-07-20T03:17:33.832 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:33.833 INFO:teuthology.orchestra.run.smithi061.stdout:] 2022-07-20T03:17:33.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.823+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18440539b0 msgr2=0x7f1844055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.823+0000 7f185ffff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18440539b0 0x7f1844055e70 secure :-1 s=READY pgs=704 cs=0 l=1 rev1=1 rx=0x7f184800c000 tx=0x7f1848008040).stop 2022-07-20T03:17:33.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.823+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 msgr2=0x7f1860063cd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.823+0000 7f185ffff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 0x7f1860063cd0 secure :-1 s=READY pgs=1466 cs=0 l=1 rev1=1 rx=0x7f1858001250 tx=0x7f18580013f0).stop 2022-07-20T03:17:33.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 shutdown_connections 2022-07-20T03:17:33.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f18440539b0 0x7f1844055e70 unknown :-1 s=CLOSED pgs=704 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f18600b5650 0x7f1860064780 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f18600af8e0 0x7f18600664b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 --2- 172.21.15.61:0/160308075 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f18600adff0 0x7f1860063cd0 unknown :-1 s=CLOSED pgs=1466 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 >> 172.21.15.61:0/160308075 conn(0x7f18600634e0 msgr2=0x7f186014a250 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 shutdown_connections 2022-07-20T03:17:33.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.827+0000 7f185ffff700 1 -- 172.21.15.61:0/160308075 wait complete. 2022-07-20T03:17:33.843 DEBUG:tasks.cephfs.filesystem:Creating filesystem 'cephfs' 2022-07-20T03:17:33.843 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_metadata 8 2022-07-20T03:17:33.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 Processor -- start 2022-07-20T03:17:33.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 -- start start 2022-07-20T03:17:33.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5d80b9da0 0x7fb5d81468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb5d80b3cf0 0x7fb5d8142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d8142f40 0x7fb5d805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d80ad5b0 con 0x7fb5d80b9da0 2022-07-20T03:17:33.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb5d80ba180 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d81461b0 con 0x7fb5d8142f40 2022-07-20T03:17:33.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5d80b9da0 0x7fb5d81468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb5d80b9da0 0x7fb5d81468f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51674/0 (socket says 172.21.15.61:51674) 2022-07-20T03:17:33.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/4116444071 learned_addr learned my addr 172.21.15.61:0/4116444071 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d77fe700 1 --2- 172.21.15.61:0/4116444071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d8142f40 0x7fb5d805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d67fc700 1 --2- 172.21.15.61:0/4116444071 >> v2:172.21.15.117:3300/0 conn(0x7fb5d80b3cf0 0x7fb5d8142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/4116444071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d8142f40 msgr2=0x7fb5d805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:33.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/4116444071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d8142f40 0x7fb5d805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/4116444071 >> v2:172.21.15.117:3300/0 conn(0x7fb5d80b3cf0 msgr2=0x7fb5d8142a50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:33.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/4116444071 >> v2:172.21.15.117:3300/0 conn(0x7fb5d80b3cf0 0x7fb5d8142a50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/4116444071 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5d8146450 con 0x7fb5d80b9da0 2022-07-20T03:17:33.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.951+0000 7fb5d67fc700 1 --2- 172.21.15.61:0/4116444071 >> v2:172.21.15.117:3300/0 conn(0x7fb5d80b3cf0 0x7fb5d8142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:33.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/4116444071 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d80b9da0 0x7fb5d81468f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc006eb0).ready entity=mon.0 client_cookie=5e7097a6ea27b2be server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d5ffb700 1 -- 172.21.15.61:0/4116444071 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc00d570 con 0x7fb5d80b9da0 2022-07-20T03:17:33.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d5ffb700 1 -- 172.21.15.61:0/4116444071 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc000e90 con 0x7fb5d80b9da0 2022-07-20T03:17:33.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d5ffb700 1 -- 172.21.15.61:0/4116444071 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc00d900 con 0x7fb5d80b9da0 2022-07-20T03:17:33.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 -- 172.21.15.61:0/4116444071 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d80b9da0 msgr2=0x7fb5d81468f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/4116444071 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d80b9da0 0x7fb5d81468f0 secure :-1 s=READY pgs=1467 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc006eb0).stop 2022-07-20T03:17:33.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 -- 172.21.15.61:0/4116444071 shutdown_connections 2022-07-20T03:17:33.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/4116444071 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d8142f40 0x7fb5d805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/4116444071 >> v2:172.21.15.117:3300/0 conn(0x7fb5d80b3cf0 0x7fb5d8142a50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/4116444071 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d80b9da0 0x7fb5d81468f0 unknown :-1 s=CLOSED pgs=1467 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 -- 172.21.15.61:0/4116444071 >> 172.21.15.61:0/4116444071 conn(0x7fb5d80678c0 msgr2=0x7fb5d80b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:33.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 -- 172.21.15.61:0/4116444071 shutdown_connections 2022-07-20T03:17:33.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 -- 172.21.15.61:0/4116444071 wait complete. 2022-07-20T03:17:33.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 Processor -- start 2022-07-20T03:17:33.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.955+0000 7fb5d7fff700 1 -- start start 2022-07-20T03:17:33.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d80b3cf0 0x7fb5d813d370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5d80b9da0 0x7fb5d8145080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d8142f40 0x7fb5d813b3e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d80ae340 con 0x7fb5d8142f40 2022-07-20T03:17:33.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8145570 con 0x7fb5d80b9da0 2022-07-20T03:17:33.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb5d8145790 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d8142f40 0x7fb5d813b3e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d8142f40 0x7fb5d813b3e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51684/0 (socket says 172.21.15.61:51684) 2022-07-20T03:17:33.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d77fe700 1 -- 172.21.15.61:0/3314357434 learned_addr learned my addr 172.21.15.61:0/3314357434 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:33.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d80b3cf0 0x7fb5d813d370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d67fc700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5d80b9da0 0x7fb5d8145080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5d80b9da0 msgr2=0x7fb5d8145080 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5d80b9da0 0x7fb5d8145080 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/3314357434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d8142f40 msgr2=0x7fb5d813b3e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d8142f40 0x7fb5d813b3e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:33.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 -- 172.21.15.61:0/3314357434 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5cc00b1e0 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d6ffd700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d80b3cf0 0x7fb5d813d370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc006be0 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb5cc0068b0 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5d8145a30 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb5cc011d70 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5d8151bb0 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.959+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb5cc01a040 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.989 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.963+0000 7fb5c7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb5b4053a40 0x7fb5b4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:33.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.963+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(145..145 src has 1..145) v4 ==== 5253+0+0 (secure 0 0 0) 0x7fb5cc06c860 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.990 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.963+0000 7fb5d67fc700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb5b4053a40 0x7fb5b4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:33.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.963+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5d8148b50 con 0x7fb5d80b3cf0 2022-07-20T03:17:33.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.963+0000 7fb5d67fc700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb5b4053a40 0x7fb5b4055f00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fb5c800c000 tx=0x7fb5c8008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:33.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:33.971+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb5d8148b50 con 0x7fb5d80b3cf0 2022-07-20T03:17:34.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.103+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8} v 0) v1 -- 0x7fb5d8148b50 con 0x7fb5d80b3cf0 2022-07-20T03:17:34.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.363+0000 7fb5c7fff700 1 -- 172.21.15.61:0/3314357434 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_metadata", "pg_num": 8}]=0 pool 'cephfs_metadata' created v146) v1 ==== 133+0+0 (secure 0 0 0) 0x7fb5d8148b50 con 0x7fb5d80b3cf0 2022-07-20T03:17:34.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb5b4053a40 msgr2=0x7fb5b4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:34.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb5b4053a40 0x7fb5b4055f00 secure :-1 s=READY pgs=705 cs=0 l=1 rev1=1 rx=0x7fb5c800c000 tx=0x7fb5c8008040).stop 2022-07-20T03:17:34.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d80b3cf0 msgr2=0x7fb5d813d370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:34.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d80b3cf0 0x7fb5d813d370 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7fb5cc002f70 tx=0x7fb5cc015000).stop 2022-07-20T03:17:34.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 shutdown_connections 2022-07-20T03:17:34.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb5b4053a40 0x7fb5b4055f00 unknown :-1 s=CLOSED pgs=705 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb5d80b9da0 0x7fb5d8145080 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5d8142f40 0x7fb5d813b3e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 --2- 172.21.15.61:0/3314357434 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5d80b3cf0 0x7fb5d813d370 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 >> 172.21.15.61:0/3314357434 conn(0x7fb5d80678c0 msgr2=0x7fb5d80b7800 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:34.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 shutdown_connections 2022-07-20T03:17:34.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.367+0000 7fb5d7fff700 1 -- 172.21.15.61:0/3314357434 wait complete. 2022-07-20T03:17:34.378 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' created 2022-07-20T03:17:34.391 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool create cephfs_data 8 2022-07-20T03:17:34.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.507+0000 7f0c98db5700 1 Processor -- start 2022-07-20T03:17:34.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.507+0000 7f0c98db5700 1 -- start start 2022-07-20T03:17:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0c940b4ee0 0x7f0c9413fa70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:34.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0c94142c40 0x7f0c9413e2a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f0c940b5d00 con 0x7f0c940b5920 2022-07-20T03:17:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9413f090 con 0x7f0c940b4ee0 2022-07-20T03:17:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0c9413f330 con 0x7f0c94142c40 2022-07-20T03:17:34.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0c940b4ee0 0x7f0c9413fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0c940b4ee0 0x7f0c9413fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51690/0 (socket says 172.21.15.61:51690) 2022-07-20T03:17:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 -- 172.21.15.61:0/1148852927 learned_addr learned my addr 172.21.15.61:0/1148852927 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c937fe700 1 --1- 172.21.15.61:0/1148852927 >> v1:172.21.15.61:6789/0 conn(0x7f0c940b5920 0x7f0c94142530 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48286/0 (socket says 172.21.15.61:48286) 2022-07-20T03:17:34.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c93fff700 1 --2- 172.21.15.61:0/1148852927 >> v2:172.21.15.117:3300/0 conn(0x7f0c94142c40 0x7f0c9413e2a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 -- 172.21.15.61:0/1148852927 >> v2:172.21.15.117:3300/0 conn(0x7f0c94142c40 msgr2=0x7f0c9413e2a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 --2- 172.21.15.61:0/1148852927 >> v2:172.21.15.117:3300/0 conn(0x7f0c94142c40 0x7f0c9413e2a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 -- 172.21.15.61:0/1148852927 >> v1:172.21.15.61:6789/0 conn(0x7f0c940b5920 legacy=0x7f0c94142530 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:34.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 -- 172.21.15.61:0/1148852927 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c9413f5d0 con 0x7f0c940b4ee0 2022-07-20T03:17:34.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c92ffd700 1 -- 172.21.15.61:0/1148852927 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 578007266 0 0) 0x7f0c940b5d00 con 0x7f0c940b5920 2022-07-20T03:17:34.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c93fff700 1 --2- 172.21.15.61:0/1148852927 >> v2:172.21.15.117:3300/0 conn(0x7f0c94142c40 0x7f0c9413e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:34.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c8bfff700 1 --2- 172.21.15.61:0/1148852927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c940b4ee0 0x7f0c9413fa70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f0c7c002f70 tx=0x7f0c7c006eb0).ready entity=mon.0 client_cookie=aea4988e35a78360 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:34.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c92ffd700 1 -- 172.21.15.61:0/1148852927 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c7c00d610 con 0x7f0c940b4ee0 2022-07-20T03:17:34.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c92ffd700 1 -- 172.21.15.61:0/1148852927 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c7c000e90 con 0x7f0c940b4ee0 2022-07-20T03:17:34.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c92ffd700 1 -- 172.21.15.61:0/1148852927 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c7c00d960 con 0x7f0c940b4ee0 2022-07-20T03:17:34.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- 172.21.15.61:0/1148852927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c940b4ee0 msgr2=0x7f0c9413fa70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:34.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1148852927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c940b4ee0 0x7f0c9413fa70 secure :-1 s=READY pgs=1469 cs=0 l=1 rev1=1 rx=0x7f0c7c002f70 tx=0x7f0c7c006eb0).stop 2022-07-20T03:17:34.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- 172.21.15.61:0/1148852927 shutdown_connections 2022-07-20T03:17:34.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1148852927 >> v2:172.21.15.117:3300/0 conn(0x7f0c94142c40 0x7f0c9413e2a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1148852927 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c940b4ee0 0x7f0c9413fa70 unknown :-1 s=CLOSED pgs=1469 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- 172.21.15.61:0/1148852927 >> 172.21.15.61:0/1148852927 conn(0x7f0c940632b0 msgr2=0x7f0c94063690 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:34.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- 172.21.15.61:0/1148852927 shutdown_connections 2022-07-20T03:17:34.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.511+0000 7f0c98db5700 1 -- 172.21.15.61:0/1148852927 wait complete. 2022-07-20T03:17:34.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 Processor -- start 2022-07-20T03:17:34.528 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 -- start start 2022-07-20T03:17:34.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940b4ee0 0x7f0c940b2bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:34.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c940b5920 0x7f0c94141880 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:34.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 0x7f0c94134ab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:34.529 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940b0870 con 0x7f0c94142c40 2022-07-20T03:17:34.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940aee30 con 0x7f0c940b4ee0 2022-07-20T03:17:34.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0c940b0120 con 0x7f0c940b5920 2022-07-20T03:17:34.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 0x7f0c94134ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:34.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 0x7f0c94134ab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51698/0 (socket says 172.21.15.61:51698) 2022-07-20T03:17:34.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 -- 172.21.15.61:0/1681484298 learned_addr learned my addr 172.21.15.61:0/1681484298 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:34.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c8bfff700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c940b5920 0x7f0c94141880 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:34.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c937fe700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940b4ee0 0x7f0c940b2bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:34.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 -- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c940b5920 msgr2=0x7f0c94141880 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:34.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c940b5920 0x7f0c94141880 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 -- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940b4ee0 msgr2=0x7f0c940b2bd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:34.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940b4ee0 0x7f0c940b2bd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:34.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 -- 172.21.15.61:0/1681484298 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0c7c00b1e0 con 0x7f0c94142c40 2022-07-20T03:17:34.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c937fe700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940b4ee0 0x7f0c940b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:34.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c93fff700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 0x7f0c94134ab0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f0c8c006cc0 tx=0x7f0c8c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:34.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c8c011780 con 0x7f0c94142c40 2022-07-20T03:17:34.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0c8c0118e0 con 0x7f0c94142c40 2022-07-20T03:17:34.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0c940b0440 con 0x7f0c94142c40 2022-07-20T03:17:34.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0c8c011c40 con 0x7f0c94142c40 2022-07-20T03:17:34.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.515+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0c940af4f0 con 0x7f0c94142c40 2022-07-20T03:17:34.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.519+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0c9409ba80 con 0x7f0c94142c40 2022-07-20T03:17:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.519+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0c8c012420 con 0x7f0c94142c40 2022-07-20T03:17:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.519+0000 7f0c917fa700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c78053920 0x7f0c78055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.519+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(146..146 src has 1..146) v4 ==== 5607+0+0 (secure 0 0 0) 0x7f0c8c06be30 con 0x7f0c94142c40 2022-07-20T03:17:34.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.519+0000 7f0c937fe700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c78053920 0x7f0c78055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:34.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.519+0000 7f0c937fe700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c78053920 0x7f0c78055de0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f0c8400d000 tx=0x7f0c84009b50).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:34.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.527+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0c8c042f00 con 0x7f0c94142c40 2022-07-20T03:17:34.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:34.655+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8} v 0) v1 -- 0x7f0c940574e0 con 0x7f0c94142c40 2022-07-20T03:17:35.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.179+0000 7f0c917fa700 1 -- 172.21.15.61:0/1681484298 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool create", "pool": "cephfs_data", "pg_num": 8}]=0 pool 'cephfs_data' created v147) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0c8c046520 con 0x7f0c94142c40 2022-07-20T03:17:35.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c78053920 msgr2=0x7f0c78055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c78053920 0x7f0c78055de0 secure :-1 s=READY pgs=706 cs=0 l=1 rev1=1 rx=0x7f0c8400d000 tx=0x7f0c84009b50).stop 2022-07-20T03:17:35.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 msgr2=0x7f0c94134ab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 0x7f0c94134ab0 secure :-1 s=READY pgs=1470 cs=0 l=1 rev1=1 rx=0x7f0c8c006cc0 tx=0x7f0c8c008040).stop 2022-07-20T03:17:35.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 shutdown_connections 2022-07-20T03:17:35.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0c78053920 0x7f0c78055de0 unknown :-1 s=CLOSED pgs=706 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0c94142c40 0x7f0c94134ab0 unknown :-1 s=CLOSED pgs=1470 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0c940b5920 0x7f0c94141880 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 --2- 172.21.15.61:0/1681484298 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0c940b4ee0 0x7f0c940b2bd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 >> 172.21.15.61:0/1681484298 conn(0x7f0c940632b0 msgr2=0x7f0c94138f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:35.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 shutdown_connections 2022-07-20T03:17:35.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.183+0000 7f0c98db5700 1 -- 172.21.15.61:0/1681484298 wait complete. 2022-07-20T03:17:35.191 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' created 2022-07-20T03:17:35.200 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs new cephfs cephfs_metadata cephfs_data --force 2022-07-20T03:17:35.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.303+0000 7ffa6bfff700 1 Processor -- start 2022-07-20T03:17:35.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.303+0000 7ffa6bfff700 1 -- start start 2022-07-20T03:17:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.303+0000 7ffa6bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b6200 0x7ffa6c0ac560 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffa6c0b4af0 0x7ffa6c13f950 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7ffa6c0677d0 con 0x7ffa6c0afc90 2022-07-20T03:17:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ffa6c0b65e0 con 0x7ffa6c0b4af0 2022-07-20T03:17:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c0aca50 con 0x7ffa6c0b6200 2022-07-20T03:17:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffa6c0b4af0 0x7ffa6c13f950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ffa6c0b4af0 0x7ffa6c13f950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51704/0 (socket says 172.21.15.61:51704) 2022-07-20T03:17:35.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 -- 172.21.15.61:0/216025935 learned_addr learned my addr 172.21.15.61:0/216025935 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:35.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6affd700 1 --2- 172.21.15.61:0/216025935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b6200 0x7ffa6c0ac560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 -- 172.21.15.61:0/216025935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b6200 msgr2=0x7ffa6c0ac560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:35.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 --2- 172.21.15.61:0/216025935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b6200 0x7ffa6c0ac560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 -- 172.21.15.61:0/216025935 >> v1:172.21.15.117:6789/0 conn(0x7ffa6c0afc90 legacy=0x7ffa6c0b11e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 -- 172.21.15.61:0/216025935 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6c067530 con 0x7ffa6c0b4af0 2022-07-20T03:17:35.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa69ffb700 1 -- 172.21.15.61:0/216025935 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2216314320 0 0) 0x7ffa6c0677d0 con 0x7ffa6c0afc90 2022-07-20T03:17:35.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6a7fc700 1 --2- 172.21.15.61:0/216025935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b4af0 0x7ffa6c13f950 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7ffa60002f70 tx=0x7ffa60006eb0).ready entity=mon.0 client_cookie=cc6f9d43b5c06233 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa69ffb700 1 -- 172.21.15.61:0/216025935 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6000d570 con 0x7ffa6c0b4af0 2022-07-20T03:17:35.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa69ffb700 1 -- 172.21.15.61:0/216025935 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa60000e90 con 0x7ffa6c0b4af0 2022-07-20T03:17:35.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa69ffb700 1 -- 172.21.15.61:0/216025935 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6000d8c0 con 0x7ffa6c0b4af0 2022-07-20T03:17:35.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- 172.21.15.61:0/216025935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b4af0 msgr2=0x7ffa6c13f950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/216025935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b4af0 0x7ffa6c13f950 secure :-1 s=READY pgs=1471 cs=0 l=1 rev1=1 rx=0x7ffa60002f70 tx=0x7ffa60006eb0).stop 2022-07-20T03:17:35.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- 172.21.15.61:0/216025935 shutdown_connections 2022-07-20T03:17:35.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/216025935 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b4af0 0x7ffa6c13f950 unknown :-1 s=CLOSED pgs=1471 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/216025935 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b6200 0x7ffa6c0ac560 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- 172.21.15.61:0/216025935 >> 172.21.15.61:0/216025935 conn(0x7ffa6c063d10 msgr2=0x7ffa6c0b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:35.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- 172.21.15.61:0/216025935 shutdown_connections 2022-07-20T03:17:35.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.307+0000 7ffa6bfff700 1 -- 172.21.15.61:0/216025935 wait complete. 2022-07-20T03:17:35.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 Processor -- start 2022-07-20T03:17:35.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 -- start start 2022-07-20T03:17:35.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 0x7ffa6c065560 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b4af0 0x7ffa6c13e9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b6200 0x7ffa6c13eeb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c142050 con 0x7ffa6c0b6200 2022-07-20T03:17:35.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c142d80 con 0x7ffa6c0afc90 2022-07-20T03:17:35.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.311+0000 7ffa6bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ffa6c143cf0 con 0x7ffa6c0b4af0 2022-07-20T03:17:35.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 0x7ffa6c065560 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.323 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6a7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b4af0 0x7ffa6c13e9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b6200 0x7ffa6c13eeb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 0x7ffa6c065560 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:33156/0 (socket says 172.21.15.61:33156) 2022-07-20T03:17:35.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6a7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b4af0 0x7ffa6c13e9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38382/0 (socket says 172.21.15.61:38382) 2022-07-20T03:17:35.324 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6affd700 1 -- 172.21.15.61:0/2996360563 learned_addr learned my addr 172.21.15.61:0/2996360563 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:35.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 -- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b4af0 msgr2=0x7ffa6c13e9c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b4af0 0x7ffa6c13e9c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 -- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 msgr2=0x7ffa6c065560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 0x7ffa6c065560 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 -- 172.21.15.61:0/2996360563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa6000b1e0 con 0x7ffa6c0b6200 2022-07-20T03:17:35.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6affd700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 0x7ffa6c065560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:35.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6b7fe700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b6200 0x7ffa6c13eeb0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7ffa64001250 tx=0x7ffa64007d30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6400f470 con 0x7ffa6c0b6200 2022-07-20T03:17:35.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa6c066200 con 0x7ffa6c0b6200 2022-07-20T03:17:35.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ffa6400f5d0 con 0x7ffa6c0b6200 2022-07-20T03:17:35.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa6c066e30 con 0x7ffa6c0b6200 2022-07-20T03:17:35.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ffa6400f930 con 0x7ffa6c0b6200 2022-07-20T03:17:35.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.315+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa6c0670d0 con 0x7ffa6c0b6200 2022-07-20T03:17:35.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.319+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ffa64019030 con 0x7ffa6c0b6200 2022-07-20T03:17:35.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.319+0000 7ffa5bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffa48053ad0 0x7ffa48055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.319+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(147..147 src has 1..147) v4 ==== 5957+0+0 (secure 0 0 0) 0x7ffa6406b730 con 0x7ffa6c0b6200 2022-07-20T03:17:35.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.319+0000 7ffa6affd700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffa48053ad0 0x7ffa48055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.319+0000 7ffa6affd700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffa48053ad0 0x7ffa48055f90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7ffa5c00b000 tx=0x7ffa5c006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.323+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ffa64042660 con 0x7ffa6c0b6200 2022-07-20T03:17:35.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.491+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true} v 0) v1 -- 0x7ffa6c142f80 con 0x7ffa6c0b6200 2022-07-20T03:17:35.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.503+0000 7ffa5bfff700 1 -- 172.21.15.61:0/2996360563 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs new", "fs_name": "cephfs", "metadata": "cephfs_metadata", "data": "cephfs_data", "force": true}]=0 new fs with metadata pool 29 and data pool 30 v82) v1 ==== 189+0+0 (secure 0 0 0) 0x7ffa64045c80 con 0x7ffa6c0b6200 2022-07-20T03:17:35.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffa48053ad0 msgr2=0x7ffa48055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffa48053ad0 0x7ffa48055f90 secure :-1 s=READY pgs=707 cs=0 l=1 rev1=1 rx=0x7ffa5c00b000 tx=0x7ffa5c006df0).stop 2022-07-20T03:17:35.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b6200 msgr2=0x7ffa6c13eeb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b6200 0x7ffa6c13eeb0 secure :-1 s=READY pgs=1472 cs=0 l=1 rev1=1 rx=0x7ffa64001250 tx=0x7ffa64007d30).stop 2022-07-20T03:17:35.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 shutdown_connections 2022-07-20T03:17:35.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ffa48053ad0 0x7ffa48055f90 secure :-1 s=CLOSED pgs=707 cs=0 l=1 rev1=1 rx=0x7ffa5c00b000 tx=0x7ffa5c006df0).stop 2022-07-20T03:17:35.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ffa6c0b6200 0x7ffa6c13eeb0 unknown :-1 s=CLOSED pgs=1472 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ffa6c0b4af0 0x7ffa6c13e9c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 --2- 172.21.15.61:0/2996360563 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ffa6c0afc90 0x7ffa6c065560 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 >> 172.21.15.61:0/2996360563 conn(0x7ffa6c063d10 msgr2=0x7ffa6c0b3ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:35.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 shutdown_connections 2022-07-20T03:17:35.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.507+0000 7ffa6bfff700 1 -- 172.21.15.61:0/2996360563 wait complete. 2022-07-20T03:17:35.518 INFO:teuthology.orchestra.run.smithi061.stderr:new fs with metadata pool 29 and data pool 30 2022-07-20T03:17:35.527 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:35.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 Processor -- start 2022-07-20T03:17:35.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 -- start start 2022-07-20T03:17:35.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f74bc13a5b0 0x7f74bc13a990 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc140810 0x7f74bc05d0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f74bc0b4410 con 0x7f74bc0b20c0 2022-07-20T03:17:35.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f74bc141ac0 con 0x7f74bc13a5b0 2022-07-20T03:17:35.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.615+0000 7f74bb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0b4170 con 0x7f74bc140810 2022-07-20T03:17:35.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f74bc0b20c0 0x7f74bc0b3650 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48318/0 (socket says 172.21.15.61:48318) 2022-07-20T03:17:35.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bad9d700 1 -- 172.21.15.61:0/1264112267 learned_addr learned my addr 172.21.15.61:0/1264112267 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:35.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 --2- 172.21.15.61:0/1264112267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc140810 0x7f74bc05d0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74ba59c700 1 --2- 172.21.15.61:0/1264112267 >> v2:172.21.15.117:3300/0 conn(0x7f74bc13a5b0 0x7f74bc13a990 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b959a700 1 -- 172.21.15.61:0/1264112267 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 840356413 0 0) 0x7f74bc0b4410 con 0x7f74bc0b20c0 2022-07-20T03:17:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b959a700 1 -- 172.21.15.61:0/1264112267 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f74a0003260 con 0x7f74bc0b20c0 2022-07-20T03:17:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 -- 172.21.15.61:0/1264112267 >> v2:172.21.15.117:3300/0 conn(0x7f74bc13a5b0 msgr2=0x7f74bc13a990 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 --2- 172.21.15.61:0/1264112267 >> v2:172.21.15.117:3300/0 conn(0x7f74bc13a5b0 0x7f74bc13a990 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 -- 172.21.15.61:0/1264112267 >> v1:172.21.15.61:6789/0 conn(0x7f74bc0b20c0 legacy=0x7f74bc0b3650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 -- 172.21.15.61:0/1264112267 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74bc05bfc0 con 0x7f74bc140810 2022-07-20T03:17:35.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 --2- 172.21.15.61:0/1264112267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc140810 0x7f74bc05d0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f74b0002f70 tx=0x7f74b0006df0).ready entity=mon.2 client_cookie=1f381151371217bc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b959a700 1 -- 172.21.15.61:0/1264112267 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b000d4a0 con 0x7f74bc140810 2022-07-20T03:17:35.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b959a700 1 -- 172.21.15.61:0/1264112267 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0000e90 con 0x7f74bc140810 2022-07-20T03:17:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b959a700 1 -- 172.21.15.61:0/1264112267 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b000d7f0 con 0x7f74bc140810 2022-07-20T03:17:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- 172.21.15.61:0/1264112267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc140810 msgr2=0x7f74bc05d0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 --2- 172.21.15.61:0/1264112267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc140810 0x7f74bc05d0c0 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7f74b0002f70 tx=0x7f74b0006df0).stop 2022-07-20T03:17:35.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- 172.21.15.61:0/1264112267 shutdown_connections 2022-07-20T03:17:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 --2- 172.21.15.61:0/1264112267 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc140810 0x7f74bc05d0c0 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 --2- 172.21.15.61:0/1264112267 >> v2:172.21.15.117:3300/0 conn(0x7f74bc13a5b0 0x7f74bc13a990 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- 172.21.15.61:0/1264112267 >> 172.21.15.61:0/1264112267 conn(0x7f74bc064be0 msgr2=0x7f74bc064fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- 172.21.15.61:0/1264112267 shutdown_connections 2022-07-20T03:17:35.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- 172.21.15.61:0/1264112267 wait complete. 2022-07-20T03:17:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 Processor -- start 2022-07-20T03:17:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- start start 2022-07-20T03:17:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc0b20c0 0x7f74bc1532d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 0x7f74bc0b79d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74bc140810 0x7f74bc0668e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0b5490 con 0x7f74bc13a5b0 2022-07-20T03:17:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0b6200 con 0x7f74bc140810 2022-07-20T03:17:35.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74bb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f74bc0b0560 con 0x7f74bc0b20c0 2022-07-20T03:17:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 0x7f74bc0b79d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 0x7f74bc0b79d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51724/0 (socket says 172.21.15.61:51724) 2022-07-20T03:17:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74b9d9b700 1 -- 172.21.15.61:0/2617255058 learned_addr learned my addr 172.21.15.61:0/2617255058 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:35.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.619+0000 7f74ba59c700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc0b20c0 0x7f74bc1532d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74bad9d700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74bc140810 0x7f74bc0668e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74b9d9b700 1 -- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc0b20c0 msgr2=0x7f74bc1532d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74b9d9b700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc0b20c0 0x7f74bc1532d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74b9d9b700 1 -- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74bc140810 msgr2=0x7f74bc0668e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74b9d9b700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74bc140810 0x7f74bc0668e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74b9d9b700 1 -- 172.21.15.61:0/2617255058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74b000b180 con 0x7f74bc13a5b0 2022-07-20T03:17:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74b9d9b700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 0x7f74bc0b79d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f74b0002f70 tx=0x7f74b0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b00139c0 con 0x7f74bc13a5b0 2022-07-20T03:17:35.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f74b0013b20 con 0x7f74bc13a5b0 2022-07-20T03:17:35.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f74bc0b0a00 con 0x7f74bc13a5b0 2022-07-20T03:17:35.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f74b0013e80 con 0x7f74bc13a5b0 2022-07-20T03:17:35.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f74bc066dd0 con 0x7f74bc13a5b0 2022-07-20T03:17:35.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f74bc0b0720 con 0x7f74bc13a5b0 2022-07-20T03:17:35.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f74b0025030 con 0x7f74bc13a5b0 2022-07-20T03:17:35.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ab7fe700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7498053a40 0x7f7498055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.648 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f74b006cbd0 con 0x7f74bc13a5b0 2022-07-20T03:17:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.623+0000 7f74ba59c700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7498053a40 0x7f7498055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.627+0000 7f74ba59c700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7498053a40 0x7f7498055f00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f74ac00b000 tx=0x7f74ac006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.649 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.631+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f74b0043b30 con 0x7f74bc13a5b0 2022-07-20T03:17:35.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.755+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f74bc0b6390 con 0x7f74bc13a5b0 2022-07-20T03:17:35.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74ab7fe700 1 -- 172.21.15.61:0/2617255058 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16113 (secure 0 0 0) 0x7f74b0023070 con 0x7f74bc13a5b0 2022-07-20T03:17:35.764 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:35.765 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":148,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:35.498013+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:34.158104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:34.713017+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:35.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7498053a40 msgr2=0x7f7498055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7498053a40 0x7f7498055f00 secure :-1 s=READY pgs=709 cs=0 l=1 rev1=1 rx=0x7f74ac00b000 tx=0x7f74ac006df0).stop 2022-07-20T03:17:35.767 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 msgr2=0x7f74bc0b79d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 0x7f74bc0b79d0 secure :-1 s=READY pgs=1474 cs=0 l=1 rev1=1 rx=0x7f74b0002f70 tx=0x7f74b0014000).stop 2022-07-20T03:17:35.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 shutdown_connections 2022-07-20T03:17:35.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7498053a40 0x7f7498055f00 unknown :-1 s=CLOSED pgs=709 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.768 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f74bc140810 0x7f74bc0668e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f74bc13a5b0 0x7f74bc0b79d0 unknown :-1 s=CLOSED pgs=1474 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 --2- 172.21.15.61:0/2617255058 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f74bc0b20c0 0x7f74bc1532d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.769 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 >> 172.21.15.61:0/2617255058 conn(0x7f74bc064be0 msgr2=0x7f74bc13fec0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:35.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 shutdown_connections 2022-07-20T03:17:35.770 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.759+0000 7f74bb59e700 1 -- 172.21.15.61:0/2617255058 wait complete. 2022-07-20T03:17:35.778 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:35.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c759e700 1 Processor -- start 2022-07-20T03:17:35.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c759e700 1 -- start start 2022-07-20T03:17:35.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c759e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb4c80b1ef0 0x7fb4c8062040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c759e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80b22d0 con 0x7fb4c8061760 2022-07-20T03:17:35.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c759e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb4c80abe50 con 0x7fb4c8098ae0 2022-07-20T03:17:35.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c759e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb4c80ac0f0 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c6d9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb4c8061760 0x7fb4c8146020 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48330/0 (socket says 172.21.15.61:48330) 2022-07-20T03:17:35.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.879+0000 7fb4c6d9d700 1 -- 172.21.15.61:0/460232151 learned_addr learned my addr 172.21.15.61:0/460232151 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:35.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c659c700 1 --2- 172.21.15.61:0/460232151 >> v2:172.21.15.117:3300/0 conn(0x7fb4c80b1ef0 0x7fb4c8062040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4225260384 0 0) 0x7fb4c80b22d0 con 0x7fb4c8061760 2022-07-20T03:17:35.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4a8003260 con 0x7fb4c8061760 2022-07-20T03:17:35.890 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 985835257 0 0) 0x7fb4c80abe50 con 0x7fb4c8098ae0 2022-07-20T03:17:35.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb4c80b22d0 con 0x7fb4c8098ae0 2022-07-20T03:17:35.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c659c700 1 -- 172.21.15.61:0/460232151 >> v1:172.21.15.117:6789/0 conn(0x7fb4c8098ae0 legacy=0x7fb4c8061050 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c659c700 1 -- 172.21.15.61:0/460232151 >> v1:172.21.15.61:6789/0 conn(0x7fb4c8061760 legacy=0x7fb4c8146020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c659c700 1 -- 172.21.15.61:0/460232151 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4c80ac390 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c659c700 1 --2- 172.21.15.61:0/460232151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8062040 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fb4b8002f70 tx=0x7fb4b8006d90).ready entity=mon.1 client_cookie=6cd682143a938daa server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4b800e630 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4b8000e90 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c559a700 1 -- 172.21.15.61:0/460232151 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4b800e980 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 -- 172.21.15.61:0/460232151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c80b1ef0 msgr2=0x7fb4c8062040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 --2- 172.21.15.61:0/460232151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8062040 secure :-1 s=READY pgs=659 cs=0 l=1 rev1=1 rx=0x7fb4b8002f70 tx=0x7fb4b8006d90).stop 2022-07-20T03:17:35.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 -- 172.21.15.61:0/460232151 shutdown_connections 2022-07-20T03:17:35.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 --2- 172.21.15.61:0/460232151 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8062040 unknown :-1 s=CLOSED pgs=659 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 -- 172.21.15.61:0/460232151 >> 172.21.15.61:0/460232151 conn(0x7fb4c80b2ca0 msgr2=0x7fb4c80b3080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:35.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 -- 172.21.15.61:0/460232151 shutdown_connections 2022-07-20T03:17:35.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.883+0000 7fb4c759e700 1 -- 172.21.15.61:0/460232151 wait complete. 2022-07-20T03:17:35.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 Processor -- start 2022-07-20T03:17:35.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 -- start start 2022-07-20T03:17:35.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c8061760 0x7fb4c81467a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4c8098ae0 0x7fb4c8156b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8157060 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c81429e0 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c814a110 con 0x7fb4c8061760 2022-07-20T03:17:35.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb4c8149bd0 con 0x7fb4c8098ae0 2022-07-20T03:17:35.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8157060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c5d9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4c8098ae0 0x7fb4c8156b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c659c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c8061760 0x7fb4c81467a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8157060 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51740/0 (socket says 172.21.15.61:51740) 2022-07-20T03:17:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 -- 172.21.15.61:0/2592520109 learned_addr learned my addr 172.21.15.61:0/2592520109 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:35.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 -- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4c8098ae0 msgr2=0x7fb4c8156b70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4c8098ae0 0x7fb4c8156b70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 -- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c8061760 msgr2=0x7fb4c81467a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c8061760 0x7fb4c81467a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:35.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 -- 172.21.15.61:0/2592520109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb4b800b120 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c5d9b700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4c8098ae0 0x7fb4c8156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c6d9d700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8157060 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7fb4c80af2e0 tx=0x7fb4c0007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4c0005d40 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb4c00070d0 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb4c8149e70 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb4c0005ea0 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.887+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb4c80b5520 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.891+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb4c80574e0 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.891+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb4c0010070 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.891+0000 7fb4bf7fe700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4ac0539d0 0x7fb4ac055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:35.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.891+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb4c006b9c0 con 0x7fb4c80b1ef0 2022-07-20T03:17:35.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.891+0000 7fb4c659c700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4ac0539d0 0x7fb4ac055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:35.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.891+0000 7fb4c659c700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4ac0539d0 0x7fb4ac055e90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fb4b8018000 tx=0x7fb4b8014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:35.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:35.899+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb4c0042810 con 0x7fb4c80b1ef0 2022-07-20T03:17:36.027 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.023+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fb4c80b2640 con 0x7fb4c80b1ef0 2022-07-20T03:17:36.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.023+0000 7fb4bf7fe700 1 -- 172.21.15.61:0/2592520109 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v148) v1 ==== 74+0+16113 (secure 0 0 0) 0x7fb4c0045e30 con 0x7fb4c80b1ef0 2022-07-20T03:17:36.030 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:36.031 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":148,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:35.498013+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:34.158104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:34.713017+0000","flags":32769,"flags_names":"hashpspool,creating","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:36.034 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4ac0539d0 msgr2=0x7fb4ac055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4ac0539d0 0x7fb4ac055e90 secure :-1 s=READY pgs=710 cs=0 l=1 rev1=1 rx=0x7fb4b8018000 tx=0x7fb4b8014000).stop 2022-07-20T03:17:36.035 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 msgr2=0x7fb4c8157060 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8157060 secure :-1 s=READY pgs=1476 cs=0 l=1 rev1=1 rx=0x7fb4c80af2e0 tx=0x7fb4c0007de0).stop 2022-07-20T03:17:36.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 shutdown_connections 2022-07-20T03:17:36.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb4ac0539d0 0x7fb4ac055e90 unknown :-1 s=CLOSED pgs=710 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb4c80b1ef0 0x7fb4c8157060 unknown :-1 s=CLOSED pgs=1476 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb4c8098ae0 0x7fb4c8156b70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.038 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 --2- 172.21.15.61:0/2592520109 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb4c8061760 0x7fb4c81467a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 >> 172.21.15.61:0/2592520109 conn(0x7fb4c80b2ca0 msgr2=0x7fb4c81433c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:36.039 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 shutdown_connections 2022-07-20T03:17:36.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.027+0000 7fb4c759e700 1 -- 172.21.15.61:0/2592520109 wait complete. 2022-07-20T03:17:36.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs set cephfs standby_count_wanted 0 2022-07-20T03:17:36.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 Processor -- start 2022-07-20T03:17:36.159 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 -- start start 2022-07-20T03:17:36.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040b59f0 0x7fb30405c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3040af940 0x7fb30405c5c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fb3040addd0 0x7fb304141cf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb304146350 con 0x7fb3040af940 2022-07-20T03:17:36.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fb30405d2f0 con 0x7fb3040addd0 2022-07-20T03:17:36.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb30890d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb3040b5dd0 con 0x7fb3040b59f0 2022-07-20T03:17:36.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3040af940 0x7fb30405c5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fb3040af940 0x7fb30405c5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51748/0 (socket says 172.21.15.61:51748) 2022-07-20T03:17:36.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 -- 172.21.15.61:0/3676231662 learned_addr learned my addr 172.21.15.61:0/3676231662 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:36.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb303fff700 1 --2- 172.21.15.61:0/3676231662 >> v2:172.21.15.117:3300/0 conn(0x7fb3040addd0 0x7fb304141cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb3037fe700 1 --2- 172.21.15.61:0/3676231662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040b59f0 0x7fb30405c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 -- 172.21.15.61:0/3676231662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040b59f0 msgr2=0x7fb30405c0d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:36.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 --2- 172.21.15.61:0/3676231662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040b59f0 0x7fb30405c0d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 -- 172.21.15.61:0/3676231662 >> v2:172.21.15.117:3300/0 conn(0x7fb3040addd0 msgr2=0x7fb304141cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:36.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 --2- 172.21.15.61:0/3676231662 >> v2:172.21.15.117:3300/0 conn(0x7fb3040addd0 0x7fb304141cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.170 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 -- 172.21.15.61:0/3676231662 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3041460b0 con 0x7fb3040af940 2022-07-20T03:17:36.171 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb303fff700 1 --2- 172.21.15.61:0/3676231662 >> v2:172.21.15.117:3300/0 conn(0x7fb3040addd0 0x7fb304141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:36.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb2faffd700 1 --2- 172.21.15.61:0/3676231662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb30405c5c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7fb2ec002f70 tx=0x7fb2ec006eb0).ready entity=mon.0 client_cookie=c3b8186247b48b93 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb302ffd700 1 -- 172.21.15.61:0/3676231662 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2ec00d570 con 0x7fb3040af940 2022-07-20T03:17:36.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb302ffd700 1 -- 172.21.15.61:0/3676231662 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec000e90 con 0x7fb3040af940 2022-07-20T03:17:36.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.155+0000 7fb302ffd700 1 -- 172.21.15.61:0/3676231662 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2ec00d900 con 0x7fb3040af940 2022-07-20T03:17:36.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 -- 172.21.15.61:0/3676231662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 msgr2=0x7fb30405c5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 --2- 172.21.15.61:0/3676231662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb30405c5c0 secure :-1 s=READY pgs=1477 cs=0 l=1 rev1=1 rx=0x7fb2ec002f70 tx=0x7fb2ec006eb0).stop 2022-07-20T03:17:36.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 -- 172.21.15.61:0/3676231662 shutdown_connections 2022-07-20T03:17:36.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 --2- 172.21.15.61:0/3676231662 >> v2:172.21.15.117:3300/0 conn(0x7fb3040addd0 0x7fb304141cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 --2- 172.21.15.61:0/3676231662 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb30405c5c0 unknown :-1 s=CLOSED pgs=1477 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 --2- 172.21.15.61:0/3676231662 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040b59f0 0x7fb30405c0d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 -- 172.21.15.61:0/3676231662 >> 172.21.15.61:0/3676231662 conn(0x7fb304063480 msgr2=0x7fb3040b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:36.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 -- 172.21.15.61:0/3676231662 shutdown_connections 2022-07-20T03:17:36.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 -- 172.21.15.61:0/3676231662 wait complete. 2022-07-20T03:17:36.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 Processor -- start 2022-07-20T03:17:36.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.159+0000 7fb30890d700 1 -- start start 2022-07-20T03:17:36.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040addd0 0x7fb3041128e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb304063a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3040b59f0 0x7fb304067380 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb30414fa50 con 0x7fb3040af940 2022-07-20T03:17:36.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb304066f50 con 0x7fb3040b59f0 2022-07-20T03:17:36.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb304152960 con 0x7fb3040addd0 2022-07-20T03:17:36.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb304063a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb304063a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51752/0 (socket says 172.21.15.61:51752) 2022-07-20T03:17:36.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 -- 172.21.15.61:0/1697473773 learned_addr learned my addr 172.21.15.61:0/1697473773 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:36.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb3037fe700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040addd0 0x7fb3041128e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb303fff700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3040b59f0 0x7fb304067380 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 -- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040addd0 msgr2=0x7fb3041128e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.191 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040addd0 0x7fb3041128e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 -- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3040b59f0 msgr2=0x7fb304067380 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3040b59f0 0x7fb304067380 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.192 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 -- 172.21.15.61:0/1697473773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2fc009470 con 0x7fb3040af940 2022-07-20T03:17:36.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb2faffd700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb304063a00 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7fb30414fb90 tx=0x7fb2ec015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.193 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2ec0117e0 con 0x7fb3040af940 2022-07-20T03:17:36.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb2ec011940 con 0x7fb3040af940 2022-07-20T03:17:36.194 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2ec00b1e0 con 0x7fb3040af940 2022-07-20T03:17:36.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb2ec011ca0 con 0x7fb3040af940 2022-07-20T03:17:36.195 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb304152c00 con 0x7fb3040af940 2022-07-20T03:17:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3040662a0 con 0x7fb3040af940 2022-07-20T03:17:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.163+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb2ec025030 con 0x7fb3040af940 2022-07-20T03:17:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.167+0000 7fb3017fa700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb2e80539b0 0x7fb2e8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.196 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.167+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(148..148 src has 1..148) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fb2ec06cf40 con 0x7fb3040af940 2022-07-20T03:17:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.167+0000 7fb3037fe700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb2e80539b0 0x7fb2e8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.167+0000 7fb3037fe700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb2e80539b0 0x7fb2e8055e70 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fb2f400b000 tx=0x7fb2f4006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.197 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.171+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb2ec043eb0 con 0x7fb3040af940 2022-07-20T03:17:36.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.331+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"} v 0) v1 -- 0x7fb3041527a0 con 0x7fb3040af940 2022-07-20T03:17:36.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.515+0000 7fb3017fa700 1 -- 172.21.15.61:0/1697473773 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs set", "fs_name": "cephfs", "var": "standby_count_wanted", "val": "0"}]=0 v83) v1 ==== 118+0+0 (secure 0 0 0) 0x7fb2ec023070 con 0x7fb3040af940 2022-07-20T03:17:36.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb2e80539b0 msgr2=0x7fb2e8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb2e80539b0 0x7fb2e8055e70 secure :-1 s=READY pgs=711 cs=0 l=1 rev1=1 rx=0x7fb2f400b000 tx=0x7fb2f4006df0).stop 2022-07-20T03:17:36.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 msgr2=0x7fb304063a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb304063a00 secure :-1 s=READY pgs=1478 cs=0 l=1 rev1=1 rx=0x7fb30414fb90 tx=0x7fb2ec015000).stop 2022-07-20T03:17:36.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 shutdown_connections 2022-07-20T03:17:36.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb2e80539b0 0x7fb2e8055e70 unknown :-1 s=CLOSED pgs=711 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb3040b59f0 0x7fb304067380 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb3040af940 0x7fb304063a00 unknown :-1 s=CLOSED pgs=1478 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 --2- 172.21.15.61:0/1697473773 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb3040addd0 0x7fb3041128e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 >> 172.21.15.61:0/1697473773 conn(0x7fb304063480 msgr2=0x7fb30413eed0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:36.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 shutdown_connections 2022-07-20T03:17:36.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.519+0000 7fb30890d700 1 -- 172.21.15.61:0/1697473773 wait complete. 2022-07-20T03:17:36.540 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:36.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 Processor -- start 2022-07-20T03:17:36.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 -- start start 2022-07-20T03:17:36.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c14d920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c0b42a0 con 0x7f5b7c0608b0 2022-07-20T03:17:36.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5b7c060c90 con 0x7f5b7c0afa10 2022-07-20T03:17:36.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0af410 con 0x7f5b7c0b3de0 2022-07-20T03:17:36.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5b7c0608b0 0x7f5b7c05bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48356/0 (socket says 172.21.15.61:48356) 2022-07-20T03:17:36.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7bfff700 1 -- 172.21.15.61:0/242420555 learned_addr learned my addr 172.21.15.61:0/242420555 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:36.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b80bf4700 1 --2- 172.21.15.61:0/242420555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c14d920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 130416303 0 0) 0x7f5b7c0b42a0 con 0x7f5b7c0608b0 2022-07-20T03:17:36.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b5c003260 con 0x7f5b7c0608b0 2022-07-20T03:17:36.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1006548106 0 0) 0x7f5b7c060c90 con 0x7f5b7c0afa10 2022-07-20T03:17:36.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5b7c0b42a0 con 0x7f5b7c0afa10 2022-07-20T03:17:36.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b80bf4700 1 -- 172.21.15.61:0/242420555 >> v1:172.21.15.117:6789/0 conn(0x7f5b7c0afa10 legacy=0x7f5b7c0b46d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b80bf4700 1 -- 172.21.15.61:0/242420555 >> v1:172.21.15.61:6789/0 conn(0x7f5b7c0608b0 legacy=0x7f5b7c05bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b80bf4700 1 -- 172.21.15.61:0/242420555 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b7c0af6b0 con 0x7f5b7c0b3de0 2022-07-20T03:17:36.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b80bf4700 1 --2- 172.21.15.61:0/242420555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c14d920 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f5b740032f0 tx=0x7f5b740091d0).ready entity=mon.2 client_cookie=d6a7635e343f817d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b7400b640 con 0x7f5b7c0b3de0 2022-07-20T03:17:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b7400b7a0 con 0x7f5b7c0b3de0 2022-07-20T03:17:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b7b7fe700 1 -- 172.21.15.61:0/242420555 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b7400b910 con 0x7f5b7c0b3de0 2022-07-20T03:17:36.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 -- 172.21.15.61:0/242420555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 msgr2=0x7f5b7c14d920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.647+0000 7f5b82657700 1 --2- 172.21.15.61:0/242420555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c14d920 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f5b740032f0 tx=0x7f5b740091d0).stop 2022-07-20T03:17:36.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- 172.21.15.61:0/242420555 shutdown_connections 2022-07-20T03:17:36.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 --2- 172.21.15.61:0/242420555 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c14d920 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- 172.21.15.61:0/242420555 >> 172.21.15.61:0/242420555 conn(0x7f5b7c0b5ee0 msgr2=0x7f5b7c0b62f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:36.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- 172.21.15.61:0/242420555 shutdown_connections 2022-07-20T03:17:36.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- 172.21.15.61:0/242420555 wait complete. 2022-07-20T03:17:36.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 Processor -- start 2022-07-20T03:17:36.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- start start 2022-07-20T03:17:36.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b7c0608b0 0x7f5b7c0b7ad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b7c0afa10 0x7f5b7c155670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c0b20f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c1479d0 con 0x7f5b7c0afa10 2022-07-20T03:17:36.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c0b1c50 con 0x7f5b7c0608b0 2022-07-20T03:17:36.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b82657700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5b7c155b60 con 0x7f5b7c0b3de0 2022-07-20T03:17:36.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.651+0000 7f5b737fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b7c0afa10 0x7f5b7c155670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b7c0608b0 0x7f5b7c0b7ad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b737fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b7c0afa10 0x7f5b7c155670 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51766/0 (socket says 172.21.15.61:51766) 2022-07-20T03:17:36.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b737fe700 1 -- 172.21.15.61:0/1536392792 learned_addr learned my addr 172.21.15.61:0/1536392792 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:36.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b80bf4700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c0b20f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 -- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 msgr2=0x7f5b7c0b20f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c0b20f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 -- 172.21.15.61:0/1536392792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b7c0afa10 msgr2=0x7f5b7c155670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b7c0afa10 0x7f5b7c155670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 -- 172.21.15.61:0/1536392792 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b74009470 con 0x7f5b7c0608b0 2022-07-20T03:17:36.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b7bfff700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b7c0608b0 0x7f5b7c0b7ad0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f5b6c001110 tx=0x7f5b6c007a60).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b6c00ce80 con 0x7f5b7c0608b0 2022-07-20T03:17:36.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b6c0070d0 con 0x7f5b7c0608b0 2022-07-20T03:17:36.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b6c007430 con 0x7f5b7c0608b0 2022-07-20T03:17:36.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b7c0b36d0 con 0x7f5b7c0608b0 2022-07-20T03:17:36.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b7c147740 con 0x7f5b7c0608b0 2022-07-20T03:17:36.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b7c0b89e0 con 0x7f5b7c0608b0 2022-07-20T03:17:36.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.655+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5b6c010070 con 0x7f5b7c0608b0 2022-07-20T03:17:36.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.659+0000 7f5b79ffb700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b60053a60 0x7f5b60055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.659+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5b6c06b450 con 0x7f5b7c0608b0 2022-07-20T03:17:36.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.659+0000 7f5b737fe700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b60053a60 0x7f5b60055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.659+0000 7f5b737fe700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b60053a60 0x7f5b60055f20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f5b6400d000 tx=0x7f5b64007a30).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.667+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5b7c0b89e0 con 0x7f5b7c0608b0 2022-07-20T03:17:36.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.831+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f5b7c147b40 con 0x7f5b7c0608b0 2022-07-20T03:17:36.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.831+0000 7f5b79ffb700 1 -- 172.21.15.61:0/1536392792 <== mon.1 v2:172.21.15.117:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f5b7c147b40 con 0x7f5b7c0608b0 2022-07-20T03:17:36.837 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:36.837 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7397,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1660470104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1660470104},{"type":"v1","addr":"172.21.15.61:6833","nonce":1660470104}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:35.498516+0000","modified":"2022-07-20T03:17:36.503758+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/1280182602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1280182602},{"type":"v1","addr":"172.21.15.117:6835","nonce":1280182602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T03:17:36.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b60053a60 msgr2=0x7f5b60055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b60053a60 0x7f5b60055f20 secure :-1 s=READY pgs=712 cs=0 l=1 rev1=1 rx=0x7f5b6400d000 tx=0x7f5b64007a30).stop 2022-07-20T03:17:36.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b7c0608b0 msgr2=0x7f5b7c0b7ad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.840 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b7c0608b0 0x7f5b7c0b7ad0 secure :-1 s=READY pgs=661 cs=0 l=1 rev1=1 rx=0x7f5b6c001110 tx=0x7f5b6c007a60).stop 2022-07-20T03:17:36.842 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 shutdown_connections 2022-07-20T03:17:36.843 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b60053a60 0x7f5b60055f20 unknown :-1 s=CLOSED pgs=712 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5b7c0b3de0 0x7f5b7c0b20f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5b7c0afa10 0x7f5b7c155670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 --2- 172.21.15.61:0/1536392792 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5b7c0608b0 0x7f5b7c0b7ad0 unknown :-1 s=CLOSED pgs=661 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.844 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 >> 172.21.15.61:0/1536392792 conn(0x7f5b7c0b5ee0 msgr2=0x7f5b7c05c4d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:36.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 shutdown_connections 2022-07-20T03:17:36.845 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.835+0000 7f5b82657700 1 -- 172.21.15.61:0/1536392792 wait complete. 2022-07-20T03:17:36.845 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-20T03:17:36.855 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:36.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 Processor -- start 2022-07-20T03:17:36.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- start start 2022-07-20T03:17:36.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc43c0b1620 0x7fc43c0acdf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fc43c0616b0 0x7fc43c14a7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c061b40 0x7fc43c14ace0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c05ff30 con 0x7fc43c0b1620 2022-07-20T03:17:36.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc43c0b1a00 con 0x7fc43c0616b0 2022-07-20T03:17:36.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c05c0d0 con 0x7fc43c061b40 2022-07-20T03:17:36.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc43c0b1620 0x7fc43c0acdf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc43c0b1620 0x7fc43c0acdf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51776/0 (socket says 172.21.15.61:51776) 2022-07-20T03:17:36.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 -- 172.21.15.61:0/4070401541 learned_addr learned my addr 172.21.15.61:0/4070401541 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:36.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43bfff700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c061b40 0x7fc43c14ace0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43affd700 1 --2- 172.21.15.61:0/4070401541 >> v2:172.21.15.117:3300/0 conn(0x7fc43c0616b0 0x7fc43c14a7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 -- 172.21.15.61:0/4070401541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c061b40 msgr2=0x7fc43c14ace0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:36.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c061b40 0x7fc43c14ace0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 -- 172.21.15.61:0/4070401541 >> v2:172.21.15.117:3300/0 conn(0x7fc43c0616b0 msgr2=0x7fc43c14a7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:36.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 --2- 172.21.15.61:0/4070401541 >> v2:172.21.15.117:3300/0 conn(0x7fc43c0616b0 0x7fc43c14a7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 -- 172.21.15.61:0/4070401541 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc43c05be30 con 0x7fc43c0b1620 2022-07-20T03:17:36.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43affd700 1 --2- 172.21.15.61:0/4070401541 >> v2:172.21.15.117:3300/0 conn(0x7fc43c0616b0 0x7fc43c14a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:36.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43bfff700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c061b40 0x7fc43c14ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:36.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43b7fe700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c0acdf0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7fc42c002f70 tx=0x7fc42c006eb0).ready entity=mon.0 client_cookie=8d405e9ac506c497 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43a7fc700 1 -- 172.21.15.61:0/4070401541 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42c00d570 con 0x7fc43c0b1620 2022-07-20T03:17:36.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43a7fc700 1 -- 172.21.15.61:0/4070401541 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc42c000e90 con 0x7fc43c0b1620 2022-07-20T03:17:36.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc43a7fc700 1 -- 172.21.15.61:0/4070401541 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc42c00d900 con 0x7fc43c0b1620 2022-07-20T03:17:36.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- 172.21.15.61:0/4070401541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 msgr2=0x7fc43c0acdf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c0acdf0 secure :-1 s=READY pgs=1480 cs=0 l=1 rev1=1 rx=0x7fc42c002f70 tx=0x7fc42c006eb0).stop 2022-07-20T03:17:36.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- 172.21.15.61:0/4070401541 shutdown_connections 2022-07-20T03:17:36.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c061b40 0x7fc43c14ace0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- 172.21.15.61:0/4070401541 >> v2:172.21.15.117:3300/0 conn(0x7fc43c0616b0 0x7fc43c14a7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 --2- 172.21.15.61:0/4070401541 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c0acdf0 unknown :-1 s=CLOSED pgs=1480 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- 172.21.15.61:0/4070401541 >> 172.21.15.61:0/4070401541 conn(0x7fc43c0b23d0 msgr2=0x7fc43c0b27e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:36.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- 172.21.15.61:0/4070401541 shutdown_connections 2022-07-20T03:17:36.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.951+0000 7fc441da2700 1 -- 172.21.15.61:0/4070401541 wait complete. 2022-07-20T03:17:36.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 Processor -- start 2022-07-20T03:17:36.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 -- start start 2022-07-20T03:17:36.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c0616b0 0x7fc43c1561f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc43c061b40 0x7fc43c09a4b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c066080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0659f0 con 0x7fc43c0b1620 2022-07-20T03:17:36.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c0637c0 con 0x7fc43c061b40 2022-07-20T03:17:36.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc441da2700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc43c065be0 con 0x7fc43c0616b0 2022-07-20T03:17:36.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc43bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c066080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc43bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c066080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51782/0 (socket says 172.21.15.61:51782) 2022-07-20T03:17:36.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc43affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc43c061b40 0x7fc43c09a4b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc43bfff700 1 -- 172.21.15.61:0/1778209795 learned_addr learned my addr 172.21.15.61:0/1778209795 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:36.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.955+0000 7fc43b7fe700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c0616b0 0x7fc43c1561f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc43bfff700 1 -- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c0616b0 msgr2=0x7fc43c1561f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc43bfff700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c0616b0 0x7fc43c1561f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc43bfff700 1 -- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc43c061b40 msgr2=0x7fc43c09a4b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:36.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc43bfff700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc43c061b40 0x7fc43c09a4b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:36.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc43bfff700 1 -- 172.21.15.61:0/1778209795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc42c00b1e0 con 0x7fc43c0b1620 2022-07-20T03:17:36.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc43bfff700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c066080 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7fc434006cc0 tx=0x7fc434008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc43400c070 con 0x7fc43c0b1620 2022-07-20T03:17:36.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc4340133f0 con 0x7fc43c0b1620 2022-07-20T03:17:36.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc43c09b330 con 0x7fc43c0b1620 2022-07-20T03:17:36.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc434013750 con 0x7fc43c0b1620 2022-07-20T03:17:36.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc43c1566e0 con 0x7fc43c0b1620 2022-07-20T03:17:36.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc43c065da0 con 0x7fc43c0b1620 2022-07-20T03:17:36.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.959+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc4340138b0 con 0x7fc43c0b1620 2022-07-20T03:17:36.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.963+0000 7fc438ff9700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc41c053ad0 0x7fc41c055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:36.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.963+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc43406ba70 con 0x7fc43c0b1620 2022-07-20T03:17:36.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.963+0000 7fc43b7fe700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc41c053ad0 0x7fc41c055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:36.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.963+0000 7fc43b7fe700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc41c053ad0 0x7fc41c055f90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fc42c01c000 tx=0x7fc42c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:36.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:36.967+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc4340429d0 con 0x7fc43c0b1620 2022-07-20T03:17:37.046 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:37.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:37.065 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:37.071 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:37.074 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:17:37.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.091+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fc43c0aca20 con 0x7fc43c0b1620 2022-07-20T03:17:37.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc438ff9700 1 -- 172.21.15.61:0/1778209795 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16100 (secure 0 0 0) 0x7fc43406be60 con 0x7fc43c0b1620 2022-07-20T03:17:37.100 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:37.100 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":149,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:36.502720+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:34.158104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:34.713017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:37.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc41c053ad0 msgr2=0x7fc41c055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc41c053ad0 0x7fc41c055f90 secure :-1 s=READY pgs=713 cs=0 l=1 rev1=1 rx=0x7fc42c01c000 tx=0x7fc42c015000).stop 2022-07-20T03:17:37.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 msgr2=0x7fc43c066080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c066080 secure :-1 s=READY pgs=1481 cs=0 l=1 rev1=1 rx=0x7fc434006cc0 tx=0x7fc434008040).stop 2022-07-20T03:17:37.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 shutdown_connections 2022-07-20T03:17:37.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc41c053ad0 0x7fc41c055f90 unknown :-1 s=CLOSED pgs=713 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc43c061b40 0x7fc43c09a4b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc43c0b1620 0x7fc43c066080 unknown :-1 s=CLOSED pgs=1481 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 --2- 172.21.15.61:0/1778209795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc43c0616b0 0x7fc43c1561f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 >> 172.21.15.61:0/1778209795 conn(0x7fc43c0b23d0 msgr2=0x7fc43c0ade10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:37.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 shutdown_connections 2022-07-20T03:17:37.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.095+0000 7fc441da2700 1 -- 172.21.15.61:0/1778209795 wait complete. 2022-07-20T03:17:37.123 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.0 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:17:37.289 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 Processor -- start 2022-07-20T03:17:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 -- start start 2022-07-20T03:17:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c0accc0 0x7f002c1432b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.290 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f002c1437a0 0x7f002c143bc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f002c095c70 0x7f002c0960c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c097250 con 0x7f002c1437a0 2022-07-20T03:17:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f002c098a80 con 0x7f002c095c70 2022-07-20T03:17:37.291 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.279+0000 7f00311ee700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c0ad0a0 con 0x7f002c0accc0 2022-07-20T03:17:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f002c1437a0 0x7f002c143bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f002c1437a0 0x7f002c143bc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51790/0 (socket says 172.21.15.61:51790) 2022-07-20T03:17:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 -- 172.21.15.61:0/2999039756 learned_addr learned my addr 172.21.15.61:0/2999039756 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:37.292 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00309ed700 1 --2- 172.21.15.61:0/2999039756 >> v2:172.21.15.117:3300/0 conn(0x7f002c095c70 0x7f002c0960c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f002bfff700 1 --2- 172.21.15.61:0/2999039756 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c0accc0 0x7f002c1432b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 -- 172.21.15.61:0/2999039756 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c0accc0 msgr2=0x7f002c1432b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:37.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 --2- 172.21.15.61:0/2999039756 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c0accc0 0x7f002c1432b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.293 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 -- 172.21.15.61:0/2999039756 >> v2:172.21.15.117:3300/0 conn(0x7f002c095c70 msgr2=0x7f002c0960c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 --2- 172.21.15.61:0/2999039756 >> v2:172.21.15.117:3300/0 conn(0x7f002c095c70 0x7f002c0960c0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 -- 172.21.15.61:0/2999039756 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f002c0abca0 con 0x7f002c1437a0 2022-07-20T03:17:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00309ed700 1 --2- 172.21.15.61:0/2999039756 >> v2:172.21.15.117:3300/0 conn(0x7f002c095c70 0x7f002c0960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00237fe700 1 --2- 172.21.15.61:0/2999039756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c143bc0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f0014002f70 tx=0x7f0014006eb0).ready entity=mon.0 client_cookie=625eb0e7e3e74690 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.294 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f002b7fe700 1 -- 172.21.15.61:0/2999039756 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f001400d570 con 0x7f002c1437a0 2022-07-20T03:17:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f002b7fe700 1 -- 172.21.15.61:0/2999039756 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0014000e90 con 0x7f002c1437a0 2022-07-20T03:17:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f002b7fe700 1 -- 172.21.15.61:0/2999039756 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f001400d900 con 0x7f002c1437a0 2022-07-20T03:17:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 -- 172.21.15.61:0/2999039756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 msgr2=0x7f002c143bc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.295 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 --2- 172.21.15.61:0/2999039756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c143bc0 secure :-1 s=READY pgs=1482 cs=0 l=1 rev1=1 rx=0x7f0014002f70 tx=0x7f0014006eb0).stop 2022-07-20T03:17:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 -- 172.21.15.61:0/2999039756 shutdown_connections 2022-07-20T03:17:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 --2- 172.21.15.61:0/2999039756 >> v2:172.21.15.117:3300/0 conn(0x7f002c095c70 0x7f002c0960c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 --2- 172.21.15.61:0/2999039756 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c143bc0 unknown :-1 s=CLOSED pgs=1482 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 --2- 172.21.15.61:0/2999039756 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c0accc0 0x7f002c1432b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.296 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 -- 172.21.15.61:0/2999039756 >> 172.21.15.61:0/2999039756 conn(0x7f002c0b6210 msgr2=0x7f002c0b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 -- 172.21.15.61:0/2999039756 shutdown_connections 2022-07-20T03:17:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 -- 172.21.15.61:0/2999039756 wait complete. 2022-07-20T03:17:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 Processor -- start 2022-07-20T03:17:37.297 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.283+0000 7f00311ee700 1 -- start start 2022-07-20T03:17:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c095c70 0x7f002c09c030 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f002c0accc0 0x7f002c067760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c09b180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c09bcc0 con 0x7f002c1437a0 2022-07-20T03:17:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f002c064f00 con 0x7f002c0accc0 2022-07-20T03:17:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f002c09ba20 con 0x7f002c095c70 2022-07-20T03:17:37.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c09b180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c09b180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51792/0 (socket says 172.21.15.61:51792) 2022-07-20T03:17:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 -- 172.21.15.61:0/3809666060 learned_addr learned my addr 172.21.15.61:0/3809666060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f002bfff700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c095c70 0x7f002c09c030 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00237fe700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f002c0accc0 0x7f002c067760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 -- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c095c70 msgr2=0x7f002c09c030 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c095c70 0x7f002c09c030 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 -- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f002c0accc0 msgr2=0x7f002c067760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f002c0accc0 0x7f002c067760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 -- 172.21.15.61:0/3809666060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0024009470 con 0x7f002c1437a0 2022-07-20T03:17:37.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00237fe700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f002c0accc0 0x7f002c067760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:37.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f002bfff700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c095c70 0x7f002c09c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:37.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00309ed700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c09b180 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f002c066170 tx=0x7f00240091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002400c920 con 0x7f002c1437a0 2022-07-20T03:17:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f002400ca80 con 0x7f002c1437a0 2022-07-20T03:17:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001400b1e0 con 0x7f002c1437a0 2022-07-20T03:17:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f002400cde0 con 0x7f002c1437a0 2022-07-20T03:17:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f002c064ba0 con 0x7f002c1437a0 2022-07-20T03:17:37.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.287+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f002c098cb0 con 0x7f002c1437a0 2022-07-20T03:17:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.291+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0024022070 con 0x7f002c1437a0 2022-07-20T03:17:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.291+0000 7f0029ffb700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0004053920 0x7f0004055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.291+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f002406cdd0 con 0x7f002c1437a0 2022-07-20T03:17:37.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.291+0000 7f002bfff700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0004053920 0x7f0004055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.291+0000 7f002bfff700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0004053920 0x7f0004055de0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f001c00b000 tx=0x7f001c008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.295+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0024043d30 con 0x7f002c1437a0 2022-07-20T03:17:37.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.459+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f002c0574e0 con 0x7f002c1437a0 2022-07-20T03:17:37.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.467+0000 7f0029ffb700 1 -- 172.21.15.61:0/3809666060 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.0", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.0 v47) v1 ==== 184+0+0 (secure 0 0 0) 0x7f00240160e0 con 0x7f002c1437a0 2022-07-20T03:17:37.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0004053920 msgr2=0x7f0004055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0004053920 0x7f0004055de0 secure :-1 s=READY pgs=714 cs=0 l=1 rev1=1 rx=0x7f001c00b000 tx=0x7f001c008040).stop 2022-07-20T03:17:37.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 msgr2=0x7f002c09b180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c09b180 secure :-1 s=READY pgs=1483 cs=0 l=1 rev1=1 rx=0x7f002c066170 tx=0x7f00240091d0).stop 2022-07-20T03:17:37.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 shutdown_connections 2022-07-20T03:17:37.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0004053920 0x7f0004055de0 unknown :-1 s=CLOSED pgs=714 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f002c0accc0 0x7f002c067760 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f002c1437a0 0x7f002c09b180 unknown :-1 s=CLOSED pgs=1483 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 --2- 172.21.15.61:0/3809666060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f002c095c70 0x7f002c09c030 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.478 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 >> 172.21.15.61:0/3809666060 conn(0x7f002c0b6210 msgr2=0x7f002c13d220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:37.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 shutdown_connections 2022-07-20T03:17:37.479 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.471+0000 7f00311ee700 1 -- 172.21.15.61:0/3809666060 wait complete. 2022-07-20T03:17:37.479 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.0 2022-07-20T03:17:37.489 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.1 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:17:37.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 Processor -- start 2022-07-20T03:17:37.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 -- start start 2022-07-20T03:17:37.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b9ef0 0x7f5ba00b0750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ba00b3650 0x7f5ba00b48b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f5ba00b3e40 con 0x7f5ba00b2290 2022-07-20T03:17:37.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f5ba00ba2d0 con 0x7f5ba00b3650 2022-07-20T03:17:37.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5ba5b22700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba00b4070 con 0x7f5ba00b9ef0 2022-07-20T03:17:37.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5b9effd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f5ba00b3650 0x7f5ba00b48b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5b9ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f5ba00b2290 0x7f5ba005bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48398/0 (socket says 172.21.15.61:48398) 2022-07-20T03:17:37.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5b9ffff700 1 -- 172.21.15.61:0/2568762813 learned_addr learned my addr 172.21.15.61:0/2568762813 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:37.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.599+0000 7f5b9f7fe700 1 --2- 172.21.15.61:0/2568762813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b9ef0 0x7f5ba00b0750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9e7fc700 1 -- 172.21.15.61:0/2568762813 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2654460797 0 0) 0x7f5ba00b3e40 con 0x7f5ba00b2290 2022-07-20T03:17:37.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9effd700 1 -- 172.21.15.61:0/2568762813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b9ef0 msgr2=0x7f5ba00b0750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:37.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9effd700 1 --2- 172.21.15.61:0/2568762813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b9ef0 0x7f5ba00b0750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9effd700 1 -- 172.21.15.61:0/2568762813 >> v1:172.21.15.61:6789/0 conn(0x7f5ba00b2290 legacy=0x7f5ba005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9effd700 1 -- 172.21.15.61:0/2568762813 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ba00b1df0 con 0x7f5ba00b3650 2022-07-20T03:17:37.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9effd700 1 --2- 172.21.15.61:0/2568762813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b3650 0x7f5ba00b48b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f5b90002f70 tx=0x7f5b90006df0).ready entity=mon.0 client_cookie=605204c6d0289fec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9e7fc700 1 -- 172.21.15.61:0/2568762813 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9000d570 con 0x7f5ba00b3650 2022-07-20T03:17:37.611 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9e7fc700 1 -- 172.21.15.61:0/2568762813 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b90000e90 con 0x7f5ba00b3650 2022-07-20T03:17:37.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5b9e7fc700 1 -- 172.21.15.61:0/2568762813 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b9000d8c0 con 0x7f5ba00b3650 2022-07-20T03:17:37.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 -- 172.21.15.61:0/2568762813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b3650 msgr2=0x7f5ba00b48b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/2568762813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b3650 0x7f5ba00b48b0 secure :-1 s=READY pgs=1485 cs=0 l=1 rev1=1 rx=0x7f5b90002f70 tx=0x7f5b90006df0).stop 2022-07-20T03:17:37.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 -- 172.21.15.61:0/2568762813 shutdown_connections 2022-07-20T03:17:37.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/2568762813 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b3650 0x7f5ba00b48b0 unknown :-1 s=CLOSED pgs=1485 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/2568762813 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b9ef0 0x7f5ba00b0750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 -- 172.21.15.61:0/2568762813 >> 172.21.15.61:0/2568762813 conn(0x7f5ba0067980 msgr2=0x7f5ba00b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:37.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 -- 172.21.15.61:0/2568762813 shutdown_connections 2022-07-20T03:17:37.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 -- 172.21.15.61:0/2568762813 wait complete. 2022-07-20T03:17:37.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 Processor -- start 2022-07-20T03:17:37.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.603+0000 7f5ba5b22700 1 -- start start 2022-07-20T03:17:37.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b2290 0x7f5ba0151d20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ba00b3650 0x7f5ba0144220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 0x7f5ba00b77b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba00acb80 con 0x7f5ba00b9ef0 2022-07-20T03:17:37.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba0147160 con 0x7f5ba00b3650 2022-07-20T03:17:37.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f5ba013d060 con 0x7f5ba00b2290 2022-07-20T03:17:37.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 0x7f5ba00b77b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b2290 0x7f5ba0151d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 0x7f5ba00b77b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51810/0 (socket says 172.21.15.61:51810) 2022-07-20T03:17:37.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9ffff700 1 -- 172.21.15.61:0/1439345832 learned_addr learned my addr 172.21.15.61:0/1439345832 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:37.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9effd700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ba00b3650 0x7f5ba0144220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 -- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ba00b3650 msgr2=0x7f5ba0144220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ba00b3650 0x7f5ba0144220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 -- 172.21.15.61:0/1439345832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 msgr2=0x7f5ba00b77b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 0x7f5ba00b77b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 -- 172.21.15.61:0/1439345832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b9000b180 con 0x7f5ba00b2290 2022-07-20T03:17:37.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9ffff700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 0x7f5ba00b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:37.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9f7fe700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b2290 0x7f5ba0151d20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5b8800c000 tx=0x7f5b88006df0).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b88011750 con 0x7f5ba00b2290 2022-07-20T03:17:37.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5b880118b0 con 0x7f5ba00b2290 2022-07-20T03:17:37.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ba00af4e0 con 0x7f5ba00b2290 2022-07-20T03:17:37.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5b88011c10 con 0x7f5ba00b2290 2022-07-20T03:17:37.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ba0152210 con 0x7f5ba00b2290 2022-07-20T03:17:37.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.607+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ba00b4da0 con 0x7f5ba00b2290 2022-07-20T03:17:37.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.611+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f5b88011d70 con 0x7f5ba00b2290 2022-07-20T03:17:37.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.611+0000 7f5b9cff9700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b84053ad0 0x7f5b84055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.611+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f5b8806c190 con 0x7f5ba00b2290 2022-07-20T03:17:37.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.611+0000 7f5b9effd700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b84053ad0 0x7f5b84055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.611+0000 7f5b9effd700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b84053ad0 0x7f5b84055f90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f5b9001a000 tx=0x7f5b90014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.619+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5b88043090 con 0x7f5ba00b2290 2022-07-20T03:17:37.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.791+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7f5ba00574e0 con 0x7f5ba00b2290 2022-07-20T03:17:37.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.803+0000 7f5b9cff9700 1 -- 172.21.15.61:0/1439345832 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.1", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.1 v48) v1 ==== 184+0+0 (secure 0 0 0) 0x7f5b880466b0 con 0x7f5ba00b2290 2022-07-20T03:17:37.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b84053ad0 msgr2=0x7f5b84055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b84053ad0 0x7f5b84055f90 secure :-1 s=READY pgs=715 cs=0 l=1 rev1=1 rx=0x7f5b9001a000 tx=0x7f5b90014000).stop 2022-07-20T03:17:37.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b2290 msgr2=0x7f5ba0151d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b2290 0x7f5ba0151d20 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f5b8800c000 tx=0x7f5b88006df0).stop 2022-07-20T03:17:37.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 shutdown_connections 2022-07-20T03:17:37.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5b84053ad0 0x7f5b84055f90 unknown :-1 s=CLOSED pgs=715 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f5ba00b3650 0x7f5ba0144220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f5ba00b9ef0 0x7f5ba00b77b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 --2- 172.21.15.61:0/1439345832 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f5ba00b2290 0x7f5ba0151d20 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 >> 172.21.15.61:0/1439345832 conn(0x7f5ba0067980 msgr2=0x7f5ba00b4680 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:37.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 shutdown_connections 2022-07-20T03:17:37.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.807+0000 7f5ba5b22700 1 -- 172.21.15.61:0/1439345832 wait complete. 2022-07-20T03:17:37.815 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.1 2022-07-20T03:17:37.825 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph auth caps client.2 mds allow mon 'allow r' osd 'allow rw pool=cephfs_data' 2022-07-20T03:17:37.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff413f5700 1 Processor -- start 2022-07-20T03:17:37.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff413f5700 1 -- start start 2022-07-20T03:17:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff413f5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7eff34043570 0x7eff340d59a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff413f5700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff34049380 con 0x7eff34048fa0 2022-07-20T03:17:37.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff413f5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7eff3403e290 con 0x7eff340d5e90 2022-07-20T03:17:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff413f5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7eff340d5260 con 0x7eff34043570 2022-07-20T03:17:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3bfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7eff34048fa0 0x7eff340d1c90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48412/0 (socket says 172.21.15.61:48412) 2022-07-20T03:17:37.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3bfff700 1 -- 172.21.15.61:0/740302154 learned_addr learned my addr 172.21.15.61:0/740302154 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:37.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3b7fe700 1 --2- 172.21.15.61:0/740302154 >> v2:172.21.15.117:3300/0 conn(0x7eff34043570 0x7eff340d59a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 686523441 0 0) 0x7eff34049380 con 0x7eff34048fa0 2022-07-20T03:17:37.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff28003260 con 0x7eff34048fa0 2022-07-20T03:17:37.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2236620366 0 0) 0x7eff3403e290 con 0x7eff340d5e90 2022-07-20T03:17:37.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7eff34049380 con 0x7eff340d5e90 2022-07-20T03:17:37.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 742072896 0 0) 0x7eff28003260 con 0x7eff34048fa0 2022-07-20T03:17:37.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 >> v2:172.21.15.117:3300/0 conn(0x7eff34043570 msgr2=0x7eff340d59a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:37.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 --2- 172.21.15.61:0/740302154 >> v2:172.21.15.117:3300/0 conn(0x7eff34043570 0x7eff340d59a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 >> v1:172.21.15.117:6789/0 conn(0x7eff340d5e90 legacy=0x7eff340cec40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff340d5500 con 0x7eff34048fa0 2022-07-20T03:17:37.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7eff30001660 con 0x7eff34048fa0 2022-07-20T03:17:37.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 >> v1:172.21.15.61:6789/0 conn(0x7eff34048fa0 legacy=0x7eff340d1c90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff280038e0 0x7eff28003cc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff28004500 0x7eff28004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff28004e70 0x7eff2800b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3bfff700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff280038e0 0x7eff28003cc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340d5260 con 0x7eff280038e0 2022-07-20T03:17:37.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff2800bc40 con 0x7eff28004e70 2022-07-20T03:17:37.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff2800bd80 con 0x7eff28004500 2022-07-20T03:17:37.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.935+0000 7eff3b7fe700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff28004500 0x7eff28004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff40bf4700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff28004e70 0x7eff2800b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3bfff700 1 -- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff28004500 msgr2=0x7eff28004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:37.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3bfff700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff28004500 0x7eff28004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3bfff700 1 -- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff28004e70 msgr2=0x7eff2800b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:37.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3bfff700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff28004e70 0x7eff2800b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.962 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3bfff700 1 -- 172.21.15.61:0/740302154 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff28003260 con 0x7eff280038e0 2022-07-20T03:17:37.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff40bf4700 1 -- 172.21.15.61:0/740302154 reap_dead start 2022-07-20T03:17:37.963 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3bfff700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff280038e0 0x7eff28003cc0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7eff30001800 tx=0x7eff34007140).ready entity=mon.0 client_cookie=eb320237139a8c49 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff3000cc70 con 0x7eff280038e0 2022-07-20T03:17:37.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3000cdd0 con 0x7eff280038e0 2022-07-20T03:17:37.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff3affd700 1 -- 172.21.15.61:0/740302154 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff300072c0 con 0x7eff280038e0 2022-07-20T03:17:37.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 -- 172.21.15.61:0/740302154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff280038e0 msgr2=0x7eff28003cc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff280038e0 0x7eff28003cc0 secure :-1 s=READY pgs=1487 cs=0 l=1 rev1=1 rx=0x7eff30001800 tx=0x7eff34007140).stop 2022-07-20T03:17:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 -- 172.21.15.61:0/740302154 shutdown_connections 2022-07-20T03:17:37.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 --2- 172.21.15.61:0/740302154 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff280038e0 0x7eff28003cc0 unknown :-1 s=CLOSED pgs=1487 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 -- 172.21.15.61:0/740302154 >> 172.21.15.61:0/740302154 conn(0x7eff3400f3a0 msgr2=0x7eff34048550 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 -- 172.21.15.61:0/740302154 shutdown_connections 2022-07-20T03:17:37.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 -- 172.21.15.61:0/740302154 wait complete. 2022-07-20T03:17:37.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 Processor -- start 2022-07-20T03:17:37.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.939+0000 7eff413f5700 1 -- start start 2022-07-20T03:17:37.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff280038e0 0x7eff340d4430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff34040320 0x7eff3403e580 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 0x7eff340a2860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340424a0 con 0x7eff340412c0 2022-07-20T03:17:37.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff340a2390 con 0x7eff280038e0 2022-07-20T03:17:37.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7eff340421c0 con 0x7eff34040320 2022-07-20T03:17:37.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 0x7eff340a2860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 0x7eff340a2860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51830/0 (socket says 172.21.15.61:51830) 2022-07-20T03:17:37.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 -- 172.21.15.61:0/35159894 learned_addr learned my addr 172.21.15.61:0/35159894 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:37.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff3bfff700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff280038e0 0x7eff340d4430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff3b7fe700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff34040320 0x7eff3403e580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 -- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff34040320 msgr2=0x7eff3403e580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff34040320 0x7eff3403e580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 -- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff280038e0 msgr2=0x7eff340d4430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:37.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff280038e0 0x7eff340d4430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:37.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 -- 172.21.15.61:0/35159894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff3403ff90 con 0x7eff340412c0 2022-07-20T03:17:37.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff3bfff700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff280038e0 0x7eff340d4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:37.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff40bf4700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 0x7eff340a2860 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7eff3400f780 tx=0x7eff3c063040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff3c06a930 con 0x7eff340412c0 2022-07-20T03:17:37.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7eff3c05dd00 con 0x7eff340412c0 2022-07-20T03:17:37.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff340a2d50 con 0x7eff340412c0 2022-07-20T03:17:37.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7eff3c026300 con 0x7eff340412c0 2022-07-20T03:17:37.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff340a31a0 con 0x7eff340412c0 2022-07-20T03:17:37.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.943+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff34042c90 con 0x7eff340412c0 2022-07-20T03:17:37.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.947+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7eff3c05d1e0 con 0x7eff340412c0 2022-07-20T03:17:37.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.947+0000 7eff397fa700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff1c053a40 0x7eff1c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:37.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.947+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7eff3c0da4e0 con 0x7eff340412c0 2022-07-20T03:17:37.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.947+0000 7eff3bfff700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff1c053a40 0x7eff1c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:37.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.947+0000 7eff3bfff700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff1c053a40 0x7eff1c055f00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7eff30017000 tx=0x7eff30015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:37.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:37.951+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7eff3c0b1340 con 0x7eff340412c0 2022-07-20T03:17:38.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.115+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]} v 0) v1 -- 0x7eff3403bd80 con 0x7eff340412c0 2022-07-20T03:17:38.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff397fa700 1 -- 172.21.15.61:0/35159894 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "auth caps", "entity": "client.2", "caps": ["mds", "allow", "mon", "allow r", "osd", "allow rw pool=cephfs_data"]}]=0 updated caps for client.2 v49) v1 ==== 184+0+0 (secure 0 0 0) 0x7eff3c0b4960 con 0x7eff340412c0 2022-07-20T03:17:38.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff1c053a40 msgr2=0x7eff1c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff1c053a40 0x7eff1c055f00 secure :-1 s=READY pgs=716 cs=0 l=1 rev1=1 rx=0x7eff30017000 tx=0x7eff30015000).stop 2022-07-20T03:17:38.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 msgr2=0x7eff340a2860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 0x7eff340a2860 secure :-1 s=READY pgs=1488 cs=0 l=1 rev1=1 rx=0x7eff3400f780 tx=0x7eff3c063040).stop 2022-07-20T03:17:38.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 shutdown_connections 2022-07-20T03:17:38.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7eff1c053a40 0x7eff1c055f00 unknown :-1 s=CLOSED pgs=716 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7eff340412c0 0x7eff340a2860 unknown :-1 s=CLOSED pgs=1488 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7eff34040320 0x7eff3403e580 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 --2- 172.21.15.61:0/35159894 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7eff280038e0 0x7eff340d4430 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 >> 172.21.15.61:0/35159894 conn(0x7eff3400f3a0 msgr2=0x7eff340cce90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:38.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 shutdown_connections 2022-07-20T03:17:38.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.123+0000 7eff413f5700 1 -- 172.21.15.61:0/35159894 wait complete. 2022-07-20T03:17:38.136 INFO:teuthology.orchestra.run.smithi061.stderr:updated caps for client.2 2022-07-20T03:17:38.140 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:38.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.243+0000 7f4e1afcb700 1 Processor -- start 2022-07-20T03:17:38.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.243+0000 7f4e1afcb700 1 -- start start 2022-07-20T03:17:38.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4e140b1200 0x7f4e140b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e140aea30 con 0x7f4e1405bd90 2022-07-20T03:17:38.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e140aec60 con 0x7f4e140b5650 2022-07-20T03:17:38.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e14142970 con 0x7f4e140b1200 2022-07-20T03:17:38.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4e140b1200 0x7f4e140b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4e140b1200 0x7f4e140b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51834/0 (socket says 172.21.15.61:51834) 2022-07-20T03:17:38.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 -- 172.21.15.61:0/295467222 learned_addr learned my addr 172.21.15.61:0/295467222 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:38.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1a7ca700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4e1405bd90 0x7f4e14142d60 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48436/0 (socket says 172.21.15.61:48436) 2022-07-20T03:17:38.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 -- 172.21.15.61:0/295467222 >> v1:172.21.15.117:6789/0 conn(0x7f4e140b5650 legacy=0x7f4e1405c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:38.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 -- 172.21.15.61:0/295467222 >> v1:172.21.15.61:6789/0 conn(0x7f4e1405bd90 legacy=0x7f4e14142d60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 -- 172.21.15.61:0/295467222 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e140af180 con 0x7f4e140b1200 2022-07-20T03:17:38.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e197c8700 1 --2- 172.21.15.61:0/295467222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b1200 0x7f4e140b5a30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f4e140608e0 tx=0x7f4e10006e50).ready entity=mon.0 client_cookie=67bd013d34464549 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:38.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e18fc7700 1 -- 172.21.15.61:0/295467222 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e1000d610 con 0x7f4e140b1200 2022-07-20T03:17:38.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e18fc7700 1 -- 172.21.15.61:0/295467222 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e10000e90 con 0x7f4e140b1200 2022-07-20T03:17:38.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e18fc7700 1 -- 172.21.15.61:0/295467222 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e1000d9a0 con 0x7f4e140b1200 2022-07-20T03:17:38.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- 172.21.15.61:0/295467222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b1200 msgr2=0x7f4e140b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/295467222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b1200 0x7f4e140b5a30 secure :-1 s=READY pgs=1490 cs=0 l=1 rev1=1 rx=0x7f4e140608e0 tx=0x7f4e10006e50).stop 2022-07-20T03:17:38.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- 172.21.15.61:0/295467222 shutdown_connections 2022-07-20T03:17:38.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/295467222 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b1200 0x7f4e140b5a30 unknown :-1 s=CLOSED pgs=1490 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- 172.21.15.61:0/295467222 >> 172.21.15.61:0/295467222 conn(0x7f4e140634e0 msgr2=0x7f4e140b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:38.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- 172.21.15.61:0/295467222 shutdown_connections 2022-07-20T03:17:38.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.247+0000 7f4e1afcb700 1 -- 172.21.15.61:0/295467222 wait complete. 2022-07-20T03:17:38.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 Processor -- start 2022-07-20T03:17:38.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 -- start start 2022-07-20T03:17:38.274 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e1405bd90 0x7f4e141461e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e140b1200 0x7f4e14152cf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.275 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 0x7f4e1413dee0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e14063f70 con 0x7f4e140b5650 2022-07-20T03:17:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e14145930 con 0x7f4e1405bd90 2022-07-20T03:17:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1afcb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e141466d0 con 0x7f4e140b1200 2022-07-20T03:17:38.276 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 0x7f4e1413dee0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 0x7f4e1413dee0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51844/0 (socket says 172.21.15.61:51844) 2022-07-20T03:17:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 -- 172.21.15.61:0/1752095342 learned_addr learned my addr 172.21.15.61:0/1752095342 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e197c8700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e140b1200 0x7f4e14152cf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.277 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e19fc9700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e1405bd90 0x7f4e141461e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 -- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e140b1200 msgr2=0x7f4e14152cf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e140b1200 0x7f4e14152cf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 -- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e1405bd90 msgr2=0x7f4e141461e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.278 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e1405bd90 0x7f4e141461e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 -- 172.21.15.61:0/1752095342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e1000b1e0 con 0x7f4e140b5650 2022-07-20T03:17:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e1a7ca700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 0x7f4e1413dee0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f4e14062420 tx=0x7f4e0c007ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e0c005ae0 con 0x7f4e140b5650 2022-07-20T03:17:38.279 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.251+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4e0c005c40 con 0x7f4e140b5650 2022-07-20T03:17:38.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e14146910 con 0x7f4e140b5650 2022-07-20T03:17:38.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4e0c0070d0 con 0x7f4e140b5650 2022-07-20T03:17:38.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e141449b0 con 0x7f4e140b5650 2022-07-20T03:17:38.280 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e140574e0 con 0x7f4e140b5650 2022-07-20T03:17:38.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4e0c010070 con 0x7f4e140b5650 2022-07-20T03:17:38.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e06ffd700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4df4053af0 0x7f4df4055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f4e0c06b750 con 0x7f4e140b5650 2022-07-20T03:17:38.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.255+0000 7f4e19fc9700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4df4053af0 0x7f4df4055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.281 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.259+0000 7f4e19fc9700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4df4053af0 0x7f4df4055fb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f4e0800a000 tx=0x7f4e08008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:38.282 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.263+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4e0c0426b0 con 0x7f4e140b5650 2022-07-20T03:17:38.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.427+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f4e14067ea0 con 0x7f4e140b5650 2022-07-20T03:17:38.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.427+0000 7f4e06ffd700 1 -- 172.21.15.61:0/1752095342 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f4e0c045cd0 con 0x7f4e140b5650 2022-07-20T03:17:38.434 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:38.434 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7397,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1660470104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1660470104},{"type":"v1","addr":"172.21.15.61:6833","nonce":1660470104}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:35.498516+0000","modified":"2022-07-20T03:17:36.503758+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/1280182602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1280182602},{"type":"v1","addr":"172.21.15.117:6835","nonce":1280182602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T03:17:38.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4df4053af0 msgr2=0x7f4df4055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4df4053af0 0x7f4df4055fb0 secure :-1 s=READY pgs=717 cs=0 l=1 rev1=1 rx=0x7f4e0800a000 tx=0x7f4e08008040).stop 2022-07-20T03:17:38.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 msgr2=0x7f4e1413dee0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 0x7f4e1413dee0 secure :-1 s=READY pgs=1491 cs=0 l=1 rev1=1 rx=0x7f4e14062420 tx=0x7f4e0c007ee0).stop 2022-07-20T03:17:38.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 shutdown_connections 2022-07-20T03:17:38.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4df4053af0 0x7f4df4055fb0 unknown :-1 s=CLOSED pgs=717 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e140b5650 0x7f4e1413dee0 unknown :-1 s=CLOSED pgs=1491 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e140b1200 0x7f4e14152cf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 --2- 172.21.15.61:0/1752095342 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e1405bd90 0x7f4e141461e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 >> 172.21.15.61:0/1752095342 conn(0x7f4e140634e0 msgr2=0x7f4e140b1710 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:38.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 shutdown_connections 2022-07-20T03:17:38.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.431+0000 7f4e1afcb700 1 -- 172.21.15.61:0/1752095342 wait complete. 2022-07-20T03:17:38.442 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-20T03:17:38.648 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:17:35.498516+0000', 'modified': '2022-07-20T03:17:36.503758+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8665': {'gid': 8665, 'name': 'b', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/1280182602', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1280182602}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1280182602}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:17:38.649 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:17:38.650 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:17:38.766 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8665,"want_state":"up:active","state":"up:active","rank_uptime":3.251989107,"mdsmap_epoch":83,"osdmap_epoch":149,"osdmap_epoch_barrier":149,"uptime":7.5906716809999999} 2022-07-20T03:17:38.767 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8665, "mdsmap_epoch": 83, "osdmap_epoch": 149, "osdmap_epoch_barrier": 149, "rank_uptime": 3.251989107, "state": "up:active", "uptime": 7.590671681, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:17:38.768 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:38.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.875+0000 7f3755bf3700 1 Processor -- start 2022-07-20T03:17:38.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.875+0000 7f3755bf3700 1 -- start start 2022-07-20T03:17:38.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f375005d160 0x7f37500b4e50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500ba2d0 con 0x7f37500b2960 2022-07-20T03:17:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37500b4560 con 0x7f37500b9ef0 2022-07-20T03:17:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37500b4800 con 0x7f375005d160 2022-07-20T03:17:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f37500b2960 0x7f3750142210 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48450/0 (socket says 172.21.15.61:48450) 2022-07-20T03:17:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374ffff700 1 -- 172.21.15.61:0/4029354469 learned_addr learned my addr 172.21.15.61:0/4029354469 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:38.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374effd700 1 --2- 172.21.15.61:0/4029354469 >> v2:172.21.15.117:3300/0 conn(0x7f375005d160 0x7f37500b4e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374e7fc700 1 -- 172.21.15.61:0/4029354469 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3365862469 0 0) 0x7f37500ba2d0 con 0x7f37500b2960 2022-07-20T03:17:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374e7fc700 1 -- 172.21.15.61:0/4029354469 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f373c003260 con 0x7f37500b2960 2022-07-20T03:17:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374effd700 1 -- 172.21.15.61:0/4029354469 >> v1:172.21.15.117:6789/0 conn(0x7f37500b9ef0 legacy=0x7f37500b2250 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374e7fc700 1 -- 172.21.15.61:0/4029354469 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2842482563 0 0) 0x7f37500b4560 con 0x7f37500b9ef0 2022-07-20T03:17:38.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374effd700 1 -- 172.21.15.61:0/4029354469 >> v1:172.21.15.61:6789/0 conn(0x7f37500b2960 legacy=0x7f3750142210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374effd700 1 -- 172.21.15.61:0/4029354469 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37500b1db0 con 0x7f375005d160 2022-07-20T03:17:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374effd700 1 --2- 172.21.15.61:0/4029354469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f375005d160 0x7f37500b4e50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f3744002f70 tx=0x7f3744006df0).ready entity=mon.1 client_cookie=701149bdbdbd0479 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374e7fc700 1 -- 172.21.15.61:0/4029354469 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374400d610 con 0x7f375005d160 2022-07-20T03:17:38.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374e7fc700 1 -- 172.21.15.61:0/4029354469 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f3744000e90 con 0x7f375005d160 2022-07-20T03:17:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f374e7fc700 1 -- 172.21.15.61:0/4029354469 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374400d960 con 0x7f375005d160 2022-07-20T03:17:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- 172.21.15.61:0/4029354469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f375005d160 msgr2=0x7f37500b4e50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 --2- 172.21.15.61:0/4029354469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f375005d160 0x7f37500b4e50 secure :-1 s=READY pgs=665 cs=0 l=1 rev1=1 rx=0x7f3744002f70 tx=0x7f3744006df0).stop 2022-07-20T03:17:38.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- 172.21.15.61:0/4029354469 shutdown_connections 2022-07-20T03:17:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 --2- 172.21.15.61:0/4029354469 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f375005d160 0x7f37500b4e50 unknown :-1 s=CLOSED pgs=665 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- 172.21.15.61:0/4029354469 >> 172.21.15.61:0/4029354469 conn(0x7f37500679d0 msgr2=0x7f37500b94a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- 172.21.15.61:0/4029354469 shutdown_connections 2022-07-20T03:17:38.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.879+0000 7f3755bf3700 1 -- 172.21.15.61:0/4029354469 wait complete. 2022-07-20T03:17:38.894 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 Processor -- start 2022-07-20T03:17:38.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 -- start start 2022-07-20T03:17:38.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f375005d160 0x7f37501452a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37500b2960 0x7f3750155680 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 0x7f37500ada60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3750144e10 con 0x7f37500b9ef0 2022-07-20T03:17:38.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3750156000 con 0x7f37500b2960 2022-07-20T03:17:38.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f3755bf3700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37501562a0 con 0x7f375005d160 2022-07-20T03:17:38.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 0x7f37500ada60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 0x7f37500ada60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51858/0 (socket says 172.21.15.61:51858) 2022-07-20T03:17:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 -- 172.21.15.61:0/1496918233 learned_addr learned my addr 172.21.15.61:0/1496918233 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374effd700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37500b2960 0x7f3750155680 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374f7fe700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f375005d160 0x7f37501452a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 -- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f375005d160 msgr2=0x7f37501452a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f375005d160 0x7f37501452a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 -- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37500b2960 msgr2=0x7f3750155680 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37500b2960 0x7f3750155680 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 -- 172.21.15.61:0/1496918233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f374400b180 con 0x7f37500b9ef0 2022-07-20T03:17:38.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374effd700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37500b2960 0x7f3750155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:38.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374ffff700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 0x7f37500ada60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7f3740001110 tx=0x7f3740007a80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:38.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.883+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f374000ce80 con 0x7f37500b9ef0 2022-07-20T03:17:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37400070d0 con 0x7f37500b9ef0 2022-07-20T03:17:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3750145790 con 0x7f37500b9ef0 2022-07-20T03:17:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f3740007430 con 0x7f37500b9ef0 2022-07-20T03:17:38.901 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3750155d90 con 0x7f37500b9ef0 2022-07-20T03:17:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37500af1c0 con 0x7f37500b9ef0 2022-07-20T03:17:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f3740010070 con 0x7f37500b9ef0 2022-07-20T03:17:38.902 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f374cff9700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37300539b0 0x7f3730055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f374006bc10 con 0x7f37500b9ef0 2022-07-20T03:17:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.887+0000 7f374f7fe700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37300539b0 0x7f3730055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:38.903 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.891+0000 7f374f7fe700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37300539b0 0x7f3730055e70 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f374800d000 tx=0x7f3748007f10).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:38.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:38.891+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f3740042a60 con 0x7f37500b9ef0 2022-07-20T03:17:39.024 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.019+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f3750144fe0 con 0x7f37500b9ef0 2022-07-20T03:17:39.026 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f374cff9700 1 -- 172.21.15.61:0/1496918233 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f3750144fe0 con 0x7f37500b9ef0 2022-07-20T03:17:39.026 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:39.027 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":149,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:36.502720+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:34.158104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:34.713017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:39.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37300539b0 msgr2=0x7f3730055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.029 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37300539b0 0x7f3730055e70 secure :-1 s=READY pgs=718 cs=0 l=1 rev1=1 rx=0x7f374800d000 tx=0x7f3748007f10).stop 2022-07-20T03:17:39.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 msgr2=0x7f37500ada60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 0x7f37500ada60 secure :-1 s=READY pgs=1493 cs=0 l=1 rev1=1 rx=0x7f3740001110 tx=0x7f3740007a80).stop 2022-07-20T03:17:39.030 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 shutdown_connections 2022-07-20T03:17:39.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37300539b0 0x7f3730055e70 unknown :-1 s=CLOSED pgs=718 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37500b2960 0x7f3750155680 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37500b9ef0 0x7f37500ada60 unknown :-1 s=CLOSED pgs=1493 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.031 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 --2- 172.21.15.61:0/1496918233 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f375005d160 0x7f37501452a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 >> 172.21.15.61:0/1496918233 conn(0x7f37500679d0 msgr2=0x7f37500b5440 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 shutdown_connections 2022-07-20T03:17:39.032 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.023+0000 7f3755bf3700 1 -- 172.21.15.61:0/1496918233 wait complete. 2022-07-20T03:17:39.040 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:17:34.158104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:17:34.713017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:17:39.041 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:17:39.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 Processor -- start 2022-07-20T03:17:39.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 -- start start 2022-07-20T03:17:39.157 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fa1a40b9da0 0x7fa1a41468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1a40b3cf0 0x7fa1a4142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fa1a40ba180 con 0x7fa1a4142f40 2022-07-20T03:17:39.158 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a40ad5b0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a9d40700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fa1a41461b0 con 0x7fa1a40b9da0 2022-07-20T03:17:39.160 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a3fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fa1a40b3cf0 0x7fa1a4142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a953f700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fa1a4142f40 0x7fa1a405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48462/0 (socket says 172.21.15.61:48462) 2022-07-20T03:17:39.161 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a953f700 1 -- 172.21.15.61:0/681077867 learned_addr learned my addr 172.21.15.61:0/681077867 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:39.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a8d3e700 1 --2- 172.21.15.61:0/681077867 >> v2:172.21.15.117:3300/0 conn(0x7fa1a40b9da0 0x7fa1a41468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a3fff700 1 -- 172.21.15.61:0/681077867 >> v2:172.21.15.117:3300/0 conn(0x7fa1a40b9da0 msgr2=0x7fa1a41468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:39.162 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a3fff700 1 --2- 172.21.15.61:0/681077867 >> v2:172.21.15.117:3300/0 conn(0x7fa1a40b9da0 0x7fa1a41468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a3fff700 1 -- 172.21.15.61:0/681077867 >> v1:172.21.15.61:6789/0 conn(0x7fa1a4142f40 legacy=0x7fa1a405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a3fff700 1 -- 172.21.15.61:0/681077867 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1a4146450 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.147+0000 7fa1a37fe700 1 -- 172.21.15.61:0/681077867 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 4043504165 0 0) 0x7fa1a40ba180 con 0x7fa1a4142f40 2022-07-20T03:17:39.163 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a3fff700 1 --2- 172.21.15.61:0/681077867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a4142a50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fa194002f70 tx=0x7fa194006df0).ready entity=mon.0 client_cookie=c84fc30337dc45d3 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a37fe700 1 -- 172.21.15.61:0/681077867 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19400d570 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a37fe700 1 -- 172.21.15.61:0/681077867 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa194000e90 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.164 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a37fe700 1 -- 172.21.15.61:0/681077867 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa19400d8c0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 -- 172.21.15.61:0/681077867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 msgr2=0x7fa1a4142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.165 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/681077867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a4142a50 secure :-1 s=READY pgs=1495 cs=0 l=1 rev1=1 rx=0x7fa194002f70 tx=0x7fa194006df0).stop 2022-07-20T03:17:39.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 -- 172.21.15.61:0/681077867 shutdown_connections 2022-07-20T03:17:39.166 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/681077867 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a4142a50 unknown :-1 s=CLOSED pgs=1495 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/681077867 >> v2:172.21.15.117:3300/0 conn(0x7fa1a40b9da0 0x7fa1a41468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.167 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 -- 172.21.15.61:0/681077867 >> 172.21.15.61:0/681077867 conn(0x7fa1a40678c0 msgr2=0x7fa1a40b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 -- 172.21.15.61:0/681077867 shutdown_connections 2022-07-20T03:17:39.168 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 -- 172.21.15.61:0/681077867 wait complete. 2022-07-20T03:17:39.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 Processor -- start 2022-07-20T03:17:39.169 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.151+0000 7fa1a9d40700 1 -- start start 2022-07-20T03:17:39.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a41483b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.177 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1a40b9da0 0x7fa1a4143f30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.178 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1a4142f40 0x7fa1a40b2960 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a4145170 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40af3a0 con 0x7fa1a4142f40 2022-07-20T03:17:39.179 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fa1a40af640 con 0x7fa1a40b9da0 2022-07-20T03:17:39.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a41483b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a41483b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51868/0 (socket says 172.21.15.61:51868) 2022-07-20T03:17:39.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 -- 172.21.15.61:0/3153944317 learned_addr learned my addr 172.21.15.61:0/3153944317 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:39.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a3fff700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1a40b9da0 0x7fa1a4143f30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.180 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a953f700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1a4142f40 0x7fa1a40b2960 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 -- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1a40b9da0 msgr2=0x7fa1a4143f30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1a40b9da0 0x7fa1a4143f30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 -- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1a4142f40 msgr2=0x7fa1a40b2960 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.181 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1a4142f40 0x7fa1a40b2960 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 -- 172.21.15.61:0/3153944317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa19400b180 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a3fff700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1a40b9da0 0x7fa1a4143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a953f700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1a4142f40 0x7fa1a40b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:39.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a8d3e700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a41483b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7fa19800c000 tx=0x7fa198006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa198010780 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa1a40b24c0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fa1980108e0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.183 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1a4145b40 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.155+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fa198010c40 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.159+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa1a40b0df0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.159+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fa19801c460 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.159+0000 7fa1a1ffb700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa184053ad0 0x7fa184055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.159+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fa19806c560 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.159+0000 7fa1a3fff700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa184053ad0 0x7fa184055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.159+0000 7fa1a3fff700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa184053ad0 0x7fa184055f90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fa194018000 tx=0x7fa194014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.163+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fa1a40b0df0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.299 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.291+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7fa1a40b0df0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.300 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.291+0000 7fa1a1ffb700 1 -- 172.21.15.61:0/3153944317 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v149) v1 ==== 110+0+10 (secure 0 0 0) 0x7fa1a40b0df0 con 0x7fa1a40b3cf0 2022-07-20T03:17:39.300 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:17:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa184053ad0 msgr2=0x7fa184055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.302 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa184053ad0 0x7fa184055f90 secure :-1 s=READY pgs=719 cs=0 l=1 rev1=1 rx=0x7fa194018000 tx=0x7fa194014000).stop 2022-07-20T03:17:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 msgr2=0x7fa1a41483b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a41483b0 secure :-1 s=READY pgs=1496 cs=0 l=1 rev1=1 rx=0x7fa19800c000 tx=0x7fa198006df0).stop 2022-07-20T03:17:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 shutdown_connections 2022-07-20T03:17:39.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fa184053ad0 0x7fa184055f90 unknown :-1 s=CLOSED pgs=719 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fa1a4142f40 0x7fa1a40b2960 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fa1a40b9da0 0x7fa1a4143f30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 --2- 172.21.15.61:0/3153944317 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fa1a40b3cf0 0x7fa1a41483b0 unknown :-1 s=CLOSED pgs=1496 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 >> 172.21.15.61:0/3153944317 conn(0x7fa1a40678c0 msgr2=0x7fa1a405bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 shutdown_connections 2022-07-20T03:17:39.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.295+0000 7fa1a9d40700 1 -- 172.21.15.61:0/3153944317 wait complete. 2022-07-20T03:17:39.312 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:17:39.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.419+0000 7fc37fcec700 1 Processor -- start 2022-07-20T03:17:39.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- start start 2022-07-20T03:17:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc37805cf40 0x7fc3780afd20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3780b1220 0x7fc37814a570 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fc3781472b0 con 0x7fc378147c90 2022-07-20T03:17:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fc3780b0920 con 0x7fc37805cf40 2022-07-20T03:17:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc378147550 con 0x7fc3780b1220 2022-07-20T03:17:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc37805cf40 0x7fc3780afd20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fc37805cf40 0x7fc3780afd20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51878/0 (socket says 172.21.15.61:51878) 2022-07-20T03:17:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 -- 172.21.15.61:0/3405136564 learned_addr learned my addr 172.21.15.61:0/3405136564 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37e289700 1 --2- 172.21.15.61:0/3405136564 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3780b1220 0x7fc37814a570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37ca86700 1 -- 172.21.15.61:0/3405136564 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3853464582 0 0) 0x7fc3781472b0 con 0x7fc378147c90 2022-07-20T03:17:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37ca86700 1 -- 172.21.15.61:0/3405136564 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fc360003260 con 0x7fc378147c90 2022-07-20T03:17:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 -- 172.21.15.61:0/3405136564 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3780b1220 msgr2=0x7fc37814a570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 --2- 172.21.15.61:0/3405136564 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3780b1220 0x7fc37814a570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 -- 172.21.15.61:0/3405136564 >> v1:172.21.15.117:6789/0 conn(0x7fc378147c90 legacy=0x7fc3780b0210 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 -- 172.21.15.61:0/3405136564 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3781477f0 con 0x7fc37805cf40 2022-07-20T03:17:39.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37da88700 1 --2- 172.21.15.61:0/3405136564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37805cf40 0x7fc3780afd20 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7fc368002f70 tx=0x7fc368006eb0).ready entity=mon.0 client_cookie=9fcdf060138b4cdc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37ca86700 1 -- 172.21.15.61:0/3405136564 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc36800d5c0 con 0x7fc37805cf40 2022-07-20T03:17:39.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37ca86700 1 -- 172.21.15.61:0/3405136564 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc368000e90 con 0x7fc37805cf40 2022-07-20T03:17:39.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37ca86700 1 -- 172.21.15.61:0/3405136564 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc36800d950 con 0x7fc37805cf40 2022-07-20T03:17:39.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- 172.21.15.61:0/3405136564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37805cf40 msgr2=0x7fc3780afd20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 --2- 172.21.15.61:0/3405136564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37805cf40 0x7fc3780afd20 secure :-1 s=READY pgs=1497 cs=0 l=1 rev1=1 rx=0x7fc368002f70 tx=0x7fc368006eb0).stop 2022-07-20T03:17:39.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- 172.21.15.61:0/3405136564 shutdown_connections 2022-07-20T03:17:39.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 --2- 172.21.15.61:0/3405136564 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc3780b1220 0x7fc37814a570 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 --2- 172.21.15.61:0/3405136564 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc37805cf40 0x7fc3780afd20 unknown :-1 s=CLOSED pgs=1497 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- 172.21.15.61:0/3405136564 >> 172.21.15.61:0/3405136564 conn(0x7fc3780b5b70 msgr2=0x7fc3780b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- 172.21.15.61:0/3405136564 shutdown_connections 2022-07-20T03:17:39.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.423+0000 7fc37fcec700 1 -- 172.21.15.61:0/3405136564 wait complete. 2022-07-20T03:17:39.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 Processor -- start 2022-07-20T03:17:39.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 -- start start 2022-07-20T03:17:39.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc37805cf40 0x7fc378141900 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 0x7fc378156ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc378147c90 0x7fc3781556d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378063ac0 con 0x7fc3780b1220 2022-07-20T03:17:39.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc378156fb0 con 0x7fc37805cf40 2022-07-20T03:17:39.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37fcec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fc378063cf0 con 0x7fc378147c90 2022-07-20T03:17:39.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 0x7fc378156ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37da88700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc37805cf40 0x7fc378141900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 0x7fc378156ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51884/0 (socket says 172.21.15.61:51884) 2022-07-20T03:17:39.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37da88700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc37805cf40 0x7fc378141900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:33330/0 (socket says 172.21.15.61:33330) 2022-07-20T03:17:39.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 -- 172.21.15.61:0/2679307625 learned_addr learned my addr 172.21.15.61:0/2679307625 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:39.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37e289700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc378147c90 0x7fc3781556d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 -- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc378147c90 msgr2=0x7fc3781556d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc378147c90 0x7fc3781556d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 -- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc37805cf40 msgr2=0x7fc378141900 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc37805cf40 0x7fc378141900 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.427+0000 7fc37d287700 1 -- 172.21.15.61:0/2679307625 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc36800b1e0 con 0x7fc3780b1220 2022-07-20T03:17:39.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc37d287700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 0x7fc378156ac0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7fc378157140 tx=0x7fc374007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc374012400 con 0x7fc3780b1220 2022-07-20T03:17:39.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fc374012560 con 0x7fc3780b1220 2022-07-20T03:17:39.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc378144770 con 0x7fc3780b1220 2022-07-20T03:17:39.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fc3740128c0 con 0x7fc3780b1220 2022-07-20T03:17:39.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc378064430 con 0x7fc3780b1220 2022-07-20T03:17:39.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc378146fa0 con 0x7fc3780b1220 2022-07-20T03:17:39.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fc374019030 con 0x7fc3780b1220 2022-07-20T03:17:39.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.431+0000 7fc366ffd700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3580539d0 0x7fc358055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.435+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fc37406b140 con 0x7fc3780b1220 2022-07-20T03:17:39.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.435+0000 7fc37da88700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3580539d0 0x7fc358055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.435+0000 7fc37da88700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3580539d0 0x7fc358055e90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fc36801c000 tx=0x7fc368015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.439+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fc3740420a0 con 0x7fc3780b1220 2022-07-20T03:17:39.570 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fc3780574e0 con 0x7fc3780b1220 2022-07-20T03:17:39.571 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc366ffd700 1 -- 172.21.15.61:0/2679307625 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v149) v1 ==== 92+0+10 (secure 0 0 0) 0x7fc3740456c0 con 0x7fc3780b1220 2022-07-20T03:17:39.572 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3580539d0 msgr2=0x7fc358055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.574 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3580539d0 0x7fc358055e90 secure :-1 s=READY pgs=720 cs=0 l=1 rev1=1 rx=0x7fc36801c000 tx=0x7fc368015000).stop 2022-07-20T03:17:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 msgr2=0x7fc378156ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 0x7fc378156ac0 secure :-1 s=READY pgs=1498 cs=0 l=1 rev1=1 rx=0x7fc378157140 tx=0x7fc374007ae0).stop 2022-07-20T03:17:39.577 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 shutdown_connections 2022-07-20T03:17:39.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fc3580539d0 0x7fc358055e90 unknown :-1 s=CLOSED pgs=720 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.578 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fc378147c90 0x7fc3781556d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.579 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fc3780b1220 0x7fc378156ac0 unknown :-1 s=CLOSED pgs=1498 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 --2- 172.21.15.61:0/2679307625 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fc37805cf40 0x7fc378141900 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.580 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 >> 172.21.15.61:0/2679307625 conn(0x7fc3780b5b70 msgr2=0x7fc37813e1b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.581 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 shutdown_connections 2022-07-20T03:17:39.582 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.567+0000 7fc37fcec700 1 -- 172.21.15.61:0/2679307625 wait complete. 2022-07-20T03:17:39.584 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:17:39.694 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 Processor -- start 2022-07-20T03:17:39.695 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 -- start start 2022-07-20T03:17:39.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7feabc0b5b50 0x7feabc0accb0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.699 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0ac5e0 con 0x7feabc0b0480 2022-07-20T03:17:39.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7feabc0b5f30 con 0x7feabc05c120 2022-07-20T03:17:39.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7feabc05cf70 con 0x7feabc0b5b50 2022-07-20T03:17:39.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabbfff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7feabc0b0480 0x7feabc14a670 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48490/0 (socket says 172.21.15.61:48490) 2022-07-20T03:17:39.700 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabbfff700 1 -- 172.21.15.61:0/3904248876 learned_addr learned my addr 172.21.15.61:0/3904248876 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:39.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac0be2700 1 --2- 172.21.15.61:0/3904248876 >> v2:172.21.15.117:3300/0 conn(0x7feabc0b5b50 0x7feabc0accb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabb7fe700 1 -- 172.21.15.61:0/3904248876 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2101146056 0 0) 0x7feabc0ac5e0 con 0x7feabc0b0480 2022-07-20T03:17:39.701 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabb7fe700 1 -- 172.21.15.61:0/3904248876 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7feaa0003260 con 0x7feabc0b0480 2022-07-20T03:17:39.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac0be2700 1 -- 172.21.15.61:0/3904248876 >> v1:172.21.15.117:6789/0 conn(0x7feabc05c120 legacy=0x7feabc144320 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.702 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac0be2700 1 -- 172.21.15.61:0/3904248876 >> v1:172.21.15.61:6789/0 conn(0x7feabc0b0480 legacy=0x7feabc14a670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac0be2700 1 -- 172.21.15.61:0/3904248876 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feabc0ac810 con 0x7feabc0b5b50 2022-07-20T03:17:39.703 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac0be2700 1 --2- 172.21.15.61:0/3904248876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc0accb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7feab0002f70 tx=0x7feab0006df0).ready entity=mon.1 client_cookie=98aed6691a441d82 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabb7fe700 1 -- 172.21.15.61:0/3904248876 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feab000d610 con 0x7feabc0b5b50 2022-07-20T03:17:39.704 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabb7fe700 1 -- 172.21.15.61:0/3904248876 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab0000e90 con 0x7feabc0b5b50 2022-07-20T03:17:39.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feabb7fe700 1 -- 172.21.15.61:0/3904248876 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feab000d960 con 0x7feabc0b5b50 2022-07-20T03:17:39.705 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 -- 172.21.15.61:0/3904248876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 msgr2=0x7feabc0accb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.691+0000 7feac2e46700 1 --2- 172.21.15.61:0/3904248876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc0accb0 secure :-1 s=READY pgs=668 cs=0 l=1 rev1=1 rx=0x7feab0002f70 tx=0x7feab0006df0).stop 2022-07-20T03:17:39.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- 172.21.15.61:0/3904248876 shutdown_connections 2022-07-20T03:17:39.706 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 --2- 172.21.15.61:0/3904248876 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc0accb0 unknown :-1 s=CLOSED pgs=668 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- 172.21.15.61:0/3904248876 >> 172.21.15.61:0/3904248876 conn(0x7feabc0635e0 msgr2=0x7feabc0b5100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.707 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- 172.21.15.61:0/3904248876 shutdown_connections 2022-07-20T03:17:39.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- 172.21.15.61:0/3904248876 wait complete. 2022-07-20T03:17:39.708 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 Processor -- start 2022-07-20T03:17:39.709 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- start start 2022-07-20T03:17:39.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 0x7feabc09e5c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feabc0b0480 0x7feabc148570 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.828 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc156df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc14a360 con 0x7feabc05c120 2022-07-20T03:17:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feabc09f7a0 con 0x7feabc0b5b50 2022-07-20T03:17:39.829 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac2e46700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7feabc1566b0 con 0x7feabc0b0480 2022-07-20T03:17:39.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.695+0000 7feac0be2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 0x7feabc09e5c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 0x7feabc09e5c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51896/0 (socket says 172.21.15.61:51896) 2022-07-20T03:17:39.830 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 -- 172.21.15.61:0/3318928461 learned_addr learned my addr 172.21.15.61:0/3318928461 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:39.835 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feabbfff700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feabc0b0480 0x7feabc148570 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac13e3700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc156df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.836 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 -- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feabc0b0480 msgr2=0x7feabc148570 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feabc0b0480 0x7feabc148570 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 -- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 msgr2=0x7feabc156df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc156df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.837 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 -- 172.21.15.61:0/3318928461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feab000b180 con 0x7feabc05c120 2022-07-20T03:17:39.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feabbfff700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feabc0b0480 0x7feabc148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:39.838 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac13e3700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac0be2700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 0x7feabc09e5c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7feab0002f70 tx=0x7feab0014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feab0013de0 con 0x7feabc05c120 2022-07-20T03:17:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7feab000d290 con 0x7feabc05c120 2022-07-20T03:17:39.839 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feabc156950 con 0x7feabc05c120 2022-07-20T03:17:39.841 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7feab001e5f0 con 0x7feabc05c120 2022-07-20T03:17:39.847 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feabc0a06f0 con 0x7feabc05c120 2022-07-20T03:17:39.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7feab0025030 con 0x7feabc05c120 2022-07-20T03:17:39.848 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feabc09f5c0 con 0x7feabc05c120 2022-07-20T03:17:39.849 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.699+0000 7feab9ffb700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea9c053af0 0x7fea9c055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.850 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.703+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7feab006ccc0 con 0x7feabc05c120 2022-07-20T03:17:39.851 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.703+0000 7feabbfff700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea9c053af0 0x7fea9c055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.703+0000 7feabbfff700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea9c053af0 0x7fea9c055fb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7feaa800d000 tx=0x7feaa8009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.852 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.711+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7feab0073080 con 0x7feabc05c120 2022-07-20T03:17:39.854 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.835+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7feabc0574e0 con 0x7feabc05c120 2022-07-20T03:17:39.855 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feab9ffb700 1 -- 172.21.15.61:0/3318928461 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+10 (secure 0 0 0) 0x7feab0043b10 con 0x7feabc05c120 2022-07-20T03:17:39.855 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:39.857 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea9c053af0 msgr2=0x7fea9c055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea9c053af0 0x7fea9c055fb0 secure :-1 s=READY pgs=721 cs=0 l=1 rev1=1 rx=0x7feaa800d000 tx=0x7feaa8009040).stop 2022-07-20T03:17:39.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 msgr2=0x7feabc09e5c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.858 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 0x7feabc09e5c0 secure :-1 s=READY pgs=1500 cs=0 l=1 rev1=1 rx=0x7feab0002f70 tx=0x7feab0014000).stop 2022-07-20T03:17:39.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 shutdown_connections 2022-07-20T03:17:39.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fea9c053af0 0x7fea9c055fb0 unknown :-1 s=CLOSED pgs=721 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7feabc0b5b50 0x7feabc156df0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.859 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7feabc0b0480 0x7feabc148570 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 --2- 172.21.15.61:0/3318928461 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7feabc05c120 0x7feabc09e5c0 unknown :-1 s=CLOSED pgs=1500 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 >> 172.21.15.61:0/3318928461 conn(0x7feabc0635e0 msgr2=0x7feabc0b50e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.860 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 shutdown_connections 2022-07-20T03:17:39.861 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.839+0000 7feac2e46700 1 -- 172.21.15.61:0/3318928461 wait complete. 2022-07-20T03:17:39.861 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:17:39.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 Processor -- start 2022-07-20T03:17:39.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 -- start start 2022-07-20T03:17:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac610 0x7f14cc0aba70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f14cc0ac9f0 0x7f14cc140520 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f14cc0aced0 con 0x7f14cc0b0b90 2022-07-20T03:17:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f14cc143c80 con 0x7f14cc0ac9f0 2022-07-20T03:17:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d2263700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc143f20 con 0x7f14cc0ac610 2022-07-20T03:17:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d1a62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f14cc0ac9f0 0x7f14cc140520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.981 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d1a62700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f14cc0ac9f0 0x7f14cc140520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51908/0 (socket says 172.21.15.61:51908) 2022-07-20T03:17:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d1a62700 1 -- 172.21.15.61:0/2107499148 learned_addr learned my addr 172.21.15.61:0/2107499148 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.971+0000 7f14d1261700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac610 0x7f14cc0aba70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d1a62700 1 -- 172.21.15.61:0/2107499148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac610 msgr2=0x7f14cc0aba70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:39.982 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d1a62700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac610 0x7f14cc0aba70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d1a62700 1 -- 172.21.15.61:0/2107499148 >> v1:172.21.15.61:6789/0 conn(0x7f14cc0b0b90 legacy=0x7f14cc05fad0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d1a62700 1 -- 172.21.15.61:0/2107499148 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14cc098b20 con 0x7f14cc0ac9f0 2022-07-20T03:17:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d1261700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac610 0x7f14cc0aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T03:17:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d1a62700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac9f0 0x7f14cc140520 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7f14c40032f0 tx=0x7f14c4009290).ready entity=mon.0 client_cookie=7f8f00078c5be567 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14bbfff700 1 -- 172.21.15.61:0/2107499148 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14c400b8a0 con 0x7f14cc0ac9f0 2022-07-20T03:17:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14bbfff700 1 -- 172.21.15.61:0/2107499148 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c400ba00 con 0x7f14cc0ac9f0 2022-07-20T03:17:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14bbfff700 1 -- 172.21.15.61:0/2107499148 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14c400bb70 con 0x7f14cc0ac9f0 2022-07-20T03:17:39.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- 172.21.15.61:0/2107499148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac9f0 msgr2=0x7f14cc140520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac9f0 0x7f14cc140520 secure :-1 s=READY pgs=1502 cs=0 l=1 rev1=1 rx=0x7f14c40032f0 tx=0x7f14c4009290).stop 2022-07-20T03:17:39.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- 172.21.15.61:0/2107499148 shutdown_connections 2022-07-20T03:17:39.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac9f0 0x7f14cc140520 unknown :-1 s=CLOSED pgs=1502 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 --2- 172.21.15.61:0/2107499148 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac610 0x7f14cc0aba70 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- 172.21.15.61:0/2107499148 >> 172.21.15.61:0/2107499148 conn(0x7f14cc0b61f0 msgr2=0x7f14cc0b65d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:39.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- 172.21.15.61:0/2107499148 shutdown_connections 2022-07-20T03:17:39.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- 172.21.15.61:0/2107499148 wait complete. 2022-07-20T03:17:39.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 Processor -- start 2022-07-20T03:17:39.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- start start 2022-07-20T03:17:39.991 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 0x7f14cc0b1300 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac9f0 0x7f14cc066c20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14cc0b0b90 0x7f14cc1558d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc066260 con 0x7f14cc0ac610 2022-07-20T03:17:39.992 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc0664e0 con 0x7f14cc0b0b90 2022-07-20T03:17:39.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.975+0000 7f14d2263700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f14cc066780 con 0x7f14cc0ac9f0 2022-07-20T03:17:39.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 0x7f14cc0b1300 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 0x7f14cc0b1300 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51910/0 (socket says 172.21.15.61:51910) 2022-07-20T03:17:39.993 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 -- 172.21.15.61:0/3137604102 learned_addr learned my addr 172.21.15.61:0/3137604102 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d0a60700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac9f0 0x7f14cc066c20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1a62700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14cc0b0b90 0x7f14cc1558d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 -- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac9f0 msgr2=0x7f14cc066c20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.994 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac9f0 0x7f14cc066c20 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 -- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14cc0b0b90 msgr2=0x7f14cc1558d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:39.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14cc0b0b90 0x7f14cc1558d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:39.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 -- 172.21.15.61:0/3137604102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f14c4009530 con 0x7f14cc0ac610 2022-07-20T03:17:39.995 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d0a60700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac9f0 0x7f14cc066c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:39.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d1261700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 0x7f14cc0b1300 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7f14c8007000 tx=0x7f14c8008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14c8010580 con 0x7f14cc0ac610 2022-07-20T03:17:39.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f14c80106e0 con 0x7f14cc0ac610 2022-07-20T03:17:39.996 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f14cc1451f0 con 0x7f14cc0ac610 2022-07-20T03:17:39.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f14c8010a40 con 0x7f14cc0ac610 2022-07-20T03:17:39.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14cc1456f0 con 0x7f14cc0ac610 2022-07-20T03:17:39.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f14cc0679a0 con 0x7f14cc0ac610 2022-07-20T03:17:39.997 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.979+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f14c8010ba0 con 0x7f14cc0ac610 2022-07-20T03:17:39.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.983+0000 7f14ba7fc700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f14ac0539b0 0x7f14ac055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:39.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.983+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f14c806c980 con 0x7f14cc0ac610 2022-07-20T03:17:39.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.983+0000 7f14d0a60700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f14ac0539b0 0x7f14ac055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:39.998 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.983+0000 7f14d0a60700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f14ac0539b0 0x7f14ac055e70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f14bc00b000 tx=0x7f14bc009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:39.999 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:39.987+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f14c8043880 con 0x7f14cc0ac610 2022-07-20T03:17:40.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.111+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f14cc0679a0 con 0x7f14cc0ac610 2022-07-20T03:17:40.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.111+0000 7f14ba7fc700 1 -- 172.21.15.61:0/3137604102 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+10 (secure 0 0 0) 0x7f14cc0679a0 con 0x7f14cc0ac610 2022-07-20T03:17:40.117 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:40.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.111+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f14ac0539b0 msgr2=0x7f14ac055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.111+0000 7f14d2263700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f14ac0539b0 0x7f14ac055e70 secure :-1 s=READY pgs=722 cs=0 l=1 rev1=1 rx=0x7f14bc00b000 tx=0x7f14bc009040).stop 2022-07-20T03:17:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.111+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 msgr2=0x7f14cc0b1300 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.111+0000 7f14d2263700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 0x7f14cc0b1300 secure :-1 s=READY pgs=1503 cs=0 l=1 rev1=1 rx=0x7f14c8007000 tx=0x7f14c8008040).stop 2022-07-20T03:17:40.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 shutdown_connections 2022-07-20T03:17:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f14ac0539b0 0x7f14ac055e70 unknown :-1 s=CLOSED pgs=722 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f14cc0b0b90 0x7f14cc1558d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f14cc0ac9f0 0x7f14cc066c20 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 --2- 172.21.15.61:0/3137604102 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f14cc0ac610 0x7f14cc0b1300 unknown :-1 s=CLOSED pgs=1503 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 >> 172.21.15.61:0/3137604102 conn(0x7f14cc0b61f0 msgr2=0x7f14cc13f6a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:40.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 shutdown_connections 2022-07-20T03:17:40.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.115+0000 7f14d2263700 1 -- 172.21.15.61:0/3137604102 wait complete. 2022-07-20T03:17:40.130 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:17:40.130 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:40.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f1339c80700 1 Processor -- start 2022-07-20T03:17:40.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f1339c80700 1 -- start start 2022-07-20T03:17:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f1339c80700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13340af8e0 0x7f13340b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f1339c80700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1334147010 con 0x7f13340adff0 2022-07-20T03:17:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f1339c80700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f13341472b0 con 0x7f13340b5650 2022-07-20T03:17:40.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f1339c80700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1334146d40 con 0x7f13340af8e0 2022-07-20T03:17:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13340af8e0 0x7f13340b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f13340af8e0 0x7f13340b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51922/0 (socket says 172.21.15.61:51922) 2022-07-20T03:17:40.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 -- 172.21.15.61:0/3570135855 learned_addr learned my addr 172.21.15.61:0/3570135855 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 -- 172.21.15.61:0/3570135855 >> v1:172.21.15.117:6789/0 conn(0x7f13340b5650 legacy=0x7f1334147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 -- 172.21.15.61:0/3570135855 >> v1:172.21.15.61:6789/0 conn(0x7f13340adff0 legacy=0x7f133405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 -- 172.21.15.61:0/3570135855 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133405c4a0 con 0x7f13340af8e0 2022-07-20T03:17:40.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f133947f700 1 --2- 172.21.15.61:0/3570135855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340af8e0 0x7f13340b5a30 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f132c0032f0 tx=0x7f132c009290).ready entity=mon.0 client_cookie=a42a4dd79a127c60 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:40.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f13337fe700 1 -- 172.21.15.61:0/3570135855 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f132c00b8a0 con 0x7f13340af8e0 2022-07-20T03:17:40.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f13337fe700 1 -- 172.21.15.61:0/3570135855 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c00ba00 con 0x7f13340af8e0 2022-07-20T03:17:40.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.235+0000 7f13337fe700 1 -- 172.21.15.61:0/3570135855 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f132c00bb70 con 0x7f13340af8e0 2022-07-20T03:17:40.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- 172.21.15.61:0/3570135855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340af8e0 msgr2=0x7f13340b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 --2- 172.21.15.61:0/3570135855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340af8e0 0x7f13340b5a30 secure :-1 s=READY pgs=1505 cs=0 l=1 rev1=1 rx=0x7f132c0032f0 tx=0x7f132c009290).stop 2022-07-20T03:17:40.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- 172.21.15.61:0/3570135855 shutdown_connections 2022-07-20T03:17:40.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 --2- 172.21.15.61:0/3570135855 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340af8e0 0x7f13340b5a30 unknown :-1 s=CLOSED pgs=1505 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- 172.21.15.61:0/3570135855 >> 172.21.15.61:0/3570135855 conn(0x7f13340634e0 msgr2=0x7f13340b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:40.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- 172.21.15.61:0/3570135855 shutdown_connections 2022-07-20T03:17:40.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- 172.21.15.61:0/3570135855 wait complete. 2022-07-20T03:17:40.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 Processor -- start 2022-07-20T03:17:40.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- start start 2022-07-20T03:17:40.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13340adff0 0x7f13341565f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13340af8e0 0x7f1334110630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 0x7f1334066840 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334065e60 con 0x7f13340b5650 2022-07-20T03:17:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1334066100 con 0x7f13340af8e0 2022-07-20T03:17:40.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.239+0000 7f1339c80700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f13340663a0 con 0x7f13340adff0 2022-07-20T03:17:40.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 0x7f1334066840 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:40.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 0x7f1334066840 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51928/0 (socket says 172.21.15.61:51928) 2022-07-20T03:17:40.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 -- 172.21.15.61:0/1997270532 learned_addr learned my addr 172.21.15.61:0/1997270532 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:40.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1338c7e700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13340adff0 0x7f13341565f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:40.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1333fff700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13340af8e0 0x7f1334110630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 -- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13340adff0 msgr2=0x7f13341565f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13340adff0 0x7f13341565f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 -- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13340af8e0 msgr2=0x7f1334110630 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13340af8e0 0x7f1334110630 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 -- 172.21.15.61:0/1997270532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f132c009530 con 0x7f13340b5650 2022-07-20T03:17:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f133947f700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 0x7f1334066840 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f132c0032f0 tx=0x7f132c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f132c01f3f0 con 0x7f13340b5650 2022-07-20T03:17:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f132c01f550 con 0x7f13340b5650 2022-07-20T03:17:40.257 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1334111410 con 0x7f13340b5650 2022-07-20T03:17:40.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f132c01f8b0 con 0x7f13340b5650 2022-07-20T03:17:40.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13340651d0 con 0x7f13340b5650 2022-07-20T03:17:40.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1334066560 con 0x7f13340b5650 2022-07-20T03:17:40.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f132c026030 con 0x7f13340b5650 2022-07-20T03:17:40.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1331ffb700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1314053ad0 0x7f1314055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:40.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f132c06cd10 con 0x7f13340b5650 2022-07-20T03:17:40.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.243+0000 7f1338c7e700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1314053ad0 0x7f1314055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:40.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.247+0000 7f1338c7e700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1314053ad0 0x7f1314055f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f132800b000 tx=0x7f1328009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:40.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.251+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f132c043c70 con 0x7f13340b5650 2022-07-20T03:17:40.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.415+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f13340574e0 con 0x7f13340b5650 2022-07-20T03:17:40.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.415+0000 7f1331ffb700 1 -- 172.21.15.61:0/1997270532 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f132c024020 con 0x7f13340b5650 2022-07-20T03:17:40.423 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:40.424 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7397,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1660470104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1660470104},{"type":"v1","addr":"172.21.15.61:6833","nonce":1660470104}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:35.498516+0000","modified":"2022-07-20T03:17:36.503758+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/1280182602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1280182602},{"type":"v1","addr":"172.21.15.117:6835","nonce":1280182602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T03:17:40.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1314053ad0 msgr2=0x7f1314055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1314053ad0 0x7f1314055f90 secure :-1 s=READY pgs=723 cs=0 l=1 rev1=1 rx=0x7f132800b000 tx=0x7f1328009040).stop 2022-07-20T03:17:40.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 msgr2=0x7f1334066840 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:40.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 0x7f1334066840 secure :-1 s=READY pgs=1506 cs=0 l=1 rev1=1 rx=0x7f132c0032f0 tx=0x7f132c016000).stop 2022-07-20T03:17:40.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 shutdown_connections 2022-07-20T03:17:40.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1314053ad0 0x7f1314055f90 unknown :-1 s=CLOSED pgs=723 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f13340af8e0 0x7f1334110630 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f13340b5650 0x7f1334066840 unknown :-1 s=CLOSED pgs=1506 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 --2- 172.21.15.61:0/1997270532 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f13340adff0 0x7f13341565f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 >> 172.21.15.61:0/1997270532 conn(0x7f13340634e0 msgr2=0x7f1334146220 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 shutdown_connections 2022-07-20T03:17:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:40.419+0000 7f1339c80700 1 -- 172.21.15.61:0/1997270532 wait complete. 2022-07-20T03:17:40.432 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-20T03:17:40.635 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:17:35.498516+0000', 'modified': '2022-07-20T03:17:36.503758+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8665': {'gid': 8665, 'name': 'b', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/1280182602', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1280182602}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1280182602}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:17:40.636 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:17:40.637 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:17:40.753 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8665,"want_state":"up:active","state":"up:active","rank_uptime":5.2385288250000004,"mdsmap_epoch":83,"osdmap_epoch":149,"osdmap_epoch_barrier":149,"uptime":9.5772127289999993} 2022-07-20T03:17:40.753 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8665, "mdsmap_epoch": 83, "osdmap_epoch": 149, "osdmap_epoch_barrier": 149, "rank_uptime": 5.238528825, "state": "up:active", "uptime": 9.577212729, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:17:40.754 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:17:40.754 INFO:tasks.cephfs.fuse_mount:Client client.0 config is {} 2022-07-20T03:17:40.755 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.0 at ubuntu@smithi124.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.0... 2022-07-20T03:17:40.755 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:40.756 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:40.765 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:17:40.826 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:40.827 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:40.842 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:40.844 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:40.845 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:40.846 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:17:40.898 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:17:40.899 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.0 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:40.903 DEBUG:teuthology.orchestra.run.smithi124:> sudo modprobe fuse 2022-07-20T03:17:40.920 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:40.921 DEBUG:teuthology.orchestra.run.smithi124:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:2022-07-20T03:17:40.971+0000 7f9af1e6a080 -1 init, newargv = 0x5601dc6a42f0 newargc=9ceph-fuse[23652]: starting ceph client 2022-07-20T03:17:40.975 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr: 2022-07-20T03:17:40.985 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:40.987 INFO:teuthology.orchestra.run.smithi124.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:40.988 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:40.988 DEBUG:teuthology.orchestra.run.smithi124:> ls /sys/fs/fuse/connections 2022-07-20T03:17:40.990 INFO:tasks.cephfs.fuse_mount.ceph-fuse.0.smithi124.stderr:ceph-fuse[23652]: starting fuse 2022-07-20T03:17:40.995 INFO:teuthology.orchestra.run.smithi124.stdout:52 2022-07-20T03:17:40.996 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:17:40.996 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:40.997 DEBUG:teuthology.orchestra.run.smithi124:> sudo python3 -c ' 2022-07-20T03:17:40.997 DEBUG:teuthology.orchestra.run.smithi124:> import glob 2022-07-20T03:17:40.998 DEBUG:teuthology.orchestra.run.smithi124:> import re 2022-07-20T03:17:40.998 DEBUG:teuthology.orchestra.run.smithi124:> import os 2022-07-20T03:17:40.999 DEBUG:teuthology.orchestra.run.smithi124:> import subprocess 2022-07-20T03:17:40.999 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:40.999 DEBUG:teuthology.orchestra.run.smithi124:> def find_socket(client_name): 2022-07-20T03:17:41.000 DEBUG:teuthology.orchestra.run.smithi124:> asok_path = "/var/run/ceph/ceph-client.0.*.asok" 2022-07-20T03:17:41.000 DEBUG:teuthology.orchestra.run.smithi124:> files = glob.glob(asok_path) 2022-07-20T03:17:41.001 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:41.001 DEBUG:teuthology.orchestra.run.smithi124:> # Given a non-glob path, it better be there 2022-07-20T03:17:41.001 DEBUG:teuthology.orchestra.run.smithi124:> if "*" not in asok_path: 2022-07-20T03:17:41.002 DEBUG:teuthology.orchestra.run.smithi124:> assert(len(files) == 1) 2022-07-20T03:17:41.002 DEBUG:teuthology.orchestra.run.smithi124:> return files[0] 2022-07-20T03:17:41.003 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:41.004 DEBUG:teuthology.orchestra.run.smithi124:> for f in files: 2022-07-20T03:17:41.004 DEBUG:teuthology.orchestra.run.smithi124:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:17:41.005 DEBUG:teuthology.orchestra.run.smithi124:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:17:41.006 DEBUG:teuthology.orchestra.run.smithi124:> return f 2022-07-20T03:17:41.006 DEBUG:teuthology.orchestra.run.smithi124:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:17:41.007 DEBUG:teuthology.orchestra.run.smithi124:> 2022-07-20T03:17:41.007 DEBUG:teuthology.orchestra.run.smithi124:> print(find_socket("client.0")) 2022-07-20T03:17:41.008 DEBUG:teuthology.orchestra.run.smithi124:> ' 2022-07-20T03:17:41.086 INFO:teuthology.orchestra.run.smithi124.stdout:/var/run/ceph/ceph-client.0.23652.asok 2022-07-20T03:17:41.093 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.0.23652.asok 2022-07-20T03:17:41.093 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:41.094 DEBUG:teuthology.orchestra.run.smithi124:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.0.23652.asok status 2022-07-20T03:17:41.182 INFO:teuthology.orchestra.run.smithi124.stdout:{ 2022-07-20T03:17:41.183 INFO:teuthology.orchestra.run.smithi124.stdout: "metadata": { 2022-07-20T03:17:41.184 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:17:41.184 INFO:teuthology.orchestra.run.smithi124.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:17:41.185 INFO:teuthology.orchestra.run.smithi124.stdout: "entity_id": "0", 2022-07-20T03:17:41.185 INFO:teuthology.orchestra.run.smithi124.stdout: "hostname": "smithi124", 2022-07-20T03:17:41.185 INFO:teuthology.orchestra.run.smithi124.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.0", 2022-07-20T03:17:41.186 INFO:teuthology.orchestra.run.smithi124.stdout: "pid": "23652", 2022-07-20T03:17:41.186 INFO:teuthology.orchestra.run.smithi124.stdout: "root": "/" 2022-07-20T03:17:41.187 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:41.187 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_count": 0, 2022-07-20T03:17:41.188 INFO:teuthology.orchestra.run.smithi124.stdout: "dentry_pinned_count": 0, 2022-07-20T03:17:41.189 INFO:teuthology.orchestra.run.smithi124.stdout: "id": 8824, 2022-07-20T03:17:41.190 INFO:teuthology.orchestra.run.smithi124.stdout: "inst": { 2022-07-20T03:17:41.190 INFO:teuthology.orchestra.run.smithi124.stdout: "name": { 2022-07-20T03:17:41.191 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "client", 2022-07-20T03:17:41.191 INFO:teuthology.orchestra.run.smithi124.stdout: "num": 8824 2022-07-20T03:17:41.192 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:41.193 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:17:41.193 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:17:41.194 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:17:41.194 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2179352926 2022-07-20T03:17:41.194 INFO:teuthology.orchestra.run.smithi124.stdout: } 2022-07-20T03:17:41.195 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:41.195 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": { 2022-07-20T03:17:41.196 INFO:teuthology.orchestra.run.smithi124.stdout: "type": "v1", 2022-07-20T03:17:41.196 INFO:teuthology.orchestra.run.smithi124.stdout: "addr": "172.21.15.124:0", 2022-07-20T03:17:41.197 INFO:teuthology.orchestra.run.smithi124.stdout: "nonce": 2179352926 2022-07-20T03:17:41.197 INFO:teuthology.orchestra.run.smithi124.stdout: }, 2022-07-20T03:17:41.198 INFO:teuthology.orchestra.run.smithi124.stdout: "inst_str": "client.8824 172.21.15.124:0/2179352926", 2022-07-20T03:17:41.198 INFO:teuthology.orchestra.run.smithi124.stdout: "addr_str": "172.21.15.124:0/2179352926", 2022-07-20T03:17:41.199 INFO:teuthology.orchestra.run.smithi124.stdout: "inode_count": 1, 2022-07-20T03:17:41.199 INFO:teuthology.orchestra.run.smithi124.stdout: "mds_epoch": 83, 2022-07-20T03:17:41.200 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch": 149, 2022-07-20T03:17:41.200 INFO:teuthology.orchestra.run.smithi124.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:17:41.200 INFO:teuthology.orchestra.run.smithi124.stdout: "blacklisted": false, 2022-07-20T03:17:41.201 INFO:teuthology.orchestra.run.smithi124.stdout: "fs_name": "cephfs" 2022-07-20T03:17:41.201 INFO:teuthology.orchestra.run.smithi124.stdout:} 2022-07-20T03:17:41.203 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:41.204 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:41.205 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:41.215 INFO:teuthology.orchestra.run.smithi124.stdout:fuseblk 2022-07-20T03:17:41.215 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.0 2022-07-20T03:17:41.216 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:41.217 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:17:41.239 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T03:17:41.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 Processor -- start 2022-07-20T03:17:41.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- start start 2022-07-20T03:17:41.366 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f471c0b1e00 0x7f471c14a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0acef0 0x7f471c146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f471c05d480 con 0x7f471c05c3f0 2022-07-20T03:17:41.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f471c0b21e0 con 0x7f471c0b1e00 2022-07-20T03:17:41.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c05cf40 con 0x7f471c0acef0 2022-07-20T03:17:41.369 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f471c0b1e00 0x7f471c14a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f471c0b1e00 0x7f471c14a6d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51932/0 (socket says 172.21.15.61:51932) 2022-07-20T03:17:41.370 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 -- 172.21.15.61:0/340289842 learned_addr learned my addr 172.21.15.61:0/340289842 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:41.371 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4722789700 1 --2- 172.21.15.61:0/340289842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0acef0 0x7f471c146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 -- 172.21.15.61:0/340289842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0acef0 msgr2=0x7f471c146410 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:41.372 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 --2- 172.21.15.61:0/340289842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0acef0 0x7f471c146410 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 -- 172.21.15.61:0/340289842 >> v1:172.21.15.61:6789/0 conn(0x7f471c05c3f0 legacy=0x7f471c0ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 -- 172.21.15.61:0/340289842 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f471c05d1e0 con 0x7f471c0b1e00 2022-07-20T03:17:41.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4721f88700 1 --2- 172.21.15.61:0/340289842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c0b1e00 0x7f471c14a6d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f47140032f0 tx=0x7f4714009290).ready entity=mon.0 client_cookie=cb0d67628dcaec7 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4720f86700 1 -- 172.21.15.61:0/340289842 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471400b8a0 con 0x7f471c0b1e00 2022-07-20T03:17:41.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4720f86700 1 -- 172.21.15.61:0/340289842 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f471400ba00 con 0x7f471c0b1e00 2022-07-20T03:17:41.375 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f4720f86700 1 -- 172.21.15.61:0/340289842 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f471400bb70 con 0x7f471c0b1e00 2022-07-20T03:17:41.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- 172.21.15.61:0/340289842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c0b1e00 msgr2=0x7f471c14a6d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 --2- 172.21.15.61:0/340289842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c0b1e00 0x7f471c14a6d0 secure :-1 s=READY pgs=1509 cs=0 l=1 rev1=1 rx=0x7f47140032f0 tx=0x7f4714009290).stop 2022-07-20T03:17:41.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- 172.21.15.61:0/340289842 shutdown_connections 2022-07-20T03:17:41.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 --2- 172.21.15.61:0/340289842 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0acef0 0x7f471c146410 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 --2- 172.21.15.61:0/340289842 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c0b1e00 0x7f471c14a6d0 unknown :-1 s=CLOSED pgs=1509 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- 172.21.15.61:0/340289842 >> 172.21.15.61:0/340289842 conn(0x7f471c0611c0 msgr2=0x7f471c062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:41.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- 172.21.15.61:0/340289842 shutdown_connections 2022-07-20T03:17:41.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.347+0000 7f47241ec700 1 -- 172.21.15.61:0/340289842 wait complete. 2022-07-20T03:17:41.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 Processor -- start 2022-07-20T03:17:41.381 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 -- start start 2022-07-20T03:17:41.382 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f471c0acef0 0x7f471c0674e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0b1e00 0x7f471c063a80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 0x7f471c09e080 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c156bb0 con 0x7f471c063f70 2022-07-20T03:17:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f471c156e50 con 0x7f471c0acef0 2022-07-20T03:17:41.385 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f471c1570f0 con 0x7f471c0b1e00 2022-07-20T03:17:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 0x7f471c09e080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 0x7f471c09e080 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51942/0 (socket says 172.21.15.61:51942) 2022-07-20T03:17:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 -- 172.21.15.61:0/4039927340 learned_addr learned my addr 172.21.15.61:0/4039927340 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:41.386 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4721787700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0b1e00 0x7f471c063a80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4721f88700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f471c0acef0 0x7f471c0674e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 -- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0b1e00 msgr2=0x7f471c063a80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0b1e00 0x7f471c063a80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 -- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f471c0acef0 msgr2=0x7f471c0674e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f471c0acef0 0x7f471c0674e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 -- 172.21.15.61:0/4039927340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4714009530 con 0x7f471c063f70 2022-07-20T03:17:41.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f4722789700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 0x7f471c09e080 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f471800a000 tx=0x7f4718008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f47180106e0 con 0x7f471c063f70 2022-07-20T03:17:41.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4718010840 con 0x7f471c063f70 2022-07-20T03:17:41.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f471c14a120 con 0x7f471c063f70 2022-07-20T03:17:41.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f471c1566a0 con 0x7f471c063f70 2022-07-20T03:17:41.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.351+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4718010ba0 con 0x7f471c063f70 2022-07-20T03:17:41.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.355+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f471c1572b0 con 0x7f471c063f70 2022-07-20T03:17:41.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.355+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f471801a030 con 0x7f471c063f70 2022-07-20T03:17:41.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.355+0000 7f470affd700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f46fc053af0 0x7f46fc055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.355+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f471806bb30 con 0x7f471c063f70 2022-07-20T03:17:41.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.355+0000 7f4721f88700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f46fc053af0 0x7f46fc055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.359+0000 7f4721f88700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f46fc053af0 0x7f46fc055fb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f471401b000 tx=0x7f4714016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.363+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4718042a30 con 0x7f471c063f70 2022-07-20T03:17:41.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.491+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f471c0b04b0 con 0x7f471c063f70 2022-07-20T03:17:41.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.491+0000 7f470affd700 1 -- 172.21.15.61:0/4039927340 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v149) v1 ==== 74+0+16100 (secure 0 0 0) 0x7f4718046050 con 0x7f471c063f70 2022-07-20T03:17:41.497 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:41.497 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":149,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:36.502720+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}},{"pool":29,"pool_name":"cephfs_metadata","create_time":"2022-07-20T03:17:34.158104+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"148","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_autoscale_bias":4,"pg_num_min":16,"recovery_priority":5},"application_metadata":{"cephfs":{"metadata":"cephfs"}}},{"pool":30,"pool_name":"cephfs_data","create_time":"2022-07-20T03:17:34.713017+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"149","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"cephfs":{"data":"cephfs"}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T03:17:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f46fc053af0 msgr2=0x7f46fc055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f46fc053af0 0x7f46fc055fb0 secure :-1 s=READY pgs=724 cs=0 l=1 rev1=1 rx=0x7f471401b000 tx=0x7f4714016000).stop 2022-07-20T03:17:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 msgr2=0x7f471c09e080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.499 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 0x7f471c09e080 secure :-1 s=READY pgs=1510 cs=0 l=1 rev1=1 rx=0x7f471800a000 tx=0x7f4718008040).stop 2022-07-20T03:17:41.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 shutdown_connections 2022-07-20T03:17:41.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f46fc053af0 0x7f46fc055fb0 unknown :-1 s=CLOSED pgs=724 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f471c063f70 0x7f471c09e080 unknown :-1 s=CLOSED pgs=1510 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f471c0b1e00 0x7f471c063a80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 --2- 172.21.15.61:0/4039927340 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f471c0acef0 0x7f471c0674e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 >> 172.21.15.61:0/4039927340 conn(0x7f471c05c3f0 msgr2=0x7f471c0afa30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:41.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 shutdown_connections 2022-07-20T03:17:41.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.495+0000 7f47241ec700 1 -- 172.21.15.61:0/4039927340 wait complete. 2022-07-20T03:17:41.515 INFO:tasks.cephfs.filesystem.ceph_manager:[{'pool': 1, 'pool_name': 'device_health_metrics', 'create_time': '2022-07-20T03:06:54.040896+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 1, 'pg_placement_num': 1, 'pg_placement_num_target': 1, 'pg_num_target': 1, 'pg_num_pending': 1, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '15', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_num_min': 1}, 'application_metadata': {'mgr_devicehealth': {}}}, {'pool': 2, 'pool_name': 'rbd', 'create_time': '2022-07-20T03:07:05.508608+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '18', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'rbd': {}}}, {'pool': 29, 'pool_name': 'cephfs_metadata', 'create_time': '2022-07-20T03:17:34.158104+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '148', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {'pg_autoscale_bias': 4, 'pg_num_min': 16, 'recovery_priority': 5}, 'application_metadata': {'cephfs': {'metadata': 'cephfs'}}}, {'pool': 30, 'pool_name': 'cephfs_data', 'create_time': '2022-07-20T03:17:34.713017+0000', 'flags': 1, 'flags_names': 'hashpspool', 'type': 1, 'size': 2, 'min_size': 1, 'crush_rule': 0, 'object_hash': 2, 'pg_autoscale_mode': 'off', 'pg_num': 8, 'pg_placement_num': 8, 'pg_placement_num_target': 8, 'pg_num_target': 8, 'pg_num_pending': 8, 'last_pg_merge_meta': {'source_pgid': '0.0', 'ready_epoch': 0, 'last_epoch_started': 0, 'last_epoch_clean': 0, 'source_version': "0'0", 'target_version': "0'0"}, 'last_change': '149', 'last_force_op_resend': '0', 'last_force_op_resend_prenautilus': '0', 'last_force_op_resend_preluminous': '0', 'auid': 0, 'snap_mode': 'selfmanaged', 'snap_seq': 0, 'snap_epoch': 0, 'pool_snaps': [], 'removed_snaps': '[]', 'quota_max_bytes': 0, 'quota_max_objects': 0, 'tiers': [], 'tier_of': -1, 'read_tier': -1, 'write_tier': -1, 'cache_mode': 'none', 'target_max_bytes': 0, 'target_max_objects': 0, 'cache_target_dirty_ratio_micro': 400000, 'cache_target_dirty_high_ratio_micro': 600000, 'cache_target_full_ratio_micro': 800000, 'cache_min_flush_age': 0, 'cache_min_evict_age': 0, 'erasure_code_profile': '', 'hit_set_params': {'type': 'none'}, 'hit_set_period': 0, 'hit_set_count': 0, 'use_gmt_hitset': True, 'min_read_recency_for_promote': 0, 'min_write_recency_for_promote': 0, 'hit_set_grade_decay_rate': 0, 'hit_set_search_last_n': 0, 'grade_table': [], 'stripe_width': 0, 'expected_num_objects': 0, 'fast_read': False, 'options': {}, 'application_metadata': {'cephfs': {'data': 'cephfs'}}}] 2022-07-20T03:17:41.516 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get device_health_metrics pg_num 2022-07-20T03:17:41.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.623+0000 7f30ab078700 1 Processor -- start 2022-07-20T03:17:41.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.623+0000 7f30ab078700 1 -- start start 2022-07-20T03:17:41.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f30a40b59f0 0x7f30a405c0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a405d2f0 con 0x7f30a40af940 2022-07-20T03:17:41.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f30a40b5dd0 con 0x7f30a40addd0 2022-07-20T03:17:41.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f30a405ccd0 con 0x7f30a40b59f0 2022-07-20T03:17:41.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30aa076700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f30a40b59f0 0x7f30a405c0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9875700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f30a40af940 0x7f30a405c5c0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48548/0 (socket says 172.21.15.61:48548) 2022-07-20T03:17:41.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9875700 1 -- 172.21.15.61:0/4009792836 learned_addr learned my addr 172.21.15.61:0/4009792836 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:41.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9074700 1 -- 172.21.15.61:0/4009792836 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 274638350 0 0) 0x7f30a405d2f0 con 0x7f30a40af940 2022-07-20T03:17:41.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9074700 1 -- 172.21.15.61:0/4009792836 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f308c003260 con 0x7f30a40af940 2022-07-20T03:17:41.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9074700 1 -- 172.21.15.61:0/4009792836 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 977304448 0 0) 0x7f30a40b5dd0 con 0x7f30a40addd0 2022-07-20T03:17:41.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30aa076700 1 -- 172.21.15.61:0/4009792836 >> v1:172.21.15.117:6789/0 conn(0x7f30a40addd0 legacy=0x7f30a4142d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30aa076700 1 -- 172.21.15.61:0/4009792836 >> v1:172.21.15.61:6789/0 conn(0x7f30a40af940 legacy=0x7f30a405c5c0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30aa076700 1 -- 172.21.15.61:0/4009792836 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a41460b0 con 0x7f30a40b59f0 2022-07-20T03:17:41.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30aa076700 1 --2- 172.21.15.61:0/4009792836 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a405c0d0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f30a0002f70 tx=0x7f30a0006df0).ready entity=mon.0 client_cookie=cd3119b7705db882 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9074700 1 -- 172.21.15.61:0/4009792836 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30a000d610 con 0x7f30a40b59f0 2022-07-20T03:17:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9074700 1 -- 172.21.15.61:0/4009792836 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f30a0000e90 con 0x7f30a40b59f0 2022-07-20T03:17:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30a9074700 1 -- 172.21.15.61:0/4009792836 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f30a000d960 con 0x7f30a40b59f0 2022-07-20T03:17:41.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- 172.21.15.61:0/4009792836 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 msgr2=0x7f30a405c0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 --2- 172.21.15.61:0/4009792836 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a405c0d0 secure :-1 s=READY pgs=1512 cs=0 l=1 rev1=1 rx=0x7f30a0002f70 tx=0x7f30a0006df0).stop 2022-07-20T03:17:41.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- 172.21.15.61:0/4009792836 shutdown_connections 2022-07-20T03:17:41.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 --2- 172.21.15.61:0/4009792836 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a405c0d0 unknown :-1 s=CLOSED pgs=1512 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- 172.21.15.61:0/4009792836 >> 172.21.15.61:0/4009792836 conn(0x7f30a4063480 msgr2=0x7f30a40b4fa0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:41.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- 172.21.15.61:0/4009792836 shutdown_connections 2022-07-20T03:17:41.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.627+0000 7f30ab078700 1 -- 172.21.15.61:0/4009792836 wait complete. 2022-07-20T03:17:41.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 Processor -- start 2022-07-20T03:17:41.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 -- start start 2022-07-20T03:17:41.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30a40addd0 0x7f30a4151bc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30a40af940 0x7f30a413fba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a413e8a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a41520b0 con 0x7f30a40b59f0 2022-07-20T03:17:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30a40645c0 con 0x7f30a40af940 2022-07-20T03:17:41.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30ab078700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f30a4064860 con 0x7f30a40addd0 2022-07-20T03:17:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30aa877700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a413e8a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30aa877700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a413e8a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51956/0 (socket says 172.21.15.61:51956) 2022-07-20T03:17:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30aa877700 1 -- 172.21.15.61:0/3949906964 learned_addr learned my addr 172.21.15.61:0/3949906964 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:41.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30aa076700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30a40addd0 0x7f30a4151bc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.631+0000 7f30a9875700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30a40af940 0x7f30a413fba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa877700 1 -- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30a40addd0 msgr2=0x7f30a4151bc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa877700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30a40addd0 0x7f30a4151bc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa877700 1 -- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30a40af940 msgr2=0x7f30a413fba0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa877700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30a40af940 0x7f30a413fba0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa877700 1 -- 172.21.15.61:0/3949906964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30a000b180 con 0x7f30a40b59f0 2022-07-20T03:17:41.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30a9875700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30a40af940 0x7f30a413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa076700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30a40addd0 0x7f30a4151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa877700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a413e8a0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f309c001250 tx=0x7f309c007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f309c00e4a0 con 0x7f30a40b59f0 2022-07-20T03:17:41.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f309c00e600 con 0x7f30a40b59f0 2022-07-20T03:17:41.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30a41551d0 con 0x7f30a40b59f0 2022-07-20T03:17:41.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f309c00e9a0 con 0x7f30a40b59f0 2022-07-20T03:17:41.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30a41435e0 con 0x7f30a40b59f0 2022-07-20T03:17:41.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30a40640c0 con 0x7f30a40b59f0 2022-07-20T03:17:41.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f309c019030 con 0x7f30a40b59f0 2022-07-20T03:17:41.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30937fe700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3084053920 0x7f3084055de0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f309c06b5c0 con 0x7f30a40b59f0 2022-07-20T03:17:41.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.635+0000 7f30aa076700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3084053920 0x7f3084055de0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.639+0000 7f30aa076700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3084053920 0x7f3084055de0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f30a0018000 tx=0x7f30a0014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.643+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f309c0425d0 con 0x7f30a40b59f0 2022-07-20T03:17:41.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.771+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"} v 0) v1 -- 0x7f30a4155360 con 0x7f30a40b59f0 2022-07-20T03:17:41.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.771+0000 7f30937fe700 1 -- 172.21.15.61:0/3949906964 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "device_health_metrics", "var": "pg_num"}]=0 v149) v1 ==== 110+0+10 (secure 0 0 0) 0x7f309c045bf0 con 0x7f30a40b59f0 2022-07-20T03:17:41.777 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 1 2022-07-20T03:17:41.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3084053920 msgr2=0x7f3084055de0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3084053920 0x7f3084055de0 secure :-1 s=READY pgs=725 cs=0 l=1 rev1=1 rx=0x7f30a0018000 tx=0x7f30a0014000).stop 2022-07-20T03:17:41.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 msgr2=0x7f30a413e8a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a413e8a0 secure :-1 s=READY pgs=1513 cs=0 l=1 rev1=1 rx=0x7f309c001250 tx=0x7f309c007dc0).stop 2022-07-20T03:17:41.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 shutdown_connections 2022-07-20T03:17:41.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f3084053920 0x7f3084055de0 unknown :-1 s=CLOSED pgs=725 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f30a40af940 0x7f30a413fba0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f30a40b59f0 0x7f30a413e8a0 unknown :-1 s=CLOSED pgs=1513 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 --2- 172.21.15.61:0/3949906964 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f30a40addd0 0x7f30a4151bc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 >> 172.21.15.61:0/3949906964 conn(0x7f30a4063480 msgr2=0x7f30a41488a0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:41.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 shutdown_connections 2022-07-20T03:17:41.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.775+0000 7f30ab078700 1 -- 172.21.15.61:0/3949906964 wait complete. 2022-07-20T03:17:41.796 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get rbd pg_num 2022-07-20T03:17:41.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.907+0000 7fd53ca16700 1 Processor -- start 2022-07-20T03:17:41.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.907+0000 7fd53ca16700 1 -- start start 2022-07-20T03:17:41.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd5380b4440 0x7fd538143590 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd538145f90 0x7fd53813fa00 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd5380b6c60 con 0x7fd5380b6880 2022-07-20T03:17:41.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fd5381429c0 con 0x7fd5380b4440 2022-07-20T03:17:41.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd538142bb0 con 0x7fd538145f90 2022-07-20T03:17:41.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd5380b4440 0x7fd538143590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fd5380b4440 0x7fd538143590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51964/0 (socket says 172.21.15.61:51964) 2022-07-20T03:17:41.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 -- 172.21.15.61:0/3675342949 learned_addr learned my addr 172.21.15.61:0/3675342949 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:41.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53759e700 1 --2- 172.21.15.61:0/3675342949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd538145f90 0x7fd53813fa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 -- 172.21.15.61:0/3675342949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd538145f90 msgr2=0x7fd53813fa00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:41.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 --2- 172.21.15.61:0/3675342949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd538145f90 0x7fd53813fa00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 -- 172.21.15.61:0/3675342949 >> v1:172.21.15.117:6789/0 conn(0x7fd5380b6880 legacy=0x7fd5380b3d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 -- 172.21.15.61:0/3675342949 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd538142e50 con 0x7fd5380b4440 2022-07-20T03:17:41.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53659c700 1 --2- 172.21.15.61:0/3675342949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b4440 0x7fd538143590 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7fd52c002f70 tx=0x7fd52c006eb0).ready entity=mon.0 client_cookie=a9da4c2fce190fa5 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd535d9b700 1 -- 172.21.15.61:0/3675342949 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd52c00d570 con 0x7fd5380b4440 2022-07-20T03:17:41.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd535d9b700 1 -- 172.21.15.61:0/3675342949 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd52c000e90 con 0x7fd5380b4440 2022-07-20T03:17:41.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd535d9b700 1 -- 172.21.15.61:0/3675342949 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd52c00d900 con 0x7fd5380b4440 2022-07-20T03:17:41.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 -- 172.21.15.61:0/3675342949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b4440 msgr2=0x7fd538143590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 --2- 172.21.15.61:0/3675342949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b4440 0x7fd538143590 secure :-1 s=READY pgs=1514 cs=0 l=1 rev1=1 rx=0x7fd52c002f70 tx=0x7fd52c006eb0).stop 2022-07-20T03:17:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 -- 172.21.15.61:0/3675342949 shutdown_connections 2022-07-20T03:17:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 --2- 172.21.15.61:0/3675342949 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd538145f90 0x7fd53813fa00 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 --2- 172.21.15.61:0/3675342949 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd5380b4440 0x7fd538143590 unknown :-1 s=CLOSED pgs=1514 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.911+0000 7fd53ca16700 1 -- 172.21.15.61:0/3675342949 >> 172.21.15.61:0/3675342949 conn(0x7fd5380b9e60 msgr2=0x7fd5380b5e30 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:41.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 -- 172.21.15.61:0/3675342949 shutdown_connections 2022-07-20T03:17:41.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 -- 172.21.15.61:0/3675342949 wait complete. 2022-07-20T03:17:41.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 Processor -- start 2022-07-20T03:17:41.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 -- start start 2022-07-20T03:17:41.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 0x7fd5380b82f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380b6880 0x7fd5381513a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 0x7fd538151890 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd538115030 con 0x7fd538145f90 2022-07-20T03:17:41.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380ae440 con 0x7fd5380b4440 2022-07-20T03:17:41.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53ca16700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd5380b87e0 con 0x7fd5380b6880 2022-07-20T03:17:41.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 0x7fd538151890 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd536d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 0x7fd5380b82f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53659c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380b6880 0x7fd5381513a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd536d9d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 0x7fd5380b82f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:33414/0 (socket says 172.21.15.61:33414) 2022-07-20T03:17:41.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd536d9d700 1 -- 172.21.15.61:0/191393866 learned_addr learned my addr 172.21.15.61:0/191393866 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:41.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.915+0000 7fd53759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 0x7fd538151890 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51970/0 (socket says 172.21.15.61:51970) 2022-07-20T03:17:41.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53759e700 1 -- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380b6880 msgr2=0x7fd5381513a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53759e700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380b6880 0x7fd5381513a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53759e700 1 -- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 msgr2=0x7fd5380b82f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:41.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53759e700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 0x7fd5380b82f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:41.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53759e700 1 -- 172.21.15.61:0/191393866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd52c00b1e0 con 0x7fd538145f90 2022-07-20T03:17:41.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd536d9d700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 0x7fd5380b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:41.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53759e700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 0x7fd538151890 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7fd53000c000 tx=0x7fd5300091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd530012600 con 0x7fd538145f90 2022-07-20T03:17:41.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd530012760 con 0x7fd538145f90 2022-07-20T03:17:41.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd530012ac0 con 0x7fd538145f90 2022-07-20T03:17:41.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd538151d80 con 0x7fd538145f90 2022-07-20T03:17:41.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd5380adfa0 con 0x7fd538145f90 2022-07-20T03:17:41.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd5380ae5b0 con 0x7fd538145f90 2022-07-20T03:17:41.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd530022070 con 0x7fd538145f90 2022-07-20T03:17:41.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd527fff700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd5140539b0 0x7fd514055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:41.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.919+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fd53006c410 con 0x7fd538145f90 2022-07-20T03:17:41.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.923+0000 7fd536d9d700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd5140539b0 0x7fd514055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:41.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.923+0000 7fd536d9d700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd5140539b0 0x7fd514055e70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fd52800d000 tx=0x7fd528009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:41.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:41.927+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd5380ae5b0 con 0x7fd538145f90 2022-07-20T03:17:42.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.055+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"} v 0) v1 -- 0x7fd5380ae5b0 con 0x7fd538145f90 2022-07-20T03:17:42.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.055+0000 7fd527fff700 1 -- 172.21.15.61:0/191393866 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "rbd", "var": "pg_num"}]=0 v149) v1 ==== 92+0+10 (secure 0 0 0) 0x7fd5380ae5b0 con 0x7fd538145f90 2022-07-20T03:17:42.059 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:42.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd5140539b0 msgr2=0x7fd514055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd5140539b0 0x7fd514055e70 secure :-1 s=READY pgs=726 cs=0 l=1 rev1=1 rx=0x7fd52800d000 tx=0x7fd528009040).stop 2022-07-20T03:17:42.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 msgr2=0x7fd538151890 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 0x7fd538151890 secure :-1 s=READY pgs=1515 cs=0 l=1 rev1=1 rx=0x7fd53000c000 tx=0x7fd5300091d0).stop 2022-07-20T03:17:42.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 shutdown_connections 2022-07-20T03:17:42.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd5140539b0 0x7fd514055e70 unknown :-1 s=CLOSED pgs=726 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd538145f90 0x7fd538151890 unknown :-1 s=CLOSED pgs=1515 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd5380b6880 0x7fd5381513a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 --2- 172.21.15.61:0/191393866 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd5380b4440 0x7fd5380b82f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 >> 172.21.15.61:0/191393866 conn(0x7fd5380b9e60 msgr2=0x7fd5380b5410 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 shutdown_connections 2022-07-20T03:17:42.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.059+0000 7fd53ca16700 1 -- 172.21.15.61:0/191393866 wait complete. 2022-07-20T03:17:42.077 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_metadata pg_num 2022-07-20T03:17:42.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 Processor -- start 2022-07-20T03:17:42.205 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- start start 2022-07-20T03:17:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601468f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10600b3cf0 0x7f1060142a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1060142f40 0x7f106005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.206 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10600ad5b0 con 0x7f10600b3cf0 2022-07-20T03:17:42.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f10600ba180 con 0x7f1060142f40 2022-07-20T03:17:42.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f10601461b0 con 0x7f10600b9da0 2022-07-20T03:17:42.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10600b3cf0 0x7f1060142a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.207 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f10600b3cf0 0x7f1060142a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51978/0 (socket says 172.21.15.61:51978) 2022-07-20T03:17:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 -- 172.21.15.61:0/1711417692 learned_addr learned my addr 172.21.15.61:0/1711417692 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105e59c700 1 --2- 172.21.15.61:0/1711417692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601468f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105ed9d700 1 --2- 172.21.15.61:0/1711417692 >> v2:172.21.15.117:3300/0 conn(0x7f1060142f40 0x7f106005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.208 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 -- 172.21.15.61:0/1711417692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 msgr2=0x7f10601468f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 --2- 172.21.15.61:0/1711417692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601468f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 -- 172.21.15.61:0/1711417692 >> v2:172.21.15.117:3300/0 conn(0x7f1060142f40 msgr2=0x7f106005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 --2- 172.21.15.61:0/1711417692 >> v2:172.21.15.117:3300/0 conn(0x7f1060142f40 0x7f106005bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 -- 172.21.15.61:0/1711417692 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1060146450 con 0x7f10600b3cf0 2022-07-20T03:17:42.209 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105ed9d700 1 --2- 172.21.15.61:0/1711417692 >> v2:172.21.15.117:3300/0 conn(0x7f1060142f40 0x7f106005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105dd9b700 1 --2- 172.21.15.61:0/1711417692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10600b3cf0 0x7f1060142a50 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f1050002f70 tx=0x7f1050006eb0).ready entity=mon.0 client_cookie=b6f0a322050c56ab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105d59a700 1 -- 172.21.15.61:0/1711417692 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f105000d570 con 0x7f10600b3cf0 2022-07-20T03:17:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105d59a700 1 -- 172.21.15.61:0/1711417692 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1050000e90 con 0x7f10600b3cf0 2022-07-20T03:17:42.210 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105d59a700 1 -- 172.21.15.61:0/1711417692 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f105000d900 con 0x7f10600b3cf0 2022-07-20T03:17:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- 172.21.15.61:0/1711417692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10600b3cf0 msgr2=0x7f1060142a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- 172.21.15.61:0/1711417692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10600b3cf0 0x7f1060142a50 secure :-1 s=READY pgs=1516 cs=0 l=1 rev1=1 rx=0x7f1050002f70 tx=0x7f1050006eb0).stop 2022-07-20T03:17:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- 172.21.15.61:0/1711417692 shutdown_connections 2022-07-20T03:17:42.211 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- 172.21.15.61:0/1711417692 >> v2:172.21.15.117:3300/0 conn(0x7f1060142f40 0x7f106005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- 172.21.15.61:0/1711417692 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f10600b3cf0 0x7f1060142a50 unknown :-1 s=CLOSED pgs=1516 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 --2- 172.21.15.61:0/1711417692 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601468f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- 172.21.15.61:0/1711417692 >> 172.21.15.61:0/1711417692 conn(0x7f10600678c0 msgr2=0x7f10600b9350 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- 172.21.15.61:0/1711417692 shutdown_connections 2022-07-20T03:17:42.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.187+0000 7f105f59e700 1 -- 172.21.15.61:0/1711417692 wait complete. 2022-07-20T03:17:42.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 Processor -- start 2022-07-20T03:17:42.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 -- start start 2022-07-20T03:17:42.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10600b3cf0 0x7f10600b5c70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601456c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1060142f40 0x7f10600b2910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10600b62a0 con 0x7f1060142f40 2022-07-20T03:17:42.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f10601541f0 con 0x7f10600b3cf0 2022-07-20T03:17:42.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1060155040 con 0x7f10600b9da0 2022-07-20T03:17:42.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1060142f40 0x7f10600b2910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601456c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105ed9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1060142f40 0x7f10600b2910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51984/0 (socket says 172.21.15.61:51984) 2022-07-20T03:17:42.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105ed9d700 1 -- 172.21.15.61:0/2934341803 learned_addr learned my addr 172.21.15.61:0/2934341803 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:42.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105e59c700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10600b3cf0 0x7f10600b5c70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 -- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10600b3cf0 msgr2=0x7f10600b5c70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10600b3cf0 0x7f10600b5c70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 -- 172.21.15.61:0/2934341803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1060142f40 msgr2=0x7f10600b2910 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1060142f40 0x7f10600b2910 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 -- 172.21.15.61:0/2934341803 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f105000b1e0 con 0x7f10600b9da0 2022-07-20T03:17:42.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105dd9b700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601456c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1050002f70 tx=0x7f1050015000).ready entity=mon.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1050011760 con 0x7f10600b9da0 2022-07-20T03:17:42.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f10500118c0 con 0x7f10600b9da0 2022-07-20T03:17:42.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f10601552c0 con 0x7f10600b9da0 2022-07-20T03:17:42.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1050011c20 con 0x7f10600b9da0 2022-07-20T03:17:42.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.191+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1060155650 con 0x7f10600b9da0 2022-07-20T03:17:42.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.195+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f10600574e0 con 0x7f10600b9da0 2022-07-20T03:17:42.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.195+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f105001a040 con 0x7f10600b9da0 2022-07-20T03:17:42.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.195+0000 7f104f7fe700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f103c0539d0 0x7f103c055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.195+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f105006cf50 con 0x7f10600b9da0 2022-07-20T03:17:42.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.195+0000 7f105e59c700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f103c0539d0 0x7f103c055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.195+0000 7f105e59c700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f103c0539d0 0x7f103c055e90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f105400c000 tx=0x7f1054008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.203+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1050043eb0 con 0x7f10600b9da0 2022-07-20T03:17:42.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.327+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"} v 0) v1 -- 0x7f10600699b0 con 0x7f10600b9da0 2022-07-20T03:17:42.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f104f7fe700 1 -- 172.21.15.61:0/2934341803 <== mon.2 v2:172.21.15.117:3301/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_metadata", "var": "pg_num"}]=0 v149) v1 ==== 104+0+10 (secure 0 0 0) 0x7f10500474d0 con 0x7f10600b9da0 2022-07-20T03:17:42.336 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:42.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f103c0539d0 msgr2=0x7f103c055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f103c0539d0 0x7f103c055e90 secure :-1 s=READY pgs=727 cs=0 l=1 rev1=1 rx=0x7f105400c000 tx=0x7f1054008040).stop 2022-07-20T03:17:42.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 msgr2=0x7f10601456c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601456c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1050002f70 tx=0x7f1050015000).stop 2022-07-20T03:17:42.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 shutdown_connections 2022-07-20T03:17:42.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f103c0539d0 0x7f103c055e90 unknown :-1 s=CLOSED pgs=727 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1060142f40 0x7f10600b2910 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f10600b9da0 0x7f10601456c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 --2- 172.21.15.61:0/2934341803 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f10600b3cf0 0x7f10600b5c70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 >> 172.21.15.61:0/2934341803 conn(0x7f10600678c0 msgr2=0x7f10601478d0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 shutdown_connections 2022-07-20T03:17:42.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.331+0000 7f105f59e700 1 -- 172.21.15.61:0/2934341803 wait complete. 2022-07-20T03:17:42.348 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool get cephfs_data pg_num 2022-07-20T03:17:42.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 Processor -- start 2022-07-20T03:17:42.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 -- start start 2022-07-20T03:17:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17c00ac280 0x7f17c013eb70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f17c00afe50 0x7f17c00b0280 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f17c00b1340 con 0x7f17c005c960 2022-07-20T03:17:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c005cd40 con 0x7f17c00ac280 2022-07-20T03:17:42.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c6245700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f17c00aba10 con 0x7f17c00afe50 2022-07-20T03:17:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17c00ac280 0x7f17c013eb70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f17c00ac280 0x7f17c013eb70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51990/0 (socket says 172.21.15.61:51990) 2022-07-20T03:17:42.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 -- 172.21.15.61:0/598303311 learned_addr learned my addr 172.21.15.61:0/598303311 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c5a44700 1 --2- 172.21.15.61:0/598303311 >> v2:172.21.15.117:3300/0 conn(0x7f17c00afe50 0x7f17c00b0280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 -- 172.21.15.61:0/598303311 >> v2:172.21.15.117:3300/0 conn(0x7f17c00afe50 msgr2=0x7f17c00b0280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 --2- 172.21.15.61:0/598303311 >> v2:172.21.15.117:3300/0 conn(0x7f17c00afe50 0x7f17c00b0280 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 -- 172.21.15.61:0/598303311 >> v1:172.21.15.117:6789/0 conn(0x7f17c005c960 legacy=0x7f17c013e460 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 -- 172.21.15.61:0/598303311 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17c00abcb0 con 0x7f17c00ac280 2022-07-20T03:17:42.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17c4a42700 1 --2- 172.21.15.61:0/598303311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00ac280 0x7f17c013eb70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7f17b0002f70 tx=0x7f17b0006f40).ready entity=mon.0 client_cookie=99d7023b7e4d810b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17affff700 1 -- 172.21.15.61:0/598303311 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17b000c880 con 0x7f17c00ac280 2022-07-20T03:17:42.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17affff700 1 -- 172.21.15.61:0/598303311 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b0000e90 con 0x7f17c00ac280 2022-07-20T03:17:42.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.451+0000 7f17affff700 1 -- 172.21.15.61:0/598303311 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17b000cc10 con 0x7f17c00ac280 2022-07-20T03:17:42.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 -- 172.21.15.61:0/598303311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00ac280 msgr2=0x7f17c013eb70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 --2- 172.21.15.61:0/598303311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00ac280 0x7f17c013eb70 secure :-1 s=READY pgs=1517 cs=0 l=1 rev1=1 rx=0x7f17b0002f70 tx=0x7f17b0006f40).stop 2022-07-20T03:17:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 -- 172.21.15.61:0/598303311 shutdown_connections 2022-07-20T03:17:42.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 --2- 172.21.15.61:0/598303311 >> v2:172.21.15.117:3300/0 conn(0x7f17c00afe50 0x7f17c00b0280 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 --2- 172.21.15.61:0/598303311 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00ac280 0x7f17c013eb70 unknown :-1 s=CLOSED pgs=1517 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 -- 172.21.15.61:0/598303311 >> 172.21.15.61:0/598303311 conn(0x7f17c00b53d0 msgr2=0x7f17c00b57e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 -- 172.21.15.61:0/598303311 shutdown_connections 2022-07-20T03:17:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 -- 172.21.15.61:0/598303311 wait complete. 2022-07-20T03:17:42.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 Processor -- start 2022-07-20T03:17:42.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.455+0000 7f17c6245700 1 -- start start 2022-07-20T03:17:42.482 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c005c960 0x7f17c0155fe0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00ac280 0x7f17c013e160 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.483 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 0x7f17c01565a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c013bd80 con 0x7f17c00afe50 2022-07-20T03:17:42.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0143480 con 0x7f17c005c960 2022-07-20T03:17:42.484 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f17c0142f40 con 0x7f17c00ac280 2022-07-20T03:17:42.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 0x7f17c01565a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 0x7f17c01565a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:51998/0 (socket says 172.21.15.61:51998) 2022-07-20T03:17:42.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 -- 172.21.15.61:0/693432881 learned_addr learned my addr 172.21.15.61:0/693432881 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:42.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c4a42700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00ac280 0x7f17c013e160 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5243700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c005c960 0x7f17c0155fe0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 -- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00ac280 msgr2=0x7f17c013e160 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00ac280 0x7f17c013e160 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 -- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c005c960 msgr2=0x7f17c0155fe0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c005c960 0x7f17c0155fe0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 -- 172.21.15.61:0/693432881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17b000a360 con 0x7f17c00afe50 2022-07-20T03:17:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c5a44700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 0x7f17c01565a0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f17c0159630 tx=0x7f17b80091d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17b800c640 con 0x7f17c00afe50 2022-07-20T03:17:42.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f17b800c7a0 con 0x7f17c00afe50 2022-07-20T03:17:42.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17c01431e0 con 0x7f17c00afe50 2022-07-20T03:17:42.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f17b800cb00 con 0x7f17c00afe50 2022-07-20T03:17:42.488 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.459+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17c00aab90 con 0x7f17c00afe50 2022-07-20T03:17:42.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.463+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17c00b0fc0 con 0x7f17c00afe50 2022-07-20T03:17:42.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.463+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f17b800cc60 con 0x7f17c00afe50 2022-07-20T03:17:42.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.463+0000 7f17ae7fc700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17a0053a60 0x7f17a0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.489 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.463+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f17b806c060 con 0x7f17c00afe50 2022-07-20T03:17:42.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.463+0000 7f17c5243700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17a0053a60 0x7f17a0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.463+0000 7f17c5243700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17a0053a60 0x7f17a0055f20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f17bc00d000 tx=0x7f17bc007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.471+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f17b8042fc0 con 0x7f17c00afe50 2022-07-20T03:17:42.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"} v 0) v1 -- 0x7f17c00574e0 con 0x7f17c00afe50 2022-07-20T03:17:42.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17ae7fc700 1 -- 172.21.15.61:0/693432881 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool get", "pool": "cephfs_data", "var": "pg_num"}]=0 v149) v1 ==== 100+0+10 (secure 0 0 0) 0x7f17b80465e0 con 0x7f17c00afe50 2022-07-20T03:17:42.605 INFO:teuthology.orchestra.run.smithi061.stdout:pg_num: 8 2022-07-20T03:17:42.606 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17a0053a60 msgr2=0x7f17a0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17a0053a60 0x7f17a0055f20 secure :-1 s=READY pgs=728 cs=0 l=1 rev1=1 rx=0x7f17bc00d000 tx=0x7f17bc007b90).stop 2022-07-20T03:17:42.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 msgr2=0x7f17c01565a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 0x7f17c01565a0 secure :-1 s=READY pgs=1518 cs=0 l=1 rev1=1 rx=0x7f17c0159630 tx=0x7f17b80091d0).stop 2022-07-20T03:17:42.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 shutdown_connections 2022-07-20T03:17:42.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f17a0053a60 0x7f17a0055f20 unknown :-1 s=CLOSED pgs=728 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f17c00afe50 0x7f17c01565a0 unknown :-1 s=CLOSED pgs=1518 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f17c00ac280 0x7f17c013e160 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 --2- 172.21.15.61:0/693432881 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f17c005c960 0x7f17c0155fe0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 >> 172.21.15.61:0/693432881 conn(0x7f17c00b53d0 msgr2=0x7f17c00b41b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 shutdown_connections 2022-07-20T03:17:42.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.599+0000 7f17c6245700 1 -- 172.21.15.61:0/693432881 wait complete. 2022-07-20T03:17:42.620 INFO:tasks.cephfs.mount:Wait for MDS to reach steady state... 2022-07-20T03:17:42.620 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:42.726 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 Processor -- start 2022-07-20T03:17:42.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- start start 2022-07-20T03:17:42.727 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3500b1e00 0x7ff35014a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7ff35005c3f0 0x7ff3500ac7e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff35005bd90 0x7ff3500accd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff35005d480 con 0x7ff35005bd90 2022-07-20T03:17:42.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7ff3500ad1c0 con 0x7ff35005c3f0 2022-07-20T03:17:42.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff3500b21e0 con 0x7ff3500b1e00 2022-07-20T03:17:42.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff35005bd90 0x7ff3500accd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7ff35005bd90 0x7ff3500accd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52006/0 (socket says 172.21.15.61:52006) 2022-07-20T03:17:42.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 -- 172.21.15.61:0/988508423 learned_addr learned my addr 172.21.15.61:0/988508423 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:42.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34f7fe700 1 --2- 172.21.15.61:0/988508423 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3500b1e00 0x7ff35014a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34effd700 1 --2- 172.21.15.61:0/988508423 >> v2:172.21.15.117:3300/0 conn(0x7ff35005c3f0 0x7ff3500ac7e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 -- 172.21.15.61:0/988508423 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3500b1e00 msgr2=0x7ff35014a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:42.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 --2- 172.21.15.61:0/988508423 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3500b1e00 0x7ff35014a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 -- 172.21.15.61:0/988508423 >> v2:172.21.15.117:3300/0 conn(0x7ff35005c3f0 msgr2=0x7ff3500ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:42.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 --2- 172.21.15.61:0/988508423 >> v2:172.21.15.117:3300/0 conn(0x7ff35005c3f0 0x7ff3500ac7e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 -- 172.21.15.61:0/988508423 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff35005cf40 con 0x7ff35005bd90 2022-07-20T03:17:42.735 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34ffff700 1 --2- 172.21.15.61:0/988508423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35005bd90 0x7ff3500accd0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7ff348002f70 tx=0x7ff348006eb0).ready entity=mon.0 client_cookie=4d899df0d59b50cc server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34e7fc700 1 -- 172.21.15.61:0/988508423 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff34800d570 con 0x7ff35005bd90 2022-07-20T03:17:42.736 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34e7fc700 1 -- 172.21.15.61:0/988508423 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348000e90 con 0x7ff35005bd90 2022-07-20T03:17:42.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff34e7fc700 1 -- 172.21.15.61:0/988508423 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff34800d900 con 0x7ff35005bd90 2022-07-20T03:17:42.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- 172.21.15.61:0/988508423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35005bd90 msgr2=0x7ff3500accd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- 172.21.15.61:0/988508423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35005bd90 0x7ff3500accd0 secure :-1 s=READY pgs=1519 cs=0 l=1 rev1=1 rx=0x7ff348002f70 tx=0x7ff348006eb0).stop 2022-07-20T03:17:42.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- 172.21.15.61:0/988508423 shutdown_connections 2022-07-20T03:17:42.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- 172.21.15.61:0/988508423 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff35005bd90 0x7ff3500accd0 unknown :-1 s=CLOSED pgs=1519 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- 172.21.15.61:0/988508423 >> v2:172.21.15.117:3300/0 conn(0x7ff35005c3f0 0x7ff3500ac7e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 --2- 172.21.15.61:0/988508423 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff3500b1e00 0x7ff35014a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.719+0000 7ff3560dd700 1 -- 172.21.15.61:0/988508423 >> 172.21.15.61:0/988508423 conn(0x7ff3500611c0 msgr2=0x7ff350062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.723+0000 7ff3560dd700 1 -- 172.21.15.61:0/988508423 shutdown_connections 2022-07-20T03:17:42.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.723+0000 7ff3560dd700 1 -- 172.21.15.61:0/988508423 wait complete. 2022-07-20T03:17:42.749 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.723+0000 7ff3560dd700 1 Processor -- start 2022-07-20T03:17:42.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.723+0000 7ff3560dd700 1 -- start start 2022-07-20T03:17:42.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff35005c3f0 0x7ff3500a07a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3500b1e00 0x7ff3500b3a10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 0x7ff35009eb40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff350149a80 con 0x7ff3501434c0 2022-07-20T03:17:42.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff350149cb0 con 0x7ff3500b1e00 2022-07-20T03:17:42.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7ff35009e900 con 0x7ff35005c3f0 2022-07-20T03:17:42.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 0x7ff35009eb40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34f7fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff35005c3f0 0x7ff3500a07a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 0x7ff35009eb40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52012/0 (socket says 172.21.15.61:52012) 2022-07-20T03:17:42.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 -- 172.21.15.61:0/1975228103 learned_addr learned my addr 172.21.15.61:0/1975228103 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:42.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34effd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3500b1e00 0x7ff3500b3a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 -- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff35005c3f0 msgr2=0x7ff3500a07a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff35005c3f0 0x7ff3500a07a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 -- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3500b1e00 msgr2=0x7ff3500b3a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3500b1e00 0x7ff3500b3a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 -- 172.21.15.61:0/1975228103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff34800b1e0 con 0x7ff3501434c0 2022-07-20T03:17:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34ffff700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 0x7ff35009eb40 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7ff348002f70 tx=0x7ff348015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff348011610 con 0x7ff3501434c0 2022-07-20T03:17:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7ff348011770 con 0x7ff3501434c0 2022-07-20T03:17:42.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff35014a280 con 0x7ff3501434c0 2022-07-20T03:17:42.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7ff348011ad0 con 0x7ff3501434c0 2022-07-20T03:17:42.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff35009f030 con 0x7ff3501434c0 2022-07-20T03:17:42.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.727+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff35009e640 con 0x7ff3501434c0 2022-07-20T03:17:42.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.731+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7ff348025030 con 0x7ff3501434c0 2022-07-20T03:17:42.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.731+0000 7ff34cff9700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff334053ad0 0x7ff334055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:42.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.731+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7ff34806d020 con 0x7ff3501434c0 2022-07-20T03:17:42.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.731+0000 7ff34f7fe700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff334053ad0 0x7ff334055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:42.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.731+0000 7ff34f7fe700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff334053ad0 0x7ff334055f90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7ff34000b000 tx=0x7ff3400091d0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:42.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.735+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7ff348043e70 con 0x7ff3501434c0 2022-07-20T03:17:42.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.899+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7ff35005d1e0 con 0x7ff3501434c0 2022-07-20T03:17:42.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.899+0000 7ff34cff9700 1 -- 172.21.15.61:0/1975228103 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2145 (secure 0 0 0) 0x7ff348023090 con 0x7ff3501434c0 2022-07-20T03:17:42.907 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:42.908 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7397,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1660470104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1660470104},{"type":"v1","addr":"172.21.15.61:6833","nonce":1660470104}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:35.498516+0000","modified":"2022-07-20T03:17:36.503758+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/1280182602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1280182602},{"type":"v1","addr":"172.21.15.117:6835","nonce":1280182602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T03:17:42.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff334053ad0 msgr2=0x7ff334055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff334053ad0 0x7ff334055f90 secure :-1 s=READY pgs=729 cs=0 l=1 rev1=1 rx=0x7ff34000b000 tx=0x7ff3400091d0).stop 2022-07-20T03:17:42.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 msgr2=0x7ff35009eb40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:42.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 0x7ff35009eb40 secure :-1 s=READY pgs=1520 cs=0 l=1 rev1=1 rx=0x7ff348002f70 tx=0x7ff348015000).stop 2022-07-20T03:17:42.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 shutdown_connections 2022-07-20T03:17:42.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7ff334053ad0 0x7ff334055f90 unknown :-1 s=CLOSED pgs=729 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7ff3500b1e00 0x7ff3500b3a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7ff3501434c0 0x7ff35009eb40 unknown :-1 s=CLOSED pgs=1520 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.911 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 --2- 172.21.15.61:0/1975228103 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7ff35005c3f0 0x7ff3500a07a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 >> 172.21.15.61:0/1975228103 conn(0x7ff35005bd90 msgr2=0x7ff35014b810 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 shutdown_connections 2022-07-20T03:17:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:42.903+0000 7ff3560dd700 1 -- 172.21.15.61:0/1975228103 wait complete. 2022-07-20T03:17:42.912 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-20T03:17:43.120 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: mds map: {'epoch': 83, 'flags': 18, 'ever_allowed_features': 0, 'explicitly_allowed_features': 0, 'created': '2022-07-20T03:17:35.498516+0000', 'modified': '2022-07-20T03:17:36.503758+0000', 'tableserver': 0, 'root': 0, 'session_timeout': 60, 'session_autoclose': 300, 'min_compat_client': '0 (unknown)', 'max_file_size': 1099511627776, 'last_failure': 0, 'last_failure_osd_epoch': 0, 'compat': {'compat': {}, 'ro_compat': {}, 'incompat': {'feature_1': 'base v0.20', 'feature_2': 'client writeable ranges', 'feature_3': 'default file layouts on dirs', 'feature_4': 'dir inode in separate object', 'feature_5': 'mds uses versioned encoding', 'feature_6': 'dirfrag is stored in omap', 'feature_8': 'no anchor table', 'feature_9': 'file layout v2', 'feature_10': 'snaprealm v2'}}, 'max_mds': 1, 'in': [0], 'up': {'mds_0': 8665}, 'failed': [], 'damaged': [], 'stopped': [], 'info': {'gid_8665': {'gid': 8665, 'name': 'b', 'rank': 0, 'incarnation': 82, 'state': 'up:active', 'state_seq': 3, 'addr': '172.21.15.117:6835/1280182602', 'addrs': {'addrvec': [{'type': 'v2', 'addr': '172.21.15.117:6834', 'nonce': 1280182602}, {'type': 'v1', 'addr': '172.21.15.117:6835', 'nonce': 1280182602}]}, 'join_fscid': -1, 'export_targets': [], 'features': 4540138292840890367, 'flags': 0}}, 'data_pools': [30], 'metadata_pool': 29, 'enabled': True, 'fs_name': 'cephfs', 'balancer': '', 'standby_count_wanted': 0} 2022-07-20T03:17:43.121 DEBUG:tasks.cephfs.filesystem:are_daemons_healthy: 1/1 2022-07-20T03:17:43.122 DEBUG:teuthology.orchestra.run.smithi117:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 900 ceph --cluster ceph --admin-daemon /var/run/ceph/ceph-mds.b.asok --format=json status 2022-07-20T03:17:43.238 INFO:teuthology.orchestra.run.smithi117.stdout:{"cluster_fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","whoami":0,"id":8665,"want_state":"up:active","state":"up:active","rank_uptime":7.7240624540000002,"mdsmap_epoch":83,"osdmap_epoch":149,"osdmap_epoch_barrier":149,"uptime":12.062743354} 2022-07-20T03:17:43.238 DEBUG:tasks.cephfs.filesystem:_json_asok output { "cluster_fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", "id": 8665, "mdsmap_epoch": 83, "osdmap_epoch": 149, "osdmap_epoch_barrier": 149, "rank_uptime": 7.724062454, "state": "up:active", "uptime": 12.062743354, "want_state": "up:active", "whoami": 0 } 2022-07-20T03:17:43.239 INFO:tasks.cephfs.mount:Ready to start FuseMount... 2022-07-20T03:17:43.239 INFO:tasks.cephfs.fuse_mount:Client client.1 config is {} 2022-07-20T03:17:43.240 INFO:tasks.cephfs.fuse_mount:Mounting ceph-fuse client.1 at ubuntu@smithi163.front.sepia.ceph.com /home/ubuntu/cephtest/mnt.1... 2022-07-20T03:17:43.240 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.240 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec mkdir -m 0000 -p /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.250 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:17:43.271 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:43.272 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:43.331 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:43.332 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:43.333 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.335 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:17:43.342 INFO:tasks.cephfs.fuse_mount:Pre-mount connections: [] 2022-07-20T03:17:43.342 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage daemon-helper kill ceph-fuse -f --name client.1 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.347 DEBUG:teuthology.orchestra.run.smithi163:> sudo modprobe fuse 2022-07-20T03:17:43.369 INFO:teuthology.orchestra.run:Running command with timeout 30 2022-07-20T03:17:43.370 DEBUG:teuthology.orchestra.run.smithi163:> sudo mount -t fusectl /sys/fs/fuse/connections /sys/fs/fuse/connections 2022-07-20T03:17:43.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:2022-07-20T03:17:43.421+0000 7fd84c953080 -1 init, newargv = 0x563abe97f3e0 newargc=9 2022-07-20T03:17:43.425 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[23111]: starting ceph client 2022-07-20T03:17:43.430 INFO:teuthology.orchestra.run.smithi163.stderr:mount: /sys/fs/fuse/connections: /sys/fs/fuse/connections already mounted or mount point busy. 2022-07-20T03:17:43.431 DEBUG:teuthology.orchestra.run:got remote process result: 32 2022-07-20T03:17:43.432 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.432 DEBUG:teuthology.orchestra.run.smithi163:> ls /sys/fs/fuse/connections 2022-07-20T03:17:43.434 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[23111]: starting fuse 2022-07-20T03:17:43.437 INFO:teuthology.orchestra.run.smithi163.stdout:52 2022-07-20T03:17:43.438 INFO:tasks.cephfs.fuse_mount:Post-mount connections: [52] 2022-07-20T03:17:43.438 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.439 DEBUG:teuthology.orchestra.run.smithi163:> sudo python3 -c ' 2022-07-20T03:17:43.440 DEBUG:teuthology.orchestra.run.smithi163:> import glob 2022-07-20T03:17:43.441 DEBUG:teuthology.orchestra.run.smithi163:> import re 2022-07-20T03:17:43.442 DEBUG:teuthology.orchestra.run.smithi163:> import os 2022-07-20T03:17:43.442 DEBUG:teuthology.orchestra.run.smithi163:> import subprocess 2022-07-20T03:17:43.443 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:43.443 DEBUG:teuthology.orchestra.run.smithi163:> def find_socket(client_name): 2022-07-20T03:17:43.444 DEBUG:teuthology.orchestra.run.smithi163:> asok_path = "/var/run/ceph/ceph-client.1.*.asok" 2022-07-20T03:17:43.444 DEBUG:teuthology.orchestra.run.smithi163:> files = glob.glob(asok_path) 2022-07-20T03:17:43.445 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:43.445 DEBUG:teuthology.orchestra.run.smithi163:> # Given a non-glob path, it better be there 2022-07-20T03:17:43.445 DEBUG:teuthology.orchestra.run.smithi163:> if "*" not in asok_path: 2022-07-20T03:17:43.446 DEBUG:teuthology.orchestra.run.smithi163:> assert(len(files) == 1) 2022-07-20T03:17:43.446 DEBUG:teuthology.orchestra.run.smithi163:> return files[0] 2022-07-20T03:17:43.447 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:43.447 DEBUG:teuthology.orchestra.run.smithi163:> for f in files: 2022-07-20T03:17:43.447 DEBUG:teuthology.orchestra.run.smithi163:> pid = re.match(".*\.(\d+)\.asok$", f).group(1) 2022-07-20T03:17:43.448 DEBUG:teuthology.orchestra.run.smithi163:> if os.path.exists("/proc/{0}".format(pid)): 2022-07-20T03:17:43.448 DEBUG:teuthology.orchestra.run.smithi163:> return f 2022-07-20T03:17:43.449 DEBUG:teuthology.orchestra.run.smithi163:> raise RuntimeError("Client socket {0} not found".format(client_name)) 2022-07-20T03:17:43.449 DEBUG:teuthology.orchestra.run.smithi163:> 2022-07-20T03:17:43.450 DEBUG:teuthology.orchestra.run.smithi163:> print(find_socket("client.1")) 2022-07-20T03:17:43.450 DEBUG:teuthology.orchestra.run.smithi163:> ' 2022-07-20T03:17:43.492 INFO:teuthology.orchestra.run.smithi163.stdout:/var/run/ceph/ceph-client.1.23111.asok 2022-07-20T03:17:43.497 INFO:tasks.cephfs.fuse_mount:Found client admin socket at /var/run/ceph/ceph-client.1.23111.asok 2022-07-20T03:17:43.497 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.498 DEBUG:teuthology.orchestra.run.smithi163:> sudo ceph --admin-daemon /var/run/ceph/ceph-client.1.23111.asok status 2022-07-20T03:17:43.631 INFO:teuthology.orchestra.run.smithi163.stdout:{ 2022-07-20T03:17:43.632 INFO:teuthology.orchestra.run.smithi163.stdout: "metadata": { 2022-07-20T03:17:43.632 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_sha1": "57544da114dbbe1c44f27b2ea18e3233e4e917e0", 2022-07-20T03:17:43.633 INFO:teuthology.orchestra.run.smithi163.stdout: "ceph_version": "ceph version 15.2.16-336-g57544da1 (57544da114dbbe1c44f27b2ea18e3233e4e917e0) octopus (stable)", 2022-07-20T03:17:43.633 INFO:teuthology.orchestra.run.smithi163.stdout: "entity_id": "1", 2022-07-20T03:17:43.634 INFO:teuthology.orchestra.run.smithi163.stdout: "hostname": "smithi163", 2022-07-20T03:17:43.634 INFO:teuthology.orchestra.run.smithi163.stdout: "mount_point": "/home/ubuntu/cephtest/mnt.1", 2022-07-20T03:17:43.634 INFO:teuthology.orchestra.run.smithi163.stdout: "pid": "23111", 2022-07-20T03:17:43.635 INFO:teuthology.orchestra.run.smithi163.stdout: "root": "/" 2022-07-20T03:17:43.635 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:43.636 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_count": 0, 2022-07-20T03:17:43.637 INFO:teuthology.orchestra.run.smithi163.stdout: "dentry_pinned_count": 0, 2022-07-20T03:17:43.637 INFO:teuthology.orchestra.run.smithi163.stdout: "id": 9006, 2022-07-20T03:17:43.637 INFO:teuthology.orchestra.run.smithi163.stdout: "inst": { 2022-07-20T03:17:43.637 INFO:teuthology.orchestra.run.smithi163.stdout: "name": { 2022-07-20T03:17:43.638 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "client", 2022-07-20T03:17:43.638 INFO:teuthology.orchestra.run.smithi163.stdout: "num": 9006 2022-07-20T03:17:43.638 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:43.638 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:17:43.639 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:17:43.639 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:17:43.639 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1305035474 2022-07-20T03:17:43.639 INFO:teuthology.orchestra.run.smithi163.stdout: } 2022-07-20T03:17:43.640 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:43.641 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": { 2022-07-20T03:17:43.642 INFO:teuthology.orchestra.run.smithi163.stdout: "type": "v1", 2022-07-20T03:17:43.642 INFO:teuthology.orchestra.run.smithi163.stdout: "addr": "172.21.15.163:0", 2022-07-20T03:17:43.643 INFO:teuthology.orchestra.run.smithi163.stdout: "nonce": 1305035474 2022-07-20T03:17:43.643 INFO:teuthology.orchestra.run.smithi163.stdout: }, 2022-07-20T03:17:43.643 INFO:teuthology.orchestra.run.smithi163.stdout: "inst_str": "client.9006 172.21.15.163:0/1305035474", 2022-07-20T03:17:43.643 INFO:teuthology.orchestra.run.smithi163.stdout: "addr_str": "172.21.15.163:0/1305035474", 2022-07-20T03:17:43.644 INFO:teuthology.orchestra.run.smithi163.stdout: "inode_count": 1, 2022-07-20T03:17:43.644 INFO:teuthology.orchestra.run.smithi163.stdout: "mds_epoch": 83, 2022-07-20T03:17:43.644 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch": 149, 2022-07-20T03:17:43.644 INFO:teuthology.orchestra.run.smithi163.stdout: "osd_epoch_barrier": 0, 2022-07-20T03:17:43.645 INFO:teuthology.orchestra.run.smithi163.stdout: "blacklisted": false, 2022-07-20T03:17:43.645 INFO:teuthology.orchestra.run.smithi163.stdout: "fs_name": "cephfs" 2022-07-20T03:17:43.645 INFO:teuthology.orchestra.run.smithi163.stdout:} 2022-07-20T03:17:43.647 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.647 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:43.648 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.659 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:17:43.659 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:43.660 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.660 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo chmod 1777 /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.681 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.682 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:43.682 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.735 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:17:43.736 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:43.737 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.738 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:43.739 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.750 INFO:teuthology.orchestra.run.smithi163.stdout:fuseblk 2022-07-20T03:17:43.750 INFO:tasks.cephfs.fuse_mount:ceph-fuse is mounted on /home/ubuntu/cephtest/mnt.1 2022-07-20T03:17:43.751 INFO:tasks.cephfs.fuse_mount:Running fusermount -u on ubuntu@smithi163.front.sepia.ceph.com... 2022-07-20T03:17:43.751 INFO:teuthology.orchestra.run:Running command with timeout 1800 2022-07-20T03:17:43.752 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec sudo fusermount -u /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.808 INFO:tasks.cephfs.fuse_mount.ceph-fuse.1.smithi163.stderr:ceph-fuse[23111]: fuse finished with error 0 and tester_r 0 2022-07-20T03:17:43.811 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:17:43.812 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:17:43.812 DEBUG:teuthology.orchestra.run.smithi163:> ' -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:43.820 INFO:teuthology.orchestra.run.smithi163.stdout:ext2/ext3 2022-07-20T03:17:43.822 DEBUG:tasks.cephfs.fuse_mount:ceph-fuse not mounted, got fs type 'ext2/ext3' 2022-07-20T03:17:43.822 INFO:teuthology.orchestra.run:waiting for 900 2022-07-20T03:17:49.926 INFO:teuthology.orchestra.run:Running command with timeout 300 2022-07-20T03:17:49.927 DEBUG:teuthology.orchestra.run.smithi163:> (cd /home/ubuntu/cephtest && exec rmdir -- /home/ubuntu/cephtest/mnt.1) 2022-07-20T03:17:49.936 INFO:teuthology.orchestra.run:Running command with timeout 10800 2022-07-20T03:17:49.937 DEBUG:teuthology.orchestra.run.smithi124:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-20T03:17:49.962 INFO:teuthology.orchestra.run.smithi124.stdout: File: "." 2022-07-20T03:17:49.963 INFO:teuthology.orchestra.run.smithi124.stdout: ID: 0 Namelen: 255 Type: fuseblk 2022-07-20T03:17:49.963 INFO:teuthology.orchestra.run.smithi124.stdout:Block size: 4194304 Fundamental block size: 4194304 2022-07-20T03:17:49.963 INFO:teuthology.orchestra.run.smithi124.stdout:Blocks: Total: 86148 Free: 86148 Available: 86148 2022-07-20T03:17:49.963 INFO:teuthology.orchestra.run.smithi124.stdout:Inodes: Total: 1 Free: 0 2022-07-20T03:17:49.964 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json-pretty 2022-07-20T03:17:50.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.067+0000 7fe977466700 1 Processor -- start 2022-07-20T03:17:50.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.067+0000 7fe977466700 1 -- start start 2022-07-20T03:17:50.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 0x7fe9700b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe9700af8e0 0x7fe97005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe970147010 con 0x7fe9700adff0 2022-07-20T03:17:50.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe9701472b0 con 0x7fe9700af8e0 2022-07-20T03:17:50.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe970146d40 con 0x7fe9700b5650 2022-07-20T03:17:50.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 0x7fe9700b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976c65700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe9700af8e0 0x7fe97005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.083 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 0x7fe9700b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38688/0 (socket says 172.21.15.61:38688) 2022-07-20T03:17:50.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 -- 172.21.15.61:0/3044101332 learned_addr learned my addr 172.21.15.61:0/3044101332 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:50.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 -- 172.21.15.61:0/3044101332 >> v2:172.21.15.117:3300/0 conn(0x7fe9700af8e0 msgr2=0x7fe97005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:50.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 --2- 172.21.15.61:0/3044101332 >> v2:172.21.15.117:3300/0 conn(0x7fe9700af8e0 0x7fe97005bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.084 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 -- 172.21.15.61:0/3044101332 >> v1:172.21.15.117:6789/0 conn(0x7fe9700adff0 legacy=0x7fe970147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 -- 172.21.15.61:0/3044101332 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9700acae0 con 0x7fe9700b5650 2022-07-20T03:17:50.085 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe976464700 1 --2- 172.21.15.61:0/3044101332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 0x7fe9700b5a30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe9680032f0 tx=0x7fe968009290).ready entity=mon.2 client_cookie=bebed5684a60fa8e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe975462700 1 -- 172.21.15.61:0/3044101332 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe96800b8a0 con 0x7fe9700b5650 2022-07-20T03:17:50.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe975462700 1 -- 172.21.15.61:0/3044101332 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe96800ba00 con 0x7fe9700b5650 2022-07-20T03:17:50.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe975462700 1 -- 172.21.15.61:0/3044101332 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe96800bb70 con 0x7fe9700b5650 2022-07-20T03:17:50.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- 172.21.15.61:0/3044101332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 msgr2=0x7fe9700b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 --2- 172.21.15.61:0/3044101332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 0x7fe9700b5a30 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7fe9680032f0 tx=0x7fe968009290).stop 2022-07-20T03:17:50.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- 172.21.15.61:0/3044101332 shutdown_connections 2022-07-20T03:17:50.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 --2- 172.21.15.61:0/3044101332 >> v2:172.21.15.117:3300/0 conn(0x7fe9700af8e0 0x7fe97005bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 --2- 172.21.15.61:0/3044101332 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700b5650 0x7fe9700b5a30 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- 172.21.15.61:0/3044101332 >> 172.21.15.61:0/3044101332 conn(0x7fe9700634e0 msgr2=0x7fe9700b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:50.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- 172.21.15.61:0/3044101332 shutdown_connections 2022-07-20T03:17:50.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.071+0000 7fe977466700 1 -- 172.21.15.61:0/3044101332 wait complete. 2022-07-20T03:17:50.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 Processor -- start 2022-07-20T03:17:50.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 -- start start 2022-07-20T03:17:50.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 0x7fe9701407a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700af8e0 0x7fe970144610 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9700b5650 0x7fe970144b00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970140060 con 0x7fe9700adff0 2022-07-20T03:17:50.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970142ee0 con 0x7fe9700b5650 2022-07-20T03:17:50.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe977466700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe970143180 con 0x7fe9700af8e0 2022-07-20T03:17:50.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe976464700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 0x7fe9701407a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe976464700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 0x7fe9701407a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52022/0 (socket says 172.21.15.61:52022) 2022-07-20T03:17:50.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe976464700 1 -- 172.21.15.61:0/915064216 learned_addr learned my addr 172.21.15.61:0/915064216 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:50.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.075+0000 7fe976c65700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9700b5650 0x7fe970144b00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe975c63700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700af8e0 0x7fe970144610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe976464700 1 -- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700af8e0 msgr2=0x7fe970144610 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe976464700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700af8e0 0x7fe970144610 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe976464700 1 -- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9700b5650 msgr2=0x7fe970144b00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe976464700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9700b5650 0x7fe970144b00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe976464700 1 -- 172.21.15.61:0/915064216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe968009530 con 0x7fe9700adff0 2022-07-20T03:17:50.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe976464700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 0x7fe9701407a0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fe9680032f0 tx=0x7fe968016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe96800b8a0 con 0x7fe9700adff0 2022-07-20T03:17:50.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe968021630 con 0x7fe9700adff0 2022-07-20T03:17:50.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe97013fce0 con 0x7fe9700adff0 2022-07-20T03:17:50.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe968021990 con 0x7fe9700adff0 2022-07-20T03:17:50.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe970144ff0 con 0x7fe9700adff0 2022-07-20T03:17:50.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9700574e0 con 0x7fe9700adff0 2022-07-20T03:17:50.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.079+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe968027020 con 0x7fe9700adff0 2022-07-20T03:17:50.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.083+0000 7fe96f7fe700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9500539d0 0x7fe950055e90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.083+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7fe96806d2f0 con 0x7fe9700adff0 2022-07-20T03:17:50.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.083+0000 7fe975c63700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9500539d0 0x7fe950055e90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.083+0000 7fe975c63700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9500539d0 0x7fe950055e90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7fe96400d000 tx=0x7fe964007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.087+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe9680440c0 con 0x7fe9700adff0 2022-07-20T03:17:50.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.211+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json-pretty"} v 0) v1 -- 0x7fe97013fed0 con 0x7fe9700adff0 2022-07-20T03:17:50.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.215+0000 7fe96f7fe700 1 -- 172.21.15.61:0/915064216 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json-pretty"}]=0 v149) v1 ==== 81+0+33707 (secure 0 0 0) 0x7fe968015020 con 0x7fe9700adff0 2022-07-20T03:17:50.222 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:50.222 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T03:17:50.223 INFO:teuthology.orchestra.run.smithi061.stdout: "epoch": 149, 2022-07-20T03:17:50.223 INFO:teuthology.orchestra.run.smithi061.stdout: "fsid": "69b2e2c1-da2e-4470-a6a9-37f8d410d6ed", 2022-07-20T03:17:50.223 INFO:teuthology.orchestra.run.smithi061.stdout: "created": "2022-07-20T03:06:50.175615+0000", 2022-07-20T03:17:50.224 INFO:teuthology.orchestra.run.smithi061.stdout: "modified": "2022-07-20T03:17:36.502720+0000", 2022-07-20T03:17:50.224 INFO:teuthology.orchestra.run.smithi061.stdout: "last_up_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:50.224 INFO:teuthology.orchestra.run.smithi061.stdout: "last_in_change": "2022-07-20T03:06:59.505879+0000", 2022-07-20T03:17:50.224 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": "sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit", 2022-07-20T03:17:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_num": 5799936, 2022-07-20T03:17:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_set": [ 2022-07-20T03:17:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: "pglog_hardlimit", 2022-07-20T03:17:50.225 INFO:teuthology.orchestra.run.smithi061.stdout: "purged_snapdirs", 2022-07-20T03:17:50.226 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_deletes", 2022-07-20T03:17:50.226 INFO:teuthology.orchestra.run.smithi061.stdout: "sortbitwise" 2022-07-20T03:17:50.226 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:50.226 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_version": 4, 2022-07-20T03:17:50.227 INFO:teuthology.orchestra.run.smithi061.stdout: "full_ratio": 0.94999998807907104, 2022-07-20T03:17:50.227 INFO:teuthology.orchestra.run.smithi061.stdout: "backfillfull_ratio": 0.89999997615814209, 2022-07-20T03:17:50.227 INFO:teuthology.orchestra.run.smithi061.stdout: "nearfull_ratio": 0.85000002384185791, 2022-07-20T03:17:50.227 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_snapshot": "", 2022-07-20T03:17:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_max": 30, 2022-07-20T03:17:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "max_osd": 8, 2022-07-20T03:17:50.228 INFO:teuthology.orchestra.run.smithi061.stdout: "require_min_compat_client": "jewel", 2022-07-20T03:17:50.229 INFO:teuthology.orchestra.run.smithi061.stdout: "min_compat_client": "jewel", 2022-07-20T03:17:50.229 INFO:teuthology.orchestra.run.smithi061.stdout: "require_osd_release": "octopus", 2022-07-20T03:17:50.230 INFO:teuthology.orchestra.run.smithi061.stdout: "pools": [ 2022-07-20T03:17:50.230 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 1, 2022-07-20T03:17:50.231 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "device_health_metrics", 2022-07-20T03:17:50.231 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:06:54.040896+0000", 2022-07-20T03:17:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:50.232 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:50.233 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 1, 2022-07-20T03:17:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 1, 2022-07-20T03:17:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 1, 2022-07-20T03:17:50.234 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 1, 2022-07-20T03:17:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 1, 2022-07-20T03:17:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:50.235 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:50.236 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "15", 2022-07-20T03:17:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:50.237 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:50.238 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:50.239 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:50.240 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:50.241 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:50.242 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:50.243 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.244 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:50.245 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:50.246 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:50.247 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:50.248 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:50.249 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:17:50.249 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 1 2022-07-20T03:17:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:50.250 INFO:teuthology.orchestra.run.smithi061.stdout: "mgr_devicehealth": {} 2022-07-20T03:17:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.251 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 2, 2022-07-20T03:17:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "rbd", 2022-07-20T03:17:50.252 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:07:05.508608+0000", 2022-07-20T03:17:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:50.253 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:50.254 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:50.255 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:50.255 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:50.256 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:50.257 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:50.258 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:50.259 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:50.260 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:50.261 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:50.261 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.262 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "18", 2022-07-20T03:17:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:50.263 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:50.264 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:50.265 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:50.266 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:50.267 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:50.268 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:50.269 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:50.270 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:50.271 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:50.272 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:17:50.273 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: "rbd": {} 2022-07-20T03:17:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.274 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 29, 2022-07-20T03:17:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_metadata", 2022-07-20T03:17:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:17:34.158104+0000", 2022-07-20T03:17:50.275 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:50.276 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:50.277 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:50.278 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:50.279 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.280 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "148", 2022-07-20T03:17:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:50.281 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:50.282 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:50.283 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:50.284 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:50.285 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:50.286 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:50.287 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:50.288 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:50.289 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:50.289 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.290 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:50.291 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:50.291 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:50.292 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:50.292 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:50.293 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:50.294 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:50.295 INFO:teuthology.orchestra.run.smithi061.stdout: "options": { 2022-07-20T03:17:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_bias": 4, 2022-07-20T03:17:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_min": 16, 2022-07-20T03:17:50.296 INFO:teuthology.orchestra.run.smithi061.stdout: "recovery_priority": 5 2022-07-20T03:17:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:17:50.297 INFO:teuthology.orchestra.run.smithi061.stdout: "metadata": "cephfs" 2022-07-20T03:17:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.298 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: "pool": 30, 2022-07-20T03:17:50.299 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_name": "cephfs_data", 2022-07-20T03:17:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "create_time": "2022-07-20T03:17:34.713017+0000", 2022-07-20T03:17:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "flags": 1, 2022-07-20T03:17:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "flags_names": "hashpspool", 2022-07-20T03:17:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "type": 1, 2022-07-20T03:17:50.300 INFO:teuthology.orchestra.run.smithi061.stdout: "size": 2, 2022-07-20T03:17:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "min_size": 1, 2022-07-20T03:17:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_rule": 0, 2022-07-20T03:17:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "object_hash": 2, 2022-07-20T03:17:50.301 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_autoscale_mode": "off", 2022-07-20T03:17:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num": 8, 2022-07-20T03:17:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num": 8, 2022-07-20T03:17:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_placement_num_target": 8, 2022-07-20T03:17:50.302 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_target": 8, 2022-07-20T03:17:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_num_pending": 8, 2022-07-20T03:17:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "last_pg_merge_meta": { 2022-07-20T03:17:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "source_pgid": "0.0", 2022-07-20T03:17:50.303 INFO:teuthology.orchestra.run.smithi061.stdout: "ready_epoch": 0, 2022-07-20T03:17:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_started": 0, 2022-07-20T03:17:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "last_epoch_clean": 0, 2022-07-20T03:17:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "source_version": "0'0", 2022-07-20T03:17:50.304 INFO:teuthology.orchestra.run.smithi061.stdout: "target_version": "0'0" 2022-07-20T03:17:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.305 INFO:teuthology.orchestra.run.smithi061.stdout: "last_change": "149", 2022-07-20T03:17:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend": "0", 2022-07-20T03:17:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_prenautilus": "0", 2022-07-20T03:17:50.306 INFO:teuthology.orchestra.run.smithi061.stdout: "last_force_op_resend_preluminous": "0", 2022-07-20T03:17:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "auid": 0, 2022-07-20T03:17:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_mode": "selfmanaged", 2022-07-20T03:17:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_seq": 0, 2022-07-20T03:17:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "snap_epoch": 0, 2022-07-20T03:17:50.307 INFO:teuthology.orchestra.run.smithi061.stdout: "pool_snaps": [], 2022-07-20T03:17:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps": "[]", 2022-07-20T03:17:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_bytes": 0, 2022-07-20T03:17:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "quota_max_objects": 0, 2022-07-20T03:17:50.308 INFO:teuthology.orchestra.run.smithi061.stdout: "tiers": [], 2022-07-20T03:17:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "tier_of": -1, 2022-07-20T03:17:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "read_tier": -1, 2022-07-20T03:17:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "write_tier": -1, 2022-07-20T03:17:50.309 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_mode": "none", 2022-07-20T03:17:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_bytes": 0, 2022-07-20T03:17:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "target_max_objects": 0, 2022-07-20T03:17:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_ratio_micro": 400000, 2022-07-20T03:17:50.310 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_dirty_high_ratio_micro": 600000, 2022-07-20T03:17:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_target_full_ratio_micro": 800000, 2022-07-20T03:17:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_flush_age": 0, 2022-07-20T03:17:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "cache_min_evict_age": 0, 2022-07-20T03:17:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profile": "", 2022-07-20T03:17:50.311 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_params": { 2022-07-20T03:17:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "none" 2022-07-20T03:17:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_period": 0, 2022-07-20T03:17:50.312 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_count": 0, 2022-07-20T03:17:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "use_gmt_hitset": true, 2022-07-20T03:17:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "min_read_recency_for_promote": 0, 2022-07-20T03:17:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "min_write_recency_for_promote": 0, 2022-07-20T03:17:50.313 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_grade_decay_rate": 0, 2022-07-20T03:17:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "hit_set_search_last_n": 0, 2022-07-20T03:17:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "grade_table": [], 2022-07-20T03:17:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "stripe_width": 0, 2022-07-20T03:17:50.314 INFO:teuthology.orchestra.run.smithi061.stdout: "expected_num_objects": 0, 2022-07-20T03:17:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "fast_read": false, 2022-07-20T03:17:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "options": {}, 2022-07-20T03:17:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "application_metadata": { 2022-07-20T03:17:50.315 INFO:teuthology.orchestra.run.smithi061.stdout: "cephfs": { 2022-07-20T03:17:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: "data": "cephfs" 2022-07-20T03:17:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.316 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: "osds": [ 2022-07-20T03:17:50.317 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.318 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:50.318 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "2994e86d-2708-44df-bf9c-63667f52059f", 2022-07-20T03:17:50.318 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.318 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.319 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.320 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.320 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 147, 2022-07-20T03:17:50.320 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.320 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.321 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6800", 2022-07-20T03:17:50.322 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.322 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.322 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.323 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.323 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6801", 2022-07-20T03:17:50.323 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.323 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.324 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.324 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.324 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.324 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.325 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.325 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.325 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6802", 2022-07-20T03:17:50.325 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.326 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6803", 2022-07-20T03:17:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.327 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.328 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6806", 2022-07-20T03:17:50.329 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.330 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.330 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.330 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6807", 2022-07-20T03:17:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.331 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.332 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.332 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.333 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.333 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.334 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.334 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6804", 2022-07-20T03:17:50.335 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.336 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.336 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.337 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.337 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6805", 2022-07-20T03:17:50.338 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22894 2022-07-20T03:17:50.338 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.339 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.339 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.340 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6801/22894", 2022-07-20T03:17:50.340 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6803/22894", 2022-07-20T03:17:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6807/22894", 2022-07-20T03:17:50.341 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6805/22894", 2022-07-20T03:17:50.342 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.342 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.343 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.343 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.343 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.344 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.344 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:50.344 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "9cd27797-c189-4c94-9e22-769611620680", 2022-07-20T03:17:50.345 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.345 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.345 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.345 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.346 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.346 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.346 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.346 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 146, 2022-07-20T03:17:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.347 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.348 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.348 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.348 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6808", 2022-07-20T03:17:50.349 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.349 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.349 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.349 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.350 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6809", 2022-07-20T03:17:50.350 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.350 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.350 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.351 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.351 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.351 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.351 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.352 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.352 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6810", 2022-07-20T03:17:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.353 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.354 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6811", 2022-07-20T03:17:50.354 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.355 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.355 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.356 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.356 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.356 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.356 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6814", 2022-07-20T03:17:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.357 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.358 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.358 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.358 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6816", 2022-07-20T03:17:50.359 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.359 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.359 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.360 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.360 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.360 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6812", 2022-07-20T03:17:50.361 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.362 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6813", 2022-07-20T03:17:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 22998 2022-07-20T03:17:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.363 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6809/22998", 2022-07-20T03:17:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6811/22998", 2022-07-20T03:17:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6816/22998", 2022-07-20T03:17:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6813/22998", 2022-07-20T03:17:50.364 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.365 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.365 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.365 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.365 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "637945c3-b1e0-4d86-933a-2723e56b0705", 2022-07-20T03:17:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.366 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.367 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 147, 2022-07-20T03:17:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.368 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.369 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6815", 2022-07-20T03:17:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.370 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6817", 2022-07-20T03:17:50.371 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.372 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.373 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.373 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.373 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.374 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.374 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6818", 2022-07-20T03:17:50.374 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6819", 2022-07-20T03:17:50.375 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.376 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.376 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.376 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.377 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.377 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.377 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.378 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.378 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6822", 2022-07-20T03:17:50.379 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.379 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.380 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.380 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.381 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6823", 2022-07-20T03:17:50.381 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.382 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.382 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.382 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.383 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.383 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.384 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.384 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.385 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6820", 2022-07-20T03:17:50.385 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.386 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.386 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.387 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.387 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6821", 2022-07-20T03:17:50.388 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23001 2022-07-20T03:17:50.388 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.389 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.389 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.390 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6817/23001", 2022-07-20T03:17:50.390 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6819/23001", 2022-07-20T03:17:50.390 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6823/23001", 2022-07-20T03:17:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6821/23001", 2022-07-20T03:17:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.391 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:50.392 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "b6079eb6-2fd9-4f0e-831f-83b051fc040f", 2022-07-20T03:17:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.393 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.394 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 147, 2022-07-20T03:17:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.395 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.396 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6824", 2022-07-20T03:17:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.397 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6825", 2022-07-20T03:17:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.398 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.399 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6826", 2022-07-20T03:17:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.400 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6827", 2022-07-20T03:17:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.401 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.402 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6830", 2022-07-20T03:17:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.403 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6831", 2022-07-20T03:17:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.404 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.405 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6828", 2022-07-20T03:17:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.406 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.61:6829", 2022-07-20T03:17:50.407 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23120 2022-07-20T03:17:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.408 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.61:6825/23120", 2022-07-20T03:17:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.61:6827/23120", 2022-07-20T03:17:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.61:6831/23120", 2022-07-20T03:17:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.61:6829/23120", 2022-07-20T03:17:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.409 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.410 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f342a61-db71-49da-a367-b177e2e65fdd", 2022-07-20T03:17:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.411 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.412 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.413 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 146, 2022-07-20T03:17:50.413 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.413 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.413 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6802", 2022-07-20T03:17:50.414 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.415 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6803", 2022-07-20T03:17:50.416 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.416 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.416 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.416 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.417 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6804", 2022-07-20T03:17:50.418 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.418 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.418 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.418 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.419 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6805", 2022-07-20T03:17:50.419 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.419 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.419 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.420 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.421 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6808", 2022-07-20T03:17:50.421 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.421 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.421 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.422 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.422 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6809", 2022-07-20T03:17:50.422 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.422 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.423 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.424 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.424 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6806", 2022-07-20T03:17:50.424 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.424 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6807", 2022-07-20T03:17:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23184 2022-07-20T03:17:50.425 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6803/23184", 2022-07-20T03:17:50.426 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6805/23184", 2022-07-20T03:17:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6809/23184", 2022-07-20T03:17:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6807/23184", 2022-07-20T03:17:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.427 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.428 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "ae86667a-16a2-48c1-9182-7694f9e01b8b", 2022-07-20T03:17:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.429 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.430 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 147, 2022-07-20T03:17:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.431 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6818", 2022-07-20T03:17:50.432 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6819", 2022-07-20T03:17:50.433 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.434 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.435 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6820", 2022-07-20T03:17:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.436 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6821", 2022-07-20T03:17:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.437 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.438 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6824", 2022-07-20T03:17:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.439 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.440 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6826", 2022-07-20T03:17:50.440 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.440 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.440 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.441 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.442 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6822", 2022-07-20T03:17:50.442 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.442 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.442 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.443 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.443 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6823", 2022-07-20T03:17:50.443 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23294 2022-07-20T03:17:50.443 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.444 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.444 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.444 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6819/23294", 2022-07-20T03:17:50.444 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6821/23294", 2022-07-20T03:17:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6826/23294", 2022-07-20T03:17:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6823/23294", 2022-07-20T03:17:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.445 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.446 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:50.447 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1558d94a-b96d-4bbb-86ad-34fe05792eb8", 2022-07-20T03:17:50.447 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.447 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.447 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.448 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.448 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.448 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.448 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 147, 2022-07-20T03:17:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.449 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.450 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.450 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.450 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6810", 2022-07-20T03:17:50.450 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.451 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.451 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.451 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.451 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6811", 2022-07-20T03:17:50.452 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.452 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.452 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.452 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.452 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.453 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6812", 2022-07-20T03:17:50.454 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.454 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.454 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.454 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.454 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6813", 2022-07-20T03:17:50.455 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.455 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.455 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.456 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.457 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.457 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6816", 2022-07-20T03:17:50.457 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.457 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6817", 2022-07-20T03:17:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.458 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.459 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.459 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.459 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.459 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6814", 2022-07-20T03:17:50.460 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6815", 2022-07-20T03:17:50.461 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23293 2022-07-20T03:17:50.462 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.462 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.462 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.462 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6811/23293", 2022-07-20T03:17:50.463 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6813/23293", 2022-07-20T03:17:50.463 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6817/23293", 2022-07-20T03:17:50.463 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6815/23293", 2022-07-20T03:17:50.463 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.464 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.464 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.464 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.464 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.464 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.465 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:50.465 INFO:teuthology.orchestra.run.smithi061.stdout: "uuid": "1f056769-9505-445e-8c01-b2276cd63a2c", 2022-07-20T03:17:50.465 INFO:teuthology.orchestra.run.smithi061.stdout: "up": 1, 2022-07-20T03:17:50.465 INFO:teuthology.orchestra.run.smithi061.stdout: "in": 1, 2022-07-20T03:17:50.466 INFO:teuthology.orchestra.run.smithi061.stdout: "weight": 1, 2022-07-20T03:17:50.466 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_affinity": 1, 2022-07-20T03:17:50.466 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_begin": 0, 2022-07-20T03:17:50.466 INFO:teuthology.orchestra.run.smithi061.stdout: "last_clean_end": 0, 2022-07-20T03:17:50.467 INFO:teuthology.orchestra.run.smithi061.stdout: "up_from": 13, 2022-07-20T03:17:50.467 INFO:teuthology.orchestra.run.smithi061.stdout: "up_thru": 147, 2022-07-20T03:17:50.467 INFO:teuthology.orchestra.run.smithi061.stdout: "down_at": 0, 2022-07-20T03:17:50.467 INFO:teuthology.orchestra.run.smithi061.stdout: "lost_at": 0, 2022-07-20T03:17:50.467 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addrs": { 2022-07-20T03:17:50.468 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.468 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.468 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.468 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6825", 2022-07-20T03:17:50.469 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.469 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.469 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.469 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.469 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6827", 2022-07-20T03:17:50.470 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.470 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.470 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.470 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.471 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addrs": { 2022-07-20T03:17:50.471 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.471 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.471 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6828", 2022-07-20T03:17:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.472 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.473 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6829", 2022-07-20T03:17:50.473 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.473 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.473 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.474 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.474 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addrs": { 2022-07-20T03:17:50.474 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.474 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6832", 2022-07-20T03:17:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.475 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.476 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.476 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6833", 2022-07-20T03:17:50.476 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.476 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.477 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.477 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.477 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addrs": { 2022-07-20T03:17:50.477 INFO:teuthology.orchestra.run.smithi061.stdout: "addrvec": [ 2022-07-20T03:17:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v2", 2022-07-20T03:17:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6830", 2022-07-20T03:17:50.478 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: "type": "v1", 2022-07-20T03:17:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: "addr": "172.21.15.117:6831", 2022-07-20T03:17:50.479 INFO:teuthology.orchestra.run.smithi061.stdout: "nonce": 23295 2022-07-20T03:17:50.480 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.480 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.480 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.480 INFO:teuthology.orchestra.run.smithi061.stdout: "public_addr": "172.21.15.117:6827/23295", 2022-07-20T03:17:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "cluster_addr": "172.21.15.117:6829/23295", 2022-07-20T03:17:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_back_addr": "172.21.15.117:6833/23295", 2022-07-20T03:17:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "heartbeat_front_addr": "172.21.15.117:6831/23295", 2022-07-20T03:17:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "state": [ 2022-07-20T03:17:50.481 INFO:teuthology.orchestra.run.smithi061.stdout: "exists", 2022-07-20T03:17:50.482 INFO:teuthology.orchestra.run.smithi061.stdout: "up" 2022-07-20T03:17:50.482 INFO:teuthology.orchestra.run.smithi061.stdout: ] 2022-07-20T03:17:50.482 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.482 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:50.483 INFO:teuthology.orchestra.run.smithi061.stdout: "osd_xinfo": [ 2022-07-20T03:17:50.483 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.483 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 0, 2022-07-20T03:17:50.483 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.484 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.485 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.755912+0000", 2022-07-20T03:17:50.485 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.485 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.485 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.485 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 1, 2022-07-20T03:17:50.486 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.486 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.486 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.486 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.487 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.487 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.840154+0000", 2022-07-20T03:17:50.487 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.487 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 2, 2022-07-20T03:17:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.488 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.773461+0000", 2022-07-20T03:17:50.489 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 3, 2022-07-20T03:17:50.490 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.491 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.811063+0000", 2022-07-20T03:17:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.492 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 4, 2022-07-20T03:17:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.493 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.494 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:57.886289+0000", 2022-07-20T03:17:50.494 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 5, 2022-07-20T03:17:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.495 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.496 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.496 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.496 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.496 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.039047+0000", 2022-07-20T03:17:50.497 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.497 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.497 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.497 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 6, 2022-07-20T03:17:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.498 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.499 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.017635+0000", 2022-07-20T03:17:50.499 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.499 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.499 INFO:teuthology.orchestra.run.smithi061.stdout: { 2022-07-20T03:17:50.500 INFO:teuthology.orchestra.run.smithi061.stdout: "osd": 7, 2022-07-20T03:17:50.500 INFO:teuthology.orchestra.run.smithi061.stdout: "down_stamp": "0.000000", 2022-07-20T03:17:50.500 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_probability": 0, 2022-07-20T03:17:50.500 INFO:teuthology.orchestra.run.smithi061.stdout: "laggy_interval": 0, 2022-07-20T03:17:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: "features": 4540138292840890367, 2022-07-20T03:17:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: "old_weight": 0, 2022-07-20T03:17:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: "last_purged_snaps_scrub": "2022-07-20T03:06:58.001653+0000", 2022-07-20T03:17:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: "dead_epoch": 0 2022-07-20T03:17:50.501 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.502 INFO:teuthology.orchestra.run.smithi061.stdout: ], 2022-07-20T03:17:50.502 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap": [], 2022-07-20T03:17:50.502 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_upmap_items": [], 2022-07-20T03:17:50.502 INFO:teuthology.orchestra.run.smithi061.stdout: "pg_temp": [], 2022-07-20T03:17:50.503 INFO:teuthology.orchestra.run.smithi061.stdout: "primary_temp": [], 2022-07-20T03:17:50.503 INFO:teuthology.orchestra.run.smithi061.stdout: "blacklist": {}, 2022-07-20T03:17:50.503 INFO:teuthology.orchestra.run.smithi061.stdout: "erasure_code_profiles": { 2022-07-20T03:17:50.503 INFO:teuthology.orchestra.run.smithi061.stdout: "default": { 2022-07-20T03:17:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "crush-failure-domain": "osd", 2022-07-20T03:17:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "k": "2", 2022-07-20T03:17:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "m": "1", 2022-07-20T03:17:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "plugin": "jerasure", 2022-07-20T03:17:50.504 INFO:teuthology.orchestra.run.smithi061.stdout: "ruleset-failure-domain": "osd", 2022-07-20T03:17:50.505 INFO:teuthology.orchestra.run.smithi061.stdout: "technique": "reed_sol_van" 2022-07-20T03:17:50.505 INFO:teuthology.orchestra.run.smithi061.stdout: } 2022-07-20T03:17:50.505 INFO:teuthology.orchestra.run.smithi061.stdout: }, 2022-07-20T03:17:50.505 INFO:teuthology.orchestra.run.smithi061.stdout: "removed_snaps_queue": [], 2022-07-20T03:17:50.506 INFO:teuthology.orchestra.run.smithi061.stdout: "new_removed_snaps": [], 2022-07-20T03:17:50.506 INFO:teuthology.orchestra.run.smithi061.stdout: "new_purged_snaps": [], 2022-07-20T03:17:50.506 INFO:teuthology.orchestra.run.smithi061.stdout: "crush_node_flags": {}, 2022-07-20T03:17:50.506 INFO:teuthology.orchestra.run.smithi061.stdout: "device_class_flags": {} 2022-07-20T03:17:50.507 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T03:17:50.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9500539d0 msgr2=0x7fe950055e90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9500539d0 0x7fe950055e90 secure :-1 s=READY pgs=730 cs=0 l=1 rev1=1 rx=0x7fe96400d000 tx=0x7fe964007b40).stop 2022-07-20T03:17:50.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 msgr2=0x7fe9701407a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 0x7fe9701407a0 secure :-1 s=READY pgs=1522 cs=0 l=1 rev1=1 rx=0x7fe9680032f0 tx=0x7fe968016000).stop 2022-07-20T03:17:50.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 shutdown_connections 2022-07-20T03:17:50.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe9500539d0 0x7fe950055e90 unknown :-1 s=CLOSED pgs=730 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe9700b5650 0x7fe970144b00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe9700af8e0 0x7fe970144610 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 --2- 172.21.15.61:0/915064216 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe9700adff0 0x7fe9701407a0 unknown :-1 s=CLOSED pgs=1522 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 >> 172.21.15.61:0/915064216 conn(0x7fe9700634e0 msgr2=0x7fe9700afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:50.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 shutdown_connections 2022-07-20T03:17:50.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.219+0000 7fe977466700 1 -- 172.21.15.61:0/915064216 wait complete. 2022-07-20T03:17:50.512 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:50.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 Processor -- start 2022-07-20T03:17:50.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 -- start start 2022-07-20T03:17:50.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f133c0b5650 0x7f133c0b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f133c05bd90 0x7f133c142d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f133c0aea30 con 0x7f133c0b1200 2022-07-20T03:17:50.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c142970 con 0x7f133c05bd90 2022-07-20T03:17:50.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342c52700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f133c0aec60 con 0x7f133c0b5650 2022-07-20T03:17:50.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f133c05bd90 0x7f133c142d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f133c05bd90 0x7f133c142d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52034/0 (socket says 172.21.15.61:52034) 2022-07-20T03:17:50.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 -- 172.21.15.61:0/1555309638 learned_addr learned my addr 172.21.15.61:0/1555309638 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:50.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1341c50700 1 --2- 172.21.15.61:0/1555309638 >> v2:172.21.15.117:3300/0 conn(0x7f133c0b5650 0x7f133c0b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 -- 172.21.15.61:0/1555309638 >> v2:172.21.15.117:3300/0 conn(0x7f133c0b5650 msgr2=0x7f133c0b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:50.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 --2- 172.21.15.61:0/1555309638 >> v2:172.21.15.117:3300/0 conn(0x7f133c0b5650 0x7f133c0b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 -- 172.21.15.61:0/1555309638 >> v1:172.21.15.117:6789/0 conn(0x7f133c0b1200 legacy=0x7f133c05c4f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.619+0000 7f1342451700 1 -- 172.21.15.61:0/1555309638 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f133c0af180 con 0x7f133c05bd90 2022-07-20T03:17:50.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342451700 1 --2- 172.21.15.61:0/1555309638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c142d60 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f13340032f0 tx=0x7f1334009290).ready entity=mon.0 client_cookie=2c6ba0226a0ea256 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1340c4e700 1 -- 172.21.15.61:0/1555309638 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f133400b870 con 0x7f133c05bd90 2022-07-20T03:17:50.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1340c4e700 1 -- 172.21.15.61:0/1555309638 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f133400b9d0 con 0x7f133c05bd90 2022-07-20T03:17:50.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1340c4e700 1 -- 172.21.15.61:0/1555309638 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f133400bb40 con 0x7f133c05bd90 2022-07-20T03:17:50.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 -- 172.21.15.61:0/1555309638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 msgr2=0x7f133c142d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 --2- 172.21.15.61:0/1555309638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c142d60 secure :-1 s=READY pgs=1523 cs=0 l=1 rev1=1 rx=0x7f13340032f0 tx=0x7f1334009290).stop 2022-07-20T03:17:50.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 -- 172.21.15.61:0/1555309638 shutdown_connections 2022-07-20T03:17:50.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 --2- 172.21.15.61:0/1555309638 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c142d60 unknown :-1 s=CLOSED pgs=1523 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 --2- 172.21.15.61:0/1555309638 >> v2:172.21.15.117:3300/0 conn(0x7f133c0b5650 0x7f133c0b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 -- 172.21.15.61:0/1555309638 >> 172.21.15.61:0/1555309638 conn(0x7f133c0634e0 msgr2=0x7f133c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:50.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 -- 172.21.15.61:0/1555309638 shutdown_connections 2022-07-20T03:17:50.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 -- 172.21.15.61:0/1555309638 wait complete. 2022-07-20T03:17:50.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 Processor -- start 2022-07-20T03:17:50.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.623+0000 7f1342c52700 1 -- start start 2022-07-20T03:17:50.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c13d3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f133c0b1200 0x7f133c145110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f133c0b5650 0x7f133c140540 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c141d10 con 0x7f133c05bd90 2022-07-20T03:17:50.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f133c141fb0 con 0x7f133c0b5650 2022-07-20T03:17:50.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f133c142250 con 0x7f133c0b1200 2022-07-20T03:17:50.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c13d3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c13d3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52036/0 (socket says 172.21.15.61:52036) 2022-07-20T03:17:50.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 -- 172.21.15.61:0/1289346092 learned_addr learned my addr 172.21.15.61:0/1289346092 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:50.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f134144f700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f133c0b1200 0x7f133c145110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342451700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f133c0b5650 0x7f133c140540 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 -- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f133c0b1200 msgr2=0x7f133c145110 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f133c0b1200 0x7f133c145110 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 -- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f133c0b5650 msgr2=0x7f133c140540 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f133c0b5650 0x7f133c140540 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 -- 172.21.15.61:0/1289346092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1334009530 con 0x7f133c05bd90 2022-07-20T03:17:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f134144f700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f133c0b1200 0x7f133c145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1341c50700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c13d3c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f13380030b0 tx=0x7f1338008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1338011780 con 0x7f133c05bd90 2022-07-20T03:17:50.644 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f133c063b90 con 0x7f133c05bd90 2022-07-20T03:17:50.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f13380118e0 con 0x7f133c05bd90 2022-07-20T03:17:50.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f133c13db50 con 0x7f133c05bd90 2022-07-20T03:17:50.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1338011c40 con 0x7f133c05bd90 2022-07-20T03:17:50.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.627+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f133c13d950 con 0x7f133c05bd90 2022-07-20T03:17:50.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.631+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f133801c460 con 0x7f133c05bd90 2022-07-20T03:17:50.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.631+0000 7f132effd700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f131c053a40 0x7f131c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.631+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f133806c160 con 0x7f133c05bd90 2022-07-20T03:17:50.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.631+0000 7f134144f700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f131c053a40 0x7f131c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.631+0000 7f134144f700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f131c053a40 0x7f131c055f00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f133000b000 tx=0x7f1330009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.647 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.635+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f1338042fc0 con 0x7f133c05bd90 2022-07-20T03:17:50.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.799+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f133c141b00 con 0x7f133c05bd90 2022-07-20T03:17:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.799+0000 7f132effd700 1 -- 172.21.15.61:0/1289346092 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 83 v83) v1 ==== 94+0+2145 (secure 0 0 0) 0x7f13380465e0 con 0x7f133c05bd90 2022-07-20T03:17:50.805 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:50.805 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":83,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7397,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1660470104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1660470104},{"type":"v1","addr":"172.21.15.61:6833","nonce":1660470104}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":81}],"filesystems":[{"mdsmap":{"epoch":83,"flags":18,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:35.498516+0000","modified":"2022-07-20T03:17:36.503758+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":0,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{"mds_0":8665},"failed":[],"damaged":[],"stopped":[],"info":{"gid_8665":{"gid":8665,"name":"b","rank":0,"incarnation":82,"state":"up:active","state_seq":3,"addr":"172.21.15.117:6835/1280182602","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6834","nonce":1280182602},{"type":"v1","addr":"172.21.15.117:6835","nonce":1280182602}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0}},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T03:17:50.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f131c053a40 msgr2=0x7f131c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f131c053a40 0x7f131c055f00 secure :-1 s=READY pgs=731 cs=0 l=1 rev1=1 rx=0x7f133000b000 tx=0x7f1330009040).stop 2022-07-20T03:17:50.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 msgr2=0x7f133c13d3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c13d3c0 secure :-1 s=READY pgs=1524 cs=0 l=1 rev1=1 rx=0x7f13380030b0 tx=0x7f1338008040).stop 2022-07-20T03:17:50.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 shutdown_connections 2022-07-20T03:17:50.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f131c053a40 0x7f131c055f00 unknown :-1 s=CLOSED pgs=731 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f133c0b5650 0x7f133c140540 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f133c0b1200 0x7f133c145110 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 --2- 172.21.15.61:0/1289346092 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f133c05bd90 0x7f133c13d3c0 unknown :-1 s=CLOSED pgs=1524 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 >> 172.21.15.61:0/1289346092 conn(0x7f133c0634e0 msgr2=0x7f133c0b2e40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:50.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 shutdown_connections 2022-07-20T03:17:50.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.803+0000 7f1342c52700 1 -- 172.21.15.61:0/1289346092 wait complete. 2022-07-20T03:17:50.814 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 83 2022-07-20T03:17:50.818 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs fail cephfs 2022-07-20T03:17:50.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 Processor -- start 2022-07-20T03:17:50.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 -- start start 2022-07-20T03:17:50.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f979c05cda0 0x7f979c05d180 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f979c0b11b0 0x7f979c05fa50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c060bb0 0x7f979c147760 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c147090 con 0x7f979c05cda0 2022-07-20T03:17:50.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f979c099080 con 0x7f979c0b11b0 2022-07-20T03:17:50.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.919+0000 7f97a3acb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c1472c0 con 0x7f979c060bb0 2022-07-20T03:17:50.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f979c05cda0 0x7f979c05d180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f979c05cda0 0x7f979c05d180 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52044/0 (socket says 172.21.15.61:52044) 2022-07-20T03:17:50.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 -- 172.21.15.61:0/1665698466 learned_addr learned my addr 172.21.15.61:0/1665698466 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:50.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1066700 1 --2- 172.21.15.61:0/1665698466 >> v2:172.21.15.117:3300/0 conn(0x7f979c0b11b0 0x7f979c05fa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a2068700 1 --2- 172.21.15.61:0/1665698466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c060bb0 0x7f979c147760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 -- 172.21.15.61:0/1665698466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c060bb0 msgr2=0x7f979c147760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:50.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 --2- 172.21.15.61:0/1665698466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c060bb0 0x7f979c147760 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 -- 172.21.15.61:0/1665698466 >> v2:172.21.15.117:3300/0 conn(0x7f979c0b11b0 msgr2=0x7f979c05fa50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:50.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 --2- 172.21.15.61:0/1665698466 >> v2:172.21.15.117:3300/0 conn(0x7f979c0b11b0 0x7f979c05fa50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 -- 172.21.15.61:0/1665698466 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f979c05ff40 con 0x7f979c05cda0 2022-07-20T03:17:50.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1066700 1 --2- 172.21.15.61:0/1665698466 >> v2:172.21.15.117:3300/0 conn(0x7f979c0b11b0 0x7f979c05fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:50.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a1867700 1 --2- 172.21.15.61:0/1665698466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c05cda0 0x7f979c05d180 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f97940032f0 tx=0x7f9794009290).ready entity=mon.0 client_cookie=7b133e2c6c6284da server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a0865700 1 -- 172.21.15.61:0/1665698466 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f979400b870 con 0x7f979c05cda0 2022-07-20T03:17:50.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a0865700 1 -- 172.21.15.61:0/1665698466 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f979400b9d0 con 0x7f979c05cda0 2022-07-20T03:17:50.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a0865700 1 -- 172.21.15.61:0/1665698466 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f979400bb40 con 0x7f979c05cda0 2022-07-20T03:17:50.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 -- 172.21.15.61:0/1665698466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c05cda0 msgr2=0x7f979c05d180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 --2- 172.21.15.61:0/1665698466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c05cda0 0x7f979c05d180 secure :-1 s=READY pgs=1525 cs=0 l=1 rev1=1 rx=0x7f97940032f0 tx=0x7f9794009290).stop 2022-07-20T03:17:50.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 -- 172.21.15.61:0/1665698466 shutdown_connections 2022-07-20T03:17:50.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 --2- 172.21.15.61:0/1665698466 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c060bb0 0x7f979c147760 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 --2- 172.21.15.61:0/1665698466 >> v2:172.21.15.117:3300/0 conn(0x7f979c0b11b0 0x7f979c05fa50 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 --2- 172.21.15.61:0/1665698466 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c05cda0 0x7f979c05d180 unknown :-1 s=CLOSED pgs=1525 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 -- 172.21.15.61:0/1665698466 >> 172.21.15.61:0/1665698466 conn(0x7f979c0b5bb0 msgr2=0x7f979c0b1b40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:50.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 -- 172.21.15.61:0/1665698466 shutdown_connections 2022-07-20T03:17:50.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 -- 172.21.15.61:0/1665698466 wait complete. 2022-07-20T03:17:50.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 Processor -- start 2022-07-20T03:17:50.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.923+0000 7f97a3acb700 1 -- start start 2022-07-20T03:17:50.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 0x7f979c156e50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c060bb0 0x7f979c14a700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f979c0b11b0 0x7f979c063ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c156250 con 0x7f979c060bb0 2022-07-20T03:17:50.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f979c156470 con 0x7f979c0b11b0 2022-07-20T03:17:50.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f979c156710 con 0x7f979c05cda0 2022-07-20T03:17:50.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1867700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 0x7f979c156e50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c060bb0 0x7f979c14a700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1867700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 0x7f979c156e50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38722/0 (socket says 172.21.15.61:38722) 2022-07-20T03:17:50.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1867700 1 -- 172.21.15.61:0/2505112686 learned_addr learned my addr 172.21.15.61:0/2505112686 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:50.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a2068700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f979c0b11b0 0x7f979c063ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 -- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 msgr2=0x7f979c156e50 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 0x7f979c156e50 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 -- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f979c0b11b0 msgr2=0x7f979c063ac0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:50.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f979c0b11b0 0x7f979c063ac0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:50.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 -- 172.21.15.61:0/2505112686 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9794009530 con 0x7f979c060bb0 2022-07-20T03:17:50.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1867700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 0x7f979c156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:50.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a1066700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c060bb0 0x7f979c14a700 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f978c00a000 tx=0x7f978c008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f978c013670 con 0x7f979c060bb0 2022-07-20T03:17:50.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f978c0137d0 con 0x7f979c060bb0 2022-07-20T03:17:50.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f979c1569b0 con 0x7f979c060bb0 2022-07-20T03:17:50.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f978c013b30 con 0x7f979c060bb0 2022-07-20T03:17:50.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.927+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f979c0650d0 con 0x7f979c060bb0 2022-07-20T03:17:50.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.931+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f979c156b40 con 0x7f979c060bb0 2022-07-20T03:17:50.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.931+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f978c01a030 con 0x7f979c060bb0 2022-07-20T03:17:50.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.931+0000 7f9792ffd700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f97840539b0 0x7f9784055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:50.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.931+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(149..149 src has 1..149) v4 ==== 6073+0+0 (secure 0 0 0) 0x7f978c06bb70 con 0x7f979c060bb0 2022-07-20T03:17:50.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.931+0000 7f97a1867700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f97840539b0 0x7f9784055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:50.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.931+0000 7f97a1867700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f97840539b0 0x7f9784055e70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f979401a000 tx=0x7f9794016000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:50.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:50.935+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f978c042ad0 con 0x7f979c060bb0 2022-07-20T03:17:51.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.099+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs fail", "fs_name": "cephfs"} v 0) v1 -- 0x7f979c0574e0 con 0x7f979c060bb0 2022-07-20T03:17:51.286 INFO:tasks.ceph.mon.a.smithi061.stderr:2022-07-20T03:17:51.283+0000 7f253721d700 -1 log_channel(cluster) log [ERR] : Health check failed: 1 filesystem is offline (MDS_ALL_DOWN) 2022-07-20T03:17:51.301 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.295+0000 7f9792ffd700 1 -- 172.21.15.61:0/2505112686 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs fail", "fs_name": "cephfs"}]=0 cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. v84) v1 ==== 161+0+0 (secure 0 0 0) 0x7f978c06d090 con 0x7f979c060bb0 2022-07-20T03:17:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f97840539b0 msgr2=0x7f9784055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f97840539b0 0x7f9784055e70 secure :-1 s=READY pgs=732 cs=0 l=1 rev1=1 rx=0x7f979401a000 tx=0x7f9794016000).stop 2022-07-20T03:17:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c060bb0 msgr2=0x7f979c14a700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c060bb0 0x7f979c14a700 secure :-1 s=READY pgs=1526 cs=0 l=1 rev1=1 rx=0x7f978c00a000 tx=0x7f978c008040).stop 2022-07-20T03:17:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 shutdown_connections 2022-07-20T03:17:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f97840539b0 0x7f9784055e70 unknown :-1 s=CLOSED pgs=732 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f979c0b11b0 0x7f979c063ac0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f979c060bb0 0x7f979c14a700 unknown :-1 s=CLOSED pgs=1526 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 --2- 172.21.15.61:0/2505112686 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f979c05cda0 0x7f979c156e50 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 >> 172.21.15.61:0/2505112686 conn(0x7f979c0b5bb0 msgr2=0x7f979c0afa40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:51.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 shutdown_connections 2022-07-20T03:17:51.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.299+0000 7f97a3acb700 1 -- 172.21.15.61:0/2505112686 wait complete. 2022-07-20T03:17:51.308 INFO:teuthology.orchestra.run.smithi061.stderr:cephfs marked not joinable; MDS cannot join the cluster. All MDS ranks marked failed. 2022-07-20T03:17:51.322 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs dump --format=json 2022-07-20T03:17:51.327 INFO:tasks.ceph.mds.b.smithi117.stdout:starting mds.b at 2022-07-20T03:17:51.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 Processor -- start 2022-07-20T03:17:51.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 -- start start 2022-07-20T03:17:51.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d380b5650 0x7f9d380b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d380adff0 0x7f9d38147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d38147010 con 0x7f9d380adff0 2022-07-20T03:17:51.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d381472b0 con 0x7f9d380b5650 2022-07-20T03:17:51.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3daa7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d38146d40 con 0x7f9d380af8e0 2022-07-20T03:17:51.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d37fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d380adff0 0x7f9d38147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.437 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.423+0000 7f9d3caa5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d380b5650 0x7f9d380b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d380b5650 0x7f9d380b5a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:33506/0 (socket says 172.21.15.61:33506) 2022-07-20T03:17:51.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 -- 172.21.15.61:0/1748986431 learned_addr learned my addr 172.21.15.61:0/1748986431 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:51.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3d2a6700 1 --2- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 -- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 msgr2=0x7f9d3805bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:51.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 --2- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3805bd90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.439 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 -- 172.21.15.61:0/1748986431 >> v2:172.21.15.61:3300/0 conn(0x7f9d380adff0 msgr2=0x7f9d38147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:51.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 --2- 172.21.15.61:0/1748986431 >> v2:172.21.15.61:3300/0 conn(0x7f9d380adff0 0x7f9d38147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 -- 172.21.15.61:0/1748986431 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d38147c40 con 0x7f9d380b5650 2022-07-20T03:17:51.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d37fff700 1 --2- 172.21.15.61:0/1748986431 >> v2:172.21.15.61:3300/0 conn(0x7f9d380adff0 0x7f9d38147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T03:17:51.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3caa5700 1 --2- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380b5650 0x7f9d380b5a30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f9d300032f0 tx=0x7f9d30009290).ready entity=mon.1 client_cookie=3825b98108d55f2e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:51.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d377fe700 1 -- 172.21.15.61:0/1748986431 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d3000b680 con 0x7f9d380b5650 2022-07-20T03:17:51.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d377fe700 1 -- 172.21.15.61:0/1748986431 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d3000b7e0 con 0x7f9d380b5650 2022-07-20T03:17:51.443 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d377fe700 1 -- 172.21.15.61:0/1748986431 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d3000b950 con 0x7f9d380b5650 2022-07-20T03:17:51.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 -- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380b5650 msgr2=0x7f9d380b5a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380b5650 0x7f9d380b5a30 secure :-1 s=READY pgs=679 cs=0 l=1 rev1=1 rx=0x7f9d300032f0 tx=0x7f9d30009290).stop 2022-07-20T03:17:51.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 -- 172.21.15.61:0/1748986431 shutdown_connections 2022-07-20T03:17:51.444 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3805bd90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/1748986431 >> v2:172.21.15.61:3300/0 conn(0x7f9d380adff0 0x7f9d38147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/1748986431 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380b5650 0x7f9d380b5a30 unknown :-1 s=CLOSED pgs=679 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 -- 172.21.15.61:0/1748986431 >> 172.21.15.61:0/1748986431 conn(0x7f9d380634e0 msgr2=0x7f9d380b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:51.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 -- 172.21.15.61:0/1748986431 shutdown_connections 2022-07-20T03:17:51.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.427+0000 7f9d3daa7700 1 -- 172.21.15.61:0/1748986431 wait complete. 2022-07-20T03:17:51.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 Processor -- start 2022-07-20T03:17:51.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 -- start start 2022-07-20T03:17:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380adff0 0x7f9d38143930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3810fbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 0x7f9d38064790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d38111c00 con 0x7f9d380b5650 2022-07-20T03:17:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d38064190 con 0x7f9d380adff0 2022-07-20T03:17:51.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d38067500 con 0x7f9d380af8e0 2022-07-20T03:17:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 0x7f9d38064790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 0x7f9d38064790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52068/0 (socket says 172.21.15.61:52068) 2022-07-20T03:17:51.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 -- 172.21.15.61:0/331663879 learned_addr learned my addr 172.21.15.61:0/331663879 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3caa5700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380adff0 0x7f9d38143930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d37fff700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3810fbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 -- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 msgr2=0x7f9d3810fbc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3810fbc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 -- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380adff0 msgr2=0x7f9d38143930 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380adff0 0x7f9d38143930 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 -- 172.21.15.61:0/331663879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c00b120 con 0x7f9d380b5650 2022-07-20T03:17:51.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3d2a6700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 0x7f9d38064790 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f9d2800c000 tx=0x7f9d28006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:51.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d28011750 con 0x7f9d380b5650 2022-07-20T03:17:51.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d280118b0 con 0x7f9d380b5650 2022-07-20T03:17:51.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d30009530 con 0x7f9d380b5650 2022-07-20T03:17:51.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d28011c10 con 0x7f9d380b5650 2022-07-20T03:17:51.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.431+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d38063990 con 0x7f9d380b5650 2022-07-20T03:17:51.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.435+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d38111a20 con 0x7f9d380b5650 2022-07-20T03:17:51.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.435+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d28011d70 con 0x7f9d380b5650 2022-07-20T03:17:51.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.435+0000 7f9d35ffb700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d180539b0 0x7f9d18055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.435+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f9d2806c1e0 con 0x7f9d380b5650 2022-07-20T03:17:51.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.435+0000 7f9d3caa5700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d180539b0 0x7f9d18055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.435+0000 7f9d3caa5700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d180539b0 0x7f9d18055e70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f9d30019000 tx=0x7f9d30017000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:51.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.443+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d280430f0 con 0x7f9d380b5650 2022-07-20T03:17:51.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.607+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs dump", "format": "json"} v 0) v1 -- 0x7f9d380574e0 con 0x7f9d380b5650 2022-07-20T03:17:51.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.607+0000 7f9d35ffb700 1 -- 172.21.15.61:0/331663879 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs dump", "format": "json"}]=0 dumped fsmap epoch 84 v84) v1 ==== 94+0+1782 (secure 0 0 0) 0x7f9d28046710 con 0x7f9d380b5650 2022-07-20T03:17:51.613 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T03:17:51.614 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":84,"default_fscid":15,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"feature_flags":{"enable_multiple":false,"ever_enabled_multiple":false},"standbys":[{"gid":7397,"name":"a","rank":-1,"incarnation":0,"state":"up:standby","state_seq":1,"addr":"172.21.15.61:6833/1660470104","addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6832","nonce":1660470104},{"type":"v1","addr":"172.21.15.61:6833","nonce":1660470104}]},"join_fscid":-1,"export_targets":[],"features":4540138292840890367,"flags":0,"epoch":81}],"filesystems":[{"mdsmap":{"epoch":84,"flags":19,"ever_allowed_features":0,"explicitly_allowed_features":0,"created":"2022-07-20T03:17:35.498516+0000","modified":"2022-07-20T03:17:51.286879+0000","tableserver":0,"root":0,"session_timeout":60,"session_autoclose":300,"min_compat_client":"0 (unknown)","max_file_size":1099511627776,"last_failure":0,"last_failure_osd_epoch":150,"compat":{"compat":{},"ro_compat":{},"incompat":{"feature_1":"base v0.20","feature_2":"client writeable ranges","feature_3":"default file layouts on dirs","feature_4":"dir inode in separate object","feature_5":"mds uses versioned encoding","feature_6":"dirfrag is stored in omap","feature_8":"no anchor table","feature_9":"file layout v2","feature_10":"snaprealm v2"}},"max_mds":1,"in":[0],"up":{},"failed":[0],"damaged":[],"stopped":[],"info":{},"data_pools":[30],"metadata_pool":29,"enabled":true,"fs_name":"cephfs","balancer":"","standby_count_wanted":0},"id":15}]} 2022-07-20T03:17:51.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d180539b0 msgr2=0x7f9d18055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d180539b0 0x7f9d18055e70 secure :-1 s=READY pgs=733 cs=0 l=1 rev1=1 rx=0x7f9d30019000 tx=0x7f9d30017000).stop 2022-07-20T03:17:51.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 msgr2=0x7f9d38064790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 0x7f9d38064790 secure :-1 s=READY pgs=1528 cs=0 l=1 rev1=1 rx=0x7f9d2800c000 tx=0x7f9d28006df0).stop 2022-07-20T03:17:51.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 shutdown_connections 2022-07-20T03:17:51.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d180539b0 0x7f9d18055e70 unknown :-1 s=CLOSED pgs=733 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d380b5650 0x7f9d38064790 unknown :-1 s=CLOSED pgs=1528 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380af8e0 0x7f9d3810fbc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 --2- 172.21.15.61:0/331663879 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d380adff0 0x7f9d38143930 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 >> 172.21.15.61:0/331663879 conn(0x7f9d380634e0 msgr2=0x7f9d3805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 shutdown_connections 2022-07-20T03:17:51.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.611+0000 7f9d3daa7700 1 -- 172.21.15.61:0/331663879 wait complete. 2022-07-20T03:17:51.619 INFO:teuthology.orchestra.run.smithi061.stderr:dumped fsmap epoch 84 2022-07-20T03:17:51.627 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph fs rm cephfs --yes-i-really-mean-it 2022-07-20T03:17:51.733 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.727+0000 7f738e2c0700 1 Processor -- start 2022-07-20T03:17:51.734 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.727+0000 7f738e2c0700 1 -- start start 2022-07-20T03:17:51.737 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f73880afb80 0x7f7388061080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.738 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f738814a750 con 0x7f7388060ca0 2022-07-20T03:17:51.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f73880b2840 con 0x7f7388097a80 2022-07-20T03:17:51.739 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f7388148430 con 0x7f73880afb80 2022-07-20T03:17:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7387fff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7388060ca0 0x7f738814ab80 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48670/0 (socket says 172.21.15.61:48670) 2022-07-20T03:17:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7387fff700 1 -- 172.21.15.61:0/3905405063 learned_addr learned my addr 172.21.15.61:0/3905405063 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T03:17:51.740 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738c85d700 1 --2- 172.21.15.61:0/3905405063 >> v2:172.21.15.117:3300/0 conn(0x7f73880afb80 0x7f7388061080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3582280053 0 0) 0x7f738814a750 con 0x7f7388060ca0 2022-07-20T03:17:51.741 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7374003260 con 0x7f7388060ca0 2022-07-20T03:17:51.742 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3027147245 0 0) 0x7f73880b2840 con 0x7f7388097a80 2022-07-20T03:17:51.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f738814a750 con 0x7f7388097a80 2022-07-20T03:17:51.743 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738c85d700 1 -- 172.21.15.61:0/3905405063 >> v1:172.21.15.117:6789/0 conn(0x7f7388097a80 legacy=0x7f7388146850 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738c85d700 1 -- 172.21.15.61:0/3905405063 >> v1:172.21.15.61:6789/0 conn(0x7f7388060ca0 legacy=0x7f738814ab80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738c85d700 1 -- 172.21.15.61:0/3905405063 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f73880ac110 con 0x7f73880afb80 2022-07-20T03:17:51.744 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738c85d700 1 --2- 172.21.15.61:0/3905405063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73880afb80 0x7f7388061080 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f7380002f70 tx=0x7f7380006df0).ready entity=mon.1 client_cookie=95180df0e818bf24 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:51.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f738000d540 con 0x7f73880afb80 2022-07-20T03:17:51.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7380000e90 con 0x7f73880afb80 2022-07-20T03:17:51.745 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f7386ffd700 1 -- 172.21.15.61:0/3905405063 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f738000d890 con 0x7f73880afb80 2022-07-20T03:17:51.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- 172.21.15.61:0/3905405063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73880afb80 msgr2=0x7f7388061080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 --2- 172.21.15.61:0/3905405063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73880afb80 0x7f7388061080 secure :-1 s=READY pgs=681 cs=0 l=1 rev1=1 rx=0x7f7380002f70 tx=0x7f7380006df0).stop 2022-07-20T03:17:51.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- 172.21.15.61:0/3905405063 shutdown_connections 2022-07-20T03:17:51.746 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 --2- 172.21.15.61:0/3905405063 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f73880afb80 0x7f7388061080 unknown :-1 s=CLOSED pgs=681 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- 172.21.15.61:0/3905405063 >> 172.21.15.61:0/3905405063 conn(0x7f73880b5cb0 msgr2=0x7f73880b1c80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:51.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- 172.21.15.61:0/3905405063 shutdown_connections 2022-07-20T03:17:51.747 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.731+0000 7f738e2c0700 1 -- 172.21.15.61:0/3905405063 wait complete. 2022-07-20T03:17:51.750 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 Processor -- start 2022-07-20T03:17:51.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 -- start start 2022-07-20T03:17:51.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 0x7f73880aea70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388097a80 0x7f7388098220 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.751 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73880afb80 0x7f7388065f60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7388065b30 con 0x7f7388060ca0 2022-07-20T03:17:51.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7388099c60 con 0x7f7388097a80 2022-07-20T03:17:51.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7388099f00 con 0x7f73880afb80 2022-07-20T03:17:51.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 0x7f73880aea70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 0x7f73880aea70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52078/0 (socket says 172.21.15.61:52078) 2022-07-20T03:17:51.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 -- 172.21.15.61:0/1424668734 learned_addr learned my addr 172.21.15.61:0/1424668734 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:51.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f73877fe700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388097a80 0x7f7388098220 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738c85d700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73880afb80 0x7f7388065f60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 -- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73880afb80 msgr2=0x7f7388065f60 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73880afb80 0x7f7388065f60 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 -- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388097a80 msgr2=0x7f7388098220 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:51.754 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388097a80 0x7f7388098220 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:51.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 -- 172.21.15.61:0/1424668734 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f738000b180 con 0x7f7388060ca0 2022-07-20T03:17:51.755 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f7387fff700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 0x7f73880aea70 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f737c0011d0 tx=0x7f737c007b50).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:51.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f737c0070d0 con 0x7f7388060ca0 2022-07-20T03:17:51.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f737c007230 con 0x7f7388060ca0 2022-07-20T03:17:51.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f73880998e0 con 0x7f7388060ca0 2022-07-20T03:17:51.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f737c005610 con 0x7f7388060ca0 2022-07-20T03:17:51.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.735+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f738809b2f0 con 0x7f7388060ca0 2022-07-20T03:17:51.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.739+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f73880abe40 con 0x7f7388060ca0 2022-07-20T03:17:51.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.739+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f737c010070 con 0x7f7388060ca0 2022-07-20T03:17:51.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.739+0000 7f73857fa700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7368053ad0 0x7f7368055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:51.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.739+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f737c06b5a0 con 0x7f7388060ca0 2022-07-20T03:17:51.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.739+0000 7f73877fe700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7368053ad0 0x7f7368055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:51.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.739+0000 7f73877fe700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7368053ad0 0x7f7368055f90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f737800d000 tx=0x7f7378007da0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:51.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.747+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f737c0423a0 con 0x7f7388060ca0 2022-07-20T03:17:51.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:51.911+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true} v 0) v1 -- 0x7f7388099ad0 con 0x7f7388060ca0 2022-07-20T03:17:52.303 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f73857fa700 1 -- 172.21.15.61:0/1424668734 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "fs rm", "fs_name": "cephfs", "yes_i_really_mean_it": true}]=0 v85) v1 ==== 104+0+0 (secure 0 0 0) 0x7f737c018070 con 0x7f7388060ca0 2022-07-20T03:17:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7368053ad0 msgr2=0x7f7368055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7368053ad0 0x7f7368055f90 secure :-1 s=READY pgs=734 cs=0 l=1 rev1=1 rx=0x7f737800d000 tx=0x7f7378007da0).stop 2022-07-20T03:17:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 msgr2=0x7f73880aea70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:52.306 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 0x7f73880aea70 secure :-1 s=READY pgs=1530 cs=0 l=1 rev1=1 rx=0x7f737c0011d0 tx=0x7f737c007b50).stop 2022-07-20T03:17:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 shutdown_connections 2022-07-20T03:17:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7368053ad0 0x7f7368055f90 unknown :-1 s=CLOSED pgs=734 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7388097a80 0x7f7388098220 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.307 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f73880afb80 0x7f7388065f60 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 --2- 172.21.15.61:0/1424668734 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7388060ca0 0x7f73880aea70 unknown :-1 s=CLOSED pgs=1530 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 >> 172.21.15.61:0/1424668734 conn(0x7f73880b5cb0 msgr2=0x7f73880b4700 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.299+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 shutdown_connections 2022-07-20T03:17:52.308 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.303+0000 7f738e2c0700 1 -- 172.21.15.61:0/1424668734 wait complete. 2022-07-20T03:17:52.319 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_metadata cephfs_metadata --yes-i-really-really-mean-it 2022-07-20T03:17:52.445 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.439+0000 7f37f359e700 1 Processor -- start 2022-07-20T03:17:52.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.439+0000 7f37f359e700 1 -- start start 2022-07-20T03:17:52.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37f40b3cf0 0x7f37f4146040 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:52.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f4141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:52.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f37f4066b10 con 0x7f37f40b6840 2022-07-20T03:17:52.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f37f40b5c90 con 0x7f37f40b3cf0 2022-07-20T03:17:52.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40b6c20 con 0x7f37f4146530 2022-07-20T03:17:52.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37f40b3cf0 0x7f37f4146040 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:52.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f37f40b3cf0 0x7f37f4146040 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52086/0 (socket says 172.21.15.61:52086) 2022-07-20T03:17:52.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 -- 172.21.15.61:0/2253381354 learned_addr learned my addr 172.21.15.61:0/2253381354 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:52.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f2d9d700 1 --2- 172.21.15.61:0/2253381354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f4141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:52.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 -- 172.21.15.61:0/2253381354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 msgr2=0x7f37f4141a90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:52.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 --2- 172.21.15.61:0/2253381354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f4141a90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 -- 172.21.15.61:0/2253381354 >> v1:172.21.15.117:6789/0 conn(0x7f37f40b6840 legacy=0x7f37f40b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:52.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f1d9b700 1 -- 172.21.15.61:0/2253381354 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2647370860 0 0) 0x7f37f4066b10 con 0x7f37f40b6840 2022-07-20T03:17:52.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 -- 172.21.15.61:0/2253381354 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37f40aead0 con 0x7f37f40b3cf0 2022-07-20T03:17:52.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37e9d9b700 1 --2- 172.21.15.61:0/2253381354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f4146040 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7f37dc002f70 tx=0x7f37dc006eb0).ready entity=mon.0 client_cookie=39c5486fa718feef server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:52.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f1d9b700 1 -- 172.21.15.61:0/2253381354 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37dc00d570 con 0x7f37f40b3cf0 2022-07-20T03:17:52.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f1d9b700 1 -- 172.21.15.61:0/2253381354 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37dc000e90 con 0x7f37f40b3cf0 2022-07-20T03:17:52.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f1d9b700 1 -- 172.21.15.61:0/2253381354 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37dc00d8c0 con 0x7f37f40b3cf0 2022-07-20T03:17:52.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- 172.21.15.61:0/2253381354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 msgr2=0x7f37f4146040 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:52.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 --2- 172.21.15.61:0/2253381354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f4146040 secure :-1 s=READY pgs=1531 cs=0 l=1 rev1=1 rx=0x7f37dc002f70 tx=0x7f37dc006eb0).stop 2022-07-20T03:17:52.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- 172.21.15.61:0/2253381354 shutdown_connections 2022-07-20T03:17:52.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 --2- 172.21.15.61:0/2253381354 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f4141a90 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 --2- 172.21.15.61:0/2253381354 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f4146040 unknown :-1 s=CLOSED pgs=1531 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- 172.21.15.61:0/2253381354 >> 172.21.15.61:0/2253381354 conn(0x7f37f40b9e10 msgr2=0x7f37f40b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:52.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- 172.21.15.61:0/2253381354 shutdown_connections 2022-07-20T03:17:52.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.443+0000 7f37f359e700 1 -- 172.21.15.61:0/2253381354 wait complete. 2022-07-20T03:17:52.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 Processor -- start 2022-07-20T03:17:52.458 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 -- start start 2022-07-20T03:17:52.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f40b0b50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:52.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37f40b6840 0x7f37f40aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:52.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f413cfb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:52.459 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40b1e00 con 0x7f37f40b3cf0 2022-07-20T03:17:52.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f4152080 con 0x7f37f40b6840 2022-07-20T03:17:52.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f359e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f37f40b1ab0 con 0x7f37f4146530 2022-07-20T03:17:52.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f40b0b50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:52.460 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f40b0b50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52092/0 (socket says 172.21.15.61:52092) 2022-07-20T03:17:52.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 -- 172.21.15.61:0/1809631331 learned_addr learned my addr 172.21.15.61:0/1809631331 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:52.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f2d9d700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f413cfb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:52.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37e9d9b700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37f40b6840 0x7f37f40aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:52.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 -- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 msgr2=0x7f37f413cfb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:52.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f413cfb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 -- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37f40b6840 msgr2=0x7f37f40aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:52.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37f40b6840 0x7f37f40aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:52.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 -- 172.21.15.61:0/1809631331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37dc00b1e0 con 0x7f37f40b3cf0 2022-07-20T03:17:52.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.447+0000 7f37f259c700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f40b0b50 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f37e4000ed0 tx=0x7f37e4007ac0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:52.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37e4012400 con 0x7f37f40b3cf0 2022-07-20T03:17:52.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f37e4012560 con 0x7f37f40b3cf0 2022-07-20T03:17:52.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f4151d20 con 0x7f37f40b3cf0 2022-07-20T03:17:52.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f37e40128c0 con 0x7f37f40b3cf0 2022-07-20T03:17:52.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f40b06e0 con 0x7f37f40b3cf0 2022-07-20T03:17:52.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4152240 con 0x7f37f40b3cf0 2022-07-20T03:17:52.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f37e4019030 con 0x7f37f40b3cf0 2022-07-20T03:17:52.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37ebfff700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37d80539b0 0x7f37d8055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:52.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(150..150 src has 1..150) v4 ==== 6159+0+0 (secure 0 0 0) 0x7f37e406b180 con 0x7f37f40b3cf0 2022-07-20T03:17:52.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.451+0000 7f37e9d9b700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37d80539b0 0x7f37d8055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:52.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.455+0000 7f37e9d9b700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37d80539b0 0x7f37d8055e70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f37dc01c000 tx=0x7f37dc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:52.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.459+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f37e40420a0 con 0x7f37f40b3cf0 2022-07-20T03:17:52.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:52.583+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f37f40aed70 con 0x7f37f40b3cf0 2022-07-20T03:17:53.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.307+0000 7f37ebfff700 1 -- 172.21.15.61:0/1809631331 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_metadata", "pool2": "cephfs_metadata", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_metadata' removed v151) v1 ==== 185+0+0 (secure 0 0 0) 0x7f37e40456c0 con 0x7f37f40b3cf0 2022-07-20T03:17:53.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37d80539b0 msgr2=0x7f37d8055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:53.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37d80539b0 0x7f37d8055e70 secure :-1 s=READY pgs=736 cs=0 l=1 rev1=1 rx=0x7f37dc01c000 tx=0x7f37dc015000).stop 2022-07-20T03:17:53.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 msgr2=0x7f37f40b0b50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:53.316 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f40b0b50 secure :-1 s=READY pgs=1532 cs=0 l=1 rev1=1 rx=0x7f37e4000ed0 tx=0x7f37e4007ac0).stop 2022-07-20T03:17:53.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 shutdown_connections 2022-07-20T03:17:53.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f37d80539b0 0x7f37d8055e70 unknown :-1 s=CLOSED pgs=736 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f37f40b6840 0x7f37f40aef70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f37f4146530 0x7f37f413cfb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 --2- 172.21.15.61:0/1809631331 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f37f40b3cf0 0x7f37f40b0b50 unknown :-1 s=CLOSED pgs=1532 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 >> 172.21.15.61:0/1809631331 conn(0x7f37f40b9e10 msgr2=0x7f37f40b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:53.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 shutdown_connections 2022-07-20T03:17:53.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.311+0000 7f37f359e700 1 -- 172.21.15.61:0/1809631331 wait complete. 2022-07-20T03:17:53.319 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_metadata' removed 2022-07-20T03:17:53.330 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd pool delete cephfs_data cephfs_data --yes-i-really-really-mean-it 2022-07-20T03:17:53.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.443+0000 7f4545a51700 1 Processor -- start 2022-07-20T03:17:53.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.443+0000 7f4545a51700 1 -- start start 2022-07-20T03:17:53.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f45400b3560 0x7f45401463d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:53.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f454005ce80 0x7f4540143080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4540061370 con 0x7f45400ab470 2022-07-20T03:17:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f45400b3940 con 0x7f45400b3560 2022-07-20T03:17:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4540061010 con 0x7f454005ce80 2022-07-20T03:17:53.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4544a4f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f45400b3560 0x7f45401463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:53.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4544a4f700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f45400b3560 0x7f45401463d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52100/0 (socket says 172.21.15.61:52100) 2022-07-20T03:17:53.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545250700 1 --2- 172.21.15.61:0/2623822243 >> v2:172.21.15.117:3300/0 conn(0x7f454005ce80 0x7f4540143080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:53.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4544a4f700 1 -- 172.21.15.61:0/2623822243 learned_addr learned my addr 172.21.15.61:0/2623822243 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:53.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545250700 1 -- 172.21.15.61:0/2623822243 >> v2:172.21.15.61:3300/0 conn(0x7f45400b3560 msgr2=0x7f45401463d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T03:17:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545250700 1 --2- 172.21.15.61:0/2623822243 >> v2:172.21.15.61:3300/0 conn(0x7f45400b3560 0x7f45401463d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545250700 1 -- 172.21.15.61:0/2623822243 >> v1:172.21.15.117:6789/0 conn(0x7f45400ab470 legacy=0x7f4540061a40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545250700 1 -- 172.21.15.61:0/2623822243 --> v2:172.21.15.117:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f45400615a0 con 0x7f454005ce80 2022-07-20T03:17:53.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545250700 1 --2- 172.21.15.61:0/2623822243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f4540143080 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f45380032f0 tx=0x7f45380091d0).ready entity=mon.1 client_cookie=866d1aafb033daa9 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:53.456 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f453f7fe700 1 -- 172.21.15.61:0/2623822243 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f453800b680 con 0x7f454005ce80 2022-07-20T03:17:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f453f7fe700 1 -- 172.21.15.61:0/2623822243 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f453800b7e0 con 0x7f454005ce80 2022-07-20T03:17:53.457 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f453f7fe700 1 -- 172.21.15.61:0/2623822243 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f453800b950 con 0x7f454005ce80 2022-07-20T03:17:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 -- 172.21.15.61:0/2623822243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 msgr2=0x7f4540143080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 --2- 172.21.15.61:0/2623822243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f4540143080 secure :-1 s=READY pgs=684 cs=0 l=1 rev1=1 rx=0x7f45380032f0 tx=0x7f45380091d0).stop 2022-07-20T03:17:53.461 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 -- 172.21.15.61:0/2623822243 shutdown_connections 2022-07-20T03:17:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 --2- 172.21.15.61:0/2623822243 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f4540143080 unknown :-1 s=CLOSED pgs=684 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 --2- 172.21.15.61:0/2623822243 >> v2:172.21.15.61:3300/0 conn(0x7f45400b3560 0x7f45401463d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.447+0000 7f4545a51700 1 -- 172.21.15.61:0/2623822243 >> 172.21.15.61:0/2623822243 conn(0x7f45400b6b20 msgr2=0x7f45400b2b10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:53.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.451+0000 7f4545a51700 1 -- 172.21.15.61:0/2623822243 shutdown_connections 2022-07-20T03:17:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.451+0000 7f4545a51700 1 -- 172.21.15.61:0/2623822243 wait complete. 2022-07-20T03:17:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.451+0000 7f4545a51700 1 Processor -- start 2022-07-20T03:17:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.451+0000 7f4545a51700 1 -- start start 2022-07-20T03:17:53.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.451+0000 7f4545a51700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f45400ade40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45400ab470 0x7f4540155f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 0x7f4540066790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f454013f3a0 con 0x7f45400b3560 2022-07-20T03:17:53.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f45401555a0 con 0x7f454005ce80 2022-07-20T03:17:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4540155840 con 0x7f45400ab470 2022-07-20T03:17:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 0x7f4540066790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 0x7f4540066790 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52110/0 (socket says 172.21.15.61:52110) 2022-07-20T03:17:53.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 -- 172.21.15.61:0/4279615412 learned_addr learned my addr 172.21.15.61:0/4279615412 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T03:17:53.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4544a4f700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f45400ade40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:53.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f453ffff700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45400ab470 0x7f4540155f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:53.470 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 -- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45400ab470 msgr2=0x7f4540155f80 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45400ab470 0x7f4540155f80 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.471 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 -- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 msgr2=0x7f45400ade40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f45400ade40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 -- 172.21.15.61:0/4279615412 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4538009470 con 0x7f45400b3560 2022-07-20T03:17:53.472 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4544a4f700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f45400ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T03:17:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545250700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 0x7f4540066790 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7f45380032f0 tx=0x7f4538016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:53.473 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4538021440 con 0x7f45400b3560 2022-07-20T03:17:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f45380215a0 con 0x7f45400b3560 2022-07-20T03:17:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4540155ae0 con 0x7f45400b3560 2022-07-20T03:17:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4538021900 con 0x7f45400b3560 2022-07-20T03:17:53.474 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4540099400 con 0x7f45400b3560 2022-07-20T03:17:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4540155ca0 con 0x7f45400b3560 2022-07-20T03:17:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.455+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4538027020 con 0x7f45400b3560 2022-07-20T03:17:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.459+0000 7f453dffb700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4520053a40 0x7f4520055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T03:17:53.475 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.459+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(151..151 src has 1..151) v4 ==== 5721+0+0 (secure 0 0 0) 0x7f453806d2a0 con 0x7f45400b3560 2022-07-20T03:17:53.476 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.459+0000 7f4544a4f700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4520053a40 0x7f4520055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T03:17:53.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.459+0000 7f4544a4f700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4520053a40 0x7f4520055f00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f4530008000 tx=0x7f4530008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T03:17:53.477 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.463+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f45380442c0 con 0x7f45400b3560 2022-07-20T03:17:53.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:53.595+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true} v 0) v1 -- 0x7f4540060d40 con 0x7f45400b3560 2022-07-20T03:17:54.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.311+0000 7f453dffb700 1 -- 172.21.15.61:0/4279615412 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd pool delete", "pool": "cephfs_data", "pool2": "cephfs_data", "yes_i_really_really_mean_it": true}]=0 pool 'cephfs_data' removed v152) v1 ==== 173+0+0 (secure 0 0 0) 0x7f45380478e0 con 0x7f45400b3560 2022-07-20T03:17:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4520053a40 msgr2=0x7f4520055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4520053a40 0x7f4520055f00 secure :-1 s=READY pgs=737 cs=0 l=1 rev1=1 rx=0x7f4530008000 tx=0x7f4530008040).stop 2022-07-20T03:17:54.325 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 msgr2=0x7f4540066790 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T03:17:54.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 0x7f4540066790 secure :-1 s=READY pgs=1533 cs=0 l=1 rev1=1 rx=0x7f45380032f0 tx=0x7f4538016000).stop 2022-07-20T03:17:54.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 shutdown_connections 2022-07-20T03:17:54.326 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4520053a40 0x7f4520055f00 unknown :-1 s=CLOSED pgs=737 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:54.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f45400b3560 0x7f4540066790 unknown :-1 s=CLOSED pgs=1533 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:54.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f45400ab470 0x7f4540155f80 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:54.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.315+0000 7f4545a51700 1 --2- 172.21.15.61:0/4279615412 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f454005ce80 0x7f45400ade40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T03:17:54.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.319+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 >> 172.21.15.61:0/4279615412 conn(0x7f45400b6b20 msgr2=0x7f45400b0a10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T03:17:54.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.319+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 shutdown_connections 2022-07-20T03:17:54.328 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T03:17:54.319+0000 7f4545a51700 1 -- 172.21.15.61:0/4279615412 wait complete. 2022-07-20T03:17:54.329 INFO:teuthology.orchestra.run.smithi061.stderr:pool 'cephfs_data' removed 2022-07-20T03:17:54.339 INFO:teuthology.orchestra.run:Running command with timeout 60 2022-07-20T03:17:54.341 DEBUG:teuthology.orchestra.run.smithi124:> cd /home/ubuntu/cephtest/mnt.0 && sudo bash -c 'stat -f .' 2022-07-20T03:18:07.141 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:07.150 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:07.155 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:07.159 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:07.163 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:24.452 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:18:24.453 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:18:24.454 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:18:37.220 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:37.224 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:37.228 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:37.231 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:37.234 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:18:54.354 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:18:54.364 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T03:19:07.307 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:07.311 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:07.315 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:07.320 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:07.324 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:37.394 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:37.400 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:37.405 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:37.410 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:19:37.413 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:07.493 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:07.499 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:07.504 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:07.508 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:07.511 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:37.567 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:37.571 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:37.574 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:37.578 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:20:37.582 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:07.670 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:07.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:07.680 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:07.684 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:07.688 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:37.745 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:37.753 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:37.756 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:37.759 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:21:37.762 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:07.811 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:07.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:07.823 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:07.829 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:07.842 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:37.897 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:37.903 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:37.907 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:37.916 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:22:37.923 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:08.007 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:08.011 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:08.015 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:08.017 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:08.027 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:38.133 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:38.149 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:38.158 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:38.163 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:23:38.170 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:08.234 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:08.243 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:08.246 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:08.257 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:08.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:38.322 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:38.377 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:38.400 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:38.423 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:24:38.429 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:08.491 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:08.501 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:08.505 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:08.512 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:08.515 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:38.567 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:38.574 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:38.579 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:38.589 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:25:38.594 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:08.677 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:08.686 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:08.699 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:08.717 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:08.735 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:38.802 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:38.807 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:38.814 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:38.820 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:26:38.824 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:08.891 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:08.904 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:08.907 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:08.910 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:08.913 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:38.969 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:38.983 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:38.992 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:39.003 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:27:39.008 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:09.079 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:09.084 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:09.088 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:09.092 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:09.095 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:39.184 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:39.213 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:39.217 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:39.234 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:28:39.245 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:09.300 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:09.306 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:09.309 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:09.312 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:09.315 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:39.370 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:39.378 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:39.388 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:39.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:29:39.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:09.463 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:09.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:09.480 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:09.487 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:09.495 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:39.574 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:39.584 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:39.591 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:39.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:30:39.618 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:09.667 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:09.670 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:09.678 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:09.689 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:09.693 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:39.753 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:39.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:39.779 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:39.784 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:31:39.806 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:09.874 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:09.889 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:09.894 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:09.897 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:09.900 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:39.953 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:39.958 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:39.976 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:39.979 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:32:39.983 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:10.050 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:10.055 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:10.059 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:10.062 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:10.068 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:24.461 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:33:24.464 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T03:33:24.468 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:33:24.471 INFO:teuthology.nuke.actions:Clearing teuthology firewall rules... 2022-07-20T03:33:24.472 DEBUG:teuthology.orchestra.run.smithi061:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:33:24.508 DEBUG:teuthology.orchestra.run.smithi117:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:33:24.536 DEBUG:teuthology.orchestra.run.smithi124:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:33:24.561 DEBUG:teuthology.orchestra.run.smithi163:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:33:24.592 DEBUG:teuthology.orchestra.run.smithi179:> sudo sh -c 'iptables-save | grep -v teuthology | iptables-restore' 2022-07-20T03:33:24.616 INFO:teuthology.nuke.actions:Cleared teuthology firewall rules. 2022-07-20T03:33:24.616 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:33:24.617 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:33:24.617 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:33:40.138 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:40.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:40.160 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:40.167 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:33:40.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:10.229 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:10.249 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:10.259 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:10.264 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:10.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:40.344 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:40.350 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:40.356 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:40.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:34:40.375 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:10.482 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:10.493 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:10.499 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:10.524 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:10.530 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:40.589 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:40.600 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:40.606 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:40.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:35:40.630 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:10.687 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:10.712 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:10.723 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:10.737 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:10.741 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:40.798 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:40.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:40.810 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:40.815 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:36:40.820 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:10.888 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:10.900 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:10.905 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:10.929 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:10.937 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:41.025 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:41.044 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:41.058 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:41.066 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:37:41.069 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:11.150 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:11.154 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:11.157 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:11.161 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:11.164 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:41.222 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:41.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:41.233 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:41.237 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:38:41.242 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:11.324 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:11.328 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:11.331 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:11.334 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:11.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:41.388 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:41.394 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:41.399 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:41.402 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:39:41.407 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:11.482 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:11.486 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:11.489 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:11.492 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:11.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:41.572 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:41.576 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:41.579 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:41.584 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:40:41.589 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:11.644 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:11.649 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:11.653 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:11.657 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:11.661 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:41.711 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:41.717 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:41.721 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:41.724 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:41:41.727 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:11.800 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:11.804 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:11.806 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:11.810 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:11.812 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:41.866 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:41.870 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:41.873 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:41.877 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:42:41.881 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:11.949 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:11.954 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:11.958 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:11.962 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:11.967 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:42.024 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:42.028 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:42.033 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:42.037 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:43:42.040 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:12.095 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:12.101 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:12.105 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:12.113 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:12.117 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:42.167 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:42.172 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:42.176 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:42.180 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:44:42.184 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:12.251 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:12.256 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:12.260 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:12.263 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:12.268 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:42.317 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:42.327 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:42.331 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:42.337 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:45:42.341 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:12.397 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:12.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:12.404 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:12.407 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:12.412 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:42.466 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:42.500 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:42.523 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:42.527 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:46:42.538 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:12.589 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:12.594 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:12.603 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:12.609 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:12.612 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:42.661 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:42.664 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:42.674 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:42.678 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:47:42.681 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:12.779 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:12.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:12.787 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:12.790 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:12.792 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:24.631 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T03:48:24.633 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T03:48:24.634 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T03:48:24.637 INFO:tasks.cephfs_test_runner:test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) ... ERROR 2022-07-20T03:48:24.638 INFO:tasks.cephfs_test_runner:ERROR 2022-07-20T03:48:24.639 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.639 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:48:24.639 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:48:24.640 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:48:24.640 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.640 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:48:24.641 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:48:24.641 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:48:24.642 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:48:24.642 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:48:24.642 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.643 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:48:24.643 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.643 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.644 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-20T03:48:24.644 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-20T03:48:24.644 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-20T03:48:24.644 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-20T03:48:24.645 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:48:24.645 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:48:24.645 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:48:24.646 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:48:24.646 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:48:24.646 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:48:24.646 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:48:24.647 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:48:24.648 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:48:24.648 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:48:24.648 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:48:24.649 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:48:24.649 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:48:24.649 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:48:24.650 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:48:24.650 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:48:24.651 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:48:24.651 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:48:24.651 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:48:24.651 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:48:24.652 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:48:24.652 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:48:24.653 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.653 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:48:24.653 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:48:24.654 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:48:24.654 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.654 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:48:24.654 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:48:24.655 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:48:24.655 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:48:24.655 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:48:24.656 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.656 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:48:24.656 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.657 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.657 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-20T03:48:24.657 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-20T03:48:24.657 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-20T03:48:24.658 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-20T03:48:24.658 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-20T03:48:24.658 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-20T03:48:24.659 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:48:24.659 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:48:24.659 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:48:24.659 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:48:24.660 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:48:24.660 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:48:24.660 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:48:24.661 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:48:24.661 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:48:24.661 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:48:24.662 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:48:24.662 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:48:24.662 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:48:24.662 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:48:24.663 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:48:24.663 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:48:24.663 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:48:24.664 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:48:24.664 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:48:24.665 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:48:24.665 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:48:24.665 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:48:24.665 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.666 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:48:24.666 INFO:tasks.cephfs_test_runner:Ran 12 tests in 2439.771s 2022-07-20T03:48:24.667 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.667 INFO:tasks.cephfs_test_runner:FAILED (errors=2) 2022-07-20T03:48:24.667 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.667 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:48:24.668 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:48:24.668 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:48:24.668 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.669 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:48:24.669 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:48:24.669 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:48:24.670 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:48:24.670 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:48:24.670 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.671 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:48:24.671 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.671 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.671 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/test_misc.py", line 28, in test_statfs_on_deleted_fs 2022-07-20T03:48:24.672 INFO:tasks.cephfs_test_runner: self.mount_a.umount_wait(force=True) 2022-07-20T03:48:24.672 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait 2022-07-20T03:48:24.672 INFO:tasks.cephfs_test_runner: if not (self.is_mounted() and self.fuse_daemon): 2022-07-20T03:48:24.673 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:48:24.673 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:48:24.673 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:48:24.673 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:48:24.674 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:48:24.674 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:48:24.674 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:48:24.675 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:48:24.675 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:48:24.675 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:48:24.676 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:48:24.676 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:48:24.676 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:48:24.677 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:48:24.677 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:48:24.677 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:48:24.677 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:48:24.678 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:48:24.678 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:48:24.678 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:48:24.679 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:48:24.679 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:48:24.680 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.680 INFO:tasks.cephfs_test_runner:====================================================================== 2022-07-20T03:48:24.680 INFO:tasks.cephfs_test_runner:ERROR: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:48:24.681 INFO:tasks.cephfs_test_runner:---------------------------------------------------------------------- 2022-07-20T03:48:24.681 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.681 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr 2022-07-20T03:48:24.682 INFO:tasks.cephfs_test_runner: out = self.in_stderr_buffer.read(nbytes, self.timeout) 2022-07-20T03:48:24.682 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read 2022-07-20T03:48:24.682 INFO:tasks.cephfs_test_runner: raise PipeTimeout() 2022-07-20T03:48:24.683 INFO:tasks.cephfs_test_runner:paramiko.buffered_pipe.PipeTimeout 2022-07-20T03:48:24.683 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.683 INFO:tasks.cephfs_test_runner:During handling of the above exception, another exception occurred: 2022-07-20T03:48:24.684 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.684 INFO:tasks.cephfs_test_runner:Traceback (most recent call last): 2022-07-20T03:48:24.684 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/cephfs_test_case.py", line 175, in tearDown 2022-07-20T03:48:24.684 INFO:tasks.cephfs_test_runner: m.teardown() 2022-07-20T03:48:24.685 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 408, in teardown 2022-07-20T03:48:24.685 INFO:tasks.cephfs_test_runner: self.umount() 2022-07-20T03:48:24.685 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 261, in umount 2022-07-20T03:48:24.686 INFO:tasks.cephfs_test_runner: if not self.is_mounted(): 2022-07-20T03:48:24.686 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted 2022-07-20T03:48:24.686 INFO:tasks.cephfs_test_runner: proc.wait() 2022-07-20T03:48:24.687 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait 2022-07-20T03:48:24.687 INFO:tasks.cephfs_test_runner: greenlet.get(block=True,timeout=60) 2022-07-20T03:48:24.687 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get 2022-07-20T03:48:24.687 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception 2022-07-20T03:48:24.688 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise 2022-07-20T03:48:24.688 INFO:tasks.cephfs_test_runner: raise value.with_traceback(tb) 2022-07-20T03:48:24.688 INFO:tasks.cephfs_test_runner: File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run 2022-07-20T03:48:24.689 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to 2022-07-20T03:48:24.689 INFO:tasks.cephfs_test_runner: copy_to_log(src, logger, capture=stream, quiet=quiet) 2022-07-20T03:48:24.689 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log 2022-07-20T03:48:24.689 INFO:tasks.cephfs_test_runner: for line in f: 2022-07-20T03:48:24.690 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ 2022-07-20T03:48:24.690 INFO:tasks.cephfs_test_runner: line = self.readline() 2022-07-20T03:48:24.690 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline 2022-07-20T03:48:24.691 INFO:tasks.cephfs_test_runner: new_data = self._read(n) 2022-07-20T03:48:24.691 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read 2022-07-20T03:48:24.691 INFO:tasks.cephfs_test_runner: return self.channel.recv_stderr(size) 2022-07-20T03:48:24.691 INFO:tasks.cephfs_test_runner: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr 2022-07-20T03:48:24.692 INFO:tasks.cephfs_test_runner: raise socket.timeout() 2022-07-20T03:48:24.692 INFO:tasks.cephfs_test_runner:socket.timeout 2022-07-20T03:48:24.692 INFO:tasks.cephfs_test_runner: 2022-07-20T03:48:24.693 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:48:24.817 ERROR:teuthology.run_tasks: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=561a546f06b543fb9cc1e018b9790ed6 Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 106, in run_tasks manager.__enter__() File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs_test_runner.py", line 211, in task raise RuntimeError("Test failure: {0}".format(", ".join(bad_tests))) RuntimeError: Test failure: test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc), test_statfs_on_deleted_fs (tasks.cephfs.test_misc.TestMisc) 2022-07-20T03:48:24.819 DEBUG:teuthology.run_tasks:Unwinding manager cephfs_test_runner 2022-07-20T03:48:24.828 DEBUG:teuthology.run_tasks:Unwinding manager ceph-fuse 2022-07-20T03:48:24.854 INFO:tasks.ceph_fuse:Unmounting ceph-fuse clients... 2022-07-20T03:48:24.855 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T03:48:24.856 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T03:48:24.856 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T03:48:43.034 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:43.038 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:43.041 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:43.043 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:48:43.046 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:13.115 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:13.119 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:13.122 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:13.125 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:13.128 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:43.181 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:43.185 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:43.187 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:43.190 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:49:43.193 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:13.249 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:13.253 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:13.256 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:13.259 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:13.261 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:43.314 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:43.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:43.321 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:43.324 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:50:43.327 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:13.393 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:13.397 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:13.400 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:13.404 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:13.408 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:43.468 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:43.471 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:43.474 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:43.476 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:51:43.479 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:13.531 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:13.535 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:13.539 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:13.541 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:13.544 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:43.600 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:43.605 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:43.608 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:43.612 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:52:43.615 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:13.681 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:13.688 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:13.696 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:13.700 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:13.703 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:43.772 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:43.784 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:43.788 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:43.791 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:53:43.795 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:13.853 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:13.859 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:13.862 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:13.867 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:13.871 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:43.920 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:43.930 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:43.934 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:43.938 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:54:43.941 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:14.003 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:14.008 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:14.011 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:14.014 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:14.018 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:44.091 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:44.096 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:44.099 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:44.104 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:55:44.108 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:14.159 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:14.163 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:14.166 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:14.169 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:14.171 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:44.243 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:44.248 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:44.255 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:44.260 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:56:44.264 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:14.310 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:14.315 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:14.318 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:14.322 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:14.325 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:44.380 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:44.385 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:44.388 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:44.391 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:57:44.394 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:14.479 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:14.483 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:14.488 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:14.493 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:14.496 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:44.608 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:44.612 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:44.615 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:44.618 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:58:44.622 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:14.676 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:14.680 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:14.684 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:14.688 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:14.692 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:44.819 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:44.822 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:44.825 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:44.828 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T03:59:44.831 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:14.912 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:14.916 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:14.919 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:14.922 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:14.924 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:45.013 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:45.017 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:45.020 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:45.022 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:00:45.029 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:15.117 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:15.124 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:15.130 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:15.137 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:15.144 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:45.190 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:45.194 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:45.197 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:45.200 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:01:45.202 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:15.313 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:15.318 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:15.323 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:15.327 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:15.331 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:45.388 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:45.392 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:45.395 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:45.398 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:02:45.401 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:15.470 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:15.475 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:15.478 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:15.481 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:15.483 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:24.883 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:03:24.884 INFO:teuthology.orchestra.run.smithi124.stderr:stat: cannot read file system information for '/home/ubuntu/cephtest/mnt.0': Connection timed out 2022-07-20T04:03:24.886 WARNING:tasks.cephfs.fuse_mount:Found stale moutn point at /home/ubuntu/cephtest/mnt.0 2022-07-20T04:03:24.887 INFO:teuthology.orchestra.run:Running command with timeout 900 2022-07-20T04:03:24.887 DEBUG:teuthology.orchestra.run.smithi124:> (cd /home/ubuntu/cephtest && exec stat --file-system '--printf=%T 2022-07-20T04:03:24.888 DEBUG:teuthology.orchestra.run.smithi124:> ' -- /home/ubuntu/cephtest/mnt.0) 2022-07-20T04:03:45.763 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:45.767 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:45.770 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:45.772 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:03:45.775 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:15.835 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:15.839 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:15.842 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:15.845 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:15.848 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:45.901 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:45.905 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:45.910 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:45.913 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:04:45.915 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:15.995 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:16.002 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:16.005 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:16.008 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:16.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:46.065 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:46.071 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:46.075 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:46.079 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:05:46.082 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:16.173 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:16.179 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:16.183 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:16.187 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:16.194 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:46.320 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:46.326 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:46.330 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:46.333 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:06:46.336 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:16.396 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:16.401 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:16.405 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:16.408 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:16.411 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:46.755 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:46.759 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:46.762 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:46.765 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:07:46.768 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:17.001 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:17.005 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:17.008 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:17.010 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:17.014 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:47.074 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:47.079 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:47.083 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:47.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:08:47.089 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:17.185 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:17.189 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:17.192 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:17.195 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:17.198 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:47.252 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:47.258 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:47.262 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:47.266 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:09:47.269 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:17.421 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:17.426 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:17.430 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:17.433 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:17.436 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:47.541 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:47.557 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:47.561 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:47.565 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:10:47.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:17.694 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:17.699 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:17.702 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:17.705 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:17.708 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:47.815 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:47.818 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:47.821 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:47.824 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:11:47.827 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:17.937 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:17.941 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:17.944 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:17.947 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:17.950 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:48.082 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:48.086 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:48.089 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:48.094 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:12:48.097 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:18.219 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:18.225 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:18.231 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:18.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:18.239 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:48.556 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:48.560 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:48.564 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:48.567 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:13:48.570 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:18.773 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:18.778 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:18.785 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:18.788 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:18.805 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:48.922 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:48.928 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:48.933 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:48.947 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:14:48.963 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:19.077 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:19.080 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:19.083 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:19.085 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:19.088 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:49.223 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:49.228 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:49.231 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:49.235 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:15:49.238 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:19.348 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:19.354 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:19.359 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:19.366 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:19.370 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:49.480 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:49.485 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:49.488 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:49.491 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:16:49.493 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:19.610 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:19.616 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:19.619 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:19.623 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:19.627 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:49.738 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:49.742 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:49.745 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:49.748 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:17:49.752 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:20.018 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:20.022 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:20.024 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:20.027 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:20.030 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:24.893 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T04:18:24.917 ERROR:teuthology:Uncaught exception (Hub) Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 699, in recv out = self.in_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1361, in _read return self.channel.recv(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 701, in recv raise socket.timeout() socket.timeout 2022-07-20T04:18:24.952 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:18:24.955 ERROR:teuthology.run_tasks:Manager failed: ceph-fuse Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T04:18:24.972 DEBUG:teuthology.run_tasks:Unwinding manager ceph 2022-07-20T04:18:24.984 INFO:tasks.ceph.ceph_manager.ceph:Canceling any pending splits or merges... 2022-07-20T04:18:24.985 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T04:18:25.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 Processor -- start 2022-07-20T04:18:25.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 -- start start 2022-07-20T04:18:25.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d3813c790 0x7f9d380b4710 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f9d3813cb70 0x7f9d38143000 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f9d38142ba0 con 0x7f9d380b3ca0 2022-07-20T04:18:25.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d381429b0 con 0x7f9d3813c790 2022-07-20T04:18:25.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3d7ac700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f9d38146330 con 0x7f9d3813cb70 2022-07-20T04:18:25.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d3813c790 0x7f9d380b4710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f9d3813c790 0x7f9d380b4710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52174/0 (socket says 172.21.15.61:52174) 2022-07-20T04:18:25.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 -- 172.21.15.61:0/639168676 learned_addr learned my addr 172.21.15.61:0/639168676 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:25.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d3cfab700 1 --2- 172.21.15.61:0/639168676 >> v2:172.21.15.117:3300/0 conn(0x7f9d3813cb70 0x7f9d38143000 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 -- 172.21.15.61:0/639168676 >> v2:172.21.15.117:3300/0 conn(0x7f9d3813cb70 msgr2=0x7f9d38143000 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:25.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 --2- 172.21.15.61:0/639168676 >> v2:172.21.15.117:3300/0 conn(0x7f9d3813cb70 0x7f9d38143000 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 -- 172.21.15.61:0/639168676 >> v1:172.21.15.117:6789/0 conn(0x7f9d380b3ca0 legacy=0x7f9d380b9a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 -- 172.21.15.61:0/639168676 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d380b4c00 con 0x7f9d3813c790 2022-07-20T04:18:25.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d37fff700 1 --2- 172.21.15.61:0/639168676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d380b4710 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f9d2c002f70 tx=0x7f9d2c006eb0).ready entity=mon.0 client_cookie=8f29fa06dd16396b server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d36ffd700 1 -- 172.21.15.61:0/639168676 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d2c00d5c0 con 0x7f9d3813c790 2022-07-20T04:18:25.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d36ffd700 1 -- 172.21.15.61:0/639168676 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d2c000e90 con 0x7f9d3813c790 2022-07-20T04:18:25.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.112+0000 7f9d36ffd700 1 -- 172.21.15.61:0/639168676 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d2c00d950 con 0x7f9d3813c790 2022-07-20T04:18:25.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/639168676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 msgr2=0x7f9d380b4710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/639168676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d380b4710 secure :-1 s=READY pgs=1534 cs=0 l=1 rev1=1 rx=0x7f9d2c002f70 tx=0x7f9d2c006eb0).stop 2022-07-20T04:18:25.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/639168676 shutdown_connections 2022-07-20T04:18:25.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/639168676 >> v2:172.21.15.117:3300/0 conn(0x7f9d3813cb70 0x7f9d38143000 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/639168676 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d380b4710 unknown :-1 s=CLOSED pgs=1534 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/639168676 >> 172.21.15.61:0/639168676 conn(0x7f9d380678b0 msgr2=0x7f9d380b9300 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:25.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/639168676 shutdown_connections 2022-07-20T04:18:25.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/639168676 wait complete. 2022-07-20T04:18:25.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 Processor -- start 2022-07-20T04:18:25.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.116+0000 7f9d3d7ac700 1 -- start start 2022-07-20T04:18:25.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380b3ca0 0x7f9d38141b30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d381555d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d3813cb70 0x7f9d380adf00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3813e810 con 0x7f9d3813c790 2022-07-20T04:18:25.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d3813eee0 con 0x7f9d3813cb70 2022-07-20T04:18:25.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9d380b06a0 con 0x7f9d380b3ca0 2022-07-20T04:18:25.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d381555d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d381555d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52180/0 (socket says 172.21.15.61:52180) 2022-07-20T04:18:25.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 -- 172.21.15.61:0/1691096754 learned_addr learned my addr 172.21.15.61:0/1691096754 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:25.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3cfab700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d3813cb70 0x7f9d380adf00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d37fff700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380b3ca0 0x7f9d38141b30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 -- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380b3ca0 msgr2=0x7f9d38141b30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380b3ca0 0x7f9d38141b30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 -- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d3813cb70 msgr2=0x7f9d380adf00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.149 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d3813cb70 0x7f9d380adf00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.150 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 -- 172.21.15.61:0/1691096754 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d2c00b1e0 con 0x7f9d3813c790 2022-07-20T04:18:25.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d377fe700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d381555d0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f9d380608b0 tx=0x7f9d28007be0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d28007030 con 0x7f9d3813c790 2022-07-20T04:18:25.151 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9d28007190 con 0x7f9d3813c790 2022-07-20T04:18:25.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9d28006510 con 0x7f9d3813c790 2022-07-20T04:18:25.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d380b0940 con 0x7f9d3813c790 2022-07-20T04:18:25.152 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.120+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d381400e0 con 0x7f9d3813c790 2022-07-20T04:18:25.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.124+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d380574e0 con 0x7f9d3813c790 2022-07-20T04:18:25.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.124+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9d28010030 con 0x7f9d3813c790 2022-07-20T04:18:25.153 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.124+0000 7f9d357fa700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d18053a40 0x7f9d18055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.124+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f9d2806b790 con 0x7f9d3813c790 2022-07-20T04:18:25.154 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.124+0000 7f9d37fff700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d18053a40 0x7f9d18055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.124+0000 7f9d37fff700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d18053a40 0x7f9d18055f00 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f9d2c01c000 tx=0x7f9d2c015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.155 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.128+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9d280428c0 con 0x7f9d3813c790 2022-07-20T04:18:25.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.260+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7f9d380b3760 con 0x7f9d3813c790 2022-07-20T04:18:25.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.260+0000 7f9d357fa700 1 -- 172.21.15.61:0/1691096754 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+13370 (secure 0 0 0) 0x7f9d28045ee0 con 0x7f9d3813c790 2022-07-20T04:18:25.261 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T04:18:25.261 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":152,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:54.309017+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.117:6835/1280182602":"2022-07-21T03:17:51.103581+0000","172.21.15.117:6834/1280182602":"2022-07-21T03:17:51.103581+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T04:18:25.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d18053a40 msgr2=0x7f9d18055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d18053a40 0x7f9d18055f00 secure :-1 s=READY pgs=804 cs=0 l=1 rev1=1 rx=0x7f9d2c01c000 tx=0x7f9d2c015000).stop 2022-07-20T04:18:25.264 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 msgr2=0x7f9d381555d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d381555d0 secure :-1 s=READY pgs=1535 cs=0 l=1 rev1=1 rx=0x7f9d380608b0 tx=0x7f9d28007be0).stop 2022-07-20T04:18:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 shutdown_connections 2022-07-20T04:18:25.265 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9d18053a40 0x7f9d18055f00 unknown :-1 s=CLOSED pgs=804 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f9d3813cb70 0x7f9d380adf00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.266 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f9d3813c790 0x7f9d381555d0 unknown :-1 s=CLOSED pgs=1535 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 --2- 172.21.15.61:0/1691096754 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f9d380b3ca0 0x7f9d38141b30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 >> 172.21.15.61:0/1691096754 conn(0x7f9d380678b0 msgr2=0x7f9d380b8080 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:25.267 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 shutdown_connections 2022-07-20T04:18:25.268 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.264+0000 7f9d3d7ac700 1 -- 172.21.15.61:0/1691096754 wait complete. 2022-07-20T04:18:25.279 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T04:18:25.383 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 Processor -- start 2022-07-20T04:18:25.384 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- start start 2022-07-20T04:18:25.387 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f85c00b5650 0x7f85c00b5a30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0147750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f85c00af8e0 0x7f85c005bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c0147010 con 0x7f85c00af8e0 2022-07-20T04:18:25.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f85c01472b0 con 0x7f85c00b5650 2022-07-20T04:18:25.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c0146d40 con 0x7f85c00adff0 2022-07-20T04:18:25.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f85c00af8e0 0x7f85c005bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f85c00af8e0 0x7f85c005bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52188/0 (socket says 172.21.15.61:52188) 2022-07-20T04:18:25.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 -- 172.21.15.61:0/349561610 learned_addr learned my addr 172.21.15.61:0/349561610 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:25.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85bf7fe700 1 --2- 172.21.15.61:0/349561610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0147750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85bffff700 1 --2- 172.21.15.61:0/349561610 >> v2:172.21.15.117:3300/0 conn(0x7f85c00b5650 0x7f85c00b5a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 -- 172.21.15.61:0/349561610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 msgr2=0x7f85c0147750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:25.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 --2- 172.21.15.61:0/349561610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0147750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 -- 172.21.15.61:0/349561610 >> v2:172.21.15.117:3300/0 conn(0x7f85c00b5650 msgr2=0x7f85c00b5a30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:25.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 --2- 172.21.15.61:0/349561610 >> v2:172.21.15.117:3300/0 conn(0x7f85c00b5650 0x7f85c00b5a30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 -- 172.21.15.61:0/349561610 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85c0147c40 con 0x7f85c00af8e0 2022-07-20T04:18:25.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c4e9b700 1 --2- 172.21.15.61:0/349561610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00af8e0 0x7f85c005bd90 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7f85b8002f70 tx=0x7f85b8006eb0).ready entity=mon.0 client_cookie=24f09540c9ebcf55 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85beffd700 1 -- 172.21.15.61:0/349561610 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85b800d570 con 0x7f85c00af8e0 2022-07-20T04:18:25.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85beffd700 1 -- 172.21.15.61:0/349561610 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8000e90 con 0x7f85c00af8e0 2022-07-20T04:18:25.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85beffd700 1 -- 172.21.15.61:0/349561610 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85b800d900 con 0x7f85c00af8e0 2022-07-20T04:18:25.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- 172.21.15.61:0/349561610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00af8e0 msgr2=0x7f85c005bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- 172.21.15.61:0/349561610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00af8e0 0x7f85c005bd90 secure :-1 s=READY pgs=1536 cs=0 l=1 rev1=1 rx=0x7f85b8002f70 tx=0x7f85b8006eb0).stop 2022-07-20T04:18:25.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- 172.21.15.61:0/349561610 shutdown_connections 2022-07-20T04:18:25.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- 172.21.15.61:0/349561610 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00af8e0 0x7f85c005bd90 unknown :-1 s=CLOSED pgs=1536 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- 172.21.15.61:0/349561610 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0147750 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 --2- 172.21.15.61:0/349561610 >> v2:172.21.15.117:3300/0 conn(0x7f85c00b5650 0x7f85c00b5a30 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.384+0000 7f85c569c700 1 -- 172.21.15.61:0/349561610 >> 172.21.15.61:0/349561610 conn(0x7f85c00634e0 msgr2=0x7f85c00b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:25.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 -- 172.21.15.61:0/349561610 shutdown_connections 2022-07-20T04:18:25.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 -- 172.21.15.61:0/349561610 wait complete. 2022-07-20T04:18:25.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 Processor -- start 2022-07-20T04:18:25.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 -- start start 2022-07-20T04:18:25.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0149120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85c00af8e0 0x7f85c0066f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 0x7f85c0067480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c01562a0 con 0x7f85c00b5650 2022-07-20T04:18:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f85c0149b40 con 0x7f85c00af8e0 2022-07-20T04:18:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.388+0000 7f85c569c700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f85c01564d0 con 0x7f85c00adff0 2022-07-20T04:18:25.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 0x7f85c0067480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bffff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0149120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 0x7f85c0067480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52196/0 (socket says 172.21.15.61:52196) 2022-07-20T04:18:25.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 -- 172.21.15.61:0/800131015 learned_addr learned my addr 172.21.15.61:0/800131015 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:25.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bf7fe700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85c00af8e0 0x7f85c0066f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 -- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 msgr2=0x7f85c0149120 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0149120 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 -- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85c00af8e0 msgr2=0x7f85c0066f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85c00af8e0 0x7f85c0066f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 -- 172.21.15.61:0/800131015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85b800b1e0 con 0x7f85c00b5650 2022-07-20T04:18:25.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c4e9b700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 0x7f85c0067480 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7f85b8002f70 tx=0x7f85b8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85b80116e0 con 0x7f85c00b5650 2022-07-20T04:18:25.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f85b8011840 con 0x7f85c00b5650 2022-07-20T04:18:25.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f85c0067970 con 0x7f85c00b5650 2022-07-20T04:18:25.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f85b8011ba0 con 0x7f85c00b5650 2022-07-20T04:18:25.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85c010fa90 con 0x7f85c00b5650 2022-07-20T04:18:25.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f85c01566d0 con 0x7f85c00b5650 2022-07-20T04:18:25.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f85b8025030 con 0x7f85c00b5650 2022-07-20T04:18:25.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bd7fa700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85980539b0 0x7f8598055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.392+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f85b806cd90 con 0x7f85c00b5650 2022-07-20T04:18:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.396+0000 7f85bffff700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85980539b0 0x7f8598055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.396+0000 7f85bffff700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85980539b0 0x7f8598055e70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f85a800b000 tx=0x7f85a8006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.400+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f85b8043ec0 con 0x7f85c00b5650 2022-07-20T04:18:25.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.524+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f85c00574e0 con 0x7f85980539b0 2022-07-20T04:18:25.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.528+0000 7f85bd7fa700 1 -- 172.21.15.61:0/800131015 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68201 (secure 0 0 0) 0x7f85c00574e0 con 0x7f85980539b0 2022-07-20T04:18:25.534 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T04:18:25.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85980539b0 msgr2=0x7f8598055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85980539b0 0x7f8598055e70 secure :-1 s=READY pgs=805 cs=0 l=1 rev1=1 rx=0x7f85a800b000 tx=0x7f85a8006df0).stop 2022-07-20T04:18:25.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 msgr2=0x7f85c0067480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 0x7f85c0067480 secure :-1 s=READY pgs=1537 cs=0 l=1 rev1=1 rx=0x7f85b8002f70 tx=0x7f85b8015000).stop 2022-07-20T04:18:25.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 shutdown_connections 2022-07-20T04:18:25.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f85980539b0 0x7f8598055e70 unknown :-1 s=CLOSED pgs=805 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f85c00af8e0 0x7f85c0066f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f85c00b5650 0x7f85c0067480 unknown :-1 s=CLOSED pgs=1537 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 --2- 172.21.15.61:0/800131015 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f85c00adff0 0x7f85c0149120 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 >> 172.21.15.61:0/800131015 conn(0x7f85c00634e0 msgr2=0x7f85c00afd70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:25.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 shutdown_connections 2022-07-20T04:18:25.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.532+0000 7f85c569c700 1 -- 172.21.15.61:0/800131015 wait complete. 2022-07-20T04:18:25.543 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-20T04:18:25.545 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2295,"stamp":"2022-07-20T04:18:24.199800+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385216,"kb_used_data":743209824,"kb_used_omap":27860,"kb_used_meta":0,"kb_avail":743209824,"statfs":{"total":767585320960,"available":761046859776,"internally_reserved":0,"allocated":761046859776,"data_stored":761046859776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28528982,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098255"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.839187+0000","last_change":"2022-07-20T03:07:00.764096+0000","last_active":"2022-07-20T03:17:53.839187+0000","last_peered":"2022-07-20T03:17:53.839187+0000","last_clean":"2022-07-20T03:17:53.839187+0000","last_became_active":"2022-07-20T03:07:00.763305+0000","last_became_peered":"2022-07-20T03:07:00.763305+0000","last_unstale":"2022-07-20T03:17:53.839187+0000","last_undegraded":"2022-07-20T03:17:53.839187+0000","last_fullsized":"2022-07-20T03:17:53.839187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_clean_scrub_stamp":"2022-07-20T03:06:54.339924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.311055+0000","last_change":"2022-07-20T03:07:06.947888+0000","last_active":"2022-07-20T03:17:53.311055+0000","last_peered":"2022-07-20T03:17:53.311055+0000","last_clean":"2022-07-20T03:17:53.311055+0000","last_became_active":"2022-07-20T03:07:06.947545+0000","last_became_peered":"2022-07-20T03:07:06.947545+0000","last_unstale":"2022-07-20T03:17:53.311055+0000","last_undegraded":"2022-07-20T03:17:53.311055+0000","last_fullsized":"2022-07-20T03:17:53.311055+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.839036+0000","last_change":"2022-07-20T03:07:06.947578+0000","last_active":"2022-07-20T03:17:53.839036+0000","last_peered":"2022-07-20T03:17:53.839036+0000","last_clean":"2022-07-20T03:17:53.839036+0000","last_became_active":"2022-07-20T03:07:06.947295+0000","last_became_peered":"2022-07-20T03:07:06.947295+0000","last_unstale":"2022-07-20T03:17:53.839036+0000","last_undegraded":"2022-07-20T03:17:53.839036+0000","last_fullsized":"2022-07-20T03:17:53.839036+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-20T03:17:51.534520+0000","last_change":"2022-07-20T03:07:06.951565+0000","last_active":"2022-07-20T03:17:51.534520+0000","last_peered":"2022-07-20T03:17:51.534520+0000","last_clean":"2022-07-20T03:17:51.534520+0000","last_became_active":"2022-07-20T03:07:06.951117+0000","last_became_peered":"2022-07-20T03:07:06.951117+0000","last_unstale":"2022-07-20T03:17:51.534520+0000","last_undegraded":"2022-07-20T03:17:51.534520+0000","last_fullsized":"2022-07-20T03:17:51.534520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.310985+0000","last_change":"2022-07-20T03:07:06.946761+0000","last_active":"2022-07-20T03:17:53.310985+0000","last_peered":"2022-07-20T03:17:53.310985+0000","last_clean":"2022-07-20T03:17:53.310985+0000","last_became_active":"2022-07-20T03:07:06.946278+0000","last_became_peered":"2022-07-20T03:07:06.946278+0000","last_unstale":"2022-07-20T03:17:53.310985+0000","last_undegraded":"2022-07-20T03:17:53.310985+0000","last_fullsized":"2022-07-20T03:17:53.310985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.311202+0000","last_change":"2022-07-20T03:07:06.951643+0000","last_active":"2022-07-20T03:17:53.311202+0000","last_peered":"2022-07-20T03:17:53.311202+0000","last_clean":"2022-07-20T03:17:53.311202+0000","last_became_active":"2022-07-20T03:07:06.951291+0000","last_became_peered":"2022-07-20T03:07:06.951291+0000","last_unstale":"2022-07-20T03:17:53.311202+0000","last_undegraded":"2022-07-20T03:17:53.311202+0000","last_fullsized":"2022-07-20T03:17:53.311202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.839347+0000","last_change":"2022-07-20T03:07:06.943629+0000","last_active":"2022-07-20T03:17:53.839347+0000","last_peered":"2022-07-20T03:17:53.839347+0000","last_clean":"2022-07-20T03:17:53.839347+0000","last_became_active":"2022-07-20T03:07:06.942970+0000","last_became_peered":"2022-07-20T03:07:06.942970+0000","last_unstale":"2022-07-20T03:17:53.839347+0000","last_undegraded":"2022-07-20T03:17:53.839347+0000","last_fullsized":"2022-07-20T03:17:53.839347+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.311325+0000","last_change":"2022-07-20T03:07:06.950534+0000","last_active":"2022-07-20T03:17:53.311325+0000","last_peered":"2022-07-20T03:17:53.311325+0000","last_clean":"2022-07-20T03:17:53.311325+0000","last_became_active":"2022-07-20T03:07:06.949930+0000","last_became_peered":"2022-07-20T03:07:06.949930+0000","last_unstale":"2022-07-20T03:17:53.311325+0000","last_undegraded":"2022-07-20T03:17:53.311325+0000","last_fullsized":"2022-07-20T03:17:53.311325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.326385+0000","last_change":"2022-07-20T03:07:06.946858+0000","last_active":"2022-07-20T03:17:53.326385+0000","last_peered":"2022-07-20T03:17:53.326385+0000","last_clean":"2022-07-20T03:17:53.326385+0000","last_became_active":"2022-07-20T03:07:06.946555+0000","last_became_peered":"2022-07-20T03:07:06.946555+0000","last_unstale":"2022-07-20T03:17:53.326385+0000","last_undegraded":"2022-07-20T03:17:53.326385+0000","last_fullsized":"2022-07-20T03:17:53.326385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575706,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3650,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3737996,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.83299999999999996,"15min":0.83799999999999997},"min":{"1min":0.56200000000000006,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.3049999999999999},"last":0.60999999999999999},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.84899999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.56599999999999995},"max":{"1min":1.1299999999999999,"5min":1.236,"15min":1.236},"last":0.49199999999999999}]},{"osd":1,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.61199999999999999,"5min":0.495,"15min":0.495},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.3660000000000001},"last":0.70199999999999996},{"interface":"front","average":{"1min":0.878,"5min":0.87,"15min":0.874},"min":{"1min":0.64300000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.107,"5min":1.2110000000000001,"15min":1.248},"last":0.75800000000000001}]},{"osd":2,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.84499999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.60899999999999999},"max":{"1min":1.0329999999999999,"5min":1.482,"15min":1.482},"last":0.95499999999999996},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91200000000000003,"15min":0.90900000000000003},"min":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.56799999999999995},"max":{"1min":1.232,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.97899999999999998}]},{"osd":3,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89100000000000001,"15min":0.872},"min":{"1min":0.67300000000000004,"5min":0.65500000000000003,"15min":0.61599999999999999},"max":{"1min":1.0249999999999999,"5min":1.413,"15min":1.413},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.90400000000000003,"15min":0.92300000000000004},"min":{"1min":0.68700000000000006,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1639999999999999,"5min":1.1799999999999999,"15min":1.2769999999999999},"last":1.1080000000000001}]},{"osd":4,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.81599999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"max":{"1min":1.0109999999999999,"5min":1.181,"15min":1.181},"last":1.0740000000000001},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.86799999999999999,"15min":0.88300000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.1279999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":1.149}]},{"osd":5,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":0.91100000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.54900000000000004},"max":{"1min":1.1220000000000001,"5min":1.2370000000000001,"15min":1.3169999999999999},"last":1.2130000000000001},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.93899999999999995},"min":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.415},"last":0.93100000000000005}]},{"osd":6,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.621},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3540000000000001},"last":1.1879999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.97399999999999998},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.627},"max":{"1min":1.173,"5min":1.532,"15min":1.532},"last":1.4610000000000001}]}]},{"osd":6,"up_from":13,"seq":55834575707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":4005,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4101914,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.86699999999999999,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.1799999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.83999999999999997,"15min":0.85099999999999998},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.0529999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.66700000000000004}]},{"osd":1,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81899999999999995,"15min":0.81000000000000005},"min":{"1min":0.626,"5min":0.59599999999999997,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":1.149,"15min":1.2030000000000001},"last":0.91900000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.86699999999999999,"15min":0.86499999999999999},"min":{"1min":0.73899999999999999,"5min":0.625,"15min":0.56499999999999995},"max":{"1min":1.0389999999999999,"5min":1.1140000000000001,"15min":1.323},"last":0.80900000000000005}]},{"osd":2,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.877,"15min":0.86199999999999999},"min":{"1min":0.66400000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.081,"5min":1.1579999999999999,"15min":1.161},"last":0.75700000000000001},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85099999999999998,"15min":0.84799999999999998},"min":{"1min":0.65600000000000003,"5min":0.63900000000000001,"15min":0.61599999999999999},"max":{"1min":1.1240000000000001,"5min":1.1259999999999999,"15min":1.129},"last":0.85099999999999998}]},{"osd":3,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.875,"15min":0.86899999999999999},"min":{"1min":0.71199999999999997,"5min":0.66700000000000004,"15min":0.63100000000000001},"max":{"1min":1.0649999999999999,"5min":1.179,"15min":1.179},"last":0.83099999999999996},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86699999999999999,"15min":0.88},"min":{"1min":0.76000000000000001,"5min":0.66800000000000004,"15min":0.63500000000000001},"max":{"1min":1.0760000000000001,"5min":1.1160000000000001,"15min":1.2110000000000001},"last":0.70599999999999996}]},{"osd":4,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78300000000000003,"15min":0.79700000000000004},"min":{"1min":0.55800000000000005,"5min":0.54300000000000004,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.218},"last":0.63200000000000001},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.88400000000000001,"15min":0.878},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.056,"5min":1.236,"15min":1.2969999999999999},"last":0.79300000000000004}]},{"osd":5,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85499999999999998,"15min":0.85299999999999998},"min":{"1min":0.65400000000000003,"5min":0.60799999999999998,"15min":0.57899999999999996},"max":{"1min":1.1499999999999999,"5min":1.204,"15min":1.2549999999999999},"last":0.73299999999999998},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.96599999999999997,"15min":0.96399999999999997},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.61699999999999999},"max":{"1min":1.131,"5min":1.165,"15min":1.327},"last":0.96499999999999997}]},{"osd":7,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95999999999999996,"15min":0.94899999999999995},"min":{"1min":0.70999999999999996,"5min":0.63800000000000001,"15min":0.58899999999999997},"max":{"1min":1.079,"5min":1.173,"15min":1.2430000000000001},"last":0.72599999999999998},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92500000000000004,"15min":0.93500000000000005},"min":{"1min":0.66400000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2010000000000001,"5min":1.216,"15min":1.3300000000000001},"last":0.76600000000000001}]}]},{"osd":5,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":2740,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2806484,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.60899999999999999,"5min":0.57599999999999996,"15min":0.48999999999999999},"max":{"1min":1.099,"5min":1.117,"15min":1.1990000000000001},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.872,"5min":0.89100000000000001,"15min":0.88900000000000001},"min":{"1min":0.72499999999999998,"5min":0.69699999999999995,"15min":0.66200000000000003},"max":{"1min":0.98799999999999999,"5min":1.139,"15min":1.2150000000000001},"last":0.98399999999999999}]},{"osd":1,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82299999999999995,"15min":0.83499999999999996},"min":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.57899999999999996},"max":{"1min":1.161,"5min":1.161,"15min":1.46},"last":0.84299999999999997},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84599999999999997,"15min":0.83699999999999997},"min":{"1min":0.67300000000000004,"5min":0.623,"15min":0.56799999999999995},"max":{"1min":1.0549999999999999,"5min":1.173,"15min":1.2589999999999999},"last":0.81399999999999995}]},{"osd":2,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85899999999999999,"15min":0.84099999999999997},"min":{"1min":0.64700000000000002,"5min":0.59199999999999997,"15min":0.56799999999999995},"max":{"1min":1.105,"5min":1.173,"15min":1.1819999999999999},"last":0.89700000000000002},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.86599999999999999,"15min":0.86799999999999999},"min":{"1min":0.64900000000000002,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":1.0449999999999999}]},{"osd":3,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83999999999999997,"15min":0.84299999999999997},"min":{"1min":0.64100000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0780000000000001,"5min":1.1899999999999999,"15min":1.482},"last":0.88},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.012,"15min":1.008},"min":{"1min":0.871,"5min":0.67900000000000005,"15min":0.623},"max":{"1min":1.274,"5min":1.274,"15min":1.3899999999999999},"last":1.1160000000000001}]},{"osd":4,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.83399999999999996,"15min":0.84899999999999998},"min":{"1min":0.58699999999999997,"5min":0.52800000000000002,"15min":0.435},"max":{"1min":1.0289999999999999,"5min":1.1599999999999999,"15min":1.266},"last":1.022},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.85499999999999998,"15min":0.874},"min":{"1min":0.61699999999999999,"5min":0.58099999999999996,"15min":0.54500000000000004},"max":{"1min":1.119,"5min":1.2889999999999999,"15min":1.325},"last":0.77500000000000002}]},{"osd":6,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.90600000000000003,"15min":0.92300000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2390000000000001,"5min":1.417,"15min":1.417},"last":0.94199999999999995},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.64000000000000001,"5min":0.59699999999999998,"15min":0.56100000000000005},"max":{"1min":1.1240000000000001,"5min":1.3879999999999999,"15min":1.452},"last":1.0740000000000001}]},{"osd":7,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.94199999999999995,"15min":0.97299999999999998},"min":{"1min":0.72799999999999998,"5min":0.58199999999999996,"15min":0.53600000000000003},"max":{"1min":1.2509999999999999,"5min":1.3300000000000001,"15min":1.4339999999999999},"last":1.0940000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.04,"15min":1.0620000000000001},"min":{"1min":0.76000000000000001,"5min":0.75,"15min":0.60899999999999999},"max":{"1min":1.3959999999999999,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.1739999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575707,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798040,"kb_used_data":92901340,"kb_used_omap":3532,"kb_used_meta":0,"kb_avail":92901340,"statfs":{"total":95948165120,"available":95130972160,"internally_reserved":0,"allocated":95130972160,"data_stored":95130972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3616802,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84499999999999997,"15min":0.84899999999999998},"min":{"1min":0.62,"5min":0.58499999999999996,"15min":0.43099999999999999},"max":{"1min":0.98099999999999998,"5min":1.1279999999999999,"15min":1.1879999999999999},"last":0.82899999999999996},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84099999999999997,"15min":0.83899999999999997},"min":{"1min":0.67200000000000004,"5min":0.56799999999999995,"15min":0.52400000000000002},"max":{"1min":1.0469999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.94799999999999995}]},{"osd":1,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.86299999999999999,"15min":0.85299999999999998},"min":{"1min":0.65500000000000003,"5min":0.61399999999999999,"15min":0.48899999999999999},"max":{"1min":0.95399999999999996,"5min":1.1499999999999999,"15min":1.1519999999999999},"last":0.88300000000000001},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.56899999999999995},"max":{"1min":1.123,"5min":1.153,"15min":1.1890000000000001},"last":0.73699999999999999}]},{"osd":2,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.64400000000000002,"5min":0.63900000000000001,"15min":0.629},"max":{"1min":1.0980000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.75900000000000001},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.88100000000000001,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.623,"15min":0.56299999999999994},"max":{"1min":1.133,"5min":1.1359999999999999,"15min":1.2270000000000001},"last":0.65200000000000002}]},{"osd":3,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85499999999999998,"15min":0.85099999999999998},"min":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.622},"max":{"1min":1.0489999999999999,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.84099999999999997},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.94799999999999995,"15min":0.93799999999999994},"min":{"1min":0.83699999999999997,"5min":0.66300000000000003,"15min":0.60499999999999998},"max":{"1min":1.111,"5min":1.264,"15min":1.79},"last":0.81599999999999995}]},{"osd":5,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.83699999999999997,"15min":0.85499999999999998},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1020000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.99399999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.54900000000000004},"max":{"1min":0.98499999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.68300000000000005}]},{"osd":6,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.85999999999999999,"15min":0.878},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.46100000000000002},"max":{"1min":1.141,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.621},{"interface":"front","average":{"1min":0.877,"5min":0.93600000000000005,"15min":0.94799999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61299999999999999},"max":{"1min":1.0740000000000001,"5min":1.22,"15min":1.2649999999999999},"last":0.92000000000000004}]},{"osd":7,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93100000000000005,"15min":0.93000000000000005},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1259999999999999,"5min":1.3120000000000001,"15min":1.337},"last":1.0129999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.002,"15min":1.01},"min":{"1min":0.71899999999999997,"5min":0.71799999999999997,"15min":0.502},"max":{"1min":1.3009999999999999,"5min":1.359,"15min":1.5229999999999999},"last":0.69599999999999995}]}]},{"osd":3,"up_from":13,"seq":55834575706,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3372,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3453227,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.59699999999999998},"min":{"1min":0.47999999999999998,"5min":0.45300000000000001,"15min":0.44900000000000001},"max":{"1min":0.84999999999999998,"5min":0.94399999999999995,"15min":1.069},"last":0.5},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.55100000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81000000000000005,"5min":0.98799999999999999,"15min":1.095},"last":0.60299999999999998}]},{"osd":1,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.70299999999999996,"15min":0.71099999999999997},"min":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.44400000000000001},"max":{"1min":0.92600000000000005,"5min":1.119,"15min":1.298},"last":0.64000000000000001},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.53100000000000003},"max":{"1min":1.014,"5min":1.046,"15min":1.1180000000000001},"last":0.71799999999999997}]},{"osd":2,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.75600000000000001,"15min":0.76200000000000001},"min":{"1min":0.58099999999999996,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0349999999999999,"5min":1.127,"15min":1.2},"last":0.76100000000000001},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77300000000000002,"15min":0.77200000000000002},"min":{"1min":0.60499999999999998,"5min":0.55300000000000005,"15min":0.51900000000000002},"max":{"1min":1.0660000000000001,"5min":1.133,"15min":1.1339999999999999},"last":0.81000000000000005}]},{"osd":4,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.871,"15min":0.85899999999999999},"min":{"1min":0.70799999999999996,"5min":0.628,"15min":0.58499999999999996},"max":{"1min":1.0409999999999999,"5min":1.246,"15min":1.2809999999999999},"last":0.70099999999999996},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.93600000000000005,"15min":0.93899999999999995},"min":{"1min":0.76700000000000002,"5min":0.71999999999999997,"15min":0.68899999999999995},"max":{"1min":1.216,"5min":1.544,"15min":1.544},"last":0.83099999999999996}]},{"osd":5,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.95199999999999996,"15min":0.95799999999999996},"min":{"1min":0.78700000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.161,"5min":1.454,"15min":1.454},"last":0.95699999999999996},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96799999999999997,"15min":0.98399999999999999},"min":{"1min":0.80600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.325,"5min":1.496,"15min":1.496},"last":0.88300000000000001}]},{"osd":6,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.79200000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.282,"5min":1.448,"15min":1.502},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.98399999999999999,"15min":0.99099999999999999},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.145,"5min":1.3500000000000001,"15min":1.3660000000000001},"last":0.86099999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0229999999999999,"15min":1.03},"min":{"1min":0.81899999999999995,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.2170000000000001,"5min":1.464,"15min":1.464},"last":0.93500000000000005},{"interface":"front","average":{"1min":1.048,"5min":1.071,"15min":1.073},"min":{"1min":0.83899999999999997,"5min":0.80800000000000005,"15min":0.69999999999999996},"max":{"1min":1.3740000000000001,"5min":1.5069999999999999,"15min":1.5409999999999999},"last":1.0149999999999999}]}]},{"osd":2,"up_from":13,"seq":55834575706,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":2842,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2910987,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.503,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.81899999999999995,"5min":0.91300000000000003,"15min":1.054},"last":0.47499999999999998},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.56299999999999994,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.875,"5min":1.153,"15min":1.153},"last":0.91800000000000004}]},{"osd":1,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73499999999999999,"15min":0.72499999999999998},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.97699999999999998,"5min":1.0589999999999999,"15min":1.147},"last":0.84899999999999998},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82699999999999996,"15min":0.82899999999999996},"min":{"1min":0.70599999999999996,"5min":0.64100000000000001,"15min":0.57199999999999995},"max":{"1min":1.0229999999999999,"5min":1.173,"15min":1.173},"last":1.1779999999999999}]},{"osd":3,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.70399999999999996,"15min":0.70099999999999996},"min":{"1min":0.57599999999999996,"5min":0.48599999999999999,"15min":0.46500000000000002},"max":{"1min":1.026,"5min":1.0840000000000001,"15min":1.1699999999999999},"last":0.51300000000000001},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.65300000000000002,"5min":0.57699999999999996,"15min":0.55700000000000005},"max":{"1min":1.0189999999999999,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.63300000000000001}]},{"osd":4,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.95199999999999996,"15min":0.94799999999999995},"min":{"1min":0.81399999999999995,"5min":0.71699999999999997,"15min":0.64700000000000002},"max":{"1min":1.1830000000000001,"5min":1.345,"15min":1.415},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.78700000000000003,"5min":0.68799999999999994,"15min":0.67400000000000004},"max":{"1min":1.091,"5min":1.2110000000000001,"15min":1.3799999999999999},"last":0.74299999999999999}]},{"osd":5,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.98499999999999999,"15min":0.98699999999999999},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.67500000000000004},"max":{"1min":1.262,"5min":1.446,"15min":1.5669999999999999},"last":1.0620000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.92500000000000004,"15min":0.94199999999999995},"min":{"1min":0.72899999999999998,"5min":0.69199999999999995,"15min":0.67200000000000004},"max":{"1min":1.1599999999999999,"5min":1.244,"15min":1.3540000000000001},"last":0.86899999999999999}]},{"osd":6,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.996,"15min":1.0089999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.2310000000000001,"5min":1.526,"15min":1.526},"last":0.91000000000000003},{"interface":"front","average":{"1min":1.069,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.91200000000000003,"5min":0.77500000000000002,"15min":0.75700000000000001},"max":{"1min":1.355,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.1200000000000001}]},{"osd":7,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0800000000000001,"15min":1.081},"min":{"1min":0.75900000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.3160000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.044,"15min":1.0449999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.71299999999999997},"max":{"1min":1.2829999999999999,"5min":1.321,"15min":1.524},"last":0.94699999999999995}]}]},{"osd":1,"up_from":13,"seq":55834575706,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3948,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4043670,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45400000000000001},"max":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":1.016},"last":0.52700000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52600000000000002,"5min":0.48599999999999999,"15min":0.47199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.135},"last":0.56899999999999995}]},{"osd":2,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.51600000000000001},"max":{"1min":1.044,"5min":1.0900000000000001,"15min":1.155},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.745},"min":{"1min":0.53000000000000003,"5min":0.50600000000000001,"15min":0.48499999999999999},"max":{"1min":1.0529999999999999,"5min":1.1519999999999999,"15min":1.2629999999999999},"last":0.71999999999999997}]},{"osd":3,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79000000000000004,"15min":0.80500000000000005},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.54600000000000004},"max":{"1min":0.92100000000000004,"5min":1.2450000000000001,"15min":1.347},"last":0.80400000000000005},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.76700000000000002},"min":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.51600000000000001},"max":{"1min":0.98099999999999998,"5min":1.1559999999999999,"15min":1.1890000000000001},"last":0.86099999999999999}]},{"osd":4,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.91700000000000004,"15min":0.91900000000000004},"min":{"1min":0.74299999999999999,"5min":0.68600000000000005,"15min":0.68000000000000005},"max":{"1min":1.1830000000000001,"5min":1.2090000000000001,"15min":1.3049999999999999},"last":0.88800000000000001},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95099999999999996,"15min":0.95899999999999996},"min":{"1min":0.745,"5min":0.73799999999999999,"15min":0.67400000000000004},"max":{"1min":1.212,"5min":1.3899999999999999,"15min":1.421},"last":0.98599999999999999}]},{"osd":5,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.98399999999999999},"min":{"1min":0.76800000000000002,"5min":0.73399999999999999,"15min":0.68999999999999995},"max":{"1min":1.266,"5min":1.3700000000000001,"15min":1.381},"last":1.2150000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0329999999999999,"15min":1.03},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.755},"max":{"1min":1.248,"5min":1.3420000000000001,"15min":1.4139999999999999},"last":1.087}]},{"osd":6,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0289999999999999,"15min":1.0409999999999999},"min":{"1min":0.79400000000000004,"5min":0.74299999999999999,"15min":0.72499999999999998},"max":{"1min":1.3180000000000001,"5min":1.3320000000000001,"15min":1.4390000000000001},"last":1.3220000000000001},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0660000000000001,"15min":1.079},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.77200000000000002},"max":{"1min":1.278,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.044}]},{"osd":7,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0609999999999999,"15min":1.0820000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.446,"5min":1.462,"15min":1.484},"last":1.194},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.1080000000000001},"min":{"1min":0.877,"5min":0.84299999999999997,"15min":0.73299999999999998},"max":{"1min":1.2929999999999999,"5min":1.329,"15min":1.5720000000000001},"last":1.2809999999999999}]}]},{"osd":0,"up_from":13,"seq":55834575707,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3767,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3857902,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91000000000000003,"5min":0.94399999999999995,"15min":1.141},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.745,"15min":0.751},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56699999999999995},"max":{"1min":0.92000000000000004,"5min":0.95999999999999996,"15min":1.129},"last":0.78900000000000003}]},{"osd":2,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.72299999999999998,"15min":0.72599999999999998},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.496},"max":{"1min":0.93000000000000005,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.72099999999999997},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.73499999999999999,"15min":0.73399999999999999},"min":{"1min":0.60499999999999998,"5min":0.50900000000000001,"15min":0.503},"max":{"1min":1.091,"5min":1.103,"15min":1.103},"last":0.68500000000000005}]},{"osd":3,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72299999999999998,"15min":0.70799999999999996},"min":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.55100000000000005},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.54800000000000004},"max":{"1min":1.0680000000000001,"5min":1.171,"15min":1.2749999999999999},"last":1.0269999999999999}]},{"osd":4,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94899999999999995,"15min":0.93999999999999995},"min":{"1min":0.748,"5min":0.71899999999999997,"15min":0.64400000000000002},"max":{"1min":1.2050000000000001,"5min":1.258,"15min":1.3180000000000001},"last":0.876},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.70399999999999996,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0960000000000001,"5min":1.3069999999999999,"15min":1.3839999999999999},"last":0.96599999999999997}]},{"osd":5,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0149999999999999,"15min":1.0229999999999999},"min":{"1min":0.85099999999999998,"5min":0.76700000000000002,"15min":0.64900000000000002},"max":{"1min":1.1950000000000001,"5min":1.448,"15min":1.571},"last":0.95299999999999996},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0109999999999999,"15min":1.012},"min":{"1min":0.89300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.98799999999999999}]},{"osd":6,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98199999999999998,"15min":0.999},"min":{"1min":0.76600000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"max":{"1min":1.1279999999999999,"5min":1.3819999999999999,"15min":1.4550000000000001},"last":0.90400000000000003},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0920000000000001,"15min":1.0900000000000001},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.71199999999999997},"max":{"1min":1.2629999999999999,"5min":1.5049999999999999,"15min":1.617},"last":1.1539999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0900000000000001,"15min":1.087},"min":{"1min":0.88800000000000001,"5min":0.75800000000000001,"15min":0.71699999999999997},"max":{"1min":1.522,"5min":1.522,"15min":1.625},"last":1.079},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0840000000000001,"15min":1.091},"min":{"1min":0.96499999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.2030000000000001,"5min":1.5109999999999999,"15min":1.571},"last":1.0169999999999999}]}]}]}} 2022-07-20T04:18:25.548 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd dump --format=json 2022-07-20T04:18:25.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.652+0000 7fe79cb10700 1 Processor -- start 2022-07-20T04:18:25.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.652+0000 7fe79cb10700 1 -- start start 2022-07-20T04:18:25.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980b13e0 0x7fe7980b0460 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fe7980af120 0x7fe798067260 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe7980b20f0 con 0x7fe7980b61d0 2022-07-20T04:18:25.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe79813efd0 con 0x7fe7980af120 2022-07-20T04:18:25.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe7980b65b0 con 0x7fe7980b13e0 2022-07-20T04:18:25.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7977fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe7980b61d0 0x7fe7980aea10 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48798/0 (socket says 172.21.15.61:48798) 2022-07-20T04:18:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7977fe700 1 -- 172.21.15.61:0/2219750790 learned_addr learned my addr 172.21.15.61:0/2219750790 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe796ffd700 1 --2- 172.21.15.61:0/2219750790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980b13e0 0x7fe7980b0460 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe797fff700 1 --2- 172.21.15.61:0/2219750790 >> v2:172.21.15.117:3300/0 conn(0x7fe7980af120 0x7fe798067260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7967fc700 1 -- 172.21.15.61:0/2219750790 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2753516521 0 0) 0x7fe7980b20f0 con 0x7fe7980b61d0 2022-07-20T04:18:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7967fc700 1 -- 172.21.15.61:0/2219750790 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fe784003260 con 0x7fe7980b61d0 2022-07-20T04:18:25.660 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe796ffd700 1 -- 172.21.15.61:0/2219750790 >> v2:172.21.15.117:3300/0 conn(0x7fe7980af120 msgr2=0x7fe798067260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:25.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe796ffd700 1 --2- 172.21.15.61:0/2219750790 >> v2:172.21.15.117:3300/0 conn(0x7fe7980af120 0x7fe798067260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe796ffd700 1 -- 172.21.15.61:0/2219750790 >> v1:172.21.15.61:6789/0 conn(0x7fe7980b61d0 legacy=0x7fe7980aea10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.661 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe796ffd700 1 -- 172.21.15.61:0/2219750790 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7980b1ad0 con 0x7fe7980b13e0 2022-07-20T04:18:25.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe797fff700 1 --2- 172.21.15.61:0/2219750790 >> v2:172.21.15.117:3300/0 conn(0x7fe7980af120 0x7fe798067260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:25.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe796ffd700 1 --2- 172.21.15.61:0/2219750790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980b13e0 0x7fe7980b0460 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe788002f70 tx=0x7fe788006df0).ready entity=mon.2 client_cookie=9625c92d3ce0d536 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.662 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7967fc700 1 -- 172.21.15.61:0/2219750790 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe78800d4a0 con 0x7fe7980b13e0 2022-07-20T04:18:25.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7967fc700 1 -- 172.21.15.61:0/2219750790 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788000e90 con 0x7fe7980b13e0 2022-07-20T04:18:25.663 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe7967fc700 1 -- 172.21.15.61:0/2219750790 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe78800d7f0 con 0x7fe7980b13e0 2022-07-20T04:18:25.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- 172.21.15.61:0/2219750790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980b13e0 msgr2=0x7fe7980b0460 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 --2- 172.21.15.61:0/2219750790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980b13e0 0x7fe7980b0460 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fe788002f70 tx=0x7fe788006df0).stop 2022-07-20T04:18:25.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- 172.21.15.61:0/2219750790 shutdown_connections 2022-07-20T04:18:25.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 --2- 172.21.15.61:0/2219750790 >> v2:172.21.15.117:3300/0 conn(0x7fe7980af120 0x7fe798067260 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 --2- 172.21.15.61:0/2219750790 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980b13e0 0x7fe7980b0460 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- 172.21.15.61:0/2219750790 >> 172.21.15.61:0/2219750790 conn(0x7fe798063cf0 msgr2=0x7fe7980b5780 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:25.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- 172.21.15.61:0/2219750790 shutdown_connections 2022-07-20T04:18:25.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.656+0000 7fe79cb10700 1 -- 172.21.15.61:0/2219750790 wait complete. 2022-07-20T04:18:25.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.660+0000 7fe79cb10700 1 Processor -- start 2022-07-20T04:18:25.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.660+0000 7fe79cb10700 1 -- start start 2022-07-20T04:18:25.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980af120 0x7fe79813caa0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 0x7fe798141ba0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7980b61d0 0x7fe79813b0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe798066fe0 con 0x7fe7980b13e0 2022-07-20T04:18:25.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe79813a700 con 0x7fe7980b61d0 2022-07-20T04:18:25.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe79813a9a0 con 0x7fe7980af120 2022-07-20T04:18:25.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 0x7fe798141ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 0x7fe798141ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52208/0 (socket says 172.21.15.61:52208) 2022-07-20T04:18:25.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe7977fe700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980af120 0x7fe79813caa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 -- 172.21.15.61:0/3396014074 learned_addr learned my addr 172.21.15.61:0/3396014074 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:25.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe797fff700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7980b61d0 0x7fe79813b0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 -- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980af120 msgr2=0x7fe79813caa0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980af120 0x7fe79813caa0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 -- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7980b61d0 msgr2=0x7fe79813b0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7980b61d0 0x7fe79813b0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 -- 172.21.15.61:0/3396014074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe790009470 con 0x7fe7980b13e0 2022-07-20T04:18:25.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe796ffd700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 0x7fe798141ba0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7fe788002f70 tx=0x7fe788014000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe7880139c0 con 0x7fe7980b13e0 2022-07-20T04:18:25.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe788013b20 con 0x7fe7980b13e0 2022-07-20T04:18:25.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe78800b180 con 0x7fe7980b13e0 2022-07-20T04:18:25.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe788013e80 con 0x7fe7980b13e0 2022-07-20T04:18:25.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe79813b5d0 con 0x7fe7980b13e0 2022-07-20T04:18:25.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.664+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7980b1d70 con 0x7fe7980b13e0 2022-07-20T04:18:25.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.668+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe788025030 con 0x7fe7980b13e0 2022-07-20T04:18:25.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.668+0000 7fe794ff9700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe7780539b0 0x7fe778055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:25.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.668+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe78806ca90 con 0x7fe7980b13e0 2022-07-20T04:18:25.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.668+0000 7fe7977fe700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe7780539b0 0x7fe778055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:25.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.668+0000 7fe7977fe700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe7780539b0 0x7fe778055e70 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7fe78c00d000 tx=0x7fe78c009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:25.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.672+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fe788043bc0 con 0x7fe7980b13e0 2022-07-20T04:18:25.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.800+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "osd dump", "format": "json"} v 0) v1 -- 0x7fe7980574e0 con 0x7fe7980b13e0 2022-07-20T04:18:25.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe794ff9700 1 -- 172.21.15.61:0/3396014074 <== mon.0 v2:172.21.15.61:3300/0 7 ==== mon_command_ack([{"prefix": "osd dump", "format": "json"}]=0 v152) v1 ==== 74+0+13370 (secure 0 0 0) 0x7fe7880471e0 con 0x7fe7980b13e0 2022-07-20T04:18:25.805 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T04:18:25.806 INFO:teuthology.orchestra.run.smithi061.stdout:{"epoch":152,"fsid":"69b2e2c1-da2e-4470-a6a9-37f8d410d6ed","created":"2022-07-20T03:06:50.175615+0000","modified":"2022-07-20T03:17:54.309017+0000","last_up_change":"2022-07-20T03:06:59.505879+0000","last_in_change":"2022-07-20T03:06:59.505879+0000","flags":"sortbitwise,recovery_deletes,purged_snapdirs,pglog_hardlimit","flags_num":5799936,"flags_set":["pglog_hardlimit","purged_snapdirs","recovery_deletes","sortbitwise"],"crush_version":4,"full_ratio":0.94999998807907104,"backfillfull_ratio":0.89999997615814209,"nearfull_ratio":0.85000002384185791,"cluster_snapshot":"","pool_max":30,"max_osd":8,"require_min_compat_client":"jewel","min_compat_client":"jewel","require_osd_release":"octopus","pools":[{"pool":1,"pool_name":"device_health_metrics","create_time":"2022-07-20T03:06:54.040896+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":1,"pg_placement_num":1,"pg_placement_num_target":1,"pg_num_target":1,"pg_num_pending":1,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"15","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{"pg_num_min":1},"application_metadata":{"mgr_devicehealth":{}}},{"pool":2,"pool_name":"rbd","create_time":"2022-07-20T03:07:05.508608+0000","flags":1,"flags_names":"hashpspool","type":1,"size":2,"min_size":1,"crush_rule":0,"object_hash":2,"pg_autoscale_mode":"off","pg_num":8,"pg_placement_num":8,"pg_placement_num_target":8,"pg_num_target":8,"pg_num_pending":8,"last_pg_merge_meta":{"source_pgid":"0.0","ready_epoch":0,"last_epoch_started":0,"last_epoch_clean":0,"source_version":"0'0","target_version":"0'0"},"last_change":"18","last_force_op_resend":"0","last_force_op_resend_prenautilus":"0","last_force_op_resend_preluminous":"0","auid":0,"snap_mode":"selfmanaged","snap_seq":0,"snap_epoch":0,"pool_snaps":[],"removed_snaps":"[]","quota_max_bytes":0,"quota_max_objects":0,"tiers":[],"tier_of":-1,"read_tier":-1,"write_tier":-1,"cache_mode":"none","target_max_bytes":0,"target_max_objects":0,"cache_target_dirty_ratio_micro":400000,"cache_target_dirty_high_ratio_micro":600000,"cache_target_full_ratio_micro":800000,"cache_min_flush_age":0,"cache_min_evict_age":0,"erasure_code_profile":"","hit_set_params":{"type":"none"},"hit_set_period":0,"hit_set_count":0,"use_gmt_hitset":true,"min_read_recency_for_promote":0,"min_write_recency_for_promote":0,"hit_set_grade_decay_rate":0,"hit_set_search_last_n":0,"grade_table":[],"stripe_width":0,"expected_num_objects":0,"fast_read":false,"options":{},"application_metadata":{"rbd":{}}}],"osds":[{"osd":0,"uuid":"2994e86d-2708-44df-bf9c-63667f52059f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6800","nonce":22894},{"type":"v1","addr":"172.21.15.61:6801","nonce":22894}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6802","nonce":22894},{"type":"v1","addr":"172.21.15.61:6803","nonce":22894}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6806","nonce":22894},{"type":"v1","addr":"172.21.15.61:6807","nonce":22894}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6804","nonce":22894},{"type":"v1","addr":"172.21.15.61:6805","nonce":22894}]},"public_addr":"172.21.15.61:6801/22894","cluster_addr":"172.21.15.61:6803/22894","heartbeat_back_addr":"172.21.15.61:6807/22894","heartbeat_front_addr":"172.21.15.61:6805/22894","state":["exists","up"]},{"osd":1,"uuid":"9cd27797-c189-4c94-9e22-769611620680","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6808","nonce":22998},{"type":"v1","addr":"172.21.15.61:6809","nonce":22998}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6810","nonce":22998},{"type":"v1","addr":"172.21.15.61:6811","nonce":22998}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6814","nonce":22998},{"type":"v1","addr":"172.21.15.61:6816","nonce":22998}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6812","nonce":22998},{"type":"v1","addr":"172.21.15.61:6813","nonce":22998}]},"public_addr":"172.21.15.61:6809/22998","cluster_addr":"172.21.15.61:6811/22998","heartbeat_back_addr":"172.21.15.61:6816/22998","heartbeat_front_addr":"172.21.15.61:6813/22998","state":["exists","up"]},{"osd":2,"uuid":"637945c3-b1e0-4d86-933a-2723e56b0705","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6815","nonce":23001},{"type":"v1","addr":"172.21.15.61:6817","nonce":23001}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6818","nonce":23001},{"type":"v1","addr":"172.21.15.61:6819","nonce":23001}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6822","nonce":23001},{"type":"v1","addr":"172.21.15.61:6823","nonce":23001}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6820","nonce":23001},{"type":"v1","addr":"172.21.15.61:6821","nonce":23001}]},"public_addr":"172.21.15.61:6817/23001","cluster_addr":"172.21.15.61:6819/23001","heartbeat_back_addr":"172.21.15.61:6823/23001","heartbeat_front_addr":"172.21.15.61:6821/23001","state":["exists","up"]},{"osd":3,"uuid":"b6079eb6-2fd9-4f0e-831f-83b051fc040f","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6824","nonce":23120},{"type":"v1","addr":"172.21.15.61:6825","nonce":23120}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6826","nonce":23120},{"type":"v1","addr":"172.21.15.61:6827","nonce":23120}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6830","nonce":23120},{"type":"v1","addr":"172.21.15.61:6831","nonce":23120}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.61:6828","nonce":23120},{"type":"v1","addr":"172.21.15.61:6829","nonce":23120}]},"public_addr":"172.21.15.61:6825/23120","cluster_addr":"172.21.15.61:6827/23120","heartbeat_back_addr":"172.21.15.61:6831/23120","heartbeat_front_addr":"172.21.15.61:6829/23120","state":["exists","up"]},{"osd":4,"uuid":"1f342a61-db71-49da-a367-b177e2e65fdd","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":146,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6802","nonce":23184},{"type":"v1","addr":"172.21.15.117:6803","nonce":23184}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6804","nonce":23184},{"type":"v1","addr":"172.21.15.117:6805","nonce":23184}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6808","nonce":23184},{"type":"v1","addr":"172.21.15.117:6809","nonce":23184}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6806","nonce":23184},{"type":"v1","addr":"172.21.15.117:6807","nonce":23184}]},"public_addr":"172.21.15.117:6803/23184","cluster_addr":"172.21.15.117:6805/23184","heartbeat_back_addr":"172.21.15.117:6809/23184","heartbeat_front_addr":"172.21.15.117:6807/23184","state":["exists","up"]},{"osd":5,"uuid":"ae86667a-16a2-48c1-9182-7694f9e01b8b","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6818","nonce":23294},{"type":"v1","addr":"172.21.15.117:6819","nonce":23294}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6820","nonce":23294},{"type":"v1","addr":"172.21.15.117:6821","nonce":23294}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6824","nonce":23294},{"type":"v1","addr":"172.21.15.117:6826","nonce":23294}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6822","nonce":23294},{"type":"v1","addr":"172.21.15.117:6823","nonce":23294}]},"public_addr":"172.21.15.117:6819/23294","cluster_addr":"172.21.15.117:6821/23294","heartbeat_back_addr":"172.21.15.117:6826/23294","heartbeat_front_addr":"172.21.15.117:6823/23294","state":["exists","up"]},{"osd":6,"uuid":"1558d94a-b96d-4bbb-86ad-34fe05792eb8","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6810","nonce":23293},{"type":"v1","addr":"172.21.15.117:6811","nonce":23293}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6812","nonce":23293},{"type":"v1","addr":"172.21.15.117:6813","nonce":23293}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6816","nonce":23293},{"type":"v1","addr":"172.21.15.117:6817","nonce":23293}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6814","nonce":23293},{"type":"v1","addr":"172.21.15.117:6815","nonce":23293}]},"public_addr":"172.21.15.117:6811/23293","cluster_addr":"172.21.15.117:6813/23293","heartbeat_back_addr":"172.21.15.117:6817/23293","heartbeat_front_addr":"172.21.15.117:6815/23293","state":["exists","up"]},{"osd":7,"uuid":"1f056769-9505-445e-8c01-b2276cd63a2c","up":1,"in":1,"weight":1,"primary_affinity":1,"last_clean_begin":0,"last_clean_end":0,"up_from":13,"up_thru":147,"down_at":0,"lost_at":0,"public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6825","nonce":23295},{"type":"v1","addr":"172.21.15.117:6827","nonce":23295}]},"cluster_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6828","nonce":23295},{"type":"v1","addr":"172.21.15.117:6829","nonce":23295}]},"heartbeat_back_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6832","nonce":23295},{"type":"v1","addr":"172.21.15.117:6833","nonce":23295}]},"heartbeat_front_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.117:6830","nonce":23295},{"type":"v1","addr":"172.21.15.117:6831","nonce":23295}]},"public_addr":"172.21.15.117:6827/23295","cluster_addr":"172.21.15.117:6829/23295","heartbeat_back_addr":"172.21.15.117:6833/23295","heartbeat_front_addr":"172.21.15.117:6831/23295","state":["exists","up"]}],"osd_xinfo":[{"osd":0,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.755912+0000","dead_epoch":0},{"osd":1,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.840154+0000","dead_epoch":0},{"osd":2,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.773461+0000","dead_epoch":0},{"osd":3,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.811063+0000","dead_epoch":0},{"osd":4,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:57.886289+0000","dead_epoch":0},{"osd":5,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.039047+0000","dead_epoch":0},{"osd":6,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.017635+0000","dead_epoch":0},{"osd":7,"down_stamp":"0.000000","laggy_probability":0,"laggy_interval":0,"features":4540138292840890367,"old_weight":0,"last_purged_snaps_scrub":"2022-07-20T03:06:58.001653+0000","dead_epoch":0}],"pg_upmap":[],"pg_upmap_items":[],"pg_temp":[],"primary_temp":[],"blacklist":{"172.21.15.117:6835/1280182602":"2022-07-21T03:17:51.103581+0000","172.21.15.117:6834/1280182602":"2022-07-21T03:17:51.103581+0000"},"erasure_code_profiles":{"default":{"crush-failure-domain":"osd","k":"2","m":"1","plugin":"jerasure","ruleset-failure-domain":"osd","technique":"reed_sol_van"}},"removed_snaps_queue":[],"new_removed_snaps":[],"new_purged_snaps":[],"crush_node_flags":{},"device_class_flags":{}} 2022-07-20T04:18:25.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe7780539b0 msgr2=0x7fe778055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe7780539b0 0x7fe778055e70 secure :-1 s=READY pgs=806 cs=0 l=1 rev1=1 rx=0x7fe78c00d000 tx=0x7fe78c009040).stop 2022-07-20T04:18:25.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 msgr2=0x7fe798141ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:25.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 0x7fe798141ba0 secure :-1 s=READY pgs=1539 cs=0 l=1 rev1=1 rx=0x7fe788002f70 tx=0x7fe788014000).stop 2022-07-20T04:18:25.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 shutdown_connections 2022-07-20T04:18:25.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe7780539b0 0x7fe778055e70 unknown :-1 s=CLOSED pgs=806 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.804+0000 7fe79cb10700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe7980b61d0 0x7fe79813b0e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.808+0000 7fe79cb10700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe7980b13e0 0x7fe798141ba0 unknown :-1 s=CLOSED pgs=1539 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.808+0000 7fe79cb10700 1 --2- 172.21.15.61:0/3396014074 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe7980af120 0x7fe79813caa0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:25.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.808+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 >> 172.21.15.61:0/3396014074 conn(0x7fe798063cf0 msgr2=0x7fe7981425e0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:25.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.808+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 shutdown_connections 2022-07-20T04:18:25.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:25.808+0000 7fe79cb10700 1 -- 172.21.15.61:0/3396014074 wait complete. 2022-07-20T04:18:26.819 INFO:tasks.ceph:Scrubbing osd.0 2022-07-20T04:18:26.820 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.0 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:26.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6bfff700 1 Processor -- start 2022-07-20T04:18:26.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6bfff700 1 -- start start 2022-07-20T04:18:26.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6bfff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1b6c0b5990 0x7f1b6c146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.944 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6bfff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c146c70 con 0x7f1b6c0af3e0 2022-07-20T04:18:26.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6bfff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1b6c0b5d70 con 0x7f1b6c05d210 2022-07-20T04:18:26.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6bfff700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1b6c0ad050 con 0x7f1b6c0b5990 2022-07-20T04:18:26.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6a7fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f1b6c0af3e0 0x7f1b6c146560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48812/0 (socket says 172.21.15.61:48812) 2022-07-20T04:18:26.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6a7fc700 1 -- 172.21.15.61:0/3944364862 learned_addr learned my addr 172.21.15.61:0/3944364862 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:26.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 --2- 172.21.15.61:0/3944364862 >> v2:172.21.15.117:3300/0 conn(0x7f1b6c0b5990 0x7f1b6c146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 941199839 0 0) 0x7f1b6c146c70 con 0x7f1b6c0af3e0 2022-07-20T04:18:26.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b50003260 con 0x7f1b6c0af3e0 2022-07-20T04:18:26.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3860144276 0 0) 0x7f1b6c0b5d70 con 0x7f1b6c05d210 2022-07-20T04:18:26.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f1b6c146c70 con 0x7f1b6c05d210 2022-07-20T04:18:26.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1164329176 0 0) 0x7f1b50003260 con 0x7f1b6c0af3e0 2022-07-20T04:18:26.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 >> v2:172.21.15.117:3300/0 conn(0x7f1b6c0b5990 msgr2=0x7f1b6c146070 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:26.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 --2- 172.21.15.61:0/3944364862 >> v2:172.21.15.117:3300/0 conn(0x7f1b6c0b5990 0x7f1b6c146070 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 >> v1:172.21.15.117:6789/0 conn(0x7f1b6c05d210 legacy=0x7f1b6c14a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 --> v1:172.21.15.61:6789/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c0b1010 con 0x7f1b6c0af3e0 2022-07-20T04:18:26.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.0 v1:172.21.15.61:6789/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (unknown 1388480323 0 0) 0x7f1b60001660 con 0x7f1b6c0af3e0 2022-07-20T04:18:26.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 >> v1:172.21.15.61:6789/0 conn(0x7f1b6c0af3e0 legacy=0x7f1b6c146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b50003930 0x7f1b50003d30 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50004500 0x7f1b50004980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b50004e70 0x7f1b5000b750 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5000bcf0 con 0x7f1b50003930 2022-07-20T04:18:26.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5000be60 con 0x7f1b50004500 2022-07-20T04:18:26.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b5000c020 con 0x7f1b50004e70 2022-07-20T04:18:26.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b50003930 0x7f1b50003d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6a7fc700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50004500 0x7f1b50004980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6b7fe700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b50004e70 0x7f1b5000b750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 -- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b50004e70 msgr2=0x7f1b5000b750 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:26.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b50004e70 0x7f1b5000b750 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 -- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50004500 msgr2=0x7f1b50004980 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:26.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50004500 0x7f1b50004980 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 -- 172.21.15.61:0/3944364862 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b50003260 con 0x7f1b50003930 2022-07-20T04:18:26.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6b7fe700 1 -- 172.21.15.61:0/3944364862 reap_dead start 2022-07-20T04:18:26.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b6affd700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b50003930 0x7f1b50003d30 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f1b5c00c000 tx=0x7f1b5c006d90).ready entity=mon.0 client_cookie=564178986fa444a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:26.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b5c013450 con 0x7f1b50003930 2022-07-20T04:18:26.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b5c0135b0 con 0x7f1b50003930 2022-07-20T04:18:26.957 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.940+0000 7f1b69ffb700 1 -- 172.21.15.61:0/3944364862 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b5c013910 con 0x7f1b50003930 2022-07-20T04:18:26.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- 172.21.15.61:0/3944364862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b50003930 msgr2=0x7f1b50003d30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b50003930 0x7f1b50003d30 secure :-1 s=READY pgs=1541 cs=0 l=1 rev1=1 rx=0x7f1b5c00c000 tx=0x7f1b5c006d90).stop 2022-07-20T04:18:26.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- 172.21.15.61:0/3944364862 shutdown_connections 2022-07-20T04:18:26.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 --2- 172.21.15.61:0/3944364862 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b50003930 0x7f1b50003d30 unknown :-1 s=CLOSED pgs=1541 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- 172.21.15.61:0/3944364862 >> 172.21.15.61:0/3944364862 conn(0x7f1b6c063480 msgr2=0x7f1b6c0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:26.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- 172.21.15.61:0/3944364862 shutdown_connections 2022-07-20T04:18:26.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- 172.21.15.61:0/3944364862 wait complete. 2022-07-20T04:18:26.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 Processor -- start 2022-07-20T04:18:26.964 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- start start 2022-07-20T04:18:26.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50003930 0x7f1b6c145190 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b6c0674c0 0x7f1b6c066650 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.965 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b6c159640 0x7f1b6c159ac0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c156b90 con 0x7f1b6c159640 2022-07-20T04:18:26.966 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c156e30 con 0x7f1b50003930 2022-07-20T04:18:26.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6bfff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1b6c067190 con 0x7f1b6c0674c0 2022-07-20T04:18:26.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b6c159640 0x7f1b6c159ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.967 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50003930 0x7f1b6c145190 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6a7fc700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b6c0674c0 0x7f1b6c066650 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.968 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6affd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50003930 0x7f1b6c145190 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:33674/0 (socket says 172.21.15.61:33674) 2022-07-20T04:18:26.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6affd700 1 -- 172.21.15.61:0/2897162227 learned_addr learned my addr 172.21.15.61:0/2897162227 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:26.969 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.944+0000 7f1b6b7fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b6c159640 0x7f1b6c159ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52230/0 (socket says 172.21.15.61:52230) 2022-07-20T04:18:26.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6b7fe700 1 -- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b6c0674c0 msgr2=0x7f1b6c066650 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6b7fe700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1b6c0674c0 0x7f1b6c066650 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.970 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6b7fe700 1 -- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50003930 msgr2=0x7f1b6c145190 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6b7fe700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1b50003930 0x7f1b6c145190 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.971 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6b7fe700 1 -- 172.21.15.61:0/2897162227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1b6c066b40 con 0x7f1b6c159640 2022-07-20T04:18:26.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6b7fe700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b6c159640 0x7f1b6c159ac0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f1b64000b60 tx=0x7f1b64007d80).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:26.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b64006c90 con 0x7f1b6c159640 2022-07-20T04:18:26.972 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1b64006df0 con 0x7f1b6c159640 2022-07-20T04:18:26.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1b6c159fb0 con 0x7f1b6c159640 2022-07-20T04:18:26.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1b64007220 con 0x7f1b6c159640 2022-07-20T04:18:26.973 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1b6c15a400 con 0x7f1b6c159640 2022-07-20T04:18:26.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1b6c1569f0 con 0x7f1b6c159640 2022-07-20T04:18:26.974 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1b64007380 con 0x7f1b6c159640 2022-07-20T04:18:26.975 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b48053af0 0x7f1b48055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1b6406b830 con 0x7f1b6c159640 2022-07-20T04:18:26.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f1b4805a1e0 0x7f1b4805c640 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:26.976 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 --> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1b4805cc20 con 0x7f1b4805a1e0 2022-07-20T04:18:26.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.948+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1b64070240 con 0x7f1b6c159640 2022-07-20T04:18:26.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.952+0000 7f1b6affd700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b48053af0 0x7f1b48055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.977 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.952+0000 7f1b6a7fc700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f1b4805a1e0 0x7f1b4805c640 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.952+0000 7f1b6a7fc700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f1b4805a1e0 0x7f1b4805c640 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:26.978 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.952+0000 7f1b6affd700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b48053af0 0x7f1b48055fb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f1b5c01c000 tx=0x7f1b5c018000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:26.979 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.952+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== osd.0 v2:172.21.15.61:6800/22894 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f1b4805cc20 con 0x7f1b4805a1e0 2022-07-20T04:18:26.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 --> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1b6c0574e0 con 0x7f1b4805a1e0 2022-07-20T04:18:26.980 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b5bfff700 1 -- 172.21.15.61:0/2897162227 <== osd.0 v2:172.21.15.61:6800/22894 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1b6c0574e0 con 0x7f1b4805a1e0 2022-07-20T04:18:26.981 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:26.981 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:26.981 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:26.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f1b4805a1e0 msgr2=0x7f1b4805c640 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.983 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:6800/22894,v1:172.21.15.61:6801/22894] conn(0x7f1b4805a1e0 0x7f1b4805c640 crc :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b48053af0 msgr2=0x7f1b48055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1b48053af0 0x7f1b48055fb0 secure :-1 s=READY pgs=807 cs=0 l=1 rev1=1 rx=0x7f1b5c01c000 tx=0x7f1b5c018000).stop 2022-07-20T04:18:26.984 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b6c159640 msgr2=0x7f1b6c159ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:26.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 --2- 172.21.15.61:0/2897162227 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1b6c159640 0x7f1b6c159ac0 secure :-1 s=READY pgs=1542 cs=0 l=1 rev1=1 rx=0x7f1b64000b60 tx=0x7f1b64007d80).stop 2022-07-20T04:18:26.985 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6b7fe700 1 -- 172.21.15.61:0/2897162227 reap_dead start 2022-07-20T04:18:26.986 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 shutdown_connections 2022-07-20T04:18:26.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 >> 172.21.15.61:0/2897162227 conn(0x7f1b6c063480 msgr2=0x7f1b6c14aa20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:26.987 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 shutdown_connections 2022-07-20T04:18:26.988 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:26.972+0000 7f1b6bfff700 1 -- 172.21.15.61:0/2897162227 wait complete. 2022-07-20T04:18:26.989 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 0 2022-07-20T04:18:27.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fb59e700 1 Processor -- start 2022-07-20T04:18:27.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fb59e700 1 -- start start 2022-07-20T04:18:27.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc0ae350 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fb59e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19fc0af360 con 0x7f19fc0b00e0 2022-07-20T04:18:27.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fb59e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f19fc0b4e70 con 0x7f19fc0b4a90 2022-07-20T04:18:27.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc0af590 con 0x7f19fc0b1790 2022-07-20T04:18:27.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fad9d700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f19fc0b00e0 0x7f19fc13e940 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48838/0 (socket says 172.21.15.61:48838) 2022-07-20T04:18:27.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19fad9d700 1 -- 172.21.15.61:0/2516470374 learned_addr learned my addr 172.21.15.61:0/2516470374 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:27.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f3fff700 1 --2- 172.21.15.61:0/2516470374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc0ae350 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1336298868 0 0) 0x7f19fc0af360 con 0x7f19fc0b00e0 2022-07-20T04:18:27.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19dc003260 con 0x7f19fc0b00e0 2022-07-20T04:18:27.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 798564609 0 0) 0x7f19fc0b4e70 con 0x7f19fc0b4a90 2022-07-20T04:18:27.118 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f19fc0af360 con 0x7f19fc0b4a90 2022-07-20T04:18:27.119 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f3fff700 1 -- 172.21.15.61:0/2516470374 >> v1:172.21.15.117:6789/0 conn(0x7f19fc0b4a90 legacy=0x7f19fc0ace20 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.120 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f3fff700 1 -- 172.21.15.61:0/2516470374 >> v1:172.21.15.61:6789/0 conn(0x7f19fc0b00e0 legacy=0x7f19fc13e940 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f3fff700 1 -- 172.21.15.61:0/2516470374 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19fc0af830 con 0x7f19fc0b1790 2022-07-20T04:18:27.121 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.100+0000 7f19f3fff700 1 --2- 172.21.15.61:0/2516470374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc0ae350 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f19e4002f70 tx=0x7f19e4006df0).ready entity=mon.2 client_cookie=d8dec114ad34fe04 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.122 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19e400d5c0 con 0x7f19fc0b1790 2022-07-20T04:18:27.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19e4000e90 con 0x7f19fc0b1790 2022-07-20T04:18:27.123 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19f9d9b700 1 -- 172.21.15.61:0/2516470374 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19e400d910 con 0x7f19fc0b1790 2022-07-20T04:18:27.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 -- 172.21.15.61:0/2516470374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 msgr2=0x7f19fc0ae350 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.124 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 --2- 172.21.15.61:0/2516470374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc0ae350 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7f19e4002f70 tx=0x7f19e4006df0).stop 2022-07-20T04:18:27.125 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 -- 172.21.15.61:0/2516470374 shutdown_connections 2022-07-20T04:18:27.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 --2- 172.21.15.61:0/2516470374 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc0ae350 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.126 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 -- 172.21.15.61:0/2516470374 >> 172.21.15.61:0/2516470374 conn(0x7f19fc063150 msgr2=0x7f19fc063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.127 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 -- 172.21.15.61:0/2516470374 shutdown_connections 2022-07-20T04:18:27.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 -- 172.21.15.61:0/2516470374 wait complete. 2022-07-20T04:18:27.128 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 Processor -- start 2022-07-20T04:18:27.129 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.104+0000 7f19fb59e700 1 -- start start 2022-07-20T04:18:27.130 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 0x7f19fc0653d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc13db40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.131 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19fc0b4a90 0x7f19fc150fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.132 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc143cc0 con 0x7f19fc0b00e0 2022-07-20T04:18:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc143a20 con 0x7f19fc0b4a90 2022-07-20T04:18:27.133 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f19fc140ee0 con 0x7f19fc0b1790 2022-07-20T04:18:27.134 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 0x7f19fc0653d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 0x7f19fc0653d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52242/0 (socket says 172.21.15.61:52242) 2022-07-20T04:18:27.135 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 -- 172.21.15.61:0/3020319644 learned_addr learned my addr 172.21.15.61:0/3020319644 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:27.136 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19f3fff700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc13db40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.137 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fad9d700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19fc0b4a90 0x7f19fc150fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 -- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 msgr2=0x7f19fc13db40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.138 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc13db40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.139 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 -- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19fc0b4a90 msgr2=0x7f19fc150fb0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19fc0b4a90 0x7f19fc150fb0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.140 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 -- 172.21.15.61:0/3020319644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e400b180 con 0x7f19fc0b00e0 2022-07-20T04:18:27.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fa59c700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 0x7f19fc0653d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f19ec001110 tx=0x7f19ec007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.141 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19ec012400 con 0x7f19fc0b00e0 2022-07-20T04:18:27.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f19ec012560 con 0x7f19fc0b00e0 2022-07-20T04:18:27.142 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f19ec0128c0 con 0x7f19fc0b00e0 2022-07-20T04:18:27.143 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19fc1436f0 con 0x7f19fc0b00e0 2022-07-20T04:18:27.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19fc143290 con 0x7f19fc0b00e0 2022-07-20T04:18:27.144 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19fc1410a0 con 0x7f19fc0b00e0 2022-07-20T04:18:27.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.108+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f19ec019030 con 0x7f19fc0b00e0 2022-07-20T04:18:27.145 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.112+0000 7f19f37fe700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19e0053ad0 0x7f19e0055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.146 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.112+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f19ec06ae60 con 0x7f19fc0b00e0 2022-07-20T04:18:27.147 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.112+0000 7f19f3fff700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19e0053ad0 0x7f19e0055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.112+0000 7f19f3fff700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19e0053ad0 0x7f19e0055f90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f19e401a000 tx=0x7f19e4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.148 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.120+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f19ec0420a0 con 0x7f19fc0b00e0 2022-07-20T04:18:27.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.248+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "0", "target": ["mon-mgr", ""]}) v1 -- 0x7f19fc0afad0 con 0x7f19e0053ad0 2022-07-20T04:18:27.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.248+0000 7f19f37fe700 1 -- 172.21.15.61:0/3020319644 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 0 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f19fc0afad0 con 0x7f19e0053ad0 2022-07-20T04:18:27.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19e0053ad0 msgr2=0x7f19e0055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19e0053ad0 0x7f19e0055f90 secure :-1 s=READY pgs=808 cs=0 l=1 rev1=1 rx=0x7f19e401a000 tx=0x7f19e4014000).stop 2022-07-20T04:18:27.255 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 msgr2=0x7f19fc0653d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 0x7f19fc0653d0 secure :-1 s=READY pgs=1544 cs=0 l=1 rev1=1 rx=0x7f19ec001110 tx=0x7f19ec007ae0).stop 2022-07-20T04:18:27.258 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 shutdown_connections 2022-07-20T04:18:27.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f19e0053ad0 0x7f19e0055f90 unknown :-1 s=CLOSED pgs=808 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.259 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f19fc0b4a90 0x7f19fc150fb0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.260 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f19fc0b1790 0x7f19fc13db40 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 --2- 172.21.15.61:0/3020319644 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f19fc0b00e0 0x7f19fc0653d0 unknown :-1 s=CLOSED pgs=1544 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.261 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 >> 172.21.15.61:0/3020319644 conn(0x7f19fc063150 msgr2=0x7f19fc0b3d00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.262 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 shutdown_connections 2022-07-20T04:18:27.263 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.252+0000 7f19fb59e700 1 -- 172.21.15.61:0/3020319644 wait complete. 2022-07-20T04:18:27.264 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 0 to deep-scrub 2022-07-20T04:18:27.271 INFO:tasks.ceph:Scrubbing osd.1 2022-07-20T04:18:27.273 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.1 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:27.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 Processor -- start 2022-07-20T04:18:27.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 -- start start 2022-07-20T04:18:27.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1ad80b6690 0x7f1ad80b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f1ad80ae670 0x7f1ad80aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f1ad80acae0 con 0x7f1ad80b59a0 2022-07-20T04:18:27.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad8147100 con 0x7f1ad80b6690 2022-07-20T04:18:27.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.392+0000 7f1adde0d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f1ad80acd10 con 0x7f1ad80ae670 2022-07-20T04:18:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1ad80b6690 0x7f1ad80b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f1ad80b6690 0x7f1ad80b6a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52252/0 (socket says 172.21.15.61:52252) 2022-07-20T04:18:27.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 -- 172.21.15.61:0/587326875 learned_addr learned my addr 172.21.15.61:0/587326875 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:27.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad7fff700 1 --2- 172.21.15.61:0/587326875 >> v2:172.21.15.117:3300/0 conn(0x7f1ad80ae670 0x7f1ad80aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 -- 172.21.15.61:0/587326875 >> v2:172.21.15.117:3300/0 conn(0x7f1ad80ae670 msgr2=0x7f1ad80aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:27.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 --2- 172.21.15.61:0/587326875 >> v2:172.21.15.117:3300/0 conn(0x7f1ad80ae670 0x7f1ad80aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 -- 172.21.15.61:0/587326875 >> v1:172.21.15.117:6789/0 conn(0x7f1ad80b59a0 legacy=0x7f1ad813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 -- 172.21.15.61:0/587326875 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad80b5df0 con 0x7f1ad80b6690 2022-07-20T04:18:27.413 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad7fff700 1 --2- 172.21.15.61:0/587326875 >> v2:172.21.15.117:3300/0 conn(0x7f1ad80ae670 0x7f1ad80aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:27.414 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad77fe700 1 --2- 172.21.15.61:0/587326875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80b6690 0x7f1ad80b6a70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f1ac8002f70 tx=0x7f1ac8006eb0).ready entity=mon.0 client_cookie=570d3b64ef393a16 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.415 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad6ffd700 1 -- 172.21.15.61:0/587326875 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ac800d5c0 con 0x7f1ad80b6690 2022-07-20T04:18:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad6ffd700 1 -- 172.21.15.61:0/587326875 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac8000e90 con 0x7f1ad80b6690 2022-07-20T04:18:27.417 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1ad6ffd700 1 -- 172.21.15.61:0/587326875 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ac800d950 con 0x7f1ad80b6690 2022-07-20T04:18:27.418 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 -- 172.21.15.61:0/587326875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80b6690 msgr2=0x7f1ad80b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.419 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 --2- 172.21.15.61:0/587326875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80b6690 0x7f1ad80b6a70 secure :-1 s=READY pgs=1545 cs=0 l=1 rev1=1 rx=0x7f1ac8002f70 tx=0x7f1ac8006eb0).stop 2022-07-20T04:18:27.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 -- 172.21.15.61:0/587326875 shutdown_connections 2022-07-20T04:18:27.420 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 --2- 172.21.15.61:0/587326875 >> v2:172.21.15.117:3300/0 conn(0x7f1ad80ae670 0x7f1ad80aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.421 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 --2- 172.21.15.61:0/587326875 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80b6690 0x7f1ad80b6a70 unknown :-1 s=CLOSED pgs=1545 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 -- 172.21.15.61:0/587326875 >> 172.21.15.61:0/587326875 conn(0x7f1ad80b9f70 msgr2=0x7f1ad80b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.422 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 -- 172.21.15.61:0/587326875 shutdown_connections 2022-07-20T04:18:27.423 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.396+0000 7f1adde0d700 1 -- 172.21.15.61:0/587326875 wait complete. 2022-07-20T04:18:27.424 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 Processor -- start 2022-07-20T04:18:27.425 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 -- start start 2022-07-20T04:18:27.426 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80ae670 0x7f1ad80b2760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.427 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ad80b59a0 0x7f1ad8141dc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ad80b6690 0x7f1ad8152500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.428 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad813ff00 con 0x7f1ad80ae670 2022-07-20T04:18:27.429 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad8141680 con 0x7f1ad80b6690 2022-07-20T04:18:27.430 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f1ad8141920 con 0x7f1ad80b59a0 2022-07-20T04:18:27.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80ae670 0x7f1ad80b2760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.431 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80ae670 0x7f1ad80b2760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52256/0 (socket says 172.21.15.61:52256) 2022-07-20T04:18:27.432 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 -- 172.21.15.61:0/133409816 learned_addr learned my addr 172.21.15.61:0/133409816 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:27.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1aceffd700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ad80b59a0 0x7f1ad8141dc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.433 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad7fff700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ad80b6690 0x7f1ad8152500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 -- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ad80b59a0 msgr2=0x7f1ad8141dc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.434 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f1ad80b59a0 0x7f1ad8141dc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.435 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 -- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ad80b6690 msgr2=0x7f1ad8152500 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f1ad80b6690 0x7f1ad8152500 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.436 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 -- 172.21.15.61:0/133409816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ad0009470 con 0x7f1ad80ae670 2022-07-20T04:18:27.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad77fe700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80ae670 0x7f1ad80b2760 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f1ac8002f70 tx=0x7f1ac8015000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.438 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ac8011760 con 0x7f1ad80ae670 2022-07-20T04:18:27.440 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1ac80118c0 con 0x7f1ad80ae670 2022-07-20T04:18:27.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ac800b1e0 con 0x7f1ad80ae670 2022-07-20T04:18:27.441 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f1ac8011c20 con 0x7f1ad80ae670 2022-07-20T04:18:27.442 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.400+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ad813f130 con 0x7f1ad80ae670 2022-07-20T04:18:27.446 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f1ad8141500 con 0x7f1ad80ae670 2022-07-20T04:18:27.447 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f1ac8025030 con 0x7f1ad80ae670 2022-07-20T04:18:27.448 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad57fa700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1abc053a40 0x7f1abc055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f1ac806cd90 con 0x7f1ad80ae670 2022-07-20T04:18:27.449 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1aceffd700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1abc053a40 0x7f1abc055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.450 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad57fa700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f1abc05a130 0x7f1abc05c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 --> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f1abc05cb70 con 0x7f1abc05a130 2022-07-20T04:18:27.451 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f1ac80722c0 con 0x7f1ad80ae670 2022-07-20T04:18:27.452 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad7fff700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f1abc05a130 0x7f1abc05c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1ad7fff700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f1abc05a130 0x7f1abc05c590 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.453 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.404+0000 7f1aceffd700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1abc053a40 0x7f1abc055f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f1ac000d000 tx=0x7f1ac0007b90).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.454 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.408+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== osd.1 v2:172.21.15.61:6808/22998 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f1abc05cb70 con 0x7f1abc05a130 2022-07-20T04:18:27.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.420+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 --> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f1ad8141b20 con 0x7f1abc05a130 2022-07-20T04:18:27.455 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.420+0000 7f1ad57fa700 1 -- 172.21.15.61:0/133409816 <== osd.1 v2:172.21.15.61:6808/22998 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f1ad0009ea0 con 0x7f1abc05a130 2022-07-20T04:18:27.456 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:27.457 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:27.458 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:27.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f1abc05a130 msgr2=0x7f1abc05c590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.462 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:6808/22998,v1:172.21.15.61:6809/22998] conn(0x7f1abc05a130 0x7f1abc05c590 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.463 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1abc053a40 msgr2=0x7f1abc055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f1abc053a40 0x7f1abc055f00 secure :-1 s=READY pgs=809 cs=0 l=1 rev1=1 rx=0x7f1ac000d000 tx=0x7f1ac0007b90).stop 2022-07-20T04:18:27.464 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80ae670 msgr2=0x7f1ad80b2760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.465 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 --2- 172.21.15.61:0/133409816 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f1ad80ae670 0x7f1ad80b2760 secure :-1 s=READY pgs=1546 cs=0 l=1 rev1=1 rx=0x7f1ac8002f70 tx=0x7f1ac8015000).stop 2022-07-20T04:18:27.466 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1ad7fff700 1 -- 172.21.15.61:0/133409816 reap_dead start 2022-07-20T04:18:27.467 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 shutdown_connections 2022-07-20T04:18:27.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 >> 172.21.15.61:0/133409816 conn(0x7f1ad80b9f70 msgr2=0x7f1ad80b8d10 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.468 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 shutdown_connections 2022-07-20T04:18:27.469 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.424+0000 7f1adde0d700 1 -- 172.21.15.61:0/133409816 wait complete. 2022-07-20T04:18:27.471 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 1 2022-07-20T04:18:27.588 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 Processor -- start 2022-07-20T04:18:27.589 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 -- start start 2022-07-20T04:18:27.599 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f76780b5630 0x7f76780b5a10 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.600 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f767813c7c0 0x7f767805bd90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.601 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f76780b0a10 con 0x7f767813cc50 2022-07-20T04:18:27.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76780b0cb0 con 0x7f767813c7c0 2022-07-20T04:18:27.602 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f767cb1d700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f76780b0f50 con 0x7f76780b5630 2022-07-20T04:18:27.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f767813c7c0 0x7f767805bd90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.603 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f767813c7c0 0x7f767805bd90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52266/0 (socket says 172.21.15.61:52266) 2022-07-20T04:18:27.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 -- 172.21.15.61:0/401893824 learned_addr learned my addr 172.21.15.61:0/401893824 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:27.604 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f76777fe700 1 --2- 172.21.15.61:0/401893824 >> v2:172.21.15.117:3300/0 conn(0x7f76780b5630 0x7f76780b5a10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.605 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 -- 172.21.15.61:0/401893824 >> v2:172.21.15.117:3300/0 conn(0x7f76780b5630 msgr2=0x7f76780b5a10 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:27.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 --2- 172.21.15.61:0/401893824 >> v2:172.21.15.117:3300/0 conn(0x7f76780b5630 0x7f76780b5a10 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.607 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 -- 172.21.15.61:0/401893824 >> v1:172.21.15.117:6789/0 conn(0x7f767813cc50 legacy=0x7f767805c280 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 -- 172.21.15.61:0/401893824 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76780afa10 con 0x7f767813c7c0 2022-07-20T04:18:27.608 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f7676ffd700 1 --2- 172.21.15.61:0/401893824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813c7c0 0x7f767805bd90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7f7668002f70 tx=0x7f7668006eb0).ready entity=mon.0 client_cookie=4df6502c143b0749 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.609 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f76767fc700 1 -- 172.21.15.61:0/401893824 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f766800d610 con 0x7f767813c7c0 2022-07-20T04:18:27.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f76767fc700 1 -- 172.21.15.61:0/401893824 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7668000e90 con 0x7f767813c7c0 2022-07-20T04:18:27.610 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.588+0000 7f76767fc700 1 -- 172.21.15.61:0/401893824 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f766800d9a0 con 0x7f767813c7c0 2022-07-20T04:18:27.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/401893824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813c7c0 msgr2=0x7f767805bd90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.612 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 --2- 172.21.15.61:0/401893824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813c7c0 0x7f767805bd90 secure :-1 s=READY pgs=1547 cs=0 l=1 rev1=1 rx=0x7f7668002f70 tx=0x7f7668006eb0).stop 2022-07-20T04:18:27.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/401893824 shutdown_connections 2022-07-20T04:18:27.613 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 --2- 172.21.15.61:0/401893824 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813c7c0 0x7f767805bd90 unknown :-1 s=CLOSED pgs=1547 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 --2- 172.21.15.61:0/401893824 >> v2:172.21.15.117:3300/0 conn(0x7f76780b5630 0x7f76780b5a10 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.614 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/401893824 >> 172.21.15.61:0/401893824 conn(0x7f7678063490 msgr2=0x7f76780b4f20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.615 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/401893824 shutdown_connections 2022-07-20T04:18:27.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/401893824 wait complete. 2022-07-20T04:18:27.616 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 Processor -- start 2022-07-20T04:18:27.617 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- start start 2022-07-20T04:18:27.618 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76780b5630 0x7f7678141620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f767813c7c0 0x7f767813e5f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.619 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 0x7f767813eae0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7678151ef0 con 0x7f767813cc50 2022-07-20T04:18:27.620 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76781425b0 con 0x7f767813c7c0 2022-07-20T04:18:27.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f76781407b0 con 0x7f76780b5630 2022-07-20T04:18:27.621 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 0x7f767813eae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.622 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 0x7f767813eae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52272/0 (socket says 172.21.15.61:52272) 2022-07-20T04:18:27.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 -- 172.21.15.61:0/1178635212 learned_addr learned my addr 172.21.15.61:0/1178635212 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:27.623 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7676ffd700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f767813c7c0 0x7f767813e5f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f76777fe700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76780b5630 0x7f7678141620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.624 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 -- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76780b5630 msgr2=0x7f7678141620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.625 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76780b5630 0x7f7678141620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 -- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f767813c7c0 msgr2=0x7f767813e5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.626 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f767813c7c0 0x7f767813e5f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 -- 172.21.15.61:0/1178635212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f766800b1e0 con 0x7f767813cc50 2022-07-20T04:18:27.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f76777fe700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76780b5630 0x7f7678141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T04:18:27.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7677fff700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 0x7f767813eae0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f76780b14d0 tx=0x7f7670007bf0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7670001d50 con 0x7f767813cc50 2022-07-20T04:18:27.629 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f767000ce30 con 0x7f767813cc50 2022-07-20T04:18:27.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76780b4b40 con 0x7f767813cc50 2022-07-20T04:18:27.630 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7670007430 con 0x7f767813cc50 2022-07-20T04:18:27.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.592+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7678140190 con 0x7f767813cc50 2022-07-20T04:18:27.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.596+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76781409b0 con 0x7f767813cc50 2022-07-20T04:18:27.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.596+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7670010030 con 0x7f767813cc50 2022-07-20T04:18:27.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.596+0000 7f7674ff9700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f765c053a40 0x7f765c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.596+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f767006bbb0 con 0x7f767813cc50 2022-07-20T04:18:27.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.596+0000 7f76777fe700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f765c053a40 0x7f765c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.596+0000 7f76777fe700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f765c053a40 0x7f765c055f00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f7660008000 tx=0x7f7660008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.600+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f7670042df0 con 0x7f767813cc50 2022-07-20T04:18:27.723 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.724+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "1", "target": ["mon-mgr", ""]}) v1 -- 0x7f7678142430 con 0x7f765c053a40 2022-07-20T04:18:27.724 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.724+0000 7f7674ff9700 1 -- 172.21.15.61:0/1178635212 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 1 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f766000f6e0 con 0x7f765c053a40 2022-07-20T04:18:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f765c053a40 msgr2=0x7f765c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.728 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f765c053a40 0x7f765c055f00 secure :-1 s=READY pgs=810 cs=0 l=1 rev1=1 rx=0x7f7660008000 tx=0x7f7660008040).stop 2022-07-20T04:18:27.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 msgr2=0x7f767813eae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 0x7f767813eae0 secure :-1 s=READY pgs=1548 cs=0 l=1 rev1=1 rx=0x7f76780b14d0 tx=0x7f7670007bf0).stop 2022-07-20T04:18:27.729 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 shutdown_connections 2022-07-20T04:18:27.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f765c053a40 0x7f765c055f00 unknown :-1 s=CLOSED pgs=810 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.730 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f767813c7c0 0x7f767813e5f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f767813cc50 0x7f767813eae0 unknown :-1 s=CLOSED pgs=1548 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 --2- 172.21.15.61:0/1178635212 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f76780b5630 0x7f7678141620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.731 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 >> 172.21.15.61:0/1178635212 conn(0x7f7678063490 msgr2=0x7f7678143310 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 shutdown_connections 2022-07-20T04:18:27.732 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.728+0000 7f767cb1d700 1 -- 172.21.15.61:0/1178635212 wait complete. 2022-07-20T04:18:27.734 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 1 to deep-scrub 2022-07-20T04:18:27.750 INFO:tasks.ceph:Scrubbing osd.2 2022-07-20T04:18:27.751 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.2 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:27.866 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.864+0000 7f7e5de0b700 1 Processor -- start 2022-07-20T04:18:27.867 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.864+0000 7f7e5de0b700 1 -- start start 2022-07-20T04:18:27.876 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580b4080 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e580b17f0 con 0x7f7e580b08e0 2022-07-20T04:18:27.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f7e580b5190 con 0x7f7e580b6200 2022-07-20T04:18:27.877 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580b4640 con 0x7f7e580b49f0 2022-07-20T04:18:27.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5d60a700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f7e580b08e0 0x7f7e58146260 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48880/0 (socket says 172.21.15.61:48880) 2022-07-20T04:18:27.878 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5d60a700 1 -- 172.21.15.61:0/888291540 learned_addr learned my addr 172.21.15.61:0/888291540 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:27.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e57fff700 1 --2- 172.21.15.61:0/888291540 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580b4080 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2955726269 0 0) 0x7f7e580b17f0 con 0x7f7e580b08e0 2022-07-20T04:18:27.879 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e3c003260 con 0x7f7e580b08e0 2022-07-20T04:18:27.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2335688332 0 0) 0x7f7e580b5190 con 0x7f7e580b6200 2022-07-20T04:18:27.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f7e580b17f0 con 0x7f7e580b6200 2022-07-20T04:18:27.880 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e57fff700 1 -- 172.21.15.61:0/888291540 >> v1:172.21.15.117:6789/0 conn(0x7f7e580b6200 legacy=0x7f7e580b01d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e57fff700 1 -- 172.21.15.61:0/888291540 >> v1:172.21.15.61:6789/0 conn(0x7f7e580b08e0 legacy=0x7f7e58146260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e57fff700 1 -- 172.21.15.61:0/888291540 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e580b65e0 con 0x7f7e580b49f0 2022-07-20T04:18:27.881 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e57fff700 1 --2- 172.21.15.61:0/888291540 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580b4080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7e44002f70 tx=0x7f7e44006df0).ready entity=mon.2 client_cookie=d472fb6c3fe89d46 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e4400d5c0 con 0x7f7e580b49f0 2022-07-20T04:18:27.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e44000e90 con 0x7f7e580b49f0 2022-07-20T04:18:27.882 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e577fe700 1 -- 172.21.15.61:0/888291540 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e4400d910 con 0x7f7e580b49f0 2022-07-20T04:18:27.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- 172.21.15.61:0/888291540 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 msgr2=0x7f7e580b4080 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 --2- 172.21.15.61:0/888291540 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580b4080 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7f7e44002f70 tx=0x7f7e44006df0).stop 2022-07-20T04:18:27.883 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- 172.21.15.61:0/888291540 shutdown_connections 2022-07-20T04:18:27.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 --2- 172.21.15.61:0/888291540 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580b4080 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.884 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- 172.21.15.61:0/888291540 >> 172.21.15.61:0/888291540 conn(0x7f7e58063d10 msgr2=0x7f7e580b57b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- 172.21.15.61:0/888291540 shutdown_connections 2022-07-20T04:18:27.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.868+0000 7f7e5de0b700 1 -- 172.21.15.61:0/888291540 wait complete. 2022-07-20T04:18:27.885 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 Processor -- start 2022-07-20T04:18:27.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 -- start start 2022-07-20T04:18:27.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e580b08e0 0x7f7e580ab5e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.886 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580ad600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e580b6200 0x7f7e5813eab0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.887 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580ae060 con 0x7f7e580b6200 2022-07-20T04:18:27.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580ac6b0 con 0x7f7e580b08e0 2022-07-20T04:18:27.888 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f7e580af070 con 0x7f7e580b49f0 2022-07-20T04:18:27.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e580b6200 0x7f7e5813eab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e580b6200 0x7f7e5813eab0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52290/0 (socket says 172.21.15.61:52290) 2022-07-20T04:18:27.889 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e57fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580ad600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 -- 172.21.15.61:0/992036642 learned_addr learned my addr 172.21.15.61:0/992036642 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:27.891 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5ce09700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e580b08e0 0x7f7e580ab5e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.892 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 -- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 msgr2=0x7f7e580ad600 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580ad600 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.893 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 -- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e580b08e0 msgr2=0x7f7e580ab5e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e580b08e0 0x7f7e580ab5e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.895 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 -- 172.21.15.61:0/992036642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7e4400b180 con 0x7f7e580b6200 2022-07-20T04:18:27.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e57fff700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f7e580b49f0 0x7f7e580ad600 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:27.896 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5ce09700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f7e580b08e0 0x7f7e580ab5e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:27.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5d60a700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e580b6200 0x7f7e5813eab0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f7e50001490 tx=0x7f7e50007e00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.897 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e50005ba0 con 0x7f7e580b6200 2022-07-20T04:18:27.898 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f7e50005d00 con 0x7f7e580b6200 2022-07-20T04:18:27.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f7e500072c0 con 0x7f7e580b6200 2022-07-20T04:18:27.899 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7e580af310 con 0x7f7e580b6200 2022-07-20T04:18:27.900 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.872+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7e580af850 con 0x7f7e580b6200 2022-07-20T04:18:27.904 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f7e580afa50 con 0x7f7e580b6200 2022-07-20T04:18:27.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f7e50010070 con 0x7f7e580b6200 2022-07-20T04:18:27.905 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e38053ad0 0x7f7e38055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f7e5006b4c0 con 0x7f7e580b6200 2022-07-20T04:18:27.906 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f7e3805a1c0 0x7f7e3805c620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:27.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 --> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f7e3805cc00 con 0x7f7e3805a1c0 2022-07-20T04:18:27.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f7e5006dad0 con 0x7f7e580b6200 2022-07-20T04:18:27.907 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e5ce09700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e38053ad0 0x7f7e38055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e57fff700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f7e3805a1c0 0x7f7e3805c620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:27.908 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e57fff700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f7e3805a1c0 0x7f7e3805c620 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.2 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e5ce09700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e38053ad0 0x7f7e38055f90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f7e4c00d000 tx=0x7f7e4c007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:27.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.876+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== osd.2 v2:172.21.15.61:6815/23001 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f7e3805cc00 con 0x7f7e3805a1c0 2022-07-20T04:18:27.909 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 --> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f7e5813f460 con 0x7f7e3805a1c0 2022-07-20T04:18:27.910 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e55ffb700 1 -- 172.21.15.61:0/992036642 <== osd.2 v2:172.21.15.61:6815/23001 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f7e5813f460 con 0x7f7e3805a1c0 2022-07-20T04:18:27.911 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:27.912 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:27.913 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:27.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f7e3805a1c0 msgr2=0x7f7e3805c620 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.916 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:6815/23001,v1:172.21.15.61:6817/23001] conn(0x7f7e3805a1c0 0x7f7e3805c620 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:27.917 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e38053ad0 msgr2=0x7f7e38055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f7e38053ad0 0x7f7e38055f90 secure :-1 s=READY pgs=811 cs=0 l=1 rev1=1 rx=0x7f7e4c00d000 tx=0x7f7e4c007a20).stop 2022-07-20T04:18:27.918 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e580b6200 msgr2=0x7f7e5813eab0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:27.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 --2- 172.21.15.61:0/992036642 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f7e580b6200 0x7f7e5813eab0 secure :-1 s=READY pgs=1550 cs=0 l=1 rev1=1 rx=0x7f7e50001490 tx=0x7f7e50007e00).stop 2022-07-20T04:18:27.919 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5d60a700 1 -- 172.21.15.61:0/992036642 reap_dead start 2022-07-20T04:18:27.920 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 shutdown_connections 2022-07-20T04:18:27.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 >> 172.21.15.61:0/992036642 conn(0x7f7e58063d10 msgr2=0x7f7e58146c60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:27.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 shutdown_connections 2022-07-20T04:18:27.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:27.896+0000 7f7e5de0b700 1 -- 172.21.15.61:0/992036642 wait complete. 2022-07-20T04:18:27.922 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 2 2022-07-20T04:18:28.036 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e092d7700 1 Processor -- start 2022-07-20T04:18:28.037 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e092d7700 1 -- start start 2022-07-20T04:18:28.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e092d7700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4e0405c7c0 0x7f4e040952b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.040 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e092d7700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e040948d0 con 0x7f4e040ab250 2022-07-20T04:18:28.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e092d7700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f4e04094b70 con 0x7f4e04146290 2022-07-20T04:18:28.041 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e092d7700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f4e04093db0 con 0x7f4e0405c7c0 2022-07-20T04:18:28.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e03fff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f4e0405c7c0 0x7f4e040952b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e08ad6700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f4e040ab250 0x7f4e041420a0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48896/0 (socket says 172.21.15.61:48896) 2022-07-20T04:18:28.042 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.036+0000 7f4e08ad6700 1 -- 172.21.15.61:0/3302520909 learned_addr learned my addr 172.21.15.61:0/3302520909 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:28.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e02ffd700 1 -- 172.21.15.61:0/3302520909 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1860959492 0 0) 0x7f4e040948d0 con 0x7f4e040ab250 2022-07-20T04:18:28.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e02ffd700 1 -- 172.21.15.61:0/3302520909 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f4df0003260 con 0x7f4e040ab250 2022-07-20T04:18:28.043 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e03fff700 1 -- 172.21.15.61:0/3302520909 >> v1:172.21.15.117:6789/0 conn(0x7f4e04146290 legacy=0x7f4e040ab760 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e03fff700 1 -- 172.21.15.61:0/3302520909 >> v1:172.21.15.61:6789/0 conn(0x7f4e040ab250 legacy=0x7f4e041420a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.044 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e03fff700 1 -- 172.21.15.61:0/3302520909 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4e04094e10 con 0x7f4e0405c7c0 2022-07-20T04:18:28.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e03fff700 1 --2- 172.21.15.61:0/3302520909 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e0405c7c0 0x7f4e040952b0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f4df4002f70 tx=0x7f4df4006df0).ready entity=mon.0 client_cookie=7253a9995bfa5063 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e02ffd700 1 -- 172.21.15.61:0/3302520909 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4df400d610 con 0x7f4e0405c7c0 2022-07-20T04:18:28.045 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e02ffd700 1 -- 172.21.15.61:0/3302520909 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df4000e90 con 0x7f4e0405c7c0 2022-07-20T04:18:28.046 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e02ffd700 1 -- 172.21.15.61:0/3302520909 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4df400d960 con 0x7f4e0405c7c0 2022-07-20T04:18:28.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 -- 172.21.15.61:0/3302520909 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e0405c7c0 msgr2=0x7f4e040952b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.050 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 --2- 172.21.15.61:0/3302520909 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e0405c7c0 0x7f4e040952b0 secure :-1 s=READY pgs=1552 cs=0 l=1 rev1=1 rx=0x7f4df4002f70 tx=0x7f4df4006df0).stop 2022-07-20T04:18:28.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 -- 172.21.15.61:0/3302520909 shutdown_connections 2022-07-20T04:18:28.051 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 --2- 172.21.15.61:0/3302520909 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e0405c7c0 0x7f4e040952b0 unknown :-1 s=CLOSED pgs=1552 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 -- 172.21.15.61:0/3302520909 >> 172.21.15.61:0/3302520909 conn(0x7f4e040b6210 msgr2=0x7f4e040b65f0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.052 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 -- 172.21.15.61:0/3302520909 shutdown_connections 2022-07-20T04:18:28.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.040+0000 7f4e092d7700 1 -- 172.21.15.61:0/3302520909 wait complete. 2022-07-20T04:18:28.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 Processor -- start 2022-07-20T04:18:28.053 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 -- start start 2022-07-20T04:18:28.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0405c7c0 0x7f4e040651d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.054 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 0x7f4e0405f9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e04146290 0x7f4e040658f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.055 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04060440 con 0x7f4e040ab250 2022-07-20T04:18:28.056 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4e040663a0 con 0x7f4e04146290 2022-07-20T04:18:28.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f4e04066640 con 0x7f4e0405c7c0 2022-07-20T04:18:28.057 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 0x7f4e0405f9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e03fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0405c7c0 0x7f4e040651d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.058 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 0x7f4e0405f9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52304/0 (socket says 172.21.15.61:52304) 2022-07-20T04:18:28.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 -- 172.21.15.61:0/1979374349 learned_addr learned my addr 172.21.15.61:0/1979374349 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.059 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e08ad6700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e04146290 0x7f4e040658f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.060 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 -- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0405c7c0 msgr2=0x7f4e040651d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0405c7c0 0x7f4e040651d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.061 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 -- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e04146290 msgr2=0x7f4e040658f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e04146290 0x7f4e040658f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.062 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 -- 172.21.15.61:0/1979374349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4df400b180 con 0x7f4e040ab250 2022-07-20T04:18:28.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e037fe700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 0x7f4e0405f9e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f4e0405d6d0 tx=0x7f4df8007c00).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.063 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4df8007e70 con 0x7f4e040ab250 2022-07-20T04:18:28.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f4df8007030 con 0x7f4e040ab250 2022-07-20T04:18:28.064 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4e0409ac90 con 0x7f4e040ab250 2022-07-20T04:18:28.065 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f4df8007390 con 0x7f4e040ab250 2022-07-20T04:18:28.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.044+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4e0409bda0 con 0x7f4e040ab250 2022-07-20T04:18:28.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.048+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4e040b2760 con 0x7f4e040ab250 2022-07-20T04:18:28.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.048+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f4df8010030 con 0x7f4e040ab250 2022-07-20T04:18:28.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.048+0000 7f4e017fa700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4de40539b0 0x7f4de4055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.048+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f4df806b630 con 0x7f4e040ab250 2022-07-20T04:18:28.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.048+0000 7f4e03fff700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4de40539b0 0x7f4de4055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.068 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.048+0000 7f4e03fff700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4de40539b0 0x7f4de4055e70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f4df401a000 tx=0x7f4df4014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.069 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.056+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f4df8042870 con 0x7f4e040ab250 2022-07-20T04:18:28.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.180+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "2", "target": ["mon-mgr", ""]}) v1 -- 0x7f4e04064f30 con 0x7f4de40539b0 2022-07-20T04:18:28.182 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.180+0000 7f4e017fa700 1 -- 172.21.15.61:0/1979374349 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 2 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f4df4017070 con 0x7f4de40539b0 2022-07-20T04:18:28.184 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4de40539b0 msgr2=0x7f4de4055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4de40539b0 0x7f4de4055e70 secure :-1 s=READY pgs=812 cs=0 l=1 rev1=1 rx=0x7f4df401a000 tx=0x7f4df4014000).stop 2022-07-20T04:18:28.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 msgr2=0x7f4e0405f9e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.185 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 0x7f4e0405f9e0 secure :-1 s=READY pgs=1553 cs=0 l=1 rev1=1 rx=0x7f4e0405d6d0 tx=0x7f4df8007c00).stop 2022-07-20T04:18:28.186 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 shutdown_connections 2022-07-20T04:18:28.187 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f4de40539b0 0x7f4de4055e70 unknown :-1 s=CLOSED pgs=812 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f4e04146290 0x7f4e040658f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f4e040ab250 0x7f4e0405f9e0 unknown :-1 s=CLOSED pgs=1553 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.188 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 --2- 172.21.15.61:0/1979374349 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f4e0405c7c0 0x7f4e040651d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.189 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 >> 172.21.15.61:0/1979374349 conn(0x7f4e040b6210 msgr2=0x7f4e040ab6c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 shutdown_connections 2022-07-20T04:18:28.190 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.184+0000 7f4e092d7700 1 -- 172.21.15.61:0/1979374349 wait complete. 2022-07-20T04:18:28.191 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 2 to deep-scrub 2022-07-20T04:18:28.194 INFO:tasks.ceph:Scrubbing osd.3 2022-07-20T04:18:28.195 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.3 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:28.304 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 Processor -- start 2022-07-20T04:18:28.305 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 -- start start 2022-07-20T04:18:28.309 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f25980b6690 0x7f25980b6a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f25980ae670 0x7f25980aeae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.310 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f25980acae0 con 0x7f25980b59a0 2022-07-20T04:18:28.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f2598147100 con 0x7f25980b6690 2022-07-20T04:18:28.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f259d9cd700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f25980acd10 con 0x7f25980ae670 2022-07-20T04:18:28.311 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f25967fc700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f25980b59a0 0x7f259813c970 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48908/0 (socket says 172.21.15.61:48908) 2022-07-20T04:18:28.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f2596ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f25980b6690 0x7f25980b6a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f25967fc700 1 -- 172.21.15.61:0/1095608648 learned_addr learned my addr 172.21.15.61:0/1095608648 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:28.312 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.304+0000 7f25977fe700 1 --2- 172.21.15.61:0/1095608648 >> v2:172.21.15.117:3300/0 conn(0x7f25980ae670 0x7f25980aeae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2595ffb700 1 -- 172.21.15.61:0/1095608648 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1640735366 0 0) 0x7f25980acae0 con 0x7f25980b59a0 2022-07-20T04:18:28.313 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2595ffb700 1 -- 172.21.15.61:0/1095608648 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f257c003260 con 0x7f25980b59a0 2022-07-20T04:18:28.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2596ffd700 1 -- 172.21.15.61:0/1095608648 >> v2:172.21.15.117:3300/0 conn(0x7f25980ae670 msgr2=0x7f25980aeae0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:28.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2595ffb700 1 -- 172.21.15.61:0/1095608648 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 1088624661 0 0) 0x7f257c003260 con 0x7f25980b59a0 2022-07-20T04:18:28.314 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2596ffd700 1 --2- 172.21.15.61:0/1095608648 >> v2:172.21.15.117:3300/0 conn(0x7f25980ae670 0x7f25980aeae0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.315 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2596ffd700 1 -- 172.21.15.61:0/1095608648 >> v1:172.21.15.61:6789/0 conn(0x7f25980b59a0 legacy=0x7f259813c970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.317 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2596ffd700 1 -- 172.21.15.61:0/1095608648 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25980b5df0 con 0x7f25980b6690 2022-07-20T04:18:28.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2596ffd700 1 --2- 172.21.15.61:0/1095608648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b6690 0x7f25980b6a70 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7f2588001800 tx=0x7f258c006df0).ready entity=mon.0 client_cookie=acc1cc221b7d2639 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2595ffb700 1 -- 172.21.15.61:0/1095608648 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2588001660 con 0x7f25980b6690 2022-07-20T04:18:28.318 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2595ffb700 1 -- 172.21.15.61:0/1095608648 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f258c000e90 con 0x7f25980b6690 2022-07-20T04:18:28.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f2595ffb700 1 -- 172.21.15.61:0/1095608648 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f258c012460 con 0x7f25980b6690 2022-07-20T04:18:28.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 -- 172.21.15.61:0/1095608648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b6690 msgr2=0x7f25980b6a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.319 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 --2- 172.21.15.61:0/1095608648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b6690 0x7f25980b6a70 secure :-1 s=READY pgs=1555 cs=0 l=1 rev1=1 rx=0x7f2588001800 tx=0x7f258c006df0).stop 2022-07-20T04:18:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 -- 172.21.15.61:0/1095608648 shutdown_connections 2022-07-20T04:18:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 --2- 172.21.15.61:0/1095608648 >> v2:172.21.15.117:3300/0 conn(0x7f25980ae670 0x7f25980aeae0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.320 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 --2- 172.21.15.61:0/1095608648 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b6690 0x7f25980b6a70 unknown :-1 s=CLOSED pgs=1555 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 -- 172.21.15.61:0/1095608648 >> 172.21.15.61:0/1095608648 conn(0x7f25980b9f70 msgr2=0x7f25980b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.321 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 -- 172.21.15.61:0/1095608648 shutdown_connections 2022-07-20T04:18:28.322 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 -- 172.21.15.61:0/1095608648 wait complete. 2022-07-20T04:18:28.327 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 Processor -- start 2022-07-20T04:18:28.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.308+0000 7f259d9cd700 1 -- start start 2022-07-20T04:18:28.329 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25980ae670 0x7f25980b2df0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.330 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b59a0 0x7f259813d2f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25980b6690 0x7f2598140af0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.331 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f25981402a0 con 0x7f25980b59a0 2022-07-20T04:18:28.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f259813f100 con 0x7f25980ae670 2022-07-20T04:18:28.332 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f259813f3a0 con 0x7f25980b6690 2022-07-20T04:18:28.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b59a0 0x7f259813d2f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.333 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b59a0 0x7f259813d2f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52318/0 (socket says 172.21.15.61:52318) 2022-07-20T04:18:28.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 -- 172.21.15.61:0/4065680982 learned_addr learned my addr 172.21.15.61:0/4065680982 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.334 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f2596ffd700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25980ae670 0x7f25980b2df0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25977fe700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25980b6690 0x7f2598140af0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.335 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 -- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25980b6690 msgr2=0x7f2598140af0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f25980b6690 0x7f2598140af0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 -- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25980ae670 msgr2=0x7f25980b2df0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.336 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f25980ae670 0x7f25980b2df0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.337 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 -- 172.21.15.61:0/4065680982 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f258c00b180 con 0x7f25980b59a0 2022-07-20T04:18:28.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f25967fc700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b59a0 0x7f259813d2f0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f2588001840 tx=0x7f2588001920).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.338 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2588007170 con 0x7f25980b59a0 2022-07-20T04:18:28.339 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f25880072d0 con 0x7f25980b59a0 2022-07-20T04:18:28.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f259813d7e0 con 0x7f25980b59a0 2022-07-20T04:18:28.340 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f2588004920 con 0x7f25980b59a0 2022-07-20T04:18:28.341 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f259813ed00 con 0x7f25980b59a0 2022-07-20T04:18:28.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.312+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f2598140120 con 0x7f25980b59a0 2022-07-20T04:18:28.342 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f2588004a80 con 0x7f25980b59a0 2022-07-20T04:18:28.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25740539b0 0x7f2574055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.343 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f258806b720 con 0x7f25980b59a0 2022-07-20T04:18:28.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7f257405a0a0 0x7f257405c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.344 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 --> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f257405cb80 con 0x7f257405a0a0 2022-07-20T04:18:28.345 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f258806ddb0 con 0x7f25980b59a0 2022-07-20T04:18:28.346 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2596ffd700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25740539b0 0x7f2574055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.348 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f25977fe700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7f257405a0a0 0x7f257405c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.349 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f25977fe700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7f257405a0a0 0x7f257405c500 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.3 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.350 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2596ffd700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25740539b0 0x7f2574055e70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f258c018000 tx=0x7f258c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.351 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.316+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== osd.3 v2:172.21.15.61:6824/23120 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f257405cb80 con 0x7f257405a0a0 2022-07-20T04:18:28.352 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 --> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f25980acfb0 con 0x7f257405a0a0 2022-07-20T04:18:28.353 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f2587fff700 1 -- 172.21.15.61:0/4065680982 <== osd.3 v2:172.21.15.61:6824/23120 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f25980acfb0 con 0x7f257405a0a0 2022-07-20T04:18:28.354 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:28.355 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:28.356 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:28.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7f257405a0a0 msgr2=0x7f257405c500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.361 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:6824/23120,v1:172.21.15.61:6825/23120] conn(0x7f257405a0a0 0x7f257405c500 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.362 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25740539b0 msgr2=0x7f2574055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f25740539b0 0x7f2574055e70 secure :-1 s=READY pgs=813 cs=0 l=1 rev1=1 rx=0x7f258c018000 tx=0x7f258c014000).stop 2022-07-20T04:18:28.363 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b59a0 msgr2=0x7f259813d2f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.364 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 --2- 172.21.15.61:0/4065680982 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f25980b59a0 0x7f259813d2f0 secure :-1 s=READY pgs=1556 cs=0 l=1 rev1=1 rx=0x7f2588001840 tx=0x7f2588001920).stop 2022-07-20T04:18:28.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f25977fe700 1 -- 172.21.15.61:0/4065680982 reap_dead start 2022-07-20T04:18:28.365 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 shutdown_connections 2022-07-20T04:18:28.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 >> 172.21.15.61:0/4065680982 conn(0x7f25980b9f70 msgr2=0x7f25980b5f80 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.367 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 shutdown_connections 2022-07-20T04:18:28.368 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.332+0000 7f259d9cd700 1 -- 172.21.15.61:0/4065680982 wait complete. 2022-07-20T04:18:28.369 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 3 2022-07-20T04:18:28.485 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 Processor -- start 2022-07-20T04:18:28.486 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 -- start start 2022-07-20T04:18:28.500 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040b4a90 0x7fce040af7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fce040aeef0 0x7fce040b0ab0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce040b0fa0 0x7fce040b1410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce040b4e70 con 0x7fce040b0fa0 2022-07-20T04:18:28.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fce040b00e0 con 0x7fce040aeef0 2022-07-20T04:18:28.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce09b2b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce040b0380 con 0x7fce040b4a90 2022-07-20T04:18:28.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.484+0000 7fce0932a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce040b0fa0 0x7fce040b1410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce03fff700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fce040aeef0 0x7fce040b0ab0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce08b29700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040b4a90 0x7fce040af7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fce040b0fa0 0x7fce040b1410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52330/0 (socket says 172.21.15.61:52330) 2022-07-20T04:18:28.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce08b29700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040b4a90 0x7fce040af7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3301/0 says I am v2:172.21.15.61:38998/0 (socket says 172.21.15.61:38998) 2022-07-20T04:18:28.519 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 -- 172.21.15.61:0/1117038318 learned_addr learned my addr 172.21.15.61:0/1117038318 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 -- 172.21.15.61:0/1117038318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040b4a90 msgr2=0x7fce040af7f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:28.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 --2- 172.21.15.61:0/1117038318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040b4a90 0x7fce040af7f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 -- 172.21.15.61:0/1117038318 >> v2:172.21.15.117:3300/0 conn(0x7fce040aeef0 msgr2=0x7fce040b0ab0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:28.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 --2- 172.21.15.61:0/1117038318 >> v2:172.21.15.117:3300/0 conn(0x7fce040aeef0 0x7fce040b0ab0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 -- 172.21.15.61:0/1117038318 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fce040ace30 con 0x7fce040b0fa0 2022-07-20T04:18:28.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce0932a700 1 --2- 172.21.15.61:0/1117038318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce040b1410 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7fcdfc0032f0 tx=0x7fcdfc009290).ready entity=mon.0 client_cookie=3838e3b05cf5df7d server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce037fe700 1 -- 172.21.15.61:0/1117038318 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdfc00b680 con 0x7fce040b0fa0 2022-07-20T04:18:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce037fe700 1 -- 172.21.15.61:0/1117038318 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdfc00b7e0 con 0x7fce040b0fa0 2022-07-20T04:18:28.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce037fe700 1 -- 172.21.15.61:0/1117038318 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdfc00b950 con 0x7fce040b0fa0 2022-07-20T04:18:28.526 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 -- 172.21.15.61:0/1117038318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 msgr2=0x7fce040b1410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1117038318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce040b1410 secure :-1 s=READY pgs=1557 cs=0 l=1 rev1=1 rx=0x7fcdfc0032f0 tx=0x7fcdfc009290).stop 2022-07-20T04:18:28.527 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 -- 172.21.15.61:0/1117038318 shutdown_connections 2022-07-20T04:18:28.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1117038318 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce040b1410 unknown :-1 s=CLOSED pgs=1557 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.530 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1117038318 >> v2:172.21.15.117:3300/0 conn(0x7fce040aeef0 0x7fce040b0ab0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.531 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1117038318 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040b4a90 0x7fce040af7f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 -- 172.21.15.61:0/1117038318 >> 172.21.15.61:0/1117038318 conn(0x7fce04063150 msgr2=0x7fce04063530 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.532 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 -- 172.21.15.61:0/1117038318 shutdown_connections 2022-07-20T04:18:28.533 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.488+0000 7fce09b2b700 1 -- 172.21.15.61:0/1117038318 wait complete. 2022-07-20T04:18:28.534 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 Processor -- start 2022-07-20T04:18:28.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 -- start start 2022-07-20T04:18:28.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040aeef0 0x7fce040b5620 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce041428f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce040b4a90 0x7fce040a9c30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04142de0 con 0x7fce040b0fa0 2022-07-20T04:18:28.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce04143000 con 0x7fce040b4a90 2022-07-20T04:18:28.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fce041432a0 con 0x7fce040aeef0 2022-07-20T04:18:28.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce041428f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce041428f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52334/0 (socket says 172.21.15.61:52334) 2022-07-20T04:18:28.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 -- 172.21.15.61:0/1380737619 learned_addr learned my addr 172.21.15.61:0/1380737619 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce08b29700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040aeef0 0x7fce040b5620 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce0932a700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce040b4a90 0x7fce040a9c30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 -- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040aeef0 msgr2=0x7fce040b5620 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.542 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040aeef0 0x7fce040b5620 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 -- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce040b4a90 msgr2=0x7fce040a9c30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.543 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce040b4a90 0x7fce040a9c30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.544 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 -- 172.21.15.61:0/1380737619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdfc009530 con 0x7fce040b0fa0 2022-07-20T04:18:28.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce08b29700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040aeef0 0x7fce040b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T04:18:28.545 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce03fff700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce041428f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7fcdf800c000 tx=0x7fcdf8006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf8010780 con 0x7fce040b0fa0 2022-07-20T04:18:28.546 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fcdf80108e0 con 0x7fce040b0fa0 2022-07-20T04:18:28.547 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce04143540 con 0x7fce040b0fa0 2022-07-20T04:18:28.548 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fcdf8010c40 con 0x7fce040b0fa0 2022-07-20T04:18:28.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.492+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce040aa7c0 con 0x7fce040b0fa0 2022-07-20T04:18:28.549 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.496+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce040574e0 con 0x7fce040b0fa0 2022-07-20T04:18:28.550 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.496+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fcdf801c460 con 0x7fce040b0fa0 2022-07-20T04:18:28.551 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.496+0000 7fce01ffb700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcde4053a60 0x7fcde4055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.552 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.496+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fcdf806bfd0 con 0x7fce040b0fa0 2022-07-20T04:18:28.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.496+0000 7fce08b29700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcde4053a60 0x7fcde4055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.553 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.496+0000 7fce08b29700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcde4053a60 0x7fcde4055f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fcdf400b000 tx=0x7fcdf4008040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.555 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.500+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fcdf80431b0 con 0x7fce040b0fa0 2022-07-20T04:18:28.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.628+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "3", "target": ["mon-mgr", ""]}) v1 -- 0x7fce040b29e0 con 0x7fcde4053a60 2022-07-20T04:18:28.628 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.628+0000 7fce01ffb700 1 -- 172.21.15.61:0/1380737619 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 3 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fce040b29e0 con 0x7fcde4053a60 2022-07-20T04:18:28.631 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.628+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcde4053a60 msgr2=0x7fcde4055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.628+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcde4053a60 0x7fcde4055f20 secure :-1 s=READY pgs=814 cs=0 l=1 rev1=1 rx=0x7fcdf400b000 tx=0x7fcdf4008040).stop 2022-07-20T04:18:28.632 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.628+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 msgr2=0x7fce041428f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.628+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce041428f0 secure :-1 s=READY pgs=1558 cs=0 l=1 rev1=1 rx=0x7fcdf800c000 tx=0x7fcdf8006df0).stop 2022-07-20T04:18:28.633 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 shutdown_connections 2022-07-20T04:18:28.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fcde4053a60 0x7fcde4055f20 unknown :-1 s=CLOSED pgs=814 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fce040b4a90 0x7fce040a9c30 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fce040b0fa0 0x7fce041428f0 unknown :-1 s=CLOSED pgs=1558 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.635 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 --2- 172.21.15.61:0/1380737619 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fce040aeef0 0x7fce040b5620 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 >> 172.21.15.61:0/1380737619 conn(0x7fce04063150 msgr2=0x7fce040b1f00 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.636 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 shutdown_connections 2022-07-20T04:18:28.637 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.632+0000 7fce09b2b700 1 -- 172.21.15.61:0/1380737619 wait complete. 2022-07-20T04:18:28.638 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 3 to deep-scrub 2022-07-20T04:18:28.642 INFO:tasks.ceph:Scrubbing osd.4 2022-07-20T04:18:28.643 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.4 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:28.752 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 Processor -- start 2022-07-20T04:18:28.753 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 -- start start 2022-07-20T04:18:28.756 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6c080ab180 0x7f6c08061ba0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f6c08142720 0x7f6c08142fd0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6c0805f840 con 0x7f6c080b34d0 2022-07-20T04:18:28.757 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c08142b90 con 0x7f6c080ab180 2022-07-20T04:18:28.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c0ddde700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f6c0805fa70 con 0x7f6c08142720 2022-07-20T04:18:28.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c06ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6c080ab180 0x7f6c08061ba0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.758 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c06ffd700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f6c080ab180 0x7f6c08061ba0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52342/0 (socket says 172.21.15.61:52342) 2022-07-20T04:18:28.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c06ffd700 1 -- 172.21.15.61:0/2309696592 learned_addr learned my addr 172.21.15.61:0/2309696592 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c077fe700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6c080b34d0 0x7f6c08142010 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48938/0 (socket says 172.21.15.61:48938) 2022-07-20T04:18:28.759 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.752+0000 7f6c07fff700 1 --2- 172.21.15.61:0/2309696592 >> v2:172.21.15.117:3300/0 conn(0x7f6c08142720 0x7f6c08142fd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c06ffd700 1 -- 172.21.15.61:0/2309696592 >> v2:172.21.15.117:3300/0 conn(0x7f6c08142720 msgr2=0x7f6c08142fd0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:28.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c06ffd700 1 --2- 172.21.15.61:0/2309696592 >> v2:172.21.15.117:3300/0 conn(0x7f6c08142720 0x7f6c08142fd0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.760 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c06ffd700 1 -- 172.21.15.61:0/2309696592 >> v1:172.21.15.61:6789/0 conn(0x7f6c080b34d0 legacy=0x7f6c08142010 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c06ffd700 1 -- 172.21.15.61:0/2309696592 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c0805fd10 con 0x7f6c080ab180 2022-07-20T04:18:28.761 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c07fff700 1 --2- 172.21.15.61:0/2309696592 >> v2:172.21.15.117:3300/0 conn(0x7f6c08142720 0x7f6c08142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_reply_more state changed! 2022-07-20T04:18:28.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c06ffd700 1 --2- 172.21.15.61:0/2309696592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c080ab180 0x7f6c08061ba0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f6c08060870 tx=0x7f6bf0006eb0).ready entity=mon.0 client_cookie=79ce1572aa8dd16e server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c067fc700 1 -- 172.21.15.61:0/2309696592 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bf000d610 con 0x7f6c080ab180 2022-07-20T04:18:28.762 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c067fc700 1 -- 172.21.15.61:0/2309696592 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6bf0000e90 con 0x7f6c080ab180 2022-07-20T04:18:28.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c067fc700 1 -- 172.21.15.61:0/2309696592 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6bf000d960 con 0x7f6c080ab180 2022-07-20T04:18:28.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2309696592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c080ab180 msgr2=0x7f6c08061ba0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.763 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 --2- 172.21.15.61:0/2309696592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c080ab180 0x7f6c08061ba0 secure :-1 s=READY pgs=1560 cs=0 l=1 rev1=1 rx=0x7f6c08060870 tx=0x7f6bf0006eb0).stop 2022-07-20T04:18:28.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2309696592 shutdown_connections 2022-07-20T04:18:28.764 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 --2- 172.21.15.61:0/2309696592 >> v2:172.21.15.117:3300/0 conn(0x7f6c08142720 0x7f6c08142fd0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 --2- 172.21.15.61:0/2309696592 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c080ab180 0x7f6c08061ba0 unknown :-1 s=CLOSED pgs=1560 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2309696592 >> 172.21.15.61:0/2309696592 conn(0x7f6c080b6c50 msgr2=0x7f6c080b2c20 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.765 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2309696592 shutdown_connections 2022-07-20T04:18:28.766 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2309696592 wait complete. 2022-07-20T04:18:28.771 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 Processor -- start 2022-07-20T04:18:28.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.756+0000 7f6c0ddde700 1 -- start start 2022-07-20T04:18:28.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c080ab180 0x7f6c080ae0e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.772 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c080b34d0 0x7f6c08099fc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.773 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c08142720 0x7f6c08152950 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c080ad6a0 con 0x7f6c08142720 2022-07-20T04:18:28.774 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c081458c0 con 0x7f6c080b34d0 2022-07-20T04:18:28.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6c0809a640 con 0x7f6c080ab180 2022-07-20T04:18:28.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c08142720 0x7f6c08152950 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.775 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c08142720 0x7f6c08152950 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52350/0 (socket says 172.21.15.61:52350) 2022-07-20T04:18:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 -- 172.21.15.61:0/2656115948 learned_addr learned my addr 172.21.15.61:0/2656115948 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c077fe700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c080ab180 0x7f6c080ae0e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.776 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c06ffd700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c080b34d0 0x7f6c08099fc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 -- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c080ab180 msgr2=0x7f6c080ae0e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6c080ab180 0x7f6c080ae0e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.777 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 -- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c080b34d0 msgr2=0x7f6c08099fc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6c080b34d0 0x7f6c08099fc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.778 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 -- 172.21.15.61:0/2656115948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6bf000b1e0 con 0x7f6c08142720 2022-07-20T04:18:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c07fff700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c08142720 0x7f6c08152950 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7f6c08145fa0 tx=0x7f6c00008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.779 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c0000f710 con 0x7f6c08142720 2022-07-20T04:18:28.780 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6c0000f870 con 0x7f6c08142720 2022-07-20T04:18:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6c0000fbd0 con 0x7f6c08142720 2022-07-20T04:18:28.781 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c08152e40 con 0x7f6c08142720 2022-07-20T04:18:28.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c0809a870 con 0x7f6c08142720 2022-07-20T04:18:28.782 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.760+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6c0809aa70 con 0x7f6c08142720 2022-07-20T04:18:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6c0001a030 con 0x7f6c08142720 2022-07-20T04:18:28.783 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c04ff9700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6bec053940 0x7f6bec055e00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.784 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f6c0006bae0 con 0x7f6c08142720 2022-07-20T04:18:28.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c04ff9700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f6bec05a0d0 0x7f6bec05c530 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.785 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 --> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6bec05cbb0 con 0x7f6bec05a0d0 2022-07-20T04:18:28.786 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6c00042c20 con 0x7f6c08142720 2022-07-20T04:18:28.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c077fe700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6bec053940 0x7f6bec055e00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.787 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c06ffd700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f6bec05a0d0 0x7f6bec05c530 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.788 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c077fe700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6bec053940 0x7f6bec055e00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f6bf800d000 tx=0x7f6bf8007b40).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.789 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.764+0000 7f6c06ffd700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f6bec05a0d0 0x7f6bec05c530 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.4 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.768+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== osd.4 v2:172.21.15.117:6802/23184 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f6bec05cbb0 con 0x7f6bec05a0d0 2022-07-20T04:18:28.792 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 --> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6c080b5960 con 0x7f6bec05a0d0 2022-07-20T04:18:28.793 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c04ff9700 1 -- 172.21.15.61:0/2656115948 <== osd.4 v2:172.21.15.117:6802/23184 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6c080b5960 con 0x7f6bec05a0d0 2022-07-20T04:18:28.794 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:28.795 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:28.795 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:28.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f6bec05a0d0 msgr2=0x7f6bec05c530 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6802/23184,v1:172.21.15.117:6803/23184] conn(0x7f6bec05a0d0 0x7f6bec05c530 crc :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6bec053940 msgr2=0x7f6bec055e00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6bec053940 0x7f6bec055e00 secure :-1 s=READY pgs=815 cs=0 l=1 rev1=1 rx=0x7f6bf800d000 tx=0x7f6bf8007b40).stop 2022-07-20T04:18:28.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c08142720 msgr2=0x7f6c08152950 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 --2- 172.21.15.61:0/2656115948 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6c08142720 0x7f6c08152950 secure :-1 s=READY pgs=1561 cs=0 l=1 rev1=1 rx=0x7f6c08145fa0 tx=0x7f6c00008040).stop 2022-07-20T04:18:28.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c07fff700 1 -- 172.21.15.61:0/2656115948 reap_dead start 2022-07-20T04:18:28.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 shutdown_connections 2022-07-20T04:18:28.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 >> 172.21.15.61:0/2656115948 conn(0x7f6c080b6c50 msgr2=0x7f6c080b1730 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 shutdown_connections 2022-07-20T04:18:28.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.784+0000 7f6c0ddde700 1 -- 172.21.15.61:0/2656115948 wait complete. 2022-07-20T04:18:28.809 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 4 2022-07-20T04:18:28.921 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb57b534700 1 Processor -- start 2022-07-20T04:18:28.922 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb57b534700 1 -- start start 2022-07-20T04:18:28.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb57b534700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b5320 0x7fb57405c4a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.926 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb57b534700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5740b0c20 con 0x7fb5740b9eb0 2022-07-20T04:18:28.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb57b534700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fb5740ba290 con 0x7fb57405c990 2022-07-20T04:18:28.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb57b534700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb574142e00 con 0x7fb5740b5320 2022-07-20T04:18:28.927 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb5792d0700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fb5740b9eb0 0x7fb57405bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48954/0 (socket says 172.21.15.61:48954) 2022-07-20T04:18:28.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.920+0000 7fb5792d0700 1 -- 172.21.15.61:0/24496670 learned_addr learned my addr 172.21.15.61:0/24496670 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:28.928 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb578acf700 1 --2- 172.21.15.61:0/24496670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b5320 0x7fb57405c4a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.929 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2465030574 0 0) 0x7fb5740b0c20 con 0x7fb5740b9eb0 2022-07-20T04:18:28.930 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb554003260 con 0x7fb5740b9eb0 2022-07-20T04:18:28.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3862095018 0 0) 0x7fb5740ba290 con 0x7fb57405c990 2022-07-20T04:18:28.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fb5740b0c20 con 0x7fb57405c990 2022-07-20T04:18:28.931 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb578acf700 1 -- 172.21.15.61:0/24496670 >> v1:172.21.15.117:6789/0 conn(0x7fb57405c990 legacy=0x7fb574146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb578acf700 1 -- 172.21.15.61:0/24496670 >> v1:172.21.15.61:6789/0 conn(0x7fb5740b9eb0 legacy=0x7fb57405bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.932 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb578acf700 1 -- 172.21.15.61:0/24496670 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5740afed0 con 0x7fb5740b5320 2022-07-20T04:18:28.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb578acf700 1 --2- 172.21.15.61:0/24496670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b5320 0x7fb57405c4a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fb564002f70 tx=0x7fb564006df0).ready entity=mon.2 client_cookie=62ea8fa19f4ed6e4 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.933 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb56400d5c0 con 0x7fb5740b5320 2022-07-20T04:18:28.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb564000e90 con 0x7fb5740b5320 2022-07-20T04:18:28.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb573fff700 1 -- 172.21.15.61:0/24496670 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb56400d910 con 0x7fb5740b5320 2022-07-20T04:18:28.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 -- 172.21.15.61:0/24496670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b5320 msgr2=0x7fb57405c4a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 --2- 172.21.15.61:0/24496670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b5320 0x7fb57405c4a0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fb564002f70 tx=0x7fb564006df0).stop 2022-07-20T04:18:28.937 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 -- 172.21.15.61:0/24496670 shutdown_connections 2022-07-20T04:18:28.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 --2- 172.21.15.61:0/24496670 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b5320 0x7fb57405c4a0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 -- 172.21.15.61:0/24496670 >> 172.21.15.61:0/24496670 conn(0x7fb574067980 msgr2=0x7fb5740b9460 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 -- 172.21.15.61:0/24496670 shutdown_connections 2022-07-20T04:18:28.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.924+0000 7fb57b534700 1 -- 172.21.15.61:0/24496670 wait complete. 2022-07-20T04:18:28.945 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 Processor -- start 2022-07-20T04:18:28.946 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 -- start start 2022-07-20T04:18:28.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 0x7fb5741463d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.947 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 0x7fb5740ae9e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b9eb0 0x7fb5741406a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574145c90 con 0x7fb5740b5320 2022-07-20T04:18:28.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb574145f30 con 0x7fb57405c990 2022-07-20T04:18:28.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fb574144c60 con 0x7fb5740b9eb0 2022-07-20T04:18:28.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb578acf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 0x7fb5740ae9e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 0x7fb5741463d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb578acf700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 0x7fb5740ae9e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52364/0 (socket says 172.21.15.61:52364) 2022-07-20T04:18:28.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 0x7fb5741463d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.117:3300/0 says I am v2:172.21.15.61:33810/0 (socket says 172.21.15.61:33810) 2022-07-20T04:18:28.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb578acf700 1 -- 172.21.15.61:0/3162062480 learned_addr learned my addr 172.21.15.61:0/3162062480 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:28.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb579ad1700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b9eb0 0x7fb5741406a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 -- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b9eb0 msgr2=0x7fb5741406a0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b9eb0 0x7fb5741406a0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 -- 172.21.15.61:0/3162062480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 msgr2=0x7fb5740ae9e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:28.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 0x7fb5740ae9e0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:28.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 -- 172.21.15.61:0/3162062480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb56400b180 con 0x7fb57405c990 2022-07-20T04:18:28.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb578acf700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 0x7fb5740ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:28.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5792d0700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 0x7fb5741463d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fb55c001490 tx=0x7fb55c007dc0).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb55c005ba0 con 0x7fb57405c990 2022-07-20T04:18:28.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5741468c0 con 0x7fb57405c990 2022-07-20T04:18:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.928+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb55c005d00 con 0x7fb57405c990 2022-07-20T04:18:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fb55c0072c0 con 0x7fb57405c990 2022-07-20T04:18:28.956 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5740aeed0 con 0x7fb57405c990 2022-07-20T04:18:28.958 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb574145ae0 con 0x7fb57405c990 2022-07-20T04:18:28.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fb55c010070 con 0x7fb57405c990 2022-07-20T04:18:28.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb5727fc700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb558053af0 0x7fb558055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:28.959 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fb55c06b470 con 0x7fb57405c990 2022-07-20T04:18:28.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb578acf700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb558053af0 0x7fb558055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:28.960 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.932+0000 7fb578acf700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb558053af0 0x7fb558055fb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fb564018000 tx=0x7fb564014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:28.961 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:28.944+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fb55c042610 con 0x7fb57405c990 2022-07-20T04:18:29.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.072+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "4", "target": ["mon-mgr", ""]}) v1 -- 0x7fb5740af180 con 0x7fb558053af0 2022-07-20T04:18:29.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.072+0000 7fb5727fc700 1 -- 172.21.15.61:0/3162062480 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 4 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7fb5740af180 con 0x7fb558053af0 2022-07-20T04:18:29.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb558053af0 msgr2=0x7fb558055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb558053af0 0x7fb558055fb0 secure :-1 s=READY pgs=816 cs=0 l=1 rev1=1 rx=0x7fb564018000 tx=0x7fb564014000).stop 2022-07-20T04:18:29.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 msgr2=0x7fb5741463d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 0x7fb5741463d0 secure :-1 s=READY pgs=693 cs=0 l=1 rev1=1 rx=0x7fb55c001490 tx=0x7fb55c007dc0).stop 2022-07-20T04:18:29.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 shutdown_connections 2022-07-20T04:18:29.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fb558053af0 0x7fb558055fb0 unknown :-1 s=CLOSED pgs=816 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fb5740b9eb0 0x7fb5741406a0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fb5740b5320 0x7fb5740ae9e0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 --2- 172.21.15.61:0/3162062480 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fb57405c990 0x7fb5741463d0 unknown :-1 s=CLOSED pgs=693 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 >> 172.21.15.61:0/3162062480 conn(0x7fb574067980 msgr2=0x7fb5740b2920 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 shutdown_connections 2022-07-20T04:18:29.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.076+0000 7fb57b534700 1 -- 172.21.15.61:0/3162062480 wait complete. 2022-07-20T04:18:29.082 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 4 to deep-scrub 2022-07-20T04:18:29.091 INFO:tasks.ceph:Scrubbing osd.5 2022-07-20T04:18:29.092 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.5 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:29.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.204+0000 7f6cd7fff700 1 Processor -- start 2022-07-20T04:18:29.212 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.204+0000 7f6cd7fff700 1 -- start start 2022-07-20T04:18:29.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b4bc0 0x7f6cd814d610 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.213 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd8066b10 con 0x7f6cd80b6840 2022-07-20T04:18:29.214 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f6cd80b6c20 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.215 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80b5c90 con 0x7f6cd80b4bc0 2022-07-20T04:18:29.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd6ffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f6cd80b6840 0x7f6cd80b5580 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:48970/0 (socket says 172.21.15.61:48970) 2022-07-20T04:18:29.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd6ffd700 1 -- 172.21.15.61:0/3604582586 learned_addr learned my addr 172.21.15.61:0/3604582586 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:29.216 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd77fe700 1 --2- 172.21.15.61:0/3604582586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b4bc0 0x7f6cd814d610 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 1736022560 0 0) 0x7f6cd8066b10 con 0x7f6cd80b6840 2022-07-20T04:18:29.217 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cbc003260 con 0x7f6cd80b6840 2022-07-20T04:18:29.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3033648590 0 0) 0x7f6cd80b6c20 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f6cd8066b10 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.218 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd77fe700 1 -- 172.21.15.61:0/3604582586 >> v1:172.21.15.117:6789/0 conn(0x7f6cd80b3cf0 legacy=0x7f6cd8146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd77fe700 1 -- 172.21.15.61:0/3604582586 >> v1:172.21.15.61:6789/0 conn(0x7f6cd80b6840 legacy=0x7f6cd80b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 <== mon.0 v1:172.21.15.61:6789/0 2 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 214+0+0 (unknown 4189349004 0 0) 0x7f6cbc003260 con 0x7f6cd80b6840 2022-07-20T04:18:29.219 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd77fe700 1 -- 172.21.15.61:0/3604582586 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd80aead0 con 0x7f6cd80b4bc0 2022-07-20T04:18:29.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd77fe700 1 --2- 172.21.15.61:0/3604582586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b4bc0 0x7f6cd814d610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f6ccc001110 tx=0x7f6cd00091d0).ready entity=mon.2 client_cookie=569ca200302e7516 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc001660 con 0x7f6cd80b4bc0 2022-07-20T04:18:29.220 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6cd000f510 con 0x7f6cd80b4bc0 2022-07-20T04:18:29.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd5ffb700 1 -- 172.21.15.61:0/3604582586 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6cd000f680 con 0x7f6cd80b4bc0 2022-07-20T04:18:29.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- 172.21.15.61:0/3604582586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b4bc0 msgr2=0x7f6cd814d610 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 --2- 172.21.15.61:0/3604582586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b4bc0 0x7f6cd814d610 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f6ccc001110 tx=0x7f6cd00091d0).stop 2022-07-20T04:18:29.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- 172.21.15.61:0/3604582586 shutdown_connections 2022-07-20T04:18:29.222 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 --2- 172.21.15.61:0/3604582586 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b4bc0 0x7f6cd814d610 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- 172.21.15.61:0/3604582586 >> 172.21.15.61:0/3604582586 conn(0x7f6cd80b9e10 msgr2=0x7f6cd80b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- 172.21.15.61:0/3604582586 shutdown_connections 2022-07-20T04:18:29.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.208+0000 7f6cd7fff700 1 -- 172.21.15.61:0/3604582586 wait complete. 2022-07-20T04:18:29.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 Processor -- start 2022-07-20T04:18:29.225 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 -- start start 2022-07-20T04:18:29.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd80b3cf0 0x7f6cd8140590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd80b4bc0 0x7f6cd80b1870 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b6840 0x7f6cd80b0e40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd813f6e0 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd80b09e0 con 0x7f6cd80b4bc0 2022-07-20T04:18:29.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd7fff700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f6cd813f440 con 0x7f6cd80b6840 2022-07-20T04:18:29.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd80b3cf0 0x7f6cd8140590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd80b3cf0 0x7f6cd8140590 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52376/0 (socket says 172.21.15.61:52376) 2022-07-20T04:18:29.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 -- 172.21.15.61:0/4212240915 learned_addr learned my addr 172.21.15.61:0/4212240915 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:29.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd67fc700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd80b4bc0 0x7f6cd80b1870 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd77fe700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b6840 0x7f6cd80b0e40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 -- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b6840 msgr2=0x7f6cd80b0e40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f6cd80b6840 0x7f6cd80b0e40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 -- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd80b4bc0 msgr2=0x7f6cd80b1870 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f6cd80b4bc0 0x7f6cd80b1870 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 -- 172.21.15.61:0/4212240915 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6cd0009470 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cd6ffd700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd80b3cf0 0x7f6cd8140590 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f6ccc001840 tx=0x7f6ccc001a40).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.212+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc007170 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f6ccc0072d0 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cd8140a80 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f6ccc004920 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cd8149e30 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7f6cd80b0860 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f6ccc01b070 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb4053a40 0x7f6cb4055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f6ccc06b360 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f6cb405a130 0x7f6cb405c590 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 --> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7f6cb405cb70 con 0x7f6cb405a130 2022-07-20T04:18:29.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7f6ccc06b620 con 0x7f6cd80b3cf0 2022-07-20T04:18:29.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cd67fc700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb4053a40 0x7f6cb4055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cd77fe700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f6cb405a130 0x7f6cb405c590 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.216+0000 7f6cd67fc700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb4053a40 0x7f6cb4055f00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f6cc800d000 tx=0x7f6cc8007a20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.220+0000 7f6cd77fe700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f6cb405a130 0x7f6cb405c590 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.5 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.220+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== osd.5 v2:172.21.15.117:6818/23294 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7f6cb405cb70 con 0x7f6cb405a130 2022-07-20T04:18:29.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 --> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7f6cd80b0ba0 con 0x7f6cb405a130 2022-07-20T04:18:29.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cc7fff700 1 -- 172.21.15.61:0/4212240915 <== osd.5 v2:172.21.15.117:6818/23294 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7f6cd000fab0 con 0x7f6cb405a130 2022-07-20T04:18:29.249 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:29.249 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:29.250 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:29.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f6cb405a130 msgr2=0x7f6cb405c590 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6818/23294,v1:172.21.15.117:6819/23294] conn(0x7f6cb405a130 0x7f6cb405c590 crc :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb4053a40 msgr2=0x7f6cb4055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f6cb4053a40 0x7f6cb4055f00 secure :-1 s=READY pgs=817 cs=0 l=1 rev1=1 rx=0x7f6cc800d000 tx=0x7f6cc8007a20).stop 2022-07-20T04:18:29.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd80b3cf0 msgr2=0x7f6cd8140590 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.253 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 --2- 172.21.15.61:0/4212240915 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f6cd80b3cf0 0x7f6cd8140590 secure :-1 s=READY pgs=1564 cs=0 l=1 rev1=1 rx=0x7f6ccc001840 tx=0x7f6ccc001a40).stop 2022-07-20T04:18:29.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd77fe700 1 -- 172.21.15.61:0/4212240915 reap_dead start 2022-07-20T04:18:29.254 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.236+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 shutdown_connections 2022-07-20T04:18:29.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.240+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 >> 172.21.15.61:0/4212240915 conn(0x7f6cd80b9e10 msgr2=0x7f6cd81480c0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.240+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 shutdown_connections 2022-07-20T04:18:29.256 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.240+0000 7f6cd7fff700 1 -- 172.21.15.61:0/4212240915 wait complete. 2022-07-20T04:18:29.257 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 5 2022-07-20T04:18:29.388 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 Processor -- start 2022-07-20T04:18:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- start start 2022-07-20T04:18:29.389 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0a780b6800 0x7f0a780aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780b3cb0 0x7f0a78145f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f0a780ad0e0 0x7f0a78146430 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.390 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a78146920 con 0x7f0a780b6800 2022-07-20T04:18:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f0a780ae280 con 0x7f0a780ad0e0 2022-07-20T04:18:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a780b6be0 con 0x7f0a780b3cb0 2022-07-20T04:18:29.391 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0a780b6800 0x7f0a780aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f0a780b6800 0x7f0a780aebf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52386/0 (socket says 172.21.15.61:52386) 2022-07-20T04:18:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 -- 172.21.15.61:0/1538873795 learned_addr learned my addr 172.21.15.61:0/1538873795 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:29.392 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a75d9b700 1 --2- 172.21.15.61:0/1538873795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780b3cb0 0x7f0a78145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a76d9d700 1 --2- 172.21.15.61:0/1538873795 >> v2:172.21.15.117:3300/0 conn(0x7f0a780ad0e0 0x7f0a78146430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.393 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 -- 172.21.15.61:0/1538873795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780b3cb0 msgr2=0x7f0a78145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:29.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 --2- 172.21.15.61:0/1538873795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780b3cb0 0x7f0a78145f40 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.394 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 -- 172.21.15.61:0/1538873795 >> v2:172.21.15.117:3300/0 conn(0x7f0a780ad0e0 msgr2=0x7f0a78146430 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:29.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 --2- 172.21.15.61:0/1538873795 >> v2:172.21.15.117:3300/0 conn(0x7f0a780ad0e0 0x7f0a78146430 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 -- 172.21.15.61:0/1538873795 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a780ae4b0 con 0x7f0a780b6800 2022-07-20T04:18:29.395 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7659c700 1 --2- 172.21.15.61:0/1538873795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780aebf0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f0a780b58c0 tx=0x7f0a60006eb0).ready entity=mon.0 client_cookie=dd027d0028b2d6d6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7559a700 1 -- 172.21.15.61:0/1538873795 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a6000e610 con 0x7f0a780b6800 2022-07-20T04:18:29.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7559a700 1 -- 172.21.15.61:0/1538873795 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a60000e90 con 0x7f0a780b6800 2022-07-20T04:18:29.396 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7559a700 1 -- 172.21.15.61:0/1538873795 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a6000e9a0 con 0x7f0a780b6800 2022-07-20T04:18:29.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- 172.21.15.61:0/1538873795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 msgr2=0x7f0a780aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- 172.21.15.61:0/1538873795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780aebf0 secure :-1 s=READY pgs=1565 cs=0 l=1 rev1=1 rx=0x7f0a780b58c0 tx=0x7f0a60006eb0).stop 2022-07-20T04:18:29.397 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- 172.21.15.61:0/1538873795 shutdown_connections 2022-07-20T04:18:29.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- 172.21.15.61:0/1538873795 >> v2:172.21.15.117:3300/0 conn(0x7f0a780ad0e0 0x7f0a78146430 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.398 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- 172.21.15.61:0/1538873795 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780b3cb0 0x7f0a78145f40 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 --2- 172.21.15.61:0/1538873795 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780aebf0 unknown :-1 s=CLOSED pgs=1565 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- 172.21.15.61:0/1538873795 >> 172.21.15.61:0/1538873795 conn(0x7f0a780b9dd0 msgr2=0x7f0a780b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.399 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.372+0000 7f0a7759e700 1 -- 172.21.15.61:0/1538873795 shutdown_connections 2022-07-20T04:18:29.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 -- 172.21.15.61:0/1538873795 wait complete. 2022-07-20T04:18:29.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 Processor -- start 2022-07-20T04:18:29.400 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 -- start start 2022-07-20T04:18:29.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780ad0e0 0x7f0a78115260 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a780b3cb0 0x7f0a78145670 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.401 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780b2200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78114b20 con 0x7f0a780b6800 2022-07-20T04:18:29.402 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78114dc0 con 0x7f0a780b3cb0 2022-07-20T04:18:29.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7759e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f0a78115750 con 0x7f0a780ad0e0 2022-07-20T04:18:29.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a76d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780b2200 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.403 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a7659c700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780ad0e0 0x7f0a78115260 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a76d9d700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780b2200 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52396/0 (socket says 172.21.15.61:52396) 2022-07-20T04:18:29.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a76d9d700 1 -- 172.21.15.61:0/857342601 learned_addr learned my addr 172.21.15.61:0/857342601 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:29.404 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.376+0000 7f0a75d9b700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a780b3cb0 0x7f0a78145670 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a76d9d700 1 -- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780ad0e0 msgr2=0x7f0a78115260 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a76d9d700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780ad0e0 0x7f0a78115260 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.405 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a76d9d700 1 -- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a780b3cb0 msgr2=0x7f0a78145670 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a76d9d700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a780b3cb0 0x7f0a78145670 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.406 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a76d9d700 1 -- 172.21.15.61:0/857342601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6000b1e0 con 0x7f0a780b6800 2022-07-20T04:18:29.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a7659c700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780ad0e0 0x7f0a78115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:29.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a76d9d700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780b2200 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f0a7000c000 tx=0x7f0a70006df0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.407 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a70011750 con 0x7f0a780b6800 2022-07-20T04:18:29.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0a700118b0 con 0x7f0a780b6800 2022-07-20T04:18:29.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a78145b60 con 0x7f0a780b6800 2022-07-20T04:18:29.408 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f0a70011c10 con 0x7f0a780b6800 2022-07-20T04:18:29.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a780b1e70 con 0x7f0a780b6800 2022-07-20T04:18:29.409 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a78114fc0 con 0x7f0a780b6800 2022-07-20T04:18:29.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f0a70011d70 con 0x7f0a780b6800 2022-07-20T04:18:29.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a6f7fe700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a5c053a40 0x7f0a5c055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.410 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f0a7006bf20 con 0x7f0a780b6800 2022-07-20T04:18:29.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.380+0000 7f0a7659c700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a5c053a40 0x7f0a5c055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.411 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.384+0000 7f0a7659c700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a5c053a40 0x7f0a5c055f00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f0a6001b000 tx=0x7f0a60015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.412 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.388+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f0a78114fc0 con 0x7f0a780b6800 2022-07-20T04:18:29.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.512+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "5", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a781149a0 con 0x7f0a5c053a40 2022-07-20T04:18:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a6f7fe700 1 -- 172.21.15.61:0/857342601 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 5 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f0a600069d0 con 0x7f0a5c053a40 2022-07-20T04:18:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a5c053a40 msgr2=0x7f0a5c055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a5c053a40 0x7f0a5c055f00 secure :-1 s=READY pgs=818 cs=0 l=1 rev1=1 rx=0x7f0a6001b000 tx=0x7f0a60015000).stop 2022-07-20T04:18:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 msgr2=0x7f0a780b2200 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780b2200 secure :-1 s=READY pgs=1566 cs=0 l=1 rev1=1 rx=0x7f0a7000c000 tx=0x7f0a70006df0).stop 2022-07-20T04:18:29.522 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 shutdown_connections 2022-07-20T04:18:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0a5c053a40 0x7f0a5c055f00 unknown :-1 s=CLOSED pgs=818 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f0a780b3cb0 0x7f0a78145670 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.523 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f0a780b6800 0x7f0a780b2200 unknown :-1 s=CLOSED pgs=1566 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 --2- 172.21.15.61:0/857342601 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f0a780ad0e0 0x7f0a78115260 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 >> 172.21.15.61:0/857342601 conn(0x7f0a780b9dd0 msgr2=0x7f0a780b5d40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.524 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 shutdown_connections 2022-07-20T04:18:29.525 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.516+0000 7f0a7759e700 1 -- 172.21.15.61:0/857342601 wait complete. 2022-07-20T04:18:29.526 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 5 to deep-scrub 2022-07-20T04:18:29.529 INFO:tasks.ceph:Scrubbing osd.6 2022-07-20T04:18:29.530 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.6 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:29.645 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.644+0000 7fec7e099700 1 Processor -- start 2022-07-20T04:18:29.646 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.644+0000 7fec7e099700 1 -- start start 2022-07-20T04:18:29.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7fec780b6840 0x7fec780b5580 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.650 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec780b4bc0 0x7fec78141a90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fec78066b10 con 0x7fec780b3cf0 2022-07-20T04:18:29.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec780b6c20 con 0x7fec780b4bc0 2022-07-20T04:18:29.651 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fec780aead0 con 0x7fec780b6840 2022-07-20T04:18:29.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec780b4bc0 0x7fec78141a90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.652 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fec780b4bc0 0x7fec78141a90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52404/0 (socket says 172.21.15.61:52404) 2022-07-20T04:18:29.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 -- 172.21.15.61:0/3308783111 learned_addr learned my addr 172.21.15.61:0/3308783111 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:29.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d097700 1 --2- 172.21.15.61:0/3308783111 >> v2:172.21.15.117:3300/0 conn(0x7fec780b6840 0x7fec780b5580 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.653 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 -- 172.21.15.61:0/3308783111 >> v2:172.21.15.117:3300/0 conn(0x7fec780b6840 msgr2=0x7fec780b5580 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:29.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 --2- 172.21.15.61:0/3308783111 >> v2:172.21.15.117:3300/0 conn(0x7fec780b6840 0x7fec780b5580 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 -- 172.21.15.61:0/3308783111 >> v1:172.21.15.117:6789/0 conn(0x7fec780b3cf0 legacy=0x7fec78146040 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.654 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 -- 172.21.15.61:0/3308783111 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec780aed70 con 0x7fec780b4bc0 2022-07-20T04:18:29.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7d898700 1 --2- 172.21.15.61:0/3308783111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b4bc0 0x7fec78141a90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7fec700032f0 tx=0x7fec70009290).ready entity=mon.0 client_cookie=fe46bf24b44706a8 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.655 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec67fff700 1 -- 172.21.15.61:0/3308783111 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec7000b870 con 0x7fec780b4bc0 2022-07-20T04:18:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec67fff700 1 -- 172.21.15.61:0/3308783111 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec7000b9d0 con 0x7fec780b4bc0 2022-07-20T04:18:29.656 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec67fff700 1 -- 172.21.15.61:0/3308783111 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec7000bb40 con 0x7fec780b4bc0 2022-07-20T04:18:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- 172.21.15.61:0/3308783111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b4bc0 msgr2=0x7fec78141a90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 --2- 172.21.15.61:0/3308783111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b4bc0 0x7fec78141a90 secure :-1 s=READY pgs=1567 cs=0 l=1 rev1=1 rx=0x7fec700032f0 tx=0x7fec70009290).stop 2022-07-20T04:18:29.657 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- 172.21.15.61:0/3308783111 shutdown_connections 2022-07-20T04:18:29.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 --2- 172.21.15.61:0/3308783111 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b4bc0 0x7fec78141a90 unknown :-1 s=CLOSED pgs=1567 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.658 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 --2- 172.21.15.61:0/3308783111 >> v2:172.21.15.117:3300/0 conn(0x7fec780b6840 0x7fec780b5580 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- 172.21.15.61:0/3308783111 >> 172.21.15.61:0/3308783111 conn(0x7fec780b9e10 msgr2=0x7fec780b5df0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- 172.21.15.61:0/3308783111 shutdown_connections 2022-07-20T04:18:29.659 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.648+0000 7fec7e099700 1 -- 172.21.15.61:0/3308783111 wait complete. 2022-07-20T04:18:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 Processor -- start 2022-07-20T04:18:29.664 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 -- start start 2022-07-20T04:18:29.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b3cf0 0x7fec78152a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.665 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec780b4bc0 0x7fec78112d30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec780b6840 0x7fec780aef70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.666 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b1ab0 con 0x7fec780b3cf0 2022-07-20T04:18:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b2e70 con 0x7fec780b4bc0 2022-07-20T04:18:29.667 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fec780b2b60 con 0x7fec780b6840 2022-07-20T04:18:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b3cf0 0x7fec78152a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.668 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b3cf0 0x7fec78152a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52406/0 (socket says 172.21.15.61:52406) 2022-07-20T04:18:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 -- 172.21.15.61:0/3518046594 learned_addr learned my addr 172.21.15.61:0/3518046594 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d898700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec780b6840 0x7fec780aef70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.669 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7c896700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec780b4bc0 0x7fec78112d30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 -- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec780b6840 msgr2=0x7fec780aef70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fec780b6840 0x7fec780aef70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.670 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 -- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec780b4bc0 msgr2=0x7fec78112d30 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fec780b4bc0 0x7fec78112d30 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 -- 172.21.15.61:0/3518046594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fec70009530 con 0x7fec780b3cf0 2022-07-20T04:18:29.671 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7d097700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b3cf0 0x7fec78152a30 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7fec740030b0 tx=0x7fec74008040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec74011780 con 0x7fec780b3cf0 2022-07-20T04:18:29.672 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fec740118e0 con 0x7fec780b3cf0 2022-07-20T04:18:29.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fec78113220 con 0x7fec780b3cf0 2022-07-20T04:18:29.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fec74011c40 con 0x7fec780b3cf0 2022-07-20T04:18:29.673 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.652+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fec781126e0 con 0x7fec780b3cf0 2022-07-20T04:18:29.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fec74012420 con 0x7fec780b3cf0 2022-07-20T04:18:29.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fec780b1c70 con 0x7fec780b3cf0 2022-07-20T04:18:29.674 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec667fc700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec580539b0 0x7fec58055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fec7406bb00 con 0x7fec780b3cf0 2022-07-20T04:18:29.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec7c896700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec580539b0 0x7fec58055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.675 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec667fc700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fec5805a0a0 0x7fec5805c500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 --> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fec5805cb80 con 0x7fec5805a0a0 2022-07-20T04:18:29.676 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7fec74070240 con 0x7fec780b3cf0 2022-07-20T04:18:29.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec7d898700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fec5805a0a0 0x7fec5805c500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.677 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec7c896700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec580539b0 0x7fec58055e70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fec6800b000 tx=0x7fec68009040).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.656+0000 7fec7d898700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fec5805a0a0 0x7fec5805c500 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.6 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.678 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.660+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== osd.6 v2:172.21.15.117:6810/23293 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fec5805cb80 con 0x7fec5805a0a0 2022-07-20T04:18:29.679 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 --> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fec780b50e0 con 0x7fec5805a0a0 2022-07-20T04:18:29.680 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec667fc700 1 -- 172.21.15.61:0/3518046594 <== osd.6 v2:172.21.15.117:6810/23293 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fec780b50e0 con 0x7fec5805a0a0 2022-07-20T04:18:29.680 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:29.681 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:29.681 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:29.682 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fec5805a0a0 msgr2=0x7fec5805c500 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6810/23293,v1:172.21.15.117:6811/23293] conn(0x7fec5805a0a0 0x7fec5805c500 crc :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.683 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec580539b0 msgr2=0x7fec58055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fec580539b0 0x7fec58055e70 secure :-1 s=READY pgs=819 cs=0 l=1 rev1=1 rx=0x7fec6800b000 tx=0x7fec68009040).stop 2022-07-20T04:18:29.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b3cf0 msgr2=0x7fec78152a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.684 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 --2- 172.21.15.61:0/3518046594 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fec780b3cf0 0x7fec78152a30 secure :-1 s=READY pgs=1568 cs=0 l=1 rev1=1 rx=0x7fec740030b0 tx=0x7fec74008040).stop 2022-07-20T04:18:29.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7d898700 1 -- 172.21.15.61:0/3518046594 reap_dead start 2022-07-20T04:18:29.685 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 shutdown_connections 2022-07-20T04:18:29.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.676+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 >> 172.21.15.61:0/3518046594 conn(0x7fec780b9e10 msgr2=0x7fec7805bd90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.680+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 shutdown_connections 2022-07-20T04:18:29.686 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.680+0000 7fec7e099700 1 -- 172.21.15.61:0/3518046594 wait complete. 2022-07-20T04:18:29.690 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 6 2022-07-20T04:18:29.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.792+0000 7f5734a1b700 1 Processor -- start 2022-07-20T04:18:29.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.792+0000 7f5734a1b700 1 -- start start 2022-07-20T04:18:29.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f57300b6800 0x7f57300aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300b3cb0 0x7f5730145f40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f57300ae280 con 0x7f57300ad0e0 2022-07-20T04:18:29.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f57300b6be0 con 0x7f57300b6800 2022-07-20T04:18:29.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57300ae4b0 con 0x7f57300b3cb0 2022-07-20T04:18:29.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572ffff700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f57300ad0e0 0x7f573005bd90 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49018/0 (socket says 172.21.15.61:49018) 2022-07-20T04:18:29.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572ffff700 1 -- 172.21.15.61:0/3685550965 learned_addr learned my addr 172.21.15.61:0/3685550965 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:29.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572f7fe700 1 --2- 172.21.15.61:0/3685550965 >> v2:172.21.15.117:3300/0 conn(0x7f57300b6800 0x7f57300aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.806 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572effd700 1 --2- 172.21.15.61:0/3685550965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300b3cb0 0x7f5730145f40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572e7fc700 1 -- 172.21.15.61:0/3685550965 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 2941203794 0 0) 0x7f57300ae280 con 0x7f57300ad0e0 2022-07-20T04:18:29.807 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572e7fc700 1 -- 172.21.15.61:0/3685550965 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f5710003260 con 0x7f57300ad0e0 2022-07-20T04:18:29.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572effd700 1 -- 172.21.15.61:0/3685550965 >> v2:172.21.15.117:3300/0 conn(0x7f57300b6800 msgr2=0x7f57300aebf0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:29.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572effd700 1 --2- 172.21.15.61:0/3685550965 >> v2:172.21.15.117:3300/0 conn(0x7f57300b6800 0x7f57300aebf0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.808 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572effd700 1 -- 172.21.15.61:0/3685550965 >> v1:172.21.15.61:6789/0 conn(0x7f57300ad0e0 legacy=0x7f573005bd90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572effd700 1 -- 172.21.15.61:0/3685550965 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57300ae750 con 0x7f57300b3cb0 2022-07-20T04:18:29.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572effd700 1 --2- 172.21.15.61:0/3685550965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300b3cb0 0x7f5730145f40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f5718002f70 tx=0x7f5718006df0).ready entity=mon.2 client_cookie=710728e5d60140ab server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572e7fc700 1 -- 172.21.15.61:0/3685550965 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571800d4a0 con 0x7f57300b3cb0 2022-07-20T04:18:29.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572e7fc700 1 -- 172.21.15.61:0/3685550965 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5718000e90 con 0x7f57300b3cb0 2022-07-20T04:18:29.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f572e7fc700 1 -- 172.21.15.61:0/3685550965 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f571800d7f0 con 0x7f57300b3cb0 2022-07-20T04:18:29.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- 172.21.15.61:0/3685550965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300b3cb0 msgr2=0x7f5730145f40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.811 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 --2- 172.21.15.61:0/3685550965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300b3cb0 0x7f5730145f40 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7f5718002f70 tx=0x7f5718006df0).stop 2022-07-20T04:18:29.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- 172.21.15.61:0/3685550965 shutdown_connections 2022-07-20T04:18:29.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 --2- 172.21.15.61:0/3685550965 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300b3cb0 0x7f5730145f40 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 --2- 172.21.15.61:0/3685550965 >> v2:172.21.15.117:3300/0 conn(0x7f57300b6800 0x7f57300aebf0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- 172.21.15.61:0/3685550965 >> 172.21.15.61:0/3685550965 conn(0x7f57300b9dd0 msgr2=0x7f57300b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- 172.21.15.61:0/3685550965 shutdown_connections 2022-07-20T04:18:29.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.796+0000 7f5734a1b700 1 -- 172.21.15.61:0/3685550965 wait complete. 2022-07-20T04:18:29.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 Processor -- start 2022-07-20T04:18:29.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 -- start start 2022-07-20T04:18:29.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300ad0e0 0x7f57300af790 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57300b3cb0 0x7f5730147020 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 0x7f57301450f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57300b1e10 con 0x7f57300b6800 2022-07-20T04:18:29.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57300afc80 con 0x7f57300b3cb0 2022-07-20T04:18:29.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f5734a1b700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f57300b1b70 con 0x7f57300ad0e0 2022-07-20T04:18:29.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f572ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 0x7f57301450f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f572ffff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 0x7f57301450f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52426/0 (socket says 172.21.15.61:52426) 2022-07-20T04:18:29.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f572ffff700 1 -- 172.21.15.61:0/709094924 learned_addr learned my addr 172.21.15.61:0/709094924 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:29.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f572f7fe700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300ad0e0 0x7f57300af790 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.800+0000 7f572effd700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57300b3cb0 0x7f5730147020 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572ffff700 1 -- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300ad0e0 msgr2=0x7f57300af790 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572ffff700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300ad0e0 0x7f57300af790 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572ffff700 1 -- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57300b3cb0 msgr2=0x7f5730147020 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572ffff700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57300b3cb0 0x7f5730147020 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572ffff700 1 -- 172.21.15.61:0/709094924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f571800b180 con 0x7f57300b6800 2022-07-20T04:18:29.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572ffff700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 0x7f57301450f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f5728001490 tx=0x7f5728007dc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f5728005c70 con 0x7f57300b6800 2022-07-20T04:18:29.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f5728005dd0 con 0x7f57300b6800 2022-07-20T04:18:29.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57300af2f0 con 0x7f57300b6800 2022-07-20T04:18:29.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f57280072c0 con 0x7f57300b6800 2022-07-20T04:18:29.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57301461b0 con 0x7f57300b6800 2022-07-20T04:18:29.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57300b19c0 con 0x7f57300b6800 2022-07-20T04:18:29.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f572801a060 con 0x7f57300b6800 2022-07-20T04:18:29.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572cff9700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5714053a40 0x7f5714055f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:29.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.804+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f572806b600 con 0x7f57300b6800 2022-07-20T04:18:29.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.808+0000 7f572f7fe700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5714053a40 0x7f5714055f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:29.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.808+0000 7f572f7fe700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5714053a40 0x7f5714055f00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f572000b000 tx=0x7f5720006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:29.827 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.808+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f5728042840 con 0x7f57300b6800 2022-07-20T04:18:29.935 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "6", "target": ["mon-mgr", ""]}) v1 -- 0x7f57300af480 con 0x7f5714053a40 2022-07-20T04:18:29.936 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f572cff9700 1 -- 172.21.15.61:0/709094924 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 6 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f57300af480 con 0x7f5714053a40 2022-07-20T04:18:29.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5714053a40 msgr2=0x7f5714055f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.938 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5714053a40 0x7f5714055f00 secure :-1 s=READY pgs=820 cs=0 l=1 rev1=1 rx=0x7f572000b000 tx=0x7f5720006df0).stop 2022-07-20T04:18:29.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 msgr2=0x7f57301450f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:29.939 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 0x7f57301450f0 secure :-1 s=READY pgs=1570 cs=0 l=1 rev1=1 rx=0x7f5728001490 tx=0x7f5728007dc0).stop 2022-07-20T04:18:29.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 shutdown_connections 2022-07-20T04:18:29.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f5714053a40 0x7f5714055f00 unknown :-1 s=CLOSED pgs=820 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.940 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f57300b3cb0 0x7f5730147020 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f57300b6800 0x7f57301450f0 unknown :-1 s=CLOSED pgs=1570 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.942 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 --2- 172.21.15.61:0/709094924 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f57300ad0e0 0x7f57300af790 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:29.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 >> 172.21.15.61:0/709094924 conn(0x7f57300b9dd0 msgr2=0x7f5730142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:29.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 shutdown_connections 2022-07-20T04:18:29.943 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:29.936+0000 7f5734a1b700 1 -- 172.21.15.61:0/709094924 wait complete. 2022-07-20T04:18:29.944 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 6 to deep-scrub 2022-07-20T04:18:29.955 INFO:tasks.ceph:Scrubbing osd.7 2022-07-20T04:18:29.955 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph tell osd.7 config set osd_debug_deep_scrub_sleep 0 2022-07-20T04:18:30.066 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.064+0000 7fe0f0a6e700 1 Processor -- start 2022-07-20T04:18:30.067 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.064+0000 7fe0f0a6e700 1 -- start start 2022-07-20T04:18:30.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0ec0b5990 0x7fe0ec146070 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec146c70 con 0x7fe0ec0af3e0 2022-07-20T04:18:30.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fe0ec0b5d70 con 0x7fe0ec05d210 2022-07-20T04:18:30.073 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7fe0ec0ad050 con 0x7fe0ec0b5990 2022-07-20T04:18:30.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eb7fe700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7fe0ec0b5990 0x7fe0ec146070 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.074 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eaffd700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fe0ec0af3e0 0x7fe0ec146560 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49030/0 (socket says 172.21.15.61:49030) 2022-07-20T04:18:30.075 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eaffd700 1 -- 172.21.15.61:0/2689093433 learned_addr learned my addr 172.21.15.61:0/2689093433 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:30.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eb7fe700 1 -- 172.21.15.61:0/2689093433 >> v1:172.21.15.117:6789/0 conn(0x7fe0ec05d210 legacy=0x7fe0ec14a360 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.076 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eb7fe700 1 -- 172.21.15.61:0/2689093433 >> v1:172.21.15.61:6789/0 conn(0x7fe0ec0af3e0 legacy=0x7fe0ec146560 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eb7fe700 1 -- 172.21.15.61:0/2689093433 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0ec0b1010 con 0x7fe0ec0b5990 2022-07-20T04:18:30.077 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0ea7fc700 1 -- 172.21.15.61:0/2689093433 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 383666971 0 0) 0x7fe0ec146c70 con 0x7fe0ec0af3e0 2022-07-20T04:18:30.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0eb7fe700 1 --2- 172.21.15.61:0/2689093433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec146070 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7fe0dc002f70 tx=0x7fe0dc006d90).ready entity=mon.0 client_cookie=ed54307baf28fbc6 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.078 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0ea7fc700 1 -- 172.21.15.61:0/2689093433 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0dc00f6d0 con 0x7fe0ec0b5990 2022-07-20T04:18:30.079 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0ea7fc700 1 -- 172.21.15.61:0/2689093433 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0dc000e90 con 0x7fe0ec0b5990 2022-07-20T04:18:30.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0ea7fc700 1 -- 172.21.15.61:0/2689093433 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0dc00fa20 con 0x7fe0ec0b5990 2022-07-20T04:18:30.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/2689093433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 msgr2=0x7fe0ec146070 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.080 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 --2- 172.21.15.61:0/2689093433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec146070 secure :-1 s=READY pgs=1572 cs=0 l=1 rev1=1 rx=0x7fe0dc002f70 tx=0x7fe0dc006d90).stop 2022-07-20T04:18:30.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/2689093433 shutdown_connections 2022-07-20T04:18:30.081 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 --2- 172.21.15.61:0/2689093433 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec146070 unknown :-1 s=CLOSED pgs=1572 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/2689093433 >> 172.21.15.61:0/2689093433 conn(0x7fe0ec063480 msgr2=0x7fe0ec0b4f40 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:30.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/2689093433 shutdown_connections 2022-07-20T04:18:30.082 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.068+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/2689093433 wait complete. 2022-07-20T04:18:30.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 Processor -- start 2022-07-20T04:18:30.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 -- start start 2022-07-20T04:18:30.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ec05d210 0x7fe0ec14a5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.088 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ec0af3e0 0x7fe0ec0665d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec066f00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec1562d0 con 0x7fe0ec0b5990 2022-07-20T04:18:30.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec063ed0 con 0x7fe0ec05d210 2022-07-20T04:18:30.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.072+0000 7fe0f0a6e700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fe0ec156520 con 0x7fe0ec0af3e0 2022-07-20T04:18:30.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec066f00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec066f00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52440/0 (socket says 172.21.15.61:52440) 2022-07-20T04:18:30.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0eb7fe700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ec05d210 0x7fe0ec14a5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.091 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 -- 172.21.15.61:0/3361404060 learned_addr learned my addr 172.21.15.61:0/3361404060 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:30.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0eaffd700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ec0af3e0 0x7fe0ec0665d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 -- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ec0af3e0 msgr2=0x7fe0ec0665d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fe0ec0af3e0 0x7fe0ec0665d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 -- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ec05d210 msgr2=0x7fe0ec14a5b0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ec05d210 0x7fe0ec14a5b0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 -- 172.21.15.61:0/3361404060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe0dc00b120 con 0x7fe0ec0b5990 2022-07-20T04:18:30.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0eb7fe700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fe0ec05d210 0x7fe0ec14a5b0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:30.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0ebfff700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec066f00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7fe0ec068060 tx=0x7fe0e4007bc0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0e4012400 con 0x7fe0ec0b5990 2022-07-20T04:18:30.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fe0e4012560 con 0x7fe0ec0b5990 2022-07-20T04:18:30.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe0ec066ac0 con 0x7fe0ec0b5990 2022-07-20T04:18:30.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fe0e40128c0 con 0x7fe0ec0b5990 2022-07-20T04:18:30.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe0ec064f50 con 0x7fe0ec0b5990 2022-07-20T04:18:30.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_get_version(what=osdmap handle=1) v1 -- 0x7fe0ec145270 con 0x7fe0ec0b5990 2022-07-20T04:18:30.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.076+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fe0e4019030 con 0x7fe0ec0b5990 2022-07-20T04:18:30.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0e8ff9700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0d0053a60 0x7fe0d0055f20 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fe0e406ae00 con 0x7fe0ec0b5990 2022-07-20T04:18:30.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0e8ff9700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7fe0d005a150 0x7fe0d005c5b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 --> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] -- command(tid 1: {"prefix": "get_command_descriptions"}) v1 -- 0x7fe0d005cb90 con 0x7fe0d005a150 2022-07-20T04:18:30.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_get_version_reply(handle=1 version=152) v2 ==== 24+0+0 (secure 0 0 0) 0x7fe0e406d490 con 0x7fe0ec0b5990 2022-07-20T04:18:30.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0eb7fe700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0d0053a60 0x7fe0d0055f20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0eaffd700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7fe0d005a150 0x7fe0d005c5b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0eb7fe700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0d0053a60 0x7fe0d0055f20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fe0dc01b000 tx=0x7fe0dc015000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0eaffd700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7fe0d005a150 0x7fe0d005c5b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).ready entity=osd.7 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.080+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== osd.7 v2:172.21.15.117:6825/23295 1 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+21404 (crc 0 0 0) 0x7fe0d005cb90 con 0x7fe0d005a150 2022-07-20T04:18:30.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 --> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] -- command(tid 2: {"prefix": "config set", "var": "osd_debug_deep_scrub_sleep", "val": ["0"]}) v1 -- 0x7fe0ec067710 con 0x7fe0d005a150 2022-07-20T04:18:30.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0e8ff9700 1 -- 172.21.15.61:0/3361404060 <== osd.7 v2:172.21.15.117:6825/23295 2 ==== command_reply(tid 2: 0 ) v1 ==== 8+0+105 (crc 0 0 0) 0x7fe0ec067710 con 0x7fe0d005a150 2022-07-20T04:18:30.103 INFO:teuthology.orchestra.run.smithi061.stdout:{ 2022-07-20T04:18:30.103 INFO:teuthology.orchestra.run.smithi061.stdout: "success": "osd_debug_deep_scrub_sleep = '0.000000' (not observed, change may require restart) " 2022-07-20T04:18:30.104 INFO:teuthology.orchestra.run.smithi061.stdout:} 2022-07-20T04:18:30.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7fe0d005a150 msgr2=0x7fe0d005c5b0 crc :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6825/23295,v1:172.21.15.117:6827/23295] conn(0x7fe0d005a150 0x7fe0d005c5b0 crc :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0d0053a60 msgr2=0x7fe0d0055f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fe0d0053a60 0x7fe0d0055f20 secure :-1 s=READY pgs=821 cs=0 l=1 rev1=1 rx=0x7fe0dc01b000 tx=0x7fe0dc015000).stop 2022-07-20T04:18:30.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 msgr2=0x7fe0ec066f00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 --2- 172.21.15.61:0/3361404060 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fe0ec0b5990 0x7fe0ec066f00 secure :-1 s=READY pgs=1573 cs=0 l=1 rev1=1 rx=0x7fe0ec068060 tx=0x7fe0e4007bc0).stop 2022-07-20T04:18:30.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0ebfff700 1 -- 172.21.15.61:0/3361404060 reap_dead start 2022-07-20T04:18:30.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 shutdown_connections 2022-07-20T04:18:30.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 >> 172.21.15.61:0/3361404060 conn(0x7fe0ec063480 msgr2=0x7fe0ec146100 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:30.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 shutdown_connections 2022-07-20T04:18:30.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.096+0000 7fe0f0a6e700 1 -- 172.21.15.61:0/3361404060 wait complete. 2022-07-20T04:18:30.115 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph osd deep-scrub 7 2022-07-20T04:18:30.223 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 Processor -- start 2022-07-20T04:18:30.224 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 -- start start 2022-07-20T04:18:30.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f072c0b3cb0 0x7f072c0aebf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0ad0e0 0x7f072c1418f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.228 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f072c0ae280 con 0x7f072c0b6800 2022-07-20T04:18:30.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f072c0b6be0 con 0x7f072c0b3cb0 2022-07-20T04:18:30.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0731cd1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0ae4b0 con 0x7f072c0ad0e0 2022-07-20T04:18:30.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072bfff700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f072c0b3cb0 0x7f072c0aebf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0730ccf700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7f072c0b6800 0x7f072c145f40 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49044/0 (socket says 172.21.15.61:49044) 2022-07-20T04:18:30.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f0730ccf700 1 -- 172.21.15.61:0/1377880377 learned_addr learned my addr 172.21.15.61:0/1377880377 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:30.230 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f07314d0700 1 --2- 172.21.15.61:0/1377880377 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0ad0e0 0x7f072c1418f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072bfff700 1 -- 172.21.15.61:0/1377880377 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0ad0e0 msgr2=0x7f072c1418f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:30.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072bfff700 1 --2- 172.21.15.61:0/1377880377 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0ad0e0 0x7f072c1418f0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072bfff700 1 -- 172.21.15.61:0/1377880377 >> v1:172.21.15.61:6789/0 conn(0x7f072c0b6800 legacy=0x7f072c145f40 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072b7fe700 1 -- 172.21.15.61:0/1377880377 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 460591269 0 0) 0x7f072c0ae280 con 0x7f072c0b6800 2022-07-20T04:18:30.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072bfff700 1 -- 172.21.15.61:0/1377880377 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f072c0ae750 con 0x7f072c0b3cb0 2022-07-20T04:18:30.232 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072bfff700 1 --2- 172.21.15.61:0/1377880377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b3cb0 0x7f072c0aebf0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f0714002f70 tx=0x7f0714006df0).ready entity=mon.0 client_cookie=6de88143bf5ca0fe server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072b7fe700 1 -- 172.21.15.61:0/1377880377 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f071400d570 con 0x7f072c0b3cb0 2022-07-20T04:18:30.233 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072b7fe700 1 -- 172.21.15.61:0/1377880377 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f0714000e90 con 0x7f072c0b3cb0 2022-07-20T04:18:30.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.224+0000 7f072b7fe700 1 -- 172.21.15.61:0/1377880377 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f071400d8c0 con 0x7f072c0b3cb0 2022-07-20T04:18:30.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 -- 172.21.15.61:0/1377880377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b3cb0 msgr2=0x7f072c0aebf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 --2- 172.21.15.61:0/1377880377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b3cb0 0x7f072c0aebf0 secure :-1 s=READY pgs=1575 cs=0 l=1 rev1=1 rx=0x7f0714002f70 tx=0x7f0714006df0).stop 2022-07-20T04:18:30.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 -- 172.21.15.61:0/1377880377 shutdown_connections 2022-07-20T04:18:30.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 --2- 172.21.15.61:0/1377880377 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0ad0e0 0x7f072c1418f0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 --2- 172.21.15.61:0/1377880377 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b3cb0 0x7f072c0aebf0 unknown :-1 s=CLOSED pgs=1575 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 -- 172.21.15.61:0/1377880377 >> 172.21.15.61:0/1377880377 conn(0x7f072c0b9dd0 msgr2=0x7f072c0b5db0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:30.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 -- 172.21.15.61:0/1377880377 shutdown_connections 2022-07-20T04:18:30.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 -- 172.21.15.61:0/1377880377 wait complete. 2022-07-20T04:18:30.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 Processor -- start 2022-07-20T04:18:30.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.228+0000 7f0731cd1700 1 -- start start 2022-07-20T04:18:30.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f072c0ad0e0 0x7f072c1128b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0b3cb0 0x7f072c112da0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b6800 0x7f072c13fa70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c0b8d30 con 0x7f072c0b6800 2022-07-20T04:18:30.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f072c152910 con 0x7f072c0ad0e0 2022-07-20T04:18:30.242 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f072c152bb0 con 0x7f072c0b3cb0 2022-07-20T04:18:30.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f07314d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b6800 0x7f072c13fa70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.243 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f07314d0700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b6800 0x7f072c13fa70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52456/0 (socket says 172.21.15.61:52456) 2022-07-20T04:18:30.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f072c0ad0e0 0x7f072c1128b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.244 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f07314d0700 1 -- 172.21.15.61:0/3502168609 learned_addr learned my addr 172.21.15.61:0/3502168609 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:30.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f072bfff700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0b3cb0 0x7f072c112da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 -- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0b3cb0 msgr2=0x7f072c112da0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.245 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0b3cb0 0x7f072c112da0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 -- 172.21.15.61:0/3502168609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b6800 msgr2=0x7f072c13fa70 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b6800 0x7f072c13fa70 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.246 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 -- 172.21.15.61:0/3502168609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f071400b180 con 0x7f072c0ad0e0 2022-07-20T04:18:30.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0730ccf700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f072c0ad0e0 0x7f072c1128b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f071c000ed0 tx=0x7f071c007c20).ready entity=mon.1 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mon.1 v2:172.21.15.117:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f071c001d30 con 0x7f072c0ad0e0 2022-07-20T04:18:30.247 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mon.1 v2:172.21.15.117:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f071c0070d0 con 0x7f072c0ad0e0 2022-07-20T04:18:30.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f072c0afd50 con 0x7f072c0ad0e0 2022-07-20T04:18:30.248 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mon.1 v2:172.21.15.117:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f071c007430 con 0x7f072c0ad0e0 2022-07-20T04:18:30.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f072c0af970 con 0x7f072c0ad0e0 2022-07-20T04:18:30.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.232+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f072c152750 con 0x7f072c0ad0e0 2022-07-20T04:18:30.249 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.236+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mon.1 v2:172.21.15.117:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f071c010070 con 0x7f072c0ad0e0 2022-07-20T04:18:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.236+0000 7f0729ffb700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0710053af0 0x7f0710055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.250 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.236+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mon.1 v2:172.21.15.117:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f071c06b230 con 0x7f072c0ad0e0 2022-07-20T04:18:30.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.236+0000 7f072bfff700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0710053af0 0x7f0710055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.251 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.236+0000 7f072bfff700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0710053af0 0x7f0710055fb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f071401a000 tx=0x7f0714014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.252 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.244+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mon.1 v2:172.21.15.117:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f071c0422e0 con 0x7f072c0ad0e0 2022-07-20T04:18:30.373 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.372+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "osd deep-scrub", "who": "7", "target": ["mon-mgr", ""]}) v1 -- 0x7f072c0b03b0 con 0x7f0710053af0 2022-07-20T04:18:30.374 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.372+0000 7f0729ffb700 1 -- 172.21.15.61:0/3502168609 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 instructed osd(s) 7 to deep-scrub) v1 ==== 41+0+0 (secure 0 0 0) 0x7f072c0b03b0 con 0x7f0710053af0 2022-07-20T04:18:30.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0710053af0 msgr2=0x7f0710055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0710053af0 0x7f0710055fb0 secure :-1 s=READY pgs=822 cs=0 l=1 rev1=1 rx=0x7f071401a000 tx=0x7f0714014000).stop 2022-07-20T04:18:30.376 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f072c0ad0e0 msgr2=0x7f072c1128b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f072c0ad0e0 0x7f072c1128b0 secure :-1 s=READY pgs=697 cs=0 l=1 rev1=1 rx=0x7f071c000ed0 tx=0x7f071c007c20).stop 2022-07-20T04:18:30.377 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 shutdown_connections 2022-07-20T04:18:30.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f0710053af0 0x7f0710055fb0 unknown :-1 s=CLOSED pgs=822 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f072c0b6800 0x7f072c13fa70 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.378 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f072c0b3cb0 0x7f072c112da0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 --2- 172.21.15.61:0/3502168609 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f072c0ad0e0 0x7f072c1128b0 unknown :-1 s=CLOSED pgs=697 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 >> 172.21.15.61:0/3502168609 conn(0x7f072c0b9dd0 msgr2=0x7f072c142f90 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:30.379 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 shutdown_connections 2022-07-20T04:18:30.380 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.376+0000 7f0731cd1700 1 -- 172.21.15.61:0/3502168609 wait complete. 2022-07-20T04:18:30.381 INFO:teuthology.orchestra.run.smithi061.stderr:instructed osd(s) 7 to deep-scrub 2022-07-20T04:18:30.387 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T04:18:30.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 Processor -- start 2022-07-20T04:18:30.487 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- start start 2022-07-20T04:18:30.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0605a0 0x7fd93c13f980 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.490 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- --> v1:172.21.15.61:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c140940 con 0x7fd93c0ab180 2022-07-20T04:18:30.491 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7fd93c0b22d0 con 0x7fd93c0b1ef0 2022-07-20T04:18:30.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c05fbc0 con 0x7fd93c0605a0 2022-07-20T04:18:30.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd942a87700 1 --1- >> v1:172.21.15.61:6789/0 conn(0x7fd93c0ab180 0x7fd93c0ab5f0 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.61:6789/0 says I am v1:172.21.15.61:49062/0 (socket says 172.21.15.61:49062) 2022-07-20T04:18:30.492 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd942a87700 1 -- 172.21.15.61:0/322976695 learned_addr learned my addr 172.21.15.61:0/322976695 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:30.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941a85700 1 --2- 172.21.15.61:0/322976695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0605a0 0x7fd93c13f980 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 <== mon.0 v1:172.21.15.61:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3509933989 0 0) 0x7fd93c140940 con 0x7fd93c0ab180 2022-07-20T04:18:30.493 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 --> v1:172.21.15.61:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd924003260 con 0x7fd93c0ab180 2022-07-20T04:18:30.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 836433910 0 0) 0x7fd93c0b22d0 con 0x7fd93c0b1ef0 2022-07-20T04:18:30.494 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7fd93c140940 con 0x7fd93c0b1ef0 2022-07-20T04:18:30.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941a85700 1 -- 172.21.15.61:0/322976695 >> v1:172.21.15.117:6789/0 conn(0x7fd93c0b1ef0 legacy=0x7fd93c0abff0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941a85700 1 -- 172.21.15.61:0/322976695 >> v1:172.21.15.61:6789/0 conn(0x7fd93c0ab180 legacy=0x7fd93c0ab5f0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.495 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941a85700 1 -- 172.21.15.61:0/322976695 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd93c05fe60 con 0x7fd93c0605a0 2022-07-20T04:18:30.496 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941a85700 1 --2- 172.21.15.61:0/322976695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0605a0 0x7fd93c13f980 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd92c002f70 tx=0x7fd92c006df0).ready entity=mon.2 client_cookie=68c2163199777282 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.501 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd92c00d5c0 con 0x7fd93c0605a0 2022-07-20T04:18:30.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd92c000e90 con 0x7fd93c0605a0 2022-07-20T04:18:30.502 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd941284700 1 -- 172.21.15.61:0/322976695 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd92c00d910 con 0x7fd93c0605a0 2022-07-20T04:18:30.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- 172.21.15.61:0/322976695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0605a0 msgr2=0x7fd93c13f980 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 --2- 172.21.15.61:0/322976695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0605a0 0x7fd93c13f980 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fd92c002f70 tx=0x7fd92c006df0).stop 2022-07-20T04:18:30.503 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- 172.21.15.61:0/322976695 shutdown_connections 2022-07-20T04:18:30.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 --2- 172.21.15.61:0/322976695 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0605a0 0x7fd93c13f980 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.504 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- 172.21.15.61:0/322976695 >> 172.21.15.61:0/322976695 conn(0x7fd93c0b2ca0 msgr2=0x7fd93c0b30b0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:30.505 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- 172.21.15.61:0/322976695 shutdown_connections 2022-07-20T04:18:30.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.488+0000 7fd943288700 1 -- 172.21.15.61:0/322976695 wait complete. 2022-07-20T04:18:30.506 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 Processor -- start 2022-07-20T04:18:30.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 -- start start 2022-07-20T04:18:30.507 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 0x7fd93c1417b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.508 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd93c0ab180 0x7fd93c142f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.509 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0b1ef0 0x7fd93c143480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c141500 con 0x7fd93c0605a0 2022-07-20T04:18:30.510 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c141ca0 con 0x7fd93c0ab180 2022-07-20T04:18:30.511 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7fd93c142620 con 0x7fd93c0b1ef0 2022-07-20T04:18:30.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 0x7fd93c1417b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.512 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 0x7fd93c1417b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52466/0 (socket says 172.21.15.61:52466) 2022-07-20T04:18:30.513 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 -- 172.21.15.61:0/2905292296 learned_addr learned my addr 172.21.15.61:0/2905292296 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:30.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 -- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0b1ef0 msgr2=0x7fd93c143480 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.514 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942a87700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0b1ef0 0x7fd93c143480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.515 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd941a85700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd93c0ab180 0x7fd93c142f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0b1ef0 0x7fd93c143480 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.516 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 -- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd93c0ab180 msgr2=0x7fd93c142f90 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd93c0ab180 0x7fd93c142f90 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.517 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 -- 172.21.15.61:0/2905292296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd92c00b180 con 0x7fd93c0605a0 2022-07-20T04:18:30.518 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd941a85700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd93c0ab180 0x7fd93c142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T04:18:30.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942a87700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0b1ef0 0x7fd93c143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).send_auth_request state changed! 2022-07-20T04:18:30.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd942286700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 0x7fd93c1417b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7fd938001110 tx=0x7fd938007ae0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.520 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd93800ce10 con 0x7fd93c0605a0 2022-07-20T04:18:30.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fd93800f470 con 0x7fd93c0605a0 2022-07-20T04:18:30.521 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd93c065300 con 0x7fd93c0605a0 2022-07-20T04:18:30.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fd93800f7d0 con 0x7fd93c0605a0 2022-07-20T04:18:30.535 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.492+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd93c0b4490 con 0x7fd93c0605a0 2022-07-20T04:18:30.536 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.496+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd93c060100 con 0x7fd93c0605a0 2022-07-20T04:18:30.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.496+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fd938019030 con 0x7fd93c0605a0 2022-07-20T04:18:30.537 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.496+0000 7fd92b7fe700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd91c0539b0 0x7fd91c055e70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:30.538 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.496+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fd93806af70 con 0x7fd93c0605a0 2022-07-20T04:18:30.539 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.496+0000 7fd941a85700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd91c0539b0 0x7fd91c055e70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:30.540 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.496+0000 7fd941a85700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd91c0539b0 0x7fd91c055e70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fd92c015000 tx=0x7fd92c014000).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:30.541 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.504+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7fd9380421b0 con 0x7fd93c0605a0 2022-07-20T04:18:30.627 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.628+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7fd93c0574e0 con 0x7fd91c0539b0 2022-07-20T04:18:30.634 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.632+0000 7fd92b7fe700 1 -- 172.21.15.61:0/2905292296 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68144 (secure 0 0 0) 0x7fd93c0574e0 con 0x7fd91c0539b0 2022-07-20T04:18:30.636 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T04:18:30.638 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd91c0539b0 msgr2=0x7fd91c055e70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.639 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd91c0539b0 0x7fd91c055e70 secure :-1 s=READY pgs=823 cs=0 l=1 rev1=1 rx=0x7fd92c015000 tx=0x7fd92c014000).stop 2022-07-20T04:18:30.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 msgr2=0x7fd93c1417b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:30.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 0x7fd93c1417b0 secure :-1 s=READY pgs=1577 cs=0 l=1 rev1=1 rx=0x7fd938001110 tx=0x7fd938007ae0).stop 2022-07-20T04:18:30.640 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 shutdown_connections 2022-07-20T04:18:30.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fd91c0539b0 0x7fd91c055e70 unknown :-1 s=CLOSED pgs=823 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7fd93c0ab180 0x7fd93c142f90 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.641 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7fd93c0b1ef0 0x7fd93c143480 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 --2- 172.21.15.61:0/2905292296 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7fd93c0605a0 0x7fd93c1417b0 unknown :-1 s=CLOSED pgs=1577 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:30.642 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 >> 172.21.15.61:0/2905292296 conn(0x7fd93c0b2ca0 msgr2=0x7fd93c13f980 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 shutdown_connections 2022-07-20T04:18:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:30.636+0000 7fd943288700 1 -- 172.21.15.61:0/2905292296 wait complete. 2022-07-20T04:18:30.643 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-20T04:18:30.650 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2298,"stamp":"2022-07-20T04:18:30.201364+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385216,"kb_used_data":743209824,"kb_used_omap":27860,"kb_used_meta":0,"kb_avail":743209824,"statfs":{"total":767585320960,"available":761046859776,"internally_reserved":0,"allocated":761046859776,"data_stored":761046859776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28528982,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098255"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":135,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.839187+0000","last_change":"2022-07-20T03:07:00.764096+0000","last_active":"2022-07-20T03:17:53.839187+0000","last_peered":"2022-07-20T03:17:53.839187+0000","last_clean":"2022-07-20T03:17:53.839187+0000","last_became_active":"2022-07-20T03:07:00.763305+0000","last_became_peered":"2022-07-20T03:07:00.763305+0000","last_unstale":"2022-07-20T03:17:53.839187+0000","last_undegraded":"2022-07-20T03:17:53.839187+0000","last_fullsized":"2022-07-20T03:17:53.839187+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:06:54.339924+0000","last_clean_scrub_stamp":"2022-07-20T03:06:54.339924+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.311055+0000","last_change":"2022-07-20T03:07:06.947888+0000","last_active":"2022-07-20T03:17:53.311055+0000","last_peered":"2022-07-20T03:17:53.311055+0000","last_clean":"2022-07-20T03:17:53.311055+0000","last_became_active":"2022-07-20T03:07:06.947545+0000","last_became_peered":"2022-07-20T03:07:06.947545+0000","last_unstale":"2022-07-20T03:17:53.311055+0000","last_undegraded":"2022-07-20T03:17:53.311055+0000","last_fullsized":"2022-07-20T03:17:53.311055+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.839036+0000","last_change":"2022-07-20T03:07:06.947578+0000","last_active":"2022-07-20T03:17:53.839036+0000","last_peered":"2022-07-20T03:17:53.839036+0000","last_clean":"2022-07-20T03:17:53.839036+0000","last_became_active":"2022-07-20T03:07:06.947295+0000","last_became_peered":"2022-07-20T03:07:06.947295+0000","last_unstale":"2022-07-20T03:17:53.839036+0000","last_undegraded":"2022-07-20T03:17:53.839036+0000","last_fullsized":"2022-07-20T03:17:53.839036+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":136,"reported_epoch":150,"state":"active+clean","last_fresh":"2022-07-20T03:17:51.534520+0000","last_change":"2022-07-20T03:07:06.951565+0000","last_active":"2022-07-20T03:17:51.534520+0000","last_peered":"2022-07-20T03:17:51.534520+0000","last_clean":"2022-07-20T03:17:51.534520+0000","last_became_active":"2022-07-20T03:07:06.951117+0000","last_became_peered":"2022-07-20T03:07:06.951117+0000","last_unstale":"2022-07-20T03:17:51.534520+0000","last_undegraded":"2022-07-20T03:17:51.534520+0000","last_fullsized":"2022-07-20T03:17:51.534520+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":139,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.310985+0000","last_change":"2022-07-20T03:07:06.946761+0000","last_active":"2022-07-20T03:17:53.310985+0000","last_peered":"2022-07-20T03:17:53.310985+0000","last_clean":"2022-07-20T03:17:53.310985+0000","last_became_active":"2022-07-20T03:07:06.946278+0000","last_became_peered":"2022-07-20T03:07:06.946278+0000","last_unstale":"2022-07-20T03:17:53.310985+0000","last_undegraded":"2022-07-20T03:17:53.310985+0000","last_fullsized":"2022-07-20T03:17:53.310985+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":136,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.311202+0000","last_change":"2022-07-20T03:07:06.951643+0000","last_active":"2022-07-20T03:17:53.311202+0000","last_peered":"2022-07-20T03:17:53.311202+0000","last_clean":"2022-07-20T03:17:53.311202+0000","last_became_active":"2022-07-20T03:07:06.951291+0000","last_became_peered":"2022-07-20T03:07:06.951291+0000","last_unstale":"2022-07-20T03:17:53.311202+0000","last_undegraded":"2022-07-20T03:17:53.311202+0000","last_fullsized":"2022-07-20T03:17:53.311202+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.839347+0000","last_change":"2022-07-20T03:07:06.943629+0000","last_active":"2022-07-20T03:17:53.839347+0000","last_peered":"2022-07-20T03:17:53.839347+0000","last_clean":"2022-07-20T03:17:53.839347+0000","last_became_active":"2022-07-20T03:07:06.942970+0000","last_became_peered":"2022-07-20T03:07:06.942970+0000","last_unstale":"2022-07-20T03:17:53.839347+0000","last_undegraded":"2022-07-20T03:17:53.839347+0000","last_fullsized":"2022-07-20T03:17:53.839347+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":136,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.311325+0000","last_change":"2022-07-20T03:07:06.950534+0000","last_active":"2022-07-20T03:17:53.311325+0000","last_peered":"2022-07-20T03:17:53.311325+0000","last_clean":"2022-07-20T03:17:53.311325+0000","last_became_active":"2022-07-20T03:07:06.949930+0000","last_became_peered":"2022-07-20T03:07:06.949930+0000","last_unstale":"2022-07-20T03:17:53.311325+0000","last_undegraded":"2022-07-20T03:17:53.311325+0000","last_fullsized":"2022-07-20T03:17:53.311325+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":132,"reported_epoch":151,"state":"active+clean","last_fresh":"2022-07-20T03:17:53.326385+0000","last_change":"2022-07-20T03:07:06.946858+0000","last_active":"2022-07-20T03:17:53.326385+0000","last_peered":"2022-07-20T03:17:53.326385+0000","last_clean":"2022-07-20T03:17:53.326385+0000","last_became_active":"2022-07-20T03:07:06.946555+0000","last_became_peered":"2022-07-20T03:07:06.946555+0000","last_unstale":"2022-07-20T03:17:53.326385+0000","last_undegraded":"2022-07-20T03:17:53.326385+0000","last_fullsized":"2022-07-20T03:17:53.326385+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T03:07:05.920131+0000","last_clean_scrub_stamp":"2022-07-20T03:07:05.920131+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575707,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3650,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3737996,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.83299999999999996,"15min":0.83799999999999997},"min":{"1min":0.56200000000000006,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.3049999999999999},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.84899999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.56599999999999995},"max":{"1min":1.1299999999999999,"5min":1.236,"15min":1.236},"last":0.96499999999999997}]},{"osd":1,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.61199999999999999,"5min":0.495,"15min":0.495},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.3660000000000001},"last":0.98699999999999999},{"interface":"front","average":{"1min":0.878,"5min":0.87,"15min":0.874},"min":{"1min":0.64300000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.107,"5min":1.2110000000000001,"15min":1.248},"last":0.63300000000000001}]},{"osd":2,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.84499999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.60899999999999999},"max":{"1min":1.0329999999999999,"5min":1.482,"15min":1.482},"last":0.80200000000000005},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91200000000000003,"15min":0.90900000000000003},"min":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.56799999999999995},"max":{"1min":1.232,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.91100000000000003}]},{"osd":3,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89100000000000001,"15min":0.872},"min":{"1min":0.67300000000000004,"5min":0.65500000000000003,"15min":0.61599999999999999},"max":{"1min":1.0249999999999999,"5min":1.413,"15min":1.413},"last":0.875},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.90400000000000003,"15min":0.92300000000000004},"min":{"1min":0.68700000000000006,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1639999999999999,"5min":1.1799999999999999,"15min":1.2769999999999999},"last":1.04}]},{"osd":4,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.81599999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"max":{"1min":1.0109999999999999,"5min":1.181,"15min":1.181},"last":0.55800000000000005},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.86799999999999999,"15min":0.88300000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.1279999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.66700000000000004}]},{"osd":5,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":0.91100000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.54900000000000004},"max":{"1min":1.1220000000000001,"5min":1.2370000000000001,"15min":1.3169999999999999},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.93899999999999995},"min":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.415},"last":0.81699999999999995}]},{"osd":6,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.621},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3540000000000001},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.97399999999999998},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.627},"max":{"1min":1.173,"5min":1.532,"15min":1.532},"last":0.94199999999999995}]}]},{"osd":6,"up_from":13,"seq":55834575708,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":4005,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4101914,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.86699999999999999,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.1799999999999999},"last":0.89300000000000002},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.83999999999999997,"15min":0.85099999999999998},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.0529999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.77300000000000002}]},{"osd":1,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81899999999999995,"15min":0.81000000000000005},"min":{"1min":0.626,"5min":0.59599999999999997,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":1.149,"15min":1.2030000000000001},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.871,"5min":0.86699999999999999,"15min":0.86499999999999999},"min":{"1min":0.73899999999999999,"5min":0.625,"15min":0.56499999999999995},"max":{"1min":1.0389999999999999,"5min":1.1140000000000001,"15min":1.323},"last":0.76300000000000001}]},{"osd":2,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.877,"15min":0.86199999999999999},"min":{"1min":0.66400000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.081,"5min":1.1579999999999999,"15min":1.161},"last":0.85199999999999998},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85099999999999998,"15min":0.84799999999999998},"min":{"1min":0.65600000000000003,"5min":0.63900000000000001,"15min":0.61599999999999999},"max":{"1min":1.1240000000000001,"5min":1.1259999999999999,"15min":1.129},"last":0.873}]},{"osd":3,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.875,"15min":0.86899999999999999},"min":{"1min":0.71199999999999997,"5min":0.66700000000000004,"15min":0.63100000000000001},"max":{"1min":1.0649999999999999,"5min":1.179,"15min":1.179},"last":0.69799999999999995},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86699999999999999,"15min":0.88},"min":{"1min":0.76000000000000001,"5min":0.66800000000000004,"15min":0.63500000000000001},"max":{"1min":1.0760000000000001,"5min":1.1160000000000001,"15min":1.2110000000000001},"last":0.80400000000000005}]},{"osd":4,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78300000000000003,"15min":0.79700000000000004},"min":{"1min":0.55800000000000005,"5min":0.54300000000000004,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.218},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.88400000000000001,"15min":0.878},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.056,"5min":1.236,"15min":1.2969999999999999},"last":0.93400000000000005}]},{"osd":5,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85499999999999998,"15min":0.85299999999999998},"min":{"1min":0.65400000000000003,"5min":0.60799999999999998,"15min":0.57899999999999996},"max":{"1min":1.1499999999999999,"5min":1.204,"15min":1.2549999999999999},"last":0.82999999999999996},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.96599999999999997,"15min":0.96399999999999997},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.61699999999999999},"max":{"1min":1.131,"5min":1.165,"15min":1.327},"last":0.97199999999999998}]},{"osd":7,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95999999999999996,"15min":0.94899999999999995},"min":{"1min":0.70999999999999996,"5min":0.63800000000000001,"15min":0.58899999999999997},"max":{"1min":1.079,"5min":1.173,"15min":1.2430000000000001},"last":1.0009999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92500000000000004,"15min":0.93500000000000005},"min":{"1min":0.66400000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2010000000000001,"5min":1.216,"15min":1.3300000000000001},"last":0.72999999999999998}]}]},{"osd":5,"up_from":13,"seq":55834575707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":2740,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2806484,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.874,"5min":0.89300000000000002,"15min":0.89400000000000002},"min":{"1min":0.60899999999999999,"5min":0.57599999999999996,"15min":0.48999999999999999},"max":{"1min":1.099,"5min":1.117,"15min":1.1990000000000001},"last":0.82399999999999995},{"interface":"front","average":{"1min":0.872,"5min":0.89100000000000001,"15min":0.88900000000000001},"min":{"1min":0.72499999999999998,"5min":0.69699999999999995,"15min":0.66200000000000003},"max":{"1min":0.98799999999999999,"5min":1.139,"15min":1.2150000000000001},"last":0.996}]},{"osd":1,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.86399999999999999,"5min":0.82299999999999995,"15min":0.83499999999999996},"min":{"1min":0.61099999999999999,"5min":0.59599999999999997,"15min":0.57899999999999996},"max":{"1min":1.161,"5min":1.161,"15min":1.46},"last":0.97799999999999998},{"interface":"front","average":{"1min":0.85899999999999999,"5min":0.84599999999999997,"15min":0.83699999999999997},"min":{"1min":0.67300000000000004,"5min":0.623,"15min":0.56799999999999995},"max":{"1min":1.0549999999999999,"5min":1.173,"15min":1.2589999999999999},"last":0.61699999999999999}]},{"osd":2,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84599999999999997,"5min":0.85899999999999999,"15min":0.84099999999999997},"min":{"1min":0.64700000000000002,"5min":0.59199999999999997,"15min":0.56799999999999995},"max":{"1min":1.105,"5min":1.173,"15min":1.1819999999999999},"last":0.92600000000000005},{"interface":"front","average":{"1min":0.91100000000000003,"5min":0.86599999999999999,"15min":0.86799999999999999},"min":{"1min":0.64900000000000002,"5min":0.61799999999999999,"15min":0.54900000000000004},"max":{"1min":1.198,"5min":1.198,"15min":1.198},"last":0.90900000000000003}]},{"osd":3,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.79200000000000004,"5min":0.83999999999999997,"15min":0.84299999999999997},"min":{"1min":0.64100000000000001,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0780000000000001,"5min":1.1899999999999999,"15min":1.482},"last":0.83899999999999997},{"interface":"front","average":{"1min":1.0429999999999999,"5min":1.012,"15min":1.008},"min":{"1min":0.871,"5min":0.67900000000000005,"15min":0.623},"max":{"1min":1.274,"5min":1.274,"15min":1.3899999999999999},"last":1.046}]},{"osd":4,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.84399999999999997,"5min":0.83399999999999996,"15min":0.84899999999999998},"min":{"1min":0.58699999999999997,"5min":0.52800000000000002,"15min":0.435},"max":{"1min":1.0289999999999999,"5min":1.1599999999999999,"15min":1.266},"last":0.75600000000000001},{"interface":"front","average":{"1min":0.84499999999999997,"5min":0.85499999999999998,"15min":0.874},"min":{"1min":0.61699999999999999,"5min":0.58099999999999996,"15min":0.54500000000000004},"max":{"1min":1.119,"5min":1.2889999999999999,"15min":1.325},"last":0.73299999999999998}]},{"osd":6,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.93799999999999994,"5min":0.90600000000000003,"15min":0.92300000000000004},"min":{"1min":0.57899999999999996,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.2390000000000001,"5min":1.417,"15min":1.417},"last":1.1080000000000001},{"interface":"front","average":{"1min":0.91800000000000004,"5min":0.95399999999999996,"15min":0.96799999999999997},"min":{"1min":0.64000000000000001,"5min":0.59699999999999998,"15min":0.56100000000000005},"max":{"1min":1.1240000000000001,"5min":1.3879999999999999,"15min":1.452},"last":1.03}]},{"osd":7,"last update":"Wed Jul 20 04:17:23 2022","interfaces":[{"interface":"back","average":{"1min":0.95499999999999996,"5min":0.94199999999999995,"15min":0.97299999999999998},"min":{"1min":0.72799999999999998,"5min":0.58199999999999996,"15min":0.53600000000000003},"max":{"1min":1.2509999999999999,"5min":1.3300000000000001,"15min":1.4339999999999999},"last":1.0840000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.04,"15min":1.0620000000000001},"min":{"1min":0.76000000000000001,"5min":0.75,"15min":0.60899999999999999},"max":{"1min":1.3959999999999999,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.1639999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575708,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798040,"kb_used_data":92901340,"kb_used_omap":3532,"kb_used_meta":0,"kb_avail":92901340,"statfs":{"total":95948165120,"available":95130972160,"internally_reserved":0,"allocated":95130972160,"data_stored":95130972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3616802,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84499999999999997,"15min":0.84899999999999998},"min":{"1min":0.62,"5min":0.58499999999999996,"15min":0.43099999999999999},"max":{"1min":0.98099999999999998,"5min":1.1279999999999999,"15min":1.1879999999999999},"last":0.98499999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84099999999999997,"15min":0.83899999999999997},"min":{"1min":0.67200000000000004,"5min":0.56799999999999995,"15min":0.52400000000000002},"max":{"1min":1.0469999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.80700000000000005}]},{"osd":1,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.86299999999999999,"15min":0.85299999999999998},"min":{"1min":0.65500000000000003,"5min":0.61399999999999999,"15min":0.48899999999999999},"max":{"1min":0.95399999999999996,"5min":1.1499999999999999,"15min":1.1519999999999999},"last":0.70399999999999996},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.56899999999999995},"max":{"1min":1.123,"5min":1.153,"15min":1.1890000000000001},"last":0.92600000000000005}]},{"osd":2,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.64400000000000002,"5min":0.63900000000000001,"15min":0.629},"max":{"1min":1.0980000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":1.04},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.88100000000000001,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.623,"15min":0.56299999999999994},"max":{"1min":1.133,"5min":1.1359999999999999,"15min":1.2270000000000001},"last":0.86899999999999999}]},{"osd":3,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85499999999999998,"15min":0.85099999999999998},"min":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.622},"max":{"1min":1.0489999999999999,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":1.0089999999999999},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.94799999999999995,"15min":0.93799999999999994},"min":{"1min":0.83699999999999997,"5min":0.66300000000000003,"15min":0.60499999999999998},"max":{"1min":1.111,"5min":1.264,"15min":1.79},"last":0.77500000000000002}]},{"osd":5,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.83699999999999997,"15min":0.85499999999999998},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1020000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":0.61699999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.54900000000000004},"max":{"1min":0.98499999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.90300000000000002}]},{"osd":6,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.85999999999999999,"15min":0.878},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.46100000000000002},"max":{"1min":1.141,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":0.82799999999999996},{"interface":"front","average":{"1min":0.877,"5min":0.93600000000000005,"15min":0.94799999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61299999999999999},"max":{"1min":1.0740000000000001,"5min":1.22,"15min":1.2649999999999999},"last":0.78800000000000003}]},{"osd":7,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93100000000000005,"15min":0.93000000000000005},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1259999999999999,"5min":1.3120000000000001,"15min":1.337},"last":0.72999999999999998},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.002,"15min":1.01},"min":{"1min":0.71899999999999997,"5min":0.71799999999999997,"15min":0.502},"max":{"1min":1.3009999999999999,"5min":1.359,"15min":1.5229999999999999},"last":0.95699999999999996}]}]},{"osd":3,"up_from":13,"seq":55834575707,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3372,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3453227,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.56999999999999995,"5min":0.57799999999999996,"15min":0.59699999999999998},"min":{"1min":0.47999999999999998,"5min":0.45300000000000001,"15min":0.44900000000000001},"max":{"1min":0.84999999999999998,"5min":0.94399999999999995,"15min":1.069},"last":0.72199999999999998},{"interface":"front","average":{"1min":0.66400000000000003,"5min":0.67400000000000004,"15min":0.67900000000000005},"min":{"1min":0.55100000000000005,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":0.81000000000000005,"5min":0.98799999999999999,"15min":1.095},"last":0.66900000000000004}]},{"osd":1,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.71599999999999997,"5min":0.70299999999999996,"15min":0.71099999999999997},"min":{"1min":0.48499999999999999,"5min":0.47399999999999998,"15min":0.44400000000000001},"max":{"1min":0.92600000000000005,"5min":1.119,"15min":1.298},"last":0.69499999999999995},{"interface":"front","average":{"1min":0.81000000000000005,"5min":0.79800000000000004,"15min":0.81000000000000005},"min":{"1min":0.69499999999999995,"5min":0.60799999999999998,"15min":0.53100000000000003},"max":{"1min":1.014,"5min":1.046,"15min":1.1180000000000001},"last":0.94999999999999996}]},{"osd":2,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.77900000000000003,"5min":0.75600000000000001,"15min":0.76200000000000001},"min":{"1min":0.58099999999999996,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":1.0349999999999999,"5min":1.127,"15min":1.2},"last":0.77800000000000002},{"interface":"front","average":{"1min":0.78000000000000003,"5min":0.77300000000000002,"15min":0.77200000000000002},"min":{"1min":0.60499999999999998,"5min":0.55300000000000005,"15min":0.51900000000000002},"max":{"1min":1.0660000000000001,"5min":1.133,"15min":1.1339999999999999},"last":0.63}]},{"osd":4,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.873,"5min":0.871,"15min":0.85899999999999999},"min":{"1min":0.70799999999999996,"5min":0.628,"15min":0.58499999999999996},"max":{"1min":1.0409999999999999,"5min":1.246,"15min":1.2809999999999999},"last":0.80500000000000005},{"interface":"front","average":{"1min":0.95699999999999996,"5min":0.93600000000000005,"15min":0.93899999999999995},"min":{"1min":0.76700000000000002,"5min":0.71999999999999997,"15min":0.68899999999999995},"max":{"1min":1.216,"5min":1.544,"15min":1.544},"last":0.96699999999999997}]},{"osd":5,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.96499999999999997,"5min":0.95199999999999996,"15min":0.95799999999999996},"min":{"1min":0.78700000000000003,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.161,"5min":1.454,"15min":1.454},"last":0.84799999999999998},{"interface":"front","average":{"1min":0.99399999999999999,"5min":0.96799999999999997,"15min":0.98399999999999999},"min":{"1min":0.80600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.325,"5min":1.496,"15min":1.496},"last":0.873}]},{"osd":6,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":0.99099999999999999,"5min":1.002,"15min":1.0109999999999999},"min":{"1min":0.79200000000000004,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.282,"5min":1.448,"15min":1.502},"last":1.0389999999999999},{"interface":"front","average":{"1min":0.95799999999999996,"5min":0.98399999999999999,"15min":0.99099999999999999},"min":{"1min":0.755,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.145,"5min":1.3500000000000001,"15min":1.3660000000000001},"last":0.90200000000000002}]},{"osd":7,"last update":"Wed Jul 20 04:17:25 2022","interfaces":[{"interface":"back","average":{"1min":1.018,"5min":1.0229999999999999,"15min":1.03},"min":{"1min":0.81899999999999995,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.2170000000000001,"5min":1.464,"15min":1.464},"last":0.99199999999999999},{"interface":"front","average":{"1min":1.048,"5min":1.071,"15min":1.073},"min":{"1min":0.83899999999999997,"5min":0.80800000000000005,"15min":0.69999999999999996},"max":{"1min":1.3740000000000001,"5min":1.5069999999999999,"15min":1.5409999999999999},"last":1.0640000000000001}]}]},{"osd":2,"up_from":13,"seq":55834575707,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":2842,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2910987,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.503,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.81899999999999995,"5min":0.91300000000000003,"15min":1.054},"last":0.58699999999999997},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.56299999999999994,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.875,"5min":1.153,"15min":1.153},"last":0.67200000000000004}]},{"osd":1,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73499999999999999,"15min":0.72499999999999998},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.97699999999999998,"5min":1.0589999999999999,"15min":1.147},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82699999999999996,"15min":0.82899999999999996},"min":{"1min":0.70599999999999996,"5min":0.64100000000000001,"15min":0.57199999999999995},"max":{"1min":1.0229999999999999,"5min":1.173,"15min":1.173},"last":0.77000000000000002}]},{"osd":3,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.70399999999999996,"15min":0.70099999999999996},"min":{"1min":0.57599999999999996,"5min":0.48599999999999999,"15min":0.46500000000000002},"max":{"1min":1.026,"5min":1.0840000000000001,"15min":1.1699999999999999},"last":0.70999999999999996},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.65300000000000002,"5min":0.57699999999999996,"15min":0.55700000000000005},"max":{"1min":1.0189999999999999,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.90300000000000002}]},{"osd":4,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.95199999999999996,"15min":0.94799999999999995},"min":{"1min":0.81399999999999995,"5min":0.71699999999999997,"15min":0.64700000000000002},"max":{"1min":1.1830000000000001,"5min":1.345,"15min":1.415},"last":1.0469999999999999},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.78700000000000003,"5min":0.68799999999999994,"15min":0.67400000000000004},"max":{"1min":1.091,"5min":1.2110000000000001,"15min":1.3799999999999999},"last":0.93500000000000005}]},{"osd":5,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.98499999999999999,"15min":0.98699999999999999},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.67500000000000004},"max":{"1min":1.262,"5min":1.446,"15min":1.5669999999999999},"last":0.84699999999999998},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.92500000000000004,"15min":0.94199999999999995},"min":{"1min":0.72899999999999998,"5min":0.69199999999999995,"15min":0.67200000000000004},"max":{"1min":1.1599999999999999,"5min":1.244,"15min":1.3540000000000001},"last":1.0780000000000001}]},{"osd":6,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.996,"15min":1.0089999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.2310000000000001,"5min":1.526,"15min":1.526},"last":1.01},{"interface":"front","average":{"1min":1.069,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.91200000000000003,"5min":0.77500000000000002,"15min":0.75700000000000001},"max":{"1min":1.355,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":0.96599999999999997}]},{"osd":7,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0800000000000001,"15min":1.081},"min":{"1min":0.75900000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.3160000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.1359999999999999},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.044,"15min":1.0449999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.71299999999999997},"max":{"1min":1.2829999999999999,"5min":1.321,"15min":1.524},"last":1.175}]}]},{"osd":1,"up_from":13,"seq":55834575707,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3948,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4043670,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45400000000000001},"max":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":1.016},"last":0.52800000000000002},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52600000000000002,"5min":0.48599999999999999,"15min":0.47199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.135},"last":0.82299999999999995}]},{"osd":2,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.51600000000000001},"max":{"1min":1.044,"5min":1.0900000000000001,"15min":1.155},"last":0.73599999999999999},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.745},"min":{"1min":0.53000000000000003,"5min":0.50600000000000001,"15min":0.48499999999999999},"max":{"1min":1.0529999999999999,"5min":1.1519999999999999,"15min":1.2629999999999999},"last":0.58799999999999997}]},{"osd":3,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79000000000000004,"15min":0.80500000000000005},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.54600000000000004},"max":{"1min":0.92100000000000004,"5min":1.2450000000000001,"15min":1.347},"last":0.91700000000000004},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.76700000000000002},"min":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.51600000000000001},"max":{"1min":0.98099999999999998,"5min":1.1559999999999999,"15min":1.1890000000000001},"last":0.66500000000000004}]},{"osd":4,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.91700000000000004,"15min":0.91900000000000004},"min":{"1min":0.74299999999999999,"5min":0.68600000000000005,"15min":0.68000000000000005},"max":{"1min":1.1830000000000001,"5min":1.2090000000000001,"15min":1.3049999999999999},"last":0.85899999999999999},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95099999999999996,"15min":0.95899999999999996},"min":{"1min":0.745,"5min":0.73799999999999999,"15min":0.67400000000000004},"max":{"1min":1.212,"5min":1.3899999999999999,"15min":1.421},"last":0.79000000000000004}]},{"osd":5,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.98399999999999999},"min":{"1min":0.76800000000000002,"5min":0.73399999999999999,"15min":0.68999999999999995},"max":{"1min":1.266,"5min":1.3700000000000001,"15min":1.381},"last":1.159},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0329999999999999,"15min":1.03},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.755},"max":{"1min":1.248,"5min":1.3420000000000001,"15min":1.4139999999999999},"last":0.94699999999999995}]},{"osd":6,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0289999999999999,"15min":1.0409999999999999},"min":{"1min":0.79400000000000004,"5min":0.74299999999999999,"15min":0.72499999999999998},"max":{"1min":1.3180000000000001,"5min":1.3320000000000001,"15min":1.4390000000000001},"last":1.222},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0660000000000001,"15min":1.079},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.77200000000000002},"max":{"1min":1.278,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.0429999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0609999999999999,"15min":1.0820000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.446,"5min":1.462,"15min":1.484},"last":0.98999999999999999},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.1080000000000001},"min":{"1min":0.877,"5min":0.84299999999999997,"15min":0.73299999999999998},"max":{"1min":1.2929999999999999,"5min":1.329,"15min":1.5720000000000001},"last":1.262}]}]},{"osd":0,"up_from":13,"seq":55834575708,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3767,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3857902,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.66300000000000003,"5min":0.66200000000000003,"15min":0.66300000000000003},"min":{"1min":0.47299999999999998,"5min":0.45500000000000002,"15min":0.45500000000000002},"max":{"1min":0.91000000000000003,"5min":0.94399999999999995,"15min":1.141},"last":0.63400000000000001},{"interface":"front","average":{"1min":0.73899999999999999,"5min":0.745,"15min":0.751},"min":{"1min":0.59699999999999998,"5min":0.59699999999999998,"15min":0.56699999999999995},"max":{"1min":0.92000000000000004,"5min":0.95999999999999996,"15min":1.129},"last":0.72499999999999998}]},{"osd":2,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.64800000000000002,"5min":0.72299999999999998,"15min":0.72599999999999998},"min":{"1min":0.53300000000000003,"5min":0.53300000000000003,"15min":0.496},"max":{"1min":0.93000000000000005,"5min":1.0309999999999999,"15min":1.2290000000000001},"last":0.53100000000000003},{"interface":"front","average":{"1min":0.81499999999999995,"5min":0.73499999999999999,"15min":0.73399999999999999},"min":{"1min":0.60499999999999998,"5min":0.50900000000000001,"15min":0.503},"max":{"1min":1.091,"5min":1.103,"15min":1.103},"last":0.59299999999999997}]},{"osd":3,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.69999999999999996,"5min":0.72299999999999998,"15min":0.70799999999999996},"min":{"1min":0.51300000000000001,"5min":0.50800000000000001,"15min":0.49199999999999999},"max":{"1min":0.96799999999999997,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.57699999999999996},{"interface":"front","average":{"1min":0.84199999999999997,"5min":0.85099999999999998,"15min":0.85399999999999998},"min":{"1min":0.64200000000000002,"5min":0.64200000000000002,"15min":0.54800000000000004},"max":{"1min":1.0680000000000001,"5min":1.171,"15min":1.2749999999999999},"last":0.69399999999999995}]},{"osd":4,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.95099999999999996,"5min":0.94899999999999995,"15min":0.93999999999999995},"min":{"1min":0.748,"5min":0.71899999999999997,"15min":0.64400000000000002},"max":{"1min":1.2050000000000001,"5min":1.258,"15min":1.3180000000000001},"last":0.92400000000000004},{"interface":"front","average":{"1min":0.88100000000000001,"5min":0.91800000000000004,"15min":0.91800000000000004},"min":{"1min":0.70399999999999996,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0960000000000001,"5min":1.3069999999999999,"15min":1.3839999999999999},"last":0.871}]},{"osd":5,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.98499999999999999,"5min":1.0149999999999999,"15min":1.0229999999999999},"min":{"1min":0.85099999999999998,"5min":0.76700000000000002,"15min":0.64900000000000002},"max":{"1min":1.1950000000000001,"5min":1.448,"15min":1.571},"last":0.81599999999999995},{"interface":"front","average":{"1min":1.0580000000000001,"5min":1.0109999999999999,"15min":1.012},"min":{"1min":0.89300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.4410000000000001,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.83899999999999997}]},{"osd":6,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":0.96899999999999997,"5min":0.98199999999999998,"15min":0.999},"min":{"1min":0.76600000000000001,"5min":0.70999999999999996,"15min":0.70199999999999996},"max":{"1min":1.1279999999999999,"5min":1.3819999999999999,"15min":1.4550000000000001},"last":0.68400000000000005},{"interface":"front","average":{"1min":1.0660000000000001,"5min":1.0920000000000001,"15min":1.0900000000000001},"min":{"1min":0.77000000000000002,"5min":0.77000000000000002,"15min":0.71199999999999997},"max":{"1min":1.2629999999999999,"5min":1.5049999999999999,"15min":1.617},"last":0.754}]},{"osd":7,"last update":"Wed Jul 20 04:17:26 2022","interfaces":[{"interface":"back","average":{"1min":1.131,"5min":1.0900000000000001,"15min":1.087},"min":{"1min":0.88800000000000001,"5min":0.75800000000000001,"15min":0.71699999999999997},"max":{"1min":1.522,"5min":1.522,"15min":1.625},"last":0.99399999999999999},{"interface":"front","average":{"1min":1.0760000000000001,"5min":1.0840000000000001,"15min":1.091},"min":{"1min":0.96499999999999997,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.2030000000000001,"5min":1.5109999999999999,"15min":1.571},"last":0.82799999999999996}]}]}]}} 2022-07-20T04:18:30.666 INFO:tasks.ceph:pgid 1.0 last_scrub_stamp 2022-07-20T03:06:54.339924+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=6, tm_sec=54, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.667 INFO:tasks.ceph:pgid 2.3 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.668 INFO:tasks.ceph:pgid 2.0 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.669 INFO:tasks.ceph:pgid 2.1 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.670 INFO:tasks.ceph:pgid 2.2 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.670 INFO:tasks.ceph:pgid 2.4 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.671 INFO:tasks.ceph:pgid 2.5 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.672 INFO:tasks.ceph:pgid 2.6 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.672 INFO:tasks.ceph:pgid 2.7 last_scrub_stamp 2022-07-20T03:07:05.920131+0000 time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=3, tm_min=7, tm_sec=5, tm_wday=2, tm_yday=201, tm_isdst=-1) <= time.struct_time(tm_year=2022, tm_mon=7, tm_mday=20, tm_hour=4, tm_min=18, tm_sec=25, tm_wday=2, tm_yday=201, tm_isdst=0) 2022-07-20T04:18:30.673 INFO:tasks.ceph:Still waiting for all pgs to be scrubbed. 2022-07-20T04:18:50.171 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:50.175 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:50.179 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:50.183 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:50.187 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:50.675 DEBUG:teuthology.orchestra.run.smithi061:> sudo adjust-ulimits ceph-coverage /home/ubuntu/cephtest/archive/coverage timeout 120 ceph --cluster ceph pg dump --format=json 2022-07-20T04:18:50.790 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.788+0000 7f93789f5700 1 Processor -- start 2022-07-20T04:18:50.791 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.788+0000 7f93789f5700 1 -- start start 2022-07-20T04:18:50.794 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7f93740b1e00 0x7f937414a6d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:50.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93740acef0 0x7f9374146410 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:50.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7f937405d480 con 0x7f937405c3f0 2022-07-20T04:18:50.795 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- --> v2:172.21.15.61:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f93740b21e0 con 0x7f93740acef0 2022-07-20T04:18:50.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7f937405cf40 con 0x7f93740b1e00 2022-07-20T04:18:50.796 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93740acef0 0x7f9374146410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:50.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 --2- >> v2:172.21.15.61:3300/0 conn(0x7f93740acef0 0x7f9374146410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52478/0 (socket says 172.21.15.61:52478) 2022-07-20T04:18:50.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 -- 172.21.15.61:0/1884263037 learned_addr learned my addr 172.21.15.61:0/1884263037 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:50.797 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937259c700 1 --2- 172.21.15.61:0/1884263037 >> v2:172.21.15.117:3300/0 conn(0x7f93740b1e00 0x7f937414a6d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:50.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937159a700 1 -- 172.21.15.61:0/1884263037 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3652679524 0 0) 0x7f937405d480 con 0x7f937405c3f0 2022-07-20T04:18:50.798 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937159a700 1 -- 172.21.15.61:0/1884263037 --> v1:172.21.15.117:6789/0 -- auth(proto 2 36 bytes epoch 0) v1 -- 0x7f9358003260 con 0x7f937405c3f0 2022-07-20T04:18:50.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 -- 172.21.15.61:0/1884263037 >> v2:172.21.15.117:3300/0 conn(0x7f93740b1e00 msgr2=0x7f937414a6d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:50.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 --2- 172.21.15.61:0/1884263037 >> v2:172.21.15.117:3300/0 conn(0x7f93740b1e00 0x7f937414a6d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.799 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 -- 172.21.15.61:0/1884263037 >> v1:172.21.15.117:6789/0 conn(0x7f937405c3f0 legacy=0x7f93740ac7e0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:50.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 -- 172.21.15.61:0/1884263037 --> v2:172.21.15.61:3300/0 -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f937405d1e0 con 0x7f93740acef0 2022-07-20T04:18:50.800 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937259c700 1 --2- 172.21.15.61:0/1884263037 >> v2:172.21.15.117:3300/0 conn(0x7f93740b1e00 0x7f937414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_auth_done state changed! 2022-07-20T04:18:50.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f9372d9d700 1 --2- 172.21.15.61:0/1884263037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740acef0 0x7f9374146410 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f936c002f70 tx=0x7f936c006eb0).ready entity=mon.0 client_cookie=c5f17008c422177c server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:50.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937159a700 1 -- 172.21.15.61:0/1884263037 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936c00d610 con 0x7f93740acef0 2022-07-20T04:18:50.801 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937159a700 1 -- 172.21.15.61:0/1884263037 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f936c000e90 con 0x7f93740acef0 2022-07-20T04:18:50.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f937159a700 1 -- 172.21.15.61:0/1884263037 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f936c00d9a0 con 0x7f93740acef0 2022-07-20T04:18:50.802 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- 172.21.15.61:0/1884263037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740acef0 msgr2=0x7f9374146410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:50.803 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 --2- 172.21.15.61:0/1884263037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740acef0 0x7f9374146410 secure :-1 s=READY pgs=1578 cs=0 l=1 rev1=1 rx=0x7f936c002f70 tx=0x7f936c006eb0).stop 2022-07-20T04:18:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- 172.21.15.61:0/1884263037 shutdown_connections 2022-07-20T04:18:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 --2- 172.21.15.61:0/1884263037 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740acef0 0x7f9374146410 unknown :-1 s=CLOSED pgs=1578 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.804 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 --2- 172.21.15.61:0/1884263037 >> v2:172.21.15.117:3300/0 conn(0x7f93740b1e00 0x7f937414a6d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- 172.21.15.61:0/1884263037 >> 172.21.15.61:0/1884263037 conn(0x7f93740611c0 msgr2=0x7f9374062af0 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:50.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.792+0000 7f93789f5700 1 -- 172.21.15.61:0/1884263037 shutdown_connections 2022-07-20T04:18:50.805 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 -- 172.21.15.61:0/1884263037 wait complete. 2022-07-20T04:18:50.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 Processor -- start 2022-07-20T04:18:50.809 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 -- start start 2022-07-20T04:18:50.810 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93740acef0 0x7f93740a1970 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:50.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 0x7f93740a0360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:50.812 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93740a0850 0x7f93740a0d00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:50.813 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937409e1a0 con 0x7f93740b1e00 2022-07-20T04:18:50.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f937409eea0 con 0x7f93740acef0 2022-07-20T04:18:50.814 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f93789f5700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7f9374149870 con 0x7f93740a0850 2022-07-20T04:18:50.815 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f9371d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 0x7f93740a0360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:50.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f9371d9b700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 0x7f93740a0360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52482/0 (socket says 172.21.15.61:52482) 2022-07-20T04:18:50.816 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f9371d9b700 1 -- 172.21.15.61:0/1275824993 learned_addr learned my addr 172.21.15.61:0/1275824993 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:50.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.796+0000 7f937259c700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93740acef0 0x7f93740a1970 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:50.817 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9372d9d700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93740a0850 0x7f93740a0d00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:50.818 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9371d9b700 1 -- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93740a0850 msgr2=0x7f93740a0d00 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:50.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9371d9b700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93740a0850 0x7f93740a0d00 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9371d9b700 1 -- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93740acef0 msgr2=0x7f93740a1970 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:50.819 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9371d9b700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93740acef0 0x7f93740a1970 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9371d9b700 1 -- 172.21.15.61:0/1275824993 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f936800b180 con 0x7f93740b1e00 2022-07-20T04:18:50.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f9371d9b700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 0x7f93740a0360 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f9364001490 tx=0x7f9364007de0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:50.820 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9364002310 con 0x7f93740b1e00 2022-07-20T04:18:50.821 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9364002470 con 0x7f93740b1e00 2022-07-20T04:18:50.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c00b1e0 con 0x7f93740b1e00 2022-07-20T04:18:50.822 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7f9364001840 con 0x7f93740b1e00 2022-07-20T04:18:50.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93740a1350 con 0x7f93740b1e00 2022-07-20T04:18:50.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f937409ece0 con 0x7f93740b1e00 2022-07-20T04:18:50.823 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7f9364010030 con 0x7f93740b1e00 2022-07-20T04:18:50.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93637fe700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9350053ad0 0x7f9350055f90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:50.824 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7f936406bb30 con 0x7f93740b1e00 2022-07-20T04:18:50.825 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.800+0000 7f937259c700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9350053ad0 0x7f9350055f90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:50.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.804+0000 7f937259c700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9350053ad0 0x7f9350055f90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f936800b000 tx=0x7f9368006df0).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:50.826 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.808+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7f9364042cd0 con 0x7f93740b1e00 2022-07-20T04:18:50.934 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.932+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 --> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] -- mgr_command(tid 0: {"prefix": "pg dump", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f93740b04b0 con 0x7f9350053ad0 2022-07-20T04:18:50.941 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93637fe700 1 -- 172.21.15.61:0/1275824993 <== mgr.4103 v2:172.21.15.117:6800/22827 1 ==== mgr_command_reply(tid 0: 0 dumped all) v1 ==== 18+0+68110 (secure 0 0 0) 0x7f93740b04b0 con 0x7f9350053ad0 2022-07-20T04:18:50.944 INFO:teuthology.orchestra.run.smithi061.stdout: 2022-07-20T04:18:50.948 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9350053ad0 msgr2=0x7f9350055f90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:50.949 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9350053ad0 0x7f9350055f90 secure :-1 s=READY pgs=824 cs=0 l=1 rev1=1 rx=0x7f936800b000 tx=0x7f9368006df0).stop 2022-07-20T04:18:50.950 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 msgr2=0x7f93740a0360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:50.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 0x7f93740a0360 secure :-1 s=READY pgs=1579 cs=0 l=1 rev1=1 rx=0x7f9364001490 tx=0x7f9364007de0).stop 2022-07-20T04:18:50.951 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 shutdown_connections 2022-07-20T04:18:50.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7f9350053ad0 0x7f9350055f90 unknown :-1 s=CLOSED pgs=824 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.952 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7f93740a0850 0x7f93740a0d00 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.953 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7f93740b1e00 0x7f93740a0360 unknown :-1 s=CLOSED pgs=1579 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 --2- 172.21.15.61:0/1275824993 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7f93740acef0 0x7f93740a1970 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:50.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 >> 172.21.15.61:0/1275824993 conn(0x7f937405c3f0 msgr2=0x7f9374143a60 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:50.954 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 shutdown_connections 2022-07-20T04:18:50.955 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:50.944+0000 7f93789f5700 1 -- 172.21.15.61:0/1275824993 wait complete. 2022-07-20T04:18:50.955 INFO:teuthology.orchestra.run.smithi061.stderr:dumped all 2022-07-20T04:18:50.956 INFO:teuthology.orchestra.run.smithi061.stdout:{"pg_ready":true,"pg_map":{"version":2308,"stamp":"2022-07-20T04:18:50.206922+0000","last_osdmap_epoch":0,"last_pg_scan":0,"pg_stats_sum":{"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":18,"acting":18,"num_store_stats":0},"osd_stats_sum":{"up_from":0,"seq":0,"num_pgs":18,"num_osds":8,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":749595040,"kb_used":6385216,"kb_used_data":743209824,"kb_used_omap":27880,"kb_used_meta":0,"kb_avail":743209824,"statfs":{"total":767585320960,"available":761046859776,"internally_reserved":0,"allocated":761046859776,"data_stored":761046859776,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":28550097,"internal_metadata":0},"hb_peers":[],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[]},"pg_stats_delta":{"stat_sum":{"num_bytes":-2304,"num_objects":-21,"num_object_clones":0,"num_object_copies":-42,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":-21,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":-44,"num_write_kb":-13,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":-13,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":0,"acting":0,"num_store_stats":0,"stamp_delta":"8.098255"},"pg_stats":[{"pgid":"1.0","version":"0'0","reported_seq":138,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:30.647297+0000","last_change":"2022-07-20T04:18:30.647394+0000","last_active":"2022-07-20T04:18:30.647297+0000","last_peered":"2022-07-20T04:18:30.647297+0000","last_clean":"2022-07-20T04:18:30.647297+0000","last_became_active":"2022-07-20T03:07:00.763305+0000","last_became_peered":"2022-07-20T03:07:00.763305+0000","last_unstale":"2022-07-20T04:18:30.647297+0000","last_undegraded":"2022-07-20T04:18:30.647297+0000","last_fullsized":"2022-07-20T04:18:30.647297+0000","mapping_epoch":13,"log_start":"0'0","ondisk_log_start":"0'0","created":4,"last_epoch_clean":14,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:30.647296+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:30.647296+0000","last_clean_scrub_stamp":"2022-07-20T04:18:30.647296+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.3","version":"0'0","reported_seq":142,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:33.972240+0000","last_change":"2022-07-20T04:18:33.972340+0000","last_active":"2022-07-20T04:18:33.972240+0000","last_peered":"2022-07-20T04:18:33.972240+0000","last_clean":"2022-07-20T04:18:33.972240+0000","last_became_active":"2022-07-20T03:07:06.947545+0000","last_became_peered":"2022-07-20T03:07:06.947545+0000","last_unstale":"2022-07-20T04:18:33.972240+0000","last_undegraded":"2022-07-20T04:18:33.972240+0000","last_fullsized":"2022-07-20T04:18:33.972240+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:33.972239+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:33.972239+0000","last_clean_scrub_stamp":"2022-07-20T04:18:33.972239+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,2],"acting":[5,2],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.0","version":"0'0","reported_seq":135,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:31.616546+0000","last_change":"2022-07-20T04:18:31.616671+0000","last_active":"2022-07-20T04:18:31.616546+0000","last_peered":"2022-07-20T04:18:31.616546+0000","last_clean":"2022-07-20T04:18:31.616546+0000","last_became_active":"2022-07-20T03:07:06.947295+0000","last_became_peered":"2022-07-20T03:07:06.947295+0000","last_unstale":"2022-07-20T04:18:31.616546+0000","last_undegraded":"2022-07-20T04:18:31.616546+0000","last_fullsized":"2022-07-20T04:18:31.616546+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:31.616545+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:31.616545+0000","last_clean_scrub_stamp":"2022-07-20T04:18:31.616545+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,1],"acting":[7,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.1","version":"0'0","reported_seq":139,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:28.762715+0000","last_change":"2022-07-20T04:18:28.762784+0000","last_active":"2022-07-20T04:18:28.762715+0000","last_peered":"2022-07-20T04:18:28.762715+0000","last_clean":"2022-07-20T04:18:28.762715+0000","last_became_active":"2022-07-20T03:07:06.951117+0000","last_became_peered":"2022-07-20T03:07:06.951117+0000","last_unstale":"2022-07-20T04:18:28.762715+0000","last_undegraded":"2022-07-20T04:18:28.762715+0000","last_fullsized":"2022-07-20T04:18:28.762715+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:28.762715+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:28.762715+0000","last_clean_scrub_stamp":"2022-07-20T04:18:28.762715+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[2,3],"acting":[2,3],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":2,"acting_primary":2,"purged_snaps":[]},{"pgid":"2.2","version":"0'0","reported_seq":142,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:29.997207+0000","last_change":"2022-07-20T04:18:29.997306+0000","last_active":"2022-07-20T04:18:29.997207+0000","last_peered":"2022-07-20T04:18:29.997207+0000","last_clean":"2022-07-20T04:18:29.997207+0000","last_became_active":"2022-07-20T03:07:06.946278+0000","last_became_peered":"2022-07-20T03:07:06.946278+0000","last_unstale":"2022-07-20T04:18:29.997207+0000","last_undegraded":"2022-07-20T04:18:29.997207+0000","last_fullsized":"2022-07-20T04:18:29.997207+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:29.997206+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:29.997206+0000","last_clean_scrub_stamp":"2022-07-20T04:18:29.997206+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[5,1],"acting":[5,1],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":5,"acting_primary":5,"purged_snaps":[]},{"pgid":"2.4","version":"0'0","reported_seq":139,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:30.411786+0000","last_change":"2022-07-20T04:18:30.411835+0000","last_active":"2022-07-20T04:18:30.411786+0000","last_peered":"2022-07-20T04:18:30.411786+0000","last_clean":"2022-07-20T04:18:30.411786+0000","last_became_active":"2022-07-20T03:07:06.951291+0000","last_became_peered":"2022-07-20T03:07:06.951291+0000","last_unstale":"2022-07-20T04:18:30.411786+0000","last_undegraded":"2022-07-20T04:18:30.411786+0000","last_fullsized":"2022-07-20T04:18:30.411786+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:30.411786+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:30.411786+0000","last_clean_scrub_stamp":"2022-07-20T04:18:30.411786+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,0],"acting":[1,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.5","version":"0'0","reported_seq":135,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:34.705975+0000","last_change":"2022-07-20T04:18:34.706073+0000","last_active":"2022-07-20T04:18:34.705975+0000","last_peered":"2022-07-20T04:18:34.705975+0000","last_clean":"2022-07-20T04:18:34.705975+0000","last_became_active":"2022-07-20T03:07:06.942970+0000","last_became_peered":"2022-07-20T03:07:06.942970+0000","last_unstale":"2022-07-20T04:18:34.705975+0000","last_undegraded":"2022-07-20T04:18:34.705975+0000","last_fullsized":"2022-07-20T04:18:34.705975+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:34.705974+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:34.705974+0000","last_clean_scrub_stamp":"2022-07-20T04:18:34.705974+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[7,0],"acting":[7,0],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":7,"acting_primary":7,"purged_snaps":[]},{"pgid":"2.6","version":"0'0","reported_seq":139,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:31.435440+0000","last_change":"2022-07-20T04:18:31.435536+0000","last_active":"2022-07-20T04:18:31.435440+0000","last_peered":"2022-07-20T04:18:31.435440+0000","last_clean":"2022-07-20T04:18:31.435440+0000","last_became_active":"2022-07-20T03:07:06.949930+0000","last_became_peered":"2022-07-20T03:07:06.949930+0000","last_unstale":"2022-07-20T04:18:31.435440+0000","last_undegraded":"2022-07-20T04:18:31.435440+0000","last_fullsized":"2022-07-20T04:18:31.435440+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:31.435439+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:31.435439+0000","last_clean_scrub_stamp":"2022-07-20T04:18:31.435439+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[1,6],"acting":[1,6],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":1,"acting_primary":1,"purged_snaps":[]},{"pgid":"2.7","version":"0'0","reported_seq":135,"reported_epoch":152,"state":"active+clean","last_fresh":"2022-07-20T04:18:30.138198+0000","last_change":"2022-07-20T04:18:30.138294+0000","last_active":"2022-07-20T04:18:30.138198+0000","last_peered":"2022-07-20T04:18:30.138198+0000","last_clean":"2022-07-20T04:18:30.138198+0000","last_became_active":"2022-07-20T03:07:06.946555+0000","last_became_peered":"2022-07-20T03:07:06.946555+0000","last_unstale":"2022-07-20T04:18:30.138198+0000","last_undegraded":"2022-07-20T04:18:30.138198+0000","last_fullsized":"2022-07-20T04:18:30.138198+0000","mapping_epoch":16,"log_start":"0'0","ondisk_log_start":"0'0","created":16,"last_epoch_clean":17,"parent":"0.0","parent_split_bits":0,"last_scrub":"0'0","last_scrub_stamp":"2022-07-20T04:18:30.138197+0000","last_deep_scrub":"0'0","last_deep_scrub_stamp":"2022-07-20T04:18:30.138197+0000","last_clean_scrub_stamp":"2022-07-20T04:18:30.138197+0000","log_size":0,"ondisk_log_size":0,"stats_invalid":false,"dirty_stats_invalid":false,"omap_stats_invalid":false,"hitset_stats_invalid":false,"hitset_bytes_stats_invalid":false,"pin_stats_invalid":false,"manifest_stats_invalid":false,"snaptrimq_len":0,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"up":[6,7],"acting":[6,7],"avail_no_missing":[],"object_location_counts":[],"blocked_by":[],"up_primary":6,"acting_primary":6,"purged_snaps":[]}],"pool_stats":[{"poolid":2,"num_pg":8,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":16,"acting":16,"num_store_stats":0},{"poolid":1,"num_pg":1,"stat_sum":{"num_bytes":0,"num_objects":0,"num_object_clones":0,"num_object_copies":0,"num_objects_missing_on_primary":0,"num_objects_missing":0,"num_objects_degraded":0,"num_objects_misplaced":0,"num_objects_unfound":0,"num_objects_dirty":0,"num_whiteouts":0,"num_read":0,"num_read_kb":0,"num_write":0,"num_write_kb":0,"num_scrub_errors":0,"num_shallow_scrub_errors":0,"num_deep_scrub_errors":0,"num_objects_recovered":0,"num_bytes_recovered":0,"num_keys_recovered":0,"num_objects_omap":0,"num_objects_hit_set_archive":0,"num_bytes_hit_set_archive":0,"num_flush":0,"num_flush_kb":0,"num_evict":0,"num_evict_kb":0,"num_promote":0,"num_flush_mode_high":0,"num_flush_mode_low":0,"num_evict_mode_some":0,"num_evict_mode_full":0,"num_objects_pinned":0,"num_legacy_snapsets":0,"num_large_omap_objects":0,"num_objects_manifest":0,"num_omap_bytes":0,"num_omap_keys":0,"num_objects_repaired":0},"store_stats":{"total":0,"available":0,"internally_reserved":0,"allocated":0,"data_stored":0,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":0,"internal_metadata":0},"log_size":0,"ondisk_log_size":0,"up":2,"acting":2,"num_store_stats":0}],"osd_stats":[{"osd":7,"up_from":13,"seq":55834575711,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3654,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3742593,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,6],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.85699999999999998,"5min":0.83299999999999996,"15min":0.83799999999999997},"min":{"1min":0.56200000000000006,"5min":0.45900000000000002,"15min":0.45900000000000002},"max":{"1min":1.196,"5min":1.196,"15min":1.3049999999999999},"last":1.1060000000000001},{"interface":"front","average":{"1min":0.83299999999999996,"5min":0.873,"15min":0.84899999999999998},"min":{"1min":0.59599999999999997,"5min":0.59599999999999997,"15min":0.56599999999999995},"max":{"1min":1.1299999999999999,"5min":1.236,"15min":1.236},"last":0.90600000000000003}]},{"osd":1,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.872,"5min":0.83099999999999996,"15min":0.84199999999999997},"min":{"1min":0.61199999999999999,"5min":0.495,"15min":0.495},"max":{"1min":1.1679999999999999,"5min":1.1679999999999999,"15min":1.3660000000000001},"last":1.02},{"interface":"front","average":{"1min":0.878,"5min":0.87,"15min":0.874},"min":{"1min":0.64300000000000002,"5min":0.58599999999999997,"15min":0.58599999999999997},"max":{"1min":1.107,"5min":1.2110000000000001,"15min":1.248},"last":1.0780000000000001}]},{"osd":2,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85799999999999998,"15min":0.84499999999999997},"min":{"1min":0.63700000000000001,"5min":0.63700000000000001,"15min":0.60899999999999999},"max":{"1min":1.0329999999999999,"5min":1.482,"15min":1.482},"last":0.73099999999999998},{"interface":"front","average":{"1min":0.92200000000000004,"5min":0.91200000000000003,"15min":0.90900000000000003},"min":{"1min":0.69799999999999995,"5min":0.63800000000000001,"15min":0.56799999999999995},"max":{"1min":1.232,"5min":1.7969999999999999,"15min":1.7969999999999999},"last":0.67100000000000004}]},{"osd":3,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.86699999999999999,"5min":0.89100000000000001,"15min":0.872},"min":{"1min":0.67300000000000004,"5min":0.65500000000000003,"15min":0.61599999999999999},"max":{"1min":1.0249999999999999,"5min":1.413,"15min":1.413},"last":0.96899999999999997},{"interface":"front","average":{"1min":0.93000000000000005,"5min":0.90400000000000003,"15min":0.92300000000000004},"min":{"1min":0.68700000000000006,"5min":0.63400000000000001,"15min":0.63400000000000001},"max":{"1min":1.1639999999999999,"5min":1.1799999999999999,"15min":1.2769999999999999},"last":0.93300000000000005}]},{"osd":4,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.77000000000000002,"5min":0.82499999999999996,"15min":0.81599999999999995},"min":{"1min":0.54900000000000004,"5min":0.54900000000000004,"15min":0.54600000000000004},"max":{"1min":1.0109999999999999,"5min":1.181,"15min":1.181},"last":0.69999999999999996},{"interface":"front","average":{"1min":0.82599999999999996,"5min":0.86799999999999999,"15min":0.88300000000000001},"min":{"1min":0.58999999999999997,"5min":0.58999999999999997,"15min":0.58999999999999997},"max":{"1min":1.1279999999999999,"5min":1.5529999999999999,"15min":1.5529999999999999},"last":0.879}]},{"osd":5,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.83899999999999997,"5min":0.86899999999999999,"15min":0.91100000000000003},"min":{"1min":0.56899999999999995,"5min":0.56899999999999995,"15min":0.54900000000000004},"max":{"1min":1.1220000000000001,"5min":1.2370000000000001,"15min":1.3169999999999999},"last":0.80300000000000005},{"interface":"front","average":{"1min":0.91300000000000003,"5min":0.93200000000000005,"15min":0.93899999999999995},"min":{"1min":0.61699999999999999,"5min":0.61499999999999999,"15min":0.57999999999999996},"max":{"1min":1.2589999999999999,"5min":1.2589999999999999,"15min":1.415},"last":0.79100000000000004}]},{"osd":6,"last update":"Wed Jul 20 04:18:11 2022","interfaces":[{"interface":"back","average":{"1min":0.98299999999999998,"5min":0.97299999999999998,"15min":0.97299999999999998},"min":{"1min":0.66300000000000003,"5min":0.66300000000000003,"15min":0.621},"max":{"1min":1.3180000000000001,"5min":1.3180000000000001,"15min":1.3540000000000001},"last":0.99299999999999999},{"interface":"front","average":{"1min":0.92500000000000004,"5min":0.98199999999999998,"15min":0.97399999999999998},"min":{"1min":0.66200000000000003,"5min":0.66200000000000003,"15min":0.627},"max":{"1min":1.173,"5min":1.532,"15min":1.532},"last":1.0549999999999999}]}]},{"osd":6,"up_from":13,"seq":55834575712,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":4008,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4104249,"internal_metadata":0},"hb_peers":[0,1,2,3,4,5,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84099999999999997,"5min":0.86699999999999999,"15min":0.84099999999999997},"min":{"1min":0.60599999999999998,"5min":0.58999999999999997,"15min":0.56200000000000006},"max":{"1min":1.0840000000000001,"5min":1.0840000000000001,"15min":1.1799999999999999},"last":0.82499999999999996},{"interface":"front","average":{"1min":0.89500000000000002,"5min":0.83999999999999997,"15min":0.85099999999999998},"min":{"1min":0.67500000000000004,"5min":0.57299999999999995,"15min":0.57099999999999995},"max":{"1min":1.0529999999999999,"5min":1.2050000000000001,"15min":1.2050000000000001},"last":0.59299999999999997}]},{"osd":1,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.81100000000000005,"5min":0.81899999999999995,"15min":0.81000000000000005},"min":{"1min":0.626,"5min":0.59599999999999997,"15min":0.51300000000000001},"max":{"1min":1.0700000000000001,"5min":1.149,"15min":1.2030000000000001},"last":0.77700000000000002},{"interface":"front","average":{"1min":0.871,"5min":0.86699999999999999,"15min":0.86499999999999999},"min":{"1min":0.73899999999999999,"5min":0.625,"15min":0.56499999999999995},"max":{"1min":1.0389999999999999,"5min":1.1140000000000001,"15min":1.323},"last":0.61399999999999999}]},{"osd":2,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84799999999999998,"5min":0.877,"15min":0.86199999999999999},"min":{"1min":0.66400000000000003,"5min":0.60199999999999998,"15min":0.60199999999999998},"max":{"1min":1.081,"5min":1.1579999999999999,"15min":1.161},"last":0.66900000000000004},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.85099999999999998,"15min":0.84799999999999998},"min":{"1min":0.65600000000000003,"5min":0.63900000000000001,"15min":0.61599999999999999},"max":{"1min":1.1240000000000001,"5min":1.1259999999999999,"15min":1.129},"last":0.85199999999999998}]},{"osd":3,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.84199999999999997,"5min":0.875,"15min":0.86899999999999999},"min":{"1min":0.71199999999999997,"5min":0.66700000000000004,"15min":0.63100000000000001},"max":{"1min":1.0649999999999999,"5min":1.179,"15min":1.179},"last":0.86599999999999999},{"interface":"front","average":{"1min":0.88800000000000001,"5min":0.86699999999999999,"15min":0.88},"min":{"1min":0.76000000000000001,"5min":0.66800000000000004,"15min":0.63500000000000001},"max":{"1min":1.0760000000000001,"5min":1.1160000000000001,"15min":1.2110000000000001},"last":1.0109999999999999}]},{"osd":4,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.78500000000000003,"5min":0.78300000000000003,"15min":0.79700000000000004},"min":{"1min":0.55800000000000005,"5min":0.54300000000000004,"15min":0.52500000000000002},"max":{"1min":1.03,"5min":1.103,"15min":1.218},"last":0.70299999999999996},{"interface":"front","average":{"1min":0.85699999999999998,"5min":0.88400000000000001,"15min":0.878},"min":{"1min":0.56999999999999995,"5min":0.56999999999999995,"15min":0.56000000000000005},"max":{"1min":1.056,"5min":1.236,"15min":1.2969999999999999},"last":0.95499999999999996}]},{"osd":5,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.82399999999999995,"5min":0.85499999999999998,"15min":0.85299999999999998},"min":{"1min":0.65400000000000003,"5min":0.60799999999999998,"15min":0.57899999999999996},"max":{"1min":1.1499999999999999,"5min":1.204,"15min":1.2549999999999999},"last":0.91600000000000004},{"interface":"front","average":{"1min":0.93999999999999995,"5min":0.96599999999999997,"15min":0.96399999999999997},"min":{"1min":0.72799999999999998,"5min":0.72799999999999998,"15min":0.61699999999999999},"max":{"1min":1.131,"5min":1.165,"15min":1.327},"last":0.98799999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:18:03 2022","interfaces":[{"interface":"back","average":{"1min":0.92800000000000005,"5min":0.95999999999999996,"15min":0.94899999999999995},"min":{"1min":0.70999999999999996,"5min":0.63800000000000001,"15min":0.58899999999999997},"max":{"1min":1.079,"5min":1.173,"15min":1.2430000000000001},"last":0.74399999999999999},{"interface":"front","average":{"1min":0.90000000000000002,"5min":0.92500000000000004,"15min":0.93500000000000005},"min":{"1min":0.66400000000000003,"5min":0.59199999999999997,"15min":0.59199999999999997},"max":{"1min":1.2010000000000001,"5min":1.216,"15min":1.3300000000000001},"last":0.92600000000000005}]}]},{"osd":5,"up_from":13,"seq":55834575711,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":2742,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2808796,"internal_metadata":0},"hb_peers":[0,1,2,3,4,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.92600000000000005,"5min":0.89500000000000002,"15min":0.90000000000000002},"min":{"1min":0.69899999999999995,"5min":0.58999999999999997,"15min":0.48999999999999999},"max":{"1min":1.1040000000000001,"5min":1.1040000000000001,"15min":1.1990000000000001},"last":1.026},{"interface":"front","average":{"1min":0.93100000000000005,"5min":0.89800000000000002,"15min":0.89100000000000001},"min":{"1min":0.79800000000000004,"5min":0.69699999999999995,"15min":0.66200000000000003},"max":{"1min":1.05,"5min":1.139,"15min":1.2150000000000001},"last":0.91100000000000003}]},{"osd":1,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.83799999999999997,"5min":0.82499999999999996,"15min":0.83299999999999996},"min":{"1min":0.59899999999999998,"5min":0.59599999999999997,"15min":0.57899999999999996},"max":{"1min":1.208,"5min":1.208,"15min":1.46},"last":0.79200000000000004},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.83799999999999997,"15min":0.83199999999999996},"min":{"1min":0.61699999999999999,"5min":0.61699999999999999,"15min":0.56799999999999995},"max":{"1min":1.121,"5min":1.173,"15min":1.1739999999999999},"last":0.92600000000000005}]},{"osd":2,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.84999999999999998,"5min":0.86499999999999999,"15min":0.84199999999999997},"min":{"1min":0.64400000000000002,"5min":0.61499999999999999,"15min":0.56799999999999995},"max":{"1min":1.1180000000000001,"5min":1.173,"15min":1.1819999999999999},"last":0.996},{"interface":"front","average":{"1min":0.82399999999999995,"5min":0.85699999999999998,"15min":0.86399999999999999},"min":{"1min":0.63700000000000001,"5min":0.61799999999999999,"15min":0.61799999999999999},"max":{"1min":1.0700000000000001,"5min":1.198,"15min":1.198},"last":0.67600000000000005}]},{"osd":3,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.80400000000000005,"5min":0.82499999999999996,"15min":0.83899999999999997},"min":{"1min":0.60699999999999998,"5min":0.57299999999999995,"15min":0.57299999999999995},"max":{"1min":1.0569999999999999,"5min":1.1899999999999999,"15min":1.482},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.997,"5min":1.0129999999999999,"15min":1.0069999999999999},"min":{"1min":0.84399999999999997,"5min":0.76000000000000001,"15min":0.623},"max":{"1min":1.1419999999999999,"5min":1.274,"15min":1.3899999999999999},"last":0.84299999999999997}]},{"osd":4,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.88300000000000001,"5min":0.84499999999999997,"15min":0.84999999999999998},"min":{"1min":0.57299999999999995,"5min":0.52800000000000002,"15min":0.435},"max":{"1min":1.1739999999999999,"5min":1.1739999999999999,"15min":1.266},"last":0.96099999999999997},{"interface":"front","average":{"1min":0.879,"5min":0.86299999999999999,"15min":0.872},"min":{"1min":0.628,"5min":0.58099999999999996,"15min":0.54500000000000004},"max":{"1min":1.161,"5min":1.2889999999999999,"15min":1.325},"last":0.82699999999999996}]},{"osd":6,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.93700000000000006,"5min":0.91900000000000004,"15min":0.92100000000000004},"min":{"1min":0.71699999999999997,"5min":0.57899999999999996,"15min":0.57899999999999996},"max":{"1min":1.1080000000000001,"5min":1.417,"15min":1.417},"last":0.93700000000000006},{"interface":"front","average":{"1min":0.999,"5min":0.95899999999999996,"15min":0.96999999999999997},"min":{"1min":0.73699999999999999,"5min":0.59699999999999998,"15min":0.56100000000000005},"max":{"1min":1.2270000000000001,"5min":1.3879999999999999,"15min":1.452},"last":1.0629999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":1.008,"5min":0.95499999999999996,"15min":0.96999999999999997},"min":{"1min":0.71599999999999997,"5min":0.58199999999999996,"15min":0.53600000000000003},"max":{"1min":1.4350000000000001,"5min":1.4350000000000001,"15min":1.4350000000000001},"last":0.97999999999999998},{"interface":"front","average":{"1min":1.079,"5min":1.056,"15min":1.0609999999999999},"min":{"1min":0.86699999999999999,"5min":0.76000000000000001,"15min":0.60899999999999999},"max":{"1min":1.361,"5min":1.5680000000000001,"15min":1.5680000000000001},"last":1.0089999999999999}]}]},{"osd":4,"up_from":13,"seq":55834575712,"num_pgs":0,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798040,"kb_used_data":92901340,"kb_used_omap":3532,"kb_used_meta":0,"kb_avail":92901340,"statfs":{"total":95948165120,"available":95130972160,"internally_reserved":0,"allocated":95130972160,"data_stored":95130972160,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3616802,"internal_metadata":0},"hb_peers":[0,1,2,3,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.79800000000000004,"5min":0.84499999999999997,"15min":0.84899999999999998},"min":{"1min":0.62,"5min":0.58499999999999996,"15min":0.43099999999999999},"max":{"1min":0.98099999999999998,"5min":1.1279999999999999,"15min":1.1879999999999999},"last":0.86799999999999999},{"interface":"front","average":{"1min":0.86799999999999999,"5min":0.84099999999999997,"15min":0.83899999999999997},"min":{"1min":0.67200000000000004,"5min":0.56799999999999995,"15min":0.52400000000000002},"max":{"1min":1.0469999999999999,"5min":1.1759999999999999,"15min":1.1759999999999999},"last":0.98599999999999999}]},{"osd":1,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.80000000000000004,"5min":0.86299999999999999,"15min":0.85299999999999998},"min":{"1min":0.65500000000000003,"5min":0.61399999999999999,"15min":0.48899999999999999},"max":{"1min":0.95399999999999996,"5min":1.1499999999999999,"15min":1.1519999999999999},"last":1.1399999999999999},{"interface":"front","average":{"1min":0.90300000000000002,"5min":0.84699999999999998,"15min":0.84399999999999997},"min":{"1min":0.61899999999999999,"5min":0.61499999999999999,"15min":0.56899999999999995},"max":{"1min":1.123,"5min":1.153,"15min":1.1890000000000001},"last":0.95999999999999996}]},{"osd":2,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85899999999999999,"5min":0.875,"15min":0.88500000000000001},"min":{"1min":0.64400000000000002,"5min":0.63900000000000001,"15min":0.629},"max":{"1min":1.0980000000000001,"5min":1.2270000000000001,"15min":1.2270000000000001},"last":0.83999999999999997},{"interface":"front","average":{"1min":0.89700000000000002,"5min":0.88100000000000001,"15min":0.86599999999999999},"min":{"1min":0.67900000000000005,"5min":0.623,"15min":0.56299999999999994},"max":{"1min":1.133,"5min":1.1359999999999999,"15min":1.2270000000000001},"last":0.76800000000000002}]},{"osd":3,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85299999999999998,"5min":0.85499999999999998,"15min":0.85099999999999998},"min":{"1min":0.67500000000000004,"5min":0.63600000000000001,"15min":0.622},"max":{"1min":1.0489999999999999,"5min":1.2330000000000001,"15min":1.2330000000000001},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.95299999999999996,"5min":0.94799999999999995,"15min":0.93799999999999994},"min":{"1min":0.83699999999999997,"5min":0.66300000000000003,"15min":0.60499999999999998},"max":{"1min":1.111,"5min":1.264,"15min":1.79},"last":1.2010000000000001}]},{"osd":5,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.85199999999999998,"5min":0.83699999999999997,"15min":0.85499999999999998},"min":{"1min":0.59299999999999997,"5min":0.56399999999999995,"15min":0.56399999999999995},"max":{"1min":1.1020000000000001,"5min":1.2410000000000001,"15min":1.2410000000000001},"last":1.0509999999999999},{"interface":"front","average":{"1min":0.76400000000000001,"5min":0.84499999999999997,"15min":0.86099999999999999},"min":{"1min":0.625,"5min":0.56599999999999995,"15min":0.54900000000000004},"max":{"1min":0.98499999999999999,"5min":1.2749999999999999,"15min":1.2749999999999999},"last":0.93600000000000005}]},{"osd":6,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.81899999999999995,"5min":0.85999999999999999,"15min":0.878},"min":{"1min":0.61199999999999999,"5min":0.61199999999999999,"15min":0.46100000000000002},"max":{"1min":1.141,"5min":1.2769999999999999,"15min":1.2769999999999999},"last":1.0149999999999999},{"interface":"front","average":{"1min":0.877,"5min":0.93600000000000005,"15min":0.94799999999999995},"min":{"1min":0.61599999999999999,"5min":0.61599999999999999,"15min":0.61299999999999999},"max":{"1min":1.0740000000000001,"5min":1.22,"15min":1.2649999999999999},"last":1.123}]},{"osd":7,"last update":"Wed Jul 20 04:18:13 2022","interfaces":[{"interface":"back","average":{"1min":0.88700000000000001,"5min":0.93100000000000005,"15min":0.93000000000000005},"min":{"1min":0.59399999999999997,"5min":0.56100000000000005,"15min":0.56100000000000005},"max":{"1min":1.1259999999999999,"5min":1.3120000000000001,"15min":1.337},"last":1.0449999999999999},{"interface":"front","average":{"1min":0.98299999999999998,"5min":1.002,"15min":1.01},"min":{"1min":0.71899999999999997,"5min":0.71799999999999997,"15min":0.502},"max":{"1min":1.3009999999999999,"5min":1.359,"15min":1.5229999999999999},"last":1.0820000000000001}]}]},{"osd":3,"up_from":13,"seq":55834575711,"num_pgs":1,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3373,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3454456,"internal_metadata":0},"hb_peers":[0,1,2,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.58399999999999996,"15min":0.59699999999999998},"min":{"1min":0.48099999999999998,"5min":0.46600000000000003,"15min":0.44900000000000001},"max":{"1min":1.0860000000000001,"5min":1.0860000000000001,"15min":1.0860000000000001},"last":0.67900000000000005},{"interface":"front","average":{"1min":0.67600000000000005,"5min":0.67200000000000004,"15min":0.67700000000000005},"min":{"1min":0.505,"5min":0.42999999999999999,"15min":0.42999999999999999},"max":{"1min":1.026,"5min":1.026,"15min":1.095},"last":0.72699999999999998}]},{"osd":1,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":0.69099999999999995,"5min":0.69599999999999995,"15min":0.70999999999999996},"min":{"1min":0.48999999999999999,"5min":0.48499999999999999,"15min":0.44400000000000001},"max":{"1min":1.0640000000000001,"5min":1.0640000000000001,"15min":1.298},"last":0.63800000000000001},{"interface":"front","average":{"1min":0.82299999999999995,"5min":0.80300000000000005,"15min":0.81200000000000006},"min":{"1min":0.54800000000000004,"5min":0.54800000000000004,"15min":0.53100000000000003},"max":{"1min":1.196,"5min":1.196,"15min":1.196},"last":0.83699999999999997}]},{"osd":2,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":0.73199999999999998,"5min":0.755,"15min":0.75800000000000001},"min":{"1min":0.56000000000000005,"5min":0.52000000000000002,"15min":0.52000000000000002},"max":{"1min":0.871,"5min":1.127,"15min":1.2},"last":0.70799999999999996},{"interface":"front","average":{"1min":0.77800000000000002,"5min":0.76800000000000002,"15min":0.77200000000000002},"min":{"1min":0.55900000000000005,"5min":0.55300000000000005,"15min":0.51900000000000002},"max":{"1min":1.2310000000000001,"5min":1.2310000000000001,"15min":1.2310000000000001},"last":0.77000000000000002}]},{"osd":4,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":0.86299999999999999,"5min":0.86299999999999999,"15min":0.85799999999999998},"min":{"1min":0.70099999999999996,"5min":0.628,"15min":0.58499999999999996},"max":{"1min":1.103,"5min":1.1200000000000001,"15min":1.2809999999999999},"last":0.79800000000000004},{"interface":"front","average":{"1min":0.91200000000000003,"5min":0.93100000000000005,"15min":0.93700000000000006},"min":{"1min":0.73899999999999999,"5min":0.71999999999999997,"15min":0.68899999999999995},"max":{"1min":1.1100000000000001,"5min":1.262,"15min":1.544},"last":1.1519999999999999}]},{"osd":5,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":0.91800000000000004,"5min":0.93500000000000005,"15min":0.95399999999999996},"min":{"1min":0.75600000000000001,"5min":0.70399999999999996,"15min":0.70399999999999996},"max":{"1min":1.012,"5min":1.1839999999999999,"15min":1.454},"last":1.216},{"interface":"front","average":{"1min":1.016,"5min":0.97399999999999998,"15min":0.98199999999999998},"min":{"1min":0.70199999999999996,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.496,"5min":1.496,"15min":1.496},"last":0.91500000000000004}]},{"osd":6,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":1.004,"5min":0.998,"15min":1.0049999999999999},"min":{"1min":0.71399999999999997,"5min":0.69899999999999995,"15min":0.69899999999999995},"max":{"1min":1.452,"5min":1.452,"15min":1.452},"last":1.0029999999999999},{"interface":"front","average":{"1min":0.96799999999999997,"5min":0.97899999999999998,"15min":0.98699999999999999},"min":{"1min":0.79600000000000004,"5min":0.64200000000000002,"15min":0.64200000000000002},"max":{"1min":1.351,"5min":1.351,"15min":1.3660000000000001},"last":0.88}]},{"osd":7,"last update":"Wed Jul 20 04:18:29 2022","interfaces":[{"interface":"back","average":{"1min":1.022,"5min":1.0189999999999999,"15min":1.0269999999999999},"min":{"1min":0.72899999999999998,"5min":0.71799999999999997,"15min":0.71799999999999997},"max":{"1min":1.427,"5min":1.464,"15min":1.464},"last":1.0389999999999999},{"interface":"front","average":{"1min":1.03,"5min":1.0569999999999999,"15min":1.0680000000000001},"min":{"1min":0.83899999999999997,"5min":0.80800000000000005,"15min":0.69999999999999996},"max":{"1min":1.1619999999999999,"5min":1.411,"15min":1.5409999999999999},"last":1.256}]}]},{"osd":2,"up_from":13,"seq":55834575711,"num_pgs":2,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":2845,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":2913372,"internal_metadata":0},"hb_peers":[0,1,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.60799999999999998,"5min":0.61499999999999999,"15min":0.62},"min":{"1min":0.503,"5min":0.47099999999999997,"15min":0.46100000000000002},"max":{"1min":0.81899999999999995,"5min":0.91300000000000003,"15min":1.054},"last":0.80700000000000005},{"interface":"front","average":{"1min":0.68100000000000005,"5min":0.70199999999999996,"15min":0.69599999999999995},"min":{"1min":0.56299999999999994,"5min":0.47799999999999998,"15min":0.47799999999999998},"max":{"1min":0.875,"5min":1.153,"15min":1.153},"last":0.61099999999999999}]},{"osd":1,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.753,"5min":0.73499999999999999,"15min":0.72499999999999998},"min":{"1min":0.55000000000000004,"5min":0.51200000000000001,"15min":0.51200000000000001},"max":{"1min":0.97699999999999998,"5min":1.0589999999999999,"15min":1.147},"last":0.57399999999999995},{"interface":"front","average":{"1min":0.82799999999999996,"5min":0.82699999999999996,"15min":0.82899999999999996},"min":{"1min":0.70599999999999996,"5min":0.64100000000000001,"15min":0.57199999999999995},"max":{"1min":1.0229999999999999,"5min":1.173,"15min":1.173},"last":1.0289999999999999}]},{"osd":3,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.74399999999999999,"5min":0.70399999999999996,"15min":0.70099999999999996},"min":{"1min":0.57599999999999996,"5min":0.48599999999999999,"15min":0.46500000000000002},"max":{"1min":1.026,"5min":1.0840000000000001,"15min":1.1699999999999999},"last":0.59299999999999997},{"interface":"front","average":{"1min":0.80700000000000005,"5min":0.79700000000000004,"15min":0.80100000000000005},"min":{"1min":0.65300000000000002,"5min":0.57699999999999996,"15min":0.55700000000000005},"max":{"1min":1.0189999999999999,"5min":1.4550000000000001,"15min":1.4550000000000001},"last":0.97799999999999998}]},{"osd":4,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":1.028,"5min":0.95199999999999996,"15min":0.94799999999999995},"min":{"1min":0.81399999999999995,"5min":0.71699999999999997,"15min":0.64700000000000002},"max":{"1min":1.1830000000000001,"5min":1.345,"15min":1.415},"last":0.78700000000000003},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.90200000000000002,"15min":0.90100000000000002},"min":{"1min":0.78700000000000003,"5min":0.68799999999999994,"15min":0.67400000000000004},"max":{"1min":1.091,"5min":1.2110000000000001,"15min":1.3799999999999999},"last":1.0609999999999999}]},{"osd":5,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.998,"5min":0.98499999999999999,"15min":0.98699999999999999},"min":{"1min":0.76200000000000001,"5min":0.69499999999999995,"15min":0.67500000000000004},"max":{"1min":1.262,"5min":1.446,"15min":1.5669999999999999},"last":0.89000000000000001},{"interface":"front","average":{"1min":0.94099999999999995,"5min":0.92500000000000004,"15min":0.94199999999999995},"min":{"1min":0.72899999999999998,"5min":0.69199999999999995,"15min":0.67200000000000004},"max":{"1min":1.1599999999999999,"5min":1.244,"15min":1.3540000000000001},"last":0.94199999999999995}]},{"osd":6,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":0.997,"5min":0.996,"15min":1.0089999999999999},"min":{"1min":0.68600000000000005,"5min":0.68600000000000005,"15min":0.68600000000000005},"max":{"1min":1.2310000000000001,"5min":1.526,"15min":1.526},"last":1.1140000000000001},{"interface":"front","average":{"1min":1.069,"5min":1.0740000000000001,"15min":1.077},"min":{"1min":0.91200000000000003,"5min":0.77500000000000002,"15min":0.75700000000000001},"max":{"1min":1.355,"5min":1.5940000000000001,"15min":1.5940000000000001},"last":1.1859999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:18:18 2022","interfaces":[{"interface":"back","average":{"1min":1.0780000000000001,"5min":1.0800000000000001,"15min":1.081},"min":{"1min":0.75900000000000001,"5min":0.72999999999999998,"15min":0.72999999999999998},"max":{"1min":1.3160000000000001,"5min":1.6459999999999999,"15min":1.6459999999999999},"last":1.139},{"interface":"front","average":{"1min":1.0569999999999999,"5min":1.044,"15min":1.0449999999999999},"min":{"1min":0.80500000000000005,"5min":0.80500000000000005,"15min":0.71299999999999997},"max":{"1min":1.2829999999999999,"5min":1.321,"15min":1.524},"last":1.153}]}]},{"osd":1,"up_from":13,"seq":55834575711,"num_pgs":4,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3953,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":4048290,"internal_metadata":0},"hb_peers":[0,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":0,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.64100000000000001,"5min":0.61199999999999999,"15min":0.61899999999999999},"min":{"1min":0.46400000000000002,"5min":0.45800000000000002,"15min":0.45400000000000001},"max":{"1min":0.88300000000000001,"5min":0.92200000000000004,"15min":1.016},"last":0.73799999999999999},{"interface":"front","average":{"1min":0.69699999999999995,"5min":0.71899999999999997,"15min":0.71699999999999997},"min":{"1min":0.52600000000000002,"5min":0.48599999999999999,"15min":0.47199999999999998},"max":{"1min":0.94399999999999995,"5min":1.0249999999999999,"15min":1.135},"last":0.64000000000000001}]},{"osd":2,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.751,"5min":0.71599999999999997,"15min":0.71699999999999997},"min":{"1min":0.53200000000000003,"5min":0.53200000000000003,"15min":0.51600000000000001},"max":{"1min":1.044,"5min":1.0900000000000001,"15min":1.155},"last":0.56100000000000005},{"interface":"front","average":{"1min":0.72499999999999998,"5min":0.75,"15min":0.745},"min":{"1min":0.53000000000000003,"5min":0.50600000000000001,"15min":0.48499999999999999},"max":{"1min":1.0529999999999999,"5min":1.1519999999999999,"15min":1.2629999999999999},"last":0.88200000000000001}]},{"osd":3,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.78700000000000003,"5min":0.79000000000000004,"15min":0.80500000000000005},"min":{"1min":0.61799999999999999,"5min":0.59099999999999997,"15min":0.54600000000000004},"max":{"1min":0.92100000000000004,"5min":1.2450000000000001,"15min":1.347},"last":0.78600000000000003},{"interface":"front","average":{"1min":0.75800000000000001,"5min":0.75800000000000001,"15min":0.76700000000000002},"min":{"1min":0.55900000000000005,"5min":0.55500000000000005,"15min":0.51600000000000001},"max":{"1min":0.98099999999999998,"5min":1.1559999999999999,"15min":1.1890000000000001},"last":0.94499999999999995}]},{"osd":4,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.93000000000000005,"5min":0.91700000000000004,"15min":0.91900000000000004},"min":{"1min":0.74299999999999999,"5min":0.68600000000000005,"15min":0.68000000000000005},"max":{"1min":1.1830000000000001,"5min":1.2090000000000001,"15min":1.3049999999999999},"last":0.81599999999999995},{"interface":"front","average":{"1min":0.91500000000000004,"5min":0.95099999999999996,"15min":0.95899999999999996},"min":{"1min":0.745,"5min":0.73799999999999999,"15min":0.67400000000000004},"max":{"1min":1.212,"5min":1.3899999999999999,"15min":1.421},"last":1.3169999999999999}]},{"osd":5,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":0.98599999999999999,"5min":0.98699999999999999,"15min":0.98399999999999999},"min":{"1min":0.76800000000000002,"5min":0.73399999999999999,"15min":0.68999999999999995},"max":{"1min":1.266,"5min":1.3700000000000001,"15min":1.381},"last":1.0980000000000001},{"interface":"front","average":{"1min":1.0349999999999999,"5min":1.0329999999999999,"15min":1.03},"min":{"1min":0.79000000000000004,"5min":0.79000000000000004,"15min":0.755},"max":{"1min":1.248,"5min":1.3420000000000001,"15min":1.4139999999999999},"last":0.92000000000000004}]},{"osd":6,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":1.0249999999999999,"5min":1.0289999999999999,"15min":1.0409999999999999},"min":{"1min":0.79400000000000004,"5min":0.74299999999999999,"15min":0.72499999999999998},"max":{"1min":1.3180000000000001,"5min":1.3320000000000001,"15min":1.4390000000000001},"last":1.157},{"interface":"front","average":{"1min":1.0509999999999999,"5min":1.0660000000000001,"15min":1.079},"min":{"1min":0.78200000000000003,"5min":0.78200000000000003,"15min":0.77200000000000002},"max":{"1min":1.278,"5min":1.6399999999999999,"15min":1.6399999999999999},"last":1.0680000000000001}]},{"osd":7,"last update":"Wed Jul 20 04:18:10 2022","interfaces":[{"interface":"back","average":{"1min":1.04,"5min":1.0609999999999999,"15min":1.0820000000000001},"min":{"1min":0.71099999999999997,"5min":0.71099999999999997,"15min":0.71099999999999997},"max":{"1min":1.446,"5min":1.462,"15min":1.484},"last":1.375},{"interface":"front","average":{"1min":1.0980000000000001,"5min":1.0980000000000001,"15min":1.1080000000000001},"min":{"1min":0.877,"5min":0.84299999999999997,"15min":0.73299999999999998},"max":{"1min":1.2929999999999999,"5min":1.329,"15min":1.5720000000000001},"last":1.1950000000000001}]}]},{"osd":0,"up_from":13,"seq":55834575712,"num_pgs":3,"num_osds":1,"num_per_pool_osds":0,"num_per_pool_omap_osds":0,"kb":93699380,"kb_used":798168,"kb_used_data":92901212,"kb_used_omap":3771,"kb_used_meta":0,"kb_avail":92901212,"statfs":{"total":95948165120,"available":95130841088,"internally_reserved":0,"allocated":95130841088,"data_stored":95130841088,"data_compressed":0,"data_compressed_allocated":0,"data_compressed_original":0,"omap_allocated":3861539,"internal_metadata":0},"hb_peers":[1,2,3,4,5,6,7],"snap_trim_queue_len":0,"num_snap_trimming":0,"num_shards_repaired":0,"op_queue_age_hist":{"histogram":[],"upper_bound":1},"perf_stat":{"commit_latency_ms":0,"apply_latency_ms":0,"commit_latency_ns":0,"apply_latency_ns":0},"alerts":[],"network_ping_times":[{"osd":1,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.68300000000000005,"5min":0.66900000000000004,"15min":0.66700000000000004},"min":{"1min":0.42499999999999999,"5min":0.42499999999999999,"15min":0.42499999999999999},"max":{"1min":1.0620000000000001,"5min":1.0620000000000001,"15min":1.141},"last":0.54000000000000004},{"interface":"front","average":{"1min":0.76300000000000001,"5min":0.75,"15min":0.749},"min":{"1min":0.59399999999999997,"5min":0.59399999999999997,"15min":0.56699999999999995},"max":{"1min":0.99199999999999999,"5min":0.99199999999999999,"15min":1.129},"last":0.67400000000000004}]},{"osd":2,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.69699999999999995,"5min":0.71699999999999997,"15min":0.72799999999999998},"min":{"1min":0.53100000000000003,"5min":0.53100000000000003,"15min":0.496},"max":{"1min":1.091,"5min":1.091,"15min":1.2290000000000001},"last":0.74099999999999999},{"interface":"front","average":{"1min":0.72399999999999998,"5min":0.73599999999999999,"15min":0.73099999999999998},"min":{"1min":0.49099999999999999,"5min":0.49099999999999999,"15min":0.49099999999999999},"max":{"1min":0.93799999999999994,"5min":1.103,"15min":1.103},"last":0.80200000000000005}]},{"osd":3,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.66700000000000004,"5min":0.71299999999999997,"15min":0.70599999999999996},"min":{"1min":0.52200000000000002,"5min":0.50800000000000001,"15min":0.49199999999999999},"max":{"1min":0.91500000000000004,"5min":1.2689999999999999,"15min":1.2689999999999999},"last":0.77000000000000002},{"interface":"front","average":{"1min":0.84299999999999997,"5min":0.85199999999999998,"15min":0.85399999999999998},"min":{"1min":0.63300000000000001,"5min":0.63300000000000001,"15min":0.54800000000000004},"max":{"1min":1.0700000000000001,"5min":1.1020000000000001,"15min":1.2749999999999999},"last":0.84199999999999997}]},{"osd":4,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.92100000000000004,"5min":0.94199999999999995,"15min":0.93999999999999995},"min":{"1min":0.69799999999999995,"5min":0.69799999999999995,"15min":0.64400000000000002},"max":{"1min":1.1910000000000001,"5min":1.206,"15min":1.3180000000000001},"last":1.0840000000000001},{"interface":"front","average":{"1min":0.89400000000000002,"5min":0.91000000000000003,"15min":0.91800000000000004},"min":{"1min":0.71199999999999997,"5min":0.65500000000000003,"15min":0.65500000000000003},"max":{"1min":1.0920000000000001,"5min":1.286,"15min":1.3839999999999999},"last":0.876}]},{"osd":5,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.97499999999999998,"5min":1.0009999999999999,"15min":1.0269999999999999},"min":{"1min":0.746,"5min":0.746,"15min":0.64900000000000002},"max":{"1min":1.163,"5min":1.448,"15min":1.571},"last":1.0609999999999999},{"interface":"front","average":{"1min":1.0089999999999999,"5min":1.0049999999999999,"15min":1.0109999999999999},"min":{"1min":0.77300000000000002,"5min":0.65400000000000003,"15min":0.65400000000000003},"max":{"1min":1.359,"5min":1.4410000000000001,"15min":1.4410000000000001},"last":0.90900000000000003}]},{"osd":6,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":0.92000000000000004,"5min":0.96799999999999997,"15min":0.995},"min":{"1min":0.65600000000000003,"5min":0.65600000000000003,"15min":0.65600000000000003},"max":{"1min":1.1080000000000001,"5min":1.3819999999999999,"15min":1.4550000000000001},"last":0.97799999999999998},{"interface":"front","average":{"1min":1.0369999999999999,"5min":1.079,"15min":1.0900000000000001},"min":{"1min":0.754,"5min":0.754,"15min":0.71199999999999997},"max":{"1min":1.528,"5min":1.528,"15min":1.617},"last":0.98699999999999999}]},{"osd":7,"last update":"Wed Jul 20 04:18:28 2022","interfaces":[{"interface":"back","average":{"1min":1.0880000000000001,"5min":1.089,"15min":1.087},"min":{"1min":0.93100000000000005,"5min":0.80800000000000005,"15min":0.71699999999999997},"max":{"1min":1.4950000000000001,"5min":1.522,"15min":1.625},"last":1.014},{"interface":"front","average":{"1min":1.03,"5min":1.0720000000000001,"15min":1.0900000000000001},"min":{"1min":0.80100000000000005,"5min":0.74399999999999999,"15min":0.74399999999999999},"max":{"1min":1.1890000000000001,"5min":1.5109999999999999,"15min":1.571},"last":1.034}]}]}]}} 2022-07-20T04:18:50.960 DEBUG:teuthology.orchestra.run.smithi061:> sudo ceph --cluster ceph config set global mon_health_to_clog false 2022-07-20T04:18:51.071 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 Processor -- start 2022-07-20T04:18:51.072 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 -- start start 2022-07-20T04:18:51.086 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 --2- >> v2:172.21.15.117:3300/0 conn(0x7faea40667e0 0x7faea4061060 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:51.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea4061550 0x7faea4142f50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:51.087 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 -- --> v1:172.21.15.117:6789/0 -- auth(proto 0 30 bytes epoch 0) v1 -- 0x7faea40b1970 con 0x7faea40b1590 2022-07-20T04:18:51.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 -- --> v2:172.21.15.117:3300/0 -- mon_getmap magic: 0 v1 -- 0x7faea4067260 con 0x7faea40667e0 2022-07-20T04:18:51.089 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeac66f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea40b1d50 con 0x7faea4061550 2022-07-20T04:18:51.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaa40b700 1 --1- >> v1:172.21.15.117:6789/0 conn(0x7faea40b1590 0x7faea4060950 :-1 s=CONNECTING_WAIT_BANNER_AND_IDENTIFY pgs=0 cs=0 l=0).handle_server_banner_and_identify peer v1:172.21.15.117:6789/0 says I am v1:172.21.15.61:57590/0 (socket says 172.21.15.61:57590) 2022-07-20T04:18:51.090 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaa40b700 1 -- 172.21.15.61:0/4011631321 learned_addr learned my addr 172.21.15.61:0/4011631321 (peer_addr_for_me v1:172.21.15.61:0/0) 2022-07-20T04:18:51.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaac0c700 1 --2- 172.21.15.61:0/4011631321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea4061550 0x7faea4142f50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:51.092 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faea9c0a700 1 --2- 172.21.15.61:0/4011631321 >> v2:172.21.15.117:3300/0 conn(0x7faea40667e0 0x7faea4061060 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:51.093 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faea9409700 1 -- 172.21.15.61:0/4011631321 <== mon.1 v1:172.21.15.117:6789/0 1 ==== auth_reply(proto 2 0 (0) Success) v1 ==== 33+0+0 (unknown 3553382713 0 0) 0x7faea40b1970 con 0x7faea40b1590 2022-07-20T04:18:51.094 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaac0c700 1 -- 172.21.15.61:0/4011631321 >> v2:172.21.15.117:3300/0 conn(0x7faea40667e0 msgr2=0x7faea4061060 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=0).mark_down 2022-07-20T04:18:51.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaac0c700 1 --2- 172.21.15.61:0/4011631321 >> v2:172.21.15.117:3300/0 conn(0x7faea40667e0 0x7faea4061060 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.095 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaac0c700 1 -- 172.21.15.61:0/4011631321 >> v1:172.21.15.117:6789/0 conn(0x7faea40b1590 legacy=0x7faea4060950 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:51.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.072+0000 7faeaac0c700 1 -- 172.21.15.61:0/4011631321 --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faea40b1ff0 con 0x7faea4061550 2022-07-20T04:18:51.096 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeaac0c700 1 --2- 172.21.15.61:0/4011631321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea4061550 0x7faea4142f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fae9c0032f0 tx=0x7fae9c0091d0).ready entity=mon.2 client_cookie=1f8b6e10f7a0e5ec server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:51.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faea9409700 1 -- 172.21.15.61:0/4011631321 <== mon.2 v2:172.21.15.117:3301/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9c00b6c0 con 0x7faea4061550 2022-07-20T04:18:51.097 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faea9409700 1 -- 172.21.15.61:0/4011631321 <== mon.2 v2:172.21.15.117:3301/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c00b820 con 0x7faea4061550 2022-07-20T04:18:51.098 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faea9409700 1 -- 172.21.15.61:0/4011631321 <== mon.2 v2:172.21.15.117:3301/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9c00b990 con 0x7faea4061550 2022-07-20T04:18:51.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 -- 172.21.15.61:0/4011631321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea4061550 msgr2=0x7faea4142f50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:51.099 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 --2- 172.21.15.61:0/4011631321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea4061550 0x7faea4142f50 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fae9c0032f0 tx=0x7fae9c0091d0).stop 2022-07-20T04:18:51.100 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 -- 172.21.15.61:0/4011631321 shutdown_connections 2022-07-20T04:18:51.101 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 --2- 172.21.15.61:0/4011631321 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea4061550 0x7faea4142f50 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 --2- 172.21.15.61:0/4011631321 >> v2:172.21.15.117:3300/0 conn(0x7faea40667e0 0x7faea4061060 unknown :-1 s=CLOSED pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.102 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 -- 172.21.15.61:0/4011631321 >> 172.21.15.61:0/4011631321 conn(0x7faea40b2340 msgr2=0x7faea40b2720 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:51.103 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 -- 172.21.15.61:0/4011631321 shutdown_connections 2022-07-20T04:18:51.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.076+0000 7faeac66f700 1 -- 172.21.15.61:0/4011631321 wait complete. 2022-07-20T04:18:51.104 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 Processor -- start 2022-07-20T04:18:51.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 -- start start 2022-07-20T04:18:51.105 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 --2- >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faea4061550 0x7faea413ddc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:51.106 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 --2- >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea40667e0 0x7faea40ae9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 0x7faea41527c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 -- --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea4143db0 con 0x7faea40b1590 2022-07-20T04:18:51.107 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 -- --> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faea40aeec0 con 0x7faea4061550 2022-07-20T04:18:51.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 -- --> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] -- mon_getmap magic: 0 v1 -- 0x7faea40af190 con 0x7faea40667e0 2022-07-20T04:18:51.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 0x7faea41527c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:51.108 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 --2- >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 0x7faea41527c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.61:3300/0 says I am v2:172.21.15.61:52498/0 (socket says 172.21.15.61:52498) 2022-07-20T04:18:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 -- 172.21.15.61:0/1593876570 learned_addr learned my addr 172.21.15.61:0/1593876570 (peer_addr_for_me v2:172.21.15.61:0/0) 2022-07-20T04:18:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaa40b700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faea4061550 0x7faea413ddc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:51.109 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faea9c0a700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea40667e0 0x7faea40ae9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 -- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea40667e0 msgr2=0x7faea40ae9d0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:51.110 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea40667e0 0x7faea40ae9d0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 -- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faea4061550 msgr2=0x7faea413ddc0 unknown :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faea4061550 0x7faea413ddc0 unknown :-1 s=AUTH_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.111 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 -- 172.21.15.61:0/1593876570 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae9c009470 con 0x7faea40b1590 2022-07-20T04:18:51.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeaac0c700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 0x7faea41527c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7fae9c0032f0 tx=0x7fae9c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:51.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 1 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9c00b6c0 con 0x7faea40b1590 2022-07-20T04:18:51.112 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faea40af430 con 0x7faea40b1590 2022-07-20T04:18:51.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fae9c021630 con 0x7faea40b1590 2022-07-20T04:18:51.113 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faea4152280 con 0x7faea40b1590 2022-07-20T04:18:51.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.080+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 3 ==== mon_map magic: 0 v1 ==== 383+0+0 (secure 0 0 0) 0x7fae9c00b990 con 0x7faea40b1590 2022-07-20T04:18:51.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.084+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faea40aad10 con 0x7faea40b1590 2022-07-20T04:18:51.114 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.084+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 4 ==== mgrmap(e 4) v1 ==== 65964+0+0 (secure 0 0 0) 0x7fae9c027020 con 0x7faea40b1590 2022-07-20T04:18:51.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.084+0000 7fae937fe700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae84053af0 0x7fae84055fb0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2022-07-20T04:18:51.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.084+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 5 ==== osd_map(152..152 src has 1..152) v4 ==== 5339+0+0 (secure 0 0 0) 0x7fae9c06cea0 con 0x7faea40b1590 2022-07-20T04:18:51.115 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.084+0000 7faeaa40b700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae84053af0 0x7fae84055fb0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2022-07-20T04:18:51.116 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.084+0000 7faeaa40b700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae84053af0 0x7fae84055fb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7faea000d000 tx=0x7faea0007c20).ready entity=mgr.4103 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2022-07-20T04:18:51.117 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.092+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+138877 (secure 0 0 0) 0x7faea40aad10 con 0x7faea40b1590 2022-07-20T04:18:51.221 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.220+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 --> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] -- mon_command([{prefix=config set, name=mon_health_to_clog}] v 0) v1 -- 0x7faea40aad10 con 0x7faea40b1590 2022-07-20T04:18:51.229 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.228+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 7 ==== config(1 keys) v1 ==== 35+0+0 (secure 0 0 0) 0x7fae9c043fe0 con 0x7faea40b1590 2022-07-20T04:18:51.231 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.228+0000 7fae937fe700 1 -- 172.21.15.61:0/1593876570 <== mon.0 v2:172.21.15.61:3300/0 8 ==== mon_command_ack([{prefix=config set, name=mon_health_to_clog}]=0 v25)=0 v25) v1 ==== 123+0+0 (secure 0 0 0) 0x7faea40aad10 con 0x7faea40b1590 2022-07-20T04:18:51.234 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae84053af0 msgr2=0x7fae84055fb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:51.235 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae84053af0 0x7fae84055fb0 secure :-1 s=READY pgs=825 cs=0 l=1 rev1=1 rx=0x7faea000d000 tx=0x7faea0007c20).stop 2022-07-20T04:18:51.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 msgr2=0x7faea41527c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2022-07-20T04:18:51.236 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 0x7faea41527c0 secure :-1 s=READY pgs=1580 cs=0 l=1 rev1=1 rx=0x7fae9c0032f0 tx=0x7fae9c016000).stop 2022-07-20T04:18:51.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 shutdown_connections 2022-07-20T04:18:51.237 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:6800/22827,v1:172.21.15.117:6801/22827] conn(0x7fae84053af0 0x7fae84055fb0 unknown :-1 s=CLOSED pgs=825 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.238 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.61:3300/0,v1:172.21.15.61:6789/0] conn(0x7faea40b1590 0x7faea41527c0 unknown :-1 s=CLOSED pgs=1580 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3301/0,v1:172.21.15.117:6790/0] conn(0x7faea40667e0 0x7faea40ae9d0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.239 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 --2- 172.21.15.61:0/1593876570 >> [v2:172.21.15.117:3300/0,v1:172.21.15.117:6789/0] conn(0x7faea4061550 0x7faea413ddc0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2022-07-20T04:18:51.240 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 >> 172.21.15.61:0/1593876570 conn(0x7faea40b2340 msgr2=0x7faea413ed70 unknown :-1 s=STATE_NONE l=0).mark_down 2022-07-20T04:18:51.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 shutdown_connections 2022-07-20T04:18:51.241 INFO:teuthology.orchestra.run.smithi061.stderr:2022-07-20T04:18:51.232+0000 7faeac66f700 1 -- 172.21.15.61:0/1593876570 wait complete. 2022-07-20T04:18:51.248 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph.py", line 1858, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T04:18:51.252 INFO:teuthology.misc:Shutting down mds daemons... 2022-07-20T04:18:51.253 DEBUG:tasks.ceph.mds.a:waiting for process to exit 2022-07-20T04:18:51.254 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.271 INFO:tasks.ceph.mds.a:Stopped 2022-07-20T04:18:51.273 DEBUG:tasks.ceph.mds.b:waiting for process to exit 2022-07-20T04:18:51.273 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.289 INFO:tasks.ceph.mds.b:Stopped 2022-07-20T04:18:51.291 INFO:teuthology.misc:Shutting down osd daemons... 2022-07-20T04:18:51.292 DEBUG:tasks.ceph.osd.0:waiting for process to exit 2022-07-20T04:18:51.294 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.315 INFO:tasks.ceph.osd.0:Stopped 2022-07-20T04:18:51.316 DEBUG:tasks.ceph.osd.1:waiting for process to exit 2022-07-20T04:18:51.317 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.377 INFO:tasks.ceph.osd.1:Stopped 2022-07-20T04:18:51.378 DEBUG:tasks.ceph.osd.2:waiting for process to exit 2022-07-20T04:18:51.379 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.396 INFO:tasks.ceph.osd.2:Stopped 2022-07-20T04:18:51.397 DEBUG:tasks.ceph.osd.3:waiting for process to exit 2022-07-20T04:18:51.397 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.419 INFO:tasks.ceph.osd.3:Stopped 2022-07-20T04:18:51.420 DEBUG:tasks.ceph.osd.4:waiting for process to exit 2022-07-20T04:18:51.421 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.440 INFO:tasks.ceph.osd.4:Stopped 2022-07-20T04:18:51.441 DEBUG:tasks.ceph.osd.5:waiting for process to exit 2022-07-20T04:18:51.442 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.463 INFO:tasks.ceph.osd.5:Stopped 2022-07-20T04:18:51.464 DEBUG:tasks.ceph.osd.6:waiting for process to exit 2022-07-20T04:18:51.465 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.485 INFO:tasks.ceph.osd.6:Stopped 2022-07-20T04:18:51.486 DEBUG:tasks.ceph.osd.7:waiting for process to exit 2022-07-20T04:18:51.486 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.503 INFO:tasks.ceph.osd.7:Stopped 2022-07-20T04:18:51.504 INFO:teuthology.misc:Shutting down mgr daemons... 2022-07-20T04:18:51.504 DEBUG:tasks.ceph.mgr.y:waiting for process to exit 2022-07-20T04:18:51.505 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.535 INFO:tasks.ceph.mgr.y:Stopped 2022-07-20T04:18:51.536 DEBUG:tasks.ceph.mgr.x:waiting for process to exit 2022-07-20T04:18:51.537 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.574 INFO:tasks.ceph.mgr.x:Stopped 2022-07-20T04:18:51.575 INFO:teuthology.misc:Shutting down mon daemons... 2022-07-20T04:18:51.576 DEBUG:tasks.ceph.mon.a:waiting for process to exit 2022-07-20T04:18:51.577 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.605 INFO:tasks.ceph.mon.a:Stopped 2022-07-20T04:18:51.606 DEBUG:tasks.ceph.mon.b:waiting for process to exit 2022-07-20T04:18:51.607 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.632 INFO:tasks.ceph.mon.b:Stopped 2022-07-20T04:18:51.634 DEBUG:tasks.ceph.mon.c:waiting for process to exit 2022-07-20T04:18:51.636 INFO:teuthology.orchestra.run:waiting for 300 2022-07-20T04:18:51.654 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 is failed for ~0s 2022-07-20T04:18:51.655 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 is failed for ~0s 2022-07-20T04:18:51.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 is failed for ~0s 2022-07-20T04:18:51.656 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 is failed for ~0s 2022-07-20T04:18:51.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 is failed for ~0s 2022-07-20T04:18:51.657 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b is failed for ~0s 2022-07-20T04:18:51.658 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c is failed for ~0s 2022-07-20T04:18:51.659 INFO:tasks.ceph.mon.c:Stopped 2022-07-20T04:18:51.661 INFO:tasks.ceph:Checking cluster log for badness... 2022-07-20T04:18:51.662 DEBUG:teuthology.orchestra.run.smithi061:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | egrep -v 'evicting unresponsive client' | egrep -v POOL_APP_NOT_ENABLED | egrep -v 'has not responded to cap revoke by MDS for over' | egrep -v MDS_CLIENT_LATE_RELEASE | egrep -v 'responding to mclientcaps' | head -n 1 2022-07-20T04:18:51.685 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-0 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T04:18:51.687 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-0 2022-07-20T04:18:51.968 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-1 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T04:18:51.970 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-1 2022-07-20T04:18:52.295 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-2 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T04:18:52.296 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-2 2022-07-20T04:18:52.476 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-3 on ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T04:18:52.477 DEBUG:teuthology.orchestra.run.smithi061:> sync && sudo umount -f /var/lib/ceph/osd/ceph-3 2022-07-20T04:18:52.650 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-4 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T04:18:52.651 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-4 2022-07-20T04:18:52.765 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-5 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T04:18:52.766 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-5 2022-07-20T04:18:53.057 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-6 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T04:18:53.059 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-6 2022-07-20T04:18:53.241 INFO:tasks.ceph:Unmounting /var/lib/ceph/osd/ceph-7 on ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T04:18:53.242 DEBUG:teuthology.orchestra.run.smithi117:> sync && sudo umount -f /var/lib/ceph/osd/ceph-7 2022-07-20T04:18:53.416 INFO:tasks.ceph:Archiving mon data... 2022-07-20T04:18:53.422 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/mon/ceph-a to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/data/mon.a.tgz 2022-07-20T04:18:53.423 DEBUG:teuthology.orchestra.run.smithi061:> mktemp 2022-07-20T04:18:53.431 INFO:teuthology.orchestra.run.smithi061.stdout:/tmp/tmp.yoCQmRVfEa 2022-07-20T04:18:53.432 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-a -- . > /tmp/tmp.yoCQmRVfEa 2022-07-20T04:18:53.661 DEBUG:teuthology.orchestra.run.smithi061:> sudo chmod 0666 /tmp/tmp.yoCQmRVfEa 2022-07-20T04:18:53.731 DEBUG:teuthology.orchestra.remote:smithi061:/tmp/tmp.yoCQmRVfEa is 568KB 2022-07-20T04:18:53.808 DEBUG:teuthology.orchestra.run.smithi061:> rm -fr /tmp/tmp.yoCQmRVfEa 2022-07-20T04:18:53.817 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-b to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/data/mon.b.tgz 2022-07-20T04:18:53.819 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T04:18:53.827 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.uq0zdzZxEr 2022-07-20T04:18:53.828 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-b -- . > /tmp/tmp.uq0zdzZxEr 2022-07-20T04:18:54.106 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.uq0zdzZxEr 2022-07-20T04:18:54.130 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.uq0zdzZxEr is 684KB 2022-07-20T04:18:54.198 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.uq0zdzZxEr 2022-07-20T04:18:54.207 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/mon/ceph-c to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/data/mon.c.tgz 2022-07-20T04:18:54.208 DEBUG:teuthology.orchestra.run.smithi117:> mktemp 2022-07-20T04:18:54.256 INFO:teuthology.orchestra.run.smithi117.stdout:/tmp/tmp.U9DRdJi9qV 2022-07-20T04:18:54.257 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/mon/ceph-c -- . > /tmp/tmp.U9DRdJi9qV 2022-07-20T04:18:54.497 DEBUG:teuthology.orchestra.run.smithi117:> sudo chmod 0666 /tmp/tmp.U9DRdJi9qV 2022-07-20T04:18:54.565 DEBUG:teuthology.orchestra.remote:smithi117:/tmp/tmp.U9DRdJi9qV is 684KB 2022-07-20T04:18:54.636 DEBUG:teuthology.orchestra.run.smithi117:> rm -fr /tmp/tmp.U9DRdJi9qV 2022-07-20T04:18:54.644 INFO:tasks.ceph:Cleaning ceph cluster... 2022-07-20T04:18:54.645 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T04:18:54.648 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T04:18:54.651 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T04:18:54.654 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T04:18:54.658 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -rf -- /etc/ceph/ceph.conf /etc/ceph/ceph.keyring /home/ubuntu/cephtest/ceph.data /home/ubuntu/cephtest/ceph.monmap /home/ubuntu/cephtest/../*.pid 2022-07-20T04:18:54.675 INFO:tasks.ceph:Checking for errors in any valgrind logs... 2022-07-20T04:18:54.676 DEBUG:teuthology.orchestra.run.smithi061:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T04:18:54.680 DEBUG:teuthology.orchestra.run.smithi117:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T04:18:54.701 INFO:teuthology.orchestra.run.smithi061.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T04:18:54.705 DEBUG:teuthology.orchestra.run.smithi124:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T04:18:54.708 DEBUG:teuthology.orchestra.run.smithi163:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T04:18:54.714 DEBUG:teuthology.orchestra.run.smithi179:> sudo zgrep '' /var/log/ceph/valgrind/* /dev/null | sort | uniq 2022-07-20T04:18:54.728 INFO:teuthology.orchestra.run.smithi117.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T04:18:54.733 INFO:teuthology.orchestra.run.smithi124.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T04:18:54.753 INFO:teuthology.orchestra.run.smithi179.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T04:18:54.820 INFO:teuthology.orchestra.run.smithi163.stderr:gzip: /var/log/ceph/valgrind/*.gz: No such file or directory 2022-07-20T04:18:54.827 INFO:tasks.ceph:Archiving crash dumps... 2022-07-20T04:18:54.830 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi061/crash 2022-07-20T04:18:54.832 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T04:18:54.854 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi117/crash 2022-07-20T04:18:54.856 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T04:18:54.878 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi124/crash 2022-07-20T04:18:54.880 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T04:18:54.903 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi163/crash 2022-07-20T04:18:54.905 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T04:18:54.927 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/lib/ceph/crash to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi179/crash 2022-07-20T04:18:54.929 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/lib/ceph/crash -- . 2022-07-20T04:18:54.951 INFO:tasks.ceph:Shutting down logrotate 2022-07-20T04:18:54.952 DEBUG:teuthology.orchestra.run.smithi061:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:54.955 DEBUG:teuthology.orchestra.run.smithi117:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:54.960 DEBUG:teuthology.orchestra.run.smithi124:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:54.967 DEBUG:teuthology.orchestra.run.smithi163:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:54.970 DEBUG:teuthology.orchestra.run.smithi179:> sudo logrotate /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:55.015 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:55.037 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:55.054 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:55.069 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:55.084 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm /etc/logrotate.d/ceph-test.conf 2022-07-20T04:18:55.102 INFO:tasks.ceph:Compressing logs... 2022-07-20T04:18:55.103 DEBUG:teuthology.orchestra.run.smithi061:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:18:55.107 DEBUG:teuthology.orchestra.run.smithi117:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:18:55.112 DEBUG:teuthology.orchestra.run.smithi124:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:18:55.115 DEBUG:teuthology.orchestra.run.smithi163:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:18:55.119 DEBUG:teuthology.orchestra.run.smithi179:> sudo find /var/log/ceph -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:18:56.660 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.3 has been restored 2022-07-20T04:18:56.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.4 has been restored 2022-07-20T04:18:56.661 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.5 has been restored 2022-07-20T04:18:56.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.6 has been restored 2022-07-20T04:18:56.662 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.osd.7 has been restored 2022-07-20T04:18:56.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.b has been restored 2022-07-20T04:18:56.663 INFO:tasks.daemonwatchdog.daemon_watchdog:daemon ceph.mon.c has been restored 2022-07-20T04:19:04.271 INFO:tasks.ceph:Archiving logs... 2022-07-20T04:19:04.272 DEBUG:teuthology.misc:Transferring archived files from smithi061:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi061/log 2022-07-20T04:19:04.274 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T04:19:06.673 DEBUG:teuthology.misc:Transferring archived files from smithi117:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi117/log 2022-07-20T04:19:06.676 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T04:19:08.537 DEBUG:teuthology.misc:Transferring archived files from smithi124:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi124/log 2022-07-20T04:19:08.539 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T04:19:08.958 DEBUG:teuthology.misc:Transferring archived files from smithi163:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi163/log 2022-07-20T04:19:08.960 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T04:19:09.001 DEBUG:teuthology.misc:Transferring archived files from smithi179:/var/log/ceph to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi179/log 2022-07-20T04:19:09.003 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /var/log/ceph -- . 2022-07-20T04:19:09.029 DEBUG:teuthology.run_tasks:Unwinding manager install 2022-07-20T04:19:09.043 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 747, in recv_stderr out = self.in_stderr_buffer.read(nbytes, self.timeout) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/buffered_pipe.py", line 164, in read raise PipeTimeout() paramiko.buffered_pipe.PipeTimeout During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/contextutil.py", line 33, in nested yield vars File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/task/install/__init__.py", line 619, in task yield File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/run_tasks.py", line 188, in run_tasks suppress = manager.__exit__(*exc_info) File "/usr/lib/python3.6/contextlib.py", line 88, in __exit__ next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/ceph_fuse.py", line 160, in task mount.umount_wait() File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 326, in umount_wait if not (self.is_mounted() and self.fuse_daemon): File "/home/teuthworker/src/github.com_ceph_ceph-c_57544da114dbbe1c44f27b2ea18e3233e4e917e0/qa/tasks/cephfs/fuse_mount.py", line 205, in is_mounted proc.wait() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 148, in wait greenlet.get(block=True,timeout=60) File "src/gevent/greenlet.py", line 803, in gevent._gevent_cgreenlet.Greenlet.get File "src/gevent/greenlet.py", line 371, in gevent._gevent_cgreenlet.Greenlet._raise_exception File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/gevent/_compat.py", line 65, in reraise raise value.with_traceback(tb) File "src/gevent/greenlet.py", line 906, in gevent._gevent_cgreenlet.Greenlet.run File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 323, in copy_file_to copy_to_log(src, logger, capture=stream, quiet=quiet) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/teuthology/orchestra/run.py", line 276, in copy_to_log for line in f: File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 125, in __next__ line = self.readline() File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/file.py", line 291, in readline new_data = self._read(n) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 1376, in _read return self.channel.recv_stderr(size) File "/home/teuthworker/src/git.ceph.com_git_teuthology_89d874d555f56b1c1f383df72487aecec0ee6cf9/virtualenv/lib/python3.6/site-packages/paramiko/channel.py", line 749, in recv_stderr raise socket.timeout() socket.timeout 2022-07-20T04:19:09.047 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits... 2022-07-20T04:19:09.048 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T04:19:09.051 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T04:19:09.054 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T04:19:09.057 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T04:19:09.060 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits 2022-07-20T04:19:09.227 DEBUG:teuthology.run_tasks:Unwinding manager clock 2022-07-20T04:19:09.240 INFO:teuthology.task.clock:Checking final clock skew... 2022-07-20T04:19:09.241 DEBUG:teuthology.orchestra.run.smithi061:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T04:19:09.245 DEBUG:teuthology.orchestra.run.smithi117:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T04:19:09.248 DEBUG:teuthology.orchestra.run.smithi124:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T04:19:09.252 DEBUG:teuthology.orchestra.run.smithi163:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T04:19:09.256 DEBUG:teuthology.orchestra.run.smithi179:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2022-07-20T04:19:09.269 INFO:teuthology.orchestra.run.smithi061.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T04:19:09.269 INFO:teuthology.orchestra.run.smithi061.stdout:============================================================================== 2022-07-20T04:19:09.270 INFO:teuthology.orchestra.run.smithi061.stdout:+hv01.front.sepi 216.240.36.24 3 u 90 128 377 0.140 2.345 0.606 2022-07-20T04:19:09.270 INFO:teuthology.orchestra.run.smithi061.stdout:*hv02.front.sepi 163.237.218.18 2 u 95 128 377 0.137 -4.451 0.627 2022-07-20T04:19:09.271 INFO:teuthology.orchestra.run.smithi061.stdout:+hv03.front.sepi 216.229.4.69 3 u 84 128 377 0.190 -3.774 0.527 2022-07-20T04:19:09.271 INFO:teuthology.orchestra.run.smithi061.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T04:19:09.272 INFO:teuthology.orchestra.run.smithi117.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T04:19:09.273 INFO:teuthology.orchestra.run.smithi117.stdout:============================================================================== 2022-07-20T04:19:09.273 INFO:teuthology.orchestra.run.smithi117.stdout:+hv01.front.sepi 216.240.36.24 3 u 3 128 377 0.157 9.966 0.658 2022-07-20T04:19:09.273 INFO:teuthology.orchestra.run.smithi117.stdout:*hv02.front.sepi 163.237.218.18 2 u 35 128 377 0.161 4.395 0.557 2022-07-20T04:19:09.274 INFO:teuthology.orchestra.run.smithi117.stdout:+hv03.front.sepi 216.229.4.69 3 u 24 128 377 0.196 4.255 0.579 2022-07-20T04:19:09.274 INFO:teuthology.orchestra.run.smithi117.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T04:19:09.275 INFO:teuthology.orchestra.run.smithi124.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T04:19:09.276 INFO:teuthology.orchestra.run.smithi124.stdout:============================================================================== 2022-07-20T04:19:09.276 INFO:teuthology.orchestra.run.smithi124.stdout:+hv01.front.sepi 216.240.36.24 3 u 40 128 377 0.172 0.901 0.433 2022-07-20T04:19:09.276 INFO:teuthology.orchestra.run.smithi124.stdout:*hv02.front.sepi 163.237.218.18 2 u 29 128 377 0.190 -5.346 0.780 2022-07-20T04:19:09.277 INFO:teuthology.orchestra.run.smithi124.stdout:+hv03.front.sepi 216.229.4.69 3 u 10 128 377 0.175 -4.575 0.416 2022-07-20T04:19:09.277 INFO:teuthology.orchestra.run.smithi124.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T04:19:09.278 INFO:teuthology.orchestra.run.smithi163.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T04:19:09.278 INFO:teuthology.orchestra.run.smithi163.stdout:============================================================================== 2022-07-20T04:19:09.278 INFO:teuthology.orchestra.run.smithi163.stdout:+hv01.front.sepi 216.240.36.24 3 u 110 128 377 0.186 3.051 0.684 2022-07-20T04:19:09.279 INFO:teuthology.orchestra.run.smithi163.stdout:*hv02.front.sepi 163.237.218.18 2 u 19 128 377 0.185 -2.383 0.341 2022-07-20T04:19:09.279 INFO:teuthology.orchestra.run.smithi163.stdout:+hv03.front.sepi 216.229.4.69 3 u 92 128 377 0.202 -2.904 0.544 2022-07-20T04:19:09.279 INFO:teuthology.orchestra.run.smithi163.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T04:19:09.281 INFO:teuthology.orchestra.run.smithi179.stdout: remote refid st t when poll reach delay offset jitter 2022-07-20T04:19:09.281 INFO:teuthology.orchestra.run.smithi179.stdout:============================================================================== 2022-07-20T04:19:09.282 INFO:teuthology.orchestra.run.smithi179.stdout:+hv01.front.sepi 216.240.36.24 3 u 87 256 377 0.146 -0.260 0.934 2022-07-20T04:19:09.282 INFO:teuthology.orchestra.run.smithi179.stdout:*hv02.front.sepi 163.237.218.18 2 u 96 256 377 0.114 -5.767 0.633 2022-07-20T04:19:09.282 INFO:teuthology.orchestra.run.smithi179.stdout:+hv03.front.sepi 216.229.4.69 3 u 227 256 377 0.170 -4.833 1.161 2022-07-20T04:19:09.283 INFO:teuthology.orchestra.run.smithi179.stdout: hv04.front.sepi .INIT. 16 u - 1024 0 0.000 0.000 0.000 2022-07-20T04:19:09.284 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2022-07-20T04:19:09.298 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2022-07-20T04:19:09.300 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2022-07-20T04:19:09.312 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2022-07-20T04:19:09.323 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2022-07-20T04:19:09.335 INFO:teuthology.task.internal:Duration was 4814.515593 seconds 2022-07-20T04:19:09.337 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2022-07-20T04:19:09.353 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2022-07-20T04:19:09.354 DEBUG:teuthology.orchestra.run.smithi061:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T04:19:09.358 DEBUG:teuthology.orchestra.run.smithi117:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T04:19:09.361 DEBUG:teuthology.orchestra.run.smithi124:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T04:19:09.365 DEBUG:teuthology.orchestra.run.smithi163:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T04:19:09.368 DEBUG:teuthology.orchestra.run.smithi179:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2022-07-20T04:19:09.423 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2022-07-20T04:19:09.424 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi061.front.sepia.ceph.com 2022-07-20T04:19:09.425 DEBUG:teuthology.orchestra.run.smithi061:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T04:19:09.443 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi117.front.sepia.ceph.com 2022-07-20T04:19:09.445 DEBUG:teuthology.orchestra.run.smithi117:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T04:19:09.461 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi124.front.sepia.ceph.com 2022-07-20T04:19:09.463 DEBUG:teuthology.orchestra.run.smithi124:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T04:19:09.480 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi163.front.sepia.ceph.com 2022-07-20T04:19:09.482 DEBUG:teuthology.orchestra.run.smithi163:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T04:19:09.500 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi179.front.sepia.ceph.com 2022-07-20T04:19:09.501 DEBUG:teuthology.orchestra.run.smithi179:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2022-07-20T04:19:09.519 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2022-07-20T04:19:09.520 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:19:09.528 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:19:09.531 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:19:09.536 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:19:09.539 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2022-07-20T04:19:09.599 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2022-07-20T04:19:09.600 DEBUG:teuthology.orchestra.run.smithi061:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T04:19:09.604 DEBUG:teuthology.orchestra.run.smithi117:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T04:19:09.609 DEBUG:teuthology.orchestra.run.smithi124:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T04:19:09.615 DEBUG:teuthology.orchestra.run.smithi163:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T04:19:09.619 DEBUG:teuthology.orchestra.run.smithi179:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2022-07-20T04:19:09.750 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2022-07-20T04:19:09.761 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2022-07-20T04:19:09.763 DEBUG:teuthology.orchestra.run.smithi061:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T04:19:09.780 DEBUG:teuthology.orchestra.run.smithi117:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T04:19:09.798 DEBUG:teuthology.orchestra.run.smithi124:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T04:19:09.814 DEBUG:teuthology.orchestra.run.smithi163:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T04:19:09.832 DEBUG:teuthology.orchestra.run.smithi179:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2022-07-20T04:19:09.853 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2022-07-20T04:19:09.872 DEBUG:teuthology.orchestra.run.smithi061:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.876 DEBUG:teuthology.orchestra.run.smithi117:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.878 DEBUG:teuthology.orchestra.run.smithi124:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.881 DEBUG:teuthology.orchestra.run.smithi163:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.885 DEBUG:teuthology.orchestra.run.smithi179:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.888 INFO:teuthology.orchestra.run.smithi061.stdout:kernel.core_pattern = core 2022-07-20T04:19:09.890 INFO:teuthology.orchestra.run.smithi117.stdout:kernel.core_pattern = core 2022-07-20T04:19:09.894 INFO:teuthology.orchestra.run.smithi124.stdout:kernel.core_pattern = core 2022-07-20T04:19:09.898 INFO:teuthology.orchestra.run.smithi163.stdout:kernel.core_pattern = core 2022-07-20T04:19:09.910 INFO:teuthology.orchestra.run.smithi179.stdout:kernel.core_pattern = core 2022-07-20T04:19:09.926 DEBUG:teuthology.orchestra.run.smithi061:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.933 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:19:09.934 DEBUG:teuthology.orchestra.run.smithi117:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.970 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:19:09.972 DEBUG:teuthology.orchestra.run.smithi124:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.978 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:19:09.979 DEBUG:teuthology.orchestra.run.smithi163:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.985 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:19:09.987 DEBUG:teuthology.orchestra.run.smithi179:> test -e /home/ubuntu/cephtest/archive/coredump 2022-07-20T04:19:09.993 DEBUG:teuthology.orchestra.run:got remote process result: 1 2022-07-20T04:19:09.995 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2022-07-20T04:19:10.006 INFO:teuthology.task.internal:Transferring archived files... 2022-07-20T04:19:10.008 DEBUG:teuthology.misc:Transferring archived files from smithi061:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi061 2022-07-20T04:19:10.008 DEBUG:teuthology.orchestra.run.smithi061:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T04:19:10.053 DEBUG:teuthology.misc:Transferring archived files from smithi117:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi117 2022-07-20T04:19:10.054 DEBUG:teuthology.orchestra.run.smithi117:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T04:19:10.099 DEBUG:teuthology.misc:Transferring archived files from smithi124:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi124 2022-07-20T04:19:10.100 DEBUG:teuthology.orchestra.run.smithi124:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T04:19:10.137 DEBUG:teuthology.misc:Transferring archived files from smithi163:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi163 2022-07-20T04:19:10.138 DEBUG:teuthology.orchestra.run.smithi163:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T04:19:10.177 DEBUG:teuthology.misc:Transferring archived files from smithi179:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2022-07-19_23:28:53-fs-octopus-release-distro-default-smithi/6939436/remote/smithi179 2022-07-20T04:19:10.177 DEBUG:teuthology.orchestra.run.smithi179:> sudo tar cz -f - -C /home/ubuntu/cephtest/archive -- . 2022-07-20T04:19:10.224 INFO:teuthology.task.internal:Removing archive directory... 2022-07-20T04:19:10.225 DEBUG:teuthology.orchestra.run.smithi061:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T04:19:10.229 DEBUG:teuthology.orchestra.run.smithi117:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T04:19:10.232 DEBUG:teuthology.orchestra.run.smithi124:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T04:19:10.235 DEBUG:teuthology.orchestra.run.smithi163:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T04:19:10.237 DEBUG:teuthology.orchestra.run.smithi179:> rm -rf -- /home/ubuntu/cephtest/archive 2022-07-20T04:19:10.249 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2022-07-20T04:19:10.263 INFO:teuthology.task.internal:Not uploading archives. 2022-07-20T04:19:10.264 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2022-07-20T04:19:10.277 INFO:teuthology.task.internal:Tidying up after the test... 2022-07-20T04:19:10.279 DEBUG:teuthology.orchestra.run.smithi061:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T04:19:10.283 DEBUG:teuthology.orchestra.run.smithi117:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T04:19:10.287 DEBUG:teuthology.orchestra.run.smithi124:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T04:19:10.291 INFO:teuthology.orchestra.run.smithi061.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 04:19 /home/ubuntu/cephtest 2022-07-20T04:19:10.294 INFO:teuthology.orchestra.run.smithi117.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 04:19 /home/ubuntu/cephtest 2022-07-20T04:19:10.294 DEBUG:teuthology.orchestra.run.smithi163:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T04:19:10.298 DEBUG:teuthology.orchestra.run.smithi179:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2022-07-20T04:19:10.303 INFO:teuthology.orchestra.run.smithi163.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 04:19 /home/ubuntu/cephtest 2022-07-20T04:19:10.306 INFO:teuthology.orchestra.run.smithi179.stdout: 524316 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jul 20 04:19 /home/ubuntu/cephtest 2022-07-20T14:59:09.434 DEBUG:teuthology.exit:Got signal 15; running 1 handler... 2022-07-20T14:59:09.466 DEBUG:teuthology.task.console_log:Killing console logger for smithi061 2022-07-20T14:59:09.468 DEBUG:teuthology.task.console_log:Killing console logger for smithi117 2022-07-20T14:59:09.468 DEBUG:teuthology.task.console_log:Killing console logger for smithi124 2022-07-20T14:59:09.469 DEBUG:teuthology.task.console_log:Killing console logger for smithi163 2022-07-20T14:59:09.470 DEBUG:teuthology.task.console_log:Killing console logger for smithi179 2022-07-20T14:59:09.471 DEBUG:teuthology.exit:Finished running handlers